-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Plugins, & getting Lektor out of the package management business #1087
Comments
I have no strong opinions here. Maybe it's worth checking if pip can be instructed these days to install into a local path for lektor. |
While I agree to remove the |
Here's a somewhat meandering brain dump... Background on "The Troubles"The headaches with the current system revolve around the issue that there seems to be no simple, stable, clean way to manage dependencies in what is, essentially, a stacked, second We start with Lektor already running, then install some more packages into a separate local package cache, then add that local package library to But what about the dependencies of those local packages? Many of them may already be installed in Lektor's environment. Ideally, we don't want to install them again — if we do one of the copies is going to be ignored. What if there are version conflicts among the dependencies? None of the package management libraries or tools (e.g. Other IssuesFor quite a while, Lektor's package management system is (appropriately) simple, with the result that it is not very flexible. How We Got HereI suspect a lot of the motivation for getting Lektor into the package management business in the first place was the desire to be able to ship Lektor as a self-contained, packaged app. Initially, Lektor did ship a GUI app for macOS (I think?), but we've since given up on that, at least for now. Is There a Reason to Stay in the Package Management Business?Use Cases: Per-Project vs Shared Lektor InstallationPer-Project Lektor InstallationMy workflow for Lektor projects (i.e. for each website I author using Lektor) involves a per-project virtualenv into which (a specific version) of Lektor is installed (along with perhaps other dependencies). In that case, I really see no reason to use Lektor's dependency management. Plugins can just as well be installed using whatever package management system I'm using to set up that per-project virtualenv. (This works, today. Just don't configure any plugins in the The advantages of this method are likely apparent to any Python developer. One gets all the features of whatever package-management tool one chooses to use. (E.g. poetry, with its lock files for reproducible builds and command-line sugar for adjusting dependencies.) Shared Lektor InstallationThere is still an alternative workflow that involves using a shared installation Lektor, wherein a single installation of Lektor is used to build multiple Lektor projects. That Lektor might be installed globally as part of an OS distribution, or it might be installed on a per-user basis using QuestionsSo, the question is, who are our users? Do we have users for whom the added requirement to use an external dependency manager (e.g. Poetry) would be particularly onerous? I don't know the answer to that, but my guess is probably not. Most Lektor projects in the wild, I would guess, have someone involved who is capable of writing the HTML templates, CSS, and JS for the site, as well as writing any project-local custom Lektor plugins, etc. In any case, I would argue that for all but the simplest of Lektor projects, the build-reproducibility that comes from using a per-project virtualenv outweighs any additional effort that might be required. My PreferenceAs mentioned above, it's already possible to use your dependency-management tool of choice to manage a per-project virtualenv into which Lektor as well as all the project's required plugins are installed. No changes are required to permit this.
|
If I understand what you are suggesting, wouldn't that lock the user into a particular dependency manager? Poetry has |
If we retain the ability for Lektor to install distributions from Note that if we relaxed Lektor's requirement on what distribution name a plugin has to come from, installing local plugins into a project-local virtualenv becomes pretty simple using just about any package management tool. One needs just declare the necessary |
IMO it is not necessary to require the local plugin to be a distribution(with a setup.py or other metadata description). It should only expose an entry file, like Yeah, that is very different from the current local |
Your confusion here is understandable! My mind slipped - I meant |
I think I like the idea of being able to use local plugins that aren't distributions. It makes sense to me that you should just be able to supply just a single script if you aren't planning on publishing the plugin. If you are publishing it, then you are probably fine using a package manager and can let Lektor find the plugin as installed by the package manager, whether local or not. If local plugins aren't distributions, how are they detected? I am in favor of using Plugins that are distributed, not local, I don't think have to require a naming scheme though, as long as the entry points are discoverable. |
While I like the idea of getting rid of lektor managing its own plugins and just using a normal package manager instead, I am not sure I that this warrants the complexity building it's own plugin discovery and loading mechanism to allow plugins that are not distributable. For two reasons: 1) Building an installable plugin is really simple (and can be even simpler for other package managers) and 2) the progression from an installable plugin to one that can be distributed through pypi is very simple / clear and something I think worth something to us as a community. For example, to have a simple plugin 'date' that just adds some date functions to the ninja environment looks like this: ❯ pwd
/path/to/lektor-project/packages/date
❯ find .
./setup.py
./lektor_date.py
❯ cat setup.py
from setuptools import setup
setup(
name='lektor-date',
version='0.1',
author=u'Martin Häcker',
author_email='my@email',
license='MIT',
py_modules=['lektor_date'],
url='https://github.com/dwt/lektor-date',
entry_points={
'lektor.plugins': [
'date = lektor_date:DatePlugin',
]
}
)
❯ cat lektor_date.py
from datetime import datetime, date
from lektor.pluginsystem import Plugin
class DatePlugin(Plugin):
name = u'Lektor Date plugin'
description = 'Add date objects to your jinja templates.'
def on_setup_env(self, **extra):
self.env.jinja_env.globals.update(
date=date, datetime=datetime,
)
And for sure, that Some documentation might go a long way - and I would be happy to provide some of it. Parting thought: I think it is easier to document how to build installable plugins very simply, than building a custom plugin loader. That would still allow us to get rid of the custom pip install stuff and would still promote the ease of publishing plugins that we already have. |
I think that's a pretty good point. I agree that leaving it out would minimize maintenance cost on us. My comment pitching a few ways of doing it I think illustrates this point. It's all unneeded if we just make people install a plugin like any other local package. Leaving out plugin loading makes things a bit more inconvenient for the user, but not a lot. Especially with good docs, a user could pretty easily modify their primary pyproject.toml (for example) to add a line to add a local plugin in editable mode. Then their env setup is basically unchanged from then on. To give a clear example, a user could run |
I disagree, I think. We've been through this before (see #875), and I suspect I'm still in the minority, but... While I agree that our documentation should include some pointers and encouragement on how to publish a plugin the PyPI and I think the plugin showcase in the docs is a great way to encourage and publicize plugins, I think it's possible to push too hard (and make it too "easy") to publish plugins to PyPI. It does a disservice to both the Python and Lektor communities to have crufty, stale, orphaned, non-maintained distributions to PyPI. It's not a good look for Lektor to have a number of It takes more than a functioning setup.py to make a healthy public distribution: e.g. hopefully, there is a GitHub repo (or similar) for the plugin to foster community feedback, some CI testing, etc. I don't think it's Lektor's place to provide a lot of guidance and leadership on these basic open-source, and python-wide best practices. Obviously, everyone who is going to be writing a Lektor plugin and then publishing it to PyPI (and then maintaining it to any degree) is going to have some level of competence in Python development. It's just not that hard to create a PyPI distribution and there's plenty of general documentation out there on how to do this. I don't think it's in Lektor's best interest to do too much hand-holding in this regard. On the other side, nearly every Lektor project I work on includes a local Lektor plugin which includes "throw-away", local-to-that-particular-project, extensions. Mostly, these include things like custom jinja globals and filters, but often it includes custom link formatting in Markdown text, etc. Including distribution metadata for these plugins (other than, perhaps, entry point declarations) is just superfluous. It's some trouble (not a lot, admittedly), but, more importantly, it's pointless — e.g. the version number in that setup.py means nothing. I suspect it would be more "beginner-friendly" to make basic local plugin authoring as simple as possible, rather than getting hung up and ensuring that all plugins are "PyPI-ready". |
I'm neutral on the idea of adding a new custom plugin loader. It would be simple enough (i.e. minimal additional code) to do something like: check a particularly directory for At the same time, once one switches to using a package manager (e.g. Poetry) to manage the Lektor plugins for a particularly Lektor project, it's really not that complicated to manually configure the appropriate entry point(s) in the top-level metadata file for a project. E.g. here's a Poetry [tool.poetry]
# Lektor currently requires that the dist name be "lektor-<plugin-name>". This should, IMO, be relaxed.
# If it were, we could name this project "test-demo-lektor-website" or something else
# more descriptive. (We could also configure more than one local plugin.)
name = "lektor-demo"
version = "0.1.0"
description = "Test"
authors = ["Jeff Dairiki <[email protected]>"]
readme = "README.md"
#packages = [{include = "demo_plugin.py"}]
[tool.poetry.plugins."lektor.plugins"]
# plugin code is in ./lektor_demo/plugin.py
demo = "lektor_demo.plugin:DemoPlugin"
[tool.poetry.dependencies]
python = "^3.7"
Lektor = "^3.3.7"
lektor-atom = "^0.4.0"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api" Note that this works just fine with current versions of Lektor. |
Oh, good point, I didn't consider it before. Yes, we can define the local plugins directly in the current project's metadata and after it is installed, the plugins are activated already. Except that it comes with a minor inconvenience -- the lektor project itself must be a PyPI-ready package, which is not usually the case for a website. It seems Poetry ensures that when initializing a project but others like Pipenv may not force that. |
For a long time now, it has become clearer that having Lektor manage the installation of a project's Python plugins is a significant pain point for maintainers and users. There are a lot of issues that have popped up over the years, from bugs, to missing features. I and some other maintainers are leaning toward minimizing Lektor's role in installing and managing a project's plugin Python packages. @mitsuhiko do we have your go-ahead? Thoughts or concerns?
Regardless of the exact goal chosen, there are the normal concerns about breakages and deprecations in order to prepare users for the changes and not make things unnecessarily hard.
We need to decided what exactly this looks like. For example, this could mean:
Rough option 1) - minimize plugin management:
lektor plugins
command, and it's sub-commands[packages]
section from project filespackages/
subdirectoryrequirements.txt
.lektor dev new-plugin
, though this would likely be modified, at least to match the changed docs instructions. I don't see this, isolated, to be very burdensome, and is potentially very helpful.In my opinion this option is the best, because
This plan has the con of potentially removing some convenience features for users, such as only having to understand and use a single program (Lektor) instead of a minimum of two (Lektor and pip/poetry/etc). The burden of choice at least can be minimized by clear instructions in our docs.
Rough option 2) - do some level of wrapping of a preferred packaging tool
Should we try to keep some pieces of
lektor plugins
, such asadd
,remove
, andlist
, and adapt them to match or wrap a chosen tool like Poetry? Should we leave in[packages]
and somehow translate that to invoke a tool like Poetry? Should we auto-install things frompackages/
? These goals are somewhat independent, and they may offload some maintenance burden, but they clearly add some too. Choosing this path could mitigate some of the drawbacks of the option 1). In my opinion, that is not a good enough tradeoff.N.B. there are also several discussions about using a different package management strategy internally to maintain this repo. This wouldn't matter much if we choose option 1), above, but the less we approach option 1), the more it might reduce mental overhead to use the same tool for integrations for projects and to manage this repo itself.
The text was updated successfully, but these errors were encountered: