pnpm for Python

[vjpr]

Could we make pnpm work for Python?
https://dev.to/zkochan/comment/9j2d

I have recently started working on a Python project and package management is a bit of a mess in the Python ecosystem. A lot of the newer Python package managers are emulating npm. pipenv is one of the newer package managers, but its incredibly slow and people are now moving away from it to an even newer one called poetry. It seems that they are many years behind all the R&D that has gone into JS package managers like yarn and pnpm.

We are also looking at creating a monorepo for our project and there is no easy solution.

I wonder how much effort would be required to get pnpm to work with Python?

Thinking about a rapid MVP approach

1. Install-time dependency version resolving. Create a custom registry that will proxy pypi.org package registry but enrich the returned packages with package.json manifests. This should allow us to use pnpm's battle-tested and flexible dependency resolution algorithm without any changes.

   Looks like there is a package format that can simply be copied to a location with no other install steps. This gives me hope that it could be as easy as I mention:

   A Distribution format containing files and metadata that only need to be moved to the correct location on the target system, to be installed. Wheel is such a format
   https://packaging.python.org/glossary/#term-built-distribution

   Or we just use pip install (like Poetry does) using Install from local archives. How we use pip install would depend on whether we are going to get benefits from the symlinking and local caching, or whether we are more interested in the dependency resolution and monorepo support.

   a. Proxy pypi as an npm registry protocol. CON: Requires separate server.
   b. Allow npm to support an alternate registry protocol (pip/pypi). Responses would still be converted to pnpm package manifest formats. But we would need to implement a new package downloader to work with the pypi registry.

2. Runtime dependency resolution. How do we make Python work with the nested node_modules structure?

   a. Would shamelessly-flatten get us most of the way there?

   b. Could we modify the Python package resolver to support Node's algorithm? The reason for this would be to support multiple versions of the same package...which I'm not sure the current Python package management solutions allow.

   Overview of Python package importing can be found below. sys.path_hooks looks interesting.

---

How Python resolves dependencies?

Good article: https://realpython.com/python-virtual-environments-a-primer/

https://docs.python.org/3/reference/import.html

It is common to use a "virtualenv". This simply modifies your PATH env var to prioritize a python binary in a certain location to ensure Python uses virtualenv-specific dependencies.

A virtualenv looks like this:

├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── easy_install
│   ├── easy_install-3.5
│   ├── pip
│   ├── pip3
│   ├── pip3.5
│   ├── python -> python3.5
│   ├── python3 -> python3.5
│   └── python3.5 -> /Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5
├── include
├── lib
│   └── python3.5
│       └── site-packages
            └── YOUR PACKAGES HERE (this is the analogous `node_modules` dir)
└── pyvenv.cfg

Python module search path (sys.path)

sys.path ("Python module search path") is similar to require.resolve.paths

PYTHONPATH is similar to NODE_PATH.

sys.path is an array of paths Python will look in to resolve an import.

DIFFERENCE: Python will not search upwards looking for packages in node_modules dirs like Node.js does.

The python binary is located at ~/dev/my-project/.venv/bin/python. sys.prefix is determined by dropping the /bin...so it becomes ~/dev/my-project/.venv (virtualenv root).

sys.path = [
  <directory of script used to invoke Python interpreter OR empty string (which searches current dir first)>,
  process.env.PYTHONPATH,
  ...some other stuff,
  `${sys.prefix}/lib/pythonX.Y/site-packages`,

Example (using poetry and pyenv):

sys.path = [
  '.',
  '~/.pyenv/versions/3.8.1/lib/python38.zip',
  '~/.pyenv/versions/3.8.1/lib/python3.8',
  '~/.pyenv/versions/3.8.1/lib/python3.8/lib-dynload',
  ~/dev/my-project/.venv/lib/python3.8/site-packages,

See https://docs.python.org/3/library/site.html and https://github.com/python/cpython/blob/8510f430781118d9b603c3a2f06945d6ebc5fe42/Lib/site.py#L557

How Poetry works?

The package installation seems to just piggy-back on pip install, and its mostly just a dependency resolver for a custom package manifest format (pyproject.toml) and a runner.

See https://github.com/python-poetry/poetry/blob/master/poetry/installation/pip_installer.py

Poetry has an option to keep a project-specific directory of packages (similar to node_modules). You must enable the setting virtualenvs.in-project = true, and then it will create a dir called my-project/.venv which contains a virtualenv like mentioned above.

[vjpr]

Managed to setup a simple npm registry that when the package manager visits /:package/:version, it downloads dists from pypi, extracts dependencies from METADATA or requires.txt, adds a package.json with the dependencies, repacks it as an npm package, and responds with the packument data.

The only difficulty is that if the package manager visits /:package and then decides to download a specific version using a tarball url, then the tarball will not be ready and there will be no sha - so verification cannot take place. As long as the package manager visits /:package/:version before downloading, then we have time to repack packages on-the-fly.

[zkochan]

I want to try this out.

[hacker-DOM]

Fwiw, pdm is a terrific packaging manager for python (I've deployed projects with it, e.g. adoc-math) with support for pnpm's functionality :-)

[tombohub]

Does pdm work like npm or we still have to activate virtual environment?

[gsemet]

i would be interested in this discussion if anyone if available.

Poetry and hatch are not the "standard" in python, and both basically does their own caching and resolution without doing what pip already does. Both also "caches" the wheel to avoid fetching dependencies already downloaded. But

• each files get recopied in the virtualenv everytime, which can be slow
• each has their own "lockfile" (diff-able of course, for some times). They prefer TOML over yaml, but each dependency has still integrity information in this lockfile (version and checksum)

What is really missing in python:

• 0 copy installation in virtualenv, allowing to have n version of the same lib installed in its own isolated environment only once
• application distribution (a pipy allowing upload of complete application + frozen dependencies), to avoid the "resolver" to fetch a different version of what has been built when the application has been constructed (the wheel does not carry the "locked" dependencies).

[johndunderhill]

I would be interested in this dicussion also. We're really struggling to set up a Python monorepo with some of the basic guarantees provided by pnpm. We have a medium-ish collection of Python code. This entire topic is fluid enough that I think it might benefit from a live discussion. I would be willing to help coordinate that if people are interested.