ci(releases): build wheels for Python package

[redeboer]

As discussed in ComPWA/mirrors-taplo#13 (comment), here is an attempt to build Python package wheels with Maturin. That would make it way easier for the Python community to integrate Taplo into their workflow and would also make it possible to host a fast Taplo pre-commit hook (#535).

A test taplo pip package is available here:
https://pypi.org/project/taplo-test

And you can use this as a test pre-commit hook:

  - repo: https://github.com/redeboer/taplo-pre-commit
    rev: v0.9.1rc1
    hooks:
      - id: taplo

OpenSSL dependency

Building wheels for MacOS and Windows was easy, but I ran into trouble with Linux because of the apparent need for OpenSSL. I thought that OpenSSL is not a dependency anymore (#302), but it is still pulled in indirectly through reqwest:

taplo/crates/taplo-cli/Cargo.toml

Lines 31 to 33 in 4bf7b53

	reqwest = { version = "0.11.9", default-features = false, features = [
	"json",
	] }

It seems that reqwest is used in several of the taplo crates, so I presume we have to deal with OpenSSL as a dependency. Manylinux does not offer OpenSSL however, which resulted in broken wheels. I therefore added openssl as an optional dependency with the vendored flag, i.e. cargo build --features vendored-openssl (open to suggestions for a better name).¹

Even though that solution solution seems to work², it only 'bakes' openssl into the wheels for Linux x86_64: the 32-bit version is still built with shared openssl lib, because its manylinux still runs into problems building with vendored OpenSSL. And I haven't managed to test whether those wheels actually work.

Important

Who can help test whether the x86 wheels work? You can try the taplo-test package or the artifacts from the job.

PyPI release

I'm happy to help maintain this repository (#403), especially the Python release. For now, however, we need to somehow share a MATURIN_PYPI_TOKEN for the workflow. Who of the Taplo maintainers is also on PyPI?

Pre-commit hook

To address #535, it will be necessary to put the .pre-commit-hooks.yaml under a different config (see example repo here). The reason is that (1) it speeds up the pre-commit clone and (2) this repo already has a pyproject.toml for the Maturin build. This would be an argument for creating a 'Taplo organization' #403 (comment). ³

Footnotes

1. 55c7e96 upgrades the lock files to keep the diff of 2ef0d97 small; you can revert that lock update if preferred. ↩

2. This is the job that published the taplo-test package. ↩

3. An organization would also make it easier to extract the VSCode extension to separate repo. ↩

[redeboer]

Probably need to remove this trigger, the jobs take too long for PRs. I don't know about your release workflow, but ideally the jobs are run before a release tag for taplo-cli can be created.

[flying-sheep]

Python itself links against system OpenSSL, so we shouldn’t bundle it unless we can commit to frequent security updates: https://peps.python.org/pep-0513/#security-implications

The manylinux project itself only bundles it when the system OpenSSL version is too old: https://github.com/pypa/manylinux/blob/ea3724609ced892adb65f4d091cc01f1231d0235/docker/build_scripts/build-openssl.sh#L28

[redeboer]

Python itself links against system OpenSSL, so we shouldn’t bundle it unless we can commit to frequent security updates: https://peps.python.org/pep-0513/#security-implications

The manylinux project itself only bundles it when the system OpenSSL version is too old: https://github.com/pypa/manylinux/blob/ea3724609ced892adb65f4d091cc01f1231d0235/docker/build_scripts/build-openssl.sh#L28

Ah okay thanks. Then I don't get why the wheels without vendored OpenSSL cannot run on Linux. See this failed job or try installing the wheels from its artifacts, e.g.:

gh run download 7814055668 -n wheels -D wheels --repo redeboer/taplo
pip install --no-index --find-links wheels/ taplo
taplo --help

[flying-sheep]

The job failure happens because the maturin action runs in some ancient container with OpenSSL 1.0.x (libssl.so.10), whereas modern systems have OpenSSL 3 (libssl.so.3). The built library then links against the ancient ABI instead of the modern one.

I think the way the manywheels action works is that it first builds against a newer OpenSSL and then removes the OpenSSL library. That way it links against the OpenSSL 3 ABI.

But maybe I’m wrong and for adequate compatibility one needs to bundle OpenSSL like you do. In that case, we’re stuck with having to do frequent releases whenever there’s some OpenSSL security patch 😞

[redeboer]

The job failure happens because the maturin action runs in some ancient container with OpenSSL 1.0.x (libssl.so.10), whereas modern systems have OpenSSL 3 (libssl.so.3).

Afaics the maturin actions just uses the manylinux_2014 container and that comes with OpenSSL 1.0.x:

$ docker run --rm -it quay.io/pypa/manylinux2014_x86_64 openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

In fact, I wonder why openssl-devel needs to be installed at all. If you don't, Taplo won't compile in manylinux at all.

taplo/.github/workflows/python.yml

Lines 45 to 46 in ebdb3e3

	before-script-linux: |
	yum install -y openssl-devel

But maybe I’m wrong and for adequate compatibility one needs to bundle OpenSSL like you do. In that case, we’re stuck with having to do frequent releases whenever there’s some OpenSSL security patch 😞

Indeed, this is the Catch-22 :S convenience (and insecurity?) of bundling OpenSSL, or pointing to shared libraries...

[flying-sheep]

Is there anything blocking this from being merged?

[redeboer]

From my side:

• support for x86 (needed?)
• bundling with with OpenSSL, as this PR does now (with a regular release flow this should be fine imho)

Details in the comments above

[flying-sheep]

Yeah, I think there’s no way around bundling at the moment, and having only 64 bit wheels is probably fine for now.

If someone wants 32 bit, they can start complaining I guess.

[bollwyvl]

This would be lovely! Is it anticipated the PyPI package would carry the taplo-cli version number, or something else? For downstreams that already have a taplo package, e.g. conda-forge, would there be a preferred name for the python package when not installed from PyPI, e.g. python-taplo?

A big thing for making this an easily-adoptable tool for the broader python ecosystem would be if taplo (even the existing fat binary CLI) could be coaxed into looking in pyproject.toml#/tool/taplo to reduce dotfile proliferation.

[redeboer]

Is it anticipated the PyPI package would carry the taplo-cli version number, or something else?

The version number of the PyPI package is the same as that of the taplo-cli crate, e.g. 0.9.0. That is, what taplo --version returns.

taplo/pyproject.toml

Line 29 in 3487ebe

manifest-path = "crates/taplo-cli/Cargo.toml"

For downstreams that already have a taplo package, e.g. conda-forge, would there be a preferred name for the python package when not installed from PyPI, e.g. python-taplo?

Good point. Not sure if a name clash should be avoided here, it's still the same CLI that is installed, right? (Or does conda-forge provide all the crates?)

I'm thinking about https://anaconda.org/conda-forge/graphviz vs https://anaconda.org/conda-forge/python-graphviz. These are actually different packages, one being the actual DOT parser, the other the Python interface. I don't think we have the same situation with taplo on PyPI and taplo on conda-forge. Both are just a CLI tool afaiu.

A big thing for making this an easily-adoptable tool for the broader python ecosystem would be if taplo (even the existing fat binary CLI) could be coaxed into looking in pyproject.toml#/tool/taplo to reduce dotfile proliferation.

I agree, would also love to see that. Haven't found an issue on this in this repo, could you post one? So pyproject.toml ad an alternative configuration file for taplo.

[bollwyvl]

it's still the same CLI that is installed,

"Same" is a... big word: unlike pip, conda really doesn't like two packages shipping the same file, so on conda-forge we'll have to resolve that somehow so that install taplo and python-taplo doesn't explode.

If the anticipated wheels don't provide any Python API (e.g. import taplo)...

...we'll probably end up continuing to ship taplo-cli as taplo, build and deployed ~6 times ({linux,osx,win}-{ppc,arm,64}) by cargo install during the build. Then we'll just need one build for the python side of the house as a noarch package so that things like pip check still work. This will avoid re-building everything, but multiplied by py{3.8,3.9,3.10,3.11,3.12}. And no, even though maturin can build magic cross-python wheels, conda does per-python-per-os-per-arch-per-ssl-per-* builds: not my circus, not my monkeys.

provide all the crates

No, we (I) just build from source and ship the taplo-cli as $PREFIX/bin/taplo as a package called taplo. Folks that want to develop against taplo could conda install rust and then cargo install that way, but so far the only dev headers/libs I know of are C-based ones that are typically handled by distribution-style package managers (or niche ones like conan).

Are there other bin/taplo-* things I should be shipping?

graphviz

😱 oh the horror of making graphviz work on Windows without conda-forge in 2024.

could you post one

#603

[redeboer]

Hmm fair enough, yes. Tbh this PR was mostly motivated by making it easier to run taplo through pre-commit (see e.g. ComPWA/mirrors-taplo#13). Seeing as this PR just builds taplo-cli, perhaps the best way forward is to name the PyPI package taplo-cli.
Providing Taplo as a Python library (import taplo) would be a larger project (which I doubt is worth pursuing -- it goes in the direction of rtoml).

If the anticipated wheels don't provide any Python API (e.g. import taplo)...

So you mean that the conda-forge feedstock would just pull from PyPI? If so, should arm, ppc also be included in the targets?

taplo/.github/workflows/python.yml

Lines 22 to 24 in 3487ebe

	target:
	- x86
	- x86_64

[bollwyvl]

conda-forge feedstock would just pull from PyPI

For most python packages, yes: we use pip install of expanded .tar.gz sdists from files.pythonhosted.org. Wheels are used in a very small number of cases, sometimes due to egregious abuse of magic PEP517 [build-system]s. As a conda-forge maintainer, I really don't want pip installing nodejs from... wherever during my build. Similarly, I wouldn't want pre-commit pulling in... whatever... from... wherever. Hence, I'm more interested in the intentional case of declaring an optional dependency in pyproject.toml (a la #603), and will happily let pre-commit take care of itself.

Back on taplo-??? on conda-forge: we probably wouldn't touch PyPI at all. As a rust project, the canonical source (for the source) of taplo would likely continue to be the tagged GitHub archive, as it would contain whatever metadata is needed. We would patch some way to not invoke maturin at all, and still leave $PREFIX/site-packages/taplo-0.9.2.dist-info/ in a state that can appropriately fool pip check, one of the few tools that we have to validate the output of the galaxy of PEP517 build backends at scale.

[redeboer]

Alright, so for this PR, what should be changed? 😅 I think a better name for PyPI would indeed be taplo-cli.

taplo/pyproject.toml

Line 6 in 3487ebe

name = "taplo"

[bollwyvl]

To be clear: I am not suggesting what conda-forge will need to do should impact what this project should do on PyPI: pip install taplo && taplo fmt pyproject.toml feels very nice!