Expose PEP 740 attestations functionality

[woodruffw]

WIP, still experimenting here. Not ready for review 🙂

TODO:

• Generate attestations for each dist
• Actually hook up twine --attestations
• Unit tests
• Documentation

See: pypi/warehouse#15871

[woodruffw]

I've confirmed that the basic version of this works as expected (attestations is currently ignored by PyPI and TestPyPI): https://github.com/woodruffw-experiments/gha-pep740-experiments/actions/runs/9008318313/job/24750093954

[webknjaz]

@woodruffw I just bumped Twine FYI. And pre-commit is fine now. Rebasing should get the blockers out of the way.

[woodruffw]

@woodruffw I just bumped Twine FYI. And pre-commit is fine now. Rebasing should get the blockers out of the way.

Good timing, so did @facutuesca 😅

[facutuesca]

I'm squashing and rebasing this branch now

[facutuesca]

@woodruffw @webknjaz The remaining lint failure is due to an error message string (_TOKEN_RETRIEVAL_FAILED_MESSAGE) being duplicated in attestations.py and oidc-exchange.py.
Would you prefer de-duplicating it by moving it into a new common.py file, or another approach?

[woodruffw]

Would you prefer de-duplicating it by moving it into a new common.py file, or another approach?

That's what I was thinking originally, although common.py won't be on sys.path so importing it won't be straightforward 🙁

I think the "clean" thing to do here would be to turn the Python files here into a project structure that gets installed as part of the Docker image's build. But that's a little heavyweight, so @webknjaz may have another idea 🙂

[webknjaz]

I'm leaning towards just being available on $PYTHONPATH, not pip installed. The way I did it @ https://github.com/re-actors/alls-green/blob/223e4bb/action.yml#L50-L53. However, such a restructuring should be separate and would probably include a src/pypi_publish/__main__.py entrypoint invoking the current main script through subprocess as the first step. I don't want a huge PR since big ones tend to cause long reviews and merge delays. Another thing to do would be to start to use our own managed virtualenv in the container rather than installing the deps globally into the shared userspace.

[webknjaz]

@woodruffw hey, it looks like GitHub rolled out their own attestations in beta: https://github.com/actions/attest-build-provenance / https://github.com/pypa/gh-action-pypi-publish/attestations / https://github.com/orgs/community/discussions/122028. I wonder if we could somehow integrate with that...

And it seems like there's even a new privilege attestations: write introduced for that on the platform: https://docs.github.com/en/actions/security-guides/using-artifact-attestations-to-establish-provenance-for-builds / https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idpermissions.

[woodruffw]

(Sorry, was on PTO -- catching up on mentions now)

@woodruffw hey, it looks like GitHub rolled out their own attestations in beta: https://github.com/actions/attest-build-provenance / https://github.com/pypa/gh-action-pypi-publish/attestations / https://github.com/orgs/community/discussions/122028. I wonder if we could somehow integrate with that...

Yep, I've been thinking about how to make integrate the two -- the last comment on the PEP discussion thread suggests an approach that would allow GitHub-generated attestations to be compatible with this PEP.

TL;DR: I think our options here are:

1. Move forward with the current "version 1" specified in the PEP. This would mean no further changes are required on the PEP side. However, it would also mean that "version 1" isn't compatible with the default attestation format generated by GitHub, and that we would have to create a "version 2" that is compatible.
2. Edit the PEP in its current form and make it compatible with GitHub's attestation format. This would be a small change to the PEP overall, but has knock-on consequences (e.g. conceptually unbounded JSON payloads being signed over).