Integrating attestation verification into brew install (and brew verify)

[woodruffw]

Verification

• This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature

I'm opening this to begin the discussion/design review process for the final piece of the build provenance work that we (@josephsweeney and I) have been doing.

For context:

1. homebrew-core has been generating attestations for a few weeks now: https://github.com/Homebrew/homebrew-core/attestations, covering an increasingly large number of "reachable" bottles
2. For pre-existing bottles that were created before the provenance rollout, we performed a one-time "backfill" attestation here: https://github.com/trailofbits/homebrew-brew-verify/attestations

Combined, this means that every currently reachable bottle in homebrew-core either has a direct attestation or a fallback "backfill" attestation.

The next step from here is to actually verify these attestations, which is what is proposed below 🙂

Proposal

TL;DR variant:

• Add a BREW_VERIFY_ATTESTATIONS environment config, which will default to false during the beta period.
• When BREW_VERIFY_ATTESTATIONS=true, brew install will additionally perform verification of each bottle's attestation as it gets downloaded (or reused from cache).
• Regardless of BREW_VERIFY_ATTESTATIONS, the separate brew verify [formulae ...] command will allow users to explicitly verify bottle(s) by their formulae names.

The above technical elements will be implemented by myself and @josephsweeney, primarily by integrating our MVP brew verify implementation here: https://github.com/trailofbits/homebrew-brew-verify. We will make sure this implementation is suitable for both API and CLI use (i.e., so that it can be used as part of brew install's steps as well as with a stand-alone brew verify invocation).

Outstanding considerations

• How long should the beta last?
• Attestation verification currently requires the gh CLI, meaning that BREW_VERIFY_ATTESTATIONS=true will require a bootstrapping phase with brew install gh or similar to start. Should this be hash-pinned or similar?

What is the motivation for the feature?

This feature was previously reviewed and approved by the Homebrew TSC and leadership at the 2023 and 2024 AGMs.

How will the feature be relevant to at least 90% of Homebrew users?

During the beta period (i.e. BREW_VERIFY_ATTESTATIONS=false), this will have no effect on Homebrew users besides those who choose to explicitly enable it.

Once enabled by default, this will enable build provenance (modulo the backfill) verification for every bottle in homebrew-core, changing Homebrew's bottle integrity guarantee from "it matches a known digest" to "it matches a known digest and it was produced on Homebrew's own CI/CD."

What alternatives to the feature have been considered?

None.

[woodruffw]

See also #16543 for the prior phase of this 🙂

[carlocab]

How difficult would it be to support verifying attestations natively in Ruby?

I'd prefer to avoid needed gh to verify attestations, since that presents challenges to verifying the gh attestation.

[MikeMcQuaid]

Thanks @woodruffw!

[ ] Add a BREW_VERIFY_ATTESTATIONS environment config, which will default to false during the beta period.

Should probably default to an empty/unset variable being false rather than handling false, just FYI.

• When BREW_VERIFY_ATTESTATIONS=true, brew install will additionally perform verification of each bottle's attestation as it gets downloaded (or reused from cache).

Should probably be =1 or really set to any, non-blank value.

• Regardless of BREW_VERIFY_ATTESTATIONS, the separate brew verify [formulae ...] command will allow users to explicitly verify bottle(s) by their formulae names.

👍🏻

• How long should the beta last?

I think it's less about the time and more about just gradual ramp-up based on bugs/issues/performance regressions seen.

Generally these things go through:

• require ENV variable
• enabled by default for HOMEBREW_DEVELOPERs
• enabled by default in CI
• optional: enabled by default for HOMEBREW_DEVCMD_RUN
• enabled on master, not yet in a stable tag
• in a stable tag

• Attestation verification currently requires the gh CLI, meaning that BREW_VERIFY_ATTESTATIONS=true will require a bootstrapping phase with brew install gh or similar to start. Should this be hash-pinned or similar?

I think this is fine while the ENV variable is opt-in or for the brew verify command. I agree that making this a hard requirement for all of brew install invocations is something we probably need to resolve before this can go out to everyone as default behaviour (and probably even go to homebrew developers/CI).

Once enabled by default, this will enable build provenance (modulo the backfill) verification for every bottle in homebrew-core, changing Homebrew's bottle integrity guarantee from "it matches a known digest" to "it matches a known digest and it was produced on Homebrew's own CI/CD."

Sounds good, assuming the performance overhead here is minimal.

[Bo98]

I'm OK relying on gh as an opt-in env for brew install (and keeping brew verify in a separate tap) initially.

I'm less OK with doing so unconditionally as we've generally been moving away from arbitrary external tools (beyond guaranteed system-provided tools on macOS). See past efforts of: moving from patchelf to patchelf.rb, readelf to rbelftools, the current PR for ldd and the existence of Portable Ruby. The chicken egg problem of verifying gh itself makes this more awkward and having to issue a new brew release every time gh is updated sounds annoying. A mini bundled gh (like Portable Ruby) used exclusively for this may however be possible if necessary.

But a lot of this can be "questions for later". I was going to point out gh-attestation being closed source but I think that got fixed in yesterday's gh release? So seems like the picture is already changing. How it actually works is now clearer and I'll take a look over the code to give myself a more informed idea of what's actually involved here rather than guessing what the black box does.

[woodruffw]

How difficult would it be to support verifying attestations natively in Ruby?

I'd prefer to avoid needed gh to verify attestations, since that presents challenges to verifying the gh attestation.

I strongly agree, although unfortunately the answer to "how difficult" is currently "very difficult" -- verifying an attestation here requires a full Sigstore verifier implementation, and there currently isn't a suitable native Ruby one that's ready for use here.

The closest implementation is probably the one that the RubyGems folks have been working on (they have similar native-Ruby-only requirements), but IIUC they're still in active development as well and probably aren't ready for downstream dependencies yet.

---

Should probably default to an empty/unset variable being false rather than handling false, just FYI.

Should probably be =1 or really set to any, non-blank value.

Makes sense! I was using shorthand here; I figure this would go in Homebrew::EnvConfig, which would abstract all the 1, bool, etc. variants 🙂

---

I'm less OK with doing so unconditionally as we've generally been moving away from arbitrary external tools (beyond guaranteed system-provided tools on macOS). See past efforts of: moving from patchelf to patchelf.rb, readelf to rbelftools, the current PR for ldd and the existence of Portable Ruby. The chicken egg problem of verifying gh itself makes this more awkward and having to issue a new brew release every time gh is updated sounds annoying. A mini bundled gh (like Portable Ruby) used exclusively for this may however be possible if necessary.

Makes sense. To align this and @carlocab's comment: is the thought that users would have to explicitly brew install gh first, only after which BREW_VERIFY_ATTESTATIONS=1 brew install ... would be effective?

If so, that makes sense to me (in the sense that it does an end-run around the bootstrapping problem) and IMO is an acceptable solution until we can vendor-in a pure Ruby attestation verifier.

My proposal with that:

• BREW_VERIFY_ATTESTATIONS=0 brew install: current, default behavior
• BREW_VERIFY_ATTESTATIONS=1 brew install with no gh installed: failure
• BREW_VERIFY_ATTESTATIONS=1 brew install with gh installed: performs attestation verification

[woodruffw]

Also: re: brew verify: keeping it in an external tap is OK by me if that's the consensus, but I'd prefer to upstream it (even if only as a dev-cmd) if possible: my reasoning is that brew verify will share almost exactly the same underlying implementation as the feature-flagged verification under brew install, so IMO it makes sense to de-dupe the code 🙂

[Bo98]

The closest implementation is probably the one that the RubyGems folks have been working on (they have similar native-Ruby-only requirements), but IIUC they're still in active development as well and probably aren't ready for downstream dependencies yet.

This might be an interesting watch. We don't necessarily have to move quickly here. If this route looks dead later in the year we can re-evaluate options but it seems worthwhile watching that space as a potential way forward removing the env-gate.

If RubyGems needs/likes any input/contributions from me then I'm happy to do that too.

[MikeMcQuaid]

Makes sense. To align this and @carlocab's comment: is the thought that users would have to explicitly brew install gh first, only after which BREW_VERIFY_ATTESTATIONS=1 brew install ... would be effective?

I think it'd be OK for it to install gh itself (we have a one-liner for such things), it's more just that it can't really be default until it's entirely in Ruby.

[Bo98]

even if only as a dev-cmd

I'd be ok as a dev-cmd if we make it clear in the command description too that it's under development. We can promote it to cmd when we're ready to remove the env-gate.

[woodruffw]

If RubyGems needs/likes any input/contributions from me then I'm happy to do that too.

Paging @segiddins 🙂

---

I'd be ok as a dev-cmd if we make it clear in the command description too that it's under development. We can promote it to cmd when we're ready to remove the env-gate.

Sounds good! We'll make that explicit.

[segiddins]

https://github.com/segiddins/sigstore-cosign-verify should almost be usable!

[woodruffw]

I've opened #17049 with the first bits of this: that PR adds HOMEBREW_VERIFY_ATTESTATIONS and directs brew update to bootstrap gh if HOMEBREW_VERIFY_ATTESTATIONS is set. Once set, brew install verifies the attestations for any bottles being fetched from homebrew-core.

[carlocab]

#17049 seems to work well, but verifying attestations seems a bit slow:

❯ brew uninstall hello
Uninstalling /opt/homebrew/Cellar/hello/2.12.1... (11 files, 177KB)
❯ time (HOMEBREW_VERIFY_ATTESTATIONS=1 brew install hello &>/dev/null)
( HOMEBREW_VERIFY_ATTESTATIONS=1 brew install hello &> /dev/null; )  0.85s user 0.27s system 37% cpu 2.982 total
❯ brew uninstall hello
Uninstalling /opt/homebrew/Cellar/hello/2.12.1... (11 files, 177KB)
❯ time (HOMEBREW_VERIFY_ATTESTATIONS= brew install hello &>/dev/null)
( HOMEBREW_VERIFY_ATTESTATIONS= brew install hello &> /dev/null; )  0.62s user 0.19s system 79% cpu 1.018 total

Note: bottle pre-fetched so download time is not included

[woodruffw]

seems to work well, but verifying attestations seems a bit slow:

Yeah, I think there's some low hanging fruit that I can optimize here: gh attestation verify is probably pulling more things down the pipe than it strictly needs to, and there may be optimizations that it can further do internally. I'll do some profiling.

[carlocab]

FYI, trying to run brew upgrade when gh is outdated and HOMEBREW_VERIFY_ATTESTATIONS is set results in:

==> Upgrading gh
  2.47.0 -> 2.48.0
==> Verifying attestation for gh
Error: The bottle for gh has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

attestation verification failed: Failure while executing; `/opt/homebrew/bin/gh attestation verify /Users/carlocab/Library/Caches/Homebrew/downloads/51240e810aca942d2fdc644c6f72d02b0c6cf00ea4e2653417b3ca959c427762--gh--2.48.0.arm64_sonoma.bottle.tar.gz --repo trailofbits/homebrew-brew-verify --format json --cert-identity https://github.com/trailofbits/homebrew-brew-verify/.github/workflows/backfill_signatures.yml@refs/heads/main` exited with 127. Here's the output:

This is likely because /opt/homebrew/bin/gh either doesn't exist or is a broken symlink at the point we tried to execute it.

[woodruffw]

Thanks for catching that! Looking at it now.

Edit: I was able to reproduce this:

==> Upgrading gh
  2.47.0 -> 2.48.0 
==> Verifying attestation for gh
Error: The bottle for gh has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

attestation verification failed: Failure while executing; `/opt/homebrew/bin/gh attestation verify /Users/william/Library/Caches/Homebrew/downloads/51240e810aca942d2fdc644c6f72d02b0c6cf00ea4e2653417b3ca959c427762--gh--2.48.0.arm64_sonoma.bottle.tar.gz --repo trailofbits/homebrew-brew-verify --format json --cert-identity https://github.com/trailofbits/homebrew-brew-verify/.github/workflows/backfill_signatures.yml@refs/heads/main` exited with 127. Here's the output:

[woodruffw]

@carlocab I think your theory is right: upgrade unlinks each previous keg right before installation, meaning that the symlink is probably broken or nonexistent. If I remember my formula building blocks correctly, I'm pretty sure this means that we need to reference the gh executable via its cellar path rather than its default prefix linkage. Trying that now.

Edit: Opened #17106 with a prospective fix.