brew update fails when (1) more than one private tap uses git protocol and (2) the ssh keys are in an ssh-agent that requires interactivity

[samhclark]

Output of brew config

HOMEBREW_VERSION: 4.2.15-48-g4501e38-dirty
ORIGIN: https://github.com/Homebrew/brew
HEAD: 4501e38bd81ac3deb0b4f4ced0376ecf338c8325
Last commit: 7 hours ago
Core tap JSON: 29 Mar 14:19 UTC
Core cask tap JSON: 29 Mar 14:20 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: hx
HOMEBREW_MAKE_JOBS: 12
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.1.4 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: dodeca-core 64-bit arm_blizzard_avalanche
Clang: 15.0.0 build 1500
Git: 2.39.3 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 8.4.0 => /usr/bin/curl
macOS: 14.3.1-arm64
CLT: 15.1.0.0.1.1700200546
Xcode: N/A
Rosetta 2: false

Output of brew doctor

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  [email protected]

Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
  /usr/local/lib/libMsgCom.dylib
  /usr/local/lib/libSysCtlCom.dylib
  /usr/local/lib/libwep_airdrop.dylib
  /usr/local/lib/libwep_burn.dylib
  /usr/local/lib/libwep_cbcarbon.dylib
  /usr/local/lib/libwep_cbcocoa.dylib
  /usr/local/lib/libwep_chrome.dylib
  /usr/local/lib/libwep_dutil.dylib
  /usr/local/lib/libwep_ff.dylib
  /usr/local/lib/libwep_icloud.dylib
  /usr/local/lib/libwep_mail.dylib
  /usr/local/lib/libwep_post.dylib
  /usr/local/lib/libwep_printer.dylib
  /usr/local/lib/libwep_screen.dylib

Warning: Found Ruby file outside tgt/brewhouse tap formula directory.
(/opt/homebrew/Library/Taps/tgt/homebrew-brewhouse/Formula):
  /opt/homebrew/Library/Taps/tgt/homebrew-brewhouse/ghe_download_strategy.rb

Found Ruby file outside go-vela/vela tap formula directory.
(/opt/homebrew/Library/Taps/go-vela/homebrew-vela/Formula):
  /opt/homebrew/Library/Taps/go-vela/homebrew-vela/template.rb

Description of issue

By "private tap" I mean one that is not on GitHub and not public, so it requires the full URL to connect and requires some sort of auth to access it.

The ssh-agent I'm using here is Secretive. The ssh private key is stored inside the Mac's Secure Enclave so it requires Touch ID for every operation.

The behavior I see is that brew update works just fine when I have just one private tap and when using Secretive. No problems at all. It prompts me for my fingerprint and then is able to update that tap no problem. But when I add a second tap, things break. The update fails for one or both taps (hard for me to tell...).

The logs for that look something like this (brew update --debug):

+ [[ -f /opt/homebrew/Library/Taps/tgt/homebrew-brewhouse/.git/TMP_FETCH_FAILURES ]]
+ cat /opt/homebrew/Library/Taps/tgt/homebrew-brewhouse/.git/TMP_FETCH_FAILURES
+ [[ -z /Library/Developer/CommandLineTools/usr/bin/git ]]
+ /Library/Developer/CommandLineTools/usr/bin/git fetch --tags --force -q origin refs/heads/master:refs/remotes/origin/master
sign_and_send_pubkey: signing failed for ECDSA "ecdsa-sha2-nistp256" from agent: agent refused operation
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

But what's strange is that the log "agent refused operation" is showing on the terminal while the Touch ID prompt is still up. So it failed before I had a chance to interact.

To be really clear, that ssh key and setup works usually. It works for normal ssh operations. It works for git. It works for Homebrew when I have only one tap. It's only once I add another that I see issues.

I found this GitHub Issue which led me to this commit which says

We execute too many git fetch at once for this to be useful. Just let them fail instead and make it up to users to setup username/password caching or SSH agents.

but editing /opt/homebrew/Library/Homebrew/cmd/update.sh to set -oBatchMode=no didn't fix it. So I'd like to keep looking for a spot to fix it, but I don't know where to look next.

Thought I should post here for some tips so I can use Secretive for more than one tap. Any thoughts?

[colindean]

I'll note that I've got a (test) setup that is as identical as can be to @samhclark — we're coworkers — except for

1. I don't use Secretive
2. I don't have a password on the SSH key used to access the internal GHE

I'm not able to reproduce this. It has to be something related to how the git fetch is occurring and the role of the SSH agent.

[samhclark]

I think this is the answer.

It feels like a wrapper for Homebrew would enable this to be handled without any changes to Homebrew. You could setup these SSH connections in advance to cache them and then terminate after update. This could also avoid needing to use a custom SSH too.

Homebrew/brew#13469

That's the direction I'll take for now.