Unknown binaries in /opt/homebrew/bin

[svallory]

Output of brew config

HOMEBREW_VERSION: 4.0.19
ORIGIN: https://github.com/Homebrew/brew
HEAD: 4446fe603cc19316036fd07e5df1124c4d15ead2
Last commit: 7 days ago
Core tap JSON: 22 May 21:01 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_BROWSER: open
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: nano
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 10
Homebrew Ruby: 2.6.10 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 14.0.0 build 1400
Git: 2.39.0 => /opt/homebrew/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.6-arm64
CLT: 14.0.0.0.1.1661618636
Xcode: 14.2
Rosetta 2: false

Output of brew doctor

➜ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: A newer Command Line Tools release is available.
Update them from Software Update in System Preferences.

If that doesn't show you any updates, run:
  sudo rm -rf /Library/Developer/CommandLineTools
  sudo xcode-select --install

Alternatively, manually download them from:
  https://developer.apple.com/download/all/.
You should download the Command Line Tools for Xcode 14.2.

Description of issue

I found a LOT of executables in the default bin folder (/opt/homebrew/bin). Most, if not all of them, are python related. I have no idea where these came from and it may have nothing to do with brew.

But it may also be that a formula went rogue and is installing things where it shouldn't. So I'm here to ask:

How can I find out if a formula did this during installation?

Fri Dec  9 18:35:53 2022  bscpylgtvcommand 
Mon Apr 24 01:14:31 2023  distro 
Mon Apr 24 01:14:31 2023  markdown_py 
Mon Apr 24 01:14:31 2023  watchmedo 
Mon Apr 24 01:14:32 2023  autoflake 
Mon Apr 24 01:14:32 2023  ghp-import 
Mon Apr 24 01:14:32 2023  mkdocs 
Mon Apr 24 01:14:32 2023  openai 
Mon Apr 24 01:14:32 2023  openapi-python-client 
Mon Apr 24 01:14:32 2023  pathy 
Mon Apr 24 01:19:53 2023  spacy 
Mon Apr 24 01:20:30 2023  pip 
Mon Apr 24 01:20:30 2023  pip3 
Mon Apr 24 01:20:30 2023  pip3.10 
Mon Apr 24 01:23:17 2023  rst2html.py 
Mon Apr 24 01:23:17 2023  rst2html4.py 
Mon Apr 24 01:23:17 2023  rst2html5.py 
Mon Apr 24 01:23:17 2023  rst2latex.py 
Mon Apr 24 01:23:17 2023  rst2man.py 
Mon Apr 24 01:23:17 2023  rst2odt_prepstyles.py 
Mon Apr 24 01:23:17 2023  rst2odt.py 
Mon Apr 24 01:23:17 2023  rst2pseudoxml.py 
Mon Apr 24 01:23:17 2023  rst2s5.py 
Mon Apr 24 01:23:17 2023  rst2xetex.py 
Mon Apr 24 01:23:17 2023  rst2xml.py 
Mon Apr 24 01:23:17 2023  rstpep2html.py 
Mon Apr 24 01:23:17 2023  pkginfo 
Mon Apr 24 01:23:17 2023  pygmentize 
Mon Apr 24 01:23:18 2023  __pycache__/
Mon Apr 24 01:23:18 2023  docutils 
Mon Apr 24 01:23:18 2023  get_objgraph 
Mon Apr 24 01:23:18 2023  markdown-it 
Mon Apr 24 01:23:18 2023  pyproject-build 
Mon Apr 24 01:23:18 2023  undill 
Mon Apr 24 01:23:19 2023  epylint 
Mon Apr 24 01:23:19 2023  keyring 
Mon Apr 24 01:23:19 2023  pylint 
Mon Apr 24 01:23:19 2023  pylint-config 
Mon Apr 24 01:23:19 2023  pyreverse 
Mon Apr 24 01:23:19 2023  symilar 
Mon Apr 24 01:23:19 2023  twine 
Mon May 22 18:01:17 2023  brew 
Sun Apr 16 15:51:26 2023  html2text 
Sun Apr 16 15:51:29 2023  normalizer 
Sun Apr 16 15:51:31 2023  tqdm 
Sun Apr 16 15:51:58 2023  dumppdf.py 
Sun Apr 16 15:51:58 2023  pdf2txt.py 
Tue Apr 18 18:49:47 2023  cpuinfo 
Tue Apr 18 18:49:47 2023  dotenv 
Tue Apr 18 18:49:47 2023  pyrsa-decrypt 
Tue Apr 18 18:49:47 2023  pyrsa-encrypt 
Tue Apr 18 18:49:47 2023  pyrsa-keygen 
Tue Apr 18 18:49:47 2023  pyrsa-priv2pub 
Tue Apr 18 18:49:47 2023  pyrsa-sign 
Tue Apr 18 18:49:47 2023  pyrsa-verify 
Tue Apr 18 18:49:47 2023  sourcery 
Tue Apr 18 18:49:47 2023  wsdump 
Tue Apr 18 18:49:48 2023  pycodestyle 
Tue Apr 18 18:49:48 2023  pyflakes 
Tue Apr 18 18:49:49 2023  nodeenv 
Tue Apr 18 18:49:50 2023  chardetect 
Tue Apr 18 18:49:50 2023  coverage 
Tue Apr 18 18:49:50 2023  coverage-3.10 
Tue Apr 18 18:49:50 2023  coverage3 
Tue Apr 18 18:49:50 2023  identify-cli 
Tue Apr 18 18:49:50 2023  isort 
Tue Apr 18 18:49:50 2023  isort-identify-imports 
Tue Apr 18 18:49:51 2023  ddgs 
Tue Apr 18 18:49:51 2023  flake8 
Tue Apr 18 18:49:51 2023  gtts-cli 
Tue Apr 18 18:49:51 2023  jsonschema 
Tue Apr 18 18:49:51 2023  pinecone 
Tue Apr 18 18:49:51 2023  py.test 
Tue Apr 18 18:49:51 2023  pytest 
Tue Apr 18 18:49:51 2023  virtualenv 
Tue Apr 18 18:49:52 2023  black 
Tue Apr 18 18:49:52 2023  blackd 
Tue Apr 18 18:49:52 2023  pre-commit 
Tue Apr 18 18:49:52 2023  py.test-benchmark 
Tue Apr 18 18:49:52 2023  pytest-benchmark 
Wed Apr 19 00:07:02 2023  isympy 
Wed Apr 19 00:07:03 2023  huggingface-cli 
Wed Apr 19 00:07:06 2023  transformers-cli 
Wed Apr 19 00:07:11 2023  convert-caffe2-to-onnx 
Wed Apr 19 00:07:11 2023  convert-onnx-to-caffe2 
Wed Apr 19 00:07:11 2023  flask 
Wed Apr 19 00:07:11 2023  httpx 
Wed Apr 19 00:07:11 2023  openplayground 
Wed Apr 19 00:07:11 2023  torchrun 
Wed Apr 19 01:14:08 2023  ray 
Wed Apr 19 01:14:08 2023  rllib 
Wed Apr 19 01:14:08 2023  serve 
Wed Apr 19 01:14:08 2023  tune 
Wed Apr 19 19:48:43 2023  f2py 
Wed Apr 19 19:48:43 2023  f2py3 
Wed Apr 19 19:48:43 2023  f2py3.10 
Wed Apr 19 19:48:48 2023  google-oauthlib-tool 
Wed Apr 19 19:48:55 2023  estimator_ckpt_converter 
Wed Apr 19 19:48:55 2023  import_pb_to_tensorboard 
Wed Apr 19 19:48:55 2023  saved_model_cli 
Wed Apr 19 19:48:55 2023  tensorboard 
Wed Apr 19 19:48:55 2023  tf_upgrade_v2 
Wed Apr 19 19:48:55 2023  tflite_convert 
Wed Apr 19 19:48:55 2023  toco 
Wed Apr 19 19:48:55 2023  toco_from_protos

Relevant casks

That is the question :)

[SMillerDev]

Run ls -al /opt/Homebrew/bin and see where they link

[gromgit]

Homebrew only puts one actual file in /opt/homebrew/bin, and that's the brew command. Since the files concerned all Python-related, I'm guessing that they were put there when you ran pip3 install blah_blah_blah.

[svallory]

Nah... I didn't manually do that. Look at the info for openai package for example:

pip3 show openai
Name: openai
Version: 0.27.2
Summary: Python client library for the OpenAI API
Home-page: https://github.com/openai/openai-python
Author: OpenAI
Author-email: [email protected]
License:
Location: /opt/homebrew/lib/python3.10/site-packages
Requires: aiohttp, requests, tqdm
Required-by: openplayground

My hope openning the discussion was that I could get some insight into how formulas are installed so I could run a search on all the formulas I have installed to try to find the culprit

[gromgit]

You're confusing Python packages (installed with pip3) with Homebrew formulae (installed with brew):

• Homebrew installs formula binaries as symlinks in /opt/homebrew/bin, Python-based or otherwise
• Python pip3 installs package binaries as executable files in <prefix>/bin, where <prefix> is the root directory of the Python3 runtime

Based on your pip3 show output, especially:

Location: /opt/homebrew/lib/python3.10/site-packages

you're using the pip3 from Homebrew's [email protected] formula, so <prefix> is /opt/homebrew, and therefore it will install Python binaries as executable files in /opt/homebrew/bin.

Additionally, all Python-based Homebrew formulae use virtualenvs to sandbox their installation, so that everything needed for the formula my_py_stuff is installed under /opt/homebrew/Cellar/my_py_stuff/<version>, then symlinked into /opt/homebrew/bin.

In summary, the files you're complaining about were installed by Python, not Homebrew. If you don't recall installing them yourself via pip3, then something else other than Homebrew installed them for you, probably some Python-based application or other.

[svallory]

Thanks for the thorough answer, @gromgit! A couple of things though... I'm not confusing pip packages with brew formulas. I made a point of NOT installing python via brew since I've had problems doing that before. And I think it's really weird to me that brew would change the default prefix of python3/pip3. (Ruby and gems via brew are also veeery messy btw)

Anyways... look at the output below:

I should have been more clear in the last message. Since I DID NOT install python via brew, nor set the pip3 prefix to use homebrew's bin folder, I imagine a formula I installed recently caused that mess.

I don't know the brew architecture to figure out in a semi-automatic way (e.g. sourgraph structural search) so I'm asking for tips regarding which files to look for inside a formula, is there a "step" I should look into?

All in all, I think I'm going to restore my installation of python/pip, simply move all the files in brew's bin leaving only brew and symlinks there and wait until something breaks.

Unless you have a better idea :)

[gromgit]

Thanks for the thorough answer, @gromgit! A couple of things though... I'm not confusing pip packages with brew formulas. I made a point of NOT installing python via brew since I've had problems doing that before. And I think it's really weird to me that brew would change the default prefix of python3/pip3. (Ruby and gems via brew are also veeery messy btw)

Homebrew does NOT change the default prefix of a system or third-party Python installation. It doesn't even care about them. See the following references for how Homebrew deals with Python at the user level and the formula author level.

Since I DID NOT install python via brew, nor set the pip3 prefix to use homebrew's bin folder,

Just to be sure, what's the output of:

env | grep PYTHON

?

You should also check your own startup files and other scripts, to see if you're setting any PYTHON* environment variables there, particularly PYTHONHOME or PYTHONUSERBASE.

You might also want to mentally review your software installation actions on the dates of the files you tagged. For instance, how did you install openai and spacy on Apr 24, and how did you update pip on that day too?

I imagine a formula I installed recently caused that mess.

As far as I can tell, the only way a formula could do so is by ignoring everything in the Homebrew Python formula author's guidelines doc, and quietly using whatever python is available in PATH. I'm pretty confident that such a formula would fail Homebrew's own CI, and therefore cannot possibly exist in Homebrew core. You might therefore want to look in any third-party taps you're using, which aren't forced to play by Homebrew's rules.

[svallory]

UPDATE: I did a full cleanup of Homebrew packages and reinstalled everything. I'm using python installed via Homebrew and every binary I install via pip gets placed at /opt/homebrew/bin and is not a link. And tbw...

❯ brew doctor
Your system is ready to brew.

❯ brew -v
Homebrew 4.0.28

@gromgit looks like this is the default behaviour

[gromgit]

I'm using python installed via Homebrew and every binary I install via pip gets placed at /opt/homebrew/bin and is not a link.

Which is exactly what I explained earlier:

• Homebrew installs formula binaries as symlinks in /opt/homebrew/bin, Python-based or otherwise
• Python pip3 installs package binaries as executable files in <prefix>/bin, where <prefix> is the root directory of the Python3 runtime

To restate the above as clearly as possible, for all M1-based macOS systems like yours:

• brew install <formula> will download precompiled binary packages to install, and symlink everything in /opt/homebrew/Cellar/<formula>/<version>/bin into /opt/homebrew/bin. It will NEVER copy files into /opt/homebrew/bin, ever.
• pip3 install <package> will download Python source packages to install, and copy executables into wherever pip3 is configured to install stuff:
  • If it's the Homebrew pip3, then it'll copy executables into /opt/homebrew/bin.

Since you stated that you used pip install to install your stuff, and not brew install, what you see is exactly what is expected.

[svallory]

oh... I must have read it in a hurry! For some reason, I was thinking of the prefix as the root folder of the installed python formula, not the homebrew prefix (which I think would be a better approach to avoid (rare) binary name conflicts)

And hey, thank you so much for your patience. Really appreciated!

[quackerex]

Why this still happen even when I do pip install inside a venv? Also how PyPi packages placing binaries in that folder without sudo privileges?

[quackerex]

This was error on my part as once I ran pip install outside of a venv some time ago .
After some testing it seems that python does not add binaries to homebrew bin folder if you are in a venv. Instead they are in the bin folder inside the venv folder.