Python: Mark Homebrew Python as externally managed

[pradyunsg]

Hi there! I'm one of the maintainers of pip, and one of the co-authors of PEP 668.

As a quick summary, that PEP is basically a design document describing a new feature's design, for Python's packaging tooling. The feature is providing a mechanism (a marker file) to inform installers like pip, that it should not try to modify files installed in a certain location since those are "externally managed" (i.e. managed by a non-Python-native package manager).

Based on my very limited understanding of Homebrew (as a "I just need git and a few more things" user), it allows installing Python packages as dependencies of a package and they're importable by default. Based on that, IIUC, Homebrew should create the marker file in the appropriate location, as a part of the Python formula. So... To have a discussion about this and track it, I've filed this here. :)

/cc @woodruffw, per https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302/39?u=pradyunsg

PS: Not sure how Homebrew discussions work, but this seems like something that needs to be tracked, and should move into issues?

[carlocab]

Hi @pradyunsg, thanks for dropping by.

PS: Not sure how Homebrew discussions work, but this seems like something that needs to be tracked, and should move into issues?

Potentially, but there may not be any need to do this (yet?), depending on how much work this transition requires exactly. If it's just a simple change, then we might not need a tracking issue at all.

I'm not much of a Python user, but I scanned PEP 668 to see what the required changes might be. These values seem to be of particular interest:

❯ /usr/local/opt/[email protected]/bin/python3 -c 'import sys, sysconfig; print(sys.prefix); print(sys.base_prefix); print(sysconfig.get_path("stdlib", sysconfig.get_default_scheme()))'
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.10
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.10
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.10/lib/python3.10

Does that mean a

/usr/local/opt/[email protected]/bin/python3 -m pip install foo

currently installs foo into

/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.10/lib/python3.10

?

If so, that's setting users up for a bad time.

[pradyunsg]

No -- Homebrew/homebrew-core#76621 has the details on the solution that Homebrew used to use, for dealing with this issue for now. Now, there's a patch to sysconfig + distutils to make this work correctly IIUC.

[pradyunsg]

❯ /opt/homebrew/Cellar/[email protected]/3.10.2/bin/python3 -m pip install sampleproject --no-deps 
ERROR: Could not find an activated virtualenv (required).
❯ PIP_REQUIRE_VENV=0 /opt/homebrew/Cellar/[email protected]/3.10.2/bin/python3 -m pip install sampleproject --no-deps
Collecting sampleproject
  Downloading sampleproject-2.0.0-py3-none-any.whl (4.2 kB)
Installing collected packages: sampleproject
Successfully installed sampleproject-2.0.0
WARNING: You are using pip version 22.0.2; however, version 22.1.2 is available.
You should consider upgrading via the '/opt/homebrew/opt/[email protected]/bin/python3.10 -m pip install --upgrade pip' command.
❯ PIP_REQUIRE_VENV=0 /opt/homebrew/Cellar/[email protected]/3.10.2/bin/python3 -m pip uninstall sampleproject        
Found existing installation: sampleproject 2.0.0
Uninstalling sampleproject-2.0.0:
  Would remove:
    /opt/homebrew/bin/sample
    /opt/homebrew/lib/python3.10/site-packages/sample/*
    /opt/homebrew/lib/python3.10/site-packages/sampleproject-2.0.0.dist-info/*
    /opt/homebrew/my_data/data_file
Proceed (Y/n)? y
  Successfully uninstalled sampleproject-2.0.0

Notice the paths used.

[pradyunsg]

FWIW, implementing this would basically involve:

• Adding a /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.10/lib/python3.10/EXTERNALLY-MANAGED file (https://peps.python.org/pep-0668/#marking-an-interpreter-as-using-an-external-package-manager).
• (once support for this PEP is implemented in pip + that version of pip is included in Homebrew's Python) It would be feasible to remove the distutils/sysconfig patches since pip won't be trying to install into this externally managed environment.
  • The only way a user would then be able to modify the environment would be by downgrading pip via some mechanism (pip wouldn't let them touch the pip-installed-by-Homebrew).

The main difference that user-facing behaviour change that this will cause is that users won't be able to install packages using pip into the Homebrew-installed Python, instead needing to create a Python virtual environment to install packages using pip.

[carlocab]

Thanks for the clarification. Given that nothing's happened on this so far, we should probably track this in a new issue at homebrew-core, lest this get buried in this discussions page.

Would you like to open the issue? Otherwise, I can do that and quote what you've said here.

[ismail]

Looks like Debian has now adopted PEP 668 - https://salsa.debian.org/python-team/packages/python-pip/-/blob/master/debian/NEWS

[robtaylor]

@carlocab This would be great to open - I've been riding an issue in Meson on where to install on homebrew for a while and no progress on resolving any time soon. I think this could help!

[eli-schwartz]

This appears to have been recently implemented? And now GitHub Actions installs of brew install python are broken and do not work.

Time to pass --break-system-packages in yet more places. And include a comment clarifying that it is NOT broken, just badly named.

[p-linnane]

I'm referring to minor versions, not patch versions. If you install [email protected] it will not be upgraded to [email protected]. If you install python that is simply an alias for whatever we have set as the default. 3.12 was promoted to default a few weeks ago, thus kicking off all this discussion.

[eli-schwartz]

Ok so why is this out of scope for homebrew to the point where you are suggesting people should use pyenv to avoid getting upgraded?

[p-linnane]

Because the vast majority of users run brew install python, or brew install python3. That will auto upgrade, since it's just picking whatever our current default is. Users can stick to a support minor release by installing it manually (brew install [email protected]), but as it moves through the deprecation cycle that will introduce its own problems obviously.

[eli-schwartz]

And the solution for this is that such users should use pyenv instead of brew install [email protected], because... why?

[woodruffw]

Homebrew distributes versioned Pythons a la [email protected] in part because we have formulae that depend on them.

Users should use pyenv if this does not align with their support requirements for a Python distribution. More precisely, if any of the following are true:

1. You need a specific patch version of Python.
2. Your application stack assumes that python or python3 do not change versions under the hood.
3. Your application stack isn't compatible with the Debian/Homebrew concept of a "system" Python distribution, i.e. can't use a virtual environment in the presence of PEP 668 enforcement.

...then you're probably better served by a pyenv-built Python than the Homebrewed one. The Homebrewed one may work, but this is not guaranteed.

[iMichka]

@pradyunsg we have now implemented this starting with Python 3.12: Homebrew/homebrew-core#150390, and Python 3.12 has become the default Python 3 in Homebrew so more users will be made aware of the change.

[fiftin]

I just installed gnupg via Homebrew and system's python3 has been broken... It is very sad (Ansible doesn't wok). Is it possible to restore system python3 as default without uninstalling Homebrew's python3?

[fiftin]

Hm, looks like problem is different. Now I have 2 pythons in Homebrew and Ansible tries to use old but can't :)

[fiftin]

Or, pip3 reinstall ansible helped.

[robtaylor]

Eli, I suggest worked through Concrete use cases would help your communication here, and check your emotions. Personally I just want meson to actually work with homebrew, and that doesn’t seem a lot to ask. All conversations have ended with you ranting about things I don’t understand and no progress being made. If your aim is to support your users, maybe start calmly from that perspective- again worked concrete usecases that the approach agreed on by the python community fails in would be a helpful way forward for you.

…

On Wed, 21 Feb 2024 at 22:16, Eli Schwartz ***@***.***> wrote: Previously fixed for Debian in ***@***.*** <mesonbuild/meson@5c479d7> Pending homebrew fix in ***@***.*** <mesonbuild/meson@a35d4d3> — Reply to this email directly, view it on GitHub <#3404 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB4O4GB5A7HEHZ2KLJR4WDYUZW5PAVCNFSM52QIMB52U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQNJUHE2DSMI> . You are receiving this because you commented.Message ID: ***@***.***>

[robtaylor]

Btw for reference, I am packaging a complex python project with large c++ dependencies: https://github.com/lip6/coriolis, so your concerns are valid to me. We’re currently forced into a fork of meson to support our macOS community.

…

On Wed, 21 Feb 2024 at 23:20, Robert Taylor ***@***.***> wrote: Eli, I suggest worked through Concrete use cases would help your communication here, and check your emotions. Personally I just want meson to actually work with homebrew, and that doesn’t seem a lot to ask. All conversations have ended with you ranting about things I don’t understand and no progress being made. If your aim is to support your users, maybe start calmly from that perspective- again worked concrete usecases that the approach agreed on by the python community fails in would be a helpful way forward for you. On Wed, 21 Feb 2024 at 22:16, Eli Schwartz ***@***.***> wrote: > Previously fixed for Debian in ***@***.*** > <mesonbuild/meson@5c479d7> > > Pending homebrew fix in ***@***.*** > <mesonbuild/meson@a35d4d3> > > — > Reply to this email directly, view it on GitHub > <#3404 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAB4O4GB5A7HEHZ2KLJR4WDYUZW5PAVCNFSM52QIMB52U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQNJUHE2DSMI> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

[woodruffw]

Thank you @robtaylor; I personally appreciate your level-headed response.

Nobody here wants to break users; adopting PEP 668 is (from my perspective) an important stepping stone towards reducing the number of error states that users of the Homebrewed Python frequently end up in. As the PEP and its original discussion thread note, it's necessarily a tradeoff between various stakeholders in the huge Python community; it's unfortunately impossible to simultaneously make changes and preserve every Python package deployment patterns.

In terms of improving things here, I think there are two pieces we need:

1. We need to know what specifically about meson's current use of the Homebrewed Python conflicts with the PEP 668 change. Most Python developers and Homebrew maintainers/members aren't experts on meson, so giving us a concrete example of breakage that we can reproduce and work through would be a huge help.
2. We need to determine whether this can be fixed while retaining PEP 668, whether it requires some kind of (acceptable) upstream workaround, or whether it can be "fixed" in user-facing documentation.

[eli-schwartz]

I'm not sure what you're asking for.

The use case like mine is mixing software from multiple sources together. To reproduce the issue, simply pip install $anything, but in particular anything that isn't packaged by brew already. Recently quite a few homebrew python packages have been removed, as well... of course, not big stuff such as numpy/scipy, but that then begs the question of "how to use those packages".

The first stumbling block encountered was by following the instructions at https://docs.brew.sh/Homebrew-and-Python#setuptools-pip-etc (which appear to not have been updated).

...

Again, keep in mind this is all running inside $GITHUB_ACTIONS environments, which means that the entire operating system is essentially one big virtualenv and doesn't need nested ones. Loosening the restrictions on CI platforms would be very helpful.

[woodruffw]

I'm not sure what you're asking for.

I'm looking for a set of steps that I can run locally (or in GHA) to reproduce the problem. I use GHA pretty extensively (including with the brewed Python) for my Python projects and haven't had pip install issues, so it would be helpful to get a link to a workflow that's failed, or an example action YAML that I can try myself and experiment with.

Again, keep in mind this is all running inside $GITHUB_ACTIONS environments, which means that the entire operating system is essentially one big virtualenv and doesn't need nested ones. Loosening the restrictions on CI platforms would be very helpful.

I agree that, conceptually, an ephemeral CI runner is one big virtual environment. However, that's not the layer of abstraction that Homebrew operates at, and hacking around CI environments in particular is likely to be too burdensome to attempt in Homebrew itself (as well as unsound in practice, since not every CI system is ephemeral like GHA -- consider self-hosted GHA runners or Jenkins).

(And note, there are idiomatic software patterns in Python where a single environment is not sufficient, e.g. software deployments where individual Python subcomponents have conflicting dependency requirements.)

[woodruffw]

@mlindner With all respect: you aren't positioned to dictate the project's job.

Homebrew is used by millions of people, including people with huge ranges in technical abilities (and the ability to break things that they can't personally unbreak). Homebrew's adoption of PEP 668 is consistent the project's historic approach of eliminating errors from the +2 SD case, leaving power users to do things off of the beaten path. That's how we approached bottles, it's how we approached build flags, and so forth. Your "user hostile" is our "makes our tool more misuse-resistant for 98+% of our user base."

(Going forwards, I'm not going to respond to this thread unless it gets updated with debugging steps. The offer still stands to help figure out how we can improve the behavior here, but that cannot happen until I have a better understanding of what's actually broken and whether it's something we actually want to support vs. being intentionally discouraged per PEP 668.)

[p-linnane]

@mlinder We help users install things in the most sustainable way we can with our limited time as volunteer maintainers. If you aren't familiar with a BSD 2-Clause license, you may be interested in this part at the bottom:

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

We owe you nothing. We are not "user hostile". We spend our free-time supporting this project. We decline to spend said limited volunteer time on things we do not think make sense for the majority of users. Your opinion is appreciated, but needs to be presented in a respectful way.

[netfortius]

I ended up here post a recent update of homebrew, which installed python 3.12.2, but missed 80-90% of the packages which existed in the previous version (/usr/local/lib/python3.11/site-packages/), so almost no python programs in need for some modules work now (the dreaded "ModuleNotFoundError: No module named '...').

Attempting to fix this "the old way" (python -m pip install --upgrade ) produces the "new"

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
xyz, where xyz is the package you are trying to
install.

Any suggestions on how to fix missing site-packages (brew install does not find specific missing modules)?

EDIT as I needed my python stuff to still work, and ASAP, I ended up with the most rudimentary solution possible: copied the content of /usr/local/lib/python3.11/site-packages/ into /usr/local/lib/python3.12/site-packages/. I know this is an unsustainable method, but short of blowing up the homebrew environment, and moving to a per individual package maintenance - with a total install of everything I need, one by one - I think this is all that can be done. I also understand and I remain very thankful for the extraordinary effort the homebrew community has put into this project, which made macos so much more useable in the past.

[iMichka]

We do give some recommendations on how to fix do it:

    If you wish to install a non-brew-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.

    If you wish to install a non-brew packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

The idea is that you should not use the "global" site-package provided by our Python, but use virtualenvs instead.

[SMillerDev]

You can still do whatever you want. You're just hindered in your ability to do things the Python project thinks you should not do.

[woodruffw]

PEP 668 is a standards track PEP which means that, by definition, it is normative. You can refer to PEP 1 for the distinction between standards, process, and informational PEPs. Concretely: informational PEPs are the least normative of the PEP categories, which PEP 668 is not.

Distributions can choose to ignore standards track PEPs if they want when it comes to packaging standards, but we don't want to. We want to do the standard thing, and our reasons have previously been extensively explained. Language lawyering over what we "have" to do does not enter into it.

[eli-schwartz]

There are plenty of other viable package managers if you don't like how we handle our project. Feel free to check out our friends at MacPorts or Nix. But I'd caution you that you probably won't be able to throw a fit and bully them into doing what you want either. That's not how large, sustainable open source projects work.

Incidentally I just checked macports and it appears their Portfile doesn't claim to be externally managed.

Nix is... an interesting topic. The only reason to use Nix is if you want all software on your system to exist in separate sysroots. It's not python specific, but depending on when you built a python package in Nix, you'll have multiple different copies of python 3.12.2 so that your python packages can depend on the recursive checksum hash of python's build recipe including micro version, list of patches applied, and micro version of all dependencies recursively.

Upgrade zlib with a trivial backport of a CVE? Rebuild python against it and get two python copies. Since python has been rebuilt, rebuild all your applications against the new python. They're byte-identical copies of python other than the RPATH entry pointing to the checksum hash directory containing the zlib library, but that's what Nix is intentionally supposed to do, so the behavior is correct.

I'm not sure what a pip install --user on Nix looks like. You'd probably have many problems with it, just none of them would have anything to do with "externally managed".

They essentially shipped every package / formula in a virtualenv, even the non python packages.

It wouldn't make sense to bully Nix about this, because that would be like bullying water to stop being wet. So foundational to its identity that the suggestion doesn't make sense.

[p-linnane]

I can't speak to how other projects do things, but I can link them and suggest someone unhappy with Homebrew takes a look at them. They might be a better fit.

[MikeMcQuaid]

PEPs are not mandatory things that must be used.

Regardless: we get shouted and and called idiots when we follow PEPs and shouted and and called idiots when we don't.

No other language community seems to have these issues.

Alternatively, maybe homebrew leadership will start listening to the userbase and stop their tirade of user-hostile actions...

It's much easier to listen to people who don't throw around words like "garbage" to describe the dedicated work of volunteers in their free time.

[p-linnane]

@mlindner Based off your GitHub history all you really do is go around to different repos and complain about things. I'm not sure if your parents taught you this, but no one wants to deal with someone who is always negative and complaining. You've contributed nothing constructive to this conversation, like others have even though they disagree with how we handle things.

@MikeMcQuaid is a colossal asset to this project, and it would be dramatically worse off without him. The opinion of the original developer is irrelevant in my opinion, and they have not been involved in this project for a very long time. You have no idea what it takes to run a project of this scale, and your complaints are no longer welcome in this discussion or the project as a whole. I'm permanently banning you from interacting with Homebrew on GitHub, which should come as a relief to you since this project has continuously been going downhill for so many years with millions of users.

[yuntan]

I can't install packages by pip3 install --user xyz using [email protected], getting annoying error "error: externally-managed-environment". Why is the flag --user not works? --user flag means installing packages in home directory (my understanding), so installing packages doesn't modify system environment. very very annoying.

[yuntan]

pynvim is essential package for neovim. It must be installed globally.

[woodruffw]

Why is the flag --user not works? --user flag means installing packages in home directory (my understanding), so installing packages doesn't modify system environment. very very annoying.

The error says "externally managed environment," not "system environment." Both the system package directories and the user package directories are considered "externally managed," since both are subject to Homebrew's versioning changes and decisions around bumping Python versions.

If you absolutely need to install pynvim in a globally reachable environment, then you can pass the --break-system-packages flag or set it in your pip.conf. I'm not super familiar with neovim, but that shouldn't be necessary: if you use virtual environments, then you should be able to pip install pynvim into a venv and set g:python3_host_prog to point to that venv's python wrapper.

[pradyunsg]

Why is the flag --user not works?

To quote from PEP 668:

This applies both to system-wide installs (sudo pip install) as well as user home directory installs (pip install --user), since packages in either location show up on the sys.path of /usr/bin/python3.

Basically, stuff you can install in --user can still break stuff that's installed by Homebrew because of how Python's import system works.

[eli-schwartz]

I would just like to make it clear that my disagreement with "externally managed" python doesn't mean I necessarily agree with any other criticisms in this thread.

But in particular:

The original developer has even disagreed with how you handle the project. You should hand over the project to people who understand its users better.

It's not obvious to me that the author of "yet another cryptobro scam" understands users at all, let alone better than homebrew. It's also not obvious to me that someone who created something 15 years ago is necessarily the best analyst of what users today need, which... may explain why the original creator of homebrew is now trying to start "homebrew but as a cryptobro scam".

I also think that homebrew has drastically improved since its initial introduction in 2009, and not everyone agrees that it was any good, or any reliable, when it was first created. It is... theoretically possible... that later homebrew leads were responsible for, to put it bluntly, making homebrew popular with users.

The "externally managed" python is relatively small potatoes here. I can disagree with the decision without thinking that homebrew is being destroyed by its leadership and doomed to extinction.

Homebrew is doing a lot of great work! Your efforts are appreciated. ᵉᵛᵉⁿ ᶦᶠ ᴵ ʷᶦˢʰ ᶦᵗ ʷᵒᵘˡᵈ ᵈᵒ ᵇᵉᵗᵗᵉʳ ʷᵒʳᵏ ʷ.ʳ.ᵗ. ᵖʸᵗʰᵒⁿ

[p-linnane]

You have been voicing your disagreements with something we are doing, while passionately, with respect. That's all we really ask here, and I do appreciate that.

[MikeMcQuaid]

@eli-schwartz Thank you for the kind words. This is why your voice is still very welcome here ❤️

[pradyunsg]

I haven't been keeping up with the discussion here (life and all) and it seems like there's a bunch of pushback and vitriol from a few vocal users who passionately disagree with this change.

To these folks: Please be respectful to Homebrew maintainers here, especially please don't go down the route of name-calling, or berating their work on Homebrew where they're doing to build something extremely useful overall. Nudging them away from following broader upstream standards that are intended to protect less experienced/familiar users against subtle failure modes is something you're OK to argue for, and having a discussion about the tradeoffs involved is also fine. Berating or name-calling them is not.

PEPs are not mandatory things that must be used.

Indeed.

To the Homebrew maintainers: Thanks for adding this file, and adopting these protections. I've said this elsewhere and, is hopefully obvious already to some of you: I think this is a good thing in the long run, especially for anyone who is using Python for the first time since it prevents a very common "I don't know what I'm doing" mistakes by replacing a subtle "this could mess things up if you don't know what you're doing" case with a clear error message.

As is hopefully also evidenced by the number of people who were either directly authors of the PEP in question or otherwise engaged in the discussions around it, this is a change for the better, even though it does break certain user workflows (in that they now need to pass an additional flag, or set an env var or run a command once to update a config file).

Thanks also for dealing with the pushback here amicably -- I realise it's not my fault per se what Someone Else On The Internet™ does and, I do also recognise that I nudged y'all to do this. 😅

[MikeMcQuaid]

To these folks: Please be respectful to Homebrew maintainers here, especially please don't go down the route of name-calling, or berating their work on Homebrew where they're doing to build something extremely useful overall.

Thanks @pradyunsg ❤️ 🙇🏻

PEPs are not mandatory things that must be used.
Indeed.

Honestly: I don't super love this, though 😢.

Frankly, when we haven't followed PEPs like this in the past, we have on occasion felt bullied into submission to follow them.

In this case, in your original post, you said "Homebrew should" do this. Not "could" or "may" or "it's totally up to you".

I really appreciate, as above, you asking people to be respectful but: I would also love to see Python take some responsibility for end-user frustration here when we're following upstream guidance.

[matthew-brett]

Sorry to butt in here - but I was going to make the point that @eli-schwartz made well below - it's a judgement call whether Homebrew is an externally managed distribution. I would say no - because the user just can't choose to use Homebrew Python packages instead of the usual pip packages. So Homebrew isn't a distribution of packages, and nor should it be. It's not a valid alternative to do brew install python-somepackage in general. Whereas that may be true for something like Debian.

[pradyunsg]

Honestly: I don't super love this, though 😢.

Neither do I, but that's the socio-technical reality of how things have evolved over longer than a decade. 😅

[netfortius]

My $0.02, with all respect due to the hard working folks maintaining homebrew, for people like me to use: if I run an upgrade that blows up my entire working environment, the feeling is one of microsoft déjà-vu, where the process of upgrade doesn't offer any hints and choice for conducting it safely, vs leaving the host in an unusable state. I know there are workarounds, but the next thing one needs to immediately consider, post a recovered_from_last_upgrade, is to how to get out from under homebrew. Thanks again - sincerely!

…

On Mon, Mar 4, 2024, 8:19 PM William Woodruff ***@***.***> wrote: PEP 668 is a standards track PEP which means that, by definition, it is normative. You can refer to PEP 1 for the distinction between standards, process, and informational PEPs. Concretely: informational PEPs are the least normative of the PEP categories, which PEP 668 is not. Distributions can choose to ignore standards track PEPs if they want when it comes to packaging standards, but we don't want to. We *want* to do the standard thing, and our reasons have previously been extensively explained. Language lawyering over what we "have" to do does not enter into it. — Reply to this email directly, view it on GitHub <#3404 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABS4BN4FAH6WYK7UQG5JMTYWTCMVAVCNFSM52QIMB52U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQNRXGEYDMOI> . You are receiving this because you commented.Message ID: ***@***.***>

[matthew-brett]

Hi - I'd like to add my perspective here, as a very long-term Python user, and a user of Homebrew Python for about a decade.

I think this change to externally-managed is a serious error - and it makes Homebrew Python useless for my (very common) case.

To summarize:

1. Homebrew does not provide, or plan to provide, the majority of packages that users install, so it is not really "externally managed", in the Linux / apt / yum sense, and
2. The "externally managed" model makes sense for people who deploy web apps, but just causes huge breakage and distraction for the rest of us, using Python for data, mathematics and science coding, and
3. The "externally managed" idea is solving a problem I've never had in 10 years installing Homebrew Python, or expect to have, of breaking the OS by installing packages via pip, and
4. I just can't use the current model - it's simply too inconvenient to install the kind of system I need - and nor can I support my students who will have no idea how to deal with the cryptic messages that an ordinary pip install will generate.

So here's a humble beg from a very long-term, and committed user of Homebrew Python, and someone who has been involved in packaging for a very long time - please revert this change.

In more detail:

To be clear - I believe the use-case that the Python Packaging Authority often has in mind, is that of a developer of web-frameworks, deploying applications. And this change could make sense for those people. For those people, the primary concern is making sure that the application (framework) stays stable to multiple installations. For those people, pipx makes perfect sense.

However, I am willing to bet some significant sum, that those people are:

a) Perfectly cabable of managing their own virtualenvs and
b) in a small minority of Homebrew Python users.

I'll document this if it's helpful - but I am confident that a large majority of your users are science / data Python users, who are installing Numpy, Pandas and other scientific Python packages. For those people, "apps" are niche to the point of being irrelevant. In particular, the only "app" these users install often, is Jupyter, in some flavor - and they very explicitly need nearly all their other packages in the same virtualenv as Jupyter.

For these users - like me - and all my students (I'm a university professor) - the externally managed model is so difficult to use that we will have to switch away from using Homebrew entirely. Indeed, I've just, very reluctantly, switched to pyenv instead - which is less convenient - but at least not broken for my use-case.

Why? Because in practice - and I have a lot of practice, the way that I and many others work, is to have one default environment, in which we install everything, where everything is nearly always Numpy, Pandas, scikit-learn, matplotlib and other pretty stable sets of packages that rarely conflict. This is how we get students started with their installs - example : https://lisds.github.io/dsip/installation_on_mac.html. Variants on those instructions were a simple route into a working Mac installation for beginners. But now it isn't, because the beginner will be told a) they can't use pip to install packages and b) be pointed to "virtualenvs", which they do not currently need, nor understand and c) will in particular be guided to pipx which is more or less exactly the wrong tool for what they need, which is just to:

pip install --user -r requirements.txt

So the "externally managed" solution, is solving a problem that I and my students don't have, at the expense of making it much harder for them to get started without learning a whole lot more about Python packaging and virtualenvs. So I think, in practice, we'll have to either downgrade to Python 3.11 and hope y'all change your minds on this one, or find another way of installing Python that doesn't cripple standard default installation. And I suspect - I say this very sadly - that the result will be that you will have a very sharp drop in the number of people using Homebrew Python, for virtually no gain in terms of real problems solved.

[eli-schwartz]

It is Homebrew's call and we are choosing to go with the direction we are advised from the PEP. I find it disingenuous for PEPs to be published and then it implied that it's up to us to follow them.

I would instead argue, that PEPs are published and it is up to homebrew to decide whether the PEP describes your use case.

The PEP describes how to declare yourself as an externally managed environment, it doesn't say you're an externally managed environment and thus must describe yourself as one.

It is Homebrew's call whether Homebrew is an environment that:

• expects to be broken if users install unapproved modules. (This depends on what a homebrew environment is. Does homebrew python get most of its useful software from brew or are users expected to install their own as a default workflow?)
• does NOT expect users to be a collaborative part of growing the Homebrew python environment. ("Power users" is the most obvious answer here. Environments that target power users are not "externally managed" even if there's a very large amount of package manager managed packages in it.)

Homebrew has made the call that it's an externally managed environment. I don't find it disingenuous to say that Homebrew is allowed to make either call, according to whatever it feels like.

And saying that Homebrew "should" make the call that it isn't one and therefore shouldn't implement the PEP isn't disingenuous for saying Homebrew shouldn't follow the PEP, it's merely a difference of opinion about whether or not following the PEP means making the call that the PEP is applicable.

If there's a way of somehow configuring our Python to do this (without causing even more breakage/complaints): this seems like a very good suggestion and then pip install --break-system-packages is the flag you should use to install Homebrew-wide (which seems appropriate).

In theory one could have homebrew install a pip.conf that enables the --user flag. That would make it practically impossible to install Homebrew-wide using pip (though it is simple to do using the low-level installer package whose CLI exists mainly for the use of distributions such as Homebrew that are packaging software into their own ecosystem and maybe don't want all of Pip's baggage to do it).

[MikeMcQuaid]

The PEP describes how to declare yourself as an externally managed environment, it doesn't say you're an externally managed environment and thus must describe yourself as one.

One of the PEP authors came in here to suggest that we should do this 🤷🏻

• expects to be broken if users install unapproved modules.

Yes, this sounds like the type of thing that:

1. often happens in Homebrew
2. we'll enable (including disrupting users) whenever we can (e.g. macOS sandbox usage, superenv compiler flag filtering, environment variable filtering)

In theory one could have homebrew install a pip.conf that enables the --user flag. That would make it practically impossible to install Homebrew-wide using pip

This sounded like a good idea until @woodruffw pointed out that it too can break things inside Homebrew.

[eli-schwartz]

To take a step back and offer another explanatory point here: apart from Python dependency problems, one of the reasons why PEP 668 reasonably restricts the user directory is because package installed entrypoints are also frequently broken by Python version changes. That alone IMO is a sufficient reason to pursue PEP 668 in Homebrew's context.

I lack familiarity or understanding on how this would be the case. Can you explain?

Entrypoints should not care about anything other than:

• having a shebang interpreter for the relevant python version
• loading the specific package used

When upgrading python in a pinned minor release line to a new patch version, my assumption is that both of these remain working.

And when upgrading to a new minor python feature release and dropping the old version, it seems intuitive to me that both virtualenvs and non virtualenvs would equally break and stop working.

And when installing a new minor python feature release in parallel to the one you already have, the topic is irrelevant since neither your entrypoints nor your python nor your user environment have changed at all.

Under what scenario do you foresee an entrypoint breaking when using the default environment, but not breaking when using a virtualenv?

[matthew-brett]

Under what scenario do you foresee an entrypoint breaking when using the default environment, but not breaking when using a virtualenv?

Yes - I think that's the key point. I personally can't think of a realistic scenario by which a --user installed package could break a Homebrew package - where installing in a virtualenv would not - but can someone come up with one to help us think about that?

[matthew-brett]

Homebrew is a general tool, and attempts to strike an (in practice, impossible) balance between a large number of user cases. That includes both cases like yours and ones that you identify with the web developer group (although IME as a non-web Python developer, this cohort is really the "everyone except data scientists" group 😉). In practice, that means leaning on the side of PyPA standards, as part of a larger level for bringing Python's different communities into alignment on packaging. At least, that's how I see it.

Yes - I don't really care whether the group I'm referring to is called "data scientist" - as long as you agree on the distinction between well-trained developers (I assume everyone on this discussion) who wouldn't miss much of a beat being forced to use virtualenvs, and students of coding or data analysis who aren't trained in software engineering - who would miss quite a few beats - enough to prefer installing Python some other way. And I was arguing that this second group is quite a lot larger, and needs more help, and that should be part of a judgment call on this change.

It's possible I guess that the PyPA believes in the rule "Never install with pip unless you're in a virtualenv" - and as a corollary - "Never do a --user install". But is that really true? I hope not, as that seems to put a peverse burden on new users, forcing them to understand and manage virtualenvs when they may never need them. "practicality beats purity", as they say.

[matthew-brett]

And otherwise install to --user?

If I was the upstream Python project: I would have instead made pip install default to --user in this case. I don't have enough context to know why they did not 🤷🏻

Totally agree. I think this was just historical - that they didn't want to change pip's behavior. Then they decided (I think) that there wasn't much advantage to giving a permissions error over just installing to --user if the user didn't have permission, giving the current behavior.

If there's a way of somehow configuring our Python to do this (without causing even more breakage/complaints): this seems like a very good suggestion and then pip install --break-system-packages is the flag you should use to install Homebrew-wide (which seems appropriate).

Yes - that seems like an excellent trade-off to me - and I'm sure @eli-schwartz would also agree that, for that case, --break-system-packages would be a perfectly acceptable level of terror to induce. Then - isn't it just a case of patching pip to insist on sudo for a system site-package install?

This is a change of course, in that packages that used to go to site-packages now go to --user . But it must be rare that this will cause a problem - I'm failing to think of anything immediately. In particular, the user will now need to put the --user binary path on their $PATH - but pip already gives good warning about that - and it's fairly easily fixed.

This unfortunately can still result in breakage: as @pradyunsg points out, Python's import system is agnostic to the concept of "user" versus "system" paths, which is why PEP 668 speaks in terms of "externally-managed" paths rather than user or system ones.

Hmm - so - as I said - even with the previous behavior - I'm pretty sure I never broke a Homebrew package with a pip install. So it must be rare. Is the argument that it will be no more rare with the --user install change? I guess this will depend on the PYTHONPATH of the Homebrew use of Python. Does it include the --user paths? But even so - won't they be last on the PYTHONPATH - meaning that the chances of breakage must be very small?

[MikeMcQuaid]

Then - isn't it just a case of patching pip to insist on sudo for a system site-package install?

Again with the just 😉.

We're not going to indefinitely carry a patch like this for pip. If upstream introduces it as a potential feature like this (and that's between you and them): we'd consider using it.

[matthew-brett]

Again with my saying sorry - honestly, in my linguistic world "just" doesn't mean "why didn't you think of it?" but "would this not be a not-invasive small change to get the effect desired?".

[MikeMcQuaid]

No worries 😁

[matthew-brett]

JFYI, this ended up in a parallel thread, not the parent one. But I'll respond here 🙂

I see that idea - but I still don't understand it? It is the shakiness restricted to the problem that the binary path with change when upgrading Python and that may surprise users? Otherwise, I'm not sure where the shakiness lies, compared to virtualenvs.

It's both that, and also the fact that the global Python environment is fundamentally not "consistent" in the way that isolated environments are -- pip and other installers have historically had to be very loose about conflicting dependency constraints because users have expected pip install --user ... to "just work," even when it's been filled with years of installs from different Python versions, different pips, etc. You can see a lot of this pain in the other things that have cropped up with 3.12, e.g. packages assuming that environments always contain setuptools, etc.

Trying to unpack the argument here - it is that, if there is one default installation location (be it --user your default Conda env or whatever) then that can get messy, the mess causes some constraints to pip development, and so it would be better to push people forcibly away from using a default installation location?

Well - speaking as someone who has been training researchers in reproducible research for a long time, and doing my own research, there is a small subset of users who really need Conda and Docker - and it would do a lot of harm to our teaching if we forced all our users to do that, for the sake of that restricted set of cases.

This is fair. I recognize that a learning environment is not necessarily an excuse (or the appropriate venue) to make students do burdensome things that are only right for a small percentage of cases.

Right - and of course I'm arguing that disabling --user installs is such an example.

With that being said, Python packaging is in a triple(?) bind here: there are (1) people who mostly follow the PEPs, (2) people who do things consistently but their own way, and (3) a huge tail of people who are just doing whatever appears to work (and they can't be blamed for this!). PEP 668 is, to my interpretation as a non-author, an effort to improve life for (1) and discourage (3), at the cost of some heartburn for (2) (and some but not as much heartburn for (1)). The alternatives to this is stasis (which I don't think anybody really wants) or more "spinout" tools that only implement the standards/pieces they care about, adding more inertia to packaging.

I think I'm not following - which group is someone like me in - who uses --user installs by default, then drops into a virtualenv for a more complex project? Aren't I (up until now) in group 1? And so yes - I get heatburn because now I can't use --user installs any more. But to what benefit? Is that so that the group who are doing something else in group 3 will realize that they should neither be doing what I was doing nor what they were doing before but use virtualenvs instead? Could we not have less heartburn by allowing --user installs (both consistent, and PEP consistent), and encouraging use of virtualenvs otherwise? And is that not exactly what we get by setting --user installs to be the default?

[woodruffw]

I think I'm not following - which group is someone like me in - who uses --user installs by default, then drops into a virtualenv for a more complex project? Aren't I (up until now) in group 1?

That puts you in (1) and (3), at least from my perspective: virtualenvs align with (1), and using a global environment for everything else is (3).

Sorry, there's too much to immediately respond to here (not your fault, I just have trouble following multiple threads). I'm still talking with the other members and some PyPA folks about how we can improve the situation here while still sticking to 668.

[matthew-brett]

Sorry, there's too much to immediately respond to here (not your fault, I just have trouble following multiple threads). I'm still talking with the other members and some PyPA folks about how we can improve the situation here while still sticking to 668.

In case it's helpful - I can't imagine a popular and successful solution that forces beginners to learn and use virtual enviroments. For example, Conda and Pyenv do provide default environments, so users can happily install with default install instructions until they realize they need isolated environments, and that's when they learn about creating and switching environments.

If we agree on that - I still see no strong argument against the --user install being that default environment, for default Python.

[MikeMcQuaid]

If we agree on that - I still see no strong argument against the --user install being that default environment, for default Python.

The strongest I can see so far based on internal discussions is "upstream does not provide a way of making --user not require --break-system-packages if you implement PEP 668 (without writing and carrying our own Homebrew patch indefinitely which we're not going to do).

[matthew-brett]

Yes - I think we agree that the only significant argument against the --user install is that the PEP is deliberately breaking it.

Does it also break --user installs if you have:

[install] 
user = true

in your pip.conf? If so - ouch! If not - then I guess a rather sad but perhaps necessary compromise would be to add a trivial command line tool that simply inserts that into the user's pip.conf, and suggest that as a prominent option when the user tries to pip install something.

[matthew-brett]

But I would strongly argue that, if Homebrew can default to --user installs, the PEP brings serious inconvenience without benefit, and that would be strong weight in favor of choosing not to identify Homebrew as a "Managed distribution".

[matthew-brett]

If we agree on that - I still see no strong argument against the --user install being that default environment, for default Python.

The strongest I can see so far based on internal discussions is "upstream does not provide a way of making --user not require --break-system-packages if you implement PEP 668 (without writing and carrying our own Homebrew patch indefinitely which we're not going to do).

And I guess agreeing with your implicit point here - it's very difficult for me to see why the --user flag should require --break-system-packages - given we've established there is no realistic way that a --user install can in fact break system packages.

[matthew-brett]

Just checking - so there is some system for which a --user install can break the system packages? How does the --user install break the system install? Isn't it designed not to do that?

[woodruffw]

To my understanding, there is no particular guarantee that the user-site is loaded after the system-site. In practice, some distributions choose to invert them (or do all kinds of patching/special casing), due to users running pip install --user setuptools or similar and being confused as to why their newer version isn't being used.

There's some original context in this pre-PEP discussion: https://mail.python.org/archives/list/[email protected]/thread/3TG2EAD4UVHLISDRU77GQKPZNNKMLZST/

[eli-schwartz]

I believe the PEP's scope was intended to include observed breakages on Linux distributions and other hosts as well, and Homebrewed Python's use on Linuxbrew is also a consideration for us.

It's probably also worth noting that the major reason Linux distributions could "afford" to say "we are externally managed, don't touch it, don't even use --user" is because e.g. Debian doesn't actually recommend using a virtualenv at all. They recommend installing whatever you wanted to install with apt.

This suggestion generally works because Linux distros are in the business of packaging "everything" subject to user demand, so it's always in scope to simply add another package recipe to the distro in the event that users tried to install a package that isn't available.

My understanding is that Homebrew explicitly isn't interested in packaging every single semi-popular module on PyPI which means that the olive branch to users which Linux distros have, Homebrew doesn't have.

Debian sees venvs as something only expert users -- probably python developers anyways -- are going to use. They even chose to delete the "venv" module from their python3 recipe and ship it as a separate recipe that isn't installed with python itself. Because they're that committed to the idea that users should "simply install everything via apt" and venvs are just for edge cases.

In that context it makes sense why they'd be okay telling pip install --user users to use venvs. They simply don't expect it to affect many people.

[matthew-brett]

Does Debian intend to deploy this? I notice that it really doesn't package very much , so, if you want a Python in which you can install most packages, you'll still be forced into virtualenv-only land. I suspect they will also see a lot of people switch to other methods of installing Python, rather than bear that inconvenience. In practice I suspect this PEP, as currently implemented, will push many people onto Conda, and have little effect on the usage of virtualenvs.

[matthew-brett]

That must be rare, that a distribution puts --user installs ahead of system installs? I have never come across it.

But in any case - why not solve that pathological case by adding an extra configuration flag to say:

USER_BREAKS_SYSTEM=True

which would then insist on --break-system-packages even with --user?

[woodruffw]

We've discussed this a bit internally, and have come to the conclusion that the "system" side of PEP 668 is still a desirable property for Homebrewed Python. Given that PEP 668 doesn't currently give us a way to indicate that that some managed Python distributions aren't necessarily broken by user-site installations (i.e. pip install --user) and that we do not want to patch pip, this is what we're considering for a path forwards here:

1. For the time being, retain PEP 668 via EXTERNALLY-MANAGED;
2. Update EXTERNALLY-MANAGED with some additional guidance for users:
   • Add a note about --break-system-packages --user being an acceptable pattern for many use cases;
   • Add a note about configuring pip.conf with break-system-packages by default, so that users only have to do it once;
3. Add a brew doctor check for break-system-packages in pip.conf, to help us with Python issue triage;
4. Investigate ways to further stymie sudo pip <install|uninstall> --break-system-packages, since users may mis-read the notes in EXTERNALLY-MANAGED as encouragement to use the non---user form.

I'm curious to hear what people think of this: it's probably not the ideal solution for everyone (things are still harder than before for --user users, and Homebrew is offering a loophole in its policy of "Homebrewed Python is for Homebrew"), but is hopefully a UX improvement over the status quo.

[pradyunsg]

Note that --user can break a project/packagr because there's a file on the import paths that it did not expect (eg: an older/newer version of requests in user vs system locations). I don't have a handy reproducer, and it might be a "good enough" solution for the problem here tho.

[matthew-brett]

Yes - a not-pathological reproducer would be handy - at the moment I still don't understand why the system wouldn't pick up the system requests.

[PatTheMav]

@woodruffw Would it be worthwhile to pin your comment? Because I think allowing --user should be a good-enough solution for Homebrew (and for now --break-system-packages works, even though it's scare-naming makes me chuckle) and it might be worth highlighting that the Homebrew team agrees but there's a compromise for the time being.

On another note, you state "Homebrewed Python is for Homebrew", though I haven't found that stated in https://docs.brew.sh/Homebrew-and-Python. This seems counterintuitive insofar as Python is always installed to be available in Homebrew's binary path and thus supersedes any other global Python version installed on the system (e.g. the system Python interpreter provided by Xcode on macOS). If Python only exists as a dependency for Homebrew formulas, then it should not be symlinked by default (at this point such a change would probably break even more Python-based use cases, but you get the point).

This makes use cases like pip-provided CLI tools (e.g. cmake-format) even more difficult to resolve, especially compared to npm, where all I have to do to get prettier to work in Terminal is npm install -g prettier and call it a day (though that might be more pip's conceptual issue to solve).

[woodruffw]

@PatTheMav Yes, I think this is probably worth pinning, although I don't have perms to do so 🙂 -- CCing @iMichka, who I think can pin it.

On another note, you state "Homebrewed Python is for Homebrew", though I haven't found that stated in https://docs.brew.sh/Homebrew-and-Python. This seems counterintuitive insofar as Python is always installed to be available in Homebrew's binary path and thus supersedes any other global Python version installed on the system (e.g. the system Python interpreter provided by Xcode on macOS). If Python only exists as a dependency for Homebrew formulas, then it should not be symlinked by default (at this point such a change would probably break even more Python-based use cases, but you get the point).

Yeah, point taken. Python is a painful situation for Homebrew (I say this both with Homebrew and PyPA hats on): we can't make it keg-only because ~hundreds of thousands of people use it directly and want it to shadow the system Python (which Apple has repeatedly signaled is not going to be around forever). At the same time, the Python packaging ecosystem has (IMO appropriately) signaled that non-system-managed base prefixes are essentially a dead-end and a significant drag on efforts to modernize Python's packaging workflows/tooling.

In a principled world, Homebrewed Python would probably be keg-only and would have to be explicitly linked in. But the cat is bag on that, so we have to balance real-world users with Homebrew's own interest in maintaining the integrity/consistent behavior of Homebrewed Python, even when users do weird things like sudo pip install ....

So, when I say something like "Homebrewed Python is for Homebrew," what I mean is "Homebrewed Python should be difficult for users to break in such a way that it breaks formulae that depend on it, but we can't actually isolate it properly because that would break even more users." But that's a mouthful.

This makes use cases like pip-provided CLI tools (e.g. cmake-format) even more difficult to resolve, especially compared to npm, where all I have to do to get prettier to work in Terminal is npm install -g prettier and call it a day (though that might be more pip's conceptual issue to solve).

Yeah, this a longstanding issue, and I think it's more generally the PyPA's responsibility to solve. In the case of Homebrew, there are two resolutions, both of which are noted in the EXTERNALLY-MANAGED message:

1. Many Python tools are packaged as Homebrew formulae, and we encourage people to create formulae for ones that aren't! cmake-format seems like a great candidate for inclusion in homebrew-core.
2. For tools that don't meet the requirements for homebrew-core, we recommend people use pipx: pipx will create a per-tool isolated environment, sidestepping the traditional problem with pip install -U (i.e. conflicting per-tool requirements).

[PatTheMav]

Yeah the real solution for such tools is to have them properly packaged as Homebrew formulas which then get all the benefits of dependency pinning and isolation out of the box.

And there is also solution 1.5 if you will: Users with GitHub accounts can run their own custom taps and create formulas for any Python-based CLI tool they might need (I might just look into that for cmake-format, which unfortunately seems abandoned but is used in a project I contribute to).

EDIT: Alas the custom formula route is not possible as cmake-format's source distribution is incomplete and broken, and it can only be installed as a wheel (its setup.py references files not shipped with the tarball).

[matthew-brett]

Another summary email after some reflection:

Summary:

Homebrew should not use the EXTERNALLY-MANAGED system until there is a mechanism for a transparent-to-the-user default virtual environment.

I have a proposed solution at the end.

Detail:

There are several separable problems here, and the PEP is addressing these problems at the same time, causing confusion as to the motivation and desirability of the change.

The PEP assumes that it is not a major burden on the user to enforce the conscious use of virtualenvs, but I believe this assumption is seriously wrong for most users of Python. Explicit use of virtualenvs is an advanced tool, and is confusing and unnecessary for most beginners.

The separable problems:

• Defending the user from breaking system Python with a user install - Defending the system.
• A --user install cannot override system packages, but sometimes this is what the user wants. Limitation of --user install.
• Having a single default installation environment makes it more likely that users will run into dependency errors. Prefer smaller environments.

The PEP wants to solve all three problems at the same time, at the expense of forcing users to understand and manage virtualenvs. It seems the PEP authors do not believe this is a significant burden, but Eli gave a good explanation of what that burden is. Certainly, as a teacher, I cannot afford to spend multiple classes explaining Python paths, pip and package management before they can get started.

Taking the three problems one by one:

• Defending the system can be achieved by defaulting to a --user install, except in rare and (it seems to me) pathological cases. Certainly this is true of macOS Homebrew. It seems to me this is already the case on default Debian and Ubuntu distributions at least - because permissions force --user installs. So this is either a solved problem, or a problem that is easy to solve.
• Limitation of --user install. Here the problem is that the user does sometimes want to override the system packages, but cannot, because the system packages shadow the user packages. It seems to me that this is not a common problem for most users, now packages like setuptools and pip have stabilized. If you enforce --user installs, then the system packages will generally be a small subset. At least - I use --user installs myself, and have done for maybe 5 years on Homebrew, and have not run into this problem. So, yes, it's a potential problem, but it is going to be a tradeoff as to the sacrifices you are prepared to make in terms of simplicity of use, in order to solve this problem - which I would argue is rather arcane and uncommon.
• Prefer smaller environments. I don't think this concern reflects the current use of Python by the majority of users, and expecially beginners. In general, Python + games + small scripts + data coders - who are, I have argued with evidence above, a majority of Python users overall, and beginners in particular - do exactly need a large environment into which they install everything, including Jupyter, Numpy, Scipy, Pandas, scikit-learn etc. This makes tools like pipx a particular hazard and burden - because the user has to understand that a) they have a separate environment for Jupyter, and b) they have to work out the pipx incantation to install everything into the environment, and c) they will also want to use that environment for terminal IPython and text editors ...

Risk:

Users (and teachers) will quickly abandon Python installations that try and force beginners to use virtualenvs, and prefer installations that do not. In particular, I predict a rapid switch to Conda installs, because they provide binary installers for Python and a transparent default environment.

Solution:

Given that it would be fairly easy to prefer --user installs, the problems of interest here are the limitations of the --user install, compared to the burden of requiring virtualenvs.

This problem could be solved substantially, by having a system that, instead of preferring --user installs, set up the user with a default and transparent virtualenv. Call this virtualenv default-venv . By default, without further configuration, and without the user having to know this was the case - pip install somepackage would install into default-venv. default-venv would not have the system packages on sys.path. Installed binaries (apps, scripts) for default-venv would, by default, be on the shell PATH. This would fully satisfy the needs of the beginning Python user, but without forcing them into understanding Python packaging and paths as soon as they start using Python. Some virtualenv mechanism would them allow them to do the equivalent of mkvirtualenv some-complex-web-app; workon some-complex-web-app, which would switch out the Python package path and binary path. They would learn to use this in due course, when they have some confidence with Python.

[pradyunsg]

I haven't fully caught up to the discussion here (health and family stuff) but @matthew-brett's most recent comment indicates to me that at least some of this discussion should probably be moved to https://discuss.python.org/c/packaging/feedback/37.

[matthew-brett]

I haven't fully caught up to the discussion here (health and family stuff) but @matthew-brett's most recent comment indicates to me that at least some of this discussion should probably be moved to https://discuss.python.org/c/packaging/feedback/37.

I'm happy to have that discussion there too, but for the moment I think there are two questions:

• Could this PEP be improved?
• It is reasonable to adopt the PEP in its current state, for Homebrew?

I was primarily arguing for a No, for the second, but of course I would also argue Yes for the first.

I'm arguing No for the second on the basis of greater cost than benefit, particularly for beginners, compared to the less invasive switch to --user installs by default.

But I am sure that some version of this PEP, that does allow a transparent default virtualenv, could be of great benefit compared to relatively small cost.

[woodruffw]

Users (and teachers) will quickly abandon Python installations that try and force beginners to use virtualenvs, and prefer installations that do not. In particular, I predict a rapid switch to Conda installs, because they provide binary installers for Python and a transparent default environment.

This has come up a few times before: Homebrew distributes Python because millions of users install formulae that depend on it, not because the Homebrewed Python is itself intended to be the ideal distribution of Python for every potential downstream use case. If people chose to switch to Anaconda or pyenv-managed Python distributions, that is perfectly fine by us: we ship casks and formulae for that purpose, and we want people to use them!

(This doesn't meant that we want to break people who directly use the Homebrewed Python rather than transitively, but we stack their interests against (1) what PyPA standard and tooling are aligned with, and (2) the transitive use case. PEP 668 adoption breaks in favor of the transitive case, especially since virtual environments will continue to work and are encouraged.)

This problem could be solved substantially, by having a system that, instead of preferring --user installs, set up the user with a default and transparent virtualenv. Call this virtualenv default-venv . By default, without further configuration, and without the user having to know this was the case - pip install somepackage would install into default-venv. default-venv would not have the system packages on sys.path. Installed binaries (apps, scripts) for default-venv would, by default, be on the shell PATH

This is better suited for a PEP or pre-PEP discussion, I think: Homebrew would have to pretty invasively patch either pip or further modify the user's shell environment to achieve this, and I don't think we want to do either unilaterally.

(I suspect that the default-venv setup still has lots of problems, which is why tools like pipx create one environment per top-level package, rather than trying to merge them all into a single environment. But I haven't thought about this extensively.)

[matthew-brett]

Just to clarify again - of course it would be good if the PEP was modified so it allowed --user installs as an option, or a default virtualenv - but I'm assuming a) that will take time to negotiate, and b) it may not succeed. So the urgent question is here - whether it's worth essentially retiring the "user" component of Homebrew's current "system+user" by applying this PEP, or applying some less invasive compromise.

[woodruffw]

I think https://github.com/orgs/Homebrew/discussions/3404#discussioncomment-8724311 summarizes Homebrew's position here: we believe that adopting PEP 668 is the right approach for the time being, one that we'll revisit when a better PEP comes along.

The compromise for us is that we're explicitly not retiring the --user path, since we'll be updating EXTERNALLY-MANAGED with instructions for how to re-enable it, including permanently via pip.conf. This puts us outside of the model the PEP wants to encourage, but we're doing it because we agree that no substantial problems happen with --user when users don't configure their sys.path pathologically. When they do, we'll catch it with brew doctor. For users that find following the steps in EXTERNALLY-MANAGED too burdensome, we continue to recommend a separately maintained Python distribution, e.g. via pyenv or Anaconda.

(I'm going to open up a PR for that later today, time permitting.)

[matthew-brett]

Users (and teachers) will quickly abandon Python installations that try and force beginners to use virtualenvs, and prefer installations that do not. In particular, I predict a rapid switch to Conda installs, because they provide binary installers for Python and a transparent default environment.

This has come up a few times before: Homebrew distributes Python because millions of users install formulae that depend on it, not because the Homebrewed Python is itself intended to be the ideal distribution of Python for every potential downstream use case. If people chose to switch to Anaconda or pyenv-managed Python distributions, that is perfectly fine by us: we ship casks and formulae for that purpose, and we want people to use them!

Right - but I think it's important to concentrate on what this PEP solves, compared to --user only installs. I think the answer to that is - very little. But the problems it creates for beginning users are substantial. So what will likely happen is that Homebrew, that used to be a very useful system+user install, now becomes a system-only install - to virtually no practical benefit.

(This doesn't meant that we want to break people who directly use the Homebrewed Python rather than transitively, but we stack their interests against (1) what PyPA standard and tooling are aligned with, and (2) the transitive use case. PEP 668 adoption breaks in favor of the transitive case, especially since virtual environments will continue to work and are encouraged.)

It's clear I think that the PyPA puts high value on multiple virtualenvs - and there is a small sub-population of advanced users for whom that pattern makes sense. Perhaps PyPA wishes that it would or should make sense for more users - but if that is the plan - I am confident it will fail - because it has such little utility for beginners, and such high cost. All that will happen is that you will push users off the installs that do choose to use this PEP - onto any other install that doesn't use this configuration. And likely, that will be Conda, and likely, that will cause development effort to switch from pip to conda. Which is unfortunate - but at least we see the benefit of competition - if one system makes a decision that doesn't take the broad range of users into account - the other can reap the benefit.

Has any other substantial distribution adopted this PEP? I'd be surprised if they have - but also interested in what discussions looked like in making the decision.

[MikeMcQuaid]

@matthew-brett It's all good. I just think we've reached the limit of useful discussion here. The rest is up to the Homebrew maintainers to decide on implementation details and implement them or for you and others to take these discussion to Python itself.

[matthew-brett]

Aha - so I think I did misundestand - I think you are saying that - more or less regardless of cost - you are are going to implement the PEP, mitigating its effects as best you can.

[MikeMcQuaid]

@matthew-brett Yes.

[matthew-brett]

Would that still be true if you knew that a large majority of your users would not be in favor of such a change?

[MikeMcQuaid]

@matthew-brett Sigh. You cannot prove this. In the absence of proof, we will decide based on the principles we've used in running this open source project over (in the case of both the project and myself) last 15 years.

Your opinions have been heard and considered. It's time to stop now, thanks.

[matthew-brett]

Just throwing it out there - but I wonder whether the way forward would be to have a light fork of pip, called, say pipu. It would only be pip with patches to defend against overwriting system packages, and releases would closely follow pip. We'd describe pipu as "A clone of Pip that won't push you into using virtualenvs". The distributions would adopt this PEP, but instead of recommending options like pipx - it would recommend using pipu for all installs where users would previously have used pip, and provide a package, e.g. brew install python-pipu.

[woodruffw]

I don't think we're going to do this within Homebrew. It's morally equivalent to patching pip itself, which isn't on the table for maintenance reasons.

(It might be a good idea for your use case, however! You might also want to look into alternate installers, like poetry, PDM, pipenv, and uv.)

[matthew-brett]

Well - there's no maintainance burden for Homebrew, Pipu would just be another Pip package. Morally - I don't think this would qualify as a hostile fork, in the sense of taking someone else's work, it would just be some patches to fix an unfortunate decision that the Pip developers have made for the benefit of what I strongly believe will be most users. And there's no sense it would be an alternative installer on the sense of those other approaches, it would just be - Pip - but configured such that doesn't force you to use virtualenvs. Think of it as Pip with different default options.

[matthew-brett]

It wouldn't be much work - draft here, just making the trival fork with global replace of pip with pipu and some edits:

https://github.com/matthew-brett/pipu

It's possible that pip install --user git+https://github.com/matthew-brett/pipu already gives you a working pipu binary - just a renamed clone of pip. I haven't had time to test it much. Then it's just a question of working out the correct patches to defend against system installs.

[MikeMcQuaid]

There is no world in which Homebrew is going to use this. By all means use it with your students.

[matthew-brett]

I thought I'd add this vignette - because it may not be obvious what the extent of the problem is going to be for beginners.

Here's an example. Let's say I install Homebrew Python 3.12. I'm a beginner, interested in Python game programming, and I found this tutorial.

Very typically, for beginner tutorials, it says that I need to install pygame
with:

pip install pygame

I run that, and I see this:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-brew-packaged Python package, create a
    virtual environment using python3 -m venv path/to/venv.  Then use
    path/to/venv/bin/python and path/to/venv/bin/pip.

    If you wish to install a non-brew packaged Python application, it may
    be easiest to use pipx install xyz, which will manage a virtual
    environment for you. Make sure you have pipx installed.

note: If you believe this is a mistake, please contact your Python
installation or OS distribution provider. You can override this, at the
risk of breaking your Python installation or OS, by passing
--break-system-packages.  hint: See PEP 668 for the detailed
specification.

Remember, I'm a beginner, so I guess I try:

$ brew install pygame
==> Downloading https://formulae.brew.sh/api/formula.jws.json
#=#=#
==> Downloading https://formulae.brew.sh/api/cask.jws.json
#=#=#
Warning: No available formula with the name "pygame". Did you mean pygments?
==> Searching for similarly named formulae and casks...
==> Formulae
pygments

To install pygments, run:
brew install pygments

Maybe I do a bit of Googling, and work out that I may want:

$ brew install python-pygame
Warning: No available formula with the name "python-pygame". Did you mean python-pytz?
==> Searching for similarly named formulae and casks...
==> Formulae
python-pytz

To install python-pytz, run:
brew install python-pytz

Now I'm a bit confused, and I go back to the original message. This is the next
suggestion, but it looks very confusing, as I have no idea what a virtual environment
is:

If you wish to install a non-brew-packaged Python package, create a
virtual environment using python3 -m venv path/to/venv.  Then use
path/to/venv/bin/python and path/to/venv/bin/pip.

Presumably I would have given up on Homebrew by now, but let's say I'm brave,
and I do eventually work out that I can do this:

python3 -m venv path/to/venv
path/to/venv/bin/pip install pygame

But I still really need to know what this virtual environment thing is, that I will have
to always run path/to/venv/bin/python3 to get an interactive console in which
I can import pygame, I'll have to remember where these files are in my
computer filesystem (which is not always easy for beginners) and so on.

Ok, but now I want to use ipython or jupyter. If I've developed some understanding by now, then I might realize I need to do:

path/to/venv/bin/pip install ipython

but now I have to work out that my next step is:

path/to/venv/bin/ipython

In short, I'll have to understand what virtualenvs are and how they work.

More plausibly, the instructions on virtualenvs above look so confusing,
that I try the next instructions:

If you wish to install a non-brew packaged Python application, it may
be easiest to use pipx install xyz, which will manage a virtual
environment for you. Make sure you have pipx installed.

OK, I work out that I first need:

brew install pipx

Then:

pipx install pygame

I get:

$ pipx install pygame
No apps associated with package pygame or its dependencies. If you are
attempting to install a library, pipx should not be used. Consider using
pip or a similar tool instead.

OK - so I'm left with this final note:

note: If you believe this is a mistake, please contact your Python
installation or OS distribution provider. You can override this, at the
risk of breaking your Python installation or OS, by passing
--break-system-packages.  hint: See PEP 668 for the detailed
specification.

That sounds far too scary - I mean - the Homebrew people wouldn't give me that
warning if they weren't serious. So I conclude Homebrew is not for me and
I install something else. Probably Conda.

[matthew-brett]

@matthew-brett Sigh. You cannot prove this. In the absence of proof, we will decide based on the principles we've used in running this open source project over (in the case of both the project and myself) last 15 years.

It's hard to prove, but not difficult to estimate.

Your opinions have been heard and considered. It's time to stop now, thanks.

Yes, I understand the parameters of the discussion now, and I will stop.

I was going to suggest you consider monitoring for the effect I predict - a rapid loss of "user" installs, as opposed to "system" installs, but on reflection, I think you're saying you're prepared to accept that, and, in any case, by the time you detect it, it will be too late.

[gibfahn]

After initially being concerned about this, the actual migration (especially after Homebrew/homebrew-core#165681 landed and clarified the guidance) was pretty straightforward. pipx for binaries works nicely, and most projects are / can easily start using virtualenvs anyway.

The only question for me is whether there are libraries that "should" be installed globally. Only one I've found so far is pynvim, but also I haven't actually had any issues after removing this library entirely and trusting that things that need it will install it.

[woodruffw]

The only question for me is whether there are libraries that "should" be installed globally. Only one I've found so far is pynvim, but also I haven't actually had any issues after removing this library entirely and trusting that things that need it will install it.

This is unfortunately a complicated question in Python packaging 😅. In principle, things that are installed globally should be managed by the system package manager, which in this case means Homebrew. In practice, Homebrew (1) doesn't manage things at the granularity of pynvim (although maybe we could?), and (2) ecosystems have ossified around the expectation that pip install [--user] mostly "just works".

For pynvim specifically: it removing it globally works, then you should probably do that! If it doesn't, some of the context in this thread may help.

[mikebeaton]

[mikebeaton]

I have fixed my GitHub runner issue using @woodruffw's very helpful pointer that you can just enter a Python virtual environment like this:

      - name: Install Dependencies
        run: |
          python3 -m venv ~/pyenv
          source ~/pyenv/bin/activate
          python3 -m pip install pyyaml

You then need to add source ~/pyenv/bin/activate to each subsequent step which expects to use the installed dependencies.

Also worth pointing out that the results of all commands are run on one after the other, so adding some additional echo commands can help to clarify which commands are actually giving problems.

[mikebeaton]

Is there any chance that the example code python3 -m venv ~/pyenv && source ~/pyenv/bin/activate could be given as a way to initiate a Python virtual environment, as required to work around this issue, in the error message shown when discouraged pip installs are attempted?

Yes, it's easy when you know how, but I'm guessing a lot of the pain felt here is caused when people don't know how! Python virtual environments may well be something many users have never needed - up to now - if they were just/mainly treating Python as a source of dependencies (or other use cases such as maths/stats as mentioned above).

[MikeMcQuaid]

Could you try and open a pull request? This document should help and we're happy to walk you through anything else.

Thanks!

[mikebeaton]

Okay, @woodruffw had already updated the help text, which is very welcome, but it looked to me like one (and possibly two) updates might be needed to that update, so I added a PR Homebrew/homebrew-core#167676 - tyvm

[woodruffw]

@mikebeaton reviewed, thank you for the changes and apologies for the confusion with the previous language!

[PatTheMav]

The unfortunate drawback of that solution is that it's always tied to the Python versioned that was available when the virtual environment was created. This is not an issue on CI (because the virtual environment is always created for each run of a workflow), but for local installations this will lead to the same problem as global installations before:

When the default Python version is upgraded by Homebrew (which might happen at any point), all references to site packages or Python binaries itself within the virtual environment are broken. The user then needs to create a new virtual environment and install the package anew.

This is not a regression though, it's been happening with globally installed PyPi packages before already. Creating a custom tap with Formulas for specific PyPi packages seems like a much better solution, but obviously has a higher barrier to overcome.

[woodruffw]

When the default Python version is upgraded by Homebrew (which might happen at any point), all references to site packages or Python binaries itself within the virtual environment are broken. The user then needs to create a new virtual environment and install the package anew.

This is not a regression though, it's been happening with globally installed PyPi packages before already. Creating a custom tap with Formulas for specific PyPi packages seems like a much better solution, but obviously has a higher barrier to overcome.

Yes, this is a basic problem in Python packaging, one that is not solved by this change or the underlying PEP. All PEP 668 does it make one specific form of sys.path breakage less common.

Generally speaking, using pyenv (which is available in Homebrew) to manage multiple Python versions and reference them stably is what I recommend. I acknowledge however that this is yet another tool you need to deal with 😅