Should Rye Exist?

[mitsuhiko]

We all know XKCD #927:

This is how I feel about all the Python packaging. And this is why I never wanted to publish rye and kept it for myself. It's also incredibly hacky internally because it was never intended to be shared. However I really like what it does (at least in theory) and I desperately want it to exist.

As it stands today, it solves my problems — sort of. I wish it solved more of my problems but to a large degree what is missing, rye cannot really solve. Rye should not exist, it should just be the out of the box Python experience. The same way as Rust comes with cargo and rustup, my dream solution for Python was that something like rye exists and becomes the standard way to manage Python projects including getting the right interpreters. However that requires that such interpreters are actually distributed as binaries. Likewise for rye to work, the underlying packages still need stuff that does not exist. Here are some things that would make a real rye work:

• A standard for declaring development dependencies
• A standard for how local dependencies should work
• A proper lockfile standard
• A way to fetch metadata from indexes, without having to fetch wheels
• Unzippable binary distributions of all major Python interpreters on all major platforms
• A standard for declaring basic scripts and tools in pyproject.toml

But even without all of this, rye could exist. In a sense it does today, but it exists to solve one developer's personal problems in his very personal way. I really have no desire to contribute more to the proliferation of Python packaging. Today the pypa org on github is what appears to be dozens of competing, largely independent efforts. In addition to that there is conda, there are now even companies appearing that are putting layers around conda as well. It's a pretty big mess.

I just want it solved, and I want to never have to think about Python packaging and project management every again. And until there is a standard tool we all rally around, I don't see that happen. I'm not sure if rye (or something like rye) could be that, but I do put the offer out.

So here is the question: should rye exist?

[amirkarimi]

Would be nice if you could compare it with existing efforts like Poetry. I didn't use Poetry extensively but my first impression was great. Curious to know what's your take on that?

[bolinocroustibat]

Are there any synergy efforts between the development of the fantastic PDM and that of the brilliant Rye?

[dre-hh]

The docs of rye mention it is using pip-tools for locking. A major difference between pip-tools and Poetry is that, poetry creates a platform independent lockfile.

pip-tools creates a lockfile for the exact os and cpu architecture you have run it on.

Python packages can specify a different dependency depending on cpu architecture . See this tensorflow isssue for example.
Tensorflow brings totally different packages for any amd64, MacOS arm64 or Linux arm64.

If you create a lockfile containing tensorflow with pip-tools on Apple Silicon , you won't be able to install it on linux

Afair, resolving something platform independent with wheels is not really 100% supported via pypa standards and poetry applies bunch of educated guesses where to find this information. Its one of the reasons for it being slow.

[con-f-use]

A major difference between pip-tools and Poetry is that, poetry creates a platform independent lockfile.

That is rather off-topic for this discussion about whether rye should exists. Maybe open an issue, but afik, pip-tools is just the stand-in option and another dependency resolver and locking tool will be used in a stable release.

[dre-hh]

I was addressing the above question in the sub thread. But if you mention it, I would argue, rye absolutely should exist, but only if it can hold up to solutions where such issues are solved. Cargo is mentioned a lot on this thread, but also cross platform lockfiles are a solved thing in bundler for a decade.
https://nokogiri.org/tutorials/installing_nokogiri.html#supported-platforms

[con-f-use]

Well, that's Rust and Ruby. Python packaging has a few complications, so it's not an entirely solved thing, because distributing interpreter versions is not a solved thing and the whole packaging system is a perpetual work-in-progress. Again, pip-tools is not the final solution, and just used because it was the quickest at the time. A cross-platform lock-file is, to my knowledge, planned, but difficult to do right in Python. Poetry imho is not a good candidate for a locking engine in rye because it is very opinionated, to heavy-weight and doesn't play nice with others.

[mrswats]

Would a series of PEPs be a way forward to solve this issues you mention here? Would the effort be worth it?

[frostming]

There are PEPs trying to address some of the problems listed above:

• A proper lockfile standard

PEP 665. Rejected due to the issue of sdist.

• A way to fetch metadata from indexes, without having to fetch wheels

PEP 658. Accepted but the implementation progress is slow.

• Unzippable binary distributions of all major Python interpreters on all major platforms

PEP 711. Draft

[flying-sheep]

There’s some discussion about a PEP 665 successor here: https://discuss.python.org/t/how-should-a-lockfile-pep-665-successor-look-like/17690

[jezdez]

@mitsuhiko Could you elaborate the sentence above, trying to understand what you mean by that:

py I think is also a problematic entity, because it's tied to cpython's distribution.

IIUC rye is tied to the python-build-standalone distribution, is there any condition that the official Python distribution on python.org would need to make py not problematic anymore?

[brettcannon]

py I think is also a problematic entity, because it's tried to cpython's distribution.

That depends on which py you're referring to. The Windows one comes with CPython, the Unix one that I created does not.

I'm also not sure if py actually plays a role today in the wider ecosystem. I at least haven't ever used it, and I don't remember it coming up in documentation of packages I use.

You're right that it doesn't play a wider role. I have some long gestating ideas on letting people participate in the interpreter search and support for subcommands, but I haven't had the time yet to make them happen. Only at that point would py (for Unix at least) start to potentially participate as something other than a launcher of Python interpreters.

[smokker1983b]

Thats a good question!

[cnpryer]

it should just be the out of the box Python experience

Yea I consider the Rust tool-chain a kind of gold-standard when it comes to experience. I understand why it's not that simple to just turn pip into more of a workflow tool, but to me the ideal scenario is one with py as a kind of rustup and pip as the cargo.

The xkcd is a good one, but I still think there's value in creating and sharing your ideas. Garnering interest and forcing more discussion is beneficial for the ecosystem, no?

[con-f-use]

Thanks for saying this, it needed saying. The way I see it, the problem is not really having "yet another standard", variatio delectat after all. I agree that having a nice plurality of ideas and having them accessible is a very good thing until, and even after, a clear "winner" emerges. What is a problem, is that these things are all presented as standards and beginners are utterly confused in what to use, how and why. Maybe people writing and advertising these tools should take more care pointing out, that these are, even if widely used, still very experimental and that no clear, blessed dev tooling exists for python yet. TL;DR, maybe people shouldn't call them "standards" at all, and not that aggressively market their solution as the bee's knees, until they are accepted, widely implemented, and generally agreed-upon PEPs.

[sultaniman]

@mitsuhiko anything new and consistent is a win for community, I think however we all need a better packaging, distribution and build system for Python.

[sztomi]

Yes, 100% should exist. You are right that something "official" should exist for Python that fills this niche, but pypa adopted several external projects that were relevant for it, so maybe rye could be such project.

[mikedlr]

The problem with that system is that there isn't a clear set of defined boundaries between those tools which allows one to work with another. There isn't also an overriding interface which allows beginners to start safely using whole infrastructure in a simplified way and then later transfer to using the particular tool for an area if they want some specialist feature.

A good summary is How to improve Python packaging, or why fourteen tools are at least twelve too many - note the author of that is part of this discussion above. Not so long ago did a "try again" with pipenv, assuming that since it was now being recommended as a standard tool it should be working. This turned out not to be the case at all and had to be backed out. That just shouldn't happen with the "official" tools.

[brettcannon]

pipenv, assuming that since it was now being recommended as a standard tool

FYI pipenv isn't "official" beyond the fact the maintainers have it under the pypa org on GitHub which isn't more than a group of projects sharing some infrastructure. Hatch itself is also under the same org, for instance, so pipenv isn't the official workflow tool for anyone except projects that choose to use it. The only requirement for a project to join the org today is for someone to ask if a project can join and other pypa org project maintainers vote on the suggestion (pipenv predates this policy BTW).

[mikedlr]

I did get that clear, but some time after having read that it was the official tool and deciding to try it. It was unable to work for my needs even with considerable willingness to find clever workarounds.

[mikedlr]

In fact, I should say specifically, that it was the above essay about fourteen tools being twelve too many which allowed me to realise that despite claims otherwise pipenv was in no way "the" official (as the Zen of Python) way of doing things.

[eli-schwartz]

Pretty sure pipenv suffered a lot from having "a maintainer of all time". It wasn't the official tool and didn't have community consensus that it should be, but was advertised as the official tool by a private overreach of authority.

However that's in the past as it is no longer advertised as such?

[href]

I don't see myself using rye quite yet, but I appreciate the use of @indygreg's standalone Python builds. Personally I use those too (on macOS), together with some shell scripts to manage my virtual environments. I use poetry as well, but that can change.

But I think rye should exist, and I'll track its development with interest.

[aglove2189]

I made a package similar to the one that you describe, all it does is downloads @indygreg's standalone builds: encant

[seekaddo]

I believe your summary already addresses the question at hand. The advantages clearly surpass any reasonable concerns. A well-defined roadmap should be created to identify the best practices, thus enabling the community to choose the package manager that best meets their needs. I haven't used poetry before, and the main idea behind rye isn't to instigate a C++ vs Rust debate, but rather to tackle significant issues that everyday developers face. I have been using conda for most scientific work and personal projects, though it has its drawbacks, such as complicating the export process. When using conda, you need to call pip freeze to export the environment, which seems counterintuitive. Furthermore, it doesn't detect all pip-installed packages - only those sourced from PyPI.

I think rye has the potential to provide simplified and transparent standards. If implemented correctly, the community could reap substantial benefits from it.

[phaustin]

Have you tried conda-lock? Works well for our use cases: https://github.com/conda/conda-lock

[Butch78]

I think something similar to cargo & rustup for python should exist! I think setting up environments in python and installation are the biggest barriers to entry. If we can reduce those I think it'll be better for everyone :)

I don't know if you've seen this project:
https://github.com/cnpryer/huak

I've used it a little bit and it seems great, if rye or huak can do all the similar things to cargo I will use them :)

[con-f-use]

Does huak manage python interpreter versions too? I.e. if I say I want python 3.12 but the system python is at 3.9, will huak install 3.12?

[mwarkentin]

Does PEP-711 (PyBI: a standard format for distributing Python Binaries) help with the python binary issue?

Caveat that it gets accepted and implemented of course.

[glyph]

mostly because the PyBI builds try to retain this framework structure for macOS that's just not needed

It's definitely needed for a whole slew of use-cases; basically anything that calls a platform API on macOS.

[mitsuhiko]

@glyph I would actually be curious for when a framework build is needed. As @indygreg mentioned a lot of macOS APIs work fine in a non framework context.

[glyph]

As @indygreg mentioned a lot of macOS APIs work fine in a non framework context.

Some do, some don't. Very few (beyond UNIX-y and CoreFoundation ones) are documented to do so; there's just a strong assumption throughout the platform frameworks that you exist in a bundle (usually an app), and that your frameworks do as well. So it's very annoying to research specific APIs because Apple won't tell you. You have to build your own examples, and then you get arbitrary undocumented failures about Mach ports which boil down to "did you build as a bundle in an app / framework? no? oh, that's your problem".

But let's assume that all APIs are actually callable from a non-framework build. If you want to distribute software for macOS, you can't do so, because all the code-signing tools break. If you want to distribute the build of Python itself, for example, unless you live entirely in CLI-land and opt out of quarantining (i.e. "files you downloaded from a web browser") entirely, it is not possible to distribute a stand-alone executable that lives outside of a bundle. Individual binary files can be code-signed, but they cannot be notarized, because there's nowhere to attach the notarization ticket. In other words, you have to end-run the entire security apparatus of the platform, and rather than it being a temporary issue for lack of attention, standardizing on non-framework builds takes a step further towards cementing it as the permanent status of Python.

If you try to distribute an app bundle using a non-framework build of Python, then you suddenly run into a bunch of other issues related to the way that codesign tool respects bundle resources and deals with dynamic linking. None of these are totally insurmountable — it is possible to ship a raw dylib in a framework bundle (and in fact Python binary wheels do have this problem as well), but it breaks a bunch of assumptions from tools like py2app that want to link against Python as a framework, and codesign, stapler, and notarytool.

Then there's the issue of callbacks. If you want to include Python as a dependency of a native Xcode project, there's no way for Swift or Objective C tooling to consume "arbitrary directory with a pile of UNIX-ish junk in it"; they expect to see a Framework with Headers, Versions, et. al.

If you ignore all of this, then sure, you can limp along and kinda call a bunch of APIs and kinda muddle through to make something distributable, but you cut off all kinds of possibilities and make natively accessing the platform arbitrarily weird and difficult and broken in edge-cases for no particular reason; we already solved framework builds and they work fine. Getting rid of them would reinforce the idea that Python is only for backend/web stuff because it's just too confusing and broken on client platforms, and the Mac port would regress further towards treating macOS as a slightly-broken version of Linux rather than its own platform.

[glyph]

You have to build your own examples, and then you get arbitrary undocumented failures about Mach ports which boil down to "did you build as a bundle in an app / framework? no? oh, that's your problem".

Case in point: CoreLocation used to be my go-to example of an API that didn't work outside of a bundle context, but it looks like maybe that does work OK now? NSUserNotificationCenter also works now but didn't used to, but it's deprecated now and in a quick test (which could be broken, btw, it's annoying to switch back and forth between framework and non-framework builds and set up the appropriate context) its un-deprecated replacement UNUserNotificationCenter appears to silently do nothing. I can't quickly find a "crash with some message about mach ports" example but there are like a hundred frameworks on the platform and I only have examples for a few of them, so this is unfortunately more like folk wisdom.

[glyph]

Oops, I double checked, and looks like I did in fact use a framework build by accident. CoreLocation no longer spits out an error message on a non-framework build, it just segfaults immediately.

[indygreg]

You can definitely call macOS APIs without being a framework. Framework-less Python builds are evidence of this. My understanding is the framework builds of Python are more to facilitate other apps calling into Python’s APIs. The framework does provide a more consistent filesystem layout / API to target. But it isn’t strictly necessary.

[danielfrg]

For what its worth I moved a small personal project to rye today and its a very good and clean experience. I had a series of scripts and used other tools like Taskfile to have a very similar experience to what rye offers.

It's a big step in the right direction of tools in other languages like volta and the mentioned rust + cargo experience. Python is behind even if you can survive without those when you come back to Python it feels like a struggle.

I guess in an ideal world it should not exist but its not the world we live in. Thanks for putting it out there!

[Abdur-rahmaanJ]

I am 100% pip tools fan but by default i appreciate all tools you suggested so far, from click to your blog engine. On the basis of past programs' design, i am optimistic for rye.

I am not a big fan of lock files though.

Edit: Like i mean using pip-tools behind the scene is too cool!

[cnpryer]

What don't you like about lockfiles?

[Abdur-rahmaanJ]

The time taken to lock + locking os-dependent files.

[con-f-use]

Lock files are optional and very much necessary in enterprise (production) contexts. No one is a big fan of how much time it costs to resolve an entire transitive dependency tree, and for most hobbyists it's overkill, but in the interest of software reliability, you really can't use python without them (until ideas like that of nix take over all of computing, which likely will never happen).

[rtpg]

This all seems very cool. I do think the XKCD is very misleading, because if you have 14 standards, and then introduce something that is overlaying over existing standards... you have 15 standards, but in theory you are sharing work and that can help proliferate good changes across the stack.

We have already seen how virtualenvs being the "bottom level" of a lot of these projects has meant that virtualenvs "just work" for many definitions of work, and efforts to improve those helps everything! Any tool that is trying hard to use what exists, and inform the greater community of specific missing parts (beyond unactionable stuff like "confusing" or "hard for beginners"... it's hard for practitioners! Maybe we should get that right first) then we'll be making loads of forward progress.

Anyways I really like how this tool should in theory work all over and doesn't involve any black magic. Will try to use it and see how far I can get

[spapas]

Sorry for this but my answer to your question is No.

Before my ostracization please allow me to explain.

First of all, not actually against this specific tool but I'm against all similar tools (i.e conda, pyenv, poetry whatever). But since you specifically actually asked I'm happy to answer this discussion. The python ecosystem is already way too crumbled with tools like this. Having options may seem like a good thing but in my opinion for the Python world it is not a good thing as the the Zen of Python reminds us:

There should be one-- and preferably only one --obvious way to do it.

Now, which is the obvious way? For me it is using the standard tools that python has to offer: pip and venv. All effort should be combined and geard towards improving these tool to support any missing features or workflows, not by introducing a new tool.

Beyond that, please consider users that are new to Python. What should they use? How can they decide? Why burden them with options? I consider myself an experienced python developer (10+ years) and yet when I see all these tools I feel overwhelmed. Why not help these people with how to use Python with the standard tools instead?

Finaly, the fact that native Windows is not fully supported is (at least for me) a very big problem for any Python package.

Ok I'd be happy to be ostracized now.

[snowe2010]

Why is the comparison here to rustup, cargo, etc.? Why aren't y'all comparing to ruby which is the same type of language but with better tooling? gem and bundler have been standardized in Ruby for decades now and have no competitors. gem is vastly easier to use than pip. Everyone using ruby knows how to use these tools, and they work perfectly. Cargo is literally written by the creators of Bundler. Use Bundler as your base level of what a tool should do in a dynamic language and then maybe move on to Cargo from there.

[eli-schwartz]

Cargo is literally written by the creators of Bundler.

Thanks, now I finally know the underlying reason why I've hated cargo for so long.

[snowe2010]

Thanks, now I finally know the underlying reason why I've hated cargo for so long.

And yet here you are in a thread for a language that has spent 30 years and still doesn't have working package management or install tools. The tool that works and is standard beats out the tool that doesn't exist.

[eli-schwartz]

But... we literally do have working package management and install tools? And they are flexible enough to let people choose how they want to install things, without preventing opinionated tools from choosing a single way and sticking with that everywhere.

No one is worried about whether python has working package management or install tools. The hot topic of the day is whether python can have a "one tool to rule them all".

[snowe2010]

you really don't. The tools are all garbage and hardly work at all across systems. I used python for years at my last job (and still partially use it at my current job), alongside ruby, java, kotlin, and rust. The difference is stark between the two. Claiming that other tools are 'opinionated' and that python has choice is just laughable. I'm sorry, but Bundler isn't opinionated, cargo isn't opinionated, maven isn't opinionated, gradle isn't opinionated. They're much more powerful tools that avoid the problems that literally every python tool deals with. You act like these other tools have limitations just so you can justify having 'choice' in the python community. I'm sorry, but it's just not accurate.

The tooling in the python world has always been terrible. Acting like you have a choice when the community recreates the package management and build tooling every year is just disingenuous. It just indicates that none of the tooling actually works. There's always a shortcoming.

The first step is admitting there's a problem.

[Kwpolska]

As the author of a slightly popular (and divisive) blog post entitled How to improve Python packaging, or why fourteen tools are at least twelve too many, my opinion is s/fourteen/fifteen/; s/twelve/thirteen/.

The thing that is the most worrisome is the use of Rust. I think Rust is a great language for lower-level systems programming; for many things, something higher-level than Rust is easier to work with and achieves equally good results. But in this specific case, we’re talking about tooling for Python. Does Python suck so much / is Python such a toy language that the packaging tools (and also code formatting tools) must be written in another language? At that point, why not skip Python entirely and write everything in Rust (or another competent language with tooling largely written in itself, like Java or C#?)

In my opinion, installing and distributing Python should be distinct from installing and distributing packages. For your average Windows user, the user experience of “download and run the latest .msi from python.org” or “click Install in the Microsoft Store” is good enough. Beginners don’t need multiple Python versions and a Python version manager (but they do need packages). Businesses may standardise on “whatever RHEL ships and supports”, or on a single version they can install from custom packages tailored to their needs. Forcing the use of a custom-built Python limits the possible users of Rye and thus leads to more tooling proliferation.

Ideally, there would be two separate tools¹:

• A binary distribution management tool — may be written in Rust or anything that’s easy to distribute and run anywhere
• A package management tool — should be written in Python (and PDM would be a good contender for the one-true tool)

Those tools would be separate, but they would also co-operate. Perhaps with a unified py frontend that can invoke whichever of these is necessary, or perhaps the Package Management Tool might say “this package requires Python 3.11, but you’re on 3.10, do you want to abvort, or do you want upgrade this project to 3.11 with the help of the Binary Distribution Management Tool?” Users who are happy with a single version of Python coming from whichever source they like most can still get a sane and unified package manager.

And even though the Steering Council killed PEP 582, I still think it’s the way to go, and that venvs are too confusing and unnecessary. But seeing the SC chose not to accept this PEP, and seeing that most of the 14 15 packaging tools are associated with the PyPA, I don’t think improvements in the packaging world (seeing as many things you’re missing are standards) are likely to appear soon. So, abandon all hope and use Rust or C# for your next project, I guess?

¹ Not the “two” I had in mind when writing the post’s headline — I accept the existence of a “scientist” tool separate from the “normal user” tool, so there could be three tools in total if the unified package management tool can’t solve all the scientists’ problems.

[con-f-use]

Me neithers, it is somewhat hard to say, who the bulk of Python developers are and how they came to Python. There's the concept of "dark matter developers" and coming partly from industry and partly from academia, I see a giant lot of those. You came to Python rather early and are, even without well-deserved flattery, somewhat exceptional. As is the whole "early wave" of Python users including devs. Just by virtue of Python's rapid growth in recent years, nothing early can be terribly representative of the community, now. But yeah, I'm not completely against C or Rust, all I'm saying is, if there's the choice between Python native and one of those other languages, tooling should lean towards native, unless there's a very good and compelling reason, because people who know Python will have a lower barrier of entry and that has all sorts of advantages when we speak of Python tooling.

That is no critique of your choice of language for rye. It is your project and you are free to judge, what the best tool for you to do the job is. Rye is very very interesting to me and elegantly solves a bunch of problems already, that I care deeply about. Rye clearly should exist for that reason alone. I'm just saying, that I hope whatever tool becomes the ultimate packaging and distribution tool, will be not need to be written entirely in something other than Python. I also see how rye could inspire that ultimate "winner", and that's why I'm here rambling in the first place. Since is done with experiments, and experiments are often repeated. That is how we learn, improve and come to robust knowledge.

[flying-sheep]

unless there's a very good and compelling reason

Speed. Linters, dev servers, and similar tools should be fast enough that the wait time is barely perceptible.

[con-f-use]

Speed. Linters, dev servers, and similar tools should be fast enough that the wait time is barely perceptible.

Sure, speed can become such a compelling reason. There's studies about that kind of thing: everything under half a second is absolutely fine, 1.5 is where it begins to be mildly irritating - you'll be loosing people to internet and coffee when it gets to the minutes. As I said, I'd argue everything other than Python should have a compelling reason and most of the time speed is overrated as one. In addition, some tools simply can't be made much faster. Pytest can't test Python faster than the interpreter can run the code. Network code, can't get data faster, then the communication allows and even than understanding and changing your tools without learning an extra language maybe worth more to people than even a 50% increase in speed in some situations. In any case, we're going offtopic a bit, arguments stand, lets move on, what say you?

[brettcannon]

The Rust in the Python ecosystem now displaces Cython, C and C++, not Python code.

I think that's true overall, but using Python to prototype something which you then later choose to re-implement in Rust is also a totally legitimate use case for Python as well. It's quite possible the prototype works well enough that you can leave it in Python, but it's also possible you simply need to squeeze every CPU cycle that you can get (although watch out for premature optimizations). But both scenarios are totally fine and use Python for its legitimate strengths of being productive to code in.

[flying-sheep]

@con-f-use I totally agree

1.5 is where it begins to be mildly irritating

When the VS Code Python extension had a bug that delayed file saving by 1.5 seconds, I was more than mildly irritated, after like half an hour of working like this, I needed a break to not get cranky.

[indygreg]

Since it was only casually mentioned in a reply, I thought I'd drop a link to https://discuss.python.org/t/announce-pybi-and-posy/23021/1 which announced PyBI (eventually PEP 711) and posy.

Posy on the surface appears very similar in scope / vision to rye.

There are significant parallels between this discussion and the linked PyBI/posy thread.

My $0.02 is that tools like rye/posy python-build-standalone/PyBI are clearly demonstrating end-user benefits versus established solutions. Their presence ultimately enriches the Python ecosystem by at the very least providing valuable feedback on how to enshrine standards via PEPs and/or iteratively improve existing/popular tools/workflows. This experimentation is good and should be encouraged. As long as the presence of these alternative tools isn't externalizing significant hardship on Python package maintainers or the larger Python ecosystem, I have no basis to discourage or criticize any work in this space.

Best of luck with Rye. I hope it meaningfully contributes to improving Python's ergonomics for n>>1 people.

[jezdez]

I concur and would call out @mitsuhiko's unclear positioning towards "standards", as it reads to me so far as if he interprets them as coming together via force of publication on social media and social coding platforms rather then engaging with the packaging community's efforts that exists (CPython, PEPs, PyPA).

"standards" aren't standards until you're going through the process of having a common body of governance review related proposals and set them in reference to existing functionality, providing a way for users to have consistent path for migration and a sensible level of maintenance. Starting a new project is the easy part, but thinking about operationalizing and long term maintenance is the actual work. The reliance on @indygreg's Python stack should come as a concern as such.

I'd encourage the rye project and particularly Armin to basically engage with the accepted governance body to make sure it's not becoming an event splitting the Python community (again). It's simply too much at stake to walk into it without fully understanding the problem space.

I know that Armin is passioned about end user experience, and understands the significance of affordance in working on these systems. This is probably the part of the release of rye I'm most excited about. That it can be a playing field to try out new patterns of interaction.

For reference and context, I'm saying this as someone working on overcoming the significant cost of duplication between the PyPA and conda ecosystems.

[Virinas-code]

Well, the project looks interesting, but why programming it using Rust as it's for Python?

[eli-schwartz]

And I don't buy your argument as a valid criticism, tbh.

Tools should be written however the tool author considers productive. That may be an indictment on the python language if the author doesn't consider python to be productive. That may raise the barrier to entry for other users to contribute to or hack on the tool. Both of these factors are not really pertinent to the question "oh, tool author, why did you make this bad mistake when choosing an implementation language".

But also, my original point was exceedingly targeted. pyenv is software for installing python executable interpreters, and the authors of pyenv felt that the best language to write a tool that installs python for you, is... NOT python.

There is a reason for that, and it's perhaps the most important reason of all: you can't install python with a program written in python, if you do not yet have python available. A program written in a compiled language that builds into a single self-contained executable file is well-positioned to handle acquiring python.

It is indeed silly to suggest writing pyenv in python, and likewise I consider it silly to suggest writing rye in python... unless rye first deletes all functionality related to installing and managing python installations.

Another valid approach might be as suggested elsewhere in this discussion: divide rye into two tools, one to manage python installations written in rust, and one to run inside a python installation and manage python packages, written in python. You likely only care about hacking on the latter, anyway.

[con-f-use]

So you can't compile gcc with gcc, huh? Interesting... 😉

It is the nature of binary distributions, that you can distribute compiled binaries, i.e. the compiling of the python runtime does not have to happen on the endusers machine. It could be bundled with the distribution binary of rye.

Sophisim aside, I was talking in general, and you are making a very rude and narrow point. Python tooling, if no compelling reason not to exist, should in general be written in Python. Exceptions exist. That is my opinion, you are allowed to yours. My criticism is as sound and valid as yours. I maintain, the original question was not particularly silly, and in any case calling questions silly isn't good form.

To re-iterate: my argument in favor of writing Python tools is Python is an accessibility and ownership one. I want the community to contribute, I want them to be represented by the tooling, I want them to participate, I want to reduce the barrier of entry. Having to learn another language, having to duplicate and re-implement and use tools specific to that other language hinders that. What makes my arguments that trying to write Python tooling in Python helps with that (in most cases) invalid or unsound?

[eli-schwartz]

So you can't compile gcc with gcc, huh? Interesting... 😉

That's a fascinating topic, and there are people who say the answer is "no": http://bootstrappable.org/projects.html for details on how they compile GCC in a multi-stage pass that begins with a stripped down C compiler capable of building a "more fully fledged C compiler like tinyCC".

It is the nature of binary distributions, that you can distribute compiled binaries, i.e. the compiling of the python runtime does not have to happen on the endusers machine. It could be bundled with the distribution binary of rye.

You could, and doing so would be quite unwieldy. Shipping a python just so you can run a tool that installs python? Single-file executable handling is not very fast either. And you remove much of the benefit of writing it in python, because actually building the binary you hacked on is a lot trickier than it is in an ecosystem where single-file executables have native support.

What makes my arguments that trying to write Python tooling in Python helps with that (in most cases) invalid or unsound?

The fact that my comparison, pyenv, made it pretty clear that my opinion was and still is, that distributing a python program which is used for installing python, is not very ergonomic -- and criticizing a project that avoids python for this task is... pretty silly. The project has a compelling reason, the compelling reason was part of the original problem statement, so I don't see where

Python tooling, if no compelling reason not to exist, should in general be written in Python.

comes in. Disregarding the compelling reason and then trying to make this a generalization for virtualenv managers (a number of which already exist, written in python no less), doesn't make my original criticism of a specificity suddenly turn into a criticism of a generalization.

It makes a lot of sense for tools such as pip, pdm, poetry, hatch, etc. to be written in python, and I absolutely agree with you there. Fortunately, they already are written in python.

It makes a certain amount of sense for rye to focus on acquiring python using rust code, and dispatching to python code once it has a python available. This would empower users to hack on the project management routines as long as they don't need to hack on the "acquire python itself" step, using nothing other than knowledge of python itself.

Getting rid of all rust code seems to be very much missing the original point of this project and it is disappointing to me to see people seriously suggest it.

[con-f-use]

Now that is a line of reasoning I can live with. To be honest, I antagonized you because the "silly question" think and general superior air rubbed me the wrong way. But just because I enjoy discussion (which is euphemistic for "I like to argue"), I'd still say, it's kind of a valid point still. It's not that much harder to bundle a standalone python interpreter with rye, than it probably would be for future prospect contributors to learn a whole other language, just to be able to contribute. Rust was probably the more practical and easier choice for Armin and that is why he made it. But the whole "should rye exist?" question is predicated on the community as well and I'd prefer the ultimate packaging tool to be governable and accessible by the whole community as much as possible.

That's a fascinating topic, and there are people who say the answer is "no"

Right? I think so too and have spend hours of fun research. It is, however, a rather academic ivory-tower subject. The fact remains, that I can download a gcc binary that runs on my machine, I can download the code of gcc and then after a long unnecessary tedium of yak shaving, I'll end up compiling the gcc code with my gcc binary on my machine. Same goes for bundling a standalone version of the python interpreter, that work has been done and Armin is using it from another project to install Python interpreters. If you ship someone else's pre-compiled interpreters anyway, you could feasibly ship them with your interpreter install tool, too.
Mostly, bootstrapping problems are a little like watching those preppers who have a lot of fun and make entertaining videos imaging they'll have to re-build civilization after the apocalypse has come and we have regressed into the stone age. How on earth would we do it? What would we have know? With how little can we start? 😁
Security vetting, novel platforms and traceability concerns aside of course, which is were solving the bootstrapping problem is of real-world import. Anyhow, thanks for not escalating my rather aggressive line of argument, I hope you enjoyed yourself as much as I. Let's not go further offtopic here. I feel mildly bad for hijacking Armin's question. Luckily github doesn't charge by character (typographic or human).

[eli-schwartz]

But just because I enjoy discussion (which is euphemistic for "I like to argue"), I'd still say, it's kind of a valid point still. It's not that much harder to bundle a standalone python interpreter with rye, than [...]

Eh... I hear the argument, I disagree (not too fond of e.g. extracting a hefty python payload to /tmp at every startup), but I acknowledge that opinions can vary here.

I hope you enjoyed yourself as much as I.

It is, indeed, a fun topic. Especially the novel platforms bit.

[csaroff]

Is it possible to make virtual environments automatic? When using npm, you don't need to explicitly activate your virtualenv, you can just cd into the directory and npm install and npm start and everything just works.

[mitsuhiko]

@csaroff if the shim is installed then python automatically picks up the right virtualenv. The same is true for rye run script which also automatically enables the virtualenv for that script. What I don't want is to create a shim for all potential scripts and mess up the global path.

[csaroff]

@marcelarie

True, npm doesn't cleanly handle projects with different node versions which can be annoying.

It does handle your package deps(node_modules) though!

[snowe2010]

with asdf you don't have to worry about any of this. cd to the directory, all your tools activate for every language.

[Kwpolska]

Tools like asdf don't necessarily solve usage from scripts. The relationship between the working directory and your packages is natively understood by node/npm (because they're looking for node_modules in or around CWD), but Python requires the use of a specific interpreter. Can you run asdf in a script? Does it always work the way you expect, even in things like CI/CD pipelines? Those aren't questions you need to worry about often in ecosystems that didn't go the venv route (i.e. most of them).

[snowe2010]

Can you run asdf in a script?

you don't 'run' asdf. you just let it handle setting the environment when you're in a specific directory. And yes you can easily refer to the active shim by just referring to the shim in the ~/.asdf/shims directory, but you don't really ever need to do so. asdf's power is that it just works. If I have a script that refers to python and that script is run from a repo that has a .tool-versions file, it will use the version from that file. For example every single one of our repos has a Makefile that calls out to a shell script. These shell scripts can refer to any tool and if that tool is declared in the .tool-versions file then it will just work. There's no setup, no version management, nothing. It works both locally and in CI/CD.

In regards to usage in CI/CD, yes, it's quite simple, even with caching. https://github.com/asdf-vm/actions

[simonw]

Quoting my tweet: https://twitter.com/simonw/status/1651614850282569728

This is pretty promising! It means you can install a single Rust binary ("rye") and then run "rye install datasette" and get a freshly installed copy of Python 3.11 plus a "~/.rye/shims/datasette" tool you can run, with safely isolated dependencies

There are some really interesting new ideas in Rye - the way it installs a pre-compiled standalone Python binary in particular (as opposed to pyenv which still insists on running a C compiler).

I think a working prototype of an idea leads to a much higher quality conversation, so on that front I'm a big fan of Rye existing already.

[pauleveritt]

@mitsuhiko Do you/rye have an opinion on PEP 582?

[mitsuhiko]

@pauleveritt my main opinion is that it has no traction, so there is no point going down that path right now. There are more important things to figure out.

[th5]

Pip + venv is now working well for my needs. For me the missing piece is how to install and manage Python versions/builds.

[tshirtman]

for that you can use pyenv or asdf, for example.

[jankatins]

Or https://github.com/jdxcode/rtx (asdf + a bit of direnv in rust)

[aglove2189]

I made a tool for handling that piece specifically @th5: encant

[majidaldo]

for data sciency use cases where i want more than one env at the same time i created hydraconda. supports monorepos where directories 'compose'. https://github.com/pnnl/hydraconda/tree/master/project
i really want to replace this with something more 'clean'.

[BruceEckel]

I'm really liking rye, and it seems clearer to me than the other tools I've tried. I notice I'm understanding things while learning rye that I didn't with those other tools. Perhaps it's just repeated exposure to the ideas, but I don't think so.

Cargo made me want something like it for Python, and that feels like the direction rye has been going in.

[benrutter]

I've just come across it, but am really liking Rye - I hope it does keep existing! I'm in no way affiliated with the python packaging authority, but whenever I hear them talk, it sounds a lot like they're actively looking at projects like these for direction (there's a good, but hour long, discussion that has a lot of members here. I really hope they'll pick something like this as a direction for python, but it sounds like that's some way off.

One thing I really love about it (that puts me off tools such as poetry) is that because it wraps existing tools such as pip, so it's perfectly easy to either:

• stop using it at any point
• use it, and collaborate with someone not using it
• use it, but additionally, use other tools alongside it (not sure quite how this would work in practice?)

I think that's important because it avoids being the "15th standard" in the xkcd comic by not being a standard at all, but just a tool conforming to standards that already exist.