-
-
Notifications
You must be signed in to change notification settings - Fork 600
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
concurrent pip-tools processes may interfere with each other #2028
Comments
I don't know why your resolution conflicts occur, but will say that when running |
Thanks, @AndydeCleyre! I tried adding that option to our Makefile setup, but I'm unfortunately still getting the Based on the traceback and the likelihood of the Do you think that's possible or have any other experience/insights about that potential risk area? |
My team uses a monorepo that contains a number of different python projects and applications. To simplify dependency management, we use a recursive Makefile strategy to run pip-compile in different directories and potentially with different python versions with a single entrypoint command.
As this process is simple, but can take some time, we've started to invoke
make
with the-j
argument (allowing different targets to be built in parallel. I've recently come to realize that there seems to be some contention or interference between these different parallel invocations ofpython -m piptools compile
.I realize this is probably fairly uncommon, but I was hopeful that adding more isolation of the dependency resolution process or temporary directories might be a simple solution. Also, please let me know if this is more of a
pip
issue than apip-tools
issue. I've done some research suggesting that while parallelpip
execution isn't a guarantee, it often works. Other than that, the traceback itself was the only thing leading me to raise the issue here.Environment Versions
Steps to replicate
Apologies for the complicated repro. I've attached a "mini monorepo" example that demonstrates the issue.
compile-error.zip
pyenv
to install the three versions of python listed in the.python-version
files.compile-error
directory (from the unzipped example project)make -j
Expected result
I hoped that these parallel executions wouldn't interfere with each other, not hit exceptions, and produce a compiled requirements list.
Actual result
The text was updated successfully, but these errors were encountered: