-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Turbo randomly hangs for 60-90 seconds in between task commands during a Docker image build #7382
Comments
Hello, I tried to repro this on linux + podman and with macos + docker desktop and can't seem to get the hang. I will mention that to make buildx play nicely with turbo you should probably ensure it is logging in 'plain' mode (rather than 'tty' mode). The latter will break formatting which (maybe..?) causes turbo to look like it's not doing anything. To give more detailed advice given I can't reproduce I would request you produce a profile (using the |
I have CI job hang indefinitely until the CI timed out (6h) Also observe the same issue on Macbook randomly, from multiple people. related: #7057 |
Hey @unional I let my machine run overnight on repobuddy with the command that failed on CI to try to collect a profile and can't repro it. If you are able to run turbo with the |
Sure, it's pretty random. I experienced that locally mostly on a private repo related to work. Not running build or doing development of repobuddy on work laptop. :) Somehow it is very consistent in the GitHub CI. |
Verify canary release
Link to code that reproduces this issue
https://github.com/ondrej/letsgo/tree/turbo-bug
What package manager are you using / does the bug impact?
Yarn v1
What operating system are you using?
Mac
Which canary version will you have in your reproduction?
1.12.4-canary.1
Describe the Bug
When executing a task (e.g.
turbo run build --filter=api...
), Turborepo 1.11.0 and higher will frequently, randomly hang for 60-90 seconds before proceeding to the next task. Debugging confirms the previous task completed (e.g.tsc && echo 'Done'
) and control seems to return back to Turbo, which then waits before successfully moving on to the next task. Running Turbo with-vvv
doesn't show Turbo generating any traces between when the previous command completes and when Turbo moves on to the next one. I was never able to reproduce the bug outside of a Docker image build.Expected Behavior
Turbo will immediately move on to the next command, as it does in versions up to 1.10.16.
To Reproduce
git clone [email protected]:ondrej/letsgo.git
git switch turbo-bug
yarn install
yarn build
cd apps/api
yarn buildx
You may need to run the last command 2-3 times to see the bug reproduce. If you do, change a source file (e.g. edit the log in
src/index.ts
) to avoid a cached layer.Additional context
I tested my reproduction with versions 1.10.16, 1.11.0, 1.12.4, and 1.12.4-canary.1. It works on version 1.10.16, fails on the others.
The repro repo uses yarn but the bug reproduces with npm as well.
The bug reproduces frequently, maybe 4/5 times. When it does reproduce, it can reproduce multiple times for multiple commands in a single task, extending total run time by 60-90 seconds for every time it hits.
I experimented with different
--concurrency=
values but it made no difference.Docker Desktop for Mac is on version 4.27.1.
The text was updated successfully, but these errors were encountered: