-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
push: hangs after data transfer to s3 compatible remote #10374
Comments
Does it freeze if you use less no. of jobs, say 8? |
Btw, when using |
Hi @skshetry, No, seems to be working only with Using 1 job seems to disable the usage of async. But that makes the transfers very slow (~1.5file/s) vs the parallel mode. My best guess is that this issue is related to s3fs/aiobotocore and most likely specifically to endpoint urls remotes. |
@tanguy-s, so |
I use minio server for testing without any issues, so I doubt it's an |
@skshetry testing with Running
Shows the 4 dvc cache files that are stuck and the 4 TCP sockets still opened. I have tried reproducing on a new remote with : On my original remote, it fails almost systematically. Do you have any hints on how I could further debug ? how could I enable boto3, botocore, s3transfer logs ? |
This issue is specific to a s3 compatible cloud remote (Scaleway) which is hanging forever on some transfers. I have switched back to aws which works fine. However, there is potentially a remaining issue in the handling of these blocked transfers. |
Bug Report
push: hangs after data transfer to s3 compatible remote
Description
When pushing files to s3 compatible storage with endpointurl with :
dvc push -vv --show-stack
or
dvc push -j 32 -vv --show-stack
DVC intermittently hangs after pushing data to storage :
No verbose output after this point however, Ctrl+C gives systematically the following traceback :
Seems like some futures of the underlying S3FileSystem in dvc-objects are never returning and do not have a timeout.
Running
dvc push -j 1 -vv
works, however seems quite slower on the status collection (approx. 30min for 18k files).Environment information
x86 Ubuntu 22.04 Docker
Output of
dvc doctor
:Same isssue using latest 3.48 DVC.
Output of
dvc config -l
:The text was updated successfully, but these errors were encountered: