-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CUDA + multiprocessing issue #404
Comments
hi hugokitano! |
Hi, However, if anyone has good ideas to get the multiprocessing to work or wants to tackle this, I am all ears. |
Describe the bug
A
Exception cudaErrorInitializationError: initialization error
occurs within the multiprocessing pool when using GPU/CUDA on two or more files. This happens in the feature_finding step but could potentially affect any time CuPY is used within the entire workflow.To Reproduce
Environment:
nvcc** --version
Using
cupy-cuda115==10.2.0
.Script: following the convention described by
test_gpu_.py
,where
test_settings.yaml
is all the defaults, with two or more files inexperiment/file_paths
Error
For three separate files
A Solution?
After some research, I was able to find the source of the problem. The combination of multiprocessing pools and CUDA is a little tricky. In short, we cannot use the CuPY API before we spawn processes. I'm not exactly sure where this happens in the code given, but I expect it's in some of the settings management. The solution I found was to set
multiprocessing.set_start_method('spawn')
('forkserver' also works).The speed and stability of the three options is up for debate, and I'm not sure if we will be able to obtain performance advantages using GPU if we cannot fork processes. I'm not an expert on multiprocessing, though.
Would like to know if you can replicate this problem and suggest a fix. Thank you.
The text was updated successfully, but these errors were encountered: