-
-
Notifications
You must be signed in to change notification settings - Fork 476
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BrokenProcessPool on run.cpu_bound #2774
Comments
Yes, a more robust solution would be awesome. But I'm not sure what the best way forward is... |
@rodja it's a tough one, so I propose to start by reasoning about the use cases and think about a solution only after we are confident about the cases we want to cover. Some thoughts and ideas to start with:
|
Sounds like a good approach. I think we should first get #2234 to work, so we have a basis for testing and verifying that everything works. |
Description
I'm trying to execute multiple parallel tasks which performs CPU bound activities, so I'm using
run.cpu_bound
.I expect a failed task to not cause the others to fail as well. What happens is that if a task launched inside a process causes the process to crash (not infrequent when you launch C/C++ apps) in case of error (or OOM issues), a
BrokenProcessPool
Exception gets raised for all the subsequent tasks. The pool becomes unusable for the rest of the nicegui app's execution, until next restart. That's just how python's standardProcessPoolExecutor
works. Excerpt from the docs: https://docs.python.org/3/library/concurrent.futures.htmlwhich outputs:
By searching a bit, I've seen some projects employ a custom logic and other ones completely re-implement the pool.
Note: One can always employ a custom solution in a case like this, but I thought it will be useful to share this problem here with the community and decide what to do about scenarios like this. The framework is pretty solid and the process pools are well integrated with the app lifecycle, so IMHO finding a solution to this can only improve the quality of the framework and the apps produced with it.
The text was updated successfully, but these errors were encountered: