New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Limit number of progress bars on larger multi-core systems #242
Comments
Example of the code from PR #243 in use in a Jupyter notebook: |
Example of the code from PR #243 in use in the IPython console: Of the 20 workers, each gets 5M rows from a 100M row |
Pandaral·lel is looking for a maintainer! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently,
pandarallel
dutifully creates one progress bar for each and every worker but on multi-core systems with a large-ish number of cores (say 128 or more) seeing so many progress bars can be overwhelming. In these situations, it may prove more valuable to display a smaller number of progress bars (not necessarily one overall) with each worker mapped to one of the displayed progress bars.What is proposed:
Additional motivation:
We have successfully used
pandarallel
on systems with a much larger number of cores than 128 where seeing as many progress bars as workers is genuinely problematic. We very much benefit from and do not want to simply disable the progress bars -- we want to monitor the progress of ourparallel_apply()
andparallel_map()
operations in a digestible way and without flooding the screen / notebook with too much information.Proposed implementation:
A working implementation has been prepared along with unittests -- a pull request will be added to this issue.
The text was updated successfully, but these errors were encountered: