New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
consider taking in global environment variable for nb_workers and possibly other parameters too #257
Comments
Pandaral·lel is looking for a maintainer! |
I'd prefer to keep the code in the core package minimal to make things easier to maintain. Wouldn't you be able to maintain the same functionality by reading in the environment variables with IMO this is a wontfix issue, unless a compelling reason is given. |
One of the tools I use, that uses pandarallel, fails consistently in a cluster environment with out-of-memory errors.
This wouldn't be a problem in the general case, but overcommiting memory is disabled on the cluster. Since the cluster comes with a lot of cores this easily eats up the entire RAM, even for processes that would be fine with 10GB of memory. If I could just limit the number of workers/subprocesses this problem wouldn't occur. Edit: Also note that I cannot just edit |
Please write here what feature
pandarallel
is missing: Would like to control the number of workers being generation without touching the code for different machines.Example: A new
pandas
API which is not (yet) supported bypandarallel
.The text was updated successfully, but these errors were encountered: