Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable more parallelization on the R action worker via: -j $(nproc) #3708

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

coatless
Copy link
Contributor

Improves the env variable MAKEFLAGS to use all available cores instead of 2 due to variability in the GH runner.

Close #3707

Improves the `env` variable `MAKEFLAGS` to use all available cores instead of 2 due to variability in the GH runner.
@coatless
Copy link
Contributor Author

Ahh, windows... Why don't you like {nprocs}? I'll poke around more tomorrow in a VM.

@shrit
Copy link
Member

shrit commented May 16, 2024

The problem with this is that mlpack needs a lot of RAM to compile, for instance I have 24 cores on my machine which require at any time around 50 ~ 70 GB or RAM only to compile mlpack (whether compiling tests or not).
I am lucky to have 256 GB or RAM on my desktop, but If you do not have that amount of RAM then you have a high chance of freezing your entire machine.

@coatless
Copy link
Contributor Author

@shrit perhaps it's a memory limitation; but, workers should have 4 cores/16 GB (ubuntu-latest, windows-latest) or 3 cores/7 GB (macos-latest). I suppose a question that needs to be answered is how much RAM is required per core assigned to compile mlpack?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch from -j 2 to -j ${nproc}
2 participants