Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DL2 OOM issues #70

Open
LukasNickel opened this issue May 12, 2023 · 3 comments
Open

DL2 OOM issues #70

LukasNickel opened this issue May 12, 2023 · 3 comments

Comments

@LukasNickel
Copy link
Collaborator

The lstchain dl2 script uses enormous amounts of memory due to loading all models and data into ram at once. Nothing we can really do about that here. We had oom issues even with 64G due to the larger models on the crab declination line, so we probably need to increase even further.
Might slow things down on the slurm side, but this is really a (known) lstchain problem.

@nbiederbeck
Copy link
Owner

I had success with 70G for the Crab models and already a commit to fix it in #67.

Need to check though whether the Crab models are the largest and change accordingly. Also might calculate it on the fly, should be possible I think. Then even smaller models get scheduled quicker.

@nbiederbeck
Copy link
Owner

nbiederbeck commented May 22, 2023

For now Fixed via #72, right?

@LukasNickel
Copy link
Collaborator Author

Yeah. I left it open as thats not a nice or long-term solution, but if you want to we can close the issue for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants