Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encounted OOM while trying to train the stage1 model #163

Open
stevenk12 opened this issue Feb 8, 2024 · 0 comments
Open

Encounted OOM while trying to train the stage1 model #163

stevenk12 opened this issue Feb 8, 2024 · 0 comments

Comments

@stevenk12
Copy link

I use the script 'uni_dist_train.sh' and try to train the stage1(track and map) model, but I encounter the error of 'OUT OF MEMORY'.
I use four 3090 GPUs, the total memory is 96GB. Recording to your words in 'Eval and Train', the first stage needs about 50GB memory. So what's the problem? Can u help me. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant