You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 134.00 MiB. GPU 1 has a total capacty of 23.65 GiB of which 31.62 MiB is free. Process 2136596 has 3.90 GiB memory in use. Process 2136599 has 2.60 GiB memory in use. Process 2136598 has 2.34 GiB memory in use. Process 2136597 has 3.32 GiB memory in use. Process 2136595 has 3.90 GiB memory in use. Process 2136600 has 2.34 GiB memory in use. Process 2136602 has 2.60 GiB memory in use. Including non-PyTorch memory, this process has 2.60 GiB memory in use. Of the allocated memory 2.22 GiB is allocated by PyTorch, and 1.80 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
报错如下,看上去是OOM错误:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 134.00 MiB. GPU 1 has a total capacty of 23.65 GiB of which 31.62 MiB is free. Process 2136596 has 3.90 GiB memory in use. Process 2136599 has 2.60 GiB memory in use. Process 2136598 has 2.34 GiB memory in use. Process 2136597 has 3.32 GiB memory in use. Process 2136595 has 3.90 GiB memory in use. Process 2136600 has 2.34 GiB memory in use. Process 2136602 has 2.60 GiB memory in use. Including non-PyTorch memory, this process has 2.60 GiB memory in use. Of the allocated memory 2.22 GiB is allocated by PyTorch, and 1.80 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.
是否可以通过张量并行或者类似的什么方案来解决这个问题?
Beta Was this translation helpful? Give feedback.
All reactions