[bug]: ROCm Out of Memory Errors - Excessive VRAM Allocation #6301
Comments
|
That is super excessive VRAM usage. Looks like it is running really, really slowly too. The ESRGAN upscale is ridiculously slow. Unfortunately, I don't have an AMD GPU to test on. Not sure if any regular contributors do. May need community help to figure this out... From a brief search, it looks like the env var If fp32 is being used instead of fp16, you'll use ~2x VRAM - try setting |
|
Setting both |
|
If you suspect the YAML file isn't being read, an easy way to test is add Sorry, I'm not sure where to go from here with the performance issue :/ |
|
Ah, that does appear in the terminal output; so I guess it is reading the file but just... not outputting color for some reason? I guess that would otherwise be somewhat reasonably safe to assume that |
|
We expect colored logs for warnings and errors only (and debug, if you set the log level accordingly). This works for me. I'm confident the precision is set correctly in normal circumstances, but who knows with the issues you are facing. |
|
Having A similar issue, Though my system is not allocating massive amounts of VRAM for txt2img, It has a staggeringly low cap. So I can generate images fine but the moment I move to inpainting I cant do anything, and get this error:
So, for some reason, on my RX 7900 XT with 20GB VRAM, only a measly 3.43GB are available.. It doesn't make sense Edit: I am on the same system, Ubuntu Linux using AMD ROCm for processing |
|
So there are few things I observed with this Tests are done with 1024x768 image generation with SD1.5 models. SDXL doesn't seem to suffer from the issue. All vram usage values were noted with Corectrl software - so it's total system Vram usage, not just what Invoke reports. I use 7900XTX 24GB
The worst case scenario in my case is when I use upscaling node on 1024x768 image (scale factor 2). Without aforementioned settings it sometimes fails to process on 24GB GPU! With theses settings it completes with ~15GB of peak Vram usage |
Is there an existing issue for this problem?
Operating system
Linux
GPU vendor
AMD (ROCm)
GPU model
RX 7800 XT
GPU VRAM
16GB
Version number
4.2.0a4
Browser
LibreWolf 125.0.2-1
Python dependencies
{ "accelerate": "0.29.2", "compel": "2.0.2", "cuda": null, "diffusers": "0.27.2", "numpy": "1.26.4", "opencv": "4.9.0.80", "onnx": "1.15.0", "pillow": "10.3.0", "python": "3.11.9", "torch": "2.2.2+rocm5.6", "torchvision": "0.17.2+rocm5.6", "transformers": "4.39.3", "xformers": null }What happened
When trying to generate an image, a huge amount of VRAM was allocated and prevented the image generation from being able to request any more to actually... generate the image.
Coming from Nvidia to AMD recently, of which my Nvidia card had only 11GBs, I find this unusual because I only tried making a 832x1480 image and that's not particularly large (at least, not large enough to trigger OOM on my Nvidia card when I used other Stable Diffusion front-ends before I came to InvokeAI today).
What you expected to happen
I expected the image to be able to generate without issue; probably even with VRAM to spare.
How to reproduce the problem
invoke.shscript to modify line 41 toHSA_OVERRIDE_GFX_VERSION=11.0.0 invokeai-web $PARAMSto get past an initial segfault bug when attempting generations9:1683214800.55and set the upscaler toESRGANDPM ++2M Karras257.5FP32)20or follow the tooltip and set it to0.7(Result is the same, regardless)Additional context
Specific Linux Distro: Gentoo (LLVM17 Built)
Kernel: 6.9.0-rc6-tkg-eevdf-gentoo-llvm-zen2
Blas Implementation: openblas
Terminal Output:
Discord username
No response
The text was updated successfully, but these errors were encountered: