Only using 65% of GPU memory

[tannercollin]

While running some benchmarks I noticed min-dalle was only using 65% of my 3070 Ti GPU's memory:

Here's the function I'm calling:

def run_dalle():
    generate_image(
        is_mega=True,
        text='rich ducks playing poker',
        seed=0,
        grid_size=3,
        top_k=256,
        image_path='generated',
        models_root='pretrained',
        fp16=True,
    )

After running it 50 times, each image takes 54.5 seconds on average to generate. It's running baremetal on 32x E5-2630 v3 threads and 64 GB RAM.

Is there a way to make it use more of the GPU? or am I reading this wrong? Thanks!

[tannercollin]

Also the GPU-Util field fluctuates between 0% and ~40%.

[78Alpha]

Doesn't seem too off. I use the bfloat16 and get a usage of 41% with 7.7 GB/ 8 GB VRAM. That's with a 3070 (non-Ti)

EDIT:

The time, however seems a little strange, as I am at 11 seconds per image

[tannercollin]

@78Alpha are you using the Mega model?

[78Alpha]

@78Alpha are you using the Mega model?

The default for the pip package

EDIT:

Checking back, it is the Mega Version. Will also try non-mega...

Non-mega went to about 9 seconds per image

[tannercollin]

I'm surprised there's only two seconds difference. Note that mega defaults as off, so you have to pass --mega into the command line.

I was also getting around 10 seconds in the non-mega version.

[78Alpha]

I had been using my own utility script for batch generation here

I have it set to is_mega=True