[Performance]: Why does vllm spend so much memory even using OPT model?

[MitchellX]

Proposal to improve performance

No response

Report of performance regression

No response

Misc discussion on performance

In the vllm, the reserved memory is always around 20GB whatever the model we use. For example, the opt model is only 125M, but the reserved memory is 20GB too. I tried to run the OPT model using a huggingface engine, which took only 568MB of memory.
Why does the vllm framework consume that much GPU memory even for a small model?

Your current environment (if you think it is necessary)

No response

[Qubitium]

@MitchellX Check your vllm config gpu_memory_utilization is set to sane value more applicable for the smaller OPT model and not default of 90% 0.9

[iohub]

Because vllm using 90% (ratio) of GPU memory to build KVCache by default, you can change gpu_memory_utilization param in your scenario.