Question regarding the nearly double GPU memory consumption.

[Zhuqln]

when i loaded vicuna7b_v1.3 with default config ,i found that gpu memory cost around 23G, but in fastchat's readme says

'requires around 14GB of GPU memory for Vicuna-7B '

anyway, the throughput is much more faster. I just want to confirm if the GPU memory usage in this scenario is normal. If it is normal, I would like to know why there is almost twice the consumption. thanks a lot

[Zhuqln]

i just read the description of class LLM, is this extra memory usage because of KV cache？

[zhuohan123]

Thank you for bringing this up. Yes, this extra memory usage is because of the KV cache. vLLM pre-allocates and reserves the maximum possible amount of memory for KV cache blocks. The KV cache generated during the inference will be written to these reserved memory blocks. You can limit the GPU memory usage by setting the parameter gpu_memory_utilization.

[Zhuqln]

Thank you for your response. I have an additional question: If I restrict the GPU memory to reduce the available space for the KV cache, will it affect performance? If it does, will this impact be significant?

[zhuohan123]

It will indeed significantly affect performance, especially when the request load is high. The memory size decides the maximum number of requests being batched together in one iteration, which directly determines the maximum throughput you can achieve. Therefore, we suggest allocating as much memory for KV cache as possible.

[Zhuqln]

got it！ thanks again:)

[humza-sami]

When I try to set it to gpu_memory_utilization = 0.8 (80%) memory then it gives:

ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine.

Although I am using llama-2 13B model which requires 26Gb VRAM on float16. I am using A40 48GB.

[oushu1zhangxiangxuan1]

How to set gpu_memory_utilization on each GPU?

[IEI-mjx]

It will indeed significantly affect performance, especially when the request load is high. The memory size decides the maximum number of requests being batched together in one iteration, which directly determines the maximum throughput you can achieve. Therefore, we suggest allocating as much memory for KV cache as possible.

How to allocating as much memory for KV cache as possible? I tried to implement long text inference with tp_size=8 on a 2B-32experts model. When i used 'watch -n 1 nvidia-smi' to monitor gpu usage, only 50GB/80GB was used. When i increase max_model_len to , OOM is encountered.