[Usage]: How to offload some layers to CPU？

[cheney369]

Your current environment

None

How would you like to use vllm

I want to load qwen2-14B-chat using VLLM, but I only have 1 RTX4090(24G).
Can vllm offload some layers to cpu and others to gpu?
As I know, the transformers-accelerate and llama.cpp can do it. But I want to use the multilora switch function in VLLM.

[eigen2017]

#3563

[eigen2017]

#627

[eigen2017]

bd-iaas-us#1

[eigen2017]

bd-iaas-us#3

[eigen2017]

it's not a good idea to use cpu mem since vllm is for inference accelerate .
there is a trade-off choice that if we can cut some weight to fit poor HBM, moe models can cut some experts etc.
see this: huggingface/transformers#30552