How to specify which gpu to use?

[Zhuqln]

If I have multiple GPUs, how can I specify which GPU to use individually?

Previously, I used 'device_map': 'sequential' with accelerate to control this. Now, with vllm_engine, is there a similar functionality available?"

exp. if i want to specify the gpu1 instead of gpu0,because they are not same type of card.i will write like this:

model_kwargs = {"torch_dtype": torch.float16, 'device_map': 'sequential'}

Then, I manually set the available memory of device0 to 0, with the purpose of using device1:

GPU_max_memory: {0: '0GiB', 1: '40GiB'}

[gesanqiu]

Use CUDA_VISIBLE_DEVICES=0,1,2,3 python -m vllm.x.x.x. [parameters] to specify the gpus you want to use in a process level.

[Zhuqln]

Thank you for the suggestion.
I am aware of this method, but due to my project's architecture, I need to be able to start my worker through a triggered request.
Using the "python" command directly in the terminal is not suitable for me. Nevertheless, I am delighted to receive your response. Thanks again.

[zhuohan123]

In addition to @gesanqiu's method, you can also use Ray's placement group feature for this. Feel free to follow up on discussion if you meet any issues or have any other suggestions!

[zhiqiangzhongddu]

A simple solution is adding this to your code:
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"

Then, only GPU-0,1 are visible.

[BJTFDGR]

Thanks!

[humza-sami]

@zhiqiangzhongddu I have 4 RTX 4090 GPUs available, and I want to run a LLM with a size of 42GB on 2 RTX 4090 GPUs (~48GB) and a separate model with a size of 22GB on 1 RTX 4090 GPU(`24GB).

from vllm import LLM
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "1,2"
llm_1 = LLM(llm_1_name,max_model_len=50,gpu_memory_utilization=0.9, tensor_parallel_size=2)

os.environ["CUDA_VISIBLE_DEVICES"] = "3"
llm_2 = LLM(llm_2_name,max_model_len=50,gpu_memory_utilization=0.9, tensor_parallel_size=1)

this still loads 2nd llm on 1 and 2 gu and gives memory error

[timeconnection]

export CUDA_VISIBLE_DEVICES=1

[sAviOr287]

Hi
is there any chance there is an example script on how to place LLMs on different GPUs?
That would be amazing and really helpful
Thanks a lot in advance.
@zhuohan123

[sparsh35]

Any solution for this