How to best set the --max-num-batched-tokens arg?

[edwardzjl]

I'm serving a gptq model with 2 GPUs with vllm for some days, and I encountered cuda OOM this morning, so I started investigating the engine args.

To my understanding, the --max-num-batched-tokens arg controls the maximum tokens can be placed in the GPU, so I need to set this arg to some value that 1. to avoid cuda OOM 2. to maximum utilize my GPU VRAM. (Correct me if I am wrong)

However, I don't know how to set this arg properly, there's little information in the doc.