ScaleLLM vs vLLM in performance

[WangErXiao]

Is there comparison performance data between ScaleLLM and vLLM

[zhyncs]

Hi @guocuimi Thanks for your outstanding work. In addition to performance comparison with vLLM, if possible, please consider adding TensorRT-LLM, LMDeploy, RTP-LLM, and TGI. And maybe we could use vLLM benchmark serving. Thanks.

[guocuimi]

thank you for your interest in ScaleLLM. Yeah, it is indeed in our roadmap. we do have some internal numbers but not ready to share yet. As part of our upcoming plans, we will do a comprehensive comparisons (in a separate ropro) in coming weeks after finishing python wrapper part. Stay tuned!

Meanwhile, feel free to conduct your own benchmarks for your specific scenarios using the vLLM benchmark serving script. Thanks.

[zhyncs]

Hi @guocuimi May you use GitHub Action to release the Python Package? Consider supporting CUDA 11.8 and CUDA 12.2, which will make it more convenient for users to use. At the same time, we can easily compare performance with other frameworks by using the compatible OpenAI Server.

[guocuimi]

thanks for your advice. yeah, it is our plan. i am working on setting up the whl build for each release. for now, i am trying to reduce whl size first. should be ready this week. stay tuned!

[guocuimi]

Hi @zhyncs A quick update for you: python is supported in latest release.
you can install scalellm with pip: pip install scalellm and start rest api server with python3 -m scalellm.serve.api_server
Please let me know if you have any questions. thanks

[zhyncs]

Hi @zhyncs A quick update for you: python is supported in latest release. you can install scalellm with pip: pip install scalellm and start rest api server with python3 -m scalellm.serve.api_server Please let me know if you have any questions. thanks

cool! I will verify it asap, thanks.

[zhyncs]

Hi @guocuimi The package you are currently compiling in GitHub Action depends on GLIBC_2.27, which is not very friendly for the widely used CentOS 7 in the industry and still requires manual compilation.

[guocuimi]

Thanks for letting me know. Let me try to downgrade GCC to 10 and republish new packages using manylinux2014
(CentOS 7 based)

Toolchain: GCC 10

[zhyncs]

republish new packages

The latest version is ok https://github.com/vectorch-ai/ScaleLLM/releases/tag/v0.1.3
And may you update the doc https://github.com/vectorch-ai/ScaleLLM/tree/main/docs/source For example, how to set up an OpenAI-compatible server. Thanks.

[zhyncs]

If the interface is compatible, then we may directly use vLLM's script for benchmark at https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_serving.py

The recent BentoML blog post https://www.bentoml.com/blog/benchmarking-llm-inference-backends can also serve as a reference.

[guocuimi]

Yeah, we can use it directly. Just sharing our plans on this: One continuous benchmark will be setup in coming weeks, comparing offline and serving metrics between scalellm, vllm and tensorrt-llm. We are working on it. Thanks, Michael Mi

…

On Mon, Jun 10, 2024 at 7:44 PM zhyncs ***@***.***> wrote: If the interface is compatible, then we may directly use vLLM's script for benchmark at https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_serving.py The recent BentoML blog post https://www.bentoml.com/blog/benchmarking-llm-inference-backends can also serve as a reference. — Reply to this email directly, view it on GitHub <#144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV4CLUVFSMEAZ3Y232FHJDZGZQBJAVCNFSM6AAAAABG4ASAOSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJZGY3DKNJZGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zhyncs]

we can use it directly

I gave it a quick try, and there seems to be a problem at the moment.

https://github.com/vllm-project/vllm/blob/351d5e7b8253d754b2a951152cd48927c4c1629d/benchmarks/backend_request_func.py#L261-L262

python3 -m scalellm.serve.api_server --model /workdir/Llama-2-13b-chat-hf

python3 benchmark_serving.py --port 8080 --dataset /workdir/ShareGPT_V3_unfiltered_cleaned_split.json --model Llama-2-13b-chat-hf --tokenizer /workdir/Llama-2-13b-chat-hf --num-prompts 1 --request-rate 1

[zhyncs]

in coming weeks

Looking forward to your results.

[guocuimi]

Thanks, never tried that benchmark script. Will try that after wrapping up current feature parity works for logprobs and best_of. Thanks