load multiple (two) model-ids

[tbugfinder]

Hello,

which would be the proper approach load load two (mor more) model-ids so that promptfoo the could be used to connect to either of those?

Thanks

[aarnphm]

We currently don't support loading more than one id.

I don't think yet it is feasible. Most of the larger models, unless you have a100, will needs all of the GPU memory to load it in, so there is not really a good reason to load more than 1 model

[aarnphm]

with that being said, for 7b models, given two a10g, you can do the following

CUDA_VISIBLE_DEVICES=0 openllm start meta-llama/Llama-2-7b-chat-hf --port 8080 --backend vllm

CUDA_VISIBLE_DEVICES=1 openllm start HuggingFaceH4/zephyr-7b-beta --port 8081 --backend vllm

You can then use a asynchronous clients like below to run comparison benchmark

import asyncio
from openllm_client import AsyncHTTPClient

llama = AsyncHTTPClient('http://localhost:8080')

zephyr = AsyncHTTPClient('http://localhost:8081')

async def comparison(prompt):
  return asyncio.gather([llama.generate(prompt), zephyr.generate(prompt)])

In your batch job somewhere

for cloud deployment I recommend BentoCloud, as it is much easier to deploy and manage these multiple models on k8s

[tbugfinder]

Thank you very much! That's helpful, although I certainly understand the resource constraints.