Model pool and model level load balance for large number of requests

[yifa-n]

Sometime I reached max token limitation for batch workload
I want this feature enable large number of requests to LLM for concurrency.
For usage scenario, there is some rate limit for single model, can we config a model pool and load balancer for LLM requests to support large number of requests concurrency.