-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: support adaptive batching #382
Labels
enhancement
New feature or request
Comments
Will create another issue to track this one. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the feature
refer to the paper:
Basically, it will adjust the batch size or wait time according to the load pressure.
This can be helpful when the traffic is not evenly distributed, which is very common. During the low-traffic period, waiting for a large batch size or a long wait time is unnecessary.
Why do you need this feature?
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: