Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scheduler filter include streaming part #2579

Open
szuecs opened this issue Sep 7, 2023 · 0 comments
Open

scheduler filter include streaming part #2579

szuecs opened this issue Sep 7, 2023 · 0 comments

Comments

@szuecs
Copy link
Member

szuecs commented Sep 7, 2023

Is your feature request related to a problem? Please describe.

As owner of a streaming application that only support N concurrent users I want to be able to limit the traffic to N.
Right now if you do have a backend that does async response where the streaming part can be long, you get much more than N requests, because the scheduler filters response part is triggered before the response body streaming is happening.

Describe the solution you would like

Make it possible to limit streaming concurrency, too.
Basically decrease the concurrency counter after body streaming was done.

Describe alternatives you've considered (optional)

We tried lifo which does not work as expected and a workaround seems to be backendRatelimit which has a lower error margin than lifo for an application which streams the response.

Additional context (optional)
internal finding

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant