-
-
Notifications
You must be signed in to change notification settings - Fork 243
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run Task One Another / Control Concurrency in Queues #364
Comments
Coravel could offer something like that. The problem with this though is that this would apply to all messages in the queue and would significantly reduce throughput across the board. So this would depend on having multiple independent queues, and then implement what you've suggested per queue. This would be a bit of work 😅. One approach that could work, is you have your queue message handler put a "job" item in storage (a database table, document, etc.). Then, using a the scheduler, every X seconds (or whatever), you can purge that database table (etc.) using the solution you suggested. This would work well because the scheduler has the option to PreventOverlap so that you can more reliably process these items. For example, the queue doesn't support preventing any overlap which could occur in the example you gave, if implemented inside of the queue message handler (e.g. your first queue handler is doing web scrapping but then the queue triggers a purge/consumation 30 seconds later. The first one isn't done yet...). You could use a That would be my suggestion at the moment: queued message complete quickly and don't get blocked/slowed by writing a job object to storage. Have a scheduled job to process those jobs every X seconds and use Have any thoughts on this approach? (It's a common pattern to use for this specific scenario and is exactly why the |
I saw this but still want to know if we can do something about this - #325
Problem Using WhenAll()
I created a WebScrapper which scraps a particular website. Users can log in and click a button to queue the scrapping request. The problem is when 50 uses queues the scrapping job of all 50 starts at the same time.
This causes immense resource problems in my server sweeping our all available RAM. I searched everywhere to see if I can control Coravel to run 5 tasks at a time or 1 or 2. I mean if the dev can control.
I understand from #325 that it's by design we use
Task.WaitAll()
, but this is a bigger concern than any workarounds.What if we add a Semaphore to control the number of concurrent tasks that can execute in a queue, Maybe something similar to this
What do you think?
The text was updated successfully, but these errors were encountered: