You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some times we need to enqueue a large batch of jobs (example: update x on every account)
we avoid doing this in a single job because that makes for a long running job.
when we use multiple jobs this can enqueue thousands of jobs. Even though these jobs are individually quick, it can take minutes for the queue to clear.
When the queue gets back up, other jobs can get delayed. What are the strategies people use to mitigate this?
Option 1: create a new queue for lower priority work. Downside, extra cost to run additional queues
Option 2: I was hoping there was some way to use Batches from sidekiq pro to limit the number of threads a batch could use. Example: enqueue this whole batch of jobs, but only use a maximum of 3 threads so we leave threads for other jobs.
Option 3: make one job enqueue the next one. Downside, much more complicated to setup.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Some times we need to enqueue a large batch of jobs (example: update x on every account)
When the queue gets back up, other jobs can get delayed. What are the strategies people use to mitigate this?
Option 1: create a new queue for lower priority work. Downside, extra cost to run additional queues
Option 2: I was hoping there was some way to use Batches from sidekiq pro to limit the number of threads a batch could use. Example: enqueue this whole batch of jobs, but only use a maximum of 3 threads so we leave threads for other jobs.
Option 3: make one job enqueue the next one. Downside, much more complicated to setup.
Anyone have any recommendations?
Beta Was this translation helpful? Give feedback.
All reactions