Allow throttle queues

[ixti]

It's possible to throttle queue by using shared throttling key, but that causes jobs to be pushed back to the end of the queue. It would be nice to have real queue throttling, where job would have been pushed back to the head of the queue, and queue have been paused from fetching for some time.

See: #122

[joevandyk]

Can you give an example of throttling a queue by a shared key? The docs only seem to mention a key_suffix?

My scenario: I have multiple job classes that run on the expensive queue. I only want one job at a time to run on the expensive queue.

[mnovelo]

@joevandyk below is what we're doing right now for throttling multiple jobs with a shared key (not necessarily throttling the entire queue, but if only jobs that have this throttle are put in the queue, then the queue is essentially throttled)

in a Rails initializer file,

# For Users::Engagement::SetSendsJob, SetOpensJob, and SetClicksJob
Sidekiq::Throttled::Registry.add(:users_engagement_set_etl,
                                 # Only allow 1 job per tenant at a time, for up to 20 tenants at at time
                                 concurrency: [
                                   { limit: 1, key_suffix: ->(args) { args['tenant'] } },
                                   { limit: 20 },
                                 ])

Then, in the individual jobs, we add

sidekiq_throttle_as :users_engagement_set_etl

[jcsrb]

I am testing the usage of this as a migration path from sidekiq-limit_fetch
I followed @mnovelo sample and registered multiple concurrency throttles based on the queues and applied it to the relevant jobs

having concurrently limited jobs in different queues causes an unexpected effect:
let's say we have 3 queues, high/normal/low , 10 threads in this example

if all jobs enqueued in the high queue have a shared concurrency limit of 3, sidekiq would run 3 as expected, but would not fill the rest of the threads with remaining low jobs

this is because how the sidekiq queues work https://github.com/sidekiq/sidekiq/wiki/Advanced-Options#queues , but need to be considered for queue-based throttling
queue weights might help adjust this

[mnovelo]

Oh that's interesting @jcsrb. We strictly use weighted queues so we've not run into that