Throttled workers seem to eat up a lot of processing time?

[vincentwoo]

Hi, I'm on the latest versions of sidekiq and sidekiq-throttled.

I have a job with max rate of 10 / hr:

  sidekiq_throttle({
    # Allow maximum 10 jobs being processed within one hour window.
    threshold: { limit: 10, period: 1.hour }
  })

However, these jobs appear in the same queue as many other jobs. If I queue up hundreds of these throttled jobs, only ten will be processed (correctly!) but all the throttled ones need to move up and down the queue every second to be discarded as throttled. This chokes out a lot of other work and reduces the overall throughput of the system.

Is it possible for sidekiq-throttled to reschedule the work for later? If we know we only want 10 in an hour, we should be able to delay until we know the bucket will be refreshed, no?

[vincentwoo]

I believe this can be addressed by changing the return value of throttled? to something like false if not throttled, true if throttled without an ETA for clearing up, or an int if we can estimate the soonest possible time to schedule work again.

[ixti]

It's a bit tricky question. First of all scheduling jobs won't help a lot, because scheduled jobs is a pretty heavyweight stuff in Sidekiq. But I definitely see couple of improvements that can help:

1. Allow throttling on per-queue level, in this case we can stop polling queue for ttl if we hit threshold
2. Allow person set whenever or not he's OK to make sidekiq-throttle handle rescheduling instead of queue rotation

[cassiopagnoncelli]

Have you tried this?

sidekiq_throttle({ threshold: { limit: 1, period: 6.minutes } })

[adamof]

@vincentwoo did you find a solution to your problem? :)

[rhetprieto]

A potential solution could be using multiple queues so instead of sending the throttled work to the beginning of the current queue, we could send it to a lower priority queue so we make sure the performance of the other workers that should be running won't be affected.

[cfc1020]

Hi Alexey @ixti ,

Thanks for your work on this gem. It’s a brilliant solution that many projects need to have.

Actually, I also encountered the problem described in this topic. As a solution, I run a separate sidekiq server in an isolated machine which processes jobs(queue) with rate limits. So it allows us not to affect other queues/servers which don’t have any rate limits.

However, it's still wasting CPU time when limits are noticeably less than the outstanding work on a certain day. I really like your proposed solution (#2) since Sidekiq itself nicely deals with scheduled jobs.

What do you think now if that's worth it to be done?

[ixti]

@cfc1020 I think it might be worth allowing to configure how jobs should be getting back to the queue immediately or via scheduler. Probably it will make sense to make configurable queue where throttled job should be re-scheduled:

sidekiq_throttled ..., :on_throttle => { :strategy => :enqueue }  # current behaviour
sidekiq_throttled ..., :on_throttle => { :strategy => :schedule } # use scheduled jobs

sidekiq_throttled ..., :on_throttle => { :strategy => :schedule, :to => "throttled" }
# use scheduled jobs, but schedule to be pushed to different queue

[ixti]

With v1.0.0 release you can now configure cooldown periods of queues:

Sidekiq::Throttled.configure do |config|
  # skip queue for 10 seconds if it returned ‹cooldown_threshold›
  # throttled jobs in a row
  config.cooldown_period = 10.0

  # if queue returned 42 throttled jobs in a row, skip it from
  # polling for the duration of ‹cooldown_period›
  config.cooldown_threshold = 42
end

[ixti]

I hope to get #150 merged soon (depending on my free time) so that there will be another way to aid this problem as well.