Resuming long running tasks from where they left off in sidekiq after SIGTERM/SIGKILL

[blaizmannen]

So on kubernetes, I have a sidekiq deployment. Whenever there is a scale-down operation/deployment update on the sidekiq deployment, the pods will receive a TERM, wait N seconds(25) and then eventually be killed. This causes long running jobs to be interrupted, and will have to be restarted from the start.

In hopes of fixing this, we came up with creating a preStop hook, that:

1. Sends a TSTP signal to the sidekiq process
2. Wait for a fixed timeout period (I have set this to 60s), but also poll the sidekiq process if there are still any active threads running. If 0, we can exit early.
3. After the execution of the preStop hook, sidekiq will then receive the SIGTERM signal, and wait N seconds (I have increased this to 45 seconds) before eventually being killed(SIGKILL).

However, when the job gets re-queued, it resumes from the very beginning.

I am unsure if I am missing something fundamental here. Is there something I can fix so that sidekiq will resume long running jobs from where they left off, or do I have to configure the job in such a way that supports this?

Here is the Worker job that I run to test this:

# For running long running sidekiq tasks
class TestWorker
  include Sidekiq::Worker
  def perform
    1.upto(400) do |i|
      # Print the number
      puts i
      # Sleep for 1 second
      sleep 1
    end
  end
end

[blaizmannen]

I will go ahead and answer my own question that indeed yes, Sidekiq by nature cannot possibly know how to resume the job from where it left off without actually changing the Job itself