New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ability to access process identity within the job #6215
Comments
That's a good summary of Sidekiq Pro's super_fetch. As for your in_progress status, I would use a database row lock instead. If your process dies, the lock should go away quickly. https://www.postgresql.org/docs/current/explicit-locking.html |
Indeed, we use this approach often but in this case, the lock will interfere with other processes and impact the performance. Plus locks are available within transactions, which brings other limitations. Tracking the identity of the responsible process seemed the easiest fix to this problem, as in this case we only lock the record for a short time to change its status and save the If forwarding the identity is too much pain, then probably having a separate table where we can store the |
Hi, I know the jobs have to be fast, idempotent, and don't rely on any state but the reality isn't always as expected. We have jobs that may run for quite a bit, which leads to jobs being returned to the queue (e.g. during deployment). We also need to make sure only one job processes a task at a time and for this reason, we set the task status to
in_progress
once it is started. And if another job starts this same task, it first locks the task in the database and checks its status, if it isin_progress
then no action is taken.The problem is, if the job hasn't finished, the task remains in progress as there is no mechanism to notify the workers to stop when Sidekiq moves into the "quiet" mode. And when this same job is restarted later it sees the task is in-progress and quits. We solve this by saving the
jid
into the task and comparing it with thejid
within the job, if it is the same we allow the job to proceed despite the in-progress status.But sometimes the process also crashes due to other reasons we can't control, and in that moment we are in trouble as the jobs are sometimes gone forever and the tasks stuck forever. In addition to
jid
, we would also like to store theidentity
of the process that started the task. Then we can have a scheduled job that would recover such orphaned tasks by simply checking the presence of the process. Once the process crashes, its identity in Redis expires after some short time and the task can be recovered.Is that something you'd be willing to add to Sidekiq?
The text was updated successfully, but these errors were encountered: