-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Jobs list on Busy page can leak #6227
Comments
Here's the results of tuning the operation. Mainly I've restructured the code to take advantage of the fact that HSET can accept multiple key/value pairs. Today the code only pushes one at a time which leads to a lot of time waiting on network latency. By restructuring, we greatly reduce the amount of time spent setting the hash data and so we greatly reduce the window of time where data can be orphaned. Original / Tuned has 25 jobs in progress. The small variant only has one job in progress.
Time constructing hash (and thus the size of the race condition window) Before: 2.37965 That's a 90% drop, a fantastic result. |
And with that optimization, I think we can move back to use MULTI, removing the race condition completely. |
If you kill -9 Sidekiq while it is updating the in-progress job data (as displayed on the Busy page), there's a possibility to leak the data by losing the expiry for the data. This will result in rows in the Jobs table which never disappear.
Original discovery and example script to fix data here.
The text was updated successfully, but these errors were encountered: