Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Purge old keys without TTL from Redis #218

Open
in03 opened this issue Aug 26, 2022 · 1 comment
Open

Purge old keys without TTL from Redis #218

in03 opened this issue Aug 26, 2022 · 1 comment

Comments

@in03
Copy link
Owner

in03 commented Aug 26, 2022

Should be easy enough, but not high priority.

We should figure out how to purge old keys that don't have a TTL. Kombu bindings to old queues with old version constraints lay around until someone cleans them up, which for most end users will be never. Deleting queues that are still in use results in those queues being created again when Celery attempts to use them. There may be some slight overhead there, but no great loss.

There are also a few stray _kombu.binding.reply.celery.pidbox keys. Ask Solem writes:

These replies are sent for remote control commands. I know flower is constantly sending out control commands, but I'm not sure there is an easy fix. The best option would be if the replies were using PUBSUB instead of being stored in a list.

celery/kombu#294

The keys are very small, but assuming we can just keep them forever may not be realistic. Especially if users have made their Redis instance persistent. We could:

  • Create a clean up command to remove old keys
  • Clean up after x amount of program runs
  • Clean up as part of install / upgrade setup

or a combination of the three.

Here's a sample of the keys from Redis Insight:

image

@in03 in03 added this to To do in Development via automation Aug 26, 2022
@in03
Copy link
Owner Author

in03 commented Oct 17, 2022

Now that we're using the native Celery AsyncResults instead of Redis pubsub, it might be a good idea to bundle this into some kind of general cleanup command. I'm not sure if RabbitMQ and some of the other popular broker backends need occasional tidying.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

1 participant