New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Investigate cluster CPU spikes #727
Comments
The job is searching for orphaned documents across all time. Since we are running this on a regular basis, we can probably restrict it to only check events within the last 3 days or so (forgot how far back we allow events to be post dated). |
I guess what would be better is to find recently deleted stacks, projects, orgs and see if there is any matching events. Ideally we wouldn't have to worry about this happening. We need to bullet proof the process of deleting projects, stacks and orgs and make sure it's impossible for for them to be deleted if there are any matching events. We might have a concurrency issue where a stack is deleted but a new event comes in at the same time and gets added to that stack even though it was just deleted. |
@ejsmith We only allow event submission for the past 3 days. Maybe we only run the full check once a month or once a week? |
Happening every 8 hours. Orphan data cleanup is the only thing that happens every 8 hours. Doesn't look like the job itself is taking a lot of CPU, but ES CPU spikes during that time. So my guess would be whatever we are doing in orphan data cleanup is very expensive in ES.
The text was updated successfully, but these errors were encountered: