Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bad performance after updating to v2.6.1 #92

Open
timmipetit opened this issue Jun 17, 2022 · 7 comments
Open

Bad performance after updating to v2.6.1 #92

timmipetit opened this issue Jun 17, 2022 · 7 comments
Labels
bug Something isn't working

Comments

@timmipetit
Copy link

After updating to v2.6.1, which introduced #91, we've seen a enormous drop in performance for our collection script. This is interesting, because the merge request is meant to increase performance.

Here's a screenshot from the average response time of our prometheus collection script:
Screenshot 2022-06-17 at 11 50 11

@LKaemmerling
Copy link
Member

Hey @timmipetit,

can you try to downgrade to v2.6.0 just to make sure it is really related to the script? How many metrics do you expose? Might it be that the redis has a lot of keys?

@LKaemmerling LKaemmerling added the bug Something isn't working label Jun 17, 2022
@timmipetit
Copy link
Author

Hi @LKaemmerling. We've downgraded back to v2.6.0, and performance is back to normal for us. We don't expose a lot of metrics, 6 histograms (with 16 buckets each) and 3 gauges (so no summaries). But our Redis instance is also used for sessions though, so we do have a lot of other keys (400k or so). The keys used for sessions have a different prefix. But since scan will iterate all keys, it will fetch all session keys (before discarding them again because they don't match)

@grogy
Copy link

grogy commented Jun 29, 2022

I have to say the same from project https://supportbox.com/ - after upgrade to 2.6.1 so bad performance. Wipe storage (Redis) does not help.

@LKaemmerling
Copy link
Member

Hello together,

i just released v2.6.2 with the revert of the given MR.

@LKaemmerling
Copy link
Member

Hey @grogy & @timmipetit,

would it be possible that you test https://github.com/PromPHP/prometheus_client_php/pull/99/files against your workload (new StorageEngine RedisNG). This basically fixes the usage of KEYS and SCAN.

@timmipetit
Copy link
Author

@LKaemmerling Thanks for the heads up. We'll test it

@timmipetit
Copy link
Author

Tested on our workload, works fine!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants