Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update rediscluster client to v2.1.3 #598

Open
jerroydmoore opened this issue Jun 16, 2021 · 3 comments
Open

Update rediscluster client to v2.1.3 #598

jerroydmoore opened this issue Jun 16, 2021 · 3 comments

Comments

@jerroydmoore
Copy link
Contributor

jerroydmoore commented Jun 16, 2021

ElastiCache will occasionally replace nodes during a configurable maintenance window ("These replacements are needed to apply mandatory software updates to your underlying host"), they refused to provide information on the maintenance failover process, but it looks similar to CLUSTER FAILOVER (source).

For ElastiCache redisclusters that do not have replica nodes for failover, this can trigger an edge case, such that rediscluster tries to free a connection referencing a local variable before it is assigned (commit). This was patched in v2.1.3.

In our case, the rediscluster client entered a bad state, and all requests to the redis node failed. The only way to recover was a manual service restart.

@spladug
Copy link
Contributor

spladug commented Jun 16, 2021

Thanks for the report! That sounds like a nasty issue.

What would you like to see changed here? Baseplate.py only sets a minimum version (https://github.com/reddit/baseplate.py/blob/develop/setup.py#L15, 2.1.2) which we can happily bump if 2.1.2 is really bad. But the actual version used by your service is entirely up to your service, not Baseplate.

@jerroydmoore
Copy link
Contributor Author

I see. I suppose bumping the minimum version to 2.1.3 is the best course of action.

@spladug
Copy link
Contributor

spladug commented Jun 17, 2021

Cool! Are you interested in submitting a patch for that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants