Bug: memphis statefulset cant start after fail over

[waruwat-kkp]

Describe the bug

I test fail over by kill all pod of memphis, memphis can't start back and got following error

container : memphis

nats-server: invalid padding size

container : reloader

2023/08/23 05:03:34 Starting Memphis Config Reloader vversion-not-set
2023/08/23 05:03:34 Error: open /var/run/memphis/memphis.pid: no such file or directory
2023/08/23 05:03:36 Error: open /var/run/memphis/memphis.pid: no such file or directory
2023/08/23 05:03:38 Error: open /var/run/memphis/memphis.pid: no such file or directory
2023/08/23 05:03:40 Error: open /var/run/memphis/memphis.pid: no such file or directory
2023/08/23 05:03:42 Error: open /var/run/memphis/memphis.pid: no such file or directory
2023/08/23 05:03:44 Error: open /var/run/memphis/memphis.pid: no such file or directory
Error: Too many errors attempting to find server process

Steps to reproduce

1. kill all pod of memphis : statefulset, deployment
2. waiting for pod re-creation

Affected services

Broker

Broker version

v1.1.1-stable

Platforms

Kubernetes

If UI - Browsers

No response

Environment

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[yanivbh1]

Hey @waruwat-kkp ,
It will be reviewed as soon as possible.
Which flavor and version of K8S are you using? If it is managed - by which vendor?
Have you deployed Memphis in cluster mode?

[idanasulin2706]

Hi @waruwat-kkp this issue you described is happening when memphis secret keys are getting changed.
That means that some encrypted values that have been encrypted with the older keys can not been decrypted with the new ones.
Now this situation might happen in one of two options:

1. You changed the memphis secrets manually.
2. You didn't set them to be constant, in this case memphis is randomly generate them on your first helm install and on helm upgrades.

So any chances that these values changed somehow?

And anyway I'm attaching the relevant section in our docs which explains about it.

Here you can make them constant for your choice on installation (creds section)

https://docs.memphis.dev/memphis/open-source-installation/kubernetes/1-installation#optional-helm-deployment-options

This one is for manual upgrade

https://docs.memphis.dev/memphis/open-source-installation/kubernetes/how-to-upgrade#upgrade-memphis-cluster-with-helm-upgrade-using-a-manual-rolling-upgrade