➤ Jeff DiNoto commented:

The design itself is already drafted, we’ll use this issue to plan out a review/discussion process to finalize the design.

So the goal here is a finalized design and a new Epic scoping out the work.

One note on how we stage this piece of work. I think we should consider supporting a manual repair/migration process first, then coming up with the fancy leader election process second.

We can do this by putting in some functionality that allows a control plane k8ssandra-operator instance to take control of an existing CassDC. This hits two birds with one stone:

1. Facilitating a migration pathway for existing DCs.
2. Allowing a new control plane to take over from an old one if it breaks somehow (by manually adding the DC).

That first step will probably throw up interesting problems around the way we've drawn boundaries between the control and data planes.

Second step will then be to do this via an inter-k8s-cluster leader election mechanism. That throws up additional questions about achieving distributed state consistency and such.

Keeping these two phases segregated will hopefully avoid some headaches.