k8ssandra repair taking days to complete

[judefdo-1]

We had a major disk corruption(AWS Volume) on one of our rack instance. After deleting the PVC and PV, We restarted the rack1.

Once pods are online, there is a huge data mismatch between the query. At some point, it will return 0 row. To fix this issue we started the repair using the reaper operator. But it is taking days, even though we stopped all our services.

[adejanovski]

Hi,

sadly this is not how replacements of broken nodes should be done. Repair is efficient for small entropy levels but is not meant to rebuild an empty node because it will re-stream data between all replicas, not just the ones that are missing the data.

There's a "Replace" CassandraTask that is meant for replacing broken nodes, which will recreate a pod with a new PVC and re-bootstrap it by replacing the old one: https://github.com/k8ssandra/cass-operator/blob/master/tests/testdata/tasks/replace_node_task.yaml