After canary upgrade, the datacenter will refuse to update StatefulSets

[burmanm]

What happened?

When trying to get all the nodes to the same state after doing a canary upgrade, we run into a situation where the check for StatefulSet being completed will always fail and the operator will output:

2024-05-16T05:41:28.555Z        INFO    waiting for upgrade to finish on statefulset    {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"test-canary-upgrade"}, "namespace": "test-canary-upgrade", "name": "dc1", "reconcileID": "56346c1d-4130-4020-b659-2f18f8ca4a0c", "namespace": "test-canary-upgrade", "datacenterName": "dc1", "clusterName": "cluster1", "statefulset": "cluster1-dc1-r1-sts", "replicas": 3, "readyReplicas": 3, "currentReplicas": 2, "updatedReplicas": 1}

There's no recovery from this, since what cass-operator expects is:

			if statefulSet.Generation != status.ObservedGeneration ||
				status.Replicas != status.ReadyReplicas ||
				status.Replicas != status.CurrentReplicas ||
				status.Replicas != status.UpdatedReplicas {

However, the StatefulSet itself will output the following status (in 1.28):

status:
  availableReplicas: 3
  collisionCount: 0
  currentReplicas: 2
  currentRevision: cluster1-dc1-r1-sts-59d575d967
  observedGeneration: 2
  readyReplicas: 3
  replicas: 3
  updateRevision: cluster1-dc1-r1-sts-5fc6f6758f
  updatedReplicas: 1

This is the final and correct state. Sadly, there's no way to tell from the status if the canary upgrade was actually reached correctly.

What did you expect to happen?

We should be able to continue the upgrade by simply removing the canary upgrade requirements.

How can we reproduce it (as minimally and precisely as possible)?

See the test in #654

cass-operator version

1.20

Kubernetes version

1.28

Method of installation

No response

Anything else we need to know?

No response