Upgrading fails and does not have a recovery or pickup-where-left-off option

[onabison]

Hi,

I noticed a behavior that I believe others will probably be facing as well.

I upgraded my cluster from 1.23 to 1.2 (I don't believe the version are relevant, but figured I'd mention anyway). If the upgrade failed for whatever reason, and you re-run the command you will be facing something like this:

INFO:eksupgrade.src.self_managed:The cluster lab-poc-eks-cluster in region us-east-1 ASGs of the self-managed nodegroups: ['eks-on-demand-96c071ab-6ec9-2249-37e0-9c8e4975b944', 'eks-spot-a2c071ab-6ecf-329a-8ec4-72654fffbe40']
INFO:eksupgrade.starter:The Manged Node Groups Found are eks-on-demand-96c071ab-6ec9-2249-37e0-9c8e4975b944,eks-spot-a2c071ab-6ecf-329a-8ec4-72654fffbe40
INFO:eksupgrade.src.boto_aws:ASG Matched = eks-on-demand-96c071ab-6ec9-2249-37e0-9c8e4975b944 ,eks-spot-a2c071ab-6ecf-329a-8ec4-72654fffbe40
INFO:eksupgrade.starter:The ASGs Found Are eks-on-demand-96c071ab-6ec9-2249-37e0-9c8e4975b944,eks-spot-a2c071ab-6ecf-329a-8ec4-72654fffbe40
INFO:eksupgrade.starter:The add-ons Update has been initiated...
INFO:eksupgrade.starter:The Addons Upgrade Started At 1677258083.007861
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:eksupgrade.src.k8s_client:The Addons Found = ['aws-load-balancer-controller-68f585b748-4cq7z', 'aws-load-balancer-controller-68f585b748-lx4fb', 'aws-node-9h8pn', 'aws-node-n75pl', 'aws-node-n8vm5', 'aws-node-x5vwb', 'coredns-79fffc5cf7-8kgkd', 'coredns-79fffc5cf7-8r6nz', 'csi-secrets-store-provider-aws-2ftmh', 'csi-secrets-store-provider-aws-4dc4p', 'csi-secrets-store-provider-aws-cr5ff', 'csi-secrets-store-provider-aws-d9lsc', 'csi-secrets-store-secrets-store-csi-driver-5gw4c', 'csi-secrets-store-secrets-store-csi-driver-btvs5', 'csi-secrets-store-secrets-store-csi-driver-pphpw', 'csi-secrets-store-secrets-store-csi-driver-w788s', 'ebs-csi-controller-78b4fdb7f5-bnm6d', 'ebs-csi-controller-78b4fdb7f5-drms5', 'ebs-csi-node-2q4rq', 'ebs-csi-node-6cvrq', 'ebs-csi-node-n4fcf', 'ebs-csi-node-srxr9', 'kube-proxy-8rfsb', 'kube-proxy-b5jfv', 'kube-proxy-j5cx6', 'kube-proxy-q9dbj']
INFO:eksupgrade.src.k8s_client:kube-proxy-8rfsb Current version: v1.24.7-minimal-eksbuild.2 Updating to: v1.24.7-eksbuild.2
INFO:eksupgrade.src.k8s_client:Updating the EKS cluster's kube-proxy add-on version via the EKS API...
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:eksupgrade.src.k8s_client:Total Pods With kube-proxy = 4
INFO:eksupgrade.src.k8s_client:Old kube-proxy pod: kube-proxy-8rfsb - New kube-proxy pod: kube-proxy-8rfsb
INFO:eksupgrade.src.k8s_client:kube-proxy-b5jfv Current version: v1.24.7-minimal-eksbuild.2 Updating to: v1.24.7-eksbuild.2
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:eksupgrade.src.k8s_client:Total Pods With kube-proxy = 4
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:eksupgrade.src.k8s_client:Total Pods With kube-proxy = 4
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:eksupgrade.src.k8s_client:Total Pods With kube-proxy = 4
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:eksupgrade.src.k8s_client:Total Pods With kube-proxy = 4
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:eksupgrade.src.k8s_client:Total Pods With kube-proxy = 4
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:eksupgrade.src.k8s_client:Total Pods With kube-proxy = 4
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials

In my case, my spot instances failed to upgrade and re-running the command to try and upgrade the spot instances will end up with the above endless loop. It seems it already went over the NodeGroup but failed to detect that there is still a component that has not been upgraded and moved on to the add-on.