[Bug] eksctl utils update-aws-node downgraded aws-node significantly instead of upgrading it

[consideRatio]

What were you trying to accomplish?

To perform a routine upgrade of a EKS cluster, upgrading aws-node daemonset along with it using eksctl utils update-aws-node.

What happened?

When following cluster upgrade docs (https://eksctl.io/usage/cluster-upgrade/), linking out to addon upgrades docs (https://eksctl.io/usage/addon-upgrade/), the step eksctl utils update-aws-node --cluster=<clusterName> --approve led to an unexpected downgrade of aws-node.

This is contradicting the help string of eksctl utils update-aws-node that sais Update aws-node add-on to latest released version.

I expected to get the version v1.18.1-eksbuild.1 as that looks currently recommended for 1.29 clusters via official EKS docs, but instead I got downgraded from v1.16.0-eksbuild.1 to v1.12.6

$ kubectl get ds -n kube-system aws-node -o yaml | grep image
        image: 602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni:v1.16.0-eksbuild.1
        imagePullPolicy: IfNotPresent
        image: 602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon/aws-network-policy-agent:v1.0.7-eksbuild.1
        imagePullPolicy: IfNotPresent
        image: 602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni-init:v1.16.0-eksbuild.1
        imagePullPolicy: IfNotPresent

$ eksctl utils update-aws-node --config-file=$CLUSTER_NAME.eksctl.yaml --approve

2024-05-15 12:59:53 [ℹ]  replaced "CustomResourceDefinition.apiextensions.k8s.io/eniconfigs.crd.k8s.amazonaws.com"
2024-05-15 12:59:53 [ℹ]  skipped existing "kube-system:ServiceAccount/aws-node"
2024-05-15 12:59:53 [ℹ]  replaced "ClusterRole.rbac.authorization.k8s.io/aws-node"
2024-05-15 12:59:54 [ℹ]  replaced "ClusterRoleBinding.rbac.authorization.k8s.io/aws-node"
2024-05-15 12:59:55 [ℹ]  replaced "kube-system:DaemonSet.apps/aws-node"
2024-05-15 12:59:55 [ℹ]  "aws-node" is now up-to-date        

$ kubectl get ds -n kube-system aws-node -o yaml | grep image
        image: 602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni:v1.12.6
        imagePullPolicy: IfNotPresent
        image: 602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni-init:v1.12.6
        imagePullPolicy: IfNotPresent

How to reproduce it?

I think it reproduces by setting up a eksctl cluster and then running the eksctl utils update-aws-node command.

Anything else we need to know?

The eksctl cluster config looks like this, plus not included nodeGroups entries.

{
   "addons": [
      {
         "name": "aws-ebs-csi-driver",
         "version": "latest",
         "wellKnownPolicies": {
            "ebsCSIController": true
         }
      }
   ],
   "apiVersion": "eksctl.io/v1alpha5",
   "availabilityZones": [
      "us-east-2a",
      "us-east-2b",
      "us-east-2c"
   ],
   "iam": {
      "withOIDC": true
   },
   "kind": "ClusterConfig",
   "metadata": {
      "name": "bican",
      "region": "us-east-2",
      "version": "1.29"
   },
   "nodeGroups": [...]
}

Versions

$ eksctl info

eksctl version: 0.176.0
kubectl version: v1.28.9
OS: linux

[github-actions]

Hello consideRatio 👋 Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[consideRatio]

It seems like the version 1.12.6 is hardcoded, so it does update to the latest aws-node version provided by eksctl, but that is older than the version you get through the initial install - so updating leads to downgrading.

In #6692 I see how the hardcoded eksctl version has been updated historically.

For k8s 1.29, version 1.14.1+ is recommended via https://github.com/aws/amazon-vpc-cni-k8s?tab=readme-ov-file#recommended-version, so in practice what happens now is that we get downgraded below the recommended version.

/cc: @wind0r and @Himangini as the author and reviewer of #6692 helping this be updated historically.

[consideRatio]

I opened #7756 to do a one-time bump.

[cPu1]

Thanks for the detailed issue! In addition to your PR, we plan on improving this experience by always installing EKS managed addons by default when addons are not specified, instead of letting EKS install self-managed addons. This should eliminate the need to maintain manifests with hard-coded versions in eksctl, as users will update addons using eksctl update addon instead.

[consideRatio]

In addition to your PR, we plan on improving this experience by always installing EKS managed addons by default when addons are not specified, instead of letting EKS install self-managed addons.

Wiee that sounds great, thank you for your work on eksctl @cPu1 and @TiberiuGC!!