DNS Regression with removal of SERVICE_IPV4_CIDR for AL2 EKS Managed Nodes

[butlern]

Description

PR 2963 removes the SERVICE_IPV4_CIDR env var in the bootstrap user-data in preference for a --service-ipv4-cidr cli arg to the /etc/eks/bootstrap.sh.

However, /etc/eks/bootstrap.sh doesn't support a --service-ipv4-cidr option.

https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh#L136 only supports a --service-ipv6-cidr option.

It does still support that as an environment variable though, https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh#L213

If my VPC is in a 10.x.x.x cidr and I specify, for example, 10.100.0.0/16 as my service-ipv4-cidr AND I set enable_bootstrap_user_data=true in my eks_managed_node_group, it will default to 172.20.0.10 as the dns resolver here:

https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh#L474

Due to the fact that TEN_RANGE is equal to 1.

If your request is for a new feature, please use the Feature request template.

• ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following first:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]: 20.8.5

• Terraform version: 1.5.7

• Provider version(s):

• provider registry.terraform.io/hashicorp/aws v5.46.0
• provider registry.terraform.io/hashicorp/cloudinit v2.3.3
• provider registry.terraform.io/hashicorp/kubernetes v2.29.0
• provider registry.terraform.io/hashicorp/null v3.2.2
• provider registry.terraform.io/hashicorp/time v0.11.1
• provider registry.terraform.io/hashicorp/tls v4.0.5

Reproduction Code [Required]

I'm sorry, I don't have a ready made example. I hope the description is sufficient. If you absolutely need one, I can try to cobble something together.

Steps to reproduce the behavior:

1. Launch EKS cluster.
2. After EKS Managed Node Group has provisioned, you can see on one of the EKS Managed nodes that clusterDNS is set to 172.20.0.10

grep -A 1 clusterDNS /etc/kubernetes/kubelet/kubelet-config.json
  "clusterDNS": [
    "172.20.0.10"

Expected behavior

The clusterDNS value be set to 10.100.0.10 when cluster_service_ipv4_cidr is set to 10.100.0.0/16

Actual behavior

The clusterDNS value was defaulting to 172.20.0.10 when cluster_service_ipv4_cidr was set to 10.100.0.0/16

Terminal Output Screenshot(s)

Additional context

Workarounds available:

1. Don't set enable_bootstrap_user_data=true. This will cause the bootstrap script to lookup the SERVICE_IPV4_CIDR correctly by calling aws eks describe-cluster
2. Set pre_bootstrap_user_data with SERVICE_IPV4_CIDR set, e.g.

      pre_bootstrap_user_data = <<-EOT
        export SERVICE_IPV4_CIDR=10.100.0.0/16
      EOT

[wanddynosios]

We encountered the same issue and the workaround works for us.