feat: Adding Pod Identity / Access Entries option for Karpenter

[rodrigobersa]

What does this PR do?

Enables Pod Identity and Access Entry feature for Karpenter addon.

• Adding variable to create Karpenter access entry.
• Changed the tests/complete
  • Use Pod Identity provided by the https://github.com/aws-ia/terraform-aws-eks-blueprints-addon module.
  • Adding Pod Identity Addon
  • Adding tags for Karpenter discovery
• Updating Karpenter namespace to kube-system, as recommended since v0.33

TO DO: Change the "aws-ia/eks-blueprints-addon/aws" to the version supporting Pod Identity, pre-commit checks will not pass without that.

Motivation

Pod Identity and Access Entries aims to be the default options for granting permissions to addons. We should adopt this in the v2.

More

• Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
• Yes, I ran pre-commit run -a with this PR

For Moderators

• E2E Test successfully complete before merge?

Additional Notes

Apply complete! Resources: 125 added, 0 changed, 0 destroyed.

Outputs:

configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name complete"
$ terraform output -raw configure_kubectl | bash
Updated context arn:aws:eks:us-west-2:123456789012:cluster/complete in .kube/config
$ kubectl get pods -A
NAMESPACE                       NAME                                            READY   STATUS    RESTARTS      AGE
aws-node-termination-handler    aws-node-termination-handler-77466dbf55-9l5hh   1/1     Running   0             15m
cert-manager                    cert-manager-6d988558d6-p2vkg                   1/1     Running   0             15m
cert-manager                    cert-manager-cainjector-6976895488-dz5kx        1/1     Running   0             15m
cert-manager                    cert-manager-webhook-fcf48cc54-f8nvv            1/1     Running   0             15m
kube-system                     aws-node-2295k                                  2/2     Running   0             14m
kube-system                     aws-node-6grmt                                  2/2     Running   0             14m
kube-system                     aws-node-7bg5t                                  2/2     Running   0             14m
kube-system                     aws-node-cnswh                                  2/2     Running   0             14m
kube-system                     aws-node-gqjn9                                  2/2     Running   0             14m
kube-system                     aws-node-j467q                                  2/2     Running   0             14m
kube-system                     coredns-848555ff5-5s8mn                         1/1     Running   0             14m
kube-system                     coredns-848555ff5-8x6m8                         1/1     Running   0             14m
kube-system                     ebs-csi-controller-8489858766-b2s9h             6/6     Running   0             14m
kube-system                     ebs-csi-controller-8489858766-txtcf             6/6     Running   0             14m
kube-system                     ebs-csi-node-2kzsp                              3/3     Running   0             14m
kube-system                     ebs-csi-node-45q6k                              3/3     Running   0             14m
kube-system                     ebs-csi-node-4p9ng                              3/3     Running   0             14m
kube-system                     ebs-csi-node-jpcbx                              3/3     Running   0             14m
kube-system                     ebs-csi-node-kslkm                              3/3     Running   0             14m
kube-system                     ebs-csi-node-v5vpn                              3/3     Running   0             14m
kube-system                     eks-pod-identity-agent-6b9st                    1/1     Running   0             14m
kube-system                     eks-pod-identity-agent-725k6                    1/1     Running   0             14m
kube-system                     eks-pod-identity-agent-7lbvf                    1/1     Running   0             14m
kube-system                     eks-pod-identity-agent-g562k                    1/1     Running   0             14m
kube-system                     eks-pod-identity-agent-pq59j                    1/1     Running   0             14m
kube-system                     eks-pod-identity-agent-pw54s                    1/1     Running   0             14m
kube-system                     karpenter-545f7bd6cd-b2rm5                      1/1     Running   0             15m
kube-system                     karpenter-545f7bd6cd-hrckk                      1/1     Running   1 (14m ago)   15m
kube-system                     kube-proxy-2dzjw                                1/1     Running   0             14m
kube-system                     kube-proxy-47lqc                                1/1     Running   0             14m
kube-system                     kube-proxy-kwxlk                                1/1     Running   0             14m
kube-system                     kube-proxy-lf7ms                                1/1     Running   0             14m
kube-system                     kube-proxy-tctgd                                1/1     Running   0             14m
kube-system                     kube-proxy-vwgkl                                1/1     Running   0             14m
kube-system                     metrics-server-5dc9dbbd5b-x5r8m                 1/1     Running   0             15m

$ kubectl apply -f example.yaml
ec2nodeclass.karpenter.k8s.aws/default created
nodepool.karpenter.sh/default created
deployment.apps/inflate created

$ kubectl scale deploy/inflate --replicas 100

$ terraform destroy -auto-approve
Destroy complete! Resources: 125 destroyed.

$ pre-commit run --files tests/complete/*
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
check for merge conflicts................................................Passed
detect private key.......................................................Passed
detect aws credentials...................................................Passed
Terraform fmt............................................................Passed
Terraform docs...........................................................Passed
Terraform validate with tflint...........................................Passed
Terraform validate.......................................................Passed
$ pre-commit run --files *               
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
check for merge conflicts................................................Passed
detect private key.......................................................Passed
detect aws credentials...................................................Passed
Terraform fmt............................................................Passed
Terraform docs...........................................................Passed
Terraform validate with tflint...........................................Passed
Terraform validate.......................................................Passed
```

[vvchik]

Will it work if karpenter_enable_spot_termination = false?
Because sqs were not created in this case.