External load balancer via targetGroupARN does not modify the security group to allow traffic from load balancer

[prakashbalaji]

Describe the bug
The pods are getting registered to the targetGroupARN specified correctly but the. security groups in the nodes are not adjusted to allow the traffic from the load balancer. Even specifically passing security group with ingress annotation is not helping.

Is this an bug here? if this is not a bug what is the correct way to adjust the node security group to allow traffic from load balancer for external load balancer.

Steps to reproduce

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-app
  labels:
    app: nginx-app
spec:
  selector:
    matchLabels:
      app: nginx-app
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-app
    spec:
      containers:
      - name: nginx-container
        image: nginx
        ports:
            - containerPort: 80        
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx-app
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: /index.html    
spec:
  type: ClusterIP
  selector:
    app: nginx-app
  ports:
    - port: 80
      targetPort: 80
---

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: nginx-manual-tg
spec:
  serviceRef:
    name: nginx-service
    port: 80
  targetGroupARN: arn:aws:elasticloadbalancing:us-west-2:<AWS_ACCOUNT_ID>:targetgroup/nginx-manual-tg/<HASH>

Expected outcome
Should security group of node be adjusted to allow traffic from load balancer for external load balancer?

Environment

AWS Load Balancer controller version : v2.7.2
Kubernetes version v1.29.3-eks-adc7111
Using EKS (yes/no), if so version? yes - v1.29.3-eks-adc7111

Additional Context:

For cases when the load balancer is created by the controller, the node security groups are adjusted correctly to allow traffic from load balancer.

[oliviassss]

@prakashbalaji hey, was the load balancer created successfully? were you trying to create a NLB or ALB?

[prakashbalaji]

@oliviassss - This is the case of we creating load balancer externally outside of the controller. We used terraform to provision the load balancer and we are using ALB here. The issue is that the pods are getting registered as targets to the load balancer but the EKS node security group rules are not amended to allow traffic from the load balancer.

When the controller creates the load balancer it augments the security group rules.

[oliviassss]

@prakashbalaji, sorry I missed that.
I doubt if it's an external LB, the controller will manage the AWS resources for it (since we rely on users to manage TG, listener, LB for external LB). Personally I haven't tested this before, I can try to repro, but check out the live doc here: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.8/guide/use_cases/self_managed_lb/

[prakashbalaji]

Thanks @oliviassss for the response, I can for sure tell you that the security group rules are not amended for the self managed lb. As I think more, it seems right for the controller not to manage the security group as we create the load balancer and security group outside and we are supposed to allow traffic from load balancer to EKS node, I think this issue can be closed as we are taking that approach and also it would be good to document it in the notes about security groups. The linked article does not talk about security groups and hence the confusion. Thanks for your inputs btw.

[oliviassss]

Thanks for the confirmation.
/kind documentation

[M00nF1sh]

@prakashbalaji
It's actually possible for a TargetGroupBinding to modify the worker node security groups.
check out the spec.networking on TargetGroupBinding. It supports to allow reach target on specific ports from specified cidr or security group(preferred to use the LB's security Group)