istio 1.19 nlb target groups are unhealthy

[chaitanya0619]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

I am trying to install second ingress gateway with istioctl (1.19.0) and below posted custom yaml file to get nlb on aws eks. target groups corresponding to ports 443 (target port 8443) and 80 (target port 8080) are showing unhealthy nodes only target group for 15021 shows healthy nodes.

Also dont see envoy listening on ports 8443 & 8080 on nlb pods. not sure why? can somebody explain what am i missing.

istio-proxy@istio-ingressgateway-nlb-5d64f78b8-vrts5:/$ netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:15000 0.0.0.0:* LISTEN 18/envoy
tcp 0 0 127.0.0.1:15004 0.0.0.0:* LISTEN 1/pilot-agent
tcp 0 0 0.0.0.0:15020 0.0.0.0:* LISTEN 1/pilot-agent
tcp 0 0 0.0.0.0:15021 0.0.0.0:* LISTEN 18/envoy
tcp 0 0 0.0.0.0:15021 0.0.0.0:* LISTEN 18/envoy
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN 18/envoy
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN 18/envoy

i checked the proxy containers for first ingressgateway (hosting aws elb)and envoy is listening on 8443 & 8080.

istio-proxy@istio-ingressgateway-757b6585bd-rw8jd:/$ netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:15020 0.0.0.0:* LISTEN 1/pilot-agent
tcp 0 0 0.0.0.0:15021 0.0.0.0:* LISTEN 16/envoy
tcp 0 0 0.0.0.0:15021 0.0.0.0:* LISTEN 16/envoy
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 16/envoy
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 16/envoy
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN 16/envoy
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN 16/envoy
tcp 0 0 127.0.0.1:15000 0.0.0.0:* LISTEN 16/envoy
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 16/envoy
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 16/envoy
tcp 0 0 127.0.0.1:15004 0.0.0.0:* LISTEN 1/pilot-agent

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:

it seems this is the only way to set autoInject policy as of right now

keep an eye out for IstioOperator API updates

values:
global:
autoscalingv2API: true
proxy:
autoInject: disabled
hub: "docker.io/istio"
meshConfig:
defaultConfig:
holdApplicationUntilProxyStarts: true
profile: "minimal"
addonComponents:
pilot:
enabled: true
components:
pilot:
k8s:
nodeSelector:
beta.kubernetes.io/os: linux
hpaSpec:
minReplicas: "2"
env:
- name: PILOT_DEBOUNCE_AFTER
value : 5s
ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
nodeSelector:
beta.kubernetes.io/os: linux
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS-1-2-2017-01"
hpaSpec:
maxReplicas: "5"
minReplicas: "1"
metrics:
- resource:
name: cpu
target:
type: Utilization
averageUtilization: 80
# targetAverageUtilization: 80
type: Resource
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-ingressgateway
resources:
limits:
cpu: "2000m"
memory: "1024Mi"
requests:
cpu: 100m
memory: 128Mi
service:
ports:
# needed first for non-tls deployments
# to be overriden later when TLS is enabled
- name: status-port
port: 15021
targetPort: 15021
- name: http2
port: 80
targetPort: 8080
- name: tls
port: 15443
targetPort: 15443
- name: cluster-local-gateway
enabled: true
label:
istio: cluster-local-gateway
app: cluster-local-gateway
k8s:
service:
type: ClusterIP
ports:
- port: 15020
name: status-port
- port: 80
name: http2
targetPort: 8080
- port: 443
name: https
targetPort: 8443
- name: istio-ingressgateway-nlb
enabled: true
k8s:
nodeSelector:
beta.kubernetes.io/os: linux
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "TCP"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
hpaSpec:
maxReplicas: "5"
minReplicas: "1"
metrics:
- resource:
name: cpu
target:
type: Utilization
averageUtilization: 80
type: Resource
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-ingressgateway-nlb
resources:
limits:
cpu: "2"
memory: 1Gi
requests:
cpu: 100m
memory: 512Mi
service:
ports:
- name: status-port
port: 15021
targetPort: 15021
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tcp
port: 31400
targetPort: 31400
- name: tls
port: 15443
targetPort: 15443
label:
app: istio-ingressgateway-nlb
istio: ingressgateway-nlb

Version

istioctl version  
client version: 1.19.0
control plane version: 1.19.0
data plane version: 1.19.0 (2 proxies)
kubectl version --short
Client Version: v1.23.6
Server Version: v1.27.8-eks-8cb36c9

Additional Information

No response

[abasitt]

You won't see any listener on port 80 and 443 in istio ingressgateway until you deploy Istio gateway resource. You should try to install gateway resource first matching with this new gateway label.

[gulshenjumayeva]

@chaitanya0619 I am having same issue. I have same setup with NLB, 443 and 80 unhealthy but 15021 healthy.
you stated that was able to get them healthy with having a gateway resource on port 443 or hit node port of 15021 port. Could you explain in details how you were able to get them healthy?

Thank you

[chaitanya0619]

create an istio gateway resource on port 443 & 80 for the health checks to pass.

[gulshenjumayeva]

I have created gateway and virtual service. still same issue with health checks

[gulshenjumayeva]

nvm it worked, thank you! had to wait a bit

[nareshlaligundla]

Run the gateway in privileged container so use below parameter in your values.yaml then health check will work.

runAsRoot: true