502 Bad Gateway and unhealthy probe in AGIC AppGW

[arunkumarguggila]

My application is with Angular FE and multiple Asp.net core web api's. I have deployed this in AKS and configured Appgw AGIC. My Ingress controller details

Name: ingress-appgw-deployment-xxxxxxxx-xxxx
Namespace: kube-system
Priority: 0
Service Account: ingress-appgw-sa
Node: aks-agentpool-xxxxxxx-vmss000000/10.224.1.255
Start Time: Fri, 01 Mar 2024 02:16:47 +0400
Labels: app=ingress-appgw
kubernetes.azure.com/managedby=aks
pod-template-hash=6d4f68fc9f
Annotations: checksum/config: 1e53b6ba909ce787812767da80976912cc41007f8a716c489edab91f215b2bd7
cluster-autoscaler.kubernetes.io/safe-to-evict: true
kubernetes.azure.com/metrics-scrape: true
prometheus.io/path: /metrics
prometheus.io/port: 8123
prometheus.io/scrape: true
resource-id:
/subscriptions/ea88e1ed-fec0-472f-87f5-a00e6a608f19/resourceGroups/AZ-BELSIO-SOFTWARES-RG/providers/Microsoft.ContainerService/managedClus...
Status: Running
IP: 10.224.x.xx
IPs:
IP: 10.224.x.xx
Controlled By: ReplicaSet/ingress-appgw-deployment-6d4f68fc9f
Containers:
ingress-appgw-container:
Container ID: containerd://de72b0a2d86ffd7e11113b0ee38a374b509b212d56eac166391d28d4ae81849c
Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2
Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:eeb1d42ebfb872478d9b0b16f6936ea938d6e5eed4a59cde332b8757556a5e1f
Port:
Host Port:
State: Running
Started: Fri, 01 Mar 2024 02:18:47 +0400
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 01 Mar 2024 02:17:55 +0400
Finished: Fri, 01 Mar 2024 02:17:56 +0400
Ready: True
Restart Count: 4
Limits:
cpu: 700m
memory: 600Mi
Requests:
cpu: 100m
memory: 20Mi
Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
ingress-appgw-cm ConfigMap Optional: false
Environment:
AGIC_POD_NAMESPACE: kube-system (v1:metadata.namespace)
KUBERNETES_PORT_443_TCP_ADDR: belsio-dns-yvu9cnmb.hcp.uaenorth.azmk8s.io
KUBERNETES_PORT: tcp://myapp-dns-yvu9cnmb.hcp.uaenorth.azmk8s.io:443
KUBERNETES_PORT_443_TCP: tcp://myapp-dns-yvu9cnmb.hcp.uaenorth.azmk8s.io:443
KUBERNETES_SERVICE_HOST: myapp-dns-yvu9cnmb.hcp.uaenorth.azmk8s.io
AZURE_CLOUD_PROVIDER_LOCATION: /etc/kubernetes/azure.json
AGIC_POD_NAME: ingress-appgw-deployment-6d4f68fc9f-xtlmk (v1:metadata.name)
Mounts:
/etc/kubernetes/azure.json from cloud-provider-config (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9kcms (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
cloud-provider-config:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/azure.json
HostPathType: File
kube-api-access-9kcms:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: CriticalAddonsOnly op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
My ingress congiguration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp
namespace: default
annotations:
kubernetes.io/ingress.class: azure/application-gateway
spec:
tls:

• hosts:
  • app.example.com
    rules:
• host: app.example.com
  http:
  paths:
  - path: /
  pathType: Exact
  backend:
  service:
  name: angular
  port:
  number: 80
  - path: /api1
  pathType: Prefix
  backend:
  service:
  name: api1
  port:
  number: 80
  - path: /api2
  pathType: Prefix
  backend:
  service:
  name: api2
  port:
  number: 80
  Expected: domain root path will route to Angular App and /api1 and /api2 to api's. I was able to achieve this in minikube using nginx IC using the same ingress configuration.

Issue: Angular is accessible but cannot access API's. I checked APIs by assigning a public and it's accessible but cannot with the ingress. In the health check, Angular pods are healthy but api pods are unhealth with the below message.

"Received invalid status code: 404 in the backend server’s HTTP response. As per the health probe configuration, 200-399 is the acceptable status code. Either modify probe configuration or resolve backend issues. To learn more visit - https://aka.ms/StatusCodeMismatch."

Please help me fix this and also suggest additional configurations that you think might help me.

Thank you in advance.

[ntsh999]

were you able to find any fix for this issue? I am also facing this issue?

[arunkumarguggila]

Yes, for me the issue was with the root prefix in the swagger config. once that's set everything worked perfectly.