Workload Identity: Failed to acquire a token

[lukaseisenring]

Describe the bug
The token of Workload Identity cannot get acquired. Logs are reporting an incomplete environment variable configuration and in documentation no more possible values are specified.

The configuration is identical to the Testing section in the PR #1498 of @akshaysngupta

To Reproduce
Steps to reproduce the behavior:

Install AGIC using Helm (Version 1.7.1) with the following parameters:

appgw.applicationGatewayID: x
armAuth.type: workloadIdentity
armAuth.identityClientID: x
kubernetes.securityContext.runAsUser: 1000
rbac.enabled: true
verbosityLevel: 1

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.
  `Name: application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6
  Namespace: application-gateway-ingress-controller
  Priority: 0
  Service Account: application-gateway-kubernetes-ingress-sa-ingress-azure
  Node: x
  Start Time: Tue, 27 Jun 2023 07:43:46 +0200
  Labels: app=ingress-azure
  azure.workload.identity/use=true
  pod-template-hash=54d9cf6f47
  release=application-gateway-kubernetes-ingress
  Annotations: checksum/config: fe8217e775b14e18c53cd5d54b2bc7719deda1da25a2f18345e9abaf101d1181
  prometheus.io/port: 8123
  prometheus.io/scrape: true
  Status: Running
  IP: x
  IPs:
  IP: x
  Controlled By: ReplicaSet/application-gateway-kubernetes-ingress-ingress-azure-54d9cf6f47
  Containers:
  ingress-azure:
  Container ID: containerd://6bde7e11a42e5c995fabd9f55ba9843ce7a2c51fdae45d5fa5e36ef8eacdeff2
  Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1
  Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:91a6648b78c65f3b6858441589daabd72146d9a53e896c0e6abf501e870f9d9b
  Port:
  Host Port:
  State: Running
  Started: Tue, 27 Jun 2023 07:43:47 +0200
  Ready: False
  Restart Count: 0
  Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
  Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
  Environment Variables from:
  application-gateway-kubernetes-ingress-cm-ingress-azure ConfigMap Optional: false
  Environment:
  AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json
  AGIC_POD_NAME: application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 (v1:metadata.name)
  AGIC_POD_NAMESPACE: application-gateway-ingress-controller (v1:metadata.namespace)
  AZURE_CLIENT_ID: x
  AZURE_TENANT_ID: x
  AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token
  AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/
  Mounts:
  /etc/appgw/ from azure (ro)
  /var/run/secrets/azure/tokens from azure-identity-token (ro)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fbjhm (ro)
  Conditions:
  Type Status
  Initialized True
  Ready False
  ContainersReady False
  PodScheduled True
  Volumes:
  azure:
  Type: HostPath (bare host directory volume)
  Path: /etc/kubernetes/
  HostPathType: Directory
  kube-api-access-fbjhm:
  Type: Projected (a volume that contains injected data from multiple sources)
  TokenExpirationSeconds: 3607
  ConfigMapName: kube-root-ca.crt
  ConfigMapOptional:
  DownwardAPI: true
  azure-identity-token:
  Type: Projected (a volume that contains injected data from multiple sources)
  TokenExpirationSeconds: 3600
  QoS Class: BestEffort
  Node-Selectors: x
  Tolerations: :NoSchedule op=Exists
  node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
  node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
  Type Reason Age From Message

  ---

  Normal Scheduled 56m default-scheduler Successfully assigned application-gateway-ingress-controller/application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 to x
  Normal Pulling 56m kubelet Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1"
  Normal Pulled 56m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1" in 261.149933ms (261.155633ms including waiting)
  Normal Created 56m kubelet Created container ingress-azure
  Normal Started 56m kubelet Started container ingress-azure
  Warning Unhealthy 67s (x374 over 55m) kubelet Readiness probe failed: Get "http://x:8123/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  `

• Output of kubectl logs <ingress controller>. I0627 05:43:47.609870 1 utils.go:114] Using verbosity level 1 from environment variable APPGW_VERBOSITY_LEVEL
  I0627 05:43:47.616327 1 main.go:81] Unable to load cloud provider config '/etc/appgw/azure.json'. Error: Reading Az Context file "/etc/appgw/azure.json" failed: open /etc/appgw/azure.json: permission denied
  I0627 05:43:48.648152 1 supported_apiversion.go:70] server version is: 1.25.6
  I0627 05:43:48.659580 1 environment.go:294] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
  I0627 05:43:48.659607 1 main.go:118] Using User Agent Suffix='application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6' when communicating with ARM
  I0627 05:43:48.659688 1 auth.go:58] Creating authorizer using Default Azure Credentials
  I0627 05:43:48.660007 1 httpserver.go:57] Starting API Server on :8123
  E0627 05:44:48.660931 1 authorizer.go:46] Error getting Azure token: DefaultAzureCredential: failed to acquire a token.
  Attempted credentials:
  EnvironmentCredential: incomplete environment variable configuration. Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set
  WorkloadIdentityCredential: unable to resolve an endpoint: server response error:
  context deadline exceeded
  `

Azure Support Request ID: 2306270050000735