Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.96.44.217:443: connect: connection refused

[ALPHA-1503]

/kind bug

Hey everyone,

I'm currently in the process of installing Kubeflow on Tanzu Kubernetes Grid, but I've hit a roadblock with the knative-serving and knative-eventing deployments. The pods aren't initializing due to the following error:

"Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.96.44.217:443: connect: connection refused"

As I'm relatively new to Kubernetes technology, I apologize if I might not grasp all the intricacies of this issue.

• Kubeflow version: 1.8
• Kubernetes platform: Tanzu Kurbenetes VMware
• Kubernetes version: v1.26.4+vmware.wcp.0
• Ubuntu (for gpu compatibility)

[AndersBennedsgaard]

This would suggest an issue with Istio, and not Kubeflow, but let's break down the error log:

Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io" suggests that Kubernetes has failed calling the Istio sidecar injector service.

The next line explains that connection refused to the Kubernetes Service istiod in the istio-system namespace on port 443 because of a connection refused:
failed to call webhook: Post "[https://istiod.istio-system.svc:443/inject?timeout=10s](https://istiod.istio-system.svc/inject?timeout=10s)": dial tcp 10.96.44.217:443: connect: connection refused

Often times connection refused means that pods aren't ready to receive traffic at this time. So check if the Istio pods are up and ready.
If it continues failing, you might want to get help from https://github.com/istio/istio/issues