AWS EKS with Calico - Telepresence interception (webhook) not working

[peter-galcik]

Description:
We struggled to find root cause why telepresence inteception for pods didnt work. After thorough investigation and support from your chat channel we were able to find rout cause. The root cause is essentialy same as nicely explained in the blog post here:
https://medium.com/@denisstortisilva/kubernetes-eks-calico-and-custom-admission-webhooks-a2956b49bd0d

In short, as there is known limitation in Calico for EKS where EKS control plane(kubeserver api) is not in same virtual network with kubernetes nodes, webmutating hooksagentInjectorWebhook.yaml (mutatingwebhookconfiguration.admissionregistration.k8s.io/agent-injector-webhook-ambassador) is not reaching kubernetes ambassador service: - agent-injector because service cant be resolved

As this is limiting factor of Calico, it cant be considered as bug in telepresence, but has functional impact, and perhaps better documentation/call diagrams could be available in certain scenarios to enable better troubleshooting in case of unexpected issues.

Solution proposal:
(disclaimer) I am not expert on kubernetes, so iI might not provide best/ideal solutions. I will describe option we have used successfully.

We exposed agent-injector service on nodePort (it means all nodes route thins nodePort which is visible to EKS to dedicated service) afterwards kubernetes-server-api in EKS was able to reach service, but we had to alter webmutating hook as there is https with certificate included to accommodate this as well + changing spec to call url instead of kubernetes service.

I would like to get possibility at least to customize telepresence helm install --values values.yaml in a way that I am able to expose webhook servcie on nodePort

spec:
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: https
      **nodePort: 32666**

And capability of altering webhook configuration (related to agentInjectorWebhook.yaml):

• certificate contains user defined hostname (as its different than defaults):
  {{- $altNames := <CUSTOM_HOSTNAMES> ( printf "agent-injector.%s" (include "traffic-manager.namespace" .)) ( printf "agent-injector.%s.svc" (include "traffic-manager.namespace" .)) -}}
• option to use url instead of k8s service:

    service:
      name: {{ .Values.agentInjector.name }}
      namespace: {{ include "traffic-manager.namespace" . }}
      path: {{ .Values.agentInjector.webhook.servicePath }}
      port: {{ .Values.agentInjector.webhook.port }}

to something as this:

  clientConfig:
    caBundle: <bundle>
    url: "https://i<NODE_DNS_NAME>:32666/traffic-agent"

I believe there might be more robust approaach, and I am showing only option we choose. In general, telepresence helm chart lack capability to have externalized (outside) cluster mutating wehbook, which i am not sure is possible to have (as webhook miht need intra cluster information to mutate kubernetes objects).

Workaround:
Manual injection of traffic-agent as per documentation:
https://www.getambassador.io/docs/telepresence/latest/reference/intercepts/manual-agent

Security question:
With approach defined above we are exposing service directly outside cluster. As we run on AWS, there is different layer of security, but it might be questionable in other deployment scenarios.

All respective changes were done on telepresence version 2.12.0