Do mutate policies require internet access?

[jseiser]

We are deployed into a a whitelist only environment, so we can not reach things like docker.io etc. We are wanting to rewrite the registries to point to our Harbor instance, which can reach out to those registries. In testing, one of the problem I noticed is that, you fail to rewrite when its not against a public registry.

❯ kubectl apply -f pods.yaml
Warning: policy prepend-registry.prepend-registry-containers: failed to mutate elements: failed to evaluate mutate.foreach[4].preconditions: failed to substitute variables in condition key: failed to resolve imageData.registry at path : failed to fetch image descriptor: harbor.tld/registry.k8s.io/ingress-nginx/controller:v1.9.5, error: failed to fetch image descriptor: harbor.tld/registry.k8s.io/ingress-nginx/controller:v1.9.5, error: failed to fetch image reference: harbor.tld/registry.k8s.io/ingress-nginx/controller:v1.9.5, error: GET https://harbor.tld/v2/registry.k8s.io/ingress-nginx/controller/manifests/v1.9.5: UNAUTHORIZED: unauthorized to access repository: registry.k8s.io/ingress-nginx/controller, action: pull: unauthorized to access repository: registry.k8s.io/ingress-nginx/controller, action: pull
pod/myapp-pod created

[chipzoller]

Only when using the pre-defined image or images variables is registry access required since these populate the actual details for context calls. If you don't have that ability, you only have the ability to use information in the field for the container image. Regex may be necessary.

[jseiser]

That brings us back to there not being any working example of an image registry rewrite for harbor. Thanks I'll see if we can utilize another tool in the mean time

Thanks.

[chipzoller]

You've got several discussions and Slack threads open on this same use case.

Let's take it from the top and I'll see what I can come up with. What I need from you is:

1. A full explanation of what you want to achieve, not how you think it should be achieved and not by showing policy samples. List any and all design or environmental constraints.

2. Examples of multiple input resources.

3. Examples of corresponding output resources.

[JimBugwadia]

A clarification: the builtin image or images do not require external registry access - they are built from the admission request data.

The imageData variable in this policy will require fetching data from the registry:

kyverno/policies#882 (comment)

@jseiser - your output shows failed to resolve imageData.registry. Do you have a context entry in your policy?

[chipzoller]

Thanks, yes I was confused after jumping between issues. imageData context variables require live registry access.

[jseiser]

@chipzoller

Basic rundown of the situation. 1 EKS cluster, in a non public AWS Environment, that has outbound/internet access on a whitelist only policy. Even DNS lookups are blocked. Harbor is accessible, and has private Image proxies configured for every common public registry. The AWS VPC that the cluster sits in, has VPC Endpoints created for AWS ECR Access, so the nodes are able to pull AWS images hosted in their private ECR registries via the node's IAM role, so we do not need/want to over write them. So kyverno can not query docker hub for instance, and then rewrite the registry, because the query to docker hub would be blocked. Kyverno can also not query the harbor proxy(s) unless its able to provide an image pull secret while doing so.

Example Inputs

nginx:1.2.3
nginx/nginx:1.2.3
docker.io/nginx/nginx:1.2.3
quay.io/nginx:1.2.3
ecr.public/nginx:1.2.3
registry.k8s.io/nginx:1.2.3
cr.fluentbit.io/nginx:1.2.3
cr.fluentbit.io/nginx:1.2.3
053166900151.dkr.ecr.us-gov-west-1.amazonaws.com/nginx:1.2.3
gcr.io/nginx:1.2.3
ghcr.io/nginx:1.2.3
harbor.tld/docker.io/nging:v1.2.3

Example Outputs

harbor.told/docker.io/nginx:1.2.3
harbor.told/docker.io/nginx/nginx:1.2.3
harbor.told/docker.io/nginx/nginx:1.2.3
harbor.told/quay.io/nginx:1.2.3
harbor.told/ecr.public/nginx:1.2.3
harbor.told/registry.k8s.io/nginx:1.2.3
harbor.told/cr.fluentbit.io/nginx:1.2.3
053166900151.dkr.ecr.us-gov-west-1.amazonaws.com/nginx:1.2.3
harbor.told/gcr.io/nginx:1.2.3
harbor.told/ghcr.io/nginx:1.2.3
harbor.tld/docker.io/nging:v1.2.3

Thanks.

[chipzoller]

Looks like there's a problem within the internals of Kyverno when parsing out the image string used for building the internal images context variable in that it doesn't like what looks like a domain to be an inner component of the image string. Otherwise, this policy would do the job:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: image-normalize-demo
spec:
  rules:
    - name: replace-image-registry-pod-containers
      match:
        any:
        - resources:
            kinds:
            - Pod
            operations:
            - CREATE
      mutate:
        foreach:
        - list: "request.object.spec.containers"
          preconditions:
            any:
            - key: "{{ element.image }}"
              operator: AnyNotIn
              value: 
                - harbor.tld/*
                - harbor.told/*
                - 053166900151.dkr.ecr.us-gov-west-1.amazonaws.com/*
          patchStrategicMerge:
            spec:
              containers:
              - name: "{{ element.name }}"
                image: harbor.told/{{ image_normalize('{{element.image}}') }}"

But because of the issue, it'll fail with something like

Error from server: error when creating "temp1.yaml": admission webhook "mutate.kyverno.svc-fail" denied the request: failed to add image information to the policy rule context: invalid image 'harbor.told/docker.io/nginx:1.2.3"' (bad image: harbor.told/docker.io/nginx:1.2.3", defaultRegistry: docker.io, enableDefaultRegistryMutation: true: invalid reference format)

See Playground link here.

I'll have to log an issue for this, but for the time being looks like you may have to resort to using another tool like Indeed's harbor-container-webhook.

cc @JimBugwadia

[chipzoller]

#9381

[chipzoller]

Please mark as answered if resolved.