Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Proposal] How to implement image policy for sealer using Kyverno #1899

Open
4 tasks
YTGhost opened this issue Nov 30, 2022 · 10 comments · May be fixed by #1982
Open
4 tasks

[Proposal] How to implement image policy for sealer using Kyverno #1899

YTGhost opened this issue Nov 30, 2022 · 10 comments · May be fixed by #1982
Labels
kind/feature Category issues or PRs related to feature request

Comments

@YTGhost
Copy link
Member

YTGhost commented Nov 30, 2022

Issue Description

Background

The Proposal(Using Kyverno to support container runtime neutrally) was discussed at an community meeting on November 16, 2022. This proposal introduces Kubernetes Pod Policy and compares Pod Security Admission, Gatekeeper, and Kyverno, and finally determines that Kyverno can be used to implement image policy.

After the meeting, we also verified whether Kyverno supports k0s/k3s, it works well.

Implementation details

Kyverno Architecture

Kyverno dynamically adds Webhook by creating Policy, which can help us implement Image policy. So in fact, in terms of implementation details, we need to pay attention to two aspects:

  1. Policy
  2. Kyverno

About Policy

Format of policy configuration file

image

The picture above shows Kyverno’s Policy rules, we can use Namespaces, Resource Kinds and Resource Names to match the corresponding Pod.

We may need to build a user-friendly Policy configuration file, and the sealer can create a corresponding Policy based on this configuration file. And maybe through this configuration file, we can do the function of getting the mirror list together. @starnop

{
  "application-list":[
    {
      "resourceKind": "deployment",
      "resourceName": "nginx-deployment",
      "namespace": "default",
      "image": "nginx:1.14.2"
    },
    ...
  ]
}

How to get the information needed to create a policy

During the build phase, sealer will cache all container images required by the cluster in the local private registry.

We can extract the application list that needs to be matched by the image policy along the way during the build process.

Where to store policy configuration file

There are currently two options, @kakaZhou719:

  1. basefs
  2. imagespec

When is the Policy created?

Since the container at the K8s layer does not require webhook to intercept, it is sufficient to create a Policy before the container at the application layer is launched.

About Kyverno

Whether to install Kyverno

In a proprietary cloud delivery scenario, the cluster network is isolated from the outside world, so we need to cache container images to a local private repository during the build phase and pull images from the private repository uniformly when the cluster starts.

But in some case, users want to not save the container image during the build process. For example, when the cluster network is not isolated from the outside world.

So we could perhaps set up two different build modes (online and offline) during the build phase and install Kyverno in offline mode only.

When is the Kyverno installed

Based on the role Kyverno needs to play, an obvious choice would be to make it part of the base and activate it along with the base.

Type: feature request

Describe what feature you want

Additional context

taskList:

  • Need to implement build mode to control whether container images are saved in the Sealer Image.
  • Implementing the logic for parsing and saving the image list
  • API definition, including determining whether a Sealer Image contains container images and getting the image list in the Sealer Image.
  • Implementation of the management of the kyverno image policy according to the image list.
@YTGhost YTGhost added the kind/feature Category issues or PRs related to feature request label Nov 30, 2022
@starnop
Copy link
Collaborator

starnop commented Dec 6, 2022

: Sorry for the late reply.

nice job 👍

And there are some suggestions for your reference:

  1. Cloud you please provide a diagram with a description to explain how to use kyvernno in the lifecycle of sealer, such as sealer build --type=app-installer will package the kyvernno, sealer run {{appImage}} will install the kyvernno if not exist, etc? It would help if you tried to answer these questions:

    1.0 the side effects of the sealer that the kyvernno will cause, such as how much the app's package size will increase
    1.1 how to store and read the image list, @kakaZhou719
    1.2when to package kyvernno and where the kyvernno will locate at
    1.3 when and how to install/uninstall kyvernno
    1.4 when to generate(add/update/delete) Kyverno’s Policy rules and where to store the policy rules
    1.5 when and how to policy the image
    1.6 ......

  2. Maybe an image list is enough for generate Kyverno’s Policy rules, we can policy all images in the list to sealer registry domain in the offline mode. So the definition of a image list can be straightforward just like the following:

[
  {
    "image": "nginx:1.14.2"
  },
  {
    "image": "nginx:1.14.2"
    "appName": "app1",
  },
  {
    "image": "image:tag",
    "appName": "app1",
    "resourceKind": "deployment",
    "resourceName": "nginx-deployment",
    "namespace": "default"
  }
]

@starnop
Copy link
Collaborator

starnop commented Dec 12, 2022

@YTGhost Any update?

@YTGhost
Copy link
Member Author

YTGhost commented Dec 13, 2022

  1. These two diagram to explain how to use kyverno in the lifecycle of sealer:

image

image

  1. Based on the previous description of Kyverno's Policy rules, I think that resourceKind, resourceName and namespace are the necessary information for kyverno to uniquely identify the resources to which the image policy needs to be applied. So the definition of a image list will just like this:
[
  {
    "image": "image:tag",
    "appName": "app1",
    "resourceKind": "deployment",
    "resourceName": "nginx-deployment",
    "namespace": "default"
  },
  ...
]

@starnop
Copy link
Collaborator

starnop commented Dec 14, 2022

I think that resourceKind, resourceName and namespace are the necessary information for kyverno to uniquely identify the resources to which the image policy needs to be applied.

In fact, you only need to mutate the image for the pod resources

@YTGhost
Copy link
Member Author

YTGhost commented Dec 28, 2022

I think that resourceKind, resourceName and namespace are the necessary information for kyverno to uniquely identify the resources to which the image policy needs to be applied.

In fact, you only need to mutate the image for the pod resources

Yes, after some research, kyverno supports the use of regex in policies to replace them, as shown in this example: https://kyverno.io/policies/other/replace_image_registry/replace_image_registry/. So in the image list, we can replace the image according to its name

@starnop
Copy link
Collaborator

starnop commented Jan 5, 2023

well done, and then cloud you please submit a to-do list so we progress to implementation in an orderly manner?

@starnop
Copy link
Collaborator

starnop commented Jan 5, 2023

attention: We don't have the command "run app" anymore, and it is unified into the "run" command

@YTGhost
Copy link
Member Author

YTGhost commented Jan 5, 2023

I think the implementation of this feature can be split into the following subtasks, @starnop WDYT:

  • sealer build support online and offline build modes. When using online build mode, sealer will not save the container image during the build process, and at this point kyverno does not need to be installed.
  • Get the image list during the parse phase, and save it into basefs.
  • During the build and package phase, kyverno is packaged as part of the base or another build-in app, which depends on the --type of sealer build.(Note: If the build mode is online, there is no need to package kyverno)
  • sealer run support launch kyverno and create image policy(Note: If you are running ClusterImage built in online mode, you do not need to start Kyverno. If you are running an application image, you also need to pay attention to whether kyverno already exists in the environment)

@starnop
Copy link
Collaborator

starnop commented Jan 6, 2023

great job! And I have a few minor suggestions.

  1. In fact, especially when collaboration is required, the API of the partnership should be the first priority, so that each collaborator can work on the API in parallel. For the current problem, I think one important API should be the storage and retrieval of the image list of Sealer Image. That way, some people can just focus on that how to package kyverno and run kyverno and manage the kyverno image policy according to the image list, and others just need to focus on that make sure that API can get the correct image list of Sealer Image.
  2. I am thinking of whether we need to deal with the issue of whether kyverno needs to be installed or not with build mode. This can be decided by the user who defines the app image. We only need to provide an app or a base Sealer Image and the corresponding guidance documents. After that, we just need to implement how to manage the kyverno image policy according to the image list.

BTW, the API related to the image list can be left to me to deal with.

@starnop
Copy link
Collaborator

starnop commented Jan 6, 2023

Supplement:

Whether to save the container images through the build mode should be necessary. And the kyverno image policy management also depends on the API that whether the Sealer Image contains the containers images.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Category issues or PRs related to feature request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants