Pre-KEP: Remove Keptn Upstream Repository

[thschue]

Motivation
When discussing the Keptn Architecture, one of the most frequent discussion points is the purpose of the upstream repository. Although it is the source of truth for deployment, it introduces some major drawbacks, especially when dealing with more deployment focussed use cases.

Example 1:
When using keptn for deployment, keptn is acting as an artifact store. Therefore, you have to publish artifacts and configurations to keptn before you can use them. In the case of a Helm chart, the chart is published in an uncommon way (keptn send artifact instead of helm push).

Example 2:
When designing the GitOps approaches, a lot of effort has been put into dealing with the keptn configuration repo, manipulating git and getting the correct configuration versions for a service deployed. Many of the problems were related to git, also some performance problems were caused by git.

Assumption
If the configuration, deployment artifact (e.g. helm chart), and the container images are stored in an artifact registry, we could get rid of the upstream repository.

Approach
In all of the current approaches, the configuration of keptn (sli.yaml, helm-charts, test configuration) is stored in a code/configuration repository maintained by developers. Therefore, it would be possible to generate an artifact out of them in the ci-process (e.g. https://oras.land/) and push them to an OCI-compliant registry. Since Helm 3.8.0, such registries are also supported for Helm Charts (https://github.com/helm/helm/releases/tag/v3.8.0) and Container images in any way. Therefore, all of the information can be stored in an OCI registry in the CI process and consumed by Keptn during the workflow. Using an OCI registry, would allow us to use common signing mechanisms for all artifacts (https://github.com/sigstore/cosign#oci-artifacts) and validate them during deployment.

Keptn specific configuration (Shipyard, Stages, Sequences, ...) could be stored in Custom Resource Definitions.

Prerequisites

• All artifacts have to be pushed before the workflow starts

Using the artifacts
When a workflow runs, the artifacts for a specific version (tagged) are fetched, extracted, and stored for deployment (git, s3, filesystem, whatever).

Benefits

• The upstream repo is not needed anymore
• decreased manipulation effort
• Unified storage for all artifacts
• Possibly more control where to store the artifacts (e.g. we could also use artifactory, s3 instead of oci)
• Portability

These are just my first ideas regarding this topic. What do you think about this?

[grabnerandi]

Hi Thomas. In a recent discussion we had I thought we came to the conclusion that the "configuration rep" (upstream or not) - is a great architectural benefit as all keptn-services only need access to that configuration repo via the Keptn API vs needing access to the source code repo where devs put their config files
Maybe you still have this in mind with this KEP - but - could be that I miss this. My point is -> I agree with most of what you said - BUT - I think we need to make sure that all artifacts available to keptn services should be managed by a Keptn Core Service such as the Configuration/Resource Service

[christian-kreuzberger-dtx]

we need to make sure that all artifacts available to keptn services should be managed by a Keptn Core Service such as the Configuration/Resource Service

Could you describe what artifacts you mean? As in Docker Images, Helm Charts, Values.yaml, etc...? Or SLO/SLI/JMeter files?

[grabnerandi]

Sure: Artifacts are all the necessary files that are needed by the keptn-services, e.g: test files, slos, monitoring configuration ...

[christian-kreuzberger-dtx]

Hi Thomas and thanks for opening the discussion.

I was one of the persons that brought up this topic after seeing this in KEP-67

Here we suggest that there will be three potentially different Git Repos involved.

In my opinion, the Config Repo, Code Repo and Keptn Upstream Repo should really be a single repo, e.g.:

.keptn folder:

For the end-user, the only thing to do is to create a project, and tell Keptn where the config files are, e.g.:

keptn create project best-project-ever --git-repo=https://github.com/christian-kreuzberger-dtx/my-jenkins-test.git --git-auth-credentials=... \
  --branch=main --keptn-config=.keptn [--slo-file=slo.yaml --remediation-file=remediation.yaml]

However, we never create a service, a stage, or add any files. This is something that the Git Repo holds as a truth (e.g., there could be a services.yaml, or it's based on directory names).

edit:
Please note, with my suggestion it is still possible to separate code from .keptn config.

[eugenbelsky]

Hi Andreas,

I`d like to add my feedback regarding what @thschue already mentioned, based on my current experience with keptn deployments.

1. To actually make use of the GitOps approach, the way the upstream repo structure looks currently makes it extremely inconvenient for an end-user to use, even if it`s convenient for keptn.
2. Regarding artifact storage - currently (even if helm charts have to be stored in the repo itself), the user would still need to ensure that the other artifacts, referenced by the chart itself ( docker images for instance ) are reachable from the keptn executors (helm-service, job-executor, etc). If we already have to do it, it`s not completely clear to me what advantage does storing the artifacts in repo brings here.
3. There are already different workarounds in place (like kia-cli), which would allow us to make the git configuration of kept deployments purely descriptive as config files to actually make sense of what we are trying to deploy. This results in having a keptn repo purely as a buffer db between the actual config and the keptn itself.

[christian-kreuzberger-dtx]

Regarding artifact storage - currently (even if helm charts have to be stored in the repo itself), the user would still need to ensure that the other artifacts, referenced by the chart itself ( docker images for instance ) are reachable from the keptn executors (helm-service, job-executor, etc). If we already have to do it, it`s not completely clear to me what advantage does storing the artifacts in repo brings here.

Unfortunately helm-service and jmeter-service as well as the Keptn tutorials imply that certain artifacts like helm charts and jmeter files must be stored in the Keptn Upstream repo and that this is a Keptn feature.

Whether you want to store your helm chart, or a config file that contains a config like this

repo: <chart-repo>
chart: <chart-name>
chartVersion: <version>
namespace: <namespace>

does not matter for Keptn, but it matters only for the execution-plane service (e.g., helm-service, job-executor-service, ...).

[eugenbelsky]

This is the current state that has been initially designed, right. But this is exactly the place to kinda discuss/question things like this. Maybe it would make sense to reevaluate if this approach is still beneficial?

[jukrut]

It would 100% fit my needs, application standalone testing, different deployment scenarios, ...

Some changes e.g. create service is doable with the UI,
would this mean keptn would need write access to this repo?

Personally I would be totally fine, if changes can only be done with files.

I really like that idea, maybe because I'm used to that because it is what gitlab, github,... also do.
Then a change to keptn related files could be done with a meaningful commit message e.g. extended load test with case x.y.z
( for some files it already works, but not all so I mostly use the cli keptn add-resource ... and keep a seperate git repo with keptn files and hope nobody overwrote my changes to the shipyard.yaml )

I actually like the approach with Custom Resources too like tekton does.

Both approaches are an real improvement in my point of view as either with custom resources or a .keptn folder it is much easier to work as a team on configs. (e.g. Pull Request for changing configs)

I personally see no problem in giving keptn-core, locust-service, or any execution-plane-service read access to the repo.
If I don't want to store the helm-chart in a registry then I still could copy it there, If I want I can also do that, so all is possible what I could think of.
If I want to use pure yaml, or jsonnet, or whatever tool comes to mind I can do that.