How are other integrating with ArgoCD?

[gespi1]

Hi all 👋

Started using devlake about a week ago to gather DORA metrics from our ArgoCD instances. Configured ArgoCD notifications to send webhooks to devlake. Our argos are running multi-source applications so our applications can have repo URLs coming from different git platforms or repo names. When theres a change from a single repo and a sync happens argo returns with a list of repos URLs and there is no differentiator on what repo the new revision came from. Sometimes changes can come from multiple repos and that can result in multiple shas as well. Dev lake only supports single repo URLs and commit sha.

So i am wondering how others are dealing with this, if any? Feel like devlake is the wrong tool for tracking deployments on argocd multisource apps.

ty

[klesh]

Hi, @gespi1 . I think you are right about the deployment webhook should accept multiple repo URLs and commit shas.

By the latest definition:

1. DeploymentCommit represent a commit of a repo of a Deployment
2. Deployment is a set of DeploymentCommit

However, The deployment webhook is actually deployment commit webhook by the current implementation since there were arguments regarding what was a deployment in the community during the earlier time of development. So, you may post multiple times to the deployment webhook for your multisource apps.

With that being said, I think we should either update the deployment webhook to accept multiple deployment commits, or rename it to /deployment_commits to better reflect its nature.

@Startrekzky Would like to hear from you.

[Startrekzky]

Hi @gespi1 . The current payload schema for the deployment webhook is

curl http://localhost:8080/plugins/webhook/1/deployments -X 'POST' -d '{
    "pipeline_id": "optional-pipeline-id",
    "environment":"PRODUCTION",
    "repo_url":"https://github.com/apache/incubator-devlake/",
    "repo_id": "optional-repo-id",
    "ref_name": "optional-release-v0.17",
    "commit_sha":"015e3d3b480e417aede5a1293bd61de9b0fd051d",
    "create_time":"2020-01-01T11:00:00+00:00",
    "start_time":"2020-01-01T12:00:00+00:00",
    "end_time":"2020-01-02T13:00:00+00:00",
    "result": "FAILURE"
  }'

Do you want something like this?

curl http://localhost:8080/plugins/webhook/1/deployments -X 'POST' -d '{
    "pipeline_id": "optional-pipeline-id",
    "deployments":[
       {
           "repo_url":"repo-1",
           "environment":"PRODUCTION",
           "ref_name": "optional-release-v0.17",
           "commit_sha":"c1"
       },
       {
           "repo_url":"repo-2",
           "environment":"PRODUCTION",
           "ref_name": "optional-release-v0.17",
           "commit_sha":"c2"
       }
    ]
    "create_time":"2020-01-01T11:00:00+00:00",
    "start_time":"2020-01-01T12:00:00+00:00",
    "end_time":"2020-01-02T13:00:00+00:00",
    "result": "FAILURE"
  }'

[klesh]

@Startrekzky I think it should be:

curl http://localhost:8080/plugins/webhook/1/deployments -X 'POST' -d '{
    "pipeline_id": "optional-pipeline-id",
    "deploymentCommits":[
       {
           "repo_url":"repo-1",
           "environment":"PRODUCTION",
           "ref_name": "optional-release-v0.17",
           "commit_sha":"c1"
       },
       {
           "repo_url":"repo-2",
           "environment":"PRODUCTION",
           "ref_name": "optional-release-v0.17",
           "commit_sha":"c2"
       }
    ]
    "create_time":"2020-01-01T11:00:00+00:00",
    "start_time":"2020-01-01T12:00:00+00:00",
    "end_time":"2020-01-02T13:00:00+00:00",
    "result": "FAILURE"
  }'

[Startrekzky]

@klesh If the pairs are deploymentCommits, should there be a deployment_id key? Or we just use pipeline_id as the deployment id?

[klesh]

deployment_id is not necessary because we are creating the deployment along with all its deployment_commits so deployment_id is unknown at this point.

[Startrekzky]

@klesh Is the deployment id an auto_increment id?

[klesh]

No, it is a composite key https://github.com/apache/incubator-devlake/blob/main/backend/plugins/webhook/api/deployments.go#L96

[Startrekzky]

Hi @gespi1 , I created #6262. Let me know if this solution will solve your problem.

[gespi1]

Wow this is amazing, the team hopped on this pretty quickly! Thanks

Some feedback. Sometimes an argo multisource app only one source is being changed and there will only be one commit sha to provide for the webhook. In this case would we leave the other deploymentCommits items with empty values for commit_sha. How would it work?

Would using deploymentCommits still count as one deployment, even if multiple sources were updated (imo one works)?

How would lead time be calculated when multiple source have changes and vary in time differences. take the average?

thanks

[Startrekzky]

Hi @gespi1 ,

Sometimes an argo multisource app only one source is being changed and there will only be one commit sha to provide for the webhook. In this case would we leave the other deploymentCommits items with empty values for commit_sha. How would it work?

In this case, you can only push the deploymentCommit that has been changed. Does it make sense?

Would using deploymentCommits still count as one deployment, even if multiple sources were updated (imo one works)?

The Argo deployment with multiple sources would be considered as ONE deployment and multiple deploymentCommits in DevLake. In DevLake v0.20 and later versions, deployment will be stored in table.cicd_deployment, deploymentCommits will be stored in table.cicd_deployment_commits.

How would lead time be calculated when multiple source have changes and vary in time differences. take the average?

The algorithm can be found here. You can look at the doc first and let me know if you still have questions. https://devlake.apache.org/docs/Metrics/LeadTimeForChanges#how-is-it-calculated

cc. @klesh

[gespi1]

ty for the clarifications