ArgoCD endlessly showing diff for empty environment variables

[velp]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

ArgoCD endlessly trying to sync deployment with empty environment variables.

This behaviour can be reproduced only with specific helm-chart other charts that contain the same values are synced.

kubectl diff does not show these variables:

--- /var/folders/sb/v5w11ncx5vscty2qslyqvx700000gn/T/LIVE-586808863/apps.v1.Deployment.openstack.cinder-backup	2024-05-09 14:32:36
+++ /var/folders/sb/v5w11ncx5vscty2qslyqvx700000gn/T/MERGED-869100274/apps.v1.Deployment.openstack.cinder-backup	2024-05-09 14:32:36
@@ -2,16 +2,14 @@
 kind: Deployment
 metadata:
   annotations:
-    argocd.argoproj.io/tracking-id: openstack-apps-cinder:apps/Deployment:openstack/cinder-backup
     deployment.kubernetes.io/revision: "1"
     kubectl.kubernetes.io/last-applied-configuration: |
       {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"argocd.argoproj.io/tracking-id":"openstack-apps-cinder:apps/Deployment:openstack/cinder-backup","openstackhelm.openstack.org/release_uuid":""},"labels":{"application":"cinder","argocd.argoproj.io/instance":"openstack-apps-cinder","component":"backup","release_group":"cinder"},"name":"cinder-backup","namespace":"openstack"},"spec":{"replicas":1,"revisionHistoryLimit":3,"selector":{"matchLabels":{"application":"cinder","component":"backup","release_group":"cinder"}},"strategy":{"rollingUpdate":{"maxSurge":3,"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"annotations":{"configmap-bin-hash":"7e73f961734d56bfe5bb6fdfcf447108d9ee0ddc5ee5309dd62cd51666bb24cd","configmap-etc-hash":"336f78b3387157ff9da41b75181c96807d0ca3aac5dbbee969066f23335af852","k8s.v1.cni.cncf.io/networks":"rook-ceph/rook-public-net@eth1","openstackhelm.openstack.org/release_uuid":""},"labels":{"application":"cinder","component":"backup","release_group":"cinder"}},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"release_group","operator":"In","values":["cinder"]},{"key":"application","operator":"In","values":["cinder"]},{"key":"component","operator":"In","values":["backup"]}]},"topologyKey":"kubernetes.io/hostname"}]}},"containers":[{"command":["/tmp/cinder-backup.sh"],"image":"cloudification/cinder:2023.2-ubuntu_jammy-20240321205704","imagePullPolicy":"IfNotPresent","name":"cinder-backup","securityContext":{"capabilities":{"add":["SYS_ADMIN"]},"readOnlyRootFilesystem":true,"redOnlyRootFilesystem":null,"runAsUser":0},"terminationMessagePath":"/var/log/termination-log","volumeMounts":[{"mountPath":"/tmp","name":"pod-tmp"},{"mountPath":"/var/lib/cinder/tmp","name":"cinder-tmp"},{"mountPath":"/tmp/cinder-backup.sh","name":"cinder-bin","readOnly":true,"subPath":"cinder-backup.sh"},{"mountPath":"/etc/cinder/cinder.conf","name":"cinder-etc","readOnly":true,"subPath":"cinder.conf"},{"mountPath":"/etc/cinder/logging.conf","name":"cinder-etc","readOnly":true,"subPath":"logging.conf"},{"mountPath":"/etc/ceph","name":"etcceph"},{"mountPath":"/etc/ceph/ceph.conf","name":"ceph-etc","readOnly":true,"subPath":"ceph.conf"},{"mountPath":"/tmp/client-keyring","name":"ceph-backup-keyring","readOnly":true,"subPath":"key"},{"mountPath":"/var/lib/cinder/coordination","name":"cinder-coordination"},{"mountPath":"/etc/sudoers.d/kolla_cinder_sudoers","name":"cinder-etc","readOnly":true,"subPath":"cinder_sudoers"},{"mountPath":"/etc/sudoers.d/kolla_cinder_volume_sudoers","name":"cinder-etc","readOnly":true,"subPath":"cinder_sudoers"},{"mountPath":"/etc/cinder/rootwrap.conf","name":"cinder-etc","readOnly":true,"subPath":"rootwrap.conf"},{"mountPath":"/etc/cinder/rootwrap.d/volume.filters","name":"cinder-etc","readOnly":true,"subPath":"volume.filters"}]}],"initContainers":[{"command":["kubernetes-entrypoint"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INTERFACE_NAME","value":"eth0"},{"name":"PATH","value":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/"},{"name":"DEPENDENCY_SERVICE","value":"openstack:cinder-api,openstack:keystone-api,openstack:rabbitmq-cinder"},{"name":"DEPENDENCY_JOBS","value":"cinder-db-sync,cinder-ks-user,cinder-ks-endpoints"},{"name":"DEPENDENCY_DAEMONSET","value":""},{"name":"DEPENDENCY_CONTAINER","value":""},{"name":"DEPENDENCY_POD_JSON","value":""},{"name":"DEPENDENCY_CUSTOM_RESOURCE","value":""}],"image":"quay.io/airshipit/kubernetes-entrypoint:v1.0.0","imagePullPolicy":"IfNotPresent","name":"init","securityContext":{"allowPrivilegeEscalation":false,"readOnlyRootFilesystem":true,"runAsUser":65534},"volumeMounts":null},{"command":["/tmp/ceph-keyring.sh"],"env":[{"name":"RBD_USER","value":"cinderbackup"}],"image":"cloudification/cinder:2023.2-ubuntu_jammy-20240321205704","imagePullPolicy":"IfNotPresent","name":"ceph-backup-keyring-placement","securityContext":{"readOnlyRootFilesystem":true,"runAsUser":0},"volumeMounts":[{"mountPath":"/tmp","name":"pod-tmp"},{"mountPath":"/etc/ceph","name":"etcceph"},{"mountPath":"/tmp/ceph-keyring.sh","name":"cinder-bin","readOnly":true,"subPath":"ceph-keyring.sh"},{"mountPath":"/tmp/client-keyring","name":"ceph-backup-keyring","readOnly":true,"subPath":"key"}]},{"command":["/tmp/ceph-keyring.sh"],"env":[{"name":"RBD_USER","value":"cinder"}],"image":"cloudification/cinder:2023.2-ubuntu_jammy-20240321205704","imagePullPolicy":"IfNotPresent","name":"ceph-keyring-placement-rbd1","securityContext":{"readOnlyRootFilesystem":true,"runAsUser":0},"volumeMounts":[{"mountPath":"/tmp","name":"pod-tmp"},{"mountPath":"/etc/ceph","name":"etcceph"},{"mountPath":"/tmp/ceph-keyring.sh","name":"cinder-bin","readOnly":true,"subPath":"ceph-keyring.sh"},{"mountPath":"/tmp/client-keyring","name":"ceph-keyring","readOnly":true,"subPath":"key"}]},{"command":["chown","-R","cinder:","/var/lib/cinder/coordination"],"image":"cloudification/cinder:2023.2-ubuntu_jammy-20240321205704","imagePullPolicy":"IfNotPresent","name":"ceph-coordination-volume-perms","securityContext":{"readOnlyRootFilesystem":true,"runAsUser":0},"volumeMounts":[{"mountPath":"/tmp","name":"pod-tmp"},{"mountPath":"/var/lib/cinder/coordination","name":"cinder-coordination"}]}],"nodeSelector":{"openstack-control-plane":"enabled"},"securityContext":{"runAsUser":42424},"serviceAccountName":"cinder-backup","volumes":[{"emptyDir":{},"name":"pod-tmp"},{"emptyDir":{},"name":"cinder-tmp"},{"name":"cinder-etc","secret":{"defaultMode":292,"secretName":"cinder-etc"}},{"configMap":{"defaultMode":365,"name":"cinder-bin"},"name":"cinder-bin"},{"emptyDir":{},"name":"etcceph"},{"configMap":{"defaultMode":292,"name":"ceph-etc"},"name":"ceph-etc"},{"name":"ceph-backup-keyring","secret":{"secretName":"cinder-backup-rbd-keyring"}},{"name":"ceph-keyring","secret":{"secretName":"cinder-volume-rbd-keyring"}},{"emptyDir":{},"name":"cinder-coordination"}]}}}}
     openstackhelm.openstack.org/release_uuid: ""
   creationTimestamp: "2024-05-09T10:43:53Z"
-  generation: 1
+  generation: 2
   labels:
     application: cinder
-    argocd.argoproj.io/instance: openstack-apps-cinder
     component: backup
     release_group: cinder
   managedFields:
@@ -21,13 +19,11 @@
       f:metadata:
         f:annotations:
           .: {}
-          f:argocd.argoproj.io/tracking-id: {}
           f:kubectl.kubernetes.io/last-applied-configuration: {}
           f:openstackhelm.openstack.org/release_uuid: {}
         f:labels:
           .: {}
           f:application: {}
-          f:argocd.argoproj.io/instance: {}
           f:component: {}
           f:release_group: {}
       f:spec:
exit status 1

In the logs nothing suspicious:

argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:41Z" level=info msg=Syncing application=argocd/openstack-apps-cinder skipHooks=true started=false syncId=01649-DUCLB
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:41Z" level=info msg="Namespace already exists" application=argocd/openstack-apps-cinder namespace=openstack syncId=01649-DUCLB
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:41Z" level=info msg="Tasks (dry-run)" application=argocd/openstack-apps-cinder syncId=01649-DUCLB tasks="[Sync/0 resource apps/Deployment:openstack/cinder-backup obj->obj (,,)]"
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:41Z" level=info msg="Applying resource Deployment/cinder-backup in cluster: https://100.64.144.1:443, namespace: openstack"
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:42Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/openstack-data-pxc
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:42Z" level=info msg="Updating operation state. phase: Running -> Running, message: '' -> 'one or more tasks are running'" application=argocd/openstack-apps-cinder syncId=01649-DUCLB
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:42Z" level=info msg="Applying resource Deployment/cinder-backup in cluster: https://100.64.144.1:443, namespace: openstack"
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:42Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'deployment.apps/cinder-backup configured'" application=argocd/openstack-apps-cinder kind=Deployment name=cinder-backup namespace=openstack phase=Sync syncId=01649-DUCLB
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:42Z" level=info msg="Updating operation state. phase: Running -> Succeeded, message: 'one or more tasks are running' -> 'successfully synced (all tasks run)'" application=argocd/openstack-apps-cinder syncId=01649-DUCLB
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:42Z" level=info msg="sync/terminate complete" application=argocd/openstack-apps-cinder duration=489.052922ms syncId=01649-DUCLB
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:42Z" level=info msg="updated 'argocd/openstack-apps-cinder' operation (phase: Succeeded)" appNamespace=argocd application=openstack-apps-cinder project=openstack-apps
argocd-application-controller-0 argocd-application-controller time="2024-05-09T12:43:42Z" level=info msg="Partial sync operation to  succeeded" application=openstack-apps-cinder dest-namespace=openstack dest-server="https://kubernetes.default.svc" reason=OperationCompleted type=Normal

What I've tried already:

• change sync strategy to ServerSideDiff=true
• remove and re-sync entire application
• sync with replace
• restart/reinstall ArgoCD

The strangest thing is that other applications also have such variables, but there are no problems with synchronisation:

diff is empty for this app:

argocd@argocd-server-775db9bdfd-r9lw6:~$ argocd app diff argocd/openstack-apps-neutron
argocd@argocd-server-775db9bdfd-r9lw6:~$

I understand that this is an environment-specific case and tried to ask about it in Slack. But I didn’t receive an answer and I ran out of ideas on what it could be.

To Reproduce

Install openstack-helm Cinder chart with cinder-backup enabled.

Expected behavior

Ignoring empty values in diff, or be able to sync them.

Screenshots

Version

argocd@argocd-server-775db9bdfd-r9lw6:~$ argocd version
argocd: v2.9.6+ba62a0a
  BuildDate: 2024-02-02T19:17:57Z
  GitCommit: ba62a0a86d19f71a65ec2b510a39ea55497e1580
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.9.6+ba62a0a
  BuildDate: 2024-02-02T19:17:57Z
  GitCommit: ba62a0a86d19f71a65ec2b510a39ea55497e1580
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.13.2+g2a2fb3b
  Kubectl Version: v0.24.17
  Jsonnet Version: v0.20.0

Logs

Paste any relevant application logs here.

[tooptoop4]

try v2.11

[srinivasmummareddy]

@velp @tooptoop4 any update on this...even I'm having same issue

[velp]

@srinivasmummareddy in my edge case upgrade to v2.11 helped. Thx @tooptoop4 for idea. I think this issue can be closed.

[srinivasmummareddy]

@tooptoop4 I'm already using v2.11. I'm having empty environment variables but argocd not considering it