[Help] Image Discovered but Not Updated

[sheldontsen-qb]

Describe the bug
Not really a bug, more like me missing something. Would appreciate a bit of help.

To Reproduce
These are the logs produced (slightly sanitised):

time="2024-01-09T18:31:48Z" level=debug msg="Processing application blabla"
time="2024-01-09T18:31:48Z" level=debug msg="Considering this image for update" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-09T18:31:48Z" level=debug msg="Using version constraint 'latest' when looking for a new tag" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-09T18:31:48Z" level=debug msg="ignoring v2 manifest sha256:5b813171a01a329e92b80d78a44c73ebbacf35d3cbb15e7fab93683adbc1adaf. Manifest platform: linux/arm64, requested: linux/arm64" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-09T18:31:48Z" level=debug msg="No metadata found for fullstack-backend:latest" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-09T18:31:48Z" level=debug msg="target parameters: image-spec= image-name=image.name, image-tag=image.tag" application=blabla image="my-registry:5000/fullstack-backend"
time="2024-01-09T18:31:48Z" level=debug msg="Image 'my-registry:5000/fullstack-backend@dummy' already on latest allowed version" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=dummy registry="my-registry:5000"
time="2024-01-09T18:31:48Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=0"
time="2024-01-09T18:33:48Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"

time="2024-01-09T18:33:48Z" level=debug msg="Processing application blabla"
time="2024-01-09T18:33:48Z" level=debug msg="Considering this image for update" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-09T18:33:48Z" level=debug msg="Using version constraint 'latest' when looking for a new tag" alias=fullstack-backend application=blablaimage_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-09T18:33:48Z" level=debug msg="ignoring v2 manifest sha256:1687db497059f4c7f107ad27bf9b1096460c606e88d06cdc99311be4547f351f. Manifest platform: linux/arm64, requested: linux/arm64" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-09T18:33:48Z" level=debug msg="No metadata found for fullstack-backend:latest" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-09T18:33:48Z" level=debug msg="target parameters: image-spec= image-name=image.name, image-tag=image.tag" application=blabla image="my-registry:5000/fullstack-backend"
time="2024-01-09T18:33:48Z" level=debug msg="Image 'my-registry:5000/fullstack-backend@dummy' already on latest allowed version" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=dummy registry="my-registry:5000"
time="2024-01-09T18:33:48Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=0"
time="2024-01-09T18:35:48Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"

Note the sha256 part changes when I push a new image using the following commands:

docker build -f /app/repo/projects/fullstack-backend/Dockerfile -t localhost:${REG_PORT}/fullstack-backend:latest /app/repo
docker push localhost:${REG_PORT}/fullstack-backend:latest

Ignore the localhost part, that is translated to the right registry called my-registry, and I can confirm the pods can pull from this private/local registry.

I've also annotated my ArgoCD app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: blabla
  annotations:
    argocd-image-updater.argoproj.io/image-list: fullstack-backend=my-registry:5000/fullstack-backend:latest
    argocd-image-updater.argoproj.io/update-strategy: digest

And my deployment spec is:

...
    spec:
      containers:
        - name: fullstack-backend
          image: my-registry:5000/fullstack-backend:latest
          imagePullPolicy: Always

When I run the CLI:

root@bd7193133e49:/app# kubectl exec -it argocd-image-updater-5576fcbc57-9bfqd -n argocd -- argocd-image-updater test my-registry:5000/fullstack-backend --registries-conf-path /app/config/registries.conf
DEBU[0000] Creating in-cluster Kubernetes client        
INFO[0000] retrieving information about image            image_alias= image_name="my-registry:5000/fullstack-backend" registry_url="my-registry:5000"
DEBU[0000] rate limiting is disabled                     prefix="my-registry:5000" registry="http://172.16.238.10:5000"
INFO[0000] Loaded 1 registry configurations from /app/config/registries.conf 
INFO[0000] Fetching available tags and metadata from registry  application=test image_alias= image_name="my-registry:5000/fullstack-backend" registry_url="my-registry:5000"
INFO[0000] Found 1 tags in registry                      application=test image_alias= image_name="my-registry:5000/fullstack-backend" registry_url="my-registry:5000"
DEBU[0000] could not parse input tag latest as semver: Invalid Semantic Version 
INFO[0000] no newer version of image found               application=test image_alias= image_name="my-registry:5000/fullstack-backend" registry_url="my-registry:5000"

It does find it.

I must be doing something wrong but I'm not sure what! Does it have something to do with Invalid Semantic Version? If so, what? Any help would be appreciated. Thank you!

Note: I just want to point to xxx:latest and let it update automagically.

PS: Really neat stuff - k8s newbie.

Update, I went through the docs again and followed the Helm parameters part, changing my helm deployment file to:

    spec:
      containers:
        - name: fullstack-backend
          image: "{{ .Values.backend.image.repository }}/{{ .Values.backend.image.name }}:{{ .Values.backend.image.tag }}"

And my values file has:

backend:
  image:
    repository: my-registry:5000
    name: fullstack-backend
    tag: latest

Still no luck :/

[ML-std]

Hi @sheldontsen-qb , you need to use alias name on strategy:

    argocd-image-updater.argoproj.io/image-list: fullstack-backend=my-registry:5000/fullstack-backend:latest
    argocd-image-updater.argoproj.io/fullstack-backend.update-strategy: digest

Also read From ArgoCD

Good Day!

[sheldontsen-qb]

Thanks @ML-std, I was following here (https://argocd-image-updater.readthedocs.io/en/stable/configuration/images/#application-wide-defaults) where it says you can do application wide, and I tried with alias earlier. Let me try again.

Update: You can see the SHA changing but it still does not update unfortunately :/

...
time="2024-01-11T11:06:32Z" level=debug msg="ignoring v2 manifest sha256:7e71f19b73b29ea74b50329def061bce9ad5ee4e94b95bf371c8b9fb5eeb0e07. Manifest platform: linux/arm64, requested: linux/arm64" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T11:06:32Z" level=debug msg="No metadata found for fullstack-backend:latest" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T11:06:32Z" level=debug msg="target parameters: image-spec= image-name=image.name, image-tag=image.tag" application=blabla image="my-registry:5000/fullstack-backend"
time="2024-01-11T11:06:32Z" level=debug msg="Image 'my-registry:5000/fullstack-backend@dummy' already on latest allowed version" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=dummy registry="my-registry:5000"



time="2024-01-11T11:06:32Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=0"
time="2024-01-11T11:08:32Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2024-01-11T11:08:32Z" level=debug msg="Processing application blabla"
time="2024-01-11T11:08:32Z" level=debug msg="Considering this image for update" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T11:08:32Z" level=debug msg="Using version constraint 'latest' when looking for a new tag" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T11:08:32Z" level=debug msg="ignoring v2 manifest sha256:3cd62f976da2364919dac25950ff5f322bf078ea052cfd78f70cb5f1ac520d1b. Manifest platform: linux/arm64, requested: linux/arm64" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T11:08:32Z" level=debug msg="No metadata found for fullstack-backend:latest" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T11:08:32Z" level=debug msg="target parameters: image-spec= image-name=image.name, image-tag=image.tag" application=blabla image="my-registry:5000/fullstack-backend"
time="2024-01-11T11:08:32Z" level=debug msg="Image 'my-registry:5000/fullstack-backend@dummy' already on latest allowed version" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=dummy registry="my-registry:5000"
time="2024-01-11T11:08:32Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=0"
time="2024-01-11T11:10:32Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"

My annotations:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: blabla
  annotations:
    argocd-image-updater.argoproj.io/image-list: fullstack-backend=my-registry:5000/fullstack-backend:latest
    argocd-image-updater.argoproj.io/fullstack-backend.update-strategy: digest

[ML-std]

@sheldontsen-qb I'm not sure about this, but can you try deleting tag from values.yaml and give it an empty string:

backend:
  image:
    repository: my-registry:5000
    name: fullstack-backend
    tag: ""

ArgoCD may try to override sha with latest, resulting no update.

[sheldontsen-qb]

Hmmm thanks for the response!

Unfortunately no luck.

This is the new values file:

image:
  name: fullstack-backend
  tag: ""

ArgoCD outright complains:

Failed to apply default image tag "my-registry:5000/fullstack-backend:": couldn't parse image reference "my-registry:5000/fullstack-backend:": invalid reference format

Should I also update the deployment to remove tag?

    spec:
      containers:
        - name: fullstack-backend
          image: "{{ .Values.backend.image.repository }}/{{ .Values.backend.image.name }}:{{ .Values.backend.image.tag }}"

But without this, how would image-updater work?

[ML-std]

have you added your private registery to image-updater config?

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-image-updater-config
    app.kubernetes.io/part-of: argocd-image-updater
  name: argocd-image-updater-config
data:
  #applications_api: argocd
  # The address of Argo CD API endpoint - defaults to argocd-server.argocd
  #argocd.server_addr: 10.1.5.227
  # Whether to use GRPC-web protocol instead of GRPC over HTTP/2
  #argocd.grpc_web: "true"
  # Whether to ignore invalid TLS cert from Argo CD API endpoint
  #argocd.insecure: "true"
  # Whether to use plain text connection (http) instead of TLS (https)
  #argocd.plaintext: "false"
  registries.conf: |
    registries:
    - name: my-registery
      api_url: my-registery-url
      ping: yes
      insecure: yes
      defaultns: library
      default: true
---

then, you can remove localhost:5000 from youe image:

  annotations:
    argocd-image-updater.argoproj.io/image-list: fullstack-backend=fullstack-backend:latest
    argocd-image-updater.argoproj.io/fullstack-backend.update-strategy: digest

you shouldnt remove the tag

[sheldontsen-qb]

Thanks @ML-std!

Yes indeed, I have my private repo added to the configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-image-updater-config
    app.kubernetes.io/part-of: argocd-image-updater
  name: argocd-image-updater-config
data:
  log.level: debug
  registries.conf: |
    registries:
    - name: my-private-registry
      prefix: my-registry:5000
      api_url: http://172.16.238.10:5000
      insecure: yes
      default: true

(I omitted defaultns)

I can confirm it is picked up because the logs right after initialisation show that it found an extra registry in the config. So defo works. Also from the logs its quite clear the registry is picked up.

But when I followed your suggestion to change from fullstack-backend=my-registry:5000/fullstack-backend:latest to fullstack-backend=fullstack-backend:latest, argocd image updated no longer detects it:

time="2024-01-11T16:56:27Z" level=debug msg="Image 'fullstack-backend' seems not to be live in this application, skipping" application=blabla
time="2024-01-11T16:56:27Z" level=info msg="Processing results: applications=1 images_considered=0 images_skipped=1 images_updated=0 errors=0"
time="2024-01-11T16:58:27Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"

Reintroducing my-registry:5000 brings us back to where we were:

time="2024-01-11T17:08:27Z" level=debug msg="Considering this image for update" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T17:08:27Z" level=debug msg="Using version constraint 'latest' when looking for a new tag" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T17:08:27Z" level=debug msg="ignoring v2 manifest sha256:1285bcf6d0223da2d37aaad4f15cef35e65733fdd679ab21b7fdac8eed94a35a. Manifest platform: linux/arm64, requested: linux/arm64" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T17:08:27Z" level=debug msg="No metadata found for fullstack-backend:latest" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=latest registry="my-registry:5000"
time="2024-01-11T17:08:27Z" level=debug msg="target parameters: image-spec= image-name=image.name, image-tag=image.tag" application=blabla image="my-registry:5000/fullstack-backend"
time="2024-01-11T17:08:27Z" level=debug msg="Image 'my-registry:5000/fullstack-backend@dummy' already on latest allowed version" alias=fullstack-backend application=blabla image_name=fullstack-backend image_tag=dummy registry="my-registry:5000"
time="2024-01-11T17:08:27Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=0"
time="2024-01-11T17:10:27Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"

:/

[ML-std]

Hmm, should it be related to registery? It has to detect the registery without you adding it to the image name. In our project, we use Harbor for OCI, and it works perfectly fine. Can it be related to that maybe?

[sheldontsen-qb]

Hmmm, I use registry:2

  my-registry:
    image: registry:2
    ports:
      - ${REG_PORT}:5000
    environment:
      - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/data
    restart: unless-stopped

Which from the docs (https://hub.docker.com/_/registry) seems to be OCI compliant.

[jtklahti]

I had a similar problem too. Hope this helps some one!

I dug around with this one for quite a while and finally figured it out. On my Mac (running in Kind with registry:2), I also got that same log line containing "ignoring v2 manifest", which is written here.

What is not logged, is the variant of the image and that was in my case "v8" resulting in "linux/arm64/v8".

Adding this to the platforms annotation solved the issue:

argocd-image-updater.argoproj.io/imagealias.platforms: linux/amd64,linux/arm64,linux/arm64/v8

Also, in the documentation, platform-annotation is missing the s (or does it work without?).