Task image can't be pulled on GKE using core_containerd base image

[daniellavoie]

Summary

After upgrading from 7.9.1 to 7.11.0, I am experimenting the same symptoms than issue #8558. My cluster is running on GKE with base os being cos_containerd. My workaround was to switch to ubuntu_containerd.

I tried switching to guardian runtime but I would experience the same issue (I find it odd that the docker daemon is still being instantiated even with guardian configured. I did confirmed my workers pods were effectively running with CONCOURSE_RUNTIME: guardian env variable.

Steps to reproduce

• Deploy Concourse 7.11.0 with helm chart 17.2.1 to GKE using cos_containerd as OS images for worker node.

Expected results

• Task Image pulling should just work 🙃

Actual results

selected worker: concourse-worker-1
waiting for docker to come up...
waiting for docker to come up...
...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
waiting for docker to come up...
time="2023-11-29T18:53:13.168974357Z" level=info msg="Starting up"
time="2023-11-29T18:53:13.169852989Z" level=info msg="containerd not running, starting managed containerd"
time="2023-11-29T18:53:13.171028458Z" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=2730
time="2023-11-29T18:53:13.234472148Z" level=info msg="starting containerd" revision=61f9fd88f79f081d64d6fa3bb1a0dc71ec870523 version=1.6.24
time="2023-11-29T18:53:13.275520318Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2023-11-29T18:53:13.275696370Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2023-11-29T18:53:13.275990831Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exec: \"modprobe\": executable file not found in $PATH \"\"): skip plugin" type=io.containerd.snapshotter.v1
time="2023-11-29T18:53:13.276040763Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2023-11-29T18:53:13.276356816Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /scratch/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-11-29T18:53:13.276397037Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2023-11-29T18:53:13.276421083Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2023-11-29T18:53:13.276440204Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2023-11-29T18:53:13.276488729Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-11-29T18:53:13.276705051Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2023-11-29T18:53:13.277037537Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /scratch/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-11-29T18:53:13.277073371Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2023-11-29T18:53:13.277113533Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2023-11-29T18:53:13.277130855Z" level=info msg="metadata content store policy set" policy=shared
time="2023-11-29T18:53:13.277351914Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2023-11-29T18:53:13.277385920Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2023-11-29T18:53:13.277409590Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2023-11-29T18:53:13.277456033Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2023-11-29T18:53:13.277482118Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2023-11-29T18:53:13.277498958Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2023-11-29T18:53:13.277514330Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2023-11-29T18:53:13.277534730Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2023-11-29T18:53:13.277550299Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2023-11-29T18:53:13.277577667Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2023-11-29T18:53:13.277592608Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2023-11-29T18:53:13.277606254Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2023-11-29T18:53:13.277679829Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2023-11-29T18:53:13.277742201Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2023-11-29T18:53:13.282477708Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2023-11-29T18:53:13.282583678Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.282663021Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2023-11-29T18:53:13.282767320Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.282809946Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.282837502Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.282872104Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.282901562Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.282966828Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.283009980Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.283035264Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.283075315Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2023-11-29T18:53:13.283182074Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.283219588Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.283246811Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2023-11-29T18:53:13.283278287Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2023-11-29T18:53:13.283317266Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
time="2023-11-29T18:53:13.283342771Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2023-11-29T18:53:13.283384536Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
time="2023-11-29T18:53:13.284316387Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2023-11-29T18:53:13.284704370Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2023-11-29T18:53:13.285067935Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2023-11-29T18:53:13.285144194Z" level=info msg="containerd successfully booted in 0.052557s"
time="2023-11-29T18:53:13.352696575Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
time="2023-11-29T18:53:13.353022453Z" level=info msg="Loading containers: start."
time="2023-11-29T18:53:13.374081605Z" level=info msg="unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument`" error="exit status 4"
time="2023-11-29T18:53:13.493113454Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2023-11-29T18:53:13.493602531Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2023-11-29T18:53:13.493678835Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument

 (exit status 4)
Docker failed to start within 120 seconds.
/opt/resource/common.sh: line 113: kill: (2722) - No such process

Additional context

Triaging info

• Concourse : 7.11.0
• Browser (if applicable): N/A
• Did this used to work? Was working properly with 7.9.1.