"Quickstart" Guide doesn't work on M1 Mac due to docker compose issue

[tedivm]

Summary

The quickstart page recommends using a supplied docker compose file to start.

Running this docker compose file on an M1 fails.

Steps to reproduce

$ curl -O https://concourse-ci.org/docker-compose.yml
$ docker-compose up -d

Expected results

Concourse launches.

Actual results

1. Attempting it directly-

 ⠋ concourse The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                                                            0.0s
...
concourse-concourse-1     | {"timestamp":"2023-03-29T18:05:21.330364719Z","level":"error","source":"worker","message":"worker.garden-runner.logging-runner-exited","data":{"error":"Exit trace for group:\ncontainerd-garden-backend exited with error: setup host network failed: create chain or flush if exists failed: running [/usr/sbin/iptables -t filter -N CONCOURSE-OPERATOR --wait]: exit status 3: iptables v1.8.7 (legacy): can't initialize iptables table `filter': iptables who? (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\n\ncontainerd exited with nil\n","session":"8"}}
...
concourse-concourse-1     | containerd-garden-backend exited with error: setup host network failed: create chain or flush if exists failed: running [/usr/sbin/iptables -t filter -N CONCOURSE-OPERATOR --wait]: exit status 3: iptables v1.8.7 (legacy): can't initialize iptables table `filter': iptables who? (do you need to insmod?)

2. Attempting it with the "dev" image, from this issue.

concourse-concourse-1     | error: the required flag `--worker-work-dir' was not specified
concourse-concourse-1 exited with code 1

Note that this does resolve the "wrong architecture" problem but otherwise seems like a broken image.

Additional context

This is with a fresh docker desktop install.

Triaging info

• Concourse version: latest
• Browser (if applicable):
• Did this used to work? No idea, I'm new to concourse. I assume it did due to the documentation recommending it.

[ajs95]

@tedivm Try with below change in docker-compose.yml for mac

  CONCOURSE_WORKER_RUNTIME: "houdini"

instead of "containerd"
https://concourse-ci.org/concourse-worker.html#configuring-runtimes

[tedivm]

That does get me a bit further but still results in an error when attempting to run the tutorial job.

Service Logs:

concourse-concourse-1     | {"timestamp":"2023-03-30T15:33:12.467050347Z","level":"error","source":"baggageclaim","message":"baggageclaim.api.volume-server.stream-out.failed-to-stream-out","data":{"error":"fork/exec /usr/bin/tar: invalid argument","session":"4.1.61","volume":"efd3e841-beeb-475e-7458-7fadded91d07"}}
concourse-concourse-1     | {"timestamp":"2023-03-30T15:33:12.469442763Z","level":"error","source":"atc","message":"atc.tracker.notify.run.task-step.failed-to-stream-metadata-file","data":{"build":"1","build_id":1,"error":"failed to stream out from volume","job":"hello-world-job","job-id":1,"pipeline":"hello-world","session":"26.29.2.3","step-name":"hello-world-task","team":"main"}}
concourse-concourse-1     | {"timestamp":"2023-03-30T15:33:12.469961930Z","level":"error","source":"atc","message":"atc.tracker.notify.run.task-step.failed-to-fetch-image-for-container","data":{"build":"1","build_id":1,"error":"stream image metadata file: failed to stream out from volume","job":"hello-world-job","job-id":1,"pipeline":"hello-world","session":"26.29.2.3","step-name":"hello-world-task","team":"main"}}
concourse-concourse-1     | {"timestamp":"2023-03-30T15:33:12.472989180Z","level":"error","source":"atc","message":"atc.tracker.notify.run.task-step.failed-to-create-container-in-garden","data":{"build":"1","build_id":1,"container":"3f1c76dc-bd3d-4c6c-4bc6-2238b1532f2d","error":"stream image metadata file: failed to stream out from volume","job":"hello-world-job","job-id":1,"pipeline":"hello-world","session":"26.29.2.3","step-name":"hello-world-task","team":"main"}}
concourse-concourse-1     | {"timestamp":"2023-03-30T15:33:12.475501388Z","level":"info","source":"atc","message":"atc.tracker.notify.run.errored","data":{"build":"1","build_id":1,"error":"find or create container on worker 06630b9c1782: stream image metadata file: failed to stream out from volume","job":"hello-world-job","pipeline":"hello-world","session":"26.29.2","team":"main"}}
concourse-concourse-1     | {"timestamp":"2023-03-30T15:33:12.491823888Z","level":"info","source":"atc","message":"atc.tracker.notify.run.finish.errored","data":{"build":"1","build_id":1,"error":"find or create container on worker 06630b9c1782: stream image metadata file: failed to stream out from volume","job":"hello-world-job","pipeline":"hello-world","session":"26.29.2.22","team":"main"}}
concourse-concourse-1     | {"timestamp":"2023-03-30T15:33:16.009880501Z","level":"info","source":"atc","message":"atc.scanner.tick.start","data":{"session":"23.29"}}
concourse-concourse-1     | {"timestamp":"2023-03-30T15:33:16.022287834Z","level":"info","source":"atc","message":"atc.scanner.tick.end","data":{"session":"23.29"}}
concourse-concourse-db-1  | 2023-03-30 15:33:24.838 UTC [27] LOG:  checkpoint starting: time

Fly CLI Output:

rhafne892@RESML-1732118 concourse % fly -t tutorial set-pipeline -p hello-world -c hello-world.yml
jobs:
  job hello-world-job has been added:
+ name: hello-world-job
+ plan:
+ - config:
+     image_resource:
+       name: ""
+       source:
+         repository: busybox
+       type: registry-image
+     platform: linux
+     run:
+       args:
+       - Hello world!
+       path: echo
+   task: hello-world-task

pipeline name: hello-world

apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http://localhost:8080/teams/main/pipelines/hello-world

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command:
    fly -t tutorial unpause-pipeline -p hello-world
  - click play next to the pipeline in the web ui
rhafne892@RESML-1732118 concourse % fly -t tutorial unpause-pipeline -p hello-world
unpaused 'hello-world'
rhafne892@RESML-1732118 concourse % fly -t tutorial trigger-job --job hello-world/hello-world-job --watch
started hello-world/hello-world-job #1

initializing
initializing check: image
selected worker: 06630b9c1782
selected worker: 06630b9c1782
fetching busybox@sha256:b5d6fe0712636ceb7430189de28819e195e8966372edfc2d9409d79402a0dc16
4b35f584bb4f [==========================================] 2.5MiB/2.5MiB
selected worker: 06630b9c1782
find or create container on worker 06630b9c1782: stream image metadata file: failed to stream out from volume
errored

[tedivm]

I've tried with the "alpine" and "python" containers as well, just in case, but end up with the same error-

find or create container on worker 06630b9c1782: stream image metadata file: failed to stream out from volume

[danlsgiga]

Here's a working docker-compose.yml file I got working on my Mac M1. The key is to use the concourse/concourse:7.9.1-ubuntu image and change / add the following env vars:

CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: naive
CONCOURSE_WORKER_BAGGAGECLAIM_DISABLE_USER_NAMESPACES: true
CONCOURSE_WORKER_RUNTIME: "houdini"

Alpine based images usually just makes things worse and all to save a few MBs. At least, from my experience, it is not worth the trouble they bring. Anyways, here is the full compose file. Hope it helps!

Full docker-compose.yml file

version: '3.9'

services:
  concourse-db:
    image: postgres
    environment:
      POSTGRES_DB: &PG_DBNAME concourse
      POSTGRES_PASSWORD: &PG_PASSWORD concourse_pass
      POSTGRES_USER: &PG_USER concourse_user
      PGDATA: /database

  concourse:
    image: concourse/concourse:7.9.1-ubuntu
    platform: linux/amd64
    command: quickstart
    privileged: true
    depends_on:
      - concourse-db
    ports:
      - "8080:8080"
    environment:
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: *PG_USER
      CONCOURSE_POSTGRES_PASSWORD: *PG_PASSWORD
      CONCOURSE_POSTGRES_DATABASE: *PG_DBNAME
      CONCOURSE_EXTERNAL_URL: http://localhost:8080
      CONCOURSE_ADD_LOCAL_USER: ptfe:ptfe
      CONCOURSE_MAIN_TEAM_LOCAL_USER: ptfe
      CONCOURSE_WORKER_SWEEP_INTERVAL: 24h
      # If running on a non M1/M2 MacOS, optionally change this to overlay
      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: naive
      CONCOURSE_WORKER_BAGGAGECLAIM_DISABLE_USER_NAMESPACES: true
      CONCOURSE_CLIENT_SECRET: Y29uY291cnNlLXdlYgo=
      CONCOURSE_TSA_CLIENT_SECRET: Y29uY291cnNlLXdvcmtlcgo=
      CONCOURSE_X_FRAME_OPTIONS: allow
      CONCOURSE_CONTENT_SECURITY_POLICY: "*"
      CONCOURSE_CLUSTER_NAME: PTFE
      CONCOURSE_WORKER_WORK_DIR: /worker-state
      # If running on a non M1/M2 MacOS, optionally change this to guardian (default) or containerd (if cgroupsv2 is supported by the host)
      CONCOURSE_WORKER_RUNTIME: "houdini"

[tedivm]

That worked! Thank you!

[ar3666]

Hi @danlsgiga ,
This issue("find or create container on worker a1acdec9f9e6: stream image metadata file: failed to stream out from volume") still persists for me even after using the exact docker-compose of yours.
Any other workaround possible and I am using Mac M1 .

selected worker: a1acdec9f9e6
fetching busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
3f4d90098f5b [==========================================] 2.1MiB/2.1MiB

selected worker: a1acdec9f9e6

find or create container on worker a1acdec9f9e6: stream image metadata file: failed to stream out from volume

docker-compose.yml
version: '3.9'

services:
concourse-db:
image: postgres
environment:
POSTGRES_DB: &PG_DBNAME concourse
POSTGRES_PASSWORD: &PG_PASSWORD concourse_pass
POSTGRES_USER: &PG_USER concourse_user
PGDATA: /database

concourse:
image: concourse/concourse:7.9.1-ubuntu
platform: linux/amd64
command: quickstart
privileged: true
depends_on:
- concourse-db
ports:
- "8080:8080"
environment:
CONCOURSE_POSTGRES_HOST: concourse-db
CONCOURSE_POSTGRES_USER: *PG_USER
CONCOURSE_POSTGRES_PASSWORD: *PG_PASSWORD
CONCOURSE_POSTGRES_DATABASE: PG_DBNAME
CONCOURSE_EXTERNAL_URL: http://localhost:8080
CONCOURSE_ADD_LOCAL_USER: ptfe:ptfe
CONCOURSE_MAIN_TEAM_LOCAL_USER: ptfe
CONCOURSE_WORKER_SWEEP_INTERVAL: 24h
# If running on a non M1/M2 MacOS, optionally change this to overlay
CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: naive
CONCOURSE_WORKER_BAGGAGECLAIM_DISABLE_USER_NAMESPACES: true
CONCOURSE_CLIENT_SECRET: Y29uY291cnNlLXdlYgo=
CONCOURSE_TSA_CLIENT_SECRET: Y29uY291cnNlLXdvcmtlcgo=
CONCOURSE_X_FRAME_OPTIONS: allow
CONCOURSE_CONTENT_SECURITY_POLICY: ""
CONCOURSE_CLUSTER_NAME: PTFE
CONCOURSE_WORKER_WORK_DIR: /worker-state
# If running on a non M1/M2 MacOS, optionally change this to guardian (default) or containerd (if cgroupsv2 is supported by the host)
CONCOURSE_WORKER_RUNTIME: "houdini"

[gsaslis]

I am also struggling getting concourse to run jobs on an arm64 mac.

With the options:

CONCOURSE_WORKER_RUNTIME: "houdini"
CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: naive
CONCOURSE_WORKER_BAGGAGECLAIM_DISABLE_USER_NAMESPACES: true

and image:

image: concourse/concourse:7.9.1-ubuntu
platform: linux/amd64

I am getting:

selected worker: 5ab99a3c2614
running build
mount: permission denied (are you root?)
FATA[0000] failed to start buildkitd: setup cgroups: exit status 1
FATA[0000] failed to run task: exit status 1
failed

(I am on a rootful=true podman machine, and privileged: true is also declared in the docker-compose.yaml)

[gsaslis]

For additional context:

      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay
      CONCOURSE_WORKER_RUNTIME: "containerd"

and:

    platform: linux/amd64
    image: concourse/concourse:7.9-ubuntu

failed with:

concourse-concourse-1     | containerd-garden-backend exited with error: setup host network failed: create chain or flush if exists failed: running [/usr/sbin/iptables -t filter -N CONCOURSE-OPERATOR --wait]: exit status 3: iptables v1.8.7 (legacy): can't initialize iptables table `filter': iptables who? (do you need to insmod?)
concourse-concourse-1     | Perhaps iptables or your kernel needs to be upgraded.

i.e. concourse itself couldn't start.

[gsaslis]

On the other hand, I was able to get this going with:

      CONCOURSE_WORKER_RUNTIME: "containerd"
      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay

and the arm64 image for concourse:

    image: rdclda/concourse:7.9.1
    platform: linux/arm64

where no errors were reported during startup of concourse, and the above mount: permission denied (are you root?) went away! 🎉

EDIT:
some more information, for my full setup:

• podman desktop v1.2.1 (not docker)
• podman v4.5.1
• rootful=true podman machine

[Sky-Syed]

platform: linux/amd64

thank it worked for me on Mac M1 without using
platform: linux/amd64

[dove-young]

I got similar error on RHEL 8.9 when follow the quick start https://concourse-ci.org/quick-start.html. docker compose up exits with these error msg

concourse-1     | {"timestamp":"2024-03-22T03:11:07.986970373Z","level":"info","source":"quickstart","message":"quickstart.web-runner.logging-runner-exited","data":{"session":"1"}}
concourse-1     | error: Exit trace for group:
concourse-1     | worker exited with error: Exit trace for group:
concourse-1     | garden exited with error: Exit trace for group:
concourse-1     | containerd-garden-backend exited with error: setup host network failed: create chain or flush if exists failed: running [/usr/sbin/iptables -t filter -N CONCOURSE-OPERATOR --wait]: exit status 3: iptables v1.8.7 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
concourse-1     | Perhaps iptables or your kernel needs to be upgraded.

here is my compose file

version: '3'

services:
  concourse-db:
    image: postgres
    environment:
      POSTGRES_DB: concourse
      POSTGRES_PASSWORD: concourse_pass
      POSTGRES_USER: concourse_user
      PGDATA: /database

  concourse:
    image: concourse/concourse
    command: quickstart
    privileged: true
    depends_on: [concourse-db]
    ports: ["8080:8080"]
    environment:
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: concourse_user
      CONCOURSE_POSTGRES_PASSWORD: concourse_pass
      CONCOURSE_POSTGRES_DATABASE: concourse
      CONCOURSE_EXTERNAL_URL: http://localhost:8080
      CONCOURSE_ADD_LOCAL_USER: test:test
      CONCOURSE_MAIN_TEAM_LOCAL_USER: test
      # instead of relying on the default "detect"
      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay
      CONCOURSE_CLIENT_SECRET: Y29uY291cnNlLXdlYgo=
      CONCOURSE_TSA_CLIENT_SECRET: Y29uY291cnNlLXdvcmtlcgo=
      CONCOURSE_X_FRAME_OPTIONS: allow
      CONCOURSE_CONTENT_SECURITY_POLICY: "*"
      CONCOURSE_CLUSTER_NAME: tutorial
      CONCOURSE_WORKER_CONTAINERD_DNS_SERVER: "8.8.8.8"
      # For ARM-based machine, change the Concourse runtime to "houdini"
      CONCOURSE_WORKER_RUNTIME: "containerd"

[dove-young]

And I can run the command iptables -t filter -N CONCOURSE-OPERATOR --wait from command line.

/home/concourse
[root@st2-]# iptables --version
iptables v1.8.5 (nf_tables)
(instana)
/home/concourse
[root@st2-]# iptables -t filter -N CONCOURSE-OPERATOR --wait
iptables: Chain already exists.

[dove-young]

Cool. I can workaround it by running this command before docker compose up
-modprobe ip_tables modprobe iptable_filter

[taylorsilva]

The troubleshooting in here has been great. In general though you are trying to run an amd64 image on an arm machine, so it's not surprising that things aren't working. I don't think Rosetta (translation layer by Apple) was designed for this use-case.

The real solution here is for Concourse to release an arm64 image. Then all these problems would likely go away and it would Just Work™️

Until that happens these are all workarounds which basically remove all the "container stuff" when running off an M1 Mac.