Current extended-base (quay.io/bioconda/base-glibc-debian-bash:3.0) for container builds yields docker images unusable on some systems

[pontus]

Since docker 25.0.0 (https://docs.docker.com/engine/release-notes/25.0/#2500),

• Unpacking layers with extended attributes onto an incompatible filesystem will now fail instead of silently discarding extended attributes. moby/moby#45464

The current image used as extended-base for container images unfortunately has one such extended attribute (user.overlay.impure) set for /etc/ (set to y).

This breaks various setups. In some cases it can be worked around (e.g. changing docker storage driver to overlay-fuse instead of overlay2), but it causes problems for users (and some situations, e.g. gitpod, make it quite difficult).

Since I can't this being actually used anywhere, maybe we can use a base image without such an extended attribute?

[daler]

@pontus the base image is built using this Dockerfile. We're in the middle of moving the image-building over into this repo (bioconda-utils), but that dockerfile is not changing.

The building itself is happening here with podman/buildah. That is currently being unified with other image building to happen here in bioconda-utils, but the commands are not changing.

It's not immediately clear to me where the issue is being introduced, whether in the dockerfile, buildah commands, or later on when creating containers with mulled/involucro. Based on the links above is there anything you see that might be causing the issue?

[pontus]

Thanks, I don't see anything that sticks out to me and can't directly replicate this when I try locally, I'll try to play a bit with that in github actions and see if I can trigger it.

[ChristopherHX]

My image has been hit by the same problem..., using buildah

I patched the source code of buildah here and the xattr + pull error is gone.

here you can find how I replaced buildah with one build from source in GitHub Actions Hosted Runners.

I have opened a PR against containers/storage, waiting for feedback

[ewels]

Sounds promising, thanks @ChristopherHX! Link to PR for others: containers/storage#1847 (took me a while to find because I didn't realise that containers/storage was an actual repo name at first 😅 )

[pontus]

Hrrm, I haven't yet been able to trigger this issue with buildah, and the commit there is large enough that I can't read it and figure out how to trigger it either.

[ChristopherHX]

Hrrm, I haven't yet been able to trigger this issue with buildah

See containers/storage#1847 for how to reproduce this by pulling an image built by buildah using latest docker:dind-rootless image.
My host system can apply that xattr without an error using a rootful docker 25.x version.

I believe docker buildx doesn't store extended attributes (xattr) in container layers.
Images built with buildah package it if a run step somehow contributes that xattr to the image.

and the commit there is large

I'm just adding

&& !strings.HasPrefix(key, "user.overlay.")

to an if condition in the code that creates an archive of layers built by buildah

The large diff is due to an implicit bump of the patched dependency, since I didn't patch the old revision. See containers/storage#1847 for the actual diff

I never had to deal with any xattr before so I have no idea of implications

[ChristopherHX]

My change seems to got an approval so it might be only a matter of time to get into buildah source tree

[pontus]

See containers/storage#1847 for how to reproduce this by pulling an image built by buildah using latest docker:dind-rootless image. My host system can apply that xattr without an error using a rootful docker 25.x version.

Yes, I've seen the issue, I just haven't been able to trigger the issue (being the built image having a that extended attribute somewhere) myself with buildah.

[ChristopherHX]

These steps for old buildah releases (like buildah version 1.23.1 (image-spec 1.0.1, runtime-spec 1.0.2-dev) as of ubuntu 22.04 like GitHub Hosted Runners) are enough to have the xattr in several places, tar command for verify, no need for docker

buildah from ubuntu:latest
buildah run ubuntu-working-container apt-get update
buildah run ubuntu-working-container apt-get install -y xattr
buildah run ubuntu-working-container sh -c "echo "TEST" > /etc/test.txt"
buildah commit ubuntu-working-container
buildah images
buildah push <shafromimages> oci:layout:test:test
tar --xattrs --xattrs-include='*' -tvvf layout/blobs/sha256/<shaoflayer-added-by-buildah>

Output of tar

drwxr-xr-x* root/root         0 2024-03-01 17:14 etc/
  x: 0 user.overlay.origin
-rw-r--r--  root/root      1748 2022-01-06 17:26 etc/inputrc
-rw-r--r--  root/root      5991 2024-03-01 17:09 etc/ld.so.cache
-rw-r--r--  root/root     72029 2022-03-21 10:12 etc/mime.types
drwxr-xr-x  root/root         0 2024-03-01 17:09 etc/python3/
-rw-r--r--  root/root        94 2024-03-01 17:09 etc/python3/debian_config
drwxr-xr-x  root/root         0 2024-03-01 17:09 etc/python3.10/
-rw-r--r--  root/root       155 2023-11-20 16:14 etc/python3.10/sitecustomize.py
-rw-r--r--  root/root         5 2024-03-01 17:14 etc/test.txt
drwxr-xr-x* root/root         0 2024-03-01 17:08 run/
  x: 0 user.overlay.origin
-rwxr-xr-x  root/root         0 2024-03-01 17:08 run/.containerenv
drwxrwxrwt* root/root         0 2024-03-01 17:09 tmp/
  x: 0 user.overlay.origin
drwxr-xr-x* root/root         0 2024-02-12 15:02 usr/
  x: 1 user.overlay.impure
  x: 0 user.overlay.origin
drwxr-xr-x* root/root         0 2024-03-01 17:09 usr/bin/
  x: 0 user.overlay.origin
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/pdb3 -> pdb3.10
lrwxrwxrwx  root/root         0 2023-11-20 16:14 usr/bin/pdb3.10 -> ../lib/python3.10/pdb.py
-rwxr-xr-x  root/root      7813 2022-08-18 12:39 usr/bin/py3clean
-rwxr-xr-x  root/root     13191 2022-08-18 12:39 usr/bin/py3compile
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/py3versions -> ../share/python3/py3versions.py
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/pydoc3 -> pydoc3.10
-rwxr-xr-x  root/root        79 2023-11-20 16:14 usr/bin/pydoc3.10
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/pygettext3 -> pygettext3.10
-rwxr-xr-x  root/root     24235 2023-06-07 00:30 usr/bin/pygettext3.10
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/python3 -> python3.10
-rwxr-xr-x  root/root   5904904 2023-11-20 16:14 usr/bin/python3.10
-rwxr-xr-x  root/root       948 2022-03-25 10:51 usr/bin/xattr
drwxr-xr-x* root/root         0 2024-03-01 17:09 usr/lib/
  x: 1 user.overlay.impure
  x: 0 user.overlay.origin

I just haven't been able to trigger the issue

buildah created a layer with changed files for you? Even an apt install is enough to have many of them... (rootless mode)

My own affected image has been built with docker buildx till 2023-08-01, all these images have no such xattr at all.

Good news, my fix/workaround has been merged in buildah see containers/buildah@662908f
Bad news, ubuntu 22.04 will never pick up this fix.

I wonder if I should create an GitHub Action for setup-buildah from source code and cache the built artifacts via actions/cache, found previously another bug caused by outdated buildah.

[pontus]

Thanks. I haven't managed to produce anything with these xattrs (and the above example does not) with Ubuntu 23.10 and buildah installed from standard sources with apt.