support unpacked images with docker/podman runners

[tomeichlersmith]

Is your feature request related to a problem? Please describe.
Unpacked images are supported natively by singularity/apptainer but I'd like to also support them with docker/podman so that a user's workflow doesn't need to change between machines.

Describe the solution you'd like
Something within denv that would allow docker/podman to load these unpacked images (perhaps docker load?). This would then allow users to have CVMFS on their laptops with docker/podman and use the same commands they would use on a HPC with apptainer/singularity.

[tomeichlersmith]

When blindly running a path via these unpacked images, I get a invalid reference format error.

ERRO[0000] cannot find mappings for user teichler: No subuid ranges found for user "teichler" in /etc/subuid
Error: unable to pull /cvmfs/singularity.opensciencegrid.org/cwinpy/cwinpy-containers/cwinpy-dev-python38:latest: error getting default registries to try: invalid reference format

CVMFS points out that updating podman's image store is necessary: https://cvmfs.readthedocs.io/en/stable/cpt-containers.html#podman-integration-pre-production
These instructions are labeled "pre production" so I am not surprised that I see an error when trying it out.

# write a new ~/.config/containers/storage.conf that has the same content as in CVMFS docs
$ podman images
Error: error retrieving label for image "c98db043bed913c5a7b59534cbf8d976122f98b75cb00baabf8af888041e4f9d": you may need to remove the image to resolve the error: error locating item named "sha256:3f57d9401f8d42f986df300f0c69192fc41da28ccc8d797829467780db3dd741" for image with ID "c98db043bed913c5a7b59534cbf8d976122f98b75cb00baabf8af888041e4f9d" (consider removing the image to resolve the issue): file does not exist   

[tomeichlersmith]

I was excited to see on the docs the possibility of using CVMFS as another image store for podman. With this in mind (and hoping to learn a bit more myself), I wanted to report an error I am seeing when attempting to use the test image inside the podmanStore on CVFMS.

I am testing at SLAC S3DF and I am not an admin so I may not be able to answer all clarification questions.

The ~/.config/containers/storage.conf file did not exist previously, so I created it with the contents provided in the docs.

[storage]
driver = "overlay"

[storage.options]
additionalimagestores = [ "/cvmfs/unpacked.cern.ch/podmanStore" ]
# mount_program = "/usr/bin/fuse-overlayfs"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

Then I get the following error

$ podman image ls
ERRO[0000] cannot find UID/GID for user eichl008: No subuid ranges found for user "eichl008" in /etc/subuid - check rootless mode in man pages.
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user
Error: error retrieving label for image "c98db043bed913c5a7b59534cbf8d976122f98b75cb00baabf8af888041e4f9d": you may need to remove the image to resolve the error: error locating item named "sha256:3f57d9401f8d42f986df300f0c69192fc41da28ccc8d797829467780db3dd741" for image with ID "c98db043bed913c5a7b59534cbf8d976122f98b75cb00baabf8af888041e4f9d" (consider removing the image to resolve the issue): file does not exist

The first two messages appear like an issue with the podman installation/configuration, so I ran a test with an image manually downloaded into my own image store.

$ mv ~/.config/containers/storage.conf{,.cvmfs}
$ podman run --rm hello-world
!... Hello Podman World ...!

         .--"--.           
       / -     - \         
      / (O)   (O) \        
   ~~~| -=(,Y,)=- |         
    .---. /`  \   |~~      
 ~/  o  o \~~~~.----. ~~   
  | =(X)= |~  / (O (O) \   
   ~~~~~~~  ~| =(Y_)=-  |   
  ~~~~    ~~~|   U      |~~ 

Project:   https://github.com/containers/podman
Website:   https://podman.io
Desktop:   https://podman-desktop.io
Documents: https://docs.podman.io
YouTube:   https://youtube.com/@Podman
X/Twitter: @Podman_io
Mastodon:  @[email protected]
$ podman image ls
REPOSITORY            TAG         IMAGE ID      CREATED     SIZE
quay.io/podman/hello  latest      b1c06f48960c  4 days ago  1.7 MB

However, when I try to run the same image that is served in CVMFS's podman store, I get an error seemingly related to the other errors I saw before.

$ podman pull busybox:latest
Resolved "busybox" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob 7b2699543f22 done  
Error: writing blob: adding layer with blob "sha256:7b2699543f22d5b8dc8d66a5873eb246767bca37232dee1e7a3b8c9956bceb0c": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /home: invalid argument
$ podman image ls
REPOSITORY            TAG         IMAGE ID      CREATED     SIZE
quay.io/podman/hello  latest      b1c06f48960c  4 days ago  1.7 MB

So perhaps this is an issue with SLAC's configuration of podman, but I find it suspect that the actual error reported by podman when attempting to read CVMFS's image store is a missing file error. Maybe this points to an issue with CVMFS's image store format?

Context

$ podman --version
podman version 4.1.1
$ cvmfs2 --version
CernVM-FS version 2.11.2
$ apptainer --version
apptainer version 1.2.5-1.el8

[tomeichlersmith]

https://docs.docker.com/reference/cli/docker/image/import/#import-from-a-local-directory gives me hope that we could at least copy an image from CVMFS into a docker/podman image format which can then be run.