New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot start containers after Docker engine v26.0.2 #19662
Comments
I've also filed an issue with moby since this seems to be related to GHSA-x84c-p2g9-rqv9 |
pinned our docker-ce and docker-ce-cli version to 26.0.1 |
Same here. I've spent several hours trying to get this working inside a systemd-nspawnd jail. The suggested workaround, using The last docker-ce / docker-ce-cli version working for me is also 26.0.1 |
Hello all - as @joeabbey says, this is related to the moby security advisory (and I'm the guilty party) ... the change is to make sure IPv6 is disabled on an interface, when only IPv4 is configured in the docker network. At the moment, dockerd always tries to write to the "/proc" file. We'll make it so that it only tries to write if it needs to change the current value. That might help in some cases. Also - when IPv6 needs to be disabled but dockerd can't do it, in the short term, we can provide a dockerd command line / daemon.json flag to restore the old behaviour, so that it just logs the problem. Then, gitpod will need to set that flag, if it's the appropriate thing to do. (Once we've made docker's IPv6 configuration simpler and more flexible, we'll remove the override and require IPv6 to be configured in the docker network when IPv6 can't be disabled on the interface.) So, a couple of questions ...
@Cellobita - with systemd-nspawnd, are you running dockerd in the jail? I guess you have a configuration that gives dockerd enough permissons to be able to create and configure networks, but blocks access to some or all of the "/proc"? |
@robmry, thank you for your reply - yes, I'm running docker inside a jail; the server is running TrueNAS SCALE (Linux-based) and the jailmaker script that a TN user put together and maintains. I did manage to get it working with |
Ah, yes - CAP_NET_ADMIN will make the /proc files writable, amongst other things - including allowing dockerd to configure iptables and its networking, which it's always needed to do. So, I'd be interested to know how that works for older moby releases - I'll take a look at the script, thank you. |
The jailmaker script does have a |
There was a discussion about this stuff, over the past few days: Jip-Hop/jailmaker#119 - some (most?) of it is way over my head, but you'll probably find it useful, @robmry |
Thank you! Is gitpod using host networking - or are these slightly different problems? (I think dockerd needs CAP_NET_ADMIN for its basic networking setup, so the gitpod problem probably isn't that it's running on a host without that capability. But, perhaps I'm missing something.) |
I'm not sure. TBH, a Google search of this issue was singularly barren, yesterday (or I did not drink enough coffee during the day) - this is the only thread I could find with the exact error I was getting, my posting here was kind of a Hail Mary... |
Sorry for the delay. We are looking into this. |
Hi @svenefftinge - to summarise from the moby side ... The problem is with an environment where "/proc/sys/net" is read-only. It'd normally be read-write on a host running dockerd, because dockerd requires I guess it'll be similar for gitpod - it'd be good to confirm, and I'd be interested to know how it's set up. The bad change in moby tries to disable IPv6 by writing to the "/proc/sys/net" file, even if it's already disabled. We'll stop it from doing that. If we can't disable IPv6, for an interface on an IPv4-only network - we'll add an environment variable override that tells dockerd to ignore the problem (requiring explicit action, because of the GHSA @joeabbey lined to above). Other options will be to explicitly enable IPv6 on the network, or make "/proc/sys/net" read-write. Those changes should be available soon, in moby 26.1.1. (Then, in a near-future moby release once we've made IPv6 easier to configure, we plan to remove support for the new env-var - and require that IPv6 is explicitly enabled on a network if dockerd can't disable it for the interface.) |
Ok, I think we will just need to make sure that we don't use 26.1 for now. |
Refer to gitpod-io/gitpod#19662 for background, this is a temporary fix From a Gitpod workspace in this repo, test with: ```bash ./build-combo.sh base ```
docker-in-docker is normally run with In current releases - if IPv6 is enabled on the host but disabled in a docker network, and dockerd can't disable it by writing the /proc file, IPv6 stays enabled on the IPv4 only interface. That's what the security advisory was about, and the reason for the change. It's a particular issue for ipvlan/macvlan docker networks, where the interface might automatically get a SLAAC assigned IPv6 address on the host's network. I guess that won't be an issue in your case (?). So, setting the environment variable before starting docker will probably be a good option. |
Refer to gitpod-io/gitpod#19662 for background, this is a temporary fix From a Gitpod workspace in this repo, test with: ```bash ./build-combo.sh base ```
👋 hey there, we're preparing a fix in gitpod-io/workspace-images#1339. It'll pin For posterity, we publish |
👋 hello again, there are a a couple workarounds that I'd like to share for the meantime:
|
Refer to gitpod-io/gitpod#19662 for background, this is a temporary fix From a Gitpod workspace in this repo, test with: ```bash ./build-combo.sh base ```
👋 The |
Bug description
Any attempts to start a container result in:
Steps to reproduce
Run:
Output:
Workspace affected
dakotaxyz-dakota-2w017zk6yxn
Expected behavior
the container starts
Example repository
No response
Anything else?
No response
The text was updated successfully, but these errors were encountered: