Containers don't have access to internet at random

[smatting]

Summary

After upgrading from 7.9.1 to 7.11.0 we noticed that concourse spawned containers that didn't have any access to the internet: When you fly hijack into a container and try to curl google.com (142.251.36.206)

root@40bf25db-c326-4a7e-5899-b761145670e5:/tmp/build/get# curl 142.251.36.206
curl: (7) Failed to connect to 142.251.36.206 port 80 after 3054 ms: No route to host

Here some characteristics:

• only containers seem to be offline, not workers: other containers on the same worker are online
• offline containers stay offline: checked by waiting and fly hijack to test later: still offline
• happens randomly: the same step might succeed in another build
• happens independently of image/step that runs: Happens on all kinds of resource "check", but also in tasks

We run concourse inside a k8s (version v1.24.17-eks-4f4795d) cluster via helm chart 17.2.1

Triaging info

• Concourse version: 7.11.0
• Did this used to work? Yes! We never had any problems with offline containers.

[taylorsilva]

The next time that happens, can you see what the veth situation is on that container. You can do that by running ip addr in the container. If ip isn't installed you can usually install it via a package named iproute (or iproute2 for ubuntu/debian images).

Output will look something like this, taken from my Concourse instance:

/tmp/build/80754af9 # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if917998: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1460 qdisc noqueue state UP
    link/ether 9a:2e:8e:03:44:25 brd ff:ff:ff:ff:ff:ff
    inet 10.254.2.50/16 brd 10.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::982e:8eff:fe03:4425/64 scope link
       valid_lft forever preferred_lft forever