CI: Conformance EKS: Installation and Connectivity Test (1.24, ca-west-1): [check-log-errors]: failed to obtain eni link list: interrupted system call

[tommyp1ckles]

Happening fairly commonly in CI EKS pipeline.

 [=] Test [check-log-errors] [65/65]
.........
  [-] Scenario [check-log-errors/no-errors-in-logs]
  [.] Action [check-log-errors/no-errors-in-logs/cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-5x8dh (cilium-agent)]
  [.] Action [check-log-errors/no-errors-in-logs/cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-5x8dh (config)]
  [.] Action [check-log-errors/no-errors-in-logs/cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-5x8dh (mount-cgroup)]
  [.] Action [check-log-errors/no-errors-in-logs/cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-5x8dh (apply-sysctl-overwrites)]
  [.] Action [check-log-errors/no-errors-in-logs/cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-5x8dh (mount-bpf-fs)]
  [.] Action [check-log-errors/no-errors-in-logs/cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-5x8dh (clean-cilium-state)]
  [.] Action [check-log-errors/no-errors-in-logs/cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-5x8dh (install-cni-binaries)]
  [.] Action [check-log-errors/no-errors-in-logs/cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-operator-844886dbf6-pxkpv (cilium-operator)]
  [.] Action [check-log-errors/no-errors-in-logs/cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-pctck (cilium-agent)]
  ❌ Found 2 logs in cilium-cilium-8049278400-1.ca-west-1.eksctl.io/kube-system/cilium-pctck (cilium-agent) matching list of errors that must be investigated:
level=error msg="Timed out waiting for ENIs to be attached" attachedENIs="map[]" error="failed to obtain eni link list: interrupted system call" expectedENIs="map[]" subsys=ipam (1 occurrences)
level=error msg="Timed out waiting for ENIs to be attached" attachedENIs="map[]" error="failed to obtain eni link list: interrupted system call" expectedENIs="map[0e:b8:fe:46:45:dc:eni-0a4378a48ca6f8443]" subsys=ipam (1 occurrences)

Example(s):

• https://github.com/cilium/cilium/actions/runs/8049278400/job/21982337763
• https://github.com/cilium/cilium/actions/runs/8044897788/job/21969262001

Sysdump:
cilium-sysdumps(40).zip

[tommyp1ckles]

Some preliminary investigation, this is failing on netlink.LinkList(), I presume the EINTR error is coming from netlinks recvfrom on the nl socket. Not sure why this is happening, I know we've had issues with netlink requests hanging before so this could be a result of a socket timeout being exceeded?

[pchaigno]

@tommyp1ckles Are you still working on this? If not, let's try to get an assignee.

It's still happening frequently in Conformance EKS.

[jasonaliyetti]

This error is something that I'm seeing in my EKS environments when trying to upgrade to 1.15. I can file a separate issue if preferred, but wanted to mention that it had been seen in the wild. This error occurs during ENI attachment for some nodes and leads to connectivity issues. A restart of the agent seems to recover it until a new ENI is attached.

[pchaigno]

@jasonaliyetti Thanks for the heads up! I think it would indeed be best to open a separate issue. This is for the flake and may be addressed differently from an actual user issue.

[jasonaliyetti]

Just circling back...I think this gets addressed by #32099.

[tommyp1ckles]

thanks @jasonaliyetti - from what I can tell this should affect all release branches so let's wait for the PR backports to be done and we can probably close this issue.

[jasonaliyetti]

@tommyp1ckles I see CI failing on the new error log (https://github.com/cilium/cilium/actions/runs/8848559648/job/24298646382#step:23:199). Is there any way to suppress this failure for this message since it just indicates a retry occurred? I'm happy to create an MR but it'd save me some time if you could point me in the right direction.

[jasonaliyetti]

I guess here? https://github.com/cilium/cilium-cli/blob/5b1f57c4460ca30b5974fd2d5c4c7c139126bc71/connectivity/tests/errors.go#L48

[giorio94]

Hit on #32471: https://github.com/cilium/cilium/actions/runs/9035428216/job/24830199789