ICMPv6 Neighbour Solicitation packets dropped preventing IPv6 from working #9807
Answered
by
rbrtbnfgl
microsec-ambrose
asked this question in
Q&A
-
|
Environmental Info: Node(s) CPU architecture, OS, and Version: Firewall rules: Cluster Configuration:
Describe the bug: Steps To Reproduce: Installed K3s on all three nodes (with --cluster-init and --server as appropriate): I upgraded from 1.28.5+k3s1 and also did some Ubuntu package upgrades at the same time, and this issue cropped up. Expected behavior: IPv6 traffic will flow correctly. Actual behavior: IPv6 traffic is blocked: Additional context / logs: Adding the following rules unblocks traffic: |
Beta Was this translation helpful? Give feedback.
Answered by
rbrtbnfgl
Apr 10, 2024
Replies: 7 comments 14 replies
-
|
We don't block any of the traffic in question; I suspect something changed in whatever host-based firewall you are using (firewalld/ufw) and the default rules are now more restrictive. Your firewall status output does specifically show a default-deny policy for ipv4 and ipv6 on eno1, so what you're reporting sounds logical. I would recommend against adding rules to the kube-router chains; these are manage by the kube-router network policy controller and will likely be discarded when you restart k3s. This doesn't appear to be related to k3s or our embedded network policy controller, so I am going to convert it to a discussion. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Some notes on my research: Looking at the NFLOG entries, the rules created by kube-router seem to not match any rules, and thus are logged as DROP: Which means none of the rules marked the packet as ALLOW (0x10000): Likely because when selecting a specific protocol (this case UDP) in a NetworkPolicy, the ICMPv6 packets will be dropped by the rule. From what I can tell, |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I managed to resolve this connection failure by adding the NDP solicited-node address to the ipset: Now it succeeds: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
The specific case where this seems to be an issue: |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
I can confirm adding |
Beta Was this translation helpful? Give feedback.
-
|
I'm confused. Are you saying that the issue is with DNS, not Neighbour Solicitation packets? Or both? |
Beta Was this translation helpful? Give feedback.
-
|
The issues with Neighbour Solicitation packets surface to my applications as DNS resolution timeouts, however it causes most IPv6 traffic to be dropped. In summary, networking fails until I add the following Ingress and Egress rules: - ipBlock:
cidr: ff02::1:ff00:0/104
- ipBlock:
cidr: fe00::/8Which just so happens to allow NDP packets through. |
Beta Was this translation helpful? Give feedback.
-
|
I managed to reproduce it in a new cluster with the following Vagrantfile and manifests: Vagrantfile# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.vm.box = "generic/ubuntu2204"
config.vm.box_version = "4.3.12"
%w[libvirt virtualbox vmware_desktop].each do |p|
config.vm.provider p do |v|
v.cpus = 2
v.memory = 2048
end
end
k3s_env = %w[
INSTALL_K3S_VERSION=v1.28.7+k3s1
K3S_KUBECONFIG_MODE=0644
]
k3s_config = {
:token => 'vagrant-k3s',
:'flannel-iface' => 'eth1',
:'flannel-backend' => 'host-gw',
:'flannel-ipv6-masq' => 'true',
:'cluster-cidr' => '10.0.0.0/13,fd10:0000:0000:0000::/56',
:'service-cidr' => '10.128.0.0/16,fd10:0000:0000:0100::/112',
:'kube-controller-manager-arg' => %w[node-cidr-mask-size-ipv4=23 node-cidr-mask-size-ipv6=64],
:disable => %w[local-storage],
}
packages = <<-SHELL
sed -i 's,https://mirrors.edge.kernel.org/ubuntu/,http://sg.archive.ubuntu.com/ubuntu/,' /etc/apt/sources.list
export DEBIAN_FRONTEND=noninteractive
#apt-get update
#apt-get upgrade -y
#apt-get install -y nfs-common
SHELL
net_write = <<-SHELL
# Undo this Vagrant setting
sed -i 's/net.ipv6.conf.all.disable_ipv6 = 1//' /etc/sysctl.conf
sysctl net.ipv6.conf.all.disable_ipv6=0
# Set up the
cat <<EOF > /etc/netplan/20-private.yaml
---
network:
version: 2
renderer: networkd
ethernets:
eth1:
addresses:
- $1
- $2
routes:
- to: default
via: "fd10:ffff::"
EOF
netplan apply
SHELL
(1..3).each do |i|
config.vm.define "node-#{i}" do |node|
ipv4 = "192.168.6.1#{i.to_s.rjust(2, '0')}/24"
ipv6 = "fd10:ffff::10#{i.to_s(16).rjust(2, '0')}/64"
node.vm.network "private_network", ip: ipv4, auto_config: false
node.vm.provision "shell", inline: packages
node.vm.provision "shell", inline: net_write, args: [ipv4, ipv6]
node.vm.hostname = "node-#{i}"
node.vm.provision :k3s, run: "once" do |k3s|
# This section runs the installation script with a configuration file,
# environmental variables and a specific version.
k3s.args = %w[server]
k3s.env = k3s_env
k3s.config_mode = '0644'
if i == 1
k3s.config = k3s_config.merge(:'cluster-init' => 'true')
else
k3s.config = k3s_config.merge(:server => 'https://192.168.6.101:6443')
k3s.skip_start = true
end
end
# Enable bash completion
node.vm.provision "shell", inline: "kubectl completion bash > /etc/bash_completion.d/kubectl"
# Enable synced folder
#node.vm.synced_folder ".", "/manifests", type: "9p"
if i == 1
# Apply default manifest
node.vm.provision "file", source: "setup.yaml", destination: "$HOME/setup.yaml"
node.vm.provision "shell", inline: "kubectl apply -f $HOME/setup.yaml", privileged: false
else
# Wait for boot
node.vm.provision "shell", inline: "while ! curl -o /dev/null --insecure https://192.168.6.101:6443; do sleep 1; done"
# Start k3s
node.vm.provision "shell", inline: "systemctl enable --now k3s"
end
end
end
endsetup.yamlapiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: test
namespace: default
spec:
egress:
- to:
- podSelector: {}
- ipBlock:
cidr: 2000::/3 # internet
- ipBlock:
cidr: 0.0.0.0/0 # internet
- ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ingress:
- from:
- podSelector: {}
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-foo
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: test-foo
template:
metadata:
labels:
app: test-foo
spec:
containers:
- image: traefik/whoami:v1.10.1@sha256:6bebf84c091b5da4d4228bf8905436e33ca371afc6f3bd52b1682b40d76b23de
name: serve
ports:
- containerPort: 80
name: http
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: test-foo
namespace: default
spec:
ipFamilyPolicy: PreferDualStack
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
selector:
app: test-foo
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-bar
namespace: default
spec:
replicas: 10
selector:
matchLabels:
app: test-bar
template:
metadata:
labels:
app: test-bar
spec:
containers:
- image: docker.io/library/alpine:3.19
command:
- sh
- -ec
- |
apk add --no-cache wget
while true; do
wget -6O /dev/null http://test-foo && echo ping || echo fail
sleep 3
done
name: poll
- image: docker.io/library/alpine:3.19
command:
- sh
- -ec
- |
apk add --no-cache tcpdump
tcpdump -ni eth0
name: dumpIn the above setup, the pods host$ vagrant ssh node-1
vm$ kubectl logs deploy/test-bar
Found 10 pods, using pod/test-bar-8f8966964-rxpwl
Defaulted container "poll" out of: poll, dump
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/4) Installing libunistring (1.1-r2)
(2/4) Installing libidn2 (2.3.4-r4)
(3/4) Installing pcre2 (10.42-r2)
(4/4) Installing wget (1.21.4-r0)
Executing busybox-1.36.1-r15.trigger
OK: 10 MiB in 19 packages
--2024-04-01 06:59:03-- http://test-foo/
Resolving test-foo (test-foo)... fd10:0:0:100::1b3a
Connecting to test-foo (test-foo)|fd10:0:0:100::1b3a|:80... failed: Host is unreachable.
failMon Apr 1 06:59:06 UTC 2024
--2024-04-01 06:59:09-- http://test-foo/
Resolving test-foo (test-foo)... fd10:0:0:100::1b3a
Connecting to test-foo (test-foo)|fd10:0:0:100::1b3a|:80... failed: Operation timed out.
Retrying.
--2024-04-01 07:01:21-- (try: 2) http://test-foo/
Connecting to test-foo (test-foo)|fd10:0:0:100::1b3a|:80... failed: Operation timed out.
Retrying.
Checking the tcpdump logs, there are NDP NS packets being sent out from all pods, and the current pod sends out NDP NA packets (but these get dropped by the outbound firewall rules): kubectl logs deploy/test-bar -c dumpHere's the firewall rules on one node: sudo ip6tables -nvLUFW is not enabled. |
Beta Was this translation helpful? Give feedback.
-
|
@manuelbuil do you have any thoughts on this? |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
@rbrtbnfgl was poking at kube-router and ipv6 a few releases ago, maybe what the user describes rings a bell? |
Beta Was this translation helpful? Give feedback.
-
|
Before the IPv6 traffic drop policy wasn't working because the ipset was wrong, now it's working as expected. The default IPv6 accepted traffic should be fixed. If the netpol aren't used it's probably better to disable it when using IPv6 for now or add the netpol to accept that traffic. |
Beta Was this translation helpful? Give feedback.
-
|
OK so just to confirm:
|
Beta Was this translation helpful? Give feedback.
-
|
Yes. I have to check which type of traffic should be allowed by default and fix kube-router rules. |
Beta Was this translation helpful? Give feedback.
-
|
A good point of reference would be UFW and RFC4890 A.6: iptables -nvL ufw-before-inputip6tables -nvL ufw6-before-inputThe rules that enable NDP are: |
Beta Was this translation helpful? Give feedback.
-
|
I investigated a bit on this. This is clearly related to the fix for the IPset rule on IPv6. This is specific to Flannel that differently from the other CNIs creates an L2 network between the pods and to manages the L2 communications IPv6 protocol uses multicast IP packets differently from IPv4. So the possible solutions could be:
|
Beta Was this translation helpful? Give feedback.
5 replies
-
|
I would vote for 1; I don't think we should make people include the flannel-specific stuff in all their network policies. If its broken without it, we should do it ourselves. |
Beta Was this translation helpful? Give feedback.
-
|
Agree, I vote for 1 + documentation about this |
Beta Was this translation helpful? Give feedback.
-
|
Why not just automatically generate the appropriate iptables rules? If I was an admin and had to manually allow this traffic in each and every one of my policies on an ipv6 cluster I'd consider this basically unusable. |
Beta Was this translation helpful? Give feedback.
-
|
I did a PR on our kube-router fork k3s-io/kube-router#85 |
Beta Was this translation helpful? Give feedback.
Answer selected by
microsec-ambrose
-
|
Thank you so much! |
Beta Was this translation helpful? Give feedback.
-
|
For those who use firewalld:
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I did a PR on our kube-router fork k3s-io/kube-router#85