etcd DNS query to host DNS #1402

mzaferyahsi · 2023-12-07T23:15:20Z

What happened?

When I've installed a fresh vcluster using the helm chart with vcluster-k8s, I noticed that my main pi-hole DNS server (outside of the cluster) is being queried for etcd nodes. I believe this is because coredns cannot resolve and therefore forwards the request to upstream server.

Logs from pi-hole

Dec  7 23:04:53 dnsmasq[206427]: reply arelon-etcd-17 is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-17.arelon-etcd-headless.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-17.arelon-etcd-headless.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-17.arelon-etcd-headless from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-17.arelon-etcd-headless is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-17.arelon-etcd-headless.vcluster-arelon from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-17.arelon-etcd-headless.vcluster-arelon is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-17.arelon-etcd-headless.vcluster-arelon from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-17.arelon-etcd-headless.vcluster-arelon is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-18.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-18.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-18.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-18.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-18 from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: forwarded arelon-etcd-18 to 10.32.0.1
Dec  7 23:04:53 dnsmasq[206427]: reply arelon-etcd-18 is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-18.arelon-etcd-headless.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-18.arelon-etcd-headless.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-18.arelon-etcd-headless.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-18.arelon-etcd-headless.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-18.arelon-etcd-headless from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-18.arelon-etcd-headless is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-18.arelon-etcd-headless.vcluster-arelon.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-18.arelon-etcd-headless.vcluster-arelon.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-18.arelon-etcd-headless.vcluster-arelon.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-18.arelon-etcd-headless.vcluster-arelon.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-18.arelon-etcd-headless.vcluster-arelon from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-18.arelon-etcd-headless.vcluster-arelon is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-19.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-19.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-19.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-19.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-19 from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: forwarded arelon-etcd-19 to 10.32.0.1
Dec  7 23:04:53 dnsmasq[206427]: reply arelon-etcd-19 is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-19.arelon-etcd-headless.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-19.arelon-etcd-headless.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-19.arelon-etcd-headless from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-19.arelon-etcd-headless is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-19.arelon-etcd-headless.vcluster-arelon.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-19.arelon-etcd-headless.vcluster-arelon.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-19.arelon-etcd-headless.vcluster-arelon from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-19.arelon-etcd-headless.vcluster-arelon is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[A] arelon-etcd-2.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-2.alroze.cloud is NXDOMAIN
Dec  7 23:04:53 dnsmasq[206427]: query[AAAA] arelon-etcd-2.alroze.cloud from 10.21.1.12
Dec  7 23:04:53 dnsmasq[206427]: cached arelon-etcd-2.alroze.cloud is NXDOMAIN

What did you expect to happen?

DNS queries to not to leak to my pi-hole.

How can we reproduce it (as minimally and precisely as possible)?

Setup pi-hole
Deploy k8s cluster with kubespray
a. Use DNS server as the pi-hole IPs
b. Use ndots:2
c. Use cluster_name: cluster.local
Deploy vcluster-k8s with following values.yaml

# Enable HA mode
enableHA: true

# Scale up syncer replicas
syncer:
  replicas: 3
  extraArgs:
    - "--tls-san=cluster.arelon.xxx.xxx"
    - "--tls-san=10.21.2.10"

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx

# Scale up etcd
etcd:
  replicas: 3

# Scale up controller manager
controller:
  replicas: 3

# Scale up api server
api:
  replicas: 3

# Scale up DNS server
coredns:
  replicas: 3

storage:
  className: longhorn

sync:
  secrets:
    enabled: true
  persistentvolumes:
    enabled: true
  volumesnapshots:
    enabled: false
  serviceaccounts:
    enabled: true
  networkpolicies:
    enabled: true
  pods:
    enabled: true
    # Sync ephemeralContainers to host cluster
    ephemeralContainers: true
    # Sync readiness gates to host cluster
    status: true

Monitor /var/logs/pihole.log

Anything else we need to know?

This issue also happens on vcluster v0.16.4

Host cluster Kubernetes version

$ kubectl version
Server Version: v1.28.3

Host cluster Kubernetes distribution

Self hosted K8s

vlcuster version

$ vcluster --version
0.18.0

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

k8s

OS and Arch

OS: Ubuntu 22.04.3 TLS
Arch: x64

The text was updated successfully, but these errors were encountered:

mzaferyahsi · 2023-12-08T20:08:59Z

It seems that the etcd nodes cannot be resolved with xxx-etcd-headless services. When I create a new service for each etcd instance, the DNS queries seem to stop.

FabianKramm · 2023-12-11T08:25:30Z

@mzaferyahsi thanks for creating this issue! I'm not sure why this is not working for you, but I don't think we should create a separate service for each replica, maybe we only need to adjust the certs itself and its enough already.

mzaferyahsi · 2023-12-11T10:48:58Z

@FabianKramm indeed that is also another solution. But the requirement on limiting the number of etcd SANs still stays in place. Therefore, I still recommend passing the number of etcd replicas as parameter for the syncer deployment. I've just tested by adjusting the etcd sans as below and it seems to be okay.

	for i := 0; i < etcdReplicaCount; i++ {
		if etcdEmbedded {
			// this is for embedded etcd
			hostname := vClusterName + "-" + strconv.Itoa(i)
			etcdSans = append(etcdSans, hostname, hostname+"."+vClusterName+"-headless", hostname+"."+vClusterName+"-headless"+"."+currentNamespace)
		} else {
			// this is for external etcd
			etcdHostname := etcdService + "-" + strconv.Itoa(i)
			// etcdSans = append(etcdSans, etcdHostname, etcdHostname+"."+etcdService+"-headless", etcdHostname+"."+etcdService+"-headless"+"."+currentNamespace)
			etcdSans = append(etcdSans, etcdHostname+"."+etcdService+"-headless"+"."+currentNamespace)
		}
	}

Shall I apply the same logic to the embedded?

FabianKramm · 2024-01-04T15:53:42Z

@mzaferyahsi we are doing some refactoring for this now, but can add that later when the refactoring is done yeah

mzaferyahsi added the kind/bug label Dec 7, 2023

This was referenced Dec 8, 2023

etcd queries for nodes that doesn't exist due to the SANs on etcd-server.crt #1403

Closed

Added individual service for etcd replicas #1404

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcd DNS query to host DNS #1402

etcd DNS query to host DNS #1402

mzaferyahsi commented Dec 7, 2023

mzaferyahsi commented Dec 8, 2023

FabianKramm commented Dec 11, 2023

mzaferyahsi commented Dec 11, 2023

FabianKramm commented Jan 4, 2024

etcd DNS query to host DNS #1402

etcd DNS query to host DNS #1402

Comments

mzaferyahsi commented Dec 7, 2023

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Host cluster Kubernetes version

Host cluster Kubernetes distribution

vlcuster version

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

OS and Arch

mzaferyahsi commented Dec 8, 2023

FabianKramm commented Dec 11, 2023

mzaferyahsi commented Dec 11, 2023

FabianKramm commented Jan 4, 2024