Failed to get CA Certs...

[TimHeckel]

Hey all - I've asked the new KH Assistant this as well, but sadly no luck. I've terraform init -upgrade to the latest I've also been upgrading the control plane in my kube.tf file -- first adding a new 3-tier cpx21 control plane, running terraform apply, then draining the old cpx11 tier in the cluster, then setting that tier to 0 and rerunning terraform apply -- but none of the new cpx21 control plane servers are able to start k3s.

My current kube.tf control plane config looks like:

control_plane_nodepools = [
    {
      name        = "sb-control-plane",
      server_type = "cpx11",
      location    = "ash",
      labels      = [],
      taints      = [],
      count       = 0

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    {
      name        = "sb-control-plane-latest",
      server_type = "cpx21",
      location    = "ash",
      labels      = [],
      taints      = [],
      count       = 3

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    }
  ]

And terraform init hangs with this:

Error: remote-exec provisioner error
│
│   with module.kube-hetzner.null_resource.control_planes["1-0-sb-control-plane-latest"],
│   on .terraform/modules/kube-hetzner/control_planes.tf line 176, in resource "null_resource" "control_planes":
│  176:   provisioner "remote-exec" {
│
│ error executing "/tmp/terraform_1859497268.sh": Process exited with status 124

cat /tmp/terraform_1859497268.sh yields:

#!/bin/sh
systemctl start k3s 2> /dev/null
mkdir -p /var/post_install /var/user_kustomize
timeout 360 bash <<EOF
  until systemctl status k3s > /dev/null; do
    systemctl start k3s 2> /dev/null
    echo "Waiting for the k3s server to start..."
    sleep 3
  done
EOF

And journalctl -u k3s on that server yields:

systemd[1]: k3s.service: Scheduled restart job, restart counter is at 2448.
systemd[1]: Starting Lightweight Kubernetes...
sh[27823]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
k3s[27830]: time="2024-05-11T19:12:43Z" level=info msg="Starting k3s v1.29.4+k3s1 (94e29e2e)"
k3s[27830]: time="2024-05-11T19:12:43Z" level=fatal msg="starting kubernetes: preparing server: failed to get CA certs: Get \"https://10.254.0.101:6443/cacerts\": dial tcp 10.254.0.101:6443: connect: connection refused"

10.254.0.101 here refers to the private IP of another control plane node - and that node has a similar error accessing the cacerts over port 6443 from other internal control plane IPs.

I noticed that the firewall didn't have 6443 opened outbound, which is perhaps odd? It does automatically inbound. I added it manually, but it doesn't seem to matter.

Any tips on how to further diagnose this one?

Thanks!

[WebSpider]

Maybe you are bit by the limitation in deleting items from a nodepool, as described here ?

The way I read this, you should

1. Add the new control nodes in your config
2. Run terraform
3. Set the count of the old control nodes to 0 in your config
4. Run terraform
5. Live with the remaining now empty definitions of the nodes in your configfiles

[TimHeckel]

Thanks @WebSpider - I think I did follow those steps, but it turned out to be easier to just migrate to a new cluster. Thanks for the recommendiations!