⚠️ Upgrade to 2.x

[mysticaltech]

You have to recreate the snapshots.

Inside your terraform folder (where your kube.tf lives), don't worry it will not affect your running cluster:

1. hcloud image list
2. hcloud image delete <run this for both MicroOS x86 and ARM>
3. rm hcloud-microos-snapshots.pkr.hcl
4. createkh, answer yes to create snapshots (this will create the snapshots)
5. terraform init -upgrade
6. terrafom apply

Then if you want the already running cluster to properly support IPv6, you will have to decomission the old nodes:

1. Double the size of your cluster, by creating one exact copy of your old nodepools for each, just modify their names as you can't have duplicated nodepools.
2. Apply, and when all new nodepools are up and running, you will have to drain and delete each node of old nodepools via kubectl. This will make all workloads shift entirely to the new nodepools.
3. When this is done, just set the old nodepool counts (for each) to 0, DON'T delete them (this is not supported, just scale to 0) and apply. Then your cluster will be again its original size, and full upgraded with full IPv6 support.

[dstockhammer]

Thanks for the update! Is the new node pool essential or is that only a way to reliably decommission the correct nodes? In other words, will new nodes in the existing node pool also use the new snapshot / have IPv6 enabled correctly?

[mysticaltech]

@dstockhammer Yes, if you replaced the underlying images (which is important to do) they will be used.

[dstockhammer]

Thanks! And just to confirm, replacing the underlying images is done by following these 6 steps including terrafom apply?

[mysticaltech]

@dstockhammer The first 4 steps will replace the image, the two others will complete the upgrade.

[dstockhammer]

Perfect, thanks for the quick reply!

[jidckii]

@mysticaltech
Many resources depend on the first_control_plane such as kustomization_backup and kubeconfig. How to update first control_plane node ?

[mysticaltech]

@jidckii It's designed to depend only during the launch of the cluster, then if you are in HA, you can even take down the first control plane completely.

[jidckii]

Resolved here #851

[mysticaltech]

If you are having issues with 2.8.x upgrades with the LB, the steps to resolve are laid out here #1028.

[mysticaltech]

📢 Starting in v2.8.3 this project needs traefik v15 and if you are upgrading you still must have previous versions of Traefik, so before attempting the upgrade of the module, you have to upgrade Traefik itself.

To do that, SSH into any control plane node (see debug section how to do that) and follow these simple steps: https://pet2cattle.com/2021/12/k3s-update-traefik-to-v2

[mysticaltech]

Starting in v2.9.x, make sure your do not have autoscaler image and version set in kube.tf, you need to let the system use the default value, we point to the correct image automatically.

[virtuman]

is there anyway to upgrade single control plane cluster if nodes need to be recreated? or will provisioning extra nodes result in failure of the cluster / or non-working nodes?

[mysticaltech]

@virtuman You can scale up and upgrade but it's delicate, as you need to move to the hetzner cloud lb manually (away from klipper), by deploying traefik again and setting enable_klipper_metal_lb to false. You can do it, but probably not without some downtime. You need to have a good understanding of the basics of kube.

In the future, we hope to make this simpler.

[virtuman]

i'm not using klipper and traefik, using nginx with direct IP for single-master setup.
if i scale up, how do i upgrade the initial first master? will it convert to multi-master during scale up, will it be able to scale back down to single-master cluster, after the upgrade? trying to save on the cluster, and keep it as a single node. if i scale up, initial node will remain the same old version, and during scale down it will continue on the same single old node, or what am i missing?

is it still safe, or too dangerous for the upgrade in place?

[mysticaltech]

@virtuman I have found out that moving from HA (at least 3 control planes) back to single control plane setup, causes etcd corruption. So once you go HA, there is no going back IMHO.

And yes, in HA, you can upgrade the all nodes, including the first control plane node.

[virtuman]

that's exactly what i was thinking, once i go up to HA, that i won't be able to scale down. i haven't had any problems with the cluster for a very long time, and it was running nice and cheap. worried that if i scale down to a lower performance node type , that even in HA, it still might start getting issues. I now have CPX31 for master, very light cluster, approximately 150-200 pods. Can someone share their experience of running a cluster on HA cpx11 nodes, and what number of pods in a lightly loaded cluster it handles nicely, or cpx11 for control plane is just not reasonable enough at all?

Thank you

[mysticaltech]

@virtuman Either ask GPT-4 or open a discussion, personally I think the control planes can be cpx11s, as it's just orchestrates everything else, but depending on your workloads agents need to be more powerful.

[yannicschroeer]

What is createkh? I am unable to follow at this point

[yannicschroeer]

Nevermind, found the instructions. For future reference:

alias createkh='tmp_script=$(mktemp) && curl -sSL -o "${tmp_script}" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh && chmod +x "${tmp_script}" && "${tmp_script}" && rm "${tmp_script}"'