remove k3s node by node annotation?

[mogliang]

Is your feature request related to a problem? Please describe.
We are developing k3s controlplane/bootstrap provider for ClusterAPI, and we are not sure what's the right way to remove a node from k3s cluster. I know there is a uninstall script on host vm, but there is no way to invoke that script from kubeconfig cluster connection.

ClusterAPI management cluster can only access workload cluster by the kubeconfig, I found out that there is way to add annotation "etcd.k3s.cattle.io/remove" to remove etcd member, i want to know if there is such annotation as well to remove node?

Thanks,

[brandond]

That is for removing the node from the etcd cluster without deleting it. If you just want it gone, use kubectl delete node. Note that it will attempt to rejoin if you restart the service though.

None of these will uninstall k3s though. If you want k3s to stop and uninstall, that requires external action. Rancher uses a system agent that run on the node as a separate service to handle this, as well as installation, configuration, and snapshot management.

[nasusoba]

If kubelet is cut off, how could we drain node?

[mogliang]

@brandond Is there way to only remove etcd, and keep kubelet still serving?
The problem we are facing is, once remove etcd by that annotation, kubelet seems stopped as well, and then drain/ detech volume operation on capi machine controller will get failed.

Although we can drain and detech volume first and then set that etcd remove annotation, but that may require duplicate capi machine controller code to k3s controlplane, which looks not tidy.

[brandond]

once remove etcd by that annotation, kubelet seems stopped as well

That is exactly what I was warning about:

Note that once you do that, you can't do much else with it because etcd shuts down, which cuts off the local apiserver, kubelet, and everything else.

Kubelet requires local apiserver requires local etcd. You should do do all your draining and such first, then remove the node from the cluster using the annotation. It is basically the last thing that node will ever do, absent some external process.

As I also mentioned earlier, this is why Rancher has a system agent on each node that runs independent of the cluster the node is a part of.

[mogliang]

Thanks @brandond for the help, i'll go discover if there is hook to use for machine deletion.

[jobcespedes]

I was looking for something similar as a way to delete a node (agent/worker) in a shutdown systemd unit in the same node. However agent/worker does not have permissions to delete itself as node from the cluster. Then thought about a the possibility to add a label/annotation to itself and have a controller with proper permission remove the labeled/annotated node. I am not sure it is possible yet.