Clarify --cluster-init vs --cluster-reset vs nothing at all

[intelfx]

Hi!

I'm currently developing some automation for management of a k3s cluster and it's not exactly clear to me how the k3s cluster is supposed to be managed beyond initial creation and adding nodes.

Searching through issues on "adding nodes" and "removing nodes" gives mentions of --cluster-init and --cluster-reset flags but it is not clear to me what is their effect and when I should use them (vs. each other and vs. nothing at all).

As far as I could understand, --cluster-init instructs k3s to use the HA etcd datastore instead of the inherently single-node SQLite datastore (thus it gives the ability to potentially use a multi-master setup), or migrate to the former from the latter, and once the HA datastore was initialized, it is not used (and as such harmless to just leave it there).

But: what is its relationship with --cluster-reset? When is it used, and do I need to drop --cluster-init when --cluster-reset is used?

Or, in general: what are the procedures for adding/removing/replacing 1) a master node or 2) an agent node?

[brandond]

--cluster-init tells the server to initialize a new etcd cluster. If there are existing etcd datastore files on disk, this flag is ignored, as the cluster has already been initialized.

--cluster-reset performs a minimal start up of the server and resets the etcd cluster membership to contain only the current node. It starts up etcd with the force-new-cluster option (see https://etcd.io/docs/v3.5/op-guide/configuration/), which retains the data in the key-value store but removes all other members from the cluster. It then halts, so you can restart the server for normal operation. When resetting the etcd cluster you can optionally also restore from a snapshot. Resetting is mandatory when restoring, as otherwise the restored snapshot would include all the members that were in the cluster at the time the snapshot was taken, some or all of which may no longer exist.

If you want to "replace" a server, you should add the new server to the cluster, and then kubectl delete the old one before stopping it, to ensure that it is cleanly removed from the etcd cluster so as to retain quorum.
If you want to replace an agent, order doesn't really matter. Agents are essentially disposable, as they contain no cluster state.

[intelfx]

Ah, I see, thanks. So it is not necessary to --cluster-reset when removing a server, given that it is reachable.

How do I handle a failed node? Is it necessary to perform a cluster-reset then, or can I still do a kubectl delete (potentially with some else on top)?

[brandond]

If you still have quorum and the cluster is running, then just use kubectl to delete it from the cluster, regardless of whether the node is reachable or not. Deleting the node will trigger a delete of the cluster member.

If you no longer have quorum and etcd is refusing to start due to insufficient cluster members, then you would need to reset the cluster and rejoin the other servers.