Rotating default kubeconfig

[Apoorva64]

How do i rotate the default kubeconfig found at /etc/rancher/k3s/k3s.yaml?
I've tried to rotate the node token(using k3s token rotate) and rotate the certificates using (k3s certificate rotate) but it doesn't change the kubeconfig file.

[brandond]

What specifically are you looking at to judge whether or not it's been regenerated? The file is rewritten every time k3s starts, and just contains an inlined version of the cert+key from /var/lib/rancher/k3s/server/tls/client-admin.crt and /var/lib/rancher/k3s/server/tls/client-admin.key. If you rotate these files, the admin kubeconfig should get updated as well next time you restart.

[Apoorva64]

To check if the kubeconfig has been rotated. I am connecting to the cluster with the kubeconfig extenally thus if the kubeconfig has been rotated i should not be able to connect to the cluster.

I've diffed client-admin.crt and client-admin.key with their respective backups and they have changed
i've done the diff between the old and the new generated kubeconfig file and they are indeed different.
But i can still connect with the old kubeconfig file

[brandond]

if the kubeconfig has been rotated i should not be able to connect to the cluster.

That is an incorrect assumption.

Rotating the certs does not invalidate old certificates, were you expecting it to do so? We do not make that claim anywhere.

Kubernetes does not actually support certificate revocation checks, so even if the old certs were revoked by the CA it would not make a difference as the CRL is not checked. The only way to completely revoke old certificates is to switch over to a new CA that does not have any common trust with the old CA - but this is a very disruptive operation, as you are rekeying the entire cluster.

Best practice is not to distribute the admin kubeconfig, and instead use Rancher or some other user tool to create user-specific kubeconfigs with their own RBAC that can be revoked.

[sass1997]

@brandond I just rotated ca as well as all certificates. And I'm still able to connect with the old client certificate which is signed by ca which isn't anymore on the cluster. Do you have any idea if I'm missing to delete a cache or something similiar. The CA's are self signed I don't know if this makes any difference.

[brandond]

How exactly did you rotate the CAs? Note that just renewing the CA certs while retaining the original root of trust will not invalidate old client certificates. You would need to start over with a completely new root of trust that shares nothing with the previous CAs.

[sass1997]

I have a 3 node cluster

1. I've created a new self signed certificate for server-ca client-ca and request-header-ca the etcd ca and service key I have copied from the existing installtion. Maybe Important all of those ca's don't share any intermediate or root ca as it would be described in the installation documentation.
2. Run the script k3s certificate rotate-ca --path /tmp/k3s-tls/ --force
3. Stopped all K3S Service on all clusters
4. Started K3S Service first on the cluster where I did the command on Step 2
5. Started K3S Service on other nodes as well
6. All the Certficates in /var/lib/rancher/k3s/server/tls were newly created and are based on my ca's which I've added in Step 1
7. Regenerated all Service Certifcates with first stopping k3s on 1 node then run k3s certificate rotate
8. Restart k3s service on nodes again
9. Checked with openssl s_connect that the kubeapi server cert is now generated from the before added ca -> which is the case
10. Still I'm able to connect with a kubeconfig which has the certifiacte authority data of the old ca as well as a client certficate which was created from a now not anymore existing ca.

[sass1997]

I was able to find the issue. I've only recreated the Certficate but still was using the "old" private key