calico-apiserver ServiceAccount is used by other services unexpectedly

[hadi2f244]

ServiceAccounts in kubernetes like kube-controller-manager SA is replaced by system:serviceaccount:calico-apiserver:calico-apiserver if calico-apiserver is installed.

Expected Behavior

Each service uses its own service account!

Current Behavior

Possible Solution

As we check this issue observed in the following Calico Versions:

• 3.26.X
• 3.27.X
  But it works great on Calico v3.25.2.
  Also If you uninstall calico api-server, everything works great.

Steps to Reproduce (for bugs)

1. Install calico >= 3.26.X
2. Install calico api-server
3. Check services like Argocd and Kube-controller-manager logs to see error like the following
   Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: connection is unauthorized: bgpfilters.crd.projectcalico.org is forbidden: User "system:serviceaccount:calico-apiserver:calico-apiserver" cannot list resource "bgpfilters" in API group "crd.projectcalico.org" at the cluster scope

Context

Related Issues:

• kube-controller-manager unable to perform normal GC  #7598
• https://stackoverflow.com/questions/76596545/argocd-cant-sync-application-user-systemserviceaccountcalico-apiservercali
• Enabling Calico API Server on Kubespray 2.24 causes Kubernetes cascade delete to fail kubernetes-sigs/kubespray#10949
• https://discuss.kubernetes.io/t/when-kubernetes-deployment-is-deleted-replicasets-are-not-being-deleted/24625

Your Environment

• Calico version >= 3.26.X
• kubernetes == 1.28.6
• Operating System and version: Ubuntu 22.04 & Ubuntu 20.04
• Kubespray v2.24.1

[caseydavenport]

I don't quite understand - are you suggesting that other components are using Calico's service accounts? If so, what evidence do you have?

Calico doesn't touch those components at all. I see you also linked to a number of other issues and it's not obvious to me how they are related to this at all.

Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: connection is unauthorized: bgpfilters.crd.projectcalico.org is forbidden: User "system:serviceaccount:calico-apiserver:calico-apiserver" cannot list resource "bgpfilters" in API group "crd.projectcalico.org" at the cluster scope

This just sounds like the Calico API server has been installed incorrectly and hasn't been given the permissions that it needs in order to operate. Likely a problem with the way you have installed Calico.

[hadi2f244]

This is the exact strange things happen when I installed calico-apiserver newer than v3.26.x ! Some critical component like kubernetes controller-manager uses the calico-apiserver SA.

I completely uninstalled calico and uses v3.26.0 and APIServer manifest and right after installing calico-apiserver I saw that some components such as argocd and kube-controller-manager uses calico-apiserver SA. I know that it is strange and confusing how it is possible that kube-controller-manager uses calico-apiserver SA, but it happens in different scenarios. Right after I downgraded to v3.25.2, it works normally.
Maybe it is not on the calico side manifests and the problem is related to kubespray but as I checked the related kubespray manifests (1 and 2) and they are similar to calico official manifests.

[caseydavenport]

What evidence do you have that other components are using Calico's service accounts? I don't see any evidence of it in this issue so far.

[hadi2f244]

This is what happened for me :

kubernetes-sigs/kubespray#10949

#7598

[caseydavenport]

Those are both fixed issues.

If you're encountering those issues, then it sounds like your RBAC is wrong. The linked issue has this clear statement in it:

This can be fixed by updating the calico-crds ClusterRole to add the resource bgpfilters to the list of resources