Multiple kubernetes clusters in same resource group, nodemanager is confused & read VMs from vmss of other clusters which are present in same resource group

[ajgupta42]

What happened:

Running 2 kubernetes clusters in same resource group.
Having Instance Groups/VMSS in both clusters
nodes.
Example:
Cluster1 VMSS:
nodes.dev01.abc.com
Cluster2 VMSS:
nodes.dev02.abc.com

Let's say nodes.dev01.abc.com VMSS has scaled up the node named node00002
But node manager of cluster1, don't know which vmss it should be looking for, it sees that nodes.dev02.abc.com doesn't have node00002, so it deletes it from the cluster.
So, node00002 never joins the cluster.

What you expected to happen:

cloud-provider-azure should have configuration which have tags, so that nodeManager only looks for the VMSS matching those tags, not all the VMSS belonging to the Azure resource group.
cluster1 nodeManager shouldn't access cluster2 VMSS in same resource group.

How to reproduce it (as minimally and precisely as possible):

Spin up 2 kubernetes clusters in the same Azure resource group.
Create VMSS/instance groups in both clusters
Example:
Cluster1 VMSS:
nodes.dev01.abc.com
Cluster2 VMSS:
nodes.dev02.abc.com

Run cloud-provider-azure on both clusters, try to scale up both VMSS.

Anything else we need to know?:

Only happens while running multiple clusters in same resource group.
It doesn't have a filter, so it tries to read all VMSS from Azure resource group

Environment:

• Kubernetes version (use kubectl version): Client Version: v1.29.3, Server Version: v1.28.7
• Cloud provider or hardware configuration: cloud-provider-azure
• OS (e.g: cat /etc/os-release): ubuntu 20.04
• Install tools: KOPS
• Network plugin and version (if this is a network-related bug): cilium