Discuss: work around SecurityRuleAddressesOrPortsPerSecurityGroupLimitReached when service floating ip disabled

[ArchangelSDY]

I'd like to discuss potential solutions for issue #2725 .

The background is we have a large cluster with > 500 nodes and > 30 services. These services have floating ip disabled, so in a NSG rule all backend node IPs are listed as dest addresses. However, NSG has a limit that sum of IPs among all rules must not exceed 4000. In above cluster the value is 500 * 30 = 15000, exceeding the limit a lot.

I don't think NSG has plan to increase such limit. So the only way in my mind is that maybe we can provide an annotation to let user explicitly specify NSG dest addresses(probably a CIDR list), instead of using all backend node IPs.

For example:

"service.beta.kubernetes.io/azure-disable-load-balancer-floating-ip-nsg-dest": "10.1.0.0/16,10.2.0.0/16"

Then we create a NSG rule with destinationAddressPrefix set to 10.1.0.0/16,10.2.0.0/16. It would only count as 2 to the limit.

Thoughts? I'm happy to submit a PR if you're ok with the change.