Creating Service type LoadBalancer does not work with "disableAzureStackCloud": false configured

[invidian]

What happened:

When creating a simple service type load balancer like:

    serviceTemplate := corev1.Service{
      ObjectMeta: metav1.ObjectMeta{
        GenerateName: "test-",
      },
      Spec: corev1.ServiceSpec{
        Type: corev1.ServiceTypeLoadBalancer,
        Ports: []corev1.ServicePort{
          {
            Name:     "http",
            Port:     80,
            Protocol: corev1.ProtocolTCP,
          },
        },
        Selector: map[string]string{
          "app": "test",
        },
      },
    }

With CCM version running v1.28.4 and configured with "disableAzureStackCloud": false, CCM produces following error in logs:

cloud-controller-manager-j9cxz cloud-controller-manager E0112 14:47:49.359522       1 controller.go:298] error processing service default/test-nlzlt (retrying with exponential backoff): failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
cloud-controller-manager-j9cxz cloud-controller-manager   "error": {
cloud-controller-manager-j9cxz cloud-controller-manager     "code": "ProbeThresholdCannotBeSpecifiedForApiVersion",
cloud-controller-manager-j9cxz cloud-controller-manager     "message": "Specified api-version 2018-11-01 does not meet the minimum required api-version 2022-01-01 to specify ProbeThreshold value for probe /subscriptions/x/resourceGroups/x/providers/Microsoft.Network/loadBalancers/x/probes/a4eff35067e924fdab52af356449c979-TCP-80.",
cloud-controller-manager-j9cxz cloud-controller-manager     "details": []
cloud-controller-manager-j9cxz cloud-controller-manager   }
cloud-controller-manager-j9cxz cloud-controller-manager }
cloud-controller-manager-j9cxz cloud-controller-manager I0112 14:47:49.359658       1 event.go:307] "Event occurred" object="default/test-nlzlt" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message=<
cloud-controller-manager-j9cxz cloud-controller-manager         Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
cloud-controller-manager-j9cxz cloud-controller-manager           "error": {
cloud-controller-manager-j9cxz cloud-controller-manager             "code": "ProbeThresholdCannotBeSpecifiedForApiVersion",
cloud-controller-manager-j9cxz cloud-controller-manager             "message": "Specified api-version 2018-11-01 does not meet the minimum required api-version 2022-01-01 to specify ProbeThreshold value for probe /subscriptions/x/resourceGroups/x/providers/Microsoft.Network/loadBalancers/x/probes/a4eff35067e924fdab52af356449c979-TCP-80.",
cloud-controller-manager-j9cxz cloud-controller-manager             "details": []
cloud-controller-manager-j9cxz cloud-controller-manager           }
cloud-controller-manager-j9cxz cloud-controller-manager         }
cloud-controller-manager-j9cxz cloud-controller-manager  >

This seems to be caused by hardcoding API version for azure stack cloud, here:

cloud-provider-azure/pkg/azureclients/loadbalancerclient/interface.go

Line 31 in 9e9079e

AzureStackCloudAPIVersion = "2018-11-01"

What you expected to happen:

Service type load balancer to properly get IP address assigned.

How to reproduce it (as minimally and precisely as possible):

Deploy CCM with:

• azurestackcloud.json file pointing to public cloud:

{
  "name": "AzurePublicCloud",
  "managementPortalURL": "https://manage.windowsazure.com/",
  "publishSettingsURL": "https://manage.windowsazure.com/publishsettings/index",
  "serviceManagementEndpoint": "https://management.core.windows.net/",
  "resourceManagerEndpoint": "https://management.azure.com/",
  "activeDirectoryEndpoint": "https://login.microsoftonline.com/",
  "galleryEndpoint": "https://gallery.azure.com/",
  "keyVaultEndpoint": "https://vault.azure.net/",
  "managedHSMEndpoint": "https://managedhsm.azure.net/",
  "graphEndpoint": "https://graph.windows.net/",
  "serviceBusEndpoint": "https://servicebus.windows.net/",
  "batchManagementEndpoint": "https://batch.core.windows.net/",
  "microsoftGraphEndpoint": "https://graph.microsoft.com/",
  "storageEndpointSuffix": "core.windows.net",
  "cosmosDBDNSSuffix": "documents.azure.com",
  "mariaDBDNSSuffix": "mariadb.database.azure.com",
  "mySqlDatabaseDNSSuffix": "mysql.database.azure.com",
  "postgresqlDatabaseDNSSuffix": "postgres.database.azure.com",
  "sqlDatabaseDNSSuffix": "database.windows.net",
  "trafficManagerDNSSuffix": "trafficmanager.net",
  "keyVaultDNSSuffix": "vault.azure.net",
  "managedHSMDNSSuffix": "managedhsm.azure.net",
  "serviceBusEndpointSuffix": "servicebus.windows.net",
  "serviceManagementVMDNSSuffix": "cloudapp.net",
  "resourceManagerVMDNSSuffix": "cloudapp.azure.com",
  "containerRegistryDNSSuffix": "azurecr.io",
  "tokenAudience": "https://management.azure.com/",
  "apiManagementHostNameSuffix": "azure-api.net",
  "synapseEndpointSuffix": "dev.azuresynapse.net",
  "datalakeSuffix": "azuredatalakestore.net",
  "resourceIdentifiers": {
    "graph": "https://graph.windows.net/",
    "keyVault": "https://vault.azure.net",
    "datalake": "https://datalake.azure.net/",
    "batch": "https://batch.core.windows.net/",
    "operationalInsights": "https://api.loganalytics.io",
    "ossRDBMS": "https://ossrdbms-aad.database.windows.net",
    "storage": "https://storage.azure.com/",
    "synapse": "https://dev.azuresynapse.net",
    "serviceBus": "https://servicebus.azure.net/",
    "sqlDatabase": "https://database.windows.net/",
    "cosmosDB": "https://cosmos.azure.com",
    "managedHSM": "https://managedhsm.azure.net",
    "microsoftGraph": "https://graph.microsoft.com/"
  }

• configure CCM to use "cloud": "AzureStackCloud", and "disableAzureStackCloud": false.
• Make CCM use azurestackcloud.json file by adding AZURE_ENVIRONMENT_FILEPATH=azurestackcloud.json environment variable, pointing to the right file.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten