extensions lib: Consider dropping EnsureKubeAPIServerService as it is no longer required after ManagedIstio/APIServerSNI is unconditionally enabled

[ialidzhikov]

How to categorize this issue?

/area scalability
/kind cleanup

What would you like to be added:
Looking into #9020 and the generic Mutator I see that the EnsureKubeAPIServerService func from the Ensurer interface is not implemented for any of the provider extensions under github.com/gardener.

This is the only mutation for an object of kind Service:

gardener/extensions/pkg/webhook/controlplane/genericmutator/mutator.go

Lines 128 to 143 in 10d7578

	switch x := new.(type) {
	case *corev1.Service:
	switch x.Name {
	case v1beta1constants.DeploymentNameKubeAPIServer:
	var oldSvc *corev1.Service
	if old != nil {
	var ok bool
	oldSvc, ok = old.(*corev1.Service)
	if !ok {
	return errors.New("could not cast old object to corev1.Service")
	}
	}
	extensionswebhook.LogMutation(m.logger, x.Kind, x.Namespace, x.Name)
	return m.ensurer.EnsureKubeAPIServerService(ctx, gctx, x, oldSvc)
	}

In the times before ManagedIstio/APIServerSNI when the kube-apiserver Service was of type LoadBalancer this extension point was used to add load balancer specific annotations to the kube-apiserver Service. Examples from the past:

• provider-aws: https://github.com/gardener/gardener-extension-provider-aws/blob/a5042666e1819ea626e01c21038f93315469f9be/pkg/webhook/controlplaneexposure/ensurer.go#L47-L65
• provider-azure: https://github.com/gardener/gardener-extension-provider-azure/blob/26b0daf4968468bdf111ec2b6fdffdd93a63d2e2/pkg/webhook/controlplaneexposure/ensurer.go#L58-L70

After ManagedIstio/APIServerSNI are unconditionally enabled, these funcs are removed.
Today, these extension webhooks mutate Services but actually don't do anything in this mutation.
For example for provider-aws the controlplaneexposure webhook looks like:

  name: controlplaneexposure.aws.extensions.gardener.cloud
  namespaceSelector:
    matchExpressions:
    - key: seed.gardener.cloud/provider
      operator: In
      values:
      - aws
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - services
    scope: '*'
  - apiGroups:
    - apps
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - deployments
    scope: '*'
  - apiGroups:
    - druid.gardener.cloud
    apiVersions:
    - v1alpha1
    operations:
    - CREATE
    - UPDATE
    resources:
    - etcds
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

while the webhook only needs to mutate ETCDs: https://github.com/gardener/gardener-extension-provider-aws/blob/aaf1ad53dfb45a14c63662f36611bb239f32bc2d/pkg/webhook/controlplaneexposure/ensurer.go#L35-L52

This means that every Service/Deployment create/update operation for Service is intercepted by the provider-aws webhook and the provider-aws webhook does not perform any mutation these Services/Deployments.
In cases where the ManagedSeed control plane and data plane are located in different continents, the latency of such webhook calls can be >100ms.

Istio Service annotations can be applied via the Seed spec:

gardener/example/50-seed.yaml

Lines 69 to 77 in 58438a7

	# loadBalancerServices:
	# annotations:
	# foo: bar
	# externalTrafficPolicy: Local
	# zones:
	# - name: europe-1a
	# annotations:
	# foo: bar
	# externalTrafficPolicy: Local

Why is this needed:
See above.