Add the VPAAndHPAForAPIServer feature gate for the gardener-operator

[ialidzhikov]

How to categorize this PR?

/area auto-scaling
/kind enhancement

What this PR does / why we need it:

Which issue(s) this PR fixes:
Part of #9562
A follow-up of #9678

Special notes for your reviewer:

• This PR is based on Introduce a new autoscaling mode (VPAAndHPA) for Shoot Kubernetes API servers #9678, hence it is in draft state until Introduce a new autoscaling mode (VPAAndHPA) for Shoot Kubernetes API servers #9678 is merged. The PR is now rebased after the merge of Introduce a new autoscaling mode (VPAAndHPA) for Shoot Kubernetes API servers #9678.

Release note:

The `VPAAndHPAForAPIServer` feature gate is now also implemented for the gardener-operator. When enabled, the virtual-garden-kube-apiserver and gardener-apiserver are scaled simultaneously by VPA and HPA on the same metric (CPU and memory usage).

[gardener-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[voelzmo]

Hey @ialidzhikov, thanks for the PR! While looking at the changes, I was wondering if we're missing the removal code for HVPA, HPA and VPA objects for the cases when the autoscalingMode is changed? This seems to have been broken already for any switch between HVPA enabled or HVPA disabled, but we never saw this?
Once we merge this and the mode gets changed from HVPA to VPAAndHPA, I guess we would still have an HVPA object and the corresponding VPA and HPA objects created by the hvpa-controller, right?

[ialidzhikov]

While looking at the changes, I was wondering if we're missing the removal code for HVPA, HPA and VPA objects for the cases when the autoscalingMode is changed? This seems to have been broken already for any switch between HVPA enabled or HVPA disabled, but we never saw this?
Once we merge this and the mode gets changed from HVPA to VPAAndHPA, I guess we would still have an HVPA object and the corresponding VPA and HPA objects created by the hvpa-controller, right?

For the kubernetes apiserver component (pkg/component/kubernetes/apiserver, used for the Shoot kube-apiserver and virtual-garden-kube-apiserver) - this is a component that is NOT deployed via GRM, but with a client. Hence, we have everywhere explicit client invocations to delete the no longer needed objects:

• gardener/pkg/component/kubernetes/apiserver/hvpa.go

  Lines 32 to 36 in 6d6c06c

  	if k.values.Autoscaling.Mode != apiserver.AutoscalingModeHVPA ||
  	k.values.Autoscaling.Replicas == nil ||
  	*k.values.Autoscaling.Replicas == 0 {
  	return kubernetesutils.DeleteObject(ctx, k.client.Client(), hvpa)
  	}

• gardener/pkg/component/kubernetes/apiserver/verticalpodautoscaler.go

  Lines 28 to 37 in 6d6c06c

  	func (k *kubeAPIServer) reconcileVerticalPodAutoscaler(ctx context.Context, verticalPodAutoscaler *vpaautoscalingv1.VerticalPodAutoscaler, deployment *appsv1.Deployment) error {
  	switch k.values.Autoscaling.Mode {
  	case apiserver.AutoscalingModeHVPA:
  	return kubernetesutils.DeleteObject(ctx, k.client.Client(), verticalPodAutoscaler)
  	case apiserver.AutoscalingModeVPAAndHPA:
  	return k.reconcileVerticalPodAutoscalerInVPAAndHPAMode(ctx, verticalPodAutoscaler, deployment)
  	default:
  	return k.reconcileVerticalPodAutoscalerInBaselineMode(ctx, verticalPodAutoscaler, deployment)
  	}
  	}

• gardener/pkg/component/kubernetes/apiserver/horizontalpodautoscaler.go

  Lines 38 to 50 in 6d6c06c

  	func (k *kubeAPIServer) reconcileHorizontalPodAutoscaler(ctx context.Context, hpa *autoscalingv2.HorizontalPodAutoscaler, deployment *appsv1.Deployment) error {
  	if k.values.Autoscaling.Mode == apiserver.AutoscalingModeHVPA ||
  	k.values.Autoscaling.Replicas == nil ||
  	*k.values.Autoscaling.Replicas == 0 {
  	return kubernetesutils.DeleteObject(ctx, k.client.Client(), hpa)
  	}
  	if k.values.Autoscaling.Mode == apiserver.AutoscalingModeVPAAndHPA {
  	return k.reconcileHorizontalPodAutoscalerInVPAAndHPAMode(ctx, hpa, deployment)
  	}
  	return k.reconcileHorizontalPodAutoscalerInBaselineMode(ctx, hpa, deployment)
  	}

For the gardener apiserver (pkg/component/gardener/apiserver) - this is a component deployed via GRM:

gardener/pkg/component/gardener/apiserver/apiserver.go

Lines 145 to 153 in 876f6f0

	runtimeResources, err := runtimeRegistry.AddAllAndSerialize(
	g.podDisruptionBudget(),
	g.serviceRuntime(),
	g.horizontalPodAutoscaler(),
	g.verticalPodAutoscaler(),
	g.hvpa(),
	g.deployment(secretCAETCD, secretETCDClient, secretGenericTokenKubeconfig, secretServer, secretAdmissionKubeconfigs, secretETCDEncryptionConfiguration, secretAuditWebhookKubeconfig, secretVirtualGardenAccess, configMapAuditPolicy, configMapAdmissionConfigs),
	g.serviceMonitor(),
	)

Hence, for the gardener apiserver component returning nil from the verticalPodAutoscaler/horizontalPodAutoscaler/hvpa funcs is enough, GRM takes care to delete the no longer desired objects.

[rfranzke]

/assign

[rfranzke]

/lgtm

[gardener-prow]

LGTM label has been added.

Git tree hash: 3fb2e5501e072f30b500c92d4353875f1e590096

[voelzmo]

/lgtm

[ialidzhikov]

/approve

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ialidzhikov, voelzmo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ialidzhikov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment