Enable dedicated interceptor/scalers with the same operator

[arschles]

Discussed in #218

^{Originally posted by rwkarg August 4, 2021}
This issue (#36) has been closed but I don't see a way to currently add annotations to the interceptor service.

The specific use case we have is using autoneg-controller to add workload instances from multiple clusters as backends to a single Load Balancer. In GCP, this requires that the service (would be the interceptor service) has two annotations to create the Network Endpoint Group (NEG) and to associate that NEG with a Backend Service on the Load Balancer (not important what it does, just that we need annotations on the interceptor service).

Looks like #206 may complicate this, if there is only one service. We can't add the annotations to individual services any more for autonet-controller to work in that case.

Is the multi-tenant interceptor something that can be opted out of?

---

Further along in that discussion, @tomkerkhove and I talked through how to allow operators to opt-out of the shared interceptor/scaler while still using the same operator. in this proposal, I outline how to enable that exact scheme.

New CRD: HTTPScalingComponents

The purpose of this new, optional, CRD, is to specify a set of running interceptor and scaler. Interceptor and scaler are always deployed together. When an HTTPScalingComponents CRD is submitted, the operator will react by ensuring the following are installed.

• interceptor Deployment
• interceptor Services for proxy and admin endpoints
• interceptor ScaledObject
• scaler Deployment
• scaler Services for gRPC and health check/admin endpoints

Sample YAML for a HTTPScalingComponents CRD would be similar to:

kind: HTTPScalingComponents
apiVersion: http.keda.sh/v1alpha1
metadata:
    name: MyApp
spec:
    interceptor:
        serviceType: ClusterIP # this is the default, but you can specify other service types
        minReplicas: 10
        maxReplicas: 100
        annotations:
            key1: val1
    scaler:
        serviceType: ClusterIP # same as spec.interceptor.serviceType
        annotations:
            key1: val1

Usage in HTTPScaledObjects

A user submitting an HTTPScaledObject will optionally be allowed to specify the components that they would like their app to use. The field, scalingComponents, would be used as follows:

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
    name: MySO
spec:
    scalingInfrastructure:
        name: MyApp
    host: myhost.com
    scaleTargetRef:
        deployment: myappdeployment
        service: myappservice
        port: 8080
    replicas:
        min: 0
        max: 1000

Note the new spec.scalingInfrastructure section. spec.scalingInfrastructure.name specifies an already-installed HTTPScalingComponents CRD and is optional. The HTTP Addon operator is configured with a default HTTPScalingComponents on startup (and one is installed by default in the official helm chart). Users who omit this name field will simply use the default

Implementation notes

The biggest changes will come to the operator if this proposal is implemented. The changes of note are:

• Create new scaler and interceptor infrastructure when a new HTTPScalingComponents is created
• Segment routing tables by HTTPScalingComponents, and write one routing table ConfigMap per HTTPScalingComponents CRD
• Tell the interceptor Deployment which aforementioned ConfigMap to consume
• On HTTPScaledObject submission, create ScaledObjects that point to the proper scaler Service

cc/ @rwkarg @tomkerkhove

[arschles]

Note that this functionality would interact with #240. See #240 (comment) for more details

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[arschles]

Reopening with label to keep the stale bot from closing this. This is still a valid feature that we'd like to explore