TLS requests are sent over HTTP instead of HTTPS

[spyrosmouchlianitis]

Report

I've setup the http-add-on with TLS enabled. The certificates are read from the add-on and TLS verification works correctly. My issues is when I send an https request to my ingress controller, the interceptor instead of sending the request to my service via https it switches to http.

According to this, shouldn't it work via https if TLS is enabled?

http-add-on/interceptor/middleware/routing.go

Lines 67 to 83 in 0793ece

	func (rm *Routing) streamFromHTTPSO(httpso *httpv1alpha1.HTTPScaledObject) (*url.URL, error) {
	if rm.tlsEnabled {
	return url.Parse(fmt.Sprintf(
	"https://%s.%s:%d",
	httpso.Spec.ScaleTargetRef.Service,
	httpso.GetNamespace(),
	httpso.Spec.ScaleTargetRef.Port,
	))
	}
	//goland:noinspection HttpUrlsUsage
	return url.Parse(fmt.Sprintf(
	"http://%s.%s:%d",
	httpso.Spec.ScaleTargetRef.Service,
	httpso.GetNamespace(),
	httpso.Spec.ScaleTargetRef.Port,
	))
	}

I've tried sending the same request the add-on is attempting but with https and it works as expected.

Ingress configuration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
  labels:
    kubernetes.mydomain.com/branch-name: devops-274
  name: apps-devops-274
  namespace: core-uat-branches
spec:
  rules:
    - host: uat.mydomain.com
      http:
        paths:
          - backend:
              service:
                name: keda-add-ons-http-interceptor-proxy
                port:
                  number: 8080
            path: /devops-274
            pathType: Prefix

HTTPScaledObject

apiVersion: http.keda.sh/v1alpha1
kind: HTTPScaledObject
metadata:
  name: site-devops-274
  namespace: core-uat-branches
spec:
  hosts:
    - uat.mydomain.com
  pathPrefixes:
    - /devops-274/
  replicas:
    max: 1
    min: 0
  scaleTargetRef:
    kind: Deployment
    name: site-devops-274
    port: 443
    service: site-devops-274
  scaledownPeriod: 28800
  scalingMetric:
    concurrency:
      targetValue: 1

Service

apiVersion: v1
kind: Service
metadata:
  labels:
    kubernetes.mydomain.com/branch-name: devops-274
  name: site-devops-274
  namespace: core-uat-branches
spec:
  ports:
    - name: https
      port: 443
      targetPort: https
  selector:
    app: site
    kubernetes.mydomain.com/branch-name: devops-274

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: site
    kubernetes.mydomain.com/branch-name: devops-274
  name: site-devops-274
  namespace: core-uat-branches
spec:
  selector:
    matchLabels:
      app: site
      kubernetes.mydomain.com/branch-name: devops-274
  template:
    metadata:
      labels:
        app: site
        kubernetes.mydomain.com/branch-name: devops-274
    spec:
      containers:
        - image: myimage:latest
          imagePullPolicy: Always
          name: site
          ports:
            - containerPort: 443
              name: https
      nodeSelector:
        kubernetes.io/os: windows

Expected Behavior

Interceptor should send requests via https

Actual Behavior

Interceptor sends requests via http

Steps to Reproduce the Problem

1. Setup keda with http-add-on like usual
2. Enable TLS
3. Add TLS certificates on both ingress controller and http-add-on
4. Send a request to the ingress controller via https

Logs from KEDA HTTP operator

2024-05-14T08:00:34Z	ERROR	LoggingMiddleware.RoutingMiddleware.CountingMiddleware.UpstreamHandler.StaticHandler	Bad Gateway	{"routingKey": "//uat.mydomain.com/devops-274/", "namespacedName": {"name":"site-devops-274","namespace":"core-uat-branches"}, "stream": "http://site-devops-274.core-uat-branches:443/devops-274", "error": "read tcp 10.121.0.23:46932->10.4.235.37:443: read: connection reset by peer"}
github.com/kedacore/http-add-on/interceptor/handler.(*Static).ServeHTTP
	github.com/kedacore/http-add-on/interceptor/handler/static.go:36
github.com/kedacore/http-add-on/interceptor/handler.(*Upstream).ServeHTTP.func2
	github.com/kedacore/http-add-on/interceptor/handler/upstream.go:54
net/http/httputil.(*ReverseProxy).ServeHTTP
	net/http/httputil/reverseproxy.go:472
github.com/kedacore/http-add-on/interceptor/handler.(*Upstream).ServeHTTP
	github.com/kedacore/http-add-on/interceptor/handler/upstream.go:57
main.runProxyServer.newForwardingHandler.func2
	github.com/kedacore/http-add-on/interceptor/proxy_handlers.go:87
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2166
github.com/kedacore/http-add-on/interceptor/middleware.(*Counting).ServeHTTP
	github.com/kedacore/http-add-on/interceptor/middleware/counting.go:35
github.com/kedacore/http-add-on/interceptor/middleware.(*Routing).ServeHTTP
	github.com/kedacore/http-add-on/interceptor/middleware/routing.go:64
github.com/kedacore/http-add-on/interceptor/middleware.(*Logging).ServeHTTP
	github.com/kedacore/http-add-on/interceptor/middleware/logging.go:42
github.com/kedacore/http-add-on/interceptor/middleware.(*Metrics).ServeHTTP
	github.com/kedacore/http-add-on/interceptor/middleware/metrics.go:24
net/http.serverHandler.ServeHTTP
	net/http/server.go:3137
net/http.(*conn).serve
	net/http/server.go:2039

HTTP Add-on Version

0.8.0

Kubernetes Version

1.28

Platform

Microsoft Azure

Anything else?

No response

[JorTurFer]

Hello,
I can see that the request is done against an https port 10.4.235.37:443 but using http protocol, which is weird. @zorocloud do you have any idea or suggestion?

[zorocloud]

Hmmm yeah very weird.

@spyrosmouchlianitis are you able to share the ingress configuration you used by any chance (details anonymised where needed)? Looking at the logs you posted, it feels like the ingress controller is trying to establish a http connection with the interceptor proxy just on port 443.

[spyrosmouchlianitis]

@zorocloud Updated with my configuration

[zorocloud]

Thanks @spyrosmouchlianitis.

Looking at your ingress configuration, it seems that you are sending the request to the interceptor proxy over HTTP (port 8080). Even though you have configured TLS to be enabled in the interceptor proxy's configuration, you need to send the request to the interceptor on it's exposed HTTPS port (8443 by default) in order to have the onward request sent over HTTPS. This is because the interceptor proxy will expose both an HTTP and HTTPS server when TLS is enabled, and the HTTP server will always evaluate if rm.tlsEnabled to false.

Could you please try updating your Ingress configuration to send traffic to the interceptor proxy over HTTPS (port 8443)? That should hopefully solve your issue 🙂.

[spyrosmouchlianitis]

I found the issue. I had to modify the interceptor's proxy service and add the 8443 port

apiVersion: v1
kind: Service
metadata:
  name: keda-add-ons-http-interceptor-proxy
spec:
  ports:
  - name: https
    port: 8443
    protocol: TCP
    targetPort: https
  - name: proxy
    port: 8080
    targetPort: proxy

I also had to update the interceptor's deployment and add the port 8443

ports:
  - containerPort: 8443
    name: https
    protocol: TCP

as well as, update my ingress to send the requests to port 8443 instead of 8080.

I think the documentation for the TLS should be enhanced to include these necessary changes. Right now, it implies that you only need to set the following parameters:

• KEDA_HTTP_PROXY_TLS_ENABLE
• KEDA_HTTP_PROXY_TLS_PORT
• KEDA_HTTP_PROXY_TLS_CERT_PATH
• KEDA_HTTP_PROXY_TLS_KEY_PATH

[zorocloud]

Sweet, glad you are up and running now 👍.

I think what we probably need to do is update the helm chart for the Add-on to add the necessary configuration to those resources when TLS is enabled. Then theoretically everything will just work out of the box without any extra manual configuration (either by hand or with Kustoimize).

@JorTurFer is a separate ticket needed in the repository where you keep the charts to get these updates in?

[zroubalik]

@zorocloud would you be willing to update the helm chart please? You can reference this issue in the PR, no need to open another one. Thanks!

[zorocloud]

Sure! Raised a PR in the chart repo though have an open question on there @zroubalik.

[gshhong]

@spyrosmouchlianitis
I would like to inquire about the detailed setup steps for the second and third steps below.
Thank you very much.
2. Enable TLS
3. Add TLS certificates on both ingress controllers and HTTP add on