Istio telemetry with als extension provider seems to have some sort of session stickiness

[joelmathew003]

I have a telemetry resource in my namespace which looks like:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
spec:
  accessLogging:
  - providers:
    - name: my-http-als
    - name: my-tcp-als

and have described these providers in the istio configmap mesh

apiVersion: v1
data:
  mesh: |-
    accessLogEncoding: JSON
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      extraStatTags:
      - request_method
      - request_path
      tracing:
        zipkin:
          address: zipkin.istio-system:9411
    enableEnvoyAccessLogService: true
    extensionProviders:
    - envoyHttpAls:
        port: 9999
        service: collector.test-ns.svc.cluster.local
      name: my-http-als
    - envoyTcpAls:
        port: 9999
        service: collector.test-ns.svc.cluster.local
      name: my-tcp-als
    enablePrometheusMerge: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'
kind: ConfigMap

My collector want the access logs of the traffic in my namespace and does some processing. I have an hpa resource created for collector which scales on 60% utilization. On some stress testing(~200 tps) and new replicas are created by hpa, I see that the older replicas receive most of the traffic, even after the new replicas are created and ready. Sometimes the older replicas cross the limit and restarts, on which the newer replicas start receiving the load. Is there some sort of session stickiness that causes this? Ideally I want the load to get distributed evenly among all ready replicas.

[howardjohn]

This is among replicas of the ALS pods, not your app pod, right?

ALS is a gRPC streaming protocol (https://github.com/envoyproxy/envoy/blob/a0cc3001b1cf42396e401f0527bd56246dd2fba9/api/envoy/service/accesslog/v3/als.proto#L28) so I suspect once envoy opens a connection to one pod it will 'stick' to that replica.

Istio has a similar issue in the connection to the control plane (Istiod); there we set a 30min max connection age to rebalance. Not perfect solution, though.

[zirain]

sorry, please add -o json/yaml after istioctl pc all

[joelmathew003]

istioctl-pc-all (1).txt

[zirain]

form the tps.txt, I cannot find any request about StreamAccessLogs?

[joelmathew003]

What should I be seeing in the logs? In my ALS pod, I see the stream access log messages coming in though. Should I try trace logs?

[zirain]

I can see OPA request /logs, it that what you saw?

[joelmathew003]

Sorry for the delayed response. Fixed the issue by setting a connection age through an EnvoyFilter and this enables connection rotation

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: als-config
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio-injection: enabled
  configPatches:
    - applyTo: CLUSTER
      match:
        cluster:
          service: "collector.test-ns.svc.cluster.local"
      patch:
        operation: MERGE
        value:
          typed_extension_protocol_options:
            envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
              "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
              common_http_protocol_options:
                max_connection_duration: "310s"
                max_stream_duration: "300s"

              explicit_http_config:
                http2_protocol_options: {}