Thanos Ruler: got 401 Body: Unauthorized

[JoshuaSmeda]

Name and Version

bitnami/thanos 15.4.4

NAME                 	NAMESPACE 	UPDATED                                	STATUS  	CHART                        	APP VERSION
kube-prometheus-stack	monitoring      2024-05-13 12:29:58.815887019 +0000 UTC	deployed	kube-prometheus-stack-56.21.4	v0.71.2
thanos               	monitoring     	2024-05-13 17:39:01.664992 +0200 SAST  	deployed	thanos-15.4.4                	0.35.0

What architecture are you using?

amd64

What steps will reproduce the bug?

I'm busy setting up a Thanos stack. So far, things have been smooth sailing.

I've deployed the stack using the thanos Helm chart, provided by Bitnami. I also have the kube-prom-stack deployed that is remote writing metrics into my thanos-receiver (which has basic auth) - this is working perfectly.

alertmanager-kube-prometheus-stack-0                        Running
kube-prometheus-stack-grafana-6b79f9bd7f-kdmc4              Running
kube-prometheus-stack-kube-state-metrics-656b554d59-xdvc2   Running
kube-prometheus-stack-operator-67b889c86d-pqpv7             Running
kube-prometheus-stack-prometheus-node-exporter-bz6pg        Running
kube-prometheus-stack-prometheus-node-exporter-cswqj        Running
kube-prometheus-stack-prometheus-node-exporter-f2l5h        Running
kube-prometheus-stack-prometheus-node-exporter-vpcff        Running
prometheus-kube-prometheus-stack-0                          Running
thanos-compactor-7fbf798589-n8tfp                           Running
thanos-query-75fd889f4-tgnfh                                Running
thanos-query-frontend-6f476cdf4b-2t2gn                      Running
thanos-receive-0                                            Running
thanos-receive-1                                            Running
thanos-ruler-0                                              Running
thanos-storegateway-0                                       Running

I've added basic authentication, via values.yaml:

auth:
  basicAuthUsers:
    xxx: xxx

which generates something similar to this:

$ k get secret thanos-http-config-secret -o json | jq '.data | map_values(@base64d)'
{
  "http-config.yml": "basic_auth_users:\n  xxx: xxx"
}

which is injected into all my pods.

My problem:

I cannot get my Thanos Ruler to authenticate to my Thanos query API.

The error I receive:

ts=2024-05-10T15:35:17.100967998Z caller=promclient.go:437 level=debug msg="querying instant" url="http://172.20.112.174:9090/api/v1/query?analyze=false&dedup=true&engine=&explain=false&partial_response=false&query=absent%28up%7Bprometheus%3D%22monitoring%2Fkube-prometheus-stack%22%7D%29&time=2024-05-10T15%3A35%3A17.0967074Z"
ts=2024-05-10T15:35:17.10288841Z caller=rule.go:926 level=error component=rules err="read query instant response: expected 2xx response, got 401. Body: Unauthorized\n" query="absent(up{prometheus=\"monitoring/kube-prometheus-stack\"})"

My arguments passed into the Ruler container, noting that everything works perfectly fine without an authentication set:

    Args:
      rule
      --log.level=debug
      --log.format=logfmt
      --grpc-address=0.0.0.0:10901
      --http-address=0.0.0.0:10902
      --data-dir=/data
      --eval-interval=1m
      --http.config=/conf/http/http-config.yml
      --alertmanagers.url=http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093
      --query=dnssrv+_http._tcp.thanos-query.monitoring.svc.cluster.local
      --alert.query-url=http://thanos-query.monitoring.svc.cluster.local:9090
      --label=replica="$(POD_NAME)"
      --label=ruler_cluster=""
      --alert.label-drop=replica
      --objstore.config-file=/conf/objstore/objstore.yml
      --rule-file=/conf/rules/*.yml

The query and http.config are passed in via the Helm chart. So I can't manually specify an additional flag of http.config because of duplicate keys, which could include my basic_auth credentials as described here.

How am I supposed to authenticate to the query API? Is there a simpler way that I may be missing?

And why is there a disparity between the different Thanos components? Why does the query API leverage basic_auth and the rest of the components require basic_auth_users configuration?

I appreciate any assistance!

Are you using any custom parameters or values?

Yes, here's a sample omitted for brevity

auth:
  basicAuthUsers:
    joshua: test

query:
  enabled: true

queryFrontend:
  enabled: true
  extraFlags:
   - --query-frontend.forward-header=Authorization

ruler:
  enabled: true
  logLevel: debug

alertmanagers:
    - http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093
  config: |-
    groups:
      - name: "metamonitoring"
        rules:
          - alert: "PrometheusDown"
            expr: absent(up{prometheus="monitoring/kube-prometheus-stack"})

What is the expected behavior?

I expect the authentication to work out seamlessly, like the other components.

What do you see instead?

ts=2024-05-10T15:35:17.100967998Z caller=promclient.go:437 level=debug msg="querying instant" url="http://172.20.112.174:9090/api/v1/query?analyze=false&dedup=true&engine=&explain=false&partial_response=false&query=absent%28up%7Bprometheus%3D%22monitoring%2Fkube-prometheus-stack%22%7D%29&time=2024-05-10T15%3A35%3A17.0967074Z"
ts=2024-05-10T15:35:17.10288841Z caller=rule.go:926 level=error component=rules err="read query instant response: expected 2xx response, got 401. Body: Unauthorized\n" query="absent(up{prometheus=\"monitoring/kube-prometheus-stack\"})"

[JoshuaSmeda]

I asked the same question on the Thanos Github discussion page, hoping for a response in case anyone else had a similar issue, but I haven't received any feedback yet.

thanos-io/thanos#7351

[carrodher]

The issue may not be directly related to the Bitnami container image or Helm chart, but rather to how the application is being utilized or configured in your specific environment.

Having said that, if you think that's not the case and are interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.