-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to properly setup remoteWrite credentials in vmagent #6280
Comments
Hey @evertonspader-tomtom ,
If you can see them, it might be caused by some wrong escape characters in your username or password. |
@Haleygo yeah, that is the problem, I don't see the flags being added, I thought the flags from env variables could be inherited in a different way. See the full logs: {"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:13","msg":"command-line flags"}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -envflag.enable=\"true\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -envflag.prefix=\"VM_\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -loggerFormat=\"json\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -promscrape.config=\"/config/scrape.yml\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -remoteWrite.label=\"[email protected],release=metric-proxy-vm\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -remoteWrite.tmpDataPath=\"/tmpData\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -remoteWrite.url=\"secret\""}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/app/vmagent/main.go:127","msg":"starting vmagent at \"[:8429]\"..."}
{"ts":"2024-05-16T08:11:42.447Z","level":"info","caller":"VictoriaMetrics/lib/memory/memory.go:42","msg":"limiting caches to 2457600000 bytes, leaving 1638400000 bytes to the OS according to -memory.allowedPercent=60"}
{"ts":"2024-05-16T08:11:42.465Z","level":"info","caller":"VictoriaMetrics/lib/persistentqueue/fastqueue.go:66","msg":"opened fast persistent queue at \"/tmpData/persistent-queue/1_2F652A4689A1A365\" with maxInmemoryBlocks=400, it contains 0 pending bytes"}
{"ts":"2024-05-16T08:11:42.623Z","level":"info","caller":"VictoriaMetrics/app/vmagent/remotewrite/client.go:166","msg":"the remote storage at \"1:secret-url\" doesn't support VictoriaMetrics remote write protocol. Switching to Prometheus remote write protocol. See https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol"}
{"ts":"2024-05-16T08:11:42.623Z","level":"info","caller":"VictoriaMetrics/app/vmagent/remotewrite/client.go:202","msg":"initialized client for -remoteWrite.url=\"1:secret-url\""}
{"ts":"2024-05-16T08:11:42.623Z","level":"info","caller":"VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:276","msg":"removing dangling queue \"1_075B615FC191DB76\""}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:283","msg":"removed 1 dangling queues from \"/tmpData\", active queues: 1"}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/app/vmagent/main.go:152","msg":"started vmagent in 0.179 seconds"}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:119","msg":"starting server at http://127.0.0.1:8429/"}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/lib/promscrape/scraper.go:113","msg":"reading scrape configs from \"/config/scrape.yml\""}
{"ts":"2024-05-16T08:11:42.626Z","level":"info","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:120","msg":"pprof handlers are exposed at http://127.0.0.1:8429/debug/pprof/"}
{"ts":"2024-05-16T08:11:42.627Z","level":"info","caller":"VictoriaMetrics/lib/promscrape/config.go:140","msg":"starting service discovery routines..."}
{"ts":"2024-05-16T08:11:42.627Z","level":"info","caller":"VictoriaMetrics/lib/promscrape/config.go:146","msg":"started 0 service discovery routines in 0.000 seconds"}
{"ts":"2024-05-16T08:12:21.148Z","level":"error","caller":"VictoriaMetrics/app/vmagent/remotewrite/client.go:461","msg":"unexpected status code received after sending a block with size 700 bytes to \"1:secret-url\" during retry #1: 401; response body=\"{\\\"status\\\":\\\"error\\\",\\\"error\\\":\\\"authentication error: no credentials provided\\\"}\"; re-sending the block in 2.016 seconds"} |
@evertonspader-tomtom Okay, that's not expected. |
@Haleygo I am using the Helm chart with the values below: extraArgs:
remoteWrite.url: <remote_write_endpoint>
# This was another attempt of using env variables
# remoteWrite.basicAuth.username: "%{BASIC_AUTH_USERNAME}"
# remoteWrite.basicAuth.password: "%{BASIC_AUTH_PASSWORD}"
envflag.enable: "true"
envflag.prefix: "VM_"
env:
- name: VM_remoteWrite_basicAuth_username
valueFrom:
secretKeyRef:
name: prometheus-secret
key: username
- name: VM_remoteWrite_basicAuth_password
valueFrom:
secretKeyRef:
name: prometheus-secret
key: password
# - name: BASIC_AUTH_USERNAME
# valueFrom:
# secretKeyRef:
# name: prometheus-secret
# key: username
# - name: BASIC_AUTH_PASSWORD
# valueFrom:
# secretKeyRef:
# name: prometheus-secret
# key: password
service:
enabled: true
resources:
limits:
memory: 4096M
cpu: 2000m
requests:
memory: 2048M
cpu: 1000m
persistence:
enabled: true
size: 18Gi
config:
scrape_configs: [] I can see the variables created with the correct values if I exec into the container: VM_remoteWrite_basicAuth_username=<username>
VM_remoteWrite_basicAuth_password=<password> And I'm installing the chart like this: helm upgrade --install \
--version 0.10.7 \
--namespace vmagent \
--create-namespace \
--values values.yaml \
vmagent \
vm/victoria-metrics-agent |
@evertonspader-tomtom |
I see it now using the value directly in the environment variable: env:
- name: VM_remoteWrite_basicAuth_username
value: username
- name: VM_remoteWrite_basicAuth_password
value: password {"ts":"2024-05-16T11:40:27.858Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -remoteWrite.basicAuth.password=\"secret\""}
{"ts":"2024-05-16T11:40:27.858Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -remoteWrite.basicAuth.username=\"\\\"username\\\\n\\\"\""} but I still get a 401. I've tried to change the username and password to not have any special characters. |
I also see this in the logs when trying to apply changes to the chart but I don't know what it means or if it's related at all: {"ts":"2024-05-16T12:04:55.734Z","level":"panic","caller":"VictoriaMetrics/lib/fs/fs.go:345","msg":"FATAL: cannot create lock file: cannot acquire lock on file \"/tmpData/persistent-queue/1_2F652A4689A1A365/flock.lock\": resource temporarily unavailable; make sure a single process has exclusive access to \"/tmpData/persistent-queue/1_2F652A4689A1A365\""}``` |
And you still can get it work by specifying cmd-flags You can find the full yaml in https://github.com/VictoriaMetrics/VictoriaMetrics/blob/test-vmalert-remotewrite-auth/deployment/docker/docker-compose.yml.
It's a panic level log which will exit the process immediately, it means the vmagent process can't acquire exclusive lock for persistent-queue directory. It could happen if you mount one filesystem to vmagent, and new pod is running while old pod hasn't finished terminating. |
Ok, I've noticed that after making any changes to the values file and applying it with |
Is your question request related to a specific component?
vmagent
Describe the question in detail
What is the proper way to set up the
remoteWrite
credentials using environment variables? I can get a successful connection usingOr by splitting the username and password in
remoteWrite.username
andremoteWrite.password
.But, of course, for security reasons, this is not ideal. So I am trying to use environment variables to pass extra flags.
So I added the following:
But it doesn't seem to pick up the credentials as now I see a 401 in the export requests. What am I missing here? Needless to day that the secret is already created with the correct keys an values.
Troubleshooting docs
The text was updated successfully, but these errors were encountered: