Problem moving postgres password to secret

[MikeKulls2]

I have airflow running under Kubernetes via ArgoCD Application using this helm chart. It is using an external postgres database. If I specify my user/pass in the application yaml file like below then everything works 100% fine. The problem is this means my DB password is stored in Git. So I want to move the user/pass to a secret, standard stuff

      - name: externalDatabase.user
        value: "airflow_user"
      - name: externalDatabase.password
        value: "xxxxx"

Following the documentation, I add these entries to the values. I have the option to remove the user/pass entry above or leave it in, I have tried it both ways

      - name: externalDatabase.passwordSecret
        value: "db-pass"
      - name: externalDatabase.passwordSecretKey
        value: "password"
      - name: externalDatabase.userSecret
        value: "db-user"
      - name: externalDatabase.userSecretKey
        value: "username"

The problem is when I do that it loses connection to the postgres database completely. This is the error I get. The error is in the db-migrations pod in the check-db container. Probably other containers get the same error also. This indicates once I use the value "externalDatabase.passwordSecret" it starts ignoring parameters such as externalDatabase.host

psycopg2.OperationalError: connection to server at "airflow-pgbouncer.airflow.svc.cluster.local" (10.101.86.111), port 6432 failed.

When I look at the yaml that comes out of the helm template the only difference is the addition of 2 environment variables referring to the user/pass secrets. This looks correct to me. It's like the pod starts to ignore the hostname for the DB and other details when these environment variables are encountered.

      env:
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: db-user
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: db-pass

I'm really baffled as to what I'm doing wrong here. I've followed the docs and this appears to be what the docs specify should be done. I've disabled the internal postgres with postgresql.enabled: false and pgbouncer is enabled. That is all the docs mention. I don't think I'm doing anything wrong because it works 100% if I remove the reference to the secrets and just specify user/pass straight up. I've also specified a secret for git-sync and that works fine. I've checked over it 100 times to make sure secret names are correct and keys in the secret are correct. I did initially have some errors like that but resolved those. I've deleted the entire application and recreated it multiple times but it hasn't helped.

If you got this far, thanks for taking the time to read all this :-)

[MikeKulls2]

I have tried also removing the values from the application yaml and putting them only in the values file like this. I get the same result. It works fine if I simply specify user/pass but not if I try to define them via a secret. This is a snippet of the values file

pgbouncer:
  enabled: true
postgresql:
  enabled: false
externalDatabase:
  type: postgres
  host: "xxxxx"
  port: 5432
  database: "airflow_db"
  user: "airflow_user"
  userSecret: "db-user"
  userSecretKey: "username"
  password: "xxxx"
  passwordSecret: "db-pass"
  passwordSecretKey: "password"
  properties: ""

[MikeKulls2]

OK, I solved it. The problem was both username and password had a linefeed at the end of them. Once encoded into base64 it wasn't evident this LF was there. Both vim, nano and echo add linefeed to the end of files. It was harder than I thought to not add a line feed in an easy to remember way while also keeping the password out of history. This is what I came up with:

set HISTCONTROL="ignorespace"
 echo -n "myuser" > username
 echo -n "mypassword" > password
kubectl create secret generic db-user-pass --from-file=username=username --from-file=password=password
rm username password

OR

set HISTCONTROL="ignorespace"
 kubectl create secret generic db-user-pass --from-literal=username=myuser --from-literal=password=mypassword

[thesuperzapper]

@MikeKulls2 thanks for the update, dangling line endings are a common problem when creating secrets in Kubernetes.

Personally, I recommend using a tool like the incredible k9s CLI for Kubernetes, as it makes issues like this easier to spot (among MANY other things).

For example, you can view the "decoded" secret by pressing x on the secret you are investigating, it will show the value using a YAML | multi-line string, which indicates a dangling new line.