Unable to retrieve data from a Vault cluster using https from build agent

[IainColledge]

Am using Vault and Build agents within a kubernetes cluster and am using the current Jetbrains plugin.

Vault is configured as a 3 node HA set and accessed via the URL: "https://vault.infrastructure.svc.cluster.local:8200"

Teamcity can connect using this URL and connection tests are succesful.

Teamcity build agents fails to retrieve data using the argument "vault:/vault/secret/cicd/blue!/foo"

However with a local vault install the same path will return:

vault read secret/cicd/blue
Key                     Value
---                     -----
refresh_interval        168h
foo                     hello

The cluster is using self signed certs however the JVM and also vault environment variables are configured as is keystore loading on container creation into the JVM.

This has also been tried using http only as well.

[VladRassokhin]

Could you please provide part of build log showing this problem?

[IainColledge]

Thanks for the reply:

HashiCorp Vault (7s)
[14:49:54][HashiCorp Vault] HashiCorp Vault token successfully fetched
[14:49:54][HashiCorp Vault] 2 Vault references to resolve: [vault:/vault/secret/cicd/blue!/foo, vault:/vault/secret/cicd/blue!/newrelic_licence_key]
[14:49:54][HashiCorp Vault] Failed to fetch data for path '/vault/secret/cicd/blue'
[14:49:54][HashiCorp Vault] Cannot resolve '/vault/secret/cicd/blue!/foo': data wasn't received from HashiCorp Vault

And later:

Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[14:49:55]Checkout directory: /home/teamcityba/buildAgent/work/74e814b1d9185321
[14:49:55]Parameter "env.READVAULT=%vault:/vault/secret/cicd/blue!/foo%" is not fully resolved, using as is.
[14:49:55]Parameter "newrelic_licence_key=%vault:/vault/secret/cicd/blue!/newrelic_licence_key%" is not fully resolved, using as is.

[IainColledge]

Is there a debug mode I can switch on at all to give some more verbose logs please?

[ghost]

@IainColledge do you have policies attached to this role?

[VladRassokhin]

I've added some logging for Failed to fetch data for path messages, if you see it again, please add here full message

[ieugen]

I think I have the same issue with teamcity 2020.2, linux agent.

teamcity_agent.2.zgn48wyhb19f@admin    | [2021-05-21 17:35:30,663]   INFO - ltLifecycleAwareSessionManager - Logged in with token: LoginToken(renewable=true, lease_duration=1198):
teamcity_agent.2.zgn48wyhb19f@admin    | [2021-05-21 17:35:30,664]   INFO - ltLifecycleAwareSessionManager - Scheduling HashiCorp Vault token refresh to Fri May 21 17:55:13 BST 2021
teamcity_agent.2.zgn48wyhb19f@admin    | [2021-05-21 17:35:30,721]   WARN - .agent.VaultParametersResolver - Cannot resolve '/kv-test/my-little-secret!/hello': data wasn't received from HashiCorp Vault
teamcity_agent.2.zgn48wyhb19f@admin    | [2021-05-21 17:35:30,723]   INFO - nt.impl.BuildRunAgentStateImpl - Stopping build on agent. Reason: fail and stop build command from the agent (Error while fetching data from HashiCorp Vault )

Locally I can get the token.
I can also get the token via vault-cli inside the build.

I'm using letsencrypt certificate to secure vault tls.
It might because of that ?!
The browser asks me to confirm the certificate - but not in the unsafe way.
Maybe the java client can't handle this?