cubestore S3 credentials regression in v0.35.17 #8163

adamstruck · 2024-04-17T18:22:43Z

Describe the bug

When I attempted to upgrade from v0.35.14 I started to see this error:

2024-04-17T18:17:19.257Z INFO  [cubestored] <pid:1> Cube Store version 0.35.17
2024-04-17T18:17:19.260Z INFO  [cubestore::http::status] <pid:1> Serving status probes at 0.0.0.0:3031
thread 'main' panicked at /build/cubestore/cubestore/src/config/mod.rs:1816:86:
called `Result::unwrap()` on an `Err` value: CubeError { message: "Failed to create S3 credentials: attohttpc: Json Error: expected value at line 1 column 1", backtrace: "", cause: Internal }
stack backtrace:
   0: rust_begin_unwind
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/core/src/result.rs:1649:5
   3: cubestore::config::injection::Injector::register::{{closure}}::{{closure}}::{{closure}}
   4: cubestore::config::injection::Injector::get_service::{{closure}}
   5: cubestore::config::injection::Injector::register::{{closure}}::{{closure}}::{{closure}}
   6: cubestore::config::injection::Injector::get_service::{{closure}}
   7: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
   8: cubestore::config::injection::Injector::get_service_typed::{{closure}}
   9: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
  10: cubestore::config::injection::Injector::get_service_typed::{{closure}}
  11: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
  12: cubestore::config::injection::Injector::get_service_typed::{{closure}}
  13: cubestored::main::{{closure}}
  14: tokio::runtime::park::CachedParkThread::block_on
  15: tokio::runtime::runtime::Runtime::block_on
  16: cubestored::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Version:
v0.35.17

The text was updated successfully, but these errors were encountered:

adamstruck · 2024-04-17T18:27:19Z

I assume this is due to one of these PRs:

#6019
#8158

ovr · 2024-04-17T22:35:22Z

Hello @adamstruck ,

Did you specify both CUBESTORE_AWS_ACCESS_KEY_ID & CUBESTORE_AWS_SECRET_ACCESS_KEY or only CUBESTORE_AWS_SECRET_ACCESS_KEY?

Thanks

adamstruck · 2024-04-17T22:52:11Z

I only set CUBESTORE_AWS_CREDS_REFRESH_EVERY_MINS. The credentials used to be picked up automatically from the pod / instance. We have been running various versions of cube for the past year and this is the first time it has been a problem.

I am able to run v0.35.16 without issue.

ovr · 2024-04-18T09:22:26Z

Is it correct that you are using STS (AWS_WEB_IDENTITY_TOKEN_FILE), IAM?

"Failed to create S3 credentials: attohttpc: Json Error: expected value at line 1 column 1

This error comes from fallback logic that is trying to resolve auth from magic IP 169.254.169.254. It's a fallback variant.

adamstruck · 2024-04-18T15:53:49Z

Yes, s3 access is being managed using IAM roles for the nodes in the EKS cluster.

adamstruck · 2024-04-19T15:54:10Z

@ovr do you think this is something you will be able to fix soon?

ovr · 2024-04-19T18:01:08Z

@adamstruck I tried to find out what had changed in the library, but I could not find anything that can affect you.

adamstruck · 2024-04-20T01:10:44Z

Did cube start doing anything differently starting in that version?

adamstruck · 2024-04-20T04:17:06Z

Or maybe it is this line:

https://github.com/cube-js/cube/pull/6019/files#diff-28d0e549290be2ac2e69b3c1da8d7e05aa0f58db287f72872c53c93c496913a0R57

I am not familiar with rust dependency management, but this seems to imply that not all features are being included after the bump?

adamstruck · 2024-04-23T22:21:31Z

I still am running into this issue with the latest version: v0.35.22

ovr · 2024-04-24T19:36:38Z

We didn't start doing anything differently. I reviewed all changes in the rust-s3 crate and found nothing that could cause this error.

In the production, we use IAM, and it works correctly.

You can check it from the pod, via curl to http://169.254.169.254/latest/meta-data/iam/security-credentials.
Next you can pass role to http://169.254.169.254/latest/meta-data/iam/security-credentials/{YOUR_ROLE}

Do you see any error? It should be a correct JSON.

Thanks

adamstruck · 2024-04-24T20:58:05Z

I think you are on to something...

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
empty role

AWS_ROLE_ARN is set in my env and so is AWS_WEB_IDENTITY_TOKEN_FILE.

Any ideas?

adamstruck · 2024-04-24T21:21:23Z

Based on https://github.com/durch/rust-s3/blob/v0.32.0/aws-creds/src/credentials.rs#L173 it looks like this should be supported.

This seems like it is caused by the bug described in durch/rust-s3#286. Could you bump rust-s3 to 0.33.0 which included the fix?

ChrisLahaye · 2024-04-25T14:44:09Z

I only set CUBESTORE_AWS_CREDS_REFRESH_EVERY_MINS. The credentials used to be picked up automatically from the pod / instance. We have been running various versions of cube for the past year and this is the first time it has been a problem.

I am able to run v0.35.16 without issue.

We are experiencing the same issue although we get the following error,

2024-04-25T14:43:31.783Z ERROR [cubestore::cluster] <pid:1> Error: CubeError { message: "AWS S3 error: serde xml: custom: missing field `Name`", backtrace: "", cause: Internal }

ovr · 2024-04-25T22:37:55Z

Right now, it's not possible to use the 0.33 release because it has bugs
At the same time, 0.34-rc has a problem with large file uploading because it doesn't control a number of parallels put(s), which causes high memory usage.

🫠

At the same time, the official SDK from AWS has problems.

@adamstruck

So, I backported the fix from durch/rust-s3#286 in https://github.com/cube-js/cube/pull/8195/files

ovr · 2024-04-26T13:16:14Z

@adamstruck Could you give a try a latest release? https://github.com/cube-js/cube/releases/tag/v0.35.24

Ty

goncaloacteixeira · 2024-04-26T13:38:39Z

@ovr the issue still persists on my side, tested v0.35.24

2024-04-26T13:38:53.370Z INFO  [cubestored] <pid:1> Cube Store version 0.35.24
2024-04-26T13:38:53.376Z INFO  [cubestore::http::status] <pid:1> Serving status probes at 0.0.0.0:3031
thread 'main' panicked at /build/cubestore/cubestore/src/config/mod.rs:1816:86:
called `Result::unwrap()` on an `Err` value: CubeError { message: "Failed to create S3 credentials: attohttpc: Json Error: EOF while parsing a value at line 1 column 0", backtrace: "", cause: Internal }
stack backtrace:
   0: rust_begin_unwind
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/core/src/result.rs:1649:5
   3: cubestore::config::injection::Injector::register::{{closure}}::{{closure}}::{{closure}}
   4: cubestore::config::injection::Injector::get_service::{{closure}}
   5: cubestore::config::injection::Injector::register::{{closure}}::{{closure}}::{{closure}}
   6: cubestore::config::injection::Injector::get_service::{{closure}}
   7: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
   8: cubestore::config::injection::Injector::get_service_typed::{{closure}}
   9: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
  10: cubestore::config::injection::Injector::get_service_typed::{{closure}}
  11: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
  12: cubestore::config::injection::Injector::get_service_typed::{{closure}}
  13: cubestored::main::{{closure}}
  14: tokio::runtime::park::CachedParkThread::block_on
  15: tokio::runtime::runtime::Runtime::block_on
  16: cubestored::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

ChrisLahaye · 2024-04-26T13:41:55Z

@ovr the issue persists here as well, although it it slightly different it was introduced at the same release.

024-04-26T13:38:47.733Z INFO  [cubestored] <pid:1> Cube Store version 0.35.24
2024-04-26T13:38:47.740Z INFO  [cubestore::http::status] <pid:1> Serving status probes at 0.0.0.0:3031
thread 'main' panicked at /build/cubestore/cubestore/src/config/mod.rs:1971:34:
called `Result::unwrap()` on an `Err` value: CubeError { message: "AWS S3 error: serde xml: custom: missing field `Name`", backtrace: "", cause: Internal }
stack backtrace:
   0: rust_begin_unwind
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/core/src/result.rs:1649:5
   3: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
   4: cubestore::config::injection::Injector::get_service_typed::{{closure}}
   5: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
   6: cubestore::config::injection::Injector::get_service_typed::{{closure}}
   7: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
   8: cubestore::config::injection::Injector::get_service_typed::{{closure}}
   9: cubestored::main::{{closure}}
  10: tokio::runtime::park::CachedParkThread::block_on
  11: tokio::runtime::runtime::Runtime::block_on
  12: cubestored::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

adamstruck · 2024-04-26T16:13:59Z

@ovr I am still seeing errors with v0.35.24

2024-04-26T16:12:16.689Z INFO  [cubestored] <pid:1> Cube Store version 0.35.24
thread 'main' panicked at /build/cubestore/cubestore/src/config/mod.rs:1816:86:
called `Result::unwrap()` on an `Err` value: CubeError { message: "Failed to create S3 credentials: attohttpc: Json Error: expected value at line 1 column 1", backtrace: "", cause: Internal }
stack backtrace:
   0: rust_begin_unwind
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/190f4c96116a3b59b7de4881cfec544be0246d84/library/core/src/result.rs:1649:5
   3: cubestore::config::injection::Injector::register::{{closure}}::{{closure}}::{{closure}}
   4: cubestore::config::injection::Injector::get_service::{{closure}}
   5: cubestore::config::injection::Injector::register::{{closure}}::{{closure}}::{{closure}}
   6: cubestore::config::injection::Injector::get_service::{{closure}}
   7: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
   8: cubestore::config::injection::Injector::get_service_typed::{{closure}}
   9: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
  10: cubestore::config::injection::Injector::get_service_typed::{{closure}}
  11: cubestore::config::injection::Injector::register_typed::{{closure}}::{{closure}}::{{closure}}
  12: cubestore::config::injection::Injector::get_service_typed::{{closure}}
  13: cubestored::main::{{closure}}
  14: tokio::runtime::park::CachedParkThread::block_on
  15: tokio::runtime::runtime::Runtime::block_on
  16: cubestored::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

adamstruck · 2024-04-27T21:02:57Z

@ovr could the from_sts method just be reverted to what was in 0.26.3 (https://github.com/durch/rust-s3/blob/0.26.3/aws-creds/src/credentials.rs#L139-L141)?

adamstruck · 2024-05-08T19:27:25Z

I hacked around this issue by authenticating via STS in a separate process and storing the returned credentials in ~/.aws/credentials so that cubestore is able to use them.

paveltiunov · 2024-05-15T07:19:59Z

@adamstruck If it's still an issue, please feel free to provide PRs to https://github.com/cube-js/rust-s3 and to the Cube Store itself.

github-actions · 2024-05-15T07:20:26Z

If you are interested in working on this issue, please go ahead and provide PR for that.
We'd be happy to review it and merge it.
If this is the first time you are contributing a Pull Request to Cube, please check our contribution guidelines.
You can also post any questions while contributing in the #contributors channel in the Cube Slack.

ovr · 2024-05-15T15:50:35Z

@adamstruck Could you give a try with v0.35.34?

Thanks

adamstruck · 2024-05-15T16:28:34Z

@ovr looks the the issue is fixed v0.35.34 - thank you for finding a fix for this!

ovr · 2024-05-15T17:19:07Z

@adamstruck Thank you for testing then I am going to close this issue.

danielli-ziprecruiter · 2024-05-23T22:02:08Z

I'm still experiencing this issue in v0.35.38 with the same error message in this comment.

ovr added bug Something isn't working backend:cube-store Issues relating to Cube Store labels Apr 18, 2024

ovr self-assigned this Apr 18, 2024

adamstruck mentioned this issue Apr 24, 2024

fix(cubestore): AWS STS credentials by bumping rust-s3 0.32.3 -> 0.33.0 #8190

Closed

4 tasks

ovr mentioned this issue Apr 25, 2024

fix(cubestore): S3 - STS credentials (backport fix for 0.32.3) #8195

Merged

ovr closed this as completed in #8195 Apr 26, 2024

ChrisLahaye mentioned this issue Apr 26, 2024

Cubestore AWS Credentials required #8197

Closed

ovr reopened this Apr 26, 2024

danielli-ziprecruiter mentioned this issue May 1, 2024

AWS S3 error in Cubestore after upgrading to 0.35 #8218

Closed

paveltiunov added the help wanted Community contributions are welcome. label May 15, 2024

ovr mentioned this issue May 15, 2024

fix(cubestore): S3 - STS credentials (align aws-creds to 0.36 in fork) #8264

Merged

ovr referenced this issue in cube-js/rust-s3 May 15, 2024

chore: Use aws-creds from upstream

3fd5a88

ovr closed this as completed in #8264 May 15, 2024

ovr reopened this May 15, 2024

ovr closed this as completed May 15, 2024

igorlukanin reopened this May 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cubestore S3 credentials regression in v0.35.17 #8163

cubestore S3 credentials regression in v0.35.17 #8163

adamstruck commented Apr 17, 2024 •

edited

adamstruck commented Apr 17, 2024

ovr commented Apr 17, 2024

adamstruck commented Apr 17, 2024 •

edited

ovr commented Apr 18, 2024

adamstruck commented Apr 18, 2024

adamstruck commented Apr 19, 2024

ovr commented Apr 19, 2024

adamstruck commented Apr 20, 2024

adamstruck commented Apr 20, 2024

adamstruck commented Apr 23, 2024

ovr commented Apr 24, 2024 •

edited

adamstruck commented Apr 24, 2024

adamstruck commented Apr 24, 2024 •

edited

ChrisLahaye commented Apr 25, 2024

ovr commented Apr 25, 2024

ovr commented Apr 26, 2024

goncaloacteixeira commented Apr 26, 2024 •

edited

ChrisLahaye commented Apr 26, 2024

adamstruck commented Apr 26, 2024

adamstruck commented Apr 27, 2024

adamstruck commented May 8, 2024

paveltiunov commented May 15, 2024

github-actions bot commented May 15, 2024

ovr commented May 15, 2024

adamstruck commented May 15, 2024

ovr commented May 15, 2024

danielli-ziprecruiter commented May 23, 2024 •

edited

cubestore S3 credentials regression in v0.35.17 #8163

cubestore S3 credentials regression in v0.35.17 #8163

Comments

adamstruck commented Apr 17, 2024 • edited

adamstruck commented Apr 17, 2024

ovr commented Apr 17, 2024

adamstruck commented Apr 17, 2024 • edited

ovr commented Apr 18, 2024

adamstruck commented Apr 18, 2024

adamstruck commented Apr 19, 2024

ovr commented Apr 19, 2024

adamstruck commented Apr 20, 2024

adamstruck commented Apr 20, 2024

adamstruck commented Apr 23, 2024

ovr commented Apr 24, 2024 • edited

adamstruck commented Apr 24, 2024

adamstruck commented Apr 24, 2024 • edited

ChrisLahaye commented Apr 25, 2024

ovr commented Apr 25, 2024

ovr commented Apr 26, 2024

goncaloacteixeira commented Apr 26, 2024 • edited

ChrisLahaye commented Apr 26, 2024

adamstruck commented Apr 26, 2024

adamstruck commented Apr 27, 2024

adamstruck commented May 8, 2024

paveltiunov commented May 15, 2024

github-actions bot commented May 15, 2024

ovr commented May 15, 2024

adamstruck commented May 15, 2024

ovr commented May 15, 2024

danielli-ziprecruiter commented May 23, 2024 • edited

adamstruck commented Apr 17, 2024 •

edited

adamstruck commented Apr 17, 2024 •

edited

ovr commented Apr 24, 2024 •

edited

adamstruck commented Apr 24, 2024 •

edited

goncaloacteixeira commented Apr 26, 2024 •

edited

danielli-ziprecruiter commented May 23, 2024 •

edited