Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"QueryAthenaPaginated: start query error: not found, ResolveEndpointV2" error when enabling aws cloud cost #2697

Open
driosalido opened this issue Apr 9, 2024 · 6 comments
Open

"QueryAthenaPaginated: start query error: not found, ResolveEndpointV2" error when enabling aws cloud cost #2697

driosalido opened this issue Apr 9, 2024 · 6 comments
Assignees
@cliffcolvin
Labels
E2 Estimated level of Effort (1 is easiest, 4 is hardest) kubecost Relevant to Kubecost's downstream project needs-follow-up needs-triage opencost OpenCost issues vs. external/downstream P2 Estimated Priority (P0 is highest, P4 is lowest)

Comments

@driosalido
Copy link

Describe the bug

When trying to enable aws cloud cost integration the Athena queries fail with GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2 error.

To Reproduce
Steps to reproduce the behavior:

  1. Follow kubecost documentation to enable CUR and Athena Reports

  2. Install via helm chart with the following values

helm upgrade --install opencost opencost-charts/opencost --namespace opencost -f values.yaml

values.yaml content

loglevel: debug

serviceAccount:
  create: false
  name: opencost

opencost:
  cloudCost:
    enabled: true

  customPricing:
    enabled: true
    provider: aws
    costModel:
      description: AWS Provider Configuration. Provides default values used if instance type or spot information is not found.
      CPU: 0.031611
      spotCPU: 0.006655
      RAM: 0.004237
      spotRAM: 0.000892
      GPU: 0.95
      storage: 0.00005479452
      zoneNetworkEgress: 0.01
      regionNetworkEgress: 0.01
      internetNetworkEgress: 0.143

      #Spot Data
      projectID: "<REDACTED>"
      spotLabel: karpenter.sh/capacity-type
      spotLabelValue: spot
      awsSpotDataRegion: eu-west-1
      awsSpotDataBucket: <REDACTED>-spot-datafeed-subscription
      awsSpotDataPrefix: spot_datafeed_subscription

      #Cloud Cost
      athenaBucketName: "s3://<REDACTED>-kubecost-report-athena-query-results"
      athenaDatabase: athenacurcfn_kubecost_report
      athenaProjectID: "<REDACTED>"
      athenaRegion: eu-west-1
      athenaTable: kubecost_report
      athenaWorkgroup: primary

  prometheus:
    internal:
      enabled: true
      serviceName: prometheus-operated
      namespaceName: monitoring
      port: 9090
  ui:
    enabled: true
  metrics:
    serviceMonitor:
      enabled: true
      namespace: monitoring

All the Athena queries fails with the same ResolveEndopointV2 error

opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.537741921Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]: Starting build back to 2024-03-25 00:00:00 +0000 UTC in blocks of 7d
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.629201778Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.630094366Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.630974887Z DBG AthenaQuerier[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: Performing Query: SELECT column_name FROM information_schema.columns WHERE table_schema = 'athenacurcfn_kubecost_report' AND table_name = 'kubecost_report'
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.632299221Z ERR CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build failed for window [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000): GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.632336757Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]:  [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000) in 3.164226ms
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.632362701Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.63238635Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.641906507Z DBG AthenaQuerier[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: Performing Query: SELECT column_name FROM information_schema.columns WHERE table_schema = 'athenacurcfn_kubecost_report' AND table_name = 'kubecost_report'
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.64239071Z ERR CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build failed for window [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000): GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.64247167Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]:  [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000) in 10.110034ms
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.642554702Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-03-25T00:00:00+0000, 2024-03-26T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.64259483Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-03-25T00:00:00+0000, 2024-03-26T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.642627046Z DBG AthenaQuerier[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: Performing Query: SELECT column_name FROM information_schema.columns WHERE table_schema = 'athenacurcfn_kubecost_report' AND table_name = 'kubecost_report'
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.64287647Z ERR CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build failed for window [2024-03-25T00:00:00+0000, 2024-03-26T00:00:00+0000): GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.643310785Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]:  [2024-03-25T00:00:00+0000, 2024-03-26T00:00:00+0000) in 755.261µs
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.644030854Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]: completed in 106.316763ms
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.537787095Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: Run[jQvAe]: running month-to-date update starting at 2024-04-01 00:00:00 +0000 UTC
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.645031307Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.645542925Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.646072828Z DBG AthenaQuerier[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: Performing Query: SELECT column_name FROM information_schema.columns WHERE table_schema = 'athenacurcfn_kubecost_report' AND table_name = 'kubecost_report'
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.66812276Z ERR CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build failed for window [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000): GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2

Expected behavior
Opencost being able to query Athena to get cloud cost data

Screenshots
Not applicable

Which version of OpenCost are you using?
Helm Chart 1.33.2
OpenCost 1.109.0

Additional context
The same configuration and Athena setup works fine with Kubecost. The ServiceAccount IRSA created has the same permissions that the IRSA created for kubecost.
@driosalido driosalido changed the title QueryAthenaPaginated: start query error: not found, ResolveEndpointV2 when enabling aws cloud cost "QueryAthenaPaginated: start query error: not found, ResolveEndpointV2" error when enabling aws cloud cost Apr 9, 2024
@driosalido
Copy link
Author

driosalido commented Apr 9, 2024 •
edited

Logs of kubecost getting data from Athena using the same athena isntance, configuration and permissions

kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:33.476703066Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-04-08T00:00:00+0000, 2024-04-10T00:00:00+0000)
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:34.568892453Z INF CloudCost: IngestionManager: deleting integration with key: <REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360051004Z INF CloudCost: IngestionManager: creating integration with key: <REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360145526Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[aBYcC]: Starting build back to 2024-01-09 00:00:00 +0000 UTC in blocks of 7d
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360339902Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[aBYcC]:  [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000) in 99.079µs
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360402412Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: Run[aBYcC]: running month-to-date update starting at 2024-04-01 00:00:00 +0000 UTC
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360441757Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000)
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360445353Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[aBYcC]:  [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000) in 81.531µs
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360460343Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000)

@AjayTripathy
Copy link
Contributor

Hi @driosalido this should have been fixed-- what version are you using?

@mattray mattray added opencost OpenCost issues vs. external/downstream P2 Estimated Priority (P0 is highest, P4 is lowest) kubecost Relevant to Kubecost's downstream project E2 Estimated level of Effort (1 is easiest, 4 is hardest) labels Apr 10, 2024
@driosalido
Copy link
Author

image: ghcr.io/opencost/opencost:1.109.0

@mattray
Copy link
Collaborator

mattray commented Apr 10, 2024

Per Slack discussion, is aws/aws-sdk-go-v2#2370 related? OpenCost 1.109.0 appears to have github.com/aws/aws-sdk-go-v2 v1.24.1

@asuprabhat
Copy link

I'm also facing same issue and with same version of opencost v1.109.0
As an workaround i used opencost v2.2.0 [ which has resolved ResolveEndpointV2 error ] but seems like v2.2.0 is removed as per this #2683

@Veeraraghavans Veeraraghavans mentioned this issue Apr 15, 2024
Closed
@mattray
Copy link
Collaborator

mattray commented Apr 16, 2024

@asuprabhat that build was an accidental release, we should be putting out OpenCost 1.110.0 soon. The 2.2 builds were inadvertent Kubecost-tagged builds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cliffcolvin cliffcolvin

Labels
E2 Estimated level of Effort (1 is easiest, 4 is hardest) kubecost Relevant to Kubecost's downstream project needs-follow-up needs-triage opencost OpenCost issues vs. external/downstream P2 Estimated Priority (P0 is highest, P4 is lowest)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mattray @AjayTripathy @driosalido @cliffcolvin @asuprabhat