Intermittent "error":"vector cannot contain metrics with the same labelset"

[lucasvscosta96]

I'm using Opencost version 1.90.0
Thanos version 0.35.0

I'm seeing intermittent "error": vector cannot contain metrics with the same labelset
Query: avg(avg_over_time(node_ram_hourly_cost{}[1d])) by (node, cluster_id, instance-type, provider_id)

[mattray]

Is that OpenCost 1.90 or 1.109?

[lucasvscosta96]

1.109.0

[mattray]

How is this expressed in the application? Are your node RAM graphs missing?

[lucasvscosta96]

No.
When UI is accessed, in console log return 500 with this message.
I'm suspicious of the time zone. Opencost is running in UTC but I need running in UTC-3, then, I put variable environment TZ = "America/Sao_Paulo", but it doesn't work

[lucasvscosta96]

I found this Warn:
WRN Cost model.ComputeAllocation: Node GPU Cost Query result missing field: " 'instance_type' field does not exist in data result vector" for node fargate

If I try put query on Thanos return successful

[mattray]

You're running Thanos instead of Prometheus? I'm not very familiar with that configuration and it's undocumented. Is that the only piece not working?

[lucasvscosta96]

Yes.
I'm using Thanos instead Prometheus because is not possible make query in Prometheus running in agent mode

[mattray]

OK, so you've got OpenCost running and the Prometheus agent scraping it and forwarding to Thanos? The typical setup is to have the opencost container scraped by Prometheus then queries to the OpenCost API are Prometheus queries behind the API calls. Did you deploy with the Helm chart and setup the external Prometheus?