OpenCost API and UI

[kumaran525]

For the past two weeks, I've been using OpenCost without any problems. However, in the last six days, I've faced an issue where metrics are no longer visible in the UI and cannot be accessed via the API. This issue has occurred previously, and restarting the pod resolved it. However, I'm reluctant to do that again now as it may result in the loss of data from previous days.Also I've enabled export csv feature but that also not working as expected.
I am using Azure service provider.

To Reproduce
Steps to reproduce the behavior:
I've followed the opencost setup installation https://www.opencost.io/docs/installation/install guidelines.

Expected behavior
Metrics needed to be rendered in UI and able to get via API also.

Screenshots
Please refer the attachments.

Which version of OpenCost are you using?
Latest Image

Additional context
Add any other context about the problem here. Kubernetes versions and which public clouds you are working with are especially important.

[mattray]

I assume you used the default Prometheus installation? Could you check on its health and see that it's working properly?

There was a build that was accidentally promoted to latest, I don't know if that's causing the issue but could you set your version to 1.109.0 just to ensure we know what you're using? The data is kept in the Prometheus pod, so restarting OpenCost won't lose data assuming Prometheus is working right.

[kumaran99]

Hey @mattray, following your advice, I've checked the Prometheus health. It appears that after port-forwarding on my local machine, I can successfully query. However, the data retrieved only extends until March 27th. Since the Opencost dashboard also shows data until March 27th, we can infer that Opencost is functioning properly. Therefore, it seems the issue lies with Prometheus, right?

image: "quay.io/prometheus/prometheus:v2.50.1".

This is the image I am using for prometheus-server. should I update this because so far I've seen every pod is running properly.

[mattray]

My suggestion is to set OpenCost to 1.109.0 and restart the pod. Prometheus scrapes OpenCost and acts as the database (OpenCost then queries Prometheus for API results), so in your case it looks like OpenCost isn't providing data to Prometheus. Restarting OpenCost should fix the issue and you'll have a few days missing in Prometheus (you should know about 10 minutes after restarting OpenCost).

[kumaran525]

Hello @mattray, despite changing the opencost image tag to 1.109.0, I'm still encountering the same issue. Opencost is only displaying data until March 27, and opencost logs indicate that it's unable to query. I'm puzzled as to why Opencost can query or calculate metrics until March 27 but not after. Should it not display data before March 27 as well? Can you assist me in identifying where I may be missing something?

[mattray]

This seems really strange. Did something happen to your Prometheus? At this point I don't have any good suggestions, you're coming up on 2 weeks of no data and the Prometheus installation only keeps 15 days by default. I think reinstalling Prometheus following the instructions closely is the only suggestion I have, which will lose the little data that you have.