You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The installation of Prometheus in a medium or larger system landscape takes some time. Meanwhile, the configuration will be constantly changing, so the metrics collected during this time will be somewhat incompatible.
I think that this is normal, and the data of the development phase can only be used in a limited way - if at all.
If you look at the time series in - e.g. - Grafana, you get "strange" effects if you extend the time range to include the setup phase.
But how to proceed if you change Prometheus configurations and especially labels later?
In relational databases, schema migration scripts are executed in such cases, which may also change the data.
Is something like this possible or desirable in Prometheus?
I was thinking of adding a version label "ver=XYZ" to all targets, which will be changed if the configuration changes in an incompatible way. In Grafana, one could add a variable "version" to the dashboards, in which the current version is entered. All PROMQL queries would need to be extended to include the expression {ver="$version}.
The effect would be that queries in Grafana would only consider the compatible time series, but the question is whether the effort is worth it.
So my question is: are there best practices for Prometehus to deal with incompatible data or configuration changes?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hi everyone,
The installation of Prometheus in a medium or larger system landscape takes some time. Meanwhile, the configuration will be constantly changing, so the metrics collected during this time will be somewhat incompatible.
I think that this is normal, and the data of the development phase can only be used in a limited way - if at all.
If you look at the time series in - e.g. - Grafana, you get "strange" effects if you extend the time range to include the setup phase.
But how to proceed if you change Prometheus configurations and especially labels later?
In relational databases, schema migration scripts are executed in such cases, which may also change the data.
Is something like this possible or desirable in Prometheus?
I was thinking of adding a version label "ver=XYZ" to all targets, which will be changed if the configuration changes in an incompatible way. In Grafana, one could add a variable "version" to the dashboards, in which the current version is entered. All PROMQL queries would need to be extended to include the expression {ver="$version}.
The effect would be that queries in Grafana would only consider the compatible time series, but the question is whether the effort is worth it.
So my question is: are there best practices for Prometehus to deal with incompatible data or configuration changes?
Beta Was this translation helpful? Give feedback.
All reactions