You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Suppose a project uses scrape package as a library in a cluster where targets are sharded across a number of instances. When a new instance joins the cluster, some portion of targets will typically be handed over to that new instance. The previous owning instance of a target that moved will attempt to inject staleness markers at the end of the scrape loop run (after delay of ~2 intervals). If out-of-order ingestion is enabled, writing staleness markers will succeed and results in occasional gaps in metrics.
Expose a method on scrape manager that allows users to disable end-of-run staleness for user-specified targets.
In above scenario, users of the scrape package will be able to detect that given target has moved to another instance and ensure that there are no unnecessary staleness markers written by the previous owning instance.
More generally, giving users control over when the staleness markers injection should or should not happen, can allow to implement more use cases correctly and with fewer data issues.
The text was updated successfully, but these errors were encountered:
Use case
Suppose a project uses
scrape
package as a library in a cluster where targets are sharded across a number of instances. When a new instance joins the cluster, some portion of targets will typically be handed over to that new instance. The previous owning instance of a target that moved will attempt to inject staleness markers at the end of the scrape loop run (after delay of ~2 intervals). If out-of-order ingestion is enabled, writing staleness markers will succeed and results in occasional gaps in metrics.Grafana Alloy faces the above issue in practice.
Proposal
Expose a method on scrape manager that allows users to disable end-of-run staleness for user-specified targets.
In above scenario, users of the
scrape
package will be able to detect that given target has moved to another instance and ensure that there are no unnecessary staleness markers written by the previous owning instance.More generally, giving users control over when the staleness markers injection should or should not happen, can allow to implement more use cases correctly and with fewer data issues.
The text was updated successfully, but these errors were encountered: