Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scrape: Allow to disable end-of-run staleness marker injection for targets #14049

Open
thampiotr opened this issue May 3, 2024 · 1 comment

Comments

@thampiotr
Copy link

thampiotr commented May 3, 2024

Use case

Suppose a project uses scrape package as a library in a cluster where targets are sharded across a number of instances. When a new instance joins the cluster, some portion of targets will typically be handed over to that new instance. The previous owning instance of a target that moved will attempt to inject staleness markers at the end of the scrape loop run (after delay of ~2 intervals). If out-of-order ingestion is enabled, writing staleness markers will succeed and results in occasional gaps in metrics.

Grafana Alloy faces the above issue in practice.

Proposal

Expose a method on scrape manager that allows users to disable end-of-run staleness for user-specified targets.

In above scenario, users of the scrape package will be able to detect that given target has moved to another instance and ensure that there are no unnecessary staleness markers written by the previous owning instance.

More generally, giving users control over when the staleness markers injection should or should not happen, can allow to implement more use cases correctly and with fewer data issues.

@GiedriusS
Copy link
Contributor

I guess #13302 is partially related?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants