Unstable on-demand metric extraction

[obostjancic]

Starting ~27th of March this alert counting the number of received on demand metrics started occasionally firing.
Most drops last for a just a couple of minutes with some of them being >10mins. These latter ones are also visible in charts that depend on on demand metrics.

We already eliminated cardinality and rate limits as potential causes and have no indication that project config is unavailable when this occurs. (All project config related metrics we have show no correlation and we also saw generated project configs during one of the drops)

Extraction of standard metrics does not seem to be affected.

Rough approximation between the counts of regular transaction metrics and on-demand metrics: link

Count of on-demand metrics: link

[iker-barriocanal]

have no indication that project config is unavailable when this occurs. (All project config related metrics we have show no correlation and we also saw generated project configs during one of the drops)

Could you expand on how this was measured? I wonder if Relay could have stale configs without required entries.

[Dav1dde]

I confirmed that metricExtractions is missing from the project config for a short period of time, by grabbing a quick snapshot when the error triggered.

@k-fish is investigating.