Poor performance when trying to select span name

[RonanQuigley]

If I have a large number of span names to retrieve, the UI will struggle to serve the span name dropdown quickly. It will take seconds to return the span names or just fail.

I don't see any obvious issues in the tempo setup that could cause this. Here's the CPU metrics.

Is there anything I can tweak to improve performance here e.g. limit the returned number of span names? The limit input field makes no difference

[joe-elliott]

What version of Tempo and Grafana are you running?
Can you use the network tab to see what endpoint Grafana is hitting? How long is taking the endpoint returning?

If I have a large number of span names to retrieve

What is a "large number"?

[joe-elliott]

Shall I try update Grafana to the latest version to see it fixes this?

Yup. I think that's your best bet. Current Grafana has made a lot of performance improvements with regard to loading longer select lists.

Furthermore, if I select a specific service, it returns span names that are unrelated to that service, so you can't workaround this problem by filtering by service name first.

Also, Tempo has added a v2 of the attribute values endpoint. This allows filtering by other fields. So in current Grafana selecting a service first will nicely scope down your span names.

[RonanQuigley]

Hey, we updated to Grafana v11.0.0 (b12ad73c52) and are still seeing issues. It is now using the v2 endpoint, but the UI will stall. It's therefore hard for me to tell if the span names are indeed being scoped down.

Shall I raise this over in the main Grafana repo? There needs to be a cap placed on how many span names are listed. This is what we could see for tags as an example.

.

[joe-elliott]

Yeah, I think Grafana is trying to render all 78K+ names at once. Filing an issue in the Grafana repo is your best bet.

Can you share how long it is taking Tempo to return that information?

[RonanQuigley]

I've submitted a PR to fix this

grafana/grafana#88569