Selectable Index Patterns for Elasticsearch DataSources #56252
Replies: 8 comments
-
Came here to report a similar thing. We have 50+ index patterns, and multiple clusters, so having to make one datasource for each means hundreds of datasources (we can script that part, but managing them in the UI is not easy). Currently, especially with Index Lifecycle Management or aliases you don't even need to specify |
Beta Was this translation helpful? Give feedback.
-
Same here! |
Beta Was this translation helpful? Give feedback.
-
I would do that, but we have almost 12000 indices and more than 40k shards housing more than 200TB of logs, so that is not realistic. |
Beta Was this translation helpful? Give feedback.
-
After looking the UI and some discussion with our internal users, I think there are two viable options for overriding index pattern: Option OneFor this option, we would add the index pattern to the query options. This would allow the user to override the index pattern for the entire panel, similar to how the Option Two (prefered)For this options we would add an Index Pattern field next the the With both of these options I think just having a way to define a wildcard pattern would be sufficient. The index pattern definition with the date pattern isn't really necessary in Elasticsearch 5+ due to optimizations made in that release (and beyond). My initial thought was to add multiple patterns to the data source with a way to specify the default. This would require a lot more work because we would need to build an index pattern editor within the datasource and then have a way to select different patterns throughout Grafana. Overriding the pattern in the query builder seems like a good compromise. @Elfo404 Thoughts? |
Beta Was this translation helpful? Give feedback.
-
For those following along, I started looking at the code today to see what it would take to make this happen. To my surprise it's pretty minimal, you can check out the draft PR here: #29477 |
Beta Was this translation helpful? Give feedback.
-
Hello, as you may have heard, we are transitioning away from using discussions to discuss feature requests. We are migrating this discussion to an issue and closing the discussion. The issue is #83012. Feel free to continue the discussion around this there. Thank you! |
Beta Was this translation helpful? Give feedback.
-
Hi to all the people in this issue, as this feature got stall, i believe we should make a comment in the new discussion: to make some traction on the topic and so to help prioritise this feature. Thanks, |
Beta Was this translation helpful? Give feedback.
-
What would you like to be added:
Allow elasticsearch index name selection when creating a query for a panel (similar to namespace selection on a CloudWatch datasource.
Why is this needed:
Presently, when setting up an elasticsearch datasource, you must enter the index name as part of the datasource configuration. This means that even though accessing the same elasticsearch cluster, I need to manually define every single index pattern as its own datasource.
If instead there was only necessity to define the timeseries pattern format (EX
(.*)-YYYY.MM.DD
), it would be very easy to provide the user with a selectable list of index patterns that match per query using theGET /_aliases
endpoint on an elasticsearch cluster, or perhaps one of the/_cat/
endpoints depending on how the information is parsed.Beta Was this translation helpful? Give feedback.
All reactions