New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Discussion: Tracking changes to index templates, component templates and ingest pipelines #108469
Comments
Pinging @elastic/es-data-management (Team:Data Management) |
My ideal scenario would be that Elasticsearch versions its assets like ingest pipeline, index templates which would not only allow to track changes (also historically) but allow to roll back changes. As this will be a massive effort, we should start simpler. Few constraints:
The above sounds a lot like an audit log. How much of this is captured today in audit logs? Instead of having this per data stream, could we have a global data stream for it with all the changes. If we have all the changes, it would allow Kibana to "stich togehther" the different changes and show it where relevant. For example if To start simple, only admin users would have access to the full changelog. Besides having the audit log, ideally also on the asset itself like ingest pipelines, the system would automatically update meta information around |
I can imagine this part getting complicated over time, but I agree that piping the audit log into a separate data stream seems like a good way to get started here. |
We already do have an audit log in ES, and it can be configured to emit request bodies, however, I would argue that its purpose is separate from the intended use-case for this. I think this new concept is more of a
To me it makes more sense to have this be global also. It could likely go into its own data stream.
We do have these for some of our configuration items (like ILM policies), and we can expand that list fairly easily. There's a little bit of a discussion around leaking username information in a |
I could see this as a low hanging fruit to get started as it at least would allow us to indicate the most recent change, no history. |
Agreed, this is a good starting point. Together with the rollover timestamp of individual datastreams this can probably go quite far in terms of providing visibility. |
Would it be worth it opening a separate more implementation focused issue around that? |
Yes, that would be useful, to keep this one a bit more focused. |
Split out this first step into #108754 |
Description
The "Logs+" initiative in Observability tries to make the experience around logs in the Elastic stack as seamless as possible.
An important part of this is detecting and mitigating ingestion issues. Most of the time ingestion issues start because something in the system changed. This can either be a change on the collection side or on the Elasticsearch side (mappings / ingest pipelines were rearranged, fleet integration packages got updated, ...)
When investigating an issue in this area, it would be very helpful to be able to understand what changes were made when things started to go south. There already is a very important building block for this - via the _ignored field and the failure store, it's possible to reconstruct when things started to act up.
The other important part is correlating the occurring errors with changes to the system - in a visual way, this is what I'm trying to get to:
It's already possible to plot the errors over time, what's challenging is to give the user access to the annotations - changes to the configuration of the system. However, having access to this information and correlating both signals should speed up time-to-resolution a lot in a lot of cases. Having this information also would allow to automate or at least to simplify getting back to a working system by rolling back applied changes.
Some rough ideas / thoughts:
.changes
index which is written to each time an index template matching the stream, a component template referenced in this index template or an ingest pipeline referenced in it is updatedAny thoughts @ruflin @dakrone @felixbarny ?
The text was updated successfully, but these errors were encountered: