Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Update input package data stream check to only look for Fleet-managed data streams #183526

Open
kpollich opened this issue May 15, 2024 · 1 comment
Labels
Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@kpollich
Copy link
Member

Currently, this check will return true if any managed data stream exists with the same index pattern defined by an input package policy:

const existingDataStreamsAreFromDifferentPackage = existingDataStreams.some(
(ds) => ds._meta?.package?.name !== pkgInfo.name
);

The intent behind this check is to prevent users from routing custom logs documents to other integration data streams that have their own index settings, mappings, etc defined. Pushing custom logs documents into an integration's ingest pipeline could result in pipeline errors, for example.

We should potentially only be checking Fleet-managed (e.g. managed_by: "fleet") data streams instead of any managed data stream matching the given index pattern.

In cases where users ingest data outside of the context of Fleet, this check can result in false positives, e.g.

  1. Run the following in Kibana dev tools
POST logs-generic-default/_doc
{
  "@timestamp": "2024-05-15T00:00:00.000Z",
  "message": "foo"
}
  1. Check the data stream that was created, e.g.
GET _data_stream/logs-*

// Response
{
  "data_streams": [
    {
      "name": "logs-generic-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-logs-generic-default-2024.05.15-000001",
          "index_uuid": "dqi4bbNAR0CJ1WtdgIgVgQ",
          "prefer_ilm": true,
          "ilm_policy": "logs",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 1,
      "_meta": {
        "description": "default logs template installed by x-pack",
        "managed": true
      },
      "status": "YELLOW",
      "template": "logs",
      "ilm_policy": "logs",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false,
      "failure_store": {
        "enabled": false,
        "rollover_on_write": false,
        "indices": []
      }
    }
  ]
}
  1. Attempt to create a custom logs policy using the generic dataset

image

  1. Observe the error thrown in the code path linked above

image

@kpollich kpollich added the Team:Fleet Team label for Observability Data Collection Fleet team label May 15, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Development

No branches or pull requests

2 participants