Indexed Fields - Splunk HEC integration - Event Endpoint Target

[tbenade]

Hoping someone can help because for the life of me I cannot get Vector to send Indexed Fields to Splunk and NOT have to include the fields in the Event message payload, which renders the exercise of indexing the fields futile.

Example given the following log payload,

{
    "event" : "{\r\n    \"message\": \"hello world!\",\r\n    \"timestamp\" : \"2023-11-16T02:49:38Z\",\r\n    \"level\": \"info\"\r\n}",
    "fields": {
        "zone" : "a",
        "instance" : "id-123"
    }
}

With fluentbit it is trivial to log the event property as the event in Splunk and index fields out of the fields property as fluent provides a event_key property. Once event_key is set you are guaranteed that is all that will be sent to Splunk. https://docs.fluentbit.io/manual/pipeline/outputs/splunk#configuration-parameters For the life of me I cannot see how this is possible with the Splunk sink in vector. Given in vector there is no way to specify the property you want posted as the event payload.

I have tried all sorts of things like nesting the fields property in the event and then removing it but nothing seems to work. Any suggestions would be much appreciated.

transforms:
    events_json_remap:
      inputs:
        - automat_events_json
      type: "remap"
      source: |
      .event = parse_json!(.event)
      .event.fields = del(.fields)
      . = .event
sinks:
    splunk_sink_json:
      type: splunk_hec_logs
      inputs:
        - events_json_remap
      compression: gzip
      endpoint: "https://http-inputs-acme.splunkcloud.com"
      endpoint_target: event
      index: "main"
      source: "splunk_sink_event"
      auto_extract_timestamp: true
      timestamp_key: "noname"
      sourcetype: "_json"
      default_token: "sssshhhh"
      encoding:
        codec: json
        except_fields: ["fields"]
      indexed_fields: ['"fields.zone"']

[tbenade]

Additional development since posting this discussion is that it appears adding any field with a . in it in the indexed_fields property causes the sink to implode.

[tbenade]

Mmmhh I need correct my inaccurate statement. The Splunk HEC Sink will implode if you supply any quoted path segments.

for example indexed_fields: ["complexity", 'fields."aws.az"'] will crash the sink. Not sure how to get the errors all I am getting in the logs is

Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.

[bruceg]

This sounds like a bug in the Splunk encoder. Could you file an incident with the relevant details?

[tbenade]

Further development. I don't believe it has much to do with the formatting of the field names but something to do with the processing of said fields and posting them the Splunk. I wish I understood the code I would be more helpful.

The Splunk HEC sink will only implode if you configure field names with a dot in that match a value in an event being processed. So if I configured a field name blah.blah which would match nothing the sink would continue to work indexed_fields: ["blah.blah"] but if I configured indexed_fields: ['x."aws.az"'] that would match a field in the event and immediately implode.