Vector Elasticsearch Sink variable index

[bebehei]

I'm currently trying to index into different indices on my configuration when using the Elasticsearch Sink. Like documented, I'm using a variable there:

[sinks.elasticsearch.bulk]
index = "{{elastic_index}}-%Y-%m-%d-%H"

The elastic_index is set here in the VRL before:

[transforms.add-elastic-index]
inputs        = [ "remote-vector" ]
type          = "remap"
drop_on_error = false

source        = '''
if .msg_type != null {
	elastic_index = .msg_type
	del(.msg_type)
} else {
        elastic_index = "cache-accesslogs"
}
'''

I'm receiving this in the vector error log:

Dec 18 18:00:08 nbg-stats-01 vector[295315]: 2023-12-18T18:00:08.921786Z ERROR sink{component_kind="sink" component_id=elasticsearch component_type=elasticsearch}: vector::internal_events::template: Failed to render template for "index". error=Missing fields on event: ["elastic_index"] error_type="template_failed" stage="processing" internal_log_rate_limit=true

Apparently elastic_index is not available as a variable anymore.

How can I pass the variable elastic_index? I assume, I have to use .elastic_index in the VRL. But this will push the variable unnecessarily onto the index, too.

Is it possible to pass variables between the different stages, which are not part of the event?

[jszwedko]

Hi @bebehei !

Variables defined in a remap transform only exist in that specific remap transform (and are independent for every event being process). I think what you want to do is assign the result to a field, as you suggested. To avoid that field ending up on the event in Elasticsearch you can use encoding.drop_fields = ["elastic_index"]. Hopefully this helps!

[bebehei]

Awesome, thank you for the fast response! It works now flawless!

Sidenote: According to current documentation, it's called except_fields: https://vector.dev/docs/reference/configuration/sinks/elasticsearch/#encoding.except_fields