Configuration option to send Buffer in newest-to-oldest order

[kj4tmp]

Use Case

The telegraf buffer is sometimes used to temporarily accommodate high write rates to an influxdb instance.

Sometimes it is preferred to see the newest data in influxdb first and backfill the older data until the production rate decreases.

Expected behavior

I expected to see a configuration option for the buffer to allow it to flush the newest data first instead of the oldest data.

Actual behavior

Only the oldest data is flushed first.

Additional info

No response

[powersj]

Hi,

The telegraf buffer is sometimes used to temporarily accommodate high write rates to an influxdb instance.

How long is your output down such that you actually see the impact of FIFO occur? Can you provide some logs giving an example? I am initially hesitant to treat the buffer like a stack versus a queue without understanding if your stack would ever flush in your situation.

fwiw, the running outputs will request a batch from the buffer, and then the buffer returns a batch size or less slice of metrics to write.

[kj4tmp]

Our application requires accommodating bursts of writes exceeding 1 million field writes / second (which is the typical limit of a single node influxdb OSS instance) for up to 20 minutes, while also reporting live down-sampled data. This can be accomplished with a telegraf buffer of about 30 GB (on our machine).

The FIFO-ness of the telegraf buffer means if we do nothing, the data in influx will be about 10 minutes behind by the end of the 20 minute burst, violating the live down-sampled data reporting requirement.

One option available to satisfy this live + buffering requirement is to use multiple influxdb output plugins and pipe the live downsampled data to a dedicated output plugin instance using some tag-based routing with aggregator.final and a processor to strip the final back off the aggregator output

###############################################################################
#                       AGGREGATOR PLUGINS                                    #
###############################################################################

# 10 Hz Aggregator
[[aggregators.final]]
  # alias = "10-Hz-Aggregator"
  ## The period on which to flush & clear the aggregator.
  period = "0.1s"

  ## If true, the original metric will be dropped by the
  ## aggregator and will not get sent to the output plugins.
  # drop_original = false

  ## The time that a series is not updated until considering it final. Ignored
  ## when output_strategy is "periodic".
  # series_timeout = "5m"

  ## Output strategy, supported values:
  ##   timeout  -- output a metric if no new input arrived for `series_timeout`
  ##   periodic -- output the last received metric every `period`
  output_strategy = "periodic"
  [aggregators.final.tags]
    aggregator = "10-Hz-Aggregator"

###############################################################################
#                       PROCESSOR PLUGINS                                     #
###############################################################################

# Trim the _final stuff from the aggregator plugin
[[processors.regex]]
  ## Other configurations for the processor...

  ## Rename metric fields to strip "_final" suffix
  [[processors.regex.field_rename]]
    ## Regular expression to match on the field name
    pattern = "(.*)_final$"
    ## Replacement expression defining the name of the new field
    replacement = "${1}"

  [processors.regex.tagpass]
    aggregator = ["10-Hz-Aggregator"]

the solution could be simpler if the FIFO buffer could be configured as LIFO. Though it may have some un-forseen consequences for users who rely on the "last-write wins" aspect of how influxdb handles duplicate data.

[kj4tmp]

here is a relevant issue
#5633

[powersj]

Thanks for the background it does help to understand the situation and desire for this better. My initial concerns are two fold:

First, that some output simply do not work with sending newer data first. For example, stackdriver absolutely requires data to be sent in order, oldest to newest. We have had issues come up in the past where this was not occurring. Adding an option like this, while opt-in, would prevent the usage of certain plugins and we would need to make that very clear somehow.

Second, I do have a concern with using the buffer like a stack, in that it is possible some metrics would get lost or sit in the stack whereas the queue ensures nothing stays in there forever.

That said, because this is opt-in, I am not opposed to adding some sort of option to allow this behavior. We are currently working on a rework of the buffer implementation to allow us to write to files and not just in-memory. I would want to make this sort of change after that work is landed.

next steps: continue work on buffer implementation, get initial re-structure in place, and look to add an option in how the buffer is read from.