feat(processors.parser): Add base64 decode option

[RodrigoDornelles]

Use Case

can be used to decode in specific fields, remove old fields parsed and add preffix/suffix.

actual

[[processors.strings]]
  [[processors.strings.base64decode]]
    field = "data"

[[processors.parser]]
  parse_fields = ["data"]
  merge = "override"
  data_format = "json"

proposal

[[processors.parser]]
  parse_fields = ["data"]
  merge = "override"
  data_format = "json"
  json_base64_decode = true 
  json_exclude_fields = true
  json_prefix_fields = "new_"

[[processors.parser]]
  parse_fields = ["data_old"]
  merge = "override"
  data_format = "json"
  json_base64_decode = true 
  json_exclude_fields = true
  json_prefix_fields = "old_"

Expected behavior

which base-64 decodes before reading the json string.

Actual behavior

Additional info

It is interesting because when using a cross data format, it can end up shortening the work. such as a jsonbase64 field in my line protocol (influx).

[srebhan]

@RodrigoDornelles I think if we do this it should be a property of the processors.parser and not of the parser itself...

[powersj]

After talking to the team, we decided we would be cool with adding an option to the parser processor to be able to base64 decode values. We would take a list of fields to decode, which must be a subset of the parse_fields option.

Happy to see a PR.

[RodrigoDornelles]

@powersj I can try to create a PR, I'm still learning Golang, can you give an example of how you would like the configuration to be?

[powersj]

Sure, here is a breakdown of what needs to happen:

First, add the config options to the sample.conf:

  ## Fields to base64 decode
  ## This list of fields must be a subset of parse_fields. Fields specified
  ## here will have base64 decode applied to them.
  # base64_fields = []

Then run make docs or add this manually to the README.md to keep the two in sync.

Second, add the new option to the parser processors' struct with toml tags.

Base64Fields []string toml:"base64_fields"`

Third, you would want to verify in the parser's Init() function that Base64Fields is a subset of ParseFields. Then in Apply update the logic to check for Base64Fields and decode them.

Finally, add some tests that cover both decoding some fields, as well as a negative test were the Base64Fields is not a subset of ParseFields.

Hope that helps!