Option to selectively include fields included in File output

[arvchristos]

Describe the solution you'd like
In the context of pdnssoc, we are storing all logs ingested from go-dnscollector in JSON files. In order to save storage space, we are using a script to remove fields not needed for our correlation and store only the minified logs. This seems like a quite common scenario so what I would suggest is:

• Providing a way to select or exclude fields from logging in the File output.

I am not entirely sure whether having this in the File output or implementing a new transformer is the optimal approach.

Describe alternatives you've considered
So far we are using a postrotate command that is opening the rotated file, running jq over it and selecting only specific fields. This is clearly sub-optimal and is impacting performance on large input as reported at #636

[dmachard]

We need to find a generic way to do that. The loki logger support this type of relabeling, it based on https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#relabel_configs

Perhaps we can do the same through a new transformer for all type of loggers.

relabelConfig: []*relabel.Config{
{
	Action: relabel.LabelDrop,
	Regex:  relabel.MustNewRegexp("job"),
},
},
labelsPattern: "{identity=\"test_id\"}",

[dmachard]

PR in progress #645

[dmachard]

implemented in next release v0.43.0
documentation here

[arvchristos]

Thank you very much @dmachard for this. Seems amazing.

Is there any specific reason why you choose to regex match the fields instead of just applying equality checks? I may be missing a clear use case but the performance of regex matching every single dns message concerns me.

[dmachard]

Regex are more efficient to delete multiple keys at once. It can be optimized if necessary. The impact in benchmark seems to be minimal but my regex are simple.