feat(new encoding): add pretty json encoding #20384

lsampras · 2024-04-26T19:13:58Z

Adds a use_pretty_json field to json encoding codec
this relies on the serde_json::to_writer_pretty implementation while using the same deserialization as plain json

Fixes : #20380

example

sources:
  dummy_logs:
    type: "demo_logs"
    format: "syslog"
    interval: 1
# Parse Syslog logs
# See the Vector Remap Language reference for more info: https://vrl.dev
transforms:
  parse_logs:
    type: "remap"
    inputs: ["dummy_logs"]
    source: |
      . = parse_syslog!(string!(.message))
# Print parsed logs to stdout
sinks:
  print:
    type: "console"
    inputs: ["parse_logs"]
    encoding:
      codec: "json"
      json:
        use_pretty_json: true

bits-bot · 2024-04-26T19:14:03Z

All committers have signed the CLA.

jszwedko

Thanks @lsampras ! This generally looks good to me as an addition granted it has somewhat limited utility outside of manual inspection. I think you'll just need to update the output assertions in the tests you copied.

Also, I'm curious if you considered making this an optional parameter to the existing json codec rather than a separate codec. I think I could go either way on it so just curious if you considered it and what your thoughts were if so.

lsampras · 2024-04-30T07:02:11Z

Also, I'm curious if you considered making this an optional parameter to the existing json codec rather than a separate codec. I think I could go either way on it so just curious if you considered it and what your thoughts were if so.

Hadn't considered adding a parameter to existing json codec, tbh I wasn't aware about the possibilty of adding more fields inside a codec...

lsampras · 2024-05-02T14:37:36Z

@jszwedko Can you take a look at this again?
Should I be adding a field in the json codec instead of a new codec?

jszwedko · 2024-05-03T18:25:13Z

Also, I'm curious if you considered making this an optional parameter to the existing json codec rather than a separate codec. I think I could go either way on it so just curious if you considered it and what your thoughts were if so.

Hadn't considered adding a parameter to existing json codec, tbh I wasn't aware about the possibilty of adding more fields inside a codec...

You can see an example with the AVRO codec: https://vector.dev/docs/reference/configuration/sinks/amqp/#encoding.avro

@jszwedko Can you take a look at this again? Should I be adding a field in the json codec instead of a new codec?

I'm having trouble coming up with super strong arguments either way but I think I have a slight preference to have it just be an option on the existing json codec since it is really just controlling "formatting" of an existing encoding. I could see us adding other formatting options to the other existing codecs where having pretty_json be its own codec would become more of an outlier. Do you think you that's something you might be able to take on? Apologies for not suggesting it sooner in the issue you had opened.

pront · 2024-05-03T19:30:23Z

@jszwedko Can you take a look at this again? Should I be adding a field in the json codec instead of a new codec?

I'm having trouble coming up with super strong arguments either way but I think I have a slight preference to have it just be an option on the existing json codec since it is really just controlling "formatting" of an existing encoding. I could see us adding other formatting options to the other existing codecs where having pretty_json be its own codec would become more of an outlier. Do you think you that's something you might be able to take on? Apologies for not suggesting it sooner in the issue you had opened.

+1 on adding a new option (vs introducing a new encoder).

Happy to help with implementation details if you get stuck. As Jesse mentioned, this https://github.com/vectordotdev/vector/blob/v0.37.1/lib/codecs/src/encoding/format/avro.rs#L42-L52 is a very good example.

… a new pretty_json codec

jszwedko

Looking good! I just have one comment about the option name, but otherwise this looks good to me.

jszwedko · 2024-05-06T18:14:11Z

config/vector.yaml

- codec: "pretty_json"
+ codec: "json"
+ json:
+ use_pretty_json: true


Suggested change

use_pretty_json: true

pretty: true

I think we can shorten this up a bit.

jszwedko

Great, thank you @lsampras !

github-actions · 2024-05-08T20:37:29Z

Regression Detector Results

Run ID: 1c54ce85-a4c3-4ae1-819d-1a6978db68bc
Baseline: e1d1e85
Comparison: f74d488
Total CPUs: 7

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	socket_to_socket_blackhole	ingress throughput	+2.31	[+2.22, +2.41]
➖	splunk_hec_route_s3	ingress throughput	+2.09	[+1.63, +2.56]
➖	otlp_http_to_blackhole	ingress throughput	+1.94	[+1.81, +2.06]
➖	otlp_grpc_to_blackhole	ingress throughput	+1.06	[+0.97, +1.15]
➖	http_elasticsearch	ingress throughput	+0.98	[+0.89, +1.08]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	+0.26	[+0.17, +0.36]
➖	http_to_http_noack	ingress throughput	+0.18	[+0.09, +0.28]
➖	syslog_regex_logs2metric_ddmetrics	ingress throughput	+0.12	[+0.01, +0.23]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	+0.11	[+0.02, +0.19]
➖	datadog_agent_remap_datadog_logs	ingress throughput	+0.08	[-0.02, +0.19]
➖	syslog_log2metric_humio_metrics	ingress throughput	+0.05	[-0.12, +0.22]
➖	http_to_http_json	ingress throughput	+0.02	[-0.05, +0.10]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	-0.00	[-0.15, +0.14]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.00	[-0.14, +0.13]
➖	syslog_splunk_hec_logs	ingress throughput	-0.01	[-0.10, +0.08]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.02	[-0.13, +0.09]
➖	enterprise_http_to_http	ingress throughput	-0.08	[-0.13, -0.02]
➖	syslog_humio_logs	ingress throughput	-0.09	[-0.23, +0.04]
➖	http_to_http_acks	ingress throughput	-0.21	[-1.57, +1.15]
➖	datadog_agent_remap_blackhole	ingress throughput	-0.25	[-0.35, -0.16]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	-0.29	[-0.45, -0.14]
➖	http_to_s3	ingress throughput	-0.32	[-0.60, -0.04]
➖	syslog_loki	ingress throughput	-0.34	[-0.42, -0.27]
➖	fluent_elasticsearch	ingress throughput	-0.42	[-0.90, +0.07]
➖	http_text_to_http_json	ingress throughput	-0.44	[-0.58, -0.30]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	-0.60	[-0.70, -0.49]
➖	file_to_blackhole	egress throughput	-1.16	[-3.76, +1.44]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

lsampras · 2024-05-14T09:26:45Z

@jszwedko Can you help here? I'm unable to understand the failure
found the following logs in the actions

The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

lsampras added 4 commits April 26, 2024 12:53

feat(new encoding): add pretty json encoding

6057dc7

chore(lint): fix cargo fmt lints

c7cd9c8

feat(new codec): update default console logger to use pretty json

4c4aa1f

docs(new codec): update docs for pretty json

ed49709

lsampras requested review from a team as code owners April 26, 2024 19:13

github-actions bot added domain: sinks Anything related to the Vector's sinks domain: codecs Anything related to Vector's codecs (encoding/decoding) domain: external docs Anything related to Vector's external, public documentation labels Apr 26, 2024

lsampras mentioned this pull request Apr 26, 2024

Add a pretty json encoding for better testing / configuring experience #20380

Open

chore(changelog): add changelog.d entry

4f875cd

lsampras force-pushed the add_pretty_json branch from 9c8b66c to 4f875cd Compare April 27, 2024 19:54

hestonhoffman approved these changes Apr 29, 2024

View reviewed changes

jszwedko approved these changes Apr 29, 2024

View reviewed changes

fix(tests): update tests for pretty_json codec

e8bf8fc

refactor(pretty_json): adds a field to existing json codec instead of…

28edc04

… a new pretty_json codec

github-actions bot removed the domain: codecs Anything related to Vector's codecs (encoding/decoding) label May 6, 2024

jszwedko reviewed May 6, 2024

View reviewed changes

refactor(codecs): rename pretty field name for json codec

5535612

jszwedko approved these changes May 8, 2024

View reviewed changes

jszwedko enabled auto-merge May 8, 2024 19:00

jszwedko added this pull request to the merge queue May 8, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(new encoding): add pretty json encoding #20384

feat(new encoding): add pretty json encoding #20384

lsampras commented Apr 26, 2024 •

edited

bits-bot commented Apr 26, 2024 •

edited

jszwedko left a comment

lsampras commented Apr 30, 2024

lsampras commented May 2, 2024

jszwedko commented May 3, 2024

pront commented May 3, 2024

jszwedko left a comment

jszwedko May 6, 2024

jszwedko left a comment

github-actions bot commented May 8, 2024

Fine details of change detection per experiment

Explanation

lsampras commented May 14, 2024

feat(new encoding): add pretty json encoding #20384

Are you sure you want to change the base?

feat(new encoding): add pretty json encoding #20384

Conversation

lsampras commented Apr 26, 2024 • edited

bits-bot commented Apr 26, 2024 • edited

jszwedko left a comment

Choose a reason for hiding this comment

lsampras commented Apr 30, 2024

lsampras commented May 2, 2024

jszwedko commented May 3, 2024

pront commented May 3, 2024

jszwedko left a comment

Choose a reason for hiding this comment

jszwedko May 6, 2024

Choose a reason for hiding this comment

jszwedko left a comment

Choose a reason for hiding this comment

github-actions bot commented May 8, 2024

Regression Detector Results

No significant changes in experiment optimization goals

Fine details of change detection per experiment

Explanation

lsampras commented May 14, 2024

lsampras commented Apr 26, 2024 •

edited

bits-bot commented Apr 26, 2024 •

edited