Kafka Avro Dumper

Kafka Avro dumping utility written in Go.

Most command-line tools that can consume Avro records from Kafka output data using Avro JSON encoding. When using Avro union types, the Avro JSON encoding includes the specific type in the field to disambiguate.

For example, for a fieldTwo field defined as type [null, string] in Avro:

{"fieldOne":100,"fieldTwo":{"string":"hello"}}

While the above is necessary for correctly encoding JSON data into Avro, it can become inconvenient when decoding plain JSON data from Avro and feeding it into non-Avro applications for processing.

For a more conveniently decoded JSON data, this tool uses the goavro library. More specifically, kadumper uses the Standard Full JSON codec which (from the documentation) provides full serialization/deserialization for JSON data that meets the expectations of regular internet JSON.

For the same example fieldTwo field shown above, kadumper outputs the following JSON data:

{"fieldOne":100,"fieldTwo":"hello"}

As added convenience, kadumper can also output raw textual data (non-Avro) from Kafka.

Usage

All currently available command-line arguments can be seen with -h/--help.

Consuming Avro data from Kafka is currently only supported in Confluent Wire format with schemas automatically downloaded from a Schema Registry instance. Downloaded schemas are cached with a configurable maximum caching age. Confluent Platform and Redpanda have both been tested.

To output raw textual data (non-Avro) from Kafka, omit the --schema-registry-url argument.

Output Kafka records are separated by a newline \n character. For each record, output timestamps, partitions, offsets, keys and values are separated by a TAB \t character. All output is sent to the standard output.

Currently, only anonymous plain-text and mutual TLS authentication connections are supported.

Building

Note: Ready-to-use binaries are available in the releases page.

To build a snapshot locally Using GoReleaser:

goreleaser build --clean --single-target --output kadumper --snapshot

License

This project is licensed under the Apache License Version 2.0.