Skip to content

Go library to deal with streaming JSON documents with line-delimited JSON (NDJSON), record separator-delimited JSON, concatenated JSON, and length-prefixed JSON

License

Notifications You must be signed in to change notification settings

bserdar/jsonstream

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

JSON Streams

GoDoc

This library supports streaming JSON streaming conventions described in this Wikipedia page.

This is for the concatenated JSON streams, where each JSON document is concatenated one after the other:

rd:=jsonstream.NewConcatReader(reader)
var entry MyEntry
for {
  err:=rd.Unmarshal(&entry)
  if err==io.EOF {
    break
  }
  if err!=nil {
    return err
  }
  processEntry(entry)
}

This is for JSON streams with a separate JSON document in each line (NDJSON):

ndLinesReader:=jsonstream.NewLineReader(reader)
ndLinesWriter:=jsonstream.NewLineWriter(writer)

This is for JSON streams separated with record separator delimiter:

seqReader:=jsonstream.NewSeqReader(reader) // 0x1e record separator
seqWriter:=jsonstream.NewSeqWriter(writer) 
seqReader:=jsonstream.NewSeqReaderWithSep(reader,'\n') // Line-separated JSON
seqWriter:=jsonstream.NewSeqWriterWithSep(writer,'\n')

This is for JSON streams with length prefixed JSON stream, where each JSON document is prefixed by its byte length:

lpReader:=jsonstream.NewLenPrefixedReader(reader)
lpWriter:=jsonstream.NewLenPrefixesWriter(writer)

APIs

All four stream readers/writers use the same APIs.

Readers

data, err:=reader.ReadRaw()

ReadRaw reads the next JSON document. Only the ConcatReader validates that the JSON document is a valid document, the remaining readers simply read until the next delimiter. The returned byte array is a newly allocated copy of the underlying read buffer. Some of the readers use buffered input, so the state of the underlying reader is unknown.

var data myStruct
err:=reader.Unmarshal(&data)

Unmarshals the next entry from the input. For concatenated JSON, errors invalidate the rest of the stream. For others stream processing can continue.

Writers

err:=writer.WriteRaw(data)

WriteRaw simply writes the []byte data to the output, with the correct delimiter. For NDJSON, WriteRaw removes the newline characters from data.

err:=writer.Marshal(data)

Marshal first encodes data to JSON, and then writes it to the output.

About

Go library to deal with streaming JSON documents with line-delimited JSON (NDJSON), record separator-delimited JSON, concatenated JSON, and length-prefixed JSON

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages