Complex Type Read Schema and Data

[Mydurian]

Hi,
I'm seeking for the approach to use ParquetSharp to read parquet with complex type. Is there any convenient way to read complex type schema and data directly?

[adamreeve]

Hi @Mydurian, by complex type, do you mean complex numbers with real and imaginary components? The Parquet format doesn't have built in support for complex numbers so I'm not sure what you're asking. Can you provide an example of the schema of a file you're trying to read?

[Mydurian]

Hi @adamreeve, not complex number. I'm reading the parquet file with complex column type like Struct/List/Map such nested data. Here's the example.

[adamreeve]

Right, yes you can read such files with ParquetSharp but there isn't a way to read the data directly as .NET lists and dictionaries. One approach is to use the Arrow based API to read the data as Arrow record batches. There is documentation on this at https://github.com/G-Research/ParquetSharp/blob/master/docs/Arrow.md.

Otherwise, if you want to use the original ParquetSharp API that is based on standard .NET types, you can't read this in a structured form directly, but can read the leaf-level columns. For example, the first column can be read as string arrays, where each row is an array of the "line1" values, and the second column will contain arrays for the "name" values. You can also read maps by separately reading arrays of keys and arrays of values. There's some documentation on this at https://github.com/G-Research/ParquetSharp/blob/master/docs/Nested.md#reading-dictionary-data

[Mydurian]

Thanks so much @adamreeve. I got to know how to read data for parquet file with complex data type. I have one more quick question. If I want to read the schema directly instead of data. For example, I just want to read the column type in top level(contains both primitive type and complex type)with List, Map<int, struct>, int, etc. Is there any convenient way to read the complex data type in schema directly? Or I should also use ParquetSharp API to do so in leaf-level and convert to top level?

[adamreeve]

Yes, you can access the Parquet schema with something like:

using var fileReader = new ParquetFileReader(path);
using var metadata = fileReader.FileMetaData;
using var schemaNode = (GroupNode) metadata.Schema.SchemaRoot;

schemaNode is a ParquetSharp.Schema.GroupNode and has a Fields property containing a node for each top-level column of the schema. Fields can be a PrimitiveNode or a GroupNode with their own subfields for more complex types like lists, maps and structs. The LogicalType property of a node will tell you if it represents a List or Map.

Alternatively, if you're working with the Arrow API you can also get the schema in Arrow format with:

using var fileReader = new ParquetSharp.Arrow.FileReader(path);
var schema = fileReader.Schema;

[Mydurian]

Hi @adamreeve , but I see that the Fields property is private property and I cannot get it from the SchemaRoot node. How could I iterate the node's Fields?

[adamreeve]

The Fields property I'm referring to is public, you can see it here:

ParquetSharp/csharp/Schema/GroupNode.cs

Line 21 in d531836

public Node[] Fields => Enumerable.Range(0, FieldCount).Select(Field).ToArray();