Read partial parquet file (ideally from external storage)

[erchirag]

Hi

I am working on an application which needs to read parquet file stored in external storage which allows reading file at different offset. I want to know how I can use .Net Interface to do the following ( I know it is possible to do in c++):
-> Read only metadata of the file stored in external storage ( including all the Column metadata)
-> Using column metadata, read the column portion from the external storage and read the column values in the c#

I might have more questions but if above is possible I guess this is MVP for me. I am happy to contribute if this is not possible today in C# library, so would appreciate help here in my success.

Thanks a lot
Chirag

[adamreeve]

Hi Chirag

This is exactly how ParquetSharp behaves if you read a file normally following the examples. When you create a ParquetFileReader, only the file metadata is read initially. Then if you iterate over the row groups and only read data for the column you are interested in, the column data will be read as required without reading data for other columns.

Does this answer your question?

Cheers,
Adam

[erchirag]

Hi Adam
Thanks for replying so fast. I want to only download the metadata part of the parquet file from external storage and then download only the column part from storage and then read them. Do you think it is possible?

[adamreeve]

Hi Chirag, what do you mean by external storage? Is it on a network file system? Or is it in cloud storage, eg. something like S3? If you can access the file with a file system path, or read it as a dotnet stream then yes this is possible.

[erchirag]

Interesting, Yes I meant some like S3 or Azure blob storage. Do you mean by using ManagedRandomAccessFile https://github.com/G-Research/ParquetSharp/blob/master/csharp/IO/ManagedRandomAccessFile.cs ?

[adamreeve]

Yes, you can use the ManagedRandomAccessFile class to wrap a stream, or we have overloads to the ParquetFileReader constructor that accept a stream directly, eg. see https://github.com/G-Research/ParquetSharp/blob/10.0.1/csharp/ParquetFileReader.cs#L25. I would assume that whatever library you use to access the external data should provide an API that lets you read the file as a stream to allow partial reads.

[erchirag]

yes it does, thanks a lot. let me get my hands dirty on parquet format now :D