Only 16 bytes of decimal length is supported

[HairPuller]

I'm stumped. I've googled, searched, hacked, coded, and tried just about everything but I am totally stumped. I have a c# parquet file reader developed with a simple drag-drop of the file over the c# form displays the parquet file in a datagrid....EXCEPT, I have a problem with a decimal field that can't be read with ParquetSharp. The field stored in the parquet field was defined in some python or other such script as to be decimal(21,18). When I get to my code for LogicalReader for that column, all I get is the message, "System.NotSupportedException: 'only 16 bytes of decimal length is supported'"

I'm fine with it being converted to a string. But no matter what I try, it seems like it's something in ParquetSharp core that refuses to read the value because it's too big of a decimal. I do know it recognizes the field as FixedLenByteArray. I just want some code that says no matter what format, just read it as a string - or something else that enables me to display it within the c# application.

[adamreeve]

Hi, yes you're right, ParquetSharp currently only handles fixed-length byte array decimal data that uses 16 byte arrays. A decimal(21, 18) type would use 9 bytes.

What do you mean by "just read it as a string"? Creating a string representation of the decimal value isn't really an option and would be more complicated than just reading it to a C# decimal. If you're happy to deal with the byte values yourself and interpret the data or show a string representation of the bytes then that's possible, you can just use the physical column reader instead of a logical column reader. For example, with something like:

using var reader = new ParquetFileReader(filePath);
using var rowGroup = reader.RowGroup(0);
var numRows = rowGroup.MetaData.NumRows;
using var column = (ColumnReader<FixedLenByteArray>) rowGroup.Column(0);
var typeLength = column.ColumnDescriptor.TypeLength;

var values = new FixedLenByteArray[numRows];
var rowsRead = column.ReadBatch(numRows, values, out var valuesRead);
var bytes = new byte[typeLength];
for (var i  = 0; i < valuesRead; ++i)
{
    Marshal.Copy(values[i].Pointer, bytes, 0, typeLength);
    Console.WriteLine(Convert.ToHexString(bytes, 0, typeLength));
}

[HairPuller]

Thanks, that's what I thought regarding the 16-byte and 9-byte, but then no idea why the error. I'll try the option you provided and see what I can figure out. Thanks for the quick reply!