Reading HDF5 that uses scaleoffset

[rschmunk]

I was sent a sample HDF5 file that uses the scaleoffset filter to compress one of its variables. Although I can open the file and read the metadata using NJ 5.5.3, I cannot read that particular variable because H5tiledLayoutBB is throwing an exception with description "Unknown filter type=6". Looking at that bit of code and also the list of filters in H5header, I see that deflate, shuffle, and fletcher32 filters are supported but skip, nbit and scaleoffset are not.

But according to Reading Zarr documentation, scaleoffset has been supported since NJ 5.5.1 for both Zarr and HDF5.

Is there something that I am missing in what I need to do to read the data, or is there a disconnect between what that page says about supporting HDF5? If I do a search for ucar.nc2.filter in the NJ code, it looks like only the Zarr code tries to use the filters in that package.

[haileyajohnson]

Are you using the last release (as opposed to a snapshot)? Do the exceptions tell you whether it's being thrown from the package ucar.nc2.internal.iosp or ucar.nc2.iosp? There are currently two implementations, and they handle filters in very different ways.

The short answer to your question though is yes, scaleoffset should be supported for both Zarr and HDF5, and both Zarr and the HDF% builder-based IOSP rely on the ucar.nc2.filter package, though I believe the HDF5 IOSP wasn't updated to use the new package until after the 5.5.3 release.

[haileyajohnson]

ok, that is a bug - looks like we're failing in the case where there is a scale but no offset. I'll get that fixed asap, but also trying to roll it up with some other fixes to the filters package this week.

[rschmunk]

Yes, the user who provided the sample dataset indicated that they were testing out compression and the data had been scaled but not offset.

[rschmunk]

@haileyajohnson, Was there any progress on dealing with the NPE in ScaleOffset? I tried looking through the commits, but it looks like that class hasn't been touched since January.

Ran into this because I was testing something else that also involved the same dataset that provoked this problem.

[haileyajohnson]

I have this particular issue fixed (though not check in apparently), but there are several other adjacent issues I am still working on. I'd expect it to be fixed in the next release, which we're targeting for end of the year.

I can merge the NPE fix sooner if that would be helpful.

[rschmunk]

I can wait, as I have versions of the dataset that do and do not use scaleOffset. Just wanted to check that this hadn't slipped off the radar, as can happen.