Skip to content
This repository has been archived by the owner on Sep 1, 2022. It is now read-only.

Chunksize can be read from an attribute now. #1238

Open
wants to merge 2 commits into
base: 5.0.0
Choose a base branch
from

Conversation

Soman2
Copy link

@Soman2 Soman2 commented Feb 22, 2019

Uncommented the part of the code which allows Chunksize set up from a variable attribute for NetCDF format.

@CLAassistant
Copy link

CLAassistant commented Feb 22, 2019

CLA assistant check
All committers have signed the CLA.

@lesserwhirls
Copy link
Collaborator

So these lines were commented out as part of a PR to fix issue #330. As part of that, some tests were written to test the fix. This PR does not break any of our tests, so that's certainly a plus. That said, I know that some core netCDF-4 attributes, like _ChunkSizes, are kind of special and are handled differently that other generic attributes. @DennisHeimbigner - what's your take?

@DennisHeimbigner
Copy link
Contributor

Ideally, this should be consistent wit the netcdf-c library behavior. Without looking in detail,
I am guessing that setting (as opposed to reading) the chunksize from attributes
is only supported by ncgen .cdl files.

@DennisHeimbigner
Copy link
Contributor

I am reluctant to merge this PR because it appears to conflict with the behavior
of the netcdf-c library. In that library, _ChunkSizes is a virtual attribute -- meaning it
does not exist as an explicit attribute in the dataset. What is the use case for this?

@Soman2
Copy link
Author

Soman2 commented Feb 25, 2019

In my project I want to use your library to write down the file with ~100000x60000 size and set up custom ChunkSize( ~4000x4000). As for know I can not set up any ChunkSize.

@lesserwhirls
Copy link
Collaborator

I see. Currently, the use of the _ChunkSizes attribute to control how data are written out is not supported (our documentation is out of date on this). This is probably something we can support again in the future, but it will take some work. I think we see an issue when trying to write a subsetted dataset using the _ChunkSizes defined in the non-subsetted dataset, but I'm not totally sure.

What you can do currently, however, is create a class that implements ucar.nc2.write.Nc4Chunking for your particular dataset and needs. Once you've implemented that, you should be able to pass it into FileWriter2 or NetcdfFileWriter using one of the signatures that accepts Nc4Chunking. A bit more work that setting an attribute and being done with it, for sure, but until we get that fixed, this should get you up and running.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants