New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create a pre-computed sample #1667

Draft

wjones127 wants to merge 3 commits into main from wjones127/filter-sampler

Contributor

wjones127 commented Nov 28, 2023 •

edited

TODO:

Change chunking to be based on row count and not distance from start.
Consider changing sampling to come after chunking
Make sure we make a mask for the no filter sampled case.
Support PyArrow array in LanceDataset.take()
Make data loader functions (lance.tf.data.from_lance_batches) take DatasetSample.
Reconsider GZIPing archive, given most of contents are already compressed.
Clean up API reference docs
Add entry to user guide

wjones127 added 2 commits

November 28, 2023 15:42


feat: create a pre-computed sample

4f0fbd1


feat: support serializing sample

3f1f326

wjones127 force-pushed the wjones127/filter-sampler branch from 2cb6d20 to 3f1f326 Compare

November 29, 2023 17:24


more todos

5def5b4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment