Support In and notIn operators in ParquetFilters.ConvertFilterToParquet #10341
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change-Id: If424bf9f83d78d2768990cce553a56866461ca4e
sririshindra
force-pushed
the
parquet_filtered_row_groups-CDPD-69432
branch
from
f2bb0c9
to
06bb5ac
Compare
Change-Id: If424bf9f83d78d2768990cce553a56866461ca4e
sririshindra
changed the title
Change-Id: Idf419106b89f356caff7a019e097bef677c46eef
Change-Id: I523d7bd39819a91c4905d007086c605bd0c53e70
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ParquetFilters.ConvertFilterToParquet performs conversion from an Iceberg filter (Expression) to a Parquet FilterPredicate. It currently only handles IS_NULL, NOT_NULL, IS_NAN, NOT_NAN, EQ, NOT_EQ, GT, GT_EQ, LT, LT_EQ. It does not handle, e.g., IN and NOT_IN, which would be useful to have.
This conversion can be used for setting a record filter in Parquet and then using that to filter row groups when reading Parquet files.