Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Athena Partitioned By format for iceberg tables #5399

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

jverhoeks
Copy link

Brief summary of the change made

fixes #5398

The parser for create table expects a ColumnDefinitionSegment at the PARTITIONED BY statement. This is expecting , but according the documentation: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-creating-tables.html. This should be just

Example:

CREATE TABLE iceberg_table (id bigint, data string, category string) 
PARTITIONED BY (category)) LOCATION 's3://DOC-EXAMPLE-BUCKET/your-folder/' 
TBLPROPERTIES ( 'table_type' = 'ICEBERG' )

Updated the parser to NakedIdentifierSegment to accept the fieldname without quotes

Are there any other side effects of this change that we should be aware of?

Pull Request checklist

  • Please confirm you have completed any of the necessary steps below.

  • Included test cases to demonstrate any code changes, which may be one or more of the following:

    • .sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with tox -e generate-fixture-yml).
  • Added appropriate documentation for the change.

  • Created GitHub issues for any relevant followup/future enhancements if appropriate.

Copy link
Contributor

@WittierDinosaur WittierDinosaur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@WittierDinosaur WittierDinosaur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to generate the new test yml, please run python3 test/generate_parse_fixture_yml.py --dialect=athena

@jverhoeks
Copy link
Author

The fix seems to be more complicated. For external tables the partioned by does require a while for the create table not. I'll work on an updated pr.

@WittierDinosaur
Copy link
Contributor

Incase it makes this easier, our general philosophy is - make it parse all valid SQL. We're less concerned about invalid combinations successfully parsing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Athena create table for iceberg with partitoned by fails
2 participants