-
-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update frame repacking to use new pyarrow data types #2470
Comments
Blocked on: |
From discussion: we could also consider a gradual migration, e.g. enabling pyarrow types for new datasets rather than applying it over everything. We could also bench the performance gains to see if they're worth it. |
Pablo noted that adding commitments to our future selves can mean that "small" data updates can expand. We should try to make more things as automatic as possible. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Don't close it. We should at least compare the performance (CPU & mem) of current repacking vs new pyarrow dtypes. |
Context
We do frame repacking to make our data frames smaller on disk and faster to work with. It has some slight annoyances, such as making more variables categorical, which complicates group-bys, for example.
What
With Pandas 2.2, we have the option to change our frame repacking to use new pyarrow data types, which are supposed to be much more efficient.
That would also bring our data catalog into compatibility with more of the data ecosystem (e.g. Polars, Nushell and friends).
The text was updated successfully, but these errors were encountered: