New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade pyarrow to 16? #2756

Open

severo opened this issue Apr 30, 2024 · 1 comment

Labels

dependencies P2 question

Collaborator

severo commented Apr 30, 2024

Release notes here: https://arrow.apache.org/blog/2024/04/20/16.0.0-release/

Are we affected by any change? Does it enable something for us?

severo added question dependencies P2 labels

Collaborator Author

severo commented Apr 30, 2024 •

edited

See the part on custom filesystems. Maybe it's an opportunity to use the hf:// scheme.

Filesystems
New Features

In addition to building the individual filesystem implementations as separate modules, users can now write and register custom filesystem implementations (GH-38309).
A new environment variable, AWS_ENDPOINT_URL_S3, has been added which allows separately overriding the endpoint for S3 operations alone (GH-38663).

Internal discussion: https://huggingface.slack.com/archives/C04L6P8KNQ5/p1714472156838979

severo mentioned this issue

upgrade to pyarrow 15? #2626

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment