Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polars Support? #9649

Open
edgBR opened this issue Mar 22, 2024 · 1 comment
Open

Polars Support? #9649

edgBR opened this issue Mar 22, 2024 · 1 comment

Comments

@edgBR
Copy link

edgBR commented Mar 22, 2024

Is your feature request related to a problem? Please describe.
Hi, we have been using GE before in our team but we are migrating most of our pandas processing pipelines to polars. However it seems that GE does not have any support for polars and transforming the polars dataset to pandas just to run the GE suite seems extremely subotimal.

Describe the solution you'd like
We would like to get polars support in GE.

Describe alternatives you've considered
We have considered to use pandera, which is going to offer full polars support very soon:

unionai-oss/pandera#1064

https://pandera--1373.org.readthedocs.build/en/1373/polars.html#polars

Additional context
We are in the situation were our data is not big enough to use spark. Therefore GE with pyspark or pydequee have been discarded for data validation.

@abekfenn
Copy link
Contributor

Jumping on the bandwagon here.
We are also exploring migrating from pandas to polars in our data processing pipelines.
Pandera is working on this and it appears support for polars is on its way there.
Polars is blazingly fast compared to pandas and this would be incredibly helpful if GX added support for a polars execution engine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants