Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REFACTOR: remove redefinition of _row_lengths and _column_widths functions in PandasOnDaskDataframe. #3780

Closed
anmyachev opened this issue Dec 4, 2021 · 4 comments · Fixed by #3781
Assignees
Labels
Code Quality 💯 Improvements or issues to improve quality of codebase P1 Important tasks that we should complete soon

Comments

@anmyachev
Copy link
Collaborator

Basic implementation should work for Dask engine. It already works for Ray engine.

Links to basic implementations:


Links to Dask redefinitions:


@vnlitvinov
Copy link
Collaborator

TBH, I would rather we add a specific Ray implementation that also gathers all objects widths in one go instead of doing a loop, this could potentially be a little bit faster.

@devin-petersohn
Copy link
Collaborator

It should be significantly faster @vnlitvinov.

anmyachev added a commit to anmyachev/modin that referenced this issue Feb 21, 2022
…funcs in 'PandasOnDaskDataframe'

Signed-off-by: Anatoly Myachev <[email protected]>
@anmyachev anmyachev added the Code Quality 💯 Improvements or issues to improve quality of codebase label Apr 21, 2022
@vnlitvinov
Copy link
Collaborator

I wonder if #4683 and #4718 actually address this. @anmyachev thoughts?

@vnlitvinov vnlitvinov added the P1 Important tasks that we should complete soon label Aug 26, 2022
@anmyachev
Copy link
Collaborator Author

I wonder if #4683 and #4718 actually address this. @anmyachev thoughts?

Really! Now the default implementation is enough, which for both Dask and Ray gather all remote objects at once.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Code Quality 💯 Improvements or issues to improve quality of codebase P1 Important tasks that we should complete soon
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants