Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PERF-#6556: do window block-wise #6290

Draft
wants to merge 18 commits into
base: master
Choose a base branch
from
Draft

PERF-#6556: do window block-wise #6290

wants to merge 18 commits into from

Conversation

christinafan
Copy link

@christinafan christinafan commented Jun 26, 2023 •
edited by mvashishtha

What do these changes do?

  • first commit message and PR title follow format outlined here

    NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

  • passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • signed commit with git commit -s
  • Resolves PERF: do window/rolling functions block-wise instead of column-wise #6556
  • tests added and passing
  • module layout described at docs/development/architecture.rst is up-to-date

@christinafan christinafan requested a review from a team as a code owner June 26, 2023 01:06
@christinafan christinafan changed the title Fix copy5 FIX-#6290: Efficient Window operator Jun 26, 2023
@christinafan christinafan changed the title FIX-#6290: Efficient Window operator Efficient Window operator Jun 26, 2023
@christinafan christinafan marked this pull request as draft June 26, 2023 01:11
christinafan and others added 12 commits July 12, 2023 20:10
@christinafan
working on naive implementation for window
1b8dbad 
naive implementation

working on efficiency improvements

efficiency has bugs

some fixes, errors inside window_function_partition

some fixes, issue with num virtual partitions

fixed map function error hopefully

fixed row-wise

fixed row actually?

removed debugging comments

attempting to fix col-wise

removing debugging comments

row window operations work

fixed col ops

query compiler changes

removing print statements

query compiler changes

more query compiler changes

center fix work in progress

continued center fixes

attempting to fix bugs update

fixed errors

corr changes but still broken

forgot to push corr and cov changes
@christinafan
begining interface changes
477727b
@christinafan
interface changes complete
54016cf
@christinafan
comment removal
6913526
@christinafan
fixing interface for base execution
bf569c4
@christinafan
style changes
e193c37
@christinafan
more style changes
afda46b
@christinafan
style and comments for window function
b489cc1
@christinafan
Make test_rolling test center=False more and fix bugs
948bd67 
Signed-off-by: mvashishtha <[email protected]>
@christinafan
changing functions to align with interface changes
6906e5b
@christinafan
bug fixes
94b777b
@christinafan
more refactoring
e8085b8
@mvashishtha mvashishtha changed the title Efficient Window operator PERF: do window block-wise Jul 13, 2023
mvashishtha
mvashishtha reviewed Jul 14, 2023
View reviewed changes
Copy link
Collaborator

@mvashishtha mvashishtha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@christinafan I'm leaving some minor comments.

Before pushing again, you should test with:

pytest modin/pandas/test/test_rolling.py -n 8 --execution=PandasOnRay
pytest modin/pandas/test/test_rolling.py -n 8 --execution=BaseOnPython

modin/pandas/window.py Show resolved Hide resolved
modin/core/storage_formats/pandas/query_compiler.py Outdated Show resolved Hide resolved
modin/core/storage_formats/pandas/query_compiler.py Outdated Show resolved Hide resolved
modin/core/dataframe/pandas/dataframe/dataframe.py Show resolved Hide resolved
modin/core/dataframe/pandas/dataframe/dataframe.py Show resolved Hide resolved
modin/core/dataframe/pandas/dataframe/dataframe.py Outdated Show resolved Hide resolved
@mvashishtha mvashishtha changed the title PERF: do window block-wise PERF-#6556: do window block-wise Sep 13, 2023
@sfc-gh-mvashishtha
Add the break that commit 4e563b7 deleted
4aa3691 
Signed-off-by: Mahesh Vashishtha <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mvashishtha mvashishtha mvashishtha left review comments

@devin-petersohn devin-petersohn Awaiting requested review from devin-petersohn devin-petersohn will be requested when the pull request is marked ready for review devin-petersohn is a code owner

@RehanSD RehanSD Awaiting requested review from RehanSD RehanSD will be requested when the pull request is marked ready for review RehanSD is a code owner

@YarShev YarShev Awaiting requested review from YarShev YarShev will be requested when the pull request is marked ready for review YarShev is a code owner

@vnlitvinov vnlitvinov Awaiting requested review from vnlitvinov vnlitvinov will be requested when the pull request is marked ready for review vnlitvinov is a code owner

@anmyachev anmyachev Awaiting requested review from anmyachev anmyachev will be requested when the pull request is marked ready for review anmyachev is a code owner

@dchigarev dchigarev Awaiting requested review from dchigarev dchigarev will be requested when the pull request is marked ready for review dchigarev is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

PERF: do window/rolling functions block-wise instead of column-wise
3 participants
@christinafan @mvashishtha @sfc-gh-mvashishtha