FEAT-#6990: Implement lazy execution for the Ray virtual partitions. #6991

AndreyPavlenko · 2024-03-01T20:32:34Z

What do these changes do?

first commit message and PR title follow format outlined here

NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.
passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
signed commit with git commit -s
Resolves Implement lazy execution for the Ray virtual partitions. #6990
tests added and passing
module layout described at docs/development/architecture.rst is up-to-date

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py

modin/core/execution/ray/common/deferred_execution.py

+ return False
+
+ @remote_function
+ def _remote_fn(obj, index): # pragma: no cover


modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py

anmyachev · 2024-05-06T11:14:43Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/lazy_virtual_partition.py

+ axis = 0
+
+ @remote_function
+ def _remote_concat(dfs): # pragma: no cover # noqa: GL08


Are you sure that the concat works as intended, given the message about the naming of the function arguments?

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/lazy_virtual_partition.py

+ axis = 1
+
+ @remote_function
+ def _remote_concat(dfs): # pragma: no cover # noqa: GL08


… partitions.

anmyachev

Judging by the annotations, we need to write a lot more tests to cover most of the changes.

anmyachev · 2024-05-06T10:31:42Z

modin/core/execution/ray/common/__init__.py

 from .utils import initialize_ray

 __all__ = [
 "initialize_ray",
 "RayWrapper",
 "MaterializationHook",
 "SignalActor",
+ "RayObjectRefTypes",


This item has been deleted

anmyachev · 2024-05-06T10:33:24Z

modin/core/execution/ray/common/engine_wrapper.py

@@ -214,7 +214,7 @@ def wait(cls, obj_ids, num_returns=None):
 num_returns : int, optional
 """
 if not isinstance(obj_ids, Sequence):
- obj_ids = list(obj_ids)
+ obj_ids = list(obj_ids) if isinstance(obj_ids, Iterable) else [obj_ids]


Can be deleted.

anmyachev · 2024-05-06T10:34:15Z

modin/config/envvars.py

@@ -868,7 +868,7 @@ class LazyExecution(EnvironmentVariable, type=str):
 """

 varname = "MODIN_LAZY_EXECUTION"
- choices = ("Auto", "On", "Off")
+ choices = ("Auto", "On", "Off", "Axis")


Why introduce a new mode?

anmyachev · 2024-05-06T10:51:12Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py

+ try:
+ ref = ray.get(ref, timeout=0)
+ except ray.exceptions.GetTimeoutError:
+ return False


If an object has been calculated and placed in distributed storage, will materialization occur here?

If this approach can be effective, then it is worth considering the possibility of using it in other places.

anmyachev · 2024-05-06T10:55:02Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py

@@ -419,7 +424,7 @@ def eager_exec(self, func, *args, length=None, width=None, **kwargs):
 LazyExecution.subscribe(_configure_lazy_exec)


-class SlicerHook(MaterializationHook):
+class SlicerHook(MaterializationHook, DeferredExecution):


What is the idea behind this change?

anmyachev · 2024-05-06T10:58:50Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/lazy_virtual_partition.py

+from .partition import PandasOnRayDataframePartition
+
+
+class PandasOnRayDataframeVirtualPartition(BaseDataframeAxisPartition):


Why not such inheritance?

Suggested change

class PandasOnRayDataframeVirtualPartition(BaseDataframeAxisPartition):

class PandasOnRayDataframeVirtualPartition(PandasDataframeAxisPartition):

anmyachev · 2024-05-06T11:07:45Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.py

@@ -42,6 +54,82 @@ class PandasOnRayDataframePartitionManager(GenericRayDataframePartitionManager):
 _execution_wrapper = RayWrapper
 materialize_futures = RayWrapper.materialize

+ if LazyExecution.get() in ("On", "Axis"):


Whether to use this function or not is determined during the first import without the possibility of further replacement. As far as I remember, in all other places, functions are defined on each call.

anmyachev · 2024-05-06T11:09:38Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.py

+
+ @classmethod
+ @_inherit_docstrings(GenericRayDataframePartitionManager.get_indices)
+ def get_indices(cls, axis, partitions, index_func=None):


Have you tried making lazy changes to the already existing get_indices? (without overriding)

anmyachev · 2024-05-06T11:11:16Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.py

+ PandasOnRayDataframeColumnPartition,
+ PandasOnRayDataframeRowPartition,
+ PandasOnRayDataframeVirtualPartition,


Have you tried making changes to existing classes?

anmyachev · 2024-05-06T11:14:43Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/lazy_virtual_partition.py

+ axis = 0
+
+ @remote_function
+ def _remote_concat(dfs): # pragma: no cover # noqa: GL08


Are you sure that the concat works as intended, given the message about the naming of the function arguments?

anmyachev · 2024-05-06T11:20:35Z

modin/core/execution/ray/common/deferred_execution.py

@@ -29,13 +29,20 @@

 import pandas
 import ray
+import ray.exceptions


It seems that a lot of the changes in this file are not directly affected by this pull request and therefore it would be great to move them into a separate pull request.

YarShev · 2024-05-23T18:34:33Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.py

@@ -12,24 +12,36 @@
 # governing permissions and limitations under the License.

 """Module houses class that implements ``GenericRayDataframePartitionManager`` using Ray."""
+import math


Suggested change

import math

import math

YarShev · 2024-05-23T18:35:51Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.py

- PandasOnRayDataframeRowPartition,
-)
+
+if LazyExecution.get() in ("On", "Axis"):


This logic should probably be placed in modin/core/execution/ray/implementations/pandas_on_ray/partitioning/init.py.

YarShev · 2024-05-23T18:47:07Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/lazy_virtual_partition.py

+# governing permissions and limitations under the License.
+
+"""Module houses classes responsible for storing a virtual partition and applying a function to it."""
+import math


Suggested change

import math

import math

YarShev · 2024-05-23T18:48:16Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/lazy_virtual_partition.py

+ """
+
+ partition_type = PandasOnRayDataframePartition
+ instance_type = ray.ObjectRef


Suggested change

instance_type = ray.ObjectRef

@anmyachev, can this be removed?

YarShev · 2024-05-23T18:53:23Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/lazy_virtual_partition.py

+ list of lengths or None
+ Estimated chunk lengths, that could be different form the real ones.
+ bool
+ Whether the specified partitions represent the full block or just the


Can you elaborate a little on this?

YarShev · 2024-05-23T19:54:33Z

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/lazy_virtual_partition.py

+ manual_partition=False,
+ **kwargs,
+ ) -> Union[List[PandasOnRayDataframePartition], PandasOnRayDataframePartition]:
+ if not manual_partition:


Why does this parameter have effect only in case of False? Should we copy the related logic from the base class?

AndreyPavlenko force-pushed the issue-6990 branch 2 times, most recently from bf2943d to ea540cc Compare March 1, 2024 20:38

github-advanced-security bot found potential problems Mar 1, 2024

View reviewed changes

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py Fixed Show fixed Hide fixed

YarShev mentioned this pull request Mar 13, 2024

FEAT-#7004: use generators when returning from _deploy_ray_func remote function. #7005

Merged

7 tasks

AndreyPavlenko force-pushed the issue-6990 branch 12 times, most recently from 2e3390b to d98324d Compare March 16, 2024 16:54

github-advanced-security bot found potential problems Mar 16, 2024

View reviewed changes

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py Fixed Show fixed Hide fixed

AndreyPavlenko force-pushed the issue-6990 branch 13 times, most recently from 128509d to 8ce0b34 Compare March 20, 2024 20:17

AndreyPavlenko force-pushed the issue-6990 branch from 06ff2d9 to 5978769 Compare March 21, 2024 23:30

github-advanced-security bot found potential problems Mar 21, 2024

View reviewed changes

AndreyPavlenko force-pushed the issue-6990 branch from 5978769 to bb497bd Compare March 21, 2024 23:57

AndreyPavlenko force-pushed the issue-6990 branch 2 times, most recently from 0009497 to cabeb86 Compare April 10, 2024 17:35

github-advanced-security bot found potential problems Apr 10, 2024

View reviewed changes

AndreyPavlenko force-pushed the issue-6990 branch 10 times, most recently from 8cc6583 to b09a944 Compare April 12, 2024 19:35

AndreyPavlenko marked this pull request as ready for review April 12, 2024 20:59

AndreyPavlenko requested review from devin-petersohn, mvashishtha, RehanSD, YarShev, vnlitvinov, anmyachev, dchigarev and a team as code owners April 12, 2024 20:59

FEAT-modin-project#6990: Implement lazy execution for the Ray virtual…

92fe2f7

… partitions.

AndreyPavlenko force-pushed the issue-6990 branch from b09a944 to 92fe2f7 Compare April 16, 2024 13:21

anmyachev reviewed May 6, 2024

View reviewed changes

YarShev reviewed May 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT-#6990: Implement lazy execution for the Ray virtual partitions. #6991

FEAT-#6990: Implement lazy execution for the Ray virtual partitions. #6991

AndreyPavlenko commented Mar 1, 2024

anmyachev May 6, 2024

anmyachev left a comment

anmyachev May 6, 2024

anmyachev May 6, 2024

anmyachev May 6, 2024

anmyachev May 6, 2024

anmyachev May 6, 2024

anmyachev May 6, 2024

anmyachev May 6, 2024

anmyachev May 6, 2024

anmyachev May 6, 2024

anmyachev May 6, 2024

anmyachev May 6, 2024

YarShev May 23, 2024

YarShev May 23, 2024

YarShev May 23, 2024

YarShev May 23, 2024

YarShev May 23, 2024

YarShev May 23, 2024

		from .partition import PandasOnRayDataframePartition


		class PandasOnRayDataframeVirtualPartition(BaseDataframeAxisPartition):

FEAT-#6990: Implement lazy execution for the Ray virtual partitions. #6991

Are you sure you want to change the base?

FEAT-#6990: Implement lazy execution for the Ray virtual partitions. #6991

Conversation

AndreyPavlenko commented Mar 1, 2024

What do these changes do?

Choose a reason for hiding this comment

anmyachev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment