Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize transferring small objects for storage #3058

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

hekaisheng
Copy link
Contributor

What do these changes do?

Related issue number

This PR implements a method that send small objects directly to receiver without split them into small blocks.

Fixes #xxxx

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@hekaisheng hekaisheng added this to the v0.10.0a1 milestone May 20, 2022
@hekaisheng hekaisheng added this to In progress in Distributed via automation May 20, 2022
@hekaisheng hekaisheng added this to PR-In progress in v0.10 Release via automation May 20, 2022
@qinxuye qinxuye changed the title Add _send_small_objects for sender to optimize sending small objects Optimize transferring small objects for storage Jun 8, 2022
@qinxuye qinxuye added the to be backported Indicate that the PR need to be backported to stable branch label Jun 8, 2022
@qinxuye
Copy link
Collaborator

qinxuye commented Jun 9, 2022

Seems vineyard cannot pass the ut for this PR, @sighingnow do you have to time to help to see what's going on?

@sighingnow
Copy link
Contributor

Seems vineyard cannot pass the ut for this PR, @sighingnow do you have to time to help to see what's going on?

I would take a look later today.

@qinxuye qinxuye modified the milestones: v0.10.0a1, v0.10.0a2 Jun 12, 2022
@sighingnow
Copy link
Contributor

Seems vineyard cannot pass the ut for this PR, @sighingnow do you have to time to help to see what's going on?

I would take a look later today.

Sorry for the delay... Will be fixed today.

@sighingnow
Copy link
Contributor

I failed to reproduce the failure if only run the single test case. But I found the test case test_execute_describe failed both for default storage and vineyard storage under this PR.

I have tried to debugging but failed to address the root cause. I think it is not a vineyard-only issue, @hekaisheng could you please help to try the test_execute_describe in your environment?

You could update the params to test the default storage only. The failre could be reproduced by

pytest -s -vvv mars/deploy/oscar/tests/test_local.py -k test_execute_describe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mod: storage to be backported Indicate that the PR need to be backported to stable branch type: enhancement request
Projects
Distributed
  
In progress
v0.10 Release
PR-In progress
Development

Successfully merging this pull request may close these issues.

None yet

3 participants