Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dspash fixes #566

Draft
wants to merge 19 commits into
base: future
Choose a base branch
from
Draft

Dspash fixes #566

wants to merge 19 commits into from

Conversation

tammam1998
Copy link
Collaborator

Fixes many bugs and reliability problems:

  • Support multiple inputs in hdfs cat "hdfs dfs -cat $IN1 $IN2 .."
  • Fix bugs with multi sink scripts
  • Support HDFS blocks that are not stored in subdir0 locally
  • Merge graph on one worker on the first aggregator (Performance improvement)
  • Use processes instead of threads in workers (fixed hanging issue)
  • Fixes other bugs and edge cases in graph splitting
  • Setup and Installation scripts improvements

@github-actions
Copy link

github-actions bot commented Jun 7, 2022

OS:ubuntu-20.04
Tue Jun 7 17:59:51 UTC 2022
intro: 2/2 tests passed.
interface: 34/34 tests passed.
compiler: 54/54 tests passed.
agg: 109/109 tests passed.

@github-actions
Copy link

github-actions bot commented Jun 7, 2022

OS:ubuntu-18.04
Tue Jun 7 18:00:15 UTC 2022
intro: 2/2 tests passed.
interface: 34/34 tests passed.
compiler: 54/54 tests passed.
agg: 109/109 tests passed.

@github-actions
Copy link

github-actions bot commented Jun 7, 2022

OS = Debian 10
CPU = Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
Ram = 15752
Hash = 226346a
Kernel= Linux 4.15.0-167-generic x86_64

benchmark tests passed failed untested unresolved unsupported not_in_use other_status
posix 494 375 41 31 6 40 1 0
intro 2 2 0 0 0 0 0 0
interface 34 34 0 0 0 0 0 0
compiler 54 54 0 0 0 0 0 0
aggregator 109 109 0 0 0 0 0 0

@angelhof
Copy link
Member

angelhof commented Jun 7, 2022

I would prefer redirecting these changes to a different branch (other than future). Let's have a dspash-future branch that we will work on temporarily, and when everything stabilizes, we can push to the normal future (to avoid stalling on PRs now). We can still make PRs to this dspash-future branch.

@tammam1998 tammam1998 marked this pull request as draft June 7, 2022 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants