[Enhancement] adjust the BE and CN schedule policy (backport #44499) #45682
Conversation
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?s=60&v=4){kind=small}
Signed-off-by: MatthewH00 <[email protected]> Signed-off-by: hmx <[email protected]> (cherry picked from commit ae7a4bf) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/lake/qe/scheduler/DefaultSharedDataWorkerProvider.java # fe/fe-core/src/main/java/com/starrocks/qe/CoordinatorPreprocessor.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/DefaultWorkerProvider.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/WorkerProvider.java # fe/fe-core/src/test/java/com/starrocks/lake/qe/scheduler/DefaultSharedDataWorkerProviderTest.java # fe/fe-core/src/test/java/com/starrocks/qe/scheduler/DefaultWorkerProviderTest.java
|
Cherry-pick of ae7a4bf has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
|
@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr |
|
@MatthewH00 can you help to solve the conflict for branch-3.2? Base on this |
OK，i will solve it later |
i have solved the conflict , please check it . the new pr is #45835 |
…45835) Signed-off-by: hmx <[email protected]>
Signed-off-by: Kevin Xiaohua Cai <[email protected]>
|
Why I'm doing:
In shared nothing mode，when deploy both BE and CN, and set
prefer_compute_node=true, current BE and CN schedule policy is not very feel perfect:In some scenarios, like when sudden business peak, need temporarily add CN to increase computational resources, current BE and CN schedule policy would cause BE nodes have low resource usage ratio，CN nodes have high resource usage ratio，especially when business type belongs to computational tasks.
What I'm doing:
Add a new session variable
computation_fragment_scheduling_policy, the value as follows:compute_nodes_only: the previous policy, as the default value.all_nodes:Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check:
This is an automatic backport of pull request #44499 done by [Mergify](https://mergify.com). Why I'm doing: In shared nothing mode，when deploy both BE and CN, and set `prefer_compute_node=true`, current BE and CN schedule policy is not very feel perfect: 1. when query internal table, except scan fragment allocate to BE, other calculate fragment would allocate to CN. 2. when query external table, all fragment would allocate to CN.
In some scenarios, like when sudden business peak, need temporarily add CN to increase computational resources, current BE and CN schedule policy would cause BE nodes have low resource usage ratio，CN nodes have high resource usage ratio，especially when business type belongs to computational tasks.
What I'm doing:
Add a new session variable
computation_fragment_scheduling_policy, the value as follows:compute_nodes_only: the previous policy, as the default value.all_nodes:Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist: