New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
getCompactionTaskCapacity is not worker category aware. #15847
Labels
Comments
My overlord dynamic config looks like this, overlord dynamic config{
"type": "equalDistributionWithCategorySpec",
"workerCategorySpec": {
"categoryMap": {
"kill": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"compact": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"single_phase_sub_task": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_dimension_cardinality": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_index_generate": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_index_generic_merge": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_range_index_generate": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_dimension_distribution": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"index_kafka": {
"defaultCategory": "xxx_category",
"categoryAffinity": {
"datasource_2": "yyy_category",
"datasource_3": "zzz_category"
}
}
},
"strong": true
}
} |
We worked around this by explicitly setting the See Update capacity for compaction tasks Example,
|
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
The getCompactionTaskCapacity function which is used to ascertain that the Druid cluster has enough task slots before the coordinator schedules additional compaction tasks doesn't take into consideration overlord dynamic config. The overlord dynamic config can prevent compaction tasks from running on specific categories of workers. This way, the compaction task capacity is incorrectly overestimated.
For example, I have two worker categories,
compaction-category
with a total of600
task slots and anotheringestion-category
with2000
slots(high number because of multiple ingestion task replicas).Using the overlord dynamic config,
compaction-category
is configured to run the following task types,ingestion-category
is configured to run,Now, getCompactionTaskCapacity would return
2600
as the total capacity, which is inaccurate since only600
slots are actually available for compaction tasks. While this might not pose a problem in a healthy cluster, it becomes critical during compaction task failures. The oversight leads to the coordinator creating excessive compaction tasks, resulting in contention on compaction slots and slowing down all compaction tasks. This creates a feedback loop where the increasing number of compaction tasks exacerbates contention, ultimately overwhelming the overlord with too many tasks to handleAffected Version
Saw this on Druid 25. Is also present in master.
The text was updated successfully, but these errors were encountered: