Close all pipelines in cluster using CLI

[Montura]

For now ozone CLI allows to get pipelines list or close pipeline by PipelineId.

I'd like to add a new command to close all pipelines (or some filtered pipeline subset):

1. get pipeline list from ScmClient
2. filter CLOSED pipelines (to close only OPEN and ALLOCATED)
3. filter pipelines by REPLICATION or REPLICATION_TYPE or REPLICATION_FACTOR (as in ListPipelinesSubcommand done)
4. close the remain pipelines set

UPD: added PR

[swamirishi]

@Montura Can you file a jira for this.

[Montura]

Sure, I'll provide Jira and PR, if my suggested solution is ok for contributors

[jojochuang]

By the way, you can do that using the following script:

ozone admin pipeline list | grep OPEN | cut -d ',' -f1 | cut -d ":" -f2 | cut -d " " -f2  | while read line; do echo $line | ozone admin pipeline close $line ; done

[Montura]

Thanks, but I suppose ozone admin pipeline closeAll will be more user-friendly

[smengcl]

or ozone admin pipeline close --all :)

[Montura]

@smengcl, better solution then adding a new subcommand I suppose

[errose28]

It seems like it would be a nice addition, but can you share more details about the use case? If mass closing of pipelines is so frequent that it needs a convenient command, there is probably something else wrong with the cluster.

[Montura]

According to discussions in #6248 for now ozone doesn't have a mechanism for fair pipeline distribution. And in some cases it's important to have fairly distributed pipelines between the datanodes.

So, I found possible solution for it by specifying the minimun number of datanodes required to exit SCM safe mode and then create pipelines. But this approach does not work very well in some fault tolerance scenarios.

So, the obvious way to achieve the desired pipeline distribution is to close all pipelines that were created during cluster startup. At this point all datanodes are (hopefully) healthy. Then BackgroundPipelineCreator can create pipelines using all healthy datanodes instead of creating as much as possible pipelines on the first 3 started datanodes.

[ivandika3]

Closed since #6577 has been merged.