Question: Option to configure node-reaper differently for specific node group

[jeesmon]

Is this a BUG REPORT or FEATURE REQUEST?:
QUESTION

What happened:
We are running node reaper in our kube cluster for reaping nodes older than 7d for security and compliance reasons. Some of our workloads are ML workloads (Apache Flink jobs) and they run on a specific node group. When we reap a node in that node group, the entire ML job need to get restarted due to the architecture of Flink (same with many ML architectures). So if we reap nodes in the node group one by one (many times a day when nodes are 7d old), the job needs to be restarted many times in a day and it is adding up significant processing lag. As reaping a node older than 7d is same as reaping all nodes in that node group, we are wondering:

1. Is there an option in node reaper to wait until all nodes in a specific node group to be older than Nd and reap them all at the same time?
2. Is there an option to reap nodes in a specific node group at a specific time window so that we can contain the downtime?
3. Any other suggestion to address our scenario?

As noted above we are looking for the advance configuration for few special node groups in addition to the regular options for all other nodes.

Thanks in advance for any help on this.

What you expected to happen:

Option to configure a specific node group differently

How to reproduce it (as minimally and precisely as possible):

N/A

Anything else we need to know?:

N/A

Environment:

• Kubernetes version: v1.23

Other debugging information (if applicable):

• relevant logs: N/A

[shreyas-badiger]

@jeesmon

Is there an option in node reaper to wait until all nodes in a specific node group to be older than Nd and reap them all at the same time?

At present, No. Such option doesn't exist.
node-reaper was designed to run as a simple cronjob that is invoked timely and reaps the nodes based on certain conditions (listed here in the flags). node-reaper doesn't have any context of what node-group does a node belong to. Because of this, terminating all the nodes in the node-group at the same time is not possible through node-reaper.

Just a thought - the nodes that are older and must be terminated, won't they go out of compliance when they are waiting for the newer nodes to age?

Is there an option to reap nodes in a specific node group at a specific time window so that we can contain the downtime?

node-reaper is a cronjob that runs every 10 mins. If the state of the cluster is relatively stable and no new nodes are getting added in 7 days, maybe you can configure this to run once every 7 days?

Any other suggestion to address our scenario?

You can explore upgrade-manager and see if it fits your requirements. upgrade-manager is a controller that will perform node rotations. You can find more details in these flow charts and additional explanation in this comment.

Note: upgrade-manager isn't a cronjob like node-reaper. Instead, it is a controller that reconciles on a custom resource - RollingUpgrade. upgrade-manager operates on each node-group individually. For a given node-group, you can set the maxUnavailable=100% in it's RollingUpgrade CR.

[jeesmon]

Thanks @shreyas-badiger for the quick response. I think we have some grace period to terminate older nodes. But mainly looking into not reaping too many times a day for a specific node group as each recycle event will trigger entire job restart. I believe we are running scanning every 20 minutes to find nodes for reaping and reap one node at a time. It would have been nicer if we can adjust configuration for certain node group(s) and reap all candidates together. But as you said it is not possible as node reaper is not aware of the node group information. Do you think it is something we can add support by reading node labels?

Thanks for the details on upgrade-manager. Will explore its capabilities.

[shreyas-badiger]

Do you think it is something we can add support by reading node labels?

We can surely talk about this further. In my opinion, this will probably be a whole new feature. This could change how we list the nodes and parse them.

We can connect over keikoproj slack , #proj-governor-reapers to discuss more about this.

Thanks for the details on upgrade-manager. Will explore its capabilities.

Currently, we use upgrade-manager to upgrade clusters that run Flink.

[jeesmon]

Thanks again. I will join slack channel and continue conversation there.