Checking cluster capacity and throttling when necessary

[devdattakulkarni]

Consider the situation where KubePlus receives a request to create an application instance, but there is no available capacity on the cluster. In this case, KubePlus should deny such a request. Currently, KubePlus will handle the request but the application Pods will remain Pending if there is not enough available capacity on the cluster.

This feature will require two things:

• First, we should require that Helm charts define requests and limits for their workload Pods. If both are defined that's great. Otherwise, at least requests need to be defined. We should flag an error if this is not the case.
• Second, we should compare the available capacity (CPU and memory) on the cluster and what is being requested in the Helm chart. If the requested amounts are less than the available amounts then we should allow the request to go through.

Performing both checks in mutating webhook can be tricky since there is a strict 30-second timeout window for mutating webhook actions. One option can be to provide a kubectl plugin to perform the first check. However, the use of this plugin cannot be enforced. So probably the best place to perform the checks will still be the mutating webhook.