🌱 Allow inplace update of fields related to deletion during Machine deletion #10589
Conversation
k8s-ci-robot
added
area/machine
|
Thanks @davidvossel change makes sense to me. Smoother deletion is actually one of the supporting use cases for inplace propagation. Let's include some unit tests. |
davidvossel
force-pushed
the
nodedrain-during-delete
branch
from
9dfb847
to
7e05934
Compare
k8s-ci-robot
added
size/M
@enxebre i extended the existing unit test to cover the case of updating a deleting machine. |
sbueringer
changed the title
davidvossel
force-pushed
the
nodedrain-during-delete
branch
from
7e05934
to
b69cea4
Compare
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 44936fae936d0eab3c39b86c432c24c5e199979d
|
| @@ -362,8 +362,21 @@ func (r *Reconciler) syncMachines(ctx context.Context, machineSet *clusterv1.Mac | |||
| log := ctrl.LoggerFrom(ctx) | |||
There was a problem hiding this comment.
Just trying to think through various cases where Machines belonging to MachineSets are deleted
- MD is deleted
The following happens:
- MD goes away
- ownerRef triggers MS deletion
- MS goes away
- ownerRef triggers Machine deletion
=> Current PR doesn't help in this scenario, because the MS will already be gone when the deletionTimestamp is set on the Machines. In this case folks would have to modify the timestamps on each Machine individually.
I recently had a discussion with @vincepri, that we should maybe consider changing our MD deletion flow. Basically adding a finalizer on MD & MS, so MD & MS stick around until all Machines are gone. If we would do this, the MS => Machine propagation of the timeouts implemented here would help for this case as well
- MD is scaled down to 0
The following happens:
- MD scales down MS to 0
- MS deletes Machine
=> This PR helps in this case because the timeouts are then propagated from MS to Machine
- MD rollout
The following happens:
- Someone updates the MD (e.g. bump the Kubernetes version)
- MD creates a new MS and scales it up
- In parallel MD scales down the old MS to 0
=> In this scenario the current PR won't help, because the MD controller does not propagate the timeouts from MD to all MS (only to the new/current one, not to the old ones)
I see how this PR addresses scenario 2. Wondering if we want to solve this problem more holistically. (maybe I also missed some cases, not sure)
There was a problem hiding this comment.
Here's what's going on... the use case is subtle, but an easy one to get trapped by.
- a MS is created with the default node drain timeout of (wait forever).
- The MS needs to scale down to zero (but not be deleted). The intent it to bring this MS back online at some point.
- The user discovers that the default node drain timeout is blocking the scale down to zero. The user likely only encounters this drain block the first time they scale down to zero because there are typically other nodes available during normal scale down operations which allows PDB to be satisfied.
The outcome is that the user is now trapped. They can't gracefully scale the MS down to zero because the default node drain timeout can't be updated on the machines. So the user is either forced to take some manual action to tear down the machines or delete the MS.
By allowing the node drain timeout to be modified while the machines are marked for deletion, we give the user a path to unblock themselves using the top level api (either MS or MD) rather than mutating individual machines or performing some other manual operation.
| if !m.DeletionTimestamp.IsZero() { | ||
| patch := client.MergeFrom(m.DeepCopy()) |
There was a problem hiding this comment.
Let's please use the patchHelper here (see e.g. l.1010), it's doing basically the same here, but I would prefer using it so we can profit from error handling & potential improvements here as well (e.g. logging etc.)
| g.Expect(reconciler.syncMachines(ctx, ms, []*clusterv1.Machine{updatedInPlaceMutatingMachine, deletingMachine})).To(Succeed()) | ||
| updatedDeletingMachine := deletingMachine.DeepCopy() | ||
|
|
||
| g.Eventually(func(g Gomega) { |
There was a problem hiding this comment.
Eventually shouldn't be required here (syncMachines does synchronous calls and GetAPIReader() reads without cache)
…ine deletion Signed-off-by: David Vossel <[email protected]>
davidvossel
force-pushed
the
nodedrain-during-delete
branch
from
b69cea4
to
da2b7c8
Compare
k8s-ci-robot
removed
the
lgtm
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: a7820897401d291e168e73cfc2ea745d5f2c8d87
|
Fixes #10588
/area machine
What this PR does / why we need it:
Machine's default to nodeDrainTimeout: 0s, which blocks indefinitely if a pod can't be evicted. We can't change the nodeDrainTimeout in-place from the MachineDeployment or MachineSet after a machine is marked for deletion.
This results in a machine that wedged forever but can't be updated using the top level objects that own the machine.
To fix this, this PR allows fields related to machine deletion to be updated in place even when the machine is marked for deletion.
NOTE - I did not add unit tests yet for this PR. I want confirmation this is an acceptable approve before investing time into testing.