Remove backends from backend services before updating

[dippynark]

I am occasionally seeing timeouts to a Service of type LoadBalancer on GKE (internal network load balancer with unmanaged instance groups for backends) when I label a Node with node.kubernetes.io/exclude-from-external-load-balancers: true and then delete the underlying instance after its health status disappears from the backend service (I am doing this to periodically force the cluster autoscaler to kick in in the hope that it'll replace on-demand VMs with spot VMs).

This only seems to happen when I remove the last instance in a zone and the corresponding unmanaged instance group is removed as a backend from the backend service. I would instead expect connection draining to apply (0 seconds by default) and close or reset any in-flight connections, but I think this is happening because the backend service is being updated rather than the instance group being explicitly removed; according to the connection draining documentation this is not one of the events that triggers connection draining.

A previous version of the backend service documentation said it can take several minutes for your changes to propagate throughout the network and there does not seem to be a way to monitor this propagation through the API so maybe I am not waiting long enough before deleting an instance?

I have found it hard to debug this scenario properly so I'm not confident that this is the cause, but does this sound like a possible bug? If so, would explicitly removing instance groups that are no longer specified as backends be an improvement to ensure connection draining applies in this situation?

[k8s-ci-robot]

This issue is currently awaiting triage.

If the repository mantainers determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aojea]

/cc @cezarygerard @swetharepakula

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[aojea]

(I am doing this to periodically force the cluster autoscaler to kick in in the hope that it'll replace on-demand VMs with spot VMs).

hi @dippynark , could you expand on this motivation to understand better what you are trying to achieve so we can offer you the best solution? there may be a better solution than having to exclude the nodes with labels :)