GPU workflows: Unable to move pods that use the entire node

[cep21]

Hi,

K8s does not support fractional GPU reservations. We have an ASG of on demand and an ASG of spot nodes that all have a single GPU.

We eventually get into a state where 3 on demand nodes are fully used ( i e a pod on each requiring a single gpu) and 3 spot nodes are fully used ( i e a pod on each requiring a single gpu). Both the spot reschedule and the auto scaler get stuck here and I'm unsure how to get out. Ideally I could boot up a spot node and wait for it to fill out. Cluster autoscaler suggests sleep containers with low priority to reserve capacity, but these low priority pods won't be considered by the spot reschedule as ignorable.

Thoughts?