Kueue multipod

[asaiacai]

This integrates kueue for multipod/node workloads. Previously triggering kueue to manage skypilot pods by including kueue.x-k8s.io/queue-name: user-queue under resource labels was causing kueue to recognize each pod as a distinct workload. This adds the additional required kueue labels/annotations anytime kueue.x-k8s.io/queue-name is included as part of a k8s skypilot Task so a multinode job registers as a single workload under kueue as well as performing sanity checks to ensure the user specified queues and priorities exist in the k8s cluster.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Manually test multinode skypilot task appears as a single workload under kueue (on GKE)

# setup kueue
kubectl apply -f --server-side tests/kubernetes/kueue-manifest.yaml
kubectl apply -f tests/kubernetes/kueue-single-clusterqueue.yaml
# launch skypilot task
sky launch  -c multikueue -d sky-kueue-multinode.yaml
# list kueue workloads
kubectl get workloads
# down task
sky down multikueue
# check workload is deleted
kubectl get workloads

sky-kueue-multinode.yaml

resources:
  cloud: kubernetes
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    kueue.x-k8s.io/priority-class: low-priority  

num_nodes: 2

run: |
  for i in {1..10000}; do
    echo "$i"
    sleep 1
  done 

[alculquicondor]

Hi folks, Kueue maintainer here.

I would love to learn more about skypilot and how we can best align our projects in the future. If anyone interested in the topic has some time, I would welcome you to present in a Kubernetes WG Batch meeting to learn about this system and the current challenges you face with Kueue, if any.

You can find details about the meeting here https://github.com/kubernetes/community/tree/master/wg-batch

[alculquicondor]

Btw, we also maintain the K8s job-controller and jobset, which are the recommended way of creating groups of pods. It's a better pattern than plain pods.

[romilbhardwaj]

Hey @alculquicondor - thanks for your interest! I've added SkyPilot to the future agenda for wg-batch meetings - let me know if 6th June works.

we also maintain the K8s job-controller and jobset, which are the recommended way of creating groups of pods. It's a better pattern than plain pods.

We've looked into jobset/job-controller, but concluded plain pods are the best way to implement the SkyPilot cluster abstraction since lifecycle management is done by the SkyPilot control plane:

• SkyPilot "clusters" do not have a fixed lifecycle like jobs do. E.g., even if the task terminates, a SkyPilot cluster still stays up and can have more tasks queued to it by users.
• Deployments are closer to what we want, but SkyPilot currently cannot handle automatic pod restarts.
• As a result, plain pods + no lifecycle management from k8s is what we built.

That said, we're open to other ideas and suggestions!