[k8s] GPU Feature discovery label formatter

[asaiacai]

Resolves #2460

This allows k8s to consume the node label nvidia.com/gpu.product created by GPU feature discovery which is commonly deployed through the NVIDIA GPU operator

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Manual test: test against GKE labels (tested against T4)
• Manual test: test against skypilot labeler script labels on EKS deployed via eks_test_cluster.yaml
• Manual tests: deploy k3s with gpu-operator using deploy_k3s.sh modified to exclude the skypilot k8s labeler, ensure the following can run

# check nvidia-smi and nvidia.com/gpu.product info
nvidia-smi --query-gpu=name --format=csv,noheader,nounits
kubectl describe node | grep nvidia.com/gpu.product
# test skypilot against gpu type
sky show-gpus --cloud kubernetes
sky launch --cloud kubernetes --gpus <GPU_TYPE>

• A100-80GB
• A100
• H100
• T4
• V100
• A10G
• P100
• P4
• L4

[Michaelvll]

This is awesome @asaiacai! It looks very reasonable to me. @romilbhardwaj for another look to make sure it does not break our other formatters : )

[romilbhardwaj]

Thanks @asaiacai!

[romilbhardwaj]

Thanks @asaiacai. Tested on A100 and H100 from Lambda. Left a comment about documenting that this labelformatter cannot be used with autoscaling, otherwise lgtm!

[asaiacai]

just added the docstring @romilbhardwaj . Thanks for the review! lmk if i this needs anything else.

[romilbhardwaj]

Thanks @asaiacai!