Runbooks - Finetune LLMs on K8s with notebooks

NOTICE: Changes coming! runbooks.git (previously substratus.git) will be refactored to focus on Notebooks on K8s.

🎵 Fine-tune LLM models with no/low code
📔 Provide a Colab style seamless Notebook experience
☁️ Provide a unified ML platform across clouds
⬆️ Easy to install with minimal dependencies

Looking for serving?
🚀 substratusai/lingo: Serve popular OSS LLM models in minutes on CPUs or GPUs

Support the project by adding a star on GitHub! ❤️

Quickstart

Create a local Kubernetes cluster using Kind.

kind create cluster --name substratus --config - <<EOF
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30080
    hostPort: 30080
EOF

Install Substratus.

kubectl apply -f https://raw.githubusercontent.com/substratusai/substratus/main/install/kind/manifests.yaml

Import a small Open Source LLM.

kubectl apply -f https://raw.githubusercontent.com/substratusai/substratus/main/examples/facebook-opt-125m/base-model.yaml

apiVersion: substratus.ai/v1
kind: Model
metadata:
  namespace: default
  name: facebook-opt-125m
spec:
  image: substratusai/model-loader-huggingface
  params:
    name: facebook/opt-125m

Serve the LLM.

kubectl apply -f https://raw.githubusercontent.com/substratusai/substratus/main/examples/facebook-opt-125m/base-server.yaml

apiVersion: substratus.ai/v1
kind: Server
metadata:
  name: facebook-opt-125m
spec:
  image: substratusai/model-server-basaran
  model:
    name: facebook-opt-125m

Checkout the progress of the Model and the Server.

kubectl get ai

When they report a Ready status, start a port-forward.

kubectl port-forward service/facebook-opt-125m-server 8080:8080

Open your browser to http://localhost:8080/ or curl the LLM's API.

PS: Because of the small size of this particular LLM, expect comically bad answers to your prompts.

curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{ \
    "model": "facebook-opt-125m", \
    "prompt": "Who was the first president of the United States? ", \
    "max_tokens": 10\
  }'

Delete the local cluster.

kind delete cluster --name substratus

If you want to try out a more capable LLM, running on substantial hardware, try Kind with GPU support, or try deploying Substratus in GKE.

Docs

• Overview
• Architecture

Creators

Feel free to contact any of us:

• Nick Stogner
• Sam Stoelinga