Install Dependencies

Prerequisites

• An operational Kubernetes cluster with recommended instance types (see Create K8s Cluster)

• Helm CLI installed

Install KAI Scheduler

OSMO uses KAI scheduler to run AI workflows at very large scale with Workflow Groups.

For more information on the scheduler, see Scheduler Configuration.

Run these commands from the root of an OSMO repository clone. The checked deployments/charts/osmo/examples/kai-values.yaml file defines the scheduler and admission behavior tested with OSMO.

The converged Quickstart, Self-contained, and Single-plane guides layer the checked deployments/charts/osmo/examples/kai-selectors.yaml file to place KAI components on nodes labeled osmo.nvidia.com/node-pool=control-plane:

global:
  nodeSelector:
    osmo.nvidia.com/node-pool: control-plane

Install the tested KAI Scheduler release with the common values:

helm upgrade --install kai-scheduler \
  https://github.com/NVIDIA/KAI-Scheduler/releases/download/v0.15.3/kai-scheduler-v0.15.3.tgz \
  --namespace kai-scheduler \
  --create-namespace \
  --values deployments/charts/osmo/examples/kai-values.yaml \
  --wait \
  --timeout 10m
kubectl --namespace kai-scheduler wait \
  --for=condition=Available=True \
  --timeout=10m config.kai.scheduler/kai-config
kubectl wait --for=condition=Available \
  --timeout=10m schedulingshard/default
kubectl --namespace kai-scheduler wait \
  --for=condition=Available \
  --timeout=10m deployment --all

Add that selector file after the common --values option only for clusters that use the documented converged node labels. Split compute clusters use their own KAI placement policy. Copy and edit the selector file when a cluster uses a different label.

Note

For newer compatible versions, refer to the official KAI Scheduler release notes .

Install the GPU Operator

The NVIDIA GPU Operator is required for GPU workloads to be discovered and scheduled.

helm repo add nvidia https://nvidia.github.io/gpu-operator
helm repo update
helm install gpu-operator nvidia/gpu-operator --namespace gpu-operator --create-namespace

Note

For optional observability components such as Grafana, Prometheus, and Kubernetes Dashboard, see Add Observability (Optional).