sflow — Declarative Workflow Descriptor for Massive GPU Clusters

01 / 18

NV-SFLOW

Workflow orchestrator for large-scale GPU clusters of any kind

Describe once. Run anywhere._

vLLMSGLangTensorRT-LLMDynamoPyTorch…

KubernetesSlurmDocker

View Documentation → Feature Map → GitHub

Scroll

02 / 18

Workload Evolution

From local host container, to cluster level multi-stage service

Deployment scalewhere it runs

BEFORE

Single local host

1 node · 1-8 GPUs · nothing to coordinate across machines

➔

NOW

Multi-node cluster service

Tens to hundreds of nodes · cross-node traffic and topology · schedulers, quotas and lifecycle

Serving architecturehow it is served

BEFORE

Agg Server→Client

One stage · one command brings the service up

➔

NOW

Router→PD Disagg→KV Transfer→Heterogeneous Compute

Many stages · many components · each with its own resources and readiness

The complexity moved: it is the same model — but getting it running went from starting one process to orchestrating a whole distributed system.

03 / 18

Perspective Shift

A wider orchestration lens: from one machine to one cluster

Single-host viewAll five layers sit on the same machine, so they line up by default — install it, run it, done.

➔

Cluster viewThe same five layers exist on every node, and they all have to agree — one layer out of step and the whole workflow fails.

↑BOTTOM-UP

node-0node-1node-2node-3

App workflowDAG · task deps · probes

Across nodes: dependencies and readiness

SoftwarevLLM · SGLang · TensorRT-LLM · images

Across nodes: matching versions and images

Infra platformSlurm · Kubernetes · Docker

Across nodes: scheduling, quota, network

DriverCUDA · NCCL · NIC drivers

Across nodes: driver and firmware parity

HardwareGPU · NVLink · IB · topology

Across nodes: same topology and visibility

So the thing being orchestrated changed: not “start one process”, but compose all five layers as a whole and keep every node sharing the same information and the same conventions.

04 / 18

Platform Fragmentation

Every platform speaks its own dialect

Slurm

#SBATCH --gres=gpu:8 srun --ntasks=16 ...

Partitions, accounts and QoS · sbatch syntax · hostlist expansion

LEARNING CURVE

Kubernetes

resources.limits: nvidia.com/gpu: 8

Manifests and CRDs · labels and affinity · RBAC and namespaces

LEARNING CURVE

Docker

docker run --gpus all --network host ...

Images and mounts · device mapping · compose syntax

LEARNING CURVE

App iteration

edit run blocked — waiting on platform work edit

Platform work

learn the syntax · match the schema · fix permissions · patch compatibility

The app is what actually gets slowed down: the model and the parameters changed by one line, but the time all goes into making it run on this platform — swap the infrastructure platform and you rewrite the whole workflow orchestration.

05 / 18

The Problem

The workload stays the same. The platform adaptation multiplies.

Stable workflow: etcd → NATS → frontend → GPU workers → benchmark

repeated concern

Slurm

Docker

Kubernetes

Launch

sbatch + srun

docker run

Pod / Job

GPU placement

GPU flags

device mapping

claims + affinity

Networking

hostlists + ports

host network

Service + DNS

Readiness

shell polling

healthcheck

probes

Logs + cleanup

sacct + files

docker logs

relay + lifecycle

5 concerns × 3 platformsthe same orchestration work, repeated around one stable workflow

06 / 18

Backend Agnostic by Design

Keep what. Swap where. Rewrite nothing.

✎

One Semantic

A deployment's logic never changes — only the infrastructure adaptation does.

⇄

Swap the Backend Block

The same sflow.yaml runs on Docker, Slurm and Kubernetes.

⚙

Delegate, Don't Reimplement

Backends defer to each platform's native ecosystem instead of rebuilding it.

07 / 18

DAG Orchestration

You define tasks and dependencies

Directed Acyclic Graph each stage fires as soon as its dependencies are ready

load_image

▼

install_dependency

gpu_monitor

▼

nats_server

etcd_server

▼

frontend_server_0

frontend_server_1

frontend_server_2

▼

nginx_server

prefill_server_0

prefill_server_1

prefill_server_2

prefill_server_3

decode_server_0

▼

benchmark_client

✎

Tasks = any command

bash, python, or a container entrypoint.

⇄

Dependencies = any DAG

Serial, parallel, fan-in and fan-out. No cap on stages.

∞

Not tied to a domain

Inference, training, evaluation, data processing.

08 / 18

Cluster-Level Orchestration at Scale

Topology-aware GPU Allocation

sflow topology plannerGPU capacity · node affinity · task dependencies

▼ backend-native assignment

node-0NVLink domain A · 4 GPUs

GPUGPU 0prefill_0

GPUGPU 1prefill_1

GPUGPU 2prefill_2

GPUGPU 3prefill_3

node-1NVLink domain B · 4 GPUs

decode_0 · 4-GPU task

GPUGPU 0decode_0

GPUGPU 1decode_0

GPUGPU 2decode_0

GPUGPU 3decode_0

Node and GPU placement, multi-node replicas and sweeps, readiness probes, and batch submission — so one descriptor drives hundreds of GPUs instead of a pile of hand-written bash.

09 / 18

Workflow Lifecycle

Define. Execute. Understand.

Define

DAGTasks + dependencies
One declarative workflow

{ }Variables + artifacts
Portable inputs and expressions

✓Probes
Readiness and failure gates

➔

Execute

GPUResource planning
Nodes, GPUs, topology

N×Replicas + sweeps
Parallel or sequential

→Backend adapters
Local, Docker, Slurm, K8S

➔

Understand

TUILive status
Tasks, logs, readiness

∿Summary + monitor
Timelines and hardware charts

⇧Results + uploads
JSON metrics and S3

10 / 18

Backend Adapters

One intent becomes four native execution plans

Portable workflow contract DAG · resources · probes · artifacts · results

▼ sflow delegates to each platform's native ecosystem

>_

Local

bash process
synthetic node/GPU map

▣

Docker

docker run
container GPU devices

#

Slurm

salloc + srun
scheduler allocations

⎈

Kubernetes

pods + claims
pinning and lifecycle

▼

Consistent outputs task.log · result.json · summary · artifacts

11 / 18

Merge Pod

Co-located GPU tasks, one fast intra-node fabric

Separate pods · isolated views

worker AGPU 0-1 visible

IB / TCP
↔

worker BGPU 2-3 visible

sflow merge
➔

One pod · combined GPU view

worker Aworker B

GPU 0GPU 1GPU 2GPU 3

CUDA IPC · NVLink · NVSwitch

Safe boundary: same workflow + backend + node + image. Example: PD KV transfer. Also fits current per-node IMEX constraints.

12 / 18

Observability

Runs explain themselves

sflow_summary.log Task Duration Chart prefill |####..........................| 58.412s READY decode |##########################....| 6m14.882s READY benchmark |..........................####| 47.906s COMPLETED Timeline 12:31:33 +00.000s prefill SUBMITTED 12:31:33 +00.000s decode SUBMITTED 12:32:31 +58.412s prefill READY 12:37:48 +06m15s decode READY 12:37:49 +06m16s benchmark SUBMITTED 12:38:36 +07m03s benchmark COMPLETED

sflow_monitor.log Metric Summary GPU util % min=42 avg=87 max=99 GPU mem used GiB min=18 avg=61 max=76 GPU power W min=310 avg=642 max=718 Timelines (cluster avg) GPU util % ▁▂▅▇████▆▅ GPU mem used GiB ▁▃▄▆▇█████ Net RX MiB/s ▁▁▂▅▇▆▃▂▁

GPU Util87%

GPU Memory61 GiB

GPU Power642 W

13 / 18

Modular Composition

Swap a leaf. Reuse the tree.

recipe/composition root

backend/

local.yamldocker.yamlslurm.yamlk8s.yaml ✓

common/

common_workflow.yaml · shared DAG

workload/

vllm/ ✓sglang/trtllm/

benchmark/

aiperf.yaml ✓infmax.yaml

▼ selected leaves compose

composed.yaml k8s + shared workflow + vLLM + AIPerf

Swap one highlighted leaf; shared branches remain unchanged.

14 / 18

Real-world Debugging

Structured Error Analysis

Workflow: b200-fp8-low-latency-tep8-1p-1d Model: DeepSeek R1 FP8 | 2 nodes × 8 GPUs | ISL=8192, OSL=1024 Allocation Map ├─ node-01 (node 0) │ GPU 0-7: prefill_server_0 (TP=8) │ Also: load_image, nats, etcd, frontend, benchmark_* └─ node-02 (node 1) GPU 0-7: decode_server_0 (TP=8) Also: load_image, gpu_monitor Timeline 01:57:08 — load_image + install_aiperf submitted 01:59:10 — load_image COMPLETED on both nodes 01:59:43 — nats_server READY 01:59:45 — etcd_server READY 02:00:31 — frontend_server_0 READY (10.x.x.x) 02:05:14 — prefill + decode READY → benchmark_4 starts 02:05:20 — HTTP 500 — all benchmark requests fail 02:05:41 — Workflow finished (8m 33s) Error from frontend logs: Invalid TCP address 'dynamo_prefill.generate-58b49ce145f56609' Invalid TCP address 'dynamo_backend.generate-58b49ce145f5660b'

Diagnosis — sflow orchestration vs application error

Layer	Status	Detail
sflow	✓ OK	DAG executed, all tasks launched, probes passed
GPU alloc	✓ OK	8 GPUs/node, no overlap, TP=8 per server
Infra	✓ OK	etcd, NATS, frontend all READY
Routing	✗ FAIL	Frontend gets service names, not host:port
Benchmark	✗ FAIL	0/800 requests succeed (all HTTP 500)

Root Cause Frontend receives discovery service names (e.g. dynamo_prefill.generate-58b49...) instead of host:port addresses. NATS/etcd service registry returns internal identifiers that the TCP router can't parse. Fix: Verify DYN_REQUEST_PLANE and frontend networking config match Dynamo disagg routing.

15 / 18

CLI at a Glance

Command	Purpose	Key Flags
`sflow run`	Execute a workflow	`--dry-run` `--tui` `--set` `-f` (multi-file)
`sflow batch`	Generate sbatch scripts	`--submit` `--bulk-input` `--row`
`sflow compose`	Merge multiple YAMLs	`--resolve` `--missable-tasks` `-o`
`sflow visualize`	Render DAG graph	`--format png/svg/mermaid`
`sflow sample`	List / copy examples	`--list` `-o`
`sflow skill`	Install AI agent skills	`--list` `-o`
`sflow upgrade`	Reinstall in place (alias: `sflow update`)	`--branch` `--dry-run` `--force`

16 / 18

Developer Experience

Agent-native workflows improve with every run

01

Author · agent-native

AGENTS.md workflow ruleswriting-sflow-yaml skillschema + examples

➔

02

Validate

sflow run --dry-runresolved variablesresource / GPU plan

➔

03

Run & observe

timestamped task eventsTUI + readiness probessflow_monitor.log

➔

04

Diagnose & evolve

sflow_summary.logtask.log + failure hintsresult.json metrics

AGENT LOOP

timestampssummarymonitorfailure hints

rerun ← adjust YAML ← read context

17 / 18

Vision

Heterogeneous Computing

One recipe can route each stage of a disaggregated pipeline to the best resource pool while preserving one workflow contract.

One DAG · specialized accelerators · one workflow contract

18 / 18

Try It

Install. Explore. Run.

Start locally, validate before allocating hardware, then use the docs and repository as your source of truth.

# install from the repository $ uv venv --python python3 && source .venv/bin/activate $ uv pip install "sflow @ git+https://github.com/NVIDIA/nv-sflow.git@main" # verify and try a local workflow $ sflow --version $ sflow sample self_contained/local/hello_world $ sflow run -f hello_world.yaml --dry-run ✓ validate first; run when the plan looks right

GITGitHub Repositorygithub.com/NVIDIA/nv-sflow→ DOCDocumentationnvidia.github.io/nv-sflow/docs→ 01Quickstartdocs/user/quickstart→ MAPFeature MapFind the right guide by intent→