sflow — Declarative Workflow Descriptor for Massive GPU Clusters

01 / 13

NV-SFLOW

Workflow orchestrator for large-scale GPU clusters of any kind

Describe once. Run anywhere._

vLLMSGLangTensorRT-LLMDynamoPyTorch…

KubernetesSlurmDocker

View Documentation → Feature Map → GitHub

Scroll

02 / 13

The Problem

The workload stays the same. The plumbing multiplies.

Stable workflow: etcd → NATS → frontend → GPU workers → benchmark

repeated concern

Slurm

Docker

Kubernetes

Launch

sbatch + srun

docker run

Pod / Job

GPU placement

GPU flags

device mapping

claims + affinity

Networking

hostlists + ports

host network

Service + DNS

Readiness

shell polling

healthcheck

probes

Logs + cleanup

sacct + files

docker logs

relay + lifecycle

5 concerns × 3 platformsthe same orchestration work, repeated around one stable workflow

03 / 13

Backend Agnostic by Design

Keep what. Swap where. Rewrite nothing.

✎

One Semantic

A deployment's logic never changes — only the infrastructure glue does.

⇄

Swap the Backend Block

The same sflow.yaml runs on Docker, Slurm and Kubernetes.

⚙

Delegate, Don't Reimplement

Backends defer to each platform's native ecosystem instead of rebuilding it.

04 / 13

DAG Orchestration

Dependencies activate the next stage

load_image

▼

install_dependency

gpu_monitor

▼

nats_server

etcd_server

▼

frontend_server_0

frontend_server_1

frontend_server_2

▼

nginx_server

prefill_server_0

prefill_server_1

prefill_server_2

prefill_server_3

decode_server_0

▼

benchmark_infmax_16

05 / 13

Cluster-Level Orchestration at Scale

Topology-aware GPU Allocation

sflow topology plannerGPU capacity · node affinity · task dependencies

▼ backend-native assignment

node-0NVLink domain A · 4 GPUs

GPUGPU 0prefill_0

GPUGPU 1prefill_1

GPUGPU 2prefill_2

GPUGPU 3prefill_3

node-1NVLink domain B · 4 GPUs

decode_0 · 4-GPU task

GPUGPU 0decode_0

GPUGPU 1decode_0

GPUGPU 2decode_0

GPUGPU 3decode_0

Node and GPU placement, multi-node replicas and sweeps, readiness probes, and batch submission — so one descriptor drives hundreds of GPUs instead of a pile of hand-written bash.

06 / 13

Workflow Lifecycle

Define. Execute. Understand.

Define

DAGTasks + dependencies
One declarative workflow

{ }Variables + artifacts
Portable inputs and expressions

✓Probes
Readiness and failure gates

➔

Execute

GPUResource planning
Nodes, GPUs, topology

N×Replicas + sweeps
Parallel or sequential

→Backend adapters
Local, Docker, Slurm, K8S

➔

Understand

TUILive status
Tasks, logs, readiness

∿Summary + monitor
Timelines and hardware charts

⇧Results + uploads
JSON metrics and S3

07 / 13

Backend Adapters

One intent becomes four native execution plans

Portable workflow contract DAG · resources · probes · artifacts · results

▼ sflow delegates to each platform's native ecosystem

>_

Local

bash process
synthetic node/GPU map

▣

Docker

docker run
container GPU devices

#

Slurm

salloc + srun
scheduler allocations

⎈

Kubernetes

pods + claims
pinning and lifecycle

▼

Consistent outputs task.log · result.json · summary · artifacts

08 / 13

Merge Pod

Co-located GPU tasks, one fast intra-node fabric

Separate pods · isolated views

worker AGPU 0-1 visible

IB / TCP
↔

worker BGPU 2-3 visible

sflow merge
➔

One pod · combined GPU view

worker Aworker B

GPU 0GPU 1GPU 2GPU 3

CUDA IPC · NVLink · NVSwitch

Safe boundary: same workflow + backend + node + image. Example: PD KV transfer. Also fits current per-node IMEX constraints.

09 / 13

Observability

Runs explain themselves

sflow_summary.log Task Duration Chart prefill |####..........................| 58.412s READY decode |##########################....| 6m14.882s READY benchmark |..........................####| 47.906s COMPLETED Timeline 12:31:33 +00.000s prefill SUBMITTED 12:31:33 +00.000s decode SUBMITTED 12:32:31 +58.412s prefill READY 12:37:48 +06m15s decode READY 12:37:49 +06m16s benchmark SUBMITTED 12:38:36 +07m03s benchmark COMPLETED

sflow_monitor.log Metric Summary GPU util % min=42 avg=87 max=99 GPU mem used GiB min=18 avg=61 max=76 GPU power W min=310 avg=642 max=718 Timelines (cluster avg) GPU util % ▁▂▅▇████▆▅ GPU mem used GiB ▁▃▄▆▇█████ Net RX MiB/s ▁▁▂▅▇▆▃▂▁

GPU Util87%

GPU Memory61 GiB

GPU Power642 W

10 / 13

Modular Composition

Swap a leaf. Reuse the tree.

recipe/composition root

backend/

local.yamldocker.yamlslurm.yamlk8s.yaml ✓

common/

common_workflow.yaml · shared DAG

workload/

vllm/ ✓sglang/trtllm/

benchmark/

aiperf.yaml ✓infmax.yaml

▼ selected leaves compose

composed.yaml k8s + shared workflow + vLLM + AIPerf

Swap one highlighted leaf; shared branches remain unchanged.

11 / 15

Real-world Debugging

Structured Error Analysis

Workflow: b200-fp8-low-latency-tep8-1p-1d Model: DeepSeek R1 FP8 | 2 nodes × 8 GPUs | ISL=8192, OSL=1024 Allocation Map ├─ slurm-node-01 (node 0) │ GPU 0-7: prefill_server_0 (TP=8) │ Also: load_image, nats, etcd, frontend, benchmark_* └─ slurm-node-02 (node 1) GPU 0-7: decode_server_0 (TP=8) Also: load_image, gpu_monitor Timeline 01:57:08 — load_image + install_aiperf submitted 01:59:10 — load_image COMPLETED on both nodes 01:59:43 — nats_server READY 01:59:45 — etcd_server READY 02:00:31 — frontend_server_0 READY (10.52.32.8) 02:05:14 — prefill + decode READY → benchmark_4 starts 02:05:20 — HTTP 500 — all benchmark requests fail 02:05:41 — Workflow finished (8m 33s) Error from frontend logs: Invalid TCP address 'dynamo_prefill.generate-58b49ce145f56609' Invalid TCP address 'dynamo_backend.generate-58b49ce145f5660b'

Diagnosis — sflow orchestration vs application error

Layer	Status	Detail
sflow	✓ OK	DAG executed, all tasks launched, probes passed
GPU alloc	✓ OK	8 GPUs/node, no overlap, TP=8 per server
Infra	✓ OK	etcd, NATS, frontend all READY
Routing	✗ FAIL	Frontend gets service names, not host:port
Benchmark	✗ FAIL	0/800 requests succeed (all HTTP 500)

Root Cause Frontend receives discovery service names (e.g. dynamo_prefill.generate-58b49...) instead of host:port addresses. NATS/etcd service registry returns internal identifiers that the TCP router can't parse. Fix: Verify DYN_REQUEST_PLANE and frontend networking config match Dynamo disagg routing.

12 / 15

CLI at a Glance

Command	Purpose	Key Flags
`sflow run`	Execute a workflow	`--dry-run` `--tui` `--set` `-f` (multi-file)
`sflow batch`	Generate sbatch scripts	`--submit` `--bulk-input` `--row`
`sflow compose`	Merge multiple YAMLs	`--resolve` `--missable-tasks` `-o`
`sflow visualize`	Render DAG graph	`--format png/svg/mermaid`
`sflow sample`	List / copy examples	`--list` `-o`
`sflow skill`	Install AI agent skills	`--list` `-o`
`sflow upgrade`	Reinstall in place (alias: `sflow update`)	`--branch` `--dry-run` `--force`

11 / 13

Developer Experience

Agent-native workflows improve with every run

01

Author · agent-native

AGENTS.md workflow ruleswriting-sflow-yaml skillschema + examples

➔

02

Validate

sflow run --dry-runresolved variablesresource / GPU plan

➔

03

Run & observe

timestamped task eventsTUI + readiness probessflow_monitor.log

➔

04

Diagnose & evolve

sflow_summary.logtask.log + failure hintsresult.json metrics

AGENT LOOP

timestampssummarymonitorfailure hints

rerun ← adjust YAML ← read context

12 / 13

Vision

Heterogeneous Computing

One recipe can route each stage of a disaggregated pipeline to the best resource pool while preserving one workflow contract.

One DAG · specialized accelerators · one workflow contract

13 / 13

Try It

Install. Explore. Run.

Start locally, validate before allocating hardware, then use the docs and repository as your source of truth.

# install from the repository $ uv venv --python python3 && source .venv/bin/activate $ uv pip install "sflow @ git+https://github.com/NVIDIA/nv-sflow.git@main" # verify and try a local workflow $ sflow --version $ sflow sample self_contained/local/hello_world $ sflow run -f hello_world.yaml --dry-run ✓ validate first; run when the plan looks right

GITGitHub Repositorygithub.com/NVIDIA/nv-sflow→ DOCDocumentationnvidia.github.io/nv-sflow/docs→ 01Quickstartdocs/user/quickstart→ MAPFeature MapFind the right guide by intent→