Why sflow

Write the workload once. Run it on any GPU infrastructure.​

A workstation, a Slurm cluster and a Kubernetes cluster can all run the same GPU job, but each one wants its own launch scripts. Teams end up maintaining one set of bash per platform for what is, logically, the same workflow.

sflow splits a workflow into two files:

• the workload: what runs, in what order, on how many GPUs, and how to tell it is healthy;
• the backend: how this particular platform launches it.

Moving to another platform means swapping the backend file. The workload does not change.

One workload, three platforms​

Here is the model server from a real example. Notice what is missing: no srun, no docker run, no pod spec, no host names.

- name: server
  operator: server
  script:
    - set -x
    - >
      vllm serve ${{ artifacts.LOCAL_MODEL_PATH.path }}
      --served-model-name ${{ variables.SERVED_MODEL_NAME }}
      --host 0.0.0.0 --port 8000
      --tensor-parallel-size ${{ variables.SERVER_GPUS }}
      --trust-remote-code
  resources:
    gpus:
      count: ${{ variables.SERVER_GPUS }}
  required_by:
    - benchmark
  probes:
    readiness:
      http_get:
        url: "http://${{ backends.cluster.nodes[0].ip_address }}:8000/health"
      timeout: 1200
      interval: 20
    failure:
      log_watch:
        match_pattern: "Traceback (most recent call last)"
        match_count: 5
      interval: 30

operator: server names a role, and backends.cluster.nodes[0] is whichever node the platform hands out. Each backend file says what the server role means there:

- name: server
  type: docker_run
  image: ${{ variables.SERVER_IMAGE }}
  extra_args:
    - --network=host          # client reaches the server on 127.0.0.1
    - --ipc=host              # NCCL / shared-memory transports between workers
    - --shm-size=${{ variables.SERVER_SHM_SIZE }}

- name: server
  type: srun
  container_image: ${{ variables.SERVER_IMAGE }}
  container_writable: true
  mpi: pmix
  ntasks_per_node: 1

- name: server
  type: k8s
  image: ${{ variables.SERVER_IMAGE }}

The three runs differ in exactly one file:

cd examples/modular/backend_agnostic
sflow run -f backends/docker.yaml -f workloads/vllm_serve.yaml -f benchmark.yaml   # one workstation
sflow run -f backends/slurm.yaml  -f workloads/vllm_serve.yaml -f benchmark.yaml   # Slurm cluster
sflow run -f backends/k8s.yaml    -f workloads/vllm_serve.yaml -f benchmark.yaml   # Kubernetes

Add --dry-run to any of them to see the whole plan (placement, GPU assignment, readiness checks) without touching a cluster. The same workload directory holds Dynamo, vLLM, SGLang and TensorRT-LLM servers, aggregated and disaggregated, all written this way: browse the example on GitHub.

What changes, and what doesn't​

What that buys you:

• Prove it small, run it big. Debug on a workstation with Docker, then submit the identical workload to the cluster.
• Write platform glue once. One person writes the Slurm or Kubernetes backend file; everyone else reuses it.
• Compare platforms fairly. Same workload, same benchmark, different backend: the comparison is like for like.

Built for the hardest case​

Multi-node LLM serving is about as hard as workflow orchestration gets: a dozen processes on several nodes that must start in the right order, find each other, share GPUs correctly, and be torn down cleanly when any one of them dies. sflow ships that as ordinary recipes:

• Disaggregated serving (separate prefill and decode workers exchanging the KV cache) with Dynamo on SGLang, vLLM and TensorRT-LLM, on Slurm and on Kubernetes (recipes).
• Workers that span nodes: prefill and decode workers at tensor parallelism 8, each spread over two 4-GPU GB200 NVL72 nodes through multi-node NVLink (recipe).
• Large MoE models: DeepSeek-R1 disaggregated serving (recipe).

Growing a run changes numbers, not structure: NUM_NODES, the GPUs per task, replicas to fan a task out, or sflow batch over a CSV of parameter rows. Along the way sflow handles:

• Placement: packs tasks onto nodes and specific GPUs, and reports conflicts when it plans the run, not ten minutes into it.
• Native launchers: srun with MPI on Slurm, pods (single or MPI) on Kubernetes, docker run on a workstation. sflow drives each platform's own tools rather than replacing them.
• Real readiness: a dependent task waits until the service it needs actually answers, not just until its process starts.
• Clean teardown on success, failure or Ctrl-C, and one output directory per run with a log per task.

The bash you stop writing​

What sflow is not​

• Not a scheduler. Slurm, Kubernetes and Docker still schedule. sflow plans the workflow and launches it through them.
• Not a service. It is a command-line tool and YAML files: no server, database or daemon to operate.
• Not only for LLMs. Anything that runs as steps depending on each other fits: build, train, evaluate; stage data, process shards in parallel, merge. LLM serving is the hardest case it is designed around, not the only one.

Try it in a minute​

uv tool install nv-sflow
sflow sample self_contained/local/hello_world -o hello.yaml
sflow run -f hello.yaml

Or paste this into your AI coding agent and let it install sflow and its skills:

Set up sflow for me by following
https://raw.githubusercontent.com/NVIDIA/nv-sflow/main/docs-site/agents-src/setup.md
step by step, then tell me what you installed and where.

Next: Quickstart · Modular workflows · Backends · GitHub