Native Sidecars¶
Native sidecar mode runs a framework's native engine beside a Dynamo sidecar that connects it to the Dynamo frontend. This guide covers configuration, worker placement, runtime compatibility, and the vLLM support timeline.
Configuration¶
Set sidecar: true on every role to run the framework's native engine process beside a CPU-only Dynamo sidecar instead of launching python3 -m dynamo.<framework>. The mode is job-wide, so every role must agree; the sidecar knobs (sidecar_port, sidecar_args, ...) stay under dynamo. dynamo.sidecar: true is the equivalent job-wide spelling. The engine and sidecar share one Slurm step and have a coupled lifecycle: if either exits, srtctl terminates the other and marks the worker failed.
vLLM jobs behind the Dynamo frontend that do not set sidecar: true still run the Python dynamo.vllm worker, and srtctl logs a warning when it loads them; sidecar mode will become the default for vLLM. The vLLM Dynamo examples set sidecar: true on every role, except features/vllm-failover.yaml: shadow engine recovery runs in the Python worker, so srtctl does not warn for engine.failover.
For SGLang, srtctl also adds --incremental-streaming-output to the engine (the sidecar consumes deltas) and sets two variables in the worker environment: SGLANG_RUST_SERVER=1, because the sidecar speaks sglang.api.v1, which only SGLang's Rust server serves, and SGLANG_RUST_BUILD_MODE=never, so the native extensions are loaded from the image instead of being rebuilt with cargo, which images that run SGLang from a source checkout cannot do. Set any of them in the role's args or env to override.
By default, srtctl launches python3 -m dynamo.<framework>.sidecar. The ai-dynamo package supplies this module and pins the matching ai-dynamo-runtime wheel, which embeds the native Rust sidecar. The configured Dynamo source or preinstalled container runtime must include the selected framework's launcher. No separate Cargo build is performed at job startup.
Nightly deployments should select an exact dynamo.source.wheel version so srtctl stages and installs the matching ai-dynamo and ai-dynamo-runtime artifacts on every worker. Set dynamo.sidecar_binary only to launch a compatible standalone executable already present in the container or a bind mount.
frontend:
type: dynamo
engine: vllm # sglang, vllm, or trtllm
roles:
agg:
nodes: 1
workers: 1
sidecar: true
args:
tensor-parallel-size: 8
dynamo:
source:
wheel: "<nightly-with-sidecars>"
sidecar_port: 19000
sidecar_args:
- --grpc-connections
- "4"
The default sidecar commands are python3 -m dynamo.sglang.sidecar, python3 -m dynamo.vllm.sidecar, and python3 -m dynamo.trtllm.sidecar. All three use the shared --grpc-endpoint flag.
TensorRT-LLM supports aggregate, prefill, and decode workers through OpenEngine gRPC. Each endpoint runs one sidecar on global MPI rank zero; prefill and decode sidecars register their disaggregation roles. The engine receives the configured parallelism and served model name, while the sidecar reads tokenizer artifacts from the model path. dynamo.sidecar_context_length can override the context length inferred from the corresponding role's args.max_seq_len. The image must include compatible TensorRT-LLM OpenEngine and Dynamo sidecar builds.
The disaggregated TRT-LLM example places a prefill and decode worker with sidecars on separate nodes.
For a TRT-LLM hf: model, set model.hf_snapshot_path to the absolute path of its commit-specific snapshot inside the container, such as /hf/hub/models--org--name/snapshots/<40-character-commit>. Set identity.model.revision to the same commit and mount the full Hugging Face hub cache at /hf/hub so the snapshot's links to blobs/ resolve. The TRT-LLM workers, sidecars, and sa-bench tokenizer then use that snapshot path; the served model name remains the configured model ID. An absent snapshot fails the launch instead of selecting a moving cache reference.
Metrics¶
A sidecar job has two kinds of worker metrics, served by different processes on different ports:
- Dynamo runtime metrics. Every Rust sidecar starts the Dynamo runtime's system status server on
DYN_SYSTEM_PORT(the worker's allocated system port) and serves thedynamo_component_*runtime, request-handler, and lifecycle families at/metrics; the vLLM and SGLang KV relays addkv_publisher_*. Only a process that runs a sidecar binds this port. - Native engine metrics. No sidecar bridges or polls the engine's own metrics (ai-dynamo/dynamo
3e72af98f8a5), so the engine's Prometheus families come from the engine's own HTTP server, on the worker's allocated HTTP port.
Tachometer scrapes both, and built-in AIPerf runners receive the same URLs in AIPERF_SERVER_METRICS_URLS. Custom benchmarks keep one URL per logical worker, aligned with SRT_*_ENDPOINTS: the leader's native engine endpoint, or the sidecar's runtime endpoint where the engine serves none (see Benchmarks). Every target has the worker's hostname, worker_index, worker_process, and worker_role metadata and the backend filter:
| Target name | URL |
|---|---|
backend_{mode}{index}_rank{rank} |
http://<node>:<system port>/metrics, the sidecar's runtime metrics |
backend_{mode}{index}_rank{rank}_engine |
http://<node>:<HTTP port>/metrics (/prometheus/metrics for TRT-LLM), the native engine's metrics |
{rank} is the process's rank within the worker: the node rank, or for vLLM hybrid DP the node's first global DP rank.
| Engine | Sidecar runtime endpoint | Native engine endpoint |
|---|---|---|
| vLLM | Every node that runs the Rust frontend: the single-node worker, each hybrid-DP node, and the leader of a cross-node TP/PP replica | Same nodes: vllm-rs serves vllm:* families labelled per DP engine (model_name, engine) on its HTTP server, which always runs beside gRPC and binds 0.0.0.0 in sidecar mode (vLLM b272cff368). --disable-log-stats turns the engine series off. Headless followers serve neither |
| SGLang | Every node that hosts Rust listeners (topology) | Every node's --port, with --enable-metrics, which srtctl passes unless the role's args set enable-metrics. On a node with Rust listeners, the first listener's HTTP server serves the node's merged series: the Rust frontends' request histograms (TTFT, ITL, end-to-end latency) and the scheduler series of the node's DP ranks (SGLang d89a1b1ae3). A node without listeners runs SGLang's health server, whose /metrics carries the scheduler series of the DP ranks on that node, labelled dp_rank. srtctl allocates every node its own HTTP port for it |
| TRT-LLM | The endpoint leader, where global MPI rank zero runs the sidecar | The leader's HTTP port at /prometheus/metrics, once the role's args opt into performance metrics (return_perf_metrics: true or perf_metrics_output_dir); otherwise none |
TRT-LLM's OpenEngine gRPC server (tensorrt_llm.commands.serve --grpc --grpc-protocol openengine) replaces the OpenAI HTTP server. Since NVIDIA/TensorRT-LLM#21034 it starts a separate Prometheus listener (--openengine-metrics-host, --openengine-metrics-port) when the engine collects performance metrics, serving the shared trtllm_* collector at /prometheus/metrics: the per-request histograms and counters, plus the iteration gauges (trtllm_kv_cache_*, running and waiting requests) when the role also sets enable_iter_perf_stats: true. srtctl binds that listener to every interface on the leader's allocated HTTP port and passes the flags only when the opt-in is on, so a recipe for a container older than that commit (which rejects the flags) keeps the opt-in off and has no engine endpoint. The sidecar bridges neither these metrics nor the GetLoad RPC. For built-in AIPerf runners, TRT-LLM sidecar URLs also follow the engine.publish_events_and_metrics gate described in Benchmarks; custom benchmarks always receive one URL per leader, the engine listener when the opt-in is on and the sidecar's runtime URL otherwise, and Tachometer scrapes every target regardless.
SGLang topology and launch requirements¶
SGLang's Rust server runs one listener per DP rank, on the node that holds the rank's first scheduler (pipeline stage 0, attention TP and CP rank 0); listener k of a node binds --port + k and --grpc-port + k (SGLang d89a1b1ae3). srtctl runs one sidecar on every node that hosts listeners and points it at the node's first listener; the sidecar discovers the others and registers all of the node's DP ranks as one worker. With attention DP across nodes, every node that owns a DP group's first rank runs a sidecar; TP-only followers and later pipeline stages host no listener and run only the engine. For example, TP8/DP2 across four nodes runs sidecars on nodes 0 and 2, and readiness requires both. Each node reserves one sidecar gRPC port per listener from dynamo.sidecar_port upward, and its HTTP port block leaves room for its HTTP listeners.
Prefill serves the KV bootstrap registry on each listener's HTTP port, so srtctl passes no --disaggregation-bootstrap-port (SGLang rejects one that differs from --port). Each prefill node's sidecar advertises that node's address with --bootstrap-host, so decode fetches KV from the node that holds it.
The multinode attention-DP example uses one logical aggregate worker across two one-GPU nodes, with a custom prebuilt image and KV routing. Set the image path before submitting. Restart the entire distributed engine group and its sidecars together after a failure.
The disaggregated multinode attention-DP example uses four one-GPU nodes: one TP2/DP2 prefill worker on the first pair and one TP2/DP2 decode worker on the second pair. Every node runs a sidecar for its DP rank. Prefill enables KV events; decode keeps prefix caching disabled by default and sets kv_events: false. KV routing can reuse prefixes on either prefill DP rank. Decode prefix-cache affinity is not enabled or implied. srtctl supplies the role, distributed rendezvous, and each prefill node's routable --bootstrap-host; the recipe selects NIXL transfer on both roles. The image must include a Dynamo SGLang sidecar that speaks sglang.api.v1, SGLang's prebuilt Rust server, and NIXL.
vLLM topology and launch requirements¶
For multi-node vLLM data parallelism, srtctl launches one Rust gRPC frontend and one Dynamo sidecar on every node using hybrid load balancing. All nodes receive the global --data-parallel-size, while --data-parallel-size-local and --data-parallel-start-rank restrict each frontend to its colocated engines. For example, DP12 on three four-GPU nodes launches local DP4 with starting ranks 0, 4, and 8. Dynamo registers three endpoints, each covering its four local ranks. No node runs --headless, and readiness requires all three sidecars. vLLM sidecars require engine.dp_launch_mode: per_node (the default; backend.dp_launch_mode in schema 1); per_gpu is rejected during configuration validation, including for single-rank jobs.
The multi-node DP command uses Python-supervised Rust frontends (python3 -m vllm.entrypoints.cli.main serve with VLLM_USE_RUST_FRONTEND=1). Python coordinates the shared DP rendezvous and starts vllm-rs frontend with each node's local engine count and starting rank; requests are handled by Rust. This is a current launcher limitation: vllm-rs serve expects to own the complete DP group and does not implement hybrid startup. Python supervision is not a fundamental requirement of the sidecar architecture. Single-node sidecars continue to use vllm-rs serve. Because Python vllm serve parses the engine arguments itself and disables its reasoning parser by omission, srtctl drops reasoning-parser: none (the vllm-rs serve value Dynamo sidecars expect) on this launcher, where the bootstrapped Rust frontend already defaults to no parser; reasoning-parser: auto has no equivalent there and is rejected.
Each Rust frontend has one API server; do not set api-server-count to another value or enable grpc (which selects the separate Python gRPC server) or data-parallel-external-lb.
For a TP/PP replica spanning nodes (including TP+EP with data-parallel-size: 1),
srtctl starts the Rust gRPC frontend and Dynamo sidecar only on the leader.
Followers run native vllm serve --headless, without an API server or sidecar.
All nodes share the leader address on the configured network interface and a
per-endpoint rendezvous port, with derived --nnodes, --node-rank, and
--distributed-executor-backend mp. TP × PP × PCP must equal the total allocated
GPUs, distributed evenly across nodes. For example, TP8 on two four-GPU nodes is
one Dynamo registration, not two. Recipe rendezvous overrides are replaced by
these allocation-derived values; api-server-count is omitted on followers.
With the default role setting critical: true, an unexpected exit on any node
fails the job and stops all workers through the job-wide process monitor. Multiple cross-node DP replicas within one endpoint
remain unsupported; use separate endpoints with DP=1 instead.
This path requires the same Python-supervised Rust frontend --grpc-port
integration described below. Native vLLM headless support alone is insufficient
to provide the leader's sidecar transport.
Hybrid sidecars require compatible changes in both projects: vLLM's Python serve command must support --grpc-port for its Rust frontend (vLLM #59659), the Rust Control service must report ParallelismInfo.data_parallel_size_local (vLLM #57116), and Dynamo's sidecar must register that local range using the reported global start rank. Equivalent backports are sufficient. The --grpc-port requirement also applies to the leader of a cross-node TP/PP replica. Without it, Python exits with an unrecognized-argument error before the sidecar starts; switching to vllm-rs serve cannot recover hybrid DP because its launcher does not implement that topology.
Pin a compatible image or source build; merely upgrading the Dynamo wheel is insufficient. srtctl does not automatically verify these capabilities in the worker image. To use a specific Rust binary in the multi-node path, set VLLM_RUST_FRONTEND_PATH in the recipe's worker environment. srtctl preserves that path and supplies the Rust frontend selection automatically.
vLLM sidecar mode sets VLLM_PLUGINS to an empty value by default. This prevents image-installed plugins from replacing native engine output types that must match the fixed vllm-rs MessagePack contract. A recipe can explicitly set VLLM_PLUGINS in a role's env when every selected plugin is compatible with the sidecar protocol.
vLLM sidecar versions and timeline¶
The following checks describe the native vLLM sidecar path, as of 2026-10-06. They are interface requirements, not an end-to-end qualification of every model, connector, or topology. Direct vLLM jobs and other backends have different requirements.
For multi-node hybrid DP, use srt-slurm with #420 (4bb5f649), a vLLM main/nightly build containing #59659 (9d2a6f52b2) and #57116 (9639cbde04), and a Dynamo main/nightly build containing #14697 (805a77f053). Equivalent backports also satisfy these interfaces. Cross-node TP/PP additionally needs srt-slurm #457 (71113bbd); its leader needs the Python gRPC-port support, while the hybrid local-DP ownership requirement applies to hybrid DP.
Minimum vLLM main/nightly for both multi-node paths: use 0.30.1rc1.dev562+g9d2a6f52b, published for x86_64 and aarch64 in the official commit wheel index, or a later main build containing commit 9d2a6f52b264c20b5816c5a094b191bfbcadd74b (merged October 2, 2026). This commit includes the earlier local-DP metadata and protocol changes. Pin the worker image or source/wheel revision to that commit or a descendant; a build dated October 2 can still predate the merge. The compatible Dynamo build above remains required for hybrid DP.
Treat this as a commit cutoff, not a package-version >= constraint: stock 0.31.0 is numerically newer but lacks the required Python CLI change. vLLM's commit-specific nightly indices allow an exact revision to be selected. The wheel labels below are copied from those indices, including the historical 0.2.1.dev* labels; they are not a monotonic compatibility scale. For container packaging changes, verify the image's source revision as well as its installed wheel.
Release dates alone do not establish that a change is included: release branches can omit changes already merged to main. The table below was checked against the tagged source, including the Python CLI field, the protobuf local-DP field, and Dynamo's local-range discovery code.
| Component / stock release | Relevant interfaces included | Remaining requirement for multi-node hybrid DP |
|---|---|---|
| vLLM 0.29.0 | Neither Python --grpc-port nor local-DP Control metadata |
Both #59659 and #57116, plus a compatible Dynamo build |
| vLLM 0.30.0 | Engine-error propagation and selected-token logprob fixes; neither hybrid prerequisite | Both #59659 and #57116, plus a compatible Dynamo build |
| vLLM 0.31.0 | Local-DP metadata, vllm-proto 0.4.0 (including the local-DP field introduced in 0.3.0), appended-output-field compatibility, and vllm-rs on the CUDA image's PATH |
Still lacks Python --grpc-port (#59659); backport it or use a main/nightly build containing it |
| Dynamo 1.5.0 | Wheel-installed vLLM sidecar launcher and complete-group DP routing | Lacks hybrid local-range ownership (#14697); use a main/nightly build containing it or an equivalent backport |
For example, vLLM 0.31.0 plus #59659, together with Dynamo containing #14697, satisfies these hybrid launch/discovery prerequisites. Stock vLLM 0.31.0 plus stock Dynamo 1.5.0 does not. Pin matching ai-dynamo and ai-dynamo-runtime artifacts, and keep Python vLLM and its Rust frontend from the same build. Single-node sidecars already use vllm-rs serve, so they do not require Python's --grpc-port addition; the image must still provide the compatible Rust executable on PATH.
Merge dates below use America/Los_Angeles time. The vllm-proto crate version is separate from the vLLM Python package version.
| Merge date (2026) | Change | Effect on srt-slurm sidecars |
|---|---|---|
| Aug 27 | Dynamo #13923 | Adds the wheel-installed python3 -m dynamo.vllm.sidecar launcher |
| Sep 11–14 | vLLM #56405, #56406 | Propagates engine generation errors and preserves selected-token logprob mode; these are serving fixes, not the hybrid launch prerequisites |
| Sep 16 | vLLM #57116 | Reports the frontend's local DP size. Published main wheel: 0.2.1.dev29+g9639cbde0. Still needs the Python --grpc-port change for hybrid DP |
| Sep 16 | vLLM #57233 | Bumps vllm-proto to 0.3.0 for downstream consumers of local-DP metadata. Published main wheel: 0.2.1.dev59+gd2a2a8295 |
| Sep 16 | vLLM #56533 | Accepts appended EngineCoreOutput fields, avoiding failures from plugins such as vLLM-Omni; does not add full Omni serving support |
| Sep 18 | Dynamo #14697 | Consumes local-DP metadata and registers each frontend's actual rank range and per-rank KV capacity |
| Sep 18 | srt-slurm #420, #457 | Launches Python-supervised Rust frontends per hybrid-DP node, or one frontend with headless followers for cross-node TP/PP |
| Sep 21 | vLLM #57606 | Exposes the bundled vllm-rs on the CUDA image's PATH for the single-node launcher. The same commit's wheel is 0.29.1rc1.dev507+gf07e227ee; installing that wheel alone does not apply the Dockerfile's PATH fix. Multi-node paths still need #59659 |
| Oct 2 | vLLM #59659 | Exposes Python vllm serve --grpc-port in 0.30.1rc1.dev562+g9d2a6f52b. Completes the vLLM launch/discovery prerequisites for hybrid DP and cross-node TP/PP with the srt-slurm and Dynamo revisions above |