Shepherd Model Gateway (SMG)¶

frontend.type: smg runs the Shepherd Model Gateway in front of direct engine workers. SMG is an engine-agnostic Rust gateway with an OpenAI-compatible API; srtctl hands it the statically allocated worker URLs, the same way it does for the other static routers. No Dynamo, NATS, or etcd is involved.

Upstream references below are pinned to SMG v1.11.0.

Supported topologies¶

Topology	Backends	What srtctl launches
Aggregated	any engine whose workers serve HTTP (`sglang`, `vllm`, `trtllm`, ...)	`smg launch --worker-urls <url> ...`
Prefill/decode	`sglang`	`smg launch --pd-disaggregation --prefill <url> <bootstrap-port> --decode <url>`
Prefill/decode	`vllm` with `grpc: true` and `connector: nixl`	`smg launch --pd-disaggregation --prefill grpc://<host:port> ... --decode grpc://<host:port>`

SMG detects each HTTP worker's engine itself (/v1/models owned_by, then /version and /server_info) and reads the served model name from the worker (detect_backend.rs, discover_metadata.rs), so srtctl passes no --backend and no model name. SMG routes a request only to workers serving the model it names (router.rs), so clients send the workers' served model name, as srtctl's benchmarks do.

A TensorRT-LLM worker is a plain trtllm-serve OpenAI server on its allocated port. Its /v1/models reports owned_by: "tensorrt_llm", which SMG does not match, and it serves /version, so SMG records it as vllm (openai_protocol.py, openai_server.py, detect_backend.rs, fallback). For an aggregated worker that label only selects the metadata reader, which takes the served model name from /v1/models (discover_metadata.rs); the HTTP router forwards the OpenAI request unchanged. --backend trtllm would not change this: over HTTP and gRPC SMG keeps auto-detection and uses --backend only to pick the routing mode (main.rs). SMG's TRT-LLM runtime is its gRPC client for trtllm-serve --grpc, which srtctl does not launch for this frontend.

SGLang P/D runs over HTTP. SMG's HTTP P/D router hands the KV cache over through SGLang's bootstrap rendezvous (bootstrap_host / bootstrap_port / bootstrap_room, pd_router.rs). A vLLM worker registered by URL carries no KV connector, so SMG sends the request through and the decode worker recomputes the prompt (upstream test note). srtctl rejects that layout.

vLLM P/D therefore runs over gRPC. With grpc: true in both roles' args each worker is vllm serve --grpc, vLLM's gRPC server, which the smg-grpc-servicer package provides (vLLM's grpc extra), and srtctl advertises it as grpc://. The servicer reports the worker's kv_transfer_config connector in GetServerInfo (servicer.py), so for NixlConnector SMG tags the prefill leg with do_remote_decode and relays the kv_transfer_params the prefill returns to the decode leg, which pulls the KV cache over NIXL (request_execution.rs, kv_transfer.rs, upstream test). engine.connector: nixl gives both roles NixlConnector and the per-worker NIXL side channel. In gRPC mode SMG tokenizes and applies the chat template itself; pass tool-call-parser / reasoning-parser in frontend.args when the model needs them. vLLM's gRPC server starts no HTTP listener, so these workers serve no Prometheus /metrics and srtctl does not scrape them; SMG's own metrics stay on its Prometheus port. See examples/vllm/smg-disagg-grpc.yaml.

TRT-LLM P/D is not available behind SMG: its gRPC P/D table has no TRT-LLM entry (pd_protocol.rs), and over HTTP SMG relays only vLLM-style kv_transfer_params, so srtctl rejects it. Use trtllm-serve disaggregated serving (frontend.type: trtllm_serve) or Dynamo for TRT-LLM P/D.

vLLM data parallel¶

A vLLM aggregated role with data-parallel-size (DP, optionally with enable-expert-parallel) is one vllm serve that load-balances its DP ranks itself. Across nodes, the leader node serves the API and the other nodes run headless engine ranks (--nnodes/--node-rank), so SMG receives one worker URL per DP deployment. This is the layout frontend.type: vllm uses, with the API on an allocated port.

SMG's --dp-aware rank pinning is not used: it discovers dp_size over HTTP only from SGLang workers and registers any other HTTP worker as a plain worker (discover_dp.rs); for vLLM it needs gRPC workers (discover_dp.rs), which srtctl does not configure. See examples/vllm/smg-dep16.yaml.

Configuration¶

frontend:
  type: smg
  container_image: smg # an image that ships SMG, e.g. lightseekorg/smg:1.11.0
  enable_multiple_frontends: false
  args:
    policy: cache_aware # any `smg launch` flag

container_image: an engine image need not ship SMG. Point this at an image with the smg command (the official lightseekorg/smg image, or any image where pip install smg was run). Without it the model container is reused.
setup_script: the recipe's setup script also runs in SMG's container before smg launch, so a script that runs pip install smg==1.11.0 (a manylinux wheel) makes the model image serve as the router image.
args: passed to smg launch as --<key> <value>; true adds a bare flag and a list repeats the flag. prometheus-port is managed by srtctl and rejected here.
enable_multiple_frontends / num_additional_frontends: as for every router, several SMG replicas behind nginx, or one SMG on the public port.

See examples/vllm/smg-agg.yaml (vLLM, aggregated), examples/vllm/smg-dep16.yaml (vLLM, two-node DP16 with EP), examples/trtllm/smg-agg.yaml (TRT-LLM, aggregated) and examples/sglang/smg-disagg.yaml (SGLang, P/D).

Launch and readiness¶

Command. smg launch (subcommand) with --host 0.0.0.0 --port <frontend port> and the workers from the allocated topology: --worker-urls for aggregated workers; --pd-disaggregation, one --prefill URL BOOTSTRAP_PORT per prefill worker and one --decode URL per decode worker for P/D (main.rs, CLI arguments, P/D arguments). The Python package (what pip install smg and the official image install) provides the same smg launch subcommand and flags (cli.py, router_args.py).
Worker probe before start. SMG waits at most --worker-startup-timeout-secs (1800 s by default, main.rs) for a startup worker to register, and a large model can load for longer, so srtctl waits for every advertised worker's /health to answer 200 (a gRPC worker: its port to accept connections) before it starts SMG.
Readiness. srtctl polls SMG's GET /workers and waits until stats.prefill_count, stats.decode_count and stats.regular_count cover the expected workers; aggregated workers count as regular (route, response).
Metrics. SMG serves Prometheus on its own listener. srtctl passes --prometheus-port 29000 (on every interface, SMG's default host) and points tachometer's frontend target there (main.rs). Workers are scraped on their own HTTP ports, as with the other static routers, at the engine's Prometheus route (prometheus_metrics_path): /metrics, or /prometheus/metrics for trtllm-serve, whose /metrics is JSON iteration stats. trtllm-serve mounts that route only with return_perf_metrics: true (openai_server.py), which srtctl sets by default for every TRT-LLM role behind a direct-worker frontend. An SGLang grpc-mode: true worker serves /metrics and the profiler routes on an HTTP sidecar instead; srtctl allocates its port, passes it as --grpc-http-sidecar-port (SGLang v0.5.11 or newer) and scrapes it there (grpc_server.py).
Logs. Each replica writes <node>_smg_<index>.out in the job log directory and runs as the Slurm step smg_<index>.

Not wired¶

srtctl advertises a worker over HTTP unless its backend reports gRPC for the role (Backend.is_grpc_mode: SGLang grpc-mode: true, vLLM grpc: true), and then as grpc://. Before SMG starts, a gRPC worker is probed by connecting to its port (vLLM's gRPC server binds it only once the engine is up), an HTTP worker on /health. SMG's ZMQ and encode (EPD) worker modes, its Kubernetes service discovery, mesh, and the cloud-provider, history, MCP and WASM features are not configured by srtctl; frontend.args can still pass any smg launch flag that needs no srtctl-managed process or port.