Skip to content

vLLM Router

frontend.type: vllm-router runs the official vLLM Router in front of direct vllm serve workers. srtctl supplies the statically allocated worker URLs; vLLM remains responsible for the DP/TP/PP engine topology inside each worker.

Responsibilities

  • The existing vLLM backend allocates endpoints and derives per_node DP, cross-node TP/PP rendezvous, device IDs, and headless followers.
  • The frontend adapter starts Router, supplies aggregate or P/D worker URLs, adds NIXL bootstrap ports for P/D, and derives Router's node-local DP expansion factor. With the Mooncake connector it starts Router only after every worker is healthy.
  • Readiness first validates Router's /workers counts, then requires HTTP 200 from every exact base /health URL advertised to Router. This prevents benchmark and eval traffic from racing a partially started DP pool.
  • Router stdout/stderr is captured in the normal job log directory alongside backend and benchmark logs. frontend.container_image can select a separate Router image; otherwise the model container is reused.

DP recipes must use upstream's default engine.dp_launch_mode: per_node. The deprecated per_gpu mode launches Dynamo registrations rather than independently routable API servers and is rejected for vLLM Router DP.

Aggregate cases

One direct engine behind Router

This is the smallest smoke topology. Router is functionally optional, but it validates the adapter and provides the same public interface as larger cases.

frontend:
  type: vllm-router
  enable_multiple_frontends: false
  args:
    policy: consistent_hash

resources:
  gpus_per_node: 8

engine: vllm
roles:
  agg:
    nodes: 1
    workers: 1
    args:
      tensor-parallel-size: 8

Multiple aggregate replicas

Each logical aggregate worker contributes one or more routable base URLs. Router receives them through --worker-urls and distributes sessions between the expanded ranks.

resources:
  gpus_per_node: 8

roles:
  agg:
    nodes: 4
    workers: 4

Node-local data parallelism

For DEP8 on two four-GPU nodes, upstream srt-slurm creates one hybrid-LB vllm serve process on each node. Router receives both base URLs as two node-local pools. srtctl does not apply --intra-node-data-parallel-size to this multi-node endpoint because the second pool owns global ranks 4 through 7; vLLM's node-local hybrid load balancer selects among those ranks.

resources:
  gpus_per_node: 4

engine: vllm
roles:
  agg:
    nodes: 2
    workers: 1
    args:
      data-parallel-size: 8
      enable-expert-parallel: true

Model parallelism spanning nodes

Upstream's native topology is preserved. If one TP/PP replica spans nodes, only its global API leader has a positive HTTP port and is advertised to Router; the remaining processes stay headless.

resources:
  gpus_per_node: 4

engine: vllm
roles:
  agg:
    nodes: 2
    workers: 1
    args:
      tensor-parallel-size: 8

Disaggregated P/D

Both pools are direct vLLM servers. Router is launched with --vllm-pd-disaggregation, repeated --prefill URL NIXL_PORT entries, and repeated --decode URL entries. The same aggregate topology rules apply independently to every prefill and decode endpoint.

frontend:
  type: vllm-router
  enable_multiple_frontends: false
  args:
    policy: consistent_hash

resources:
  gpus_per_node: 4

engine:
  type: vllm
  connector: nixl
roles:
  prefill:
    nodes: 1
    workers: 1
    args:
      data-parallel-size: 4
  decode:
    nodes: 2
    workers: 2
    args:
      data-parallel-size: 4

Router has one global --intra-node-data-parallel-size, so srtctl uses it only when every logical endpoint has one advertised base URL. Multi-node hybrid-LB endpoints remain unexpanded node-local pools because each later pool starts at a nonzero global DP rank. Do not set the option manually unless it exactly matches the allocated topology.

Mooncake

engine.connector: mooncake runs the P/D pair over vLLM's MooncakeConnector. Its prefill returns no kv_transfer_params; the Router supplies them instead. It reads each prefill's engine IDs from that prefill's Mooncake bootstrap server and adds the bootstrap address and engine ID to every decode request.

frontend:
  type: vllm-router
  enable_multiple_frontends: false
  args:
    policy: consistent_hash

engine:
  type: vllm
  connector: mooncake
Piece srtctl sets Why
Router command --kv-connector mooncake, and --prefill URL PORT for each prefill Mooncake mode in vllm-project/router (CliArgs.kv_connector in src/main.rs); without a PORT the Router queries 8998
kv_connector / kv_role MooncakeConnector; kv_producer on prefill, kv_consumer on decode The connector's role follows the worker's mode
VLLM_MOONCAKE_BOOTSTRAP_PORT a separate allocator-owned bootstrap port, shared by every node of a prefill engine and advertised to the Router vLLM's TP0/PP0 rank opens the server; follower ranks register at the same leader address and port. Colocated prefill engines get distinct ports. Decode receives the address in request metadata and needs no bootstrap environment variable
Router start after every advertised worker answers /health The Router queries each bootstrap server once, while it starts, and keeps the answer. It exits after about 4.5 minutes of retries, and an answer from before the prefill's workers registered (in register_kv_caches) has no engine IDs, so decode then recomputes every prompt

Both prefill and decode must run the connector, and frontend.args.kv-connector is either unset or exactly mooncake (case-sensitive). A role's own kv-transfer-config replaces the preset. The effective config selects the Router mode and bootstrap port, including a raw MooncakeConnector config without the shorthand or a MultiConnector wrapping Mooncake with SimpleCPUOffloadConnector. Replacing Mooncake on only one role is rejected; replacing it on both removes Mooncake routing. The connector's per-worker ZMQ side channel binds a port the kernel picks.

Mooncake requires frontend.type: vllm-router, a prefill/decode topology, and enabled health checks (health_check: null is rejected). Prefill must use data-parallel-size: 1; multiple independent prefill workers and cross-node TP/PP are supported, but prefill DP pools are not yet supported. Prefill restart must be never: the Router caches engine IDs at startup and cannot refresh them after a worker relaunch. These rules fail at config load, before a job is submitted.

Upstream behavior: vllm-project/router db02f358 (v0.1.16) src/routers/http/vllm_pd_router.rs, vllm-project/vllm 96710245 vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py. examples/vllm/vllm-router-mooncake-disagg.yaml is the reference recipe.

MoRI-IO discovery

engine.connector: moriio runs the P/D pair over AMD's MoRI-IO connector (ROCm). MoRI-IO workers do not take peer addresses on the command line; they register with the Router over a ZMQ discovery endpoint and learn each other's transfer addresses from it. srtctl therefore launches the Router in discovery mode and builds each worker's connector config from the realized topology instead of listing URLs:

frontend:
  type: vllm-router
  enable_multiple_frontends: false

engine:
  type: vllm
  connector: moriio
roles:
  prefill:
    nodes: 1
    workers: 1
    gpus: 1
  decode:
    nodes: colocate
    workers: 1
    gpus: 1
Piece srtctl sets Why
Router command --vllm-pd-disaggregation --kv-connector moriio --vllm-discovery-address 0.0.0.0:36367, no --prefill/--decode Discovery mode in vllm-project/router (RouterArgs.vllm_discovery_address); the address is the one listener workers register with
kv_connector / kv_role MoRIIOConnector; kv_producer on prefill, kv_consumer on decode The connector's role follows the worker's mode
proxy_ip, proxy_ping_port the head node's IP on the cluster interface, 36367 Where the worker registers; hence one Router, on the head node
http_port, host_ip the worker's allocated HTTP port and its own routable IP What the Router routes to and what peers connect to; upstream falls back to its own interface guess for host_ip, which is wrong on multi-homed nodes
handshake_port, notify_port allocated per process by NodePortAllocator (moriio_handshake, moriio_notify kinds); both reserve one port per local rank vLLM adds rank offsets to both bases, so colocated workers need disjoint blocks. Defaults start at 26000/27000, below Linux's default ephemeral range, to avoid MoRI's other dynamically allocated listeners
read_mode true unless the recipe's template sets it The Router's discovery flow serves both modes: true has decode pull the KV (RDMA READ), false has prefill push it into blocks decode allocated first (RDMA WRITE, vllm-router 0.1.15 or later). A non-boolean, or prefill and decode disagreeing, fails when srtctl loads the config
VLLM_PORT unset for these workers MoRI-IO opens its other listeners inside the TP child processes, which inherit VLLM_PORT; a fixed scan base there hands several ranks the same unbound port, so vLLM takes ephemeral ports from the kernel

Readiness is the Router's /health: it answers 503 Waiting for discovered workers until a prefill and a decode have registered, then 200 (vllm-project/router 43140bc8e2). This is a workaround for a gap in the Router, not a MoRI-IO property: in discovery mode the Router keeps registered workers in a separate registry but still serves /workers and /get_server_info from the static URL list, which is empty, so the count-based /workers gate srtctl uses for static routing has nothing to count. /health is the only endpoint that reads the discovery registry, and it only proves one worker of each role, so with several workers per role the benchmark can start before the Router knows all of them; the per-worker /health gate that follows proves the workers are up, not that they registered. The upstream fix is for /workers to merge the discovery registry (src/routers/router_manager.rs); when it does, VLLMRouterFrontend.probe_ready loses its discovery branch and the count-based probe covers both modes.

Rules the recipe must satisfy: both prefill and decode run the connector (an engine-wide connector: moriio, or the same value in both roles' args), the topology is prefill/decode, frontend.enable_multiple_frontends is false, and frontend.orchestrator_placement is head. srtctl dry-run rejects anything else, and any other frontend.type with this connector. Upstream key names are those moriio_common.py reads (vllm-project/vllm 9679173788). examples/vllm/vllm-router-moriio-disagg.yaml is the reference recipe.

Discovery templates and CPU offload

Keep engine.connector: moriio to select discovery and allocate listeners. roles.<role>.args.kv-transfer-config (or kv_transfer_config) may supply a JSON object or JSON string containing exactly one MoRIIOConnector, directly or inside a connector chain. Supplying both argument spellings is rejected. The renderer preserves failure policies, transport options and sibling connectors, and fills the discovery child's role and topology fields. Explicit roles or binding values that conflict with the allocated topology are rejected when the worker command is built. Matching port values may be integers or decimal strings; rendered ports use the allocator's string values. Do not hard-code worker addresses or listener ports in a reusable recipe. read_mode is not a binding: it defaults to true (RDMA READ), and a template may set false (RDMA WRITE, which needs vllm-router 0.1.15 or later). Set the same value in both roles; srtctl rejects a mismatch when it loads the config. This default is the opposite of vLLM's, which runs WRITE when the key is missing, and srtctl takes only a JSON boolean where vLLM also parses strings.

For example, a prefill template can combine RDMA READ with CPU offload:

engine:
  type: vllm
  connector: moriio
roles:
  prefill:
    args:
      kv-transfer-config:
        kv_connector: MultiConnector
        kv_role: kv_both
        kv_load_failure_policy: fail
        kv_connector_extra_config:
          connectors:
            - kv_connector: MoRIIOConnector
              kv_role: kv_producer
              kv_load_failure_policy: fail
              kv_connector_extra_config:
                backend: rdma
                qp_per_transfer: 8
            - kv_connector: SimpleCPUOffloadConnector
              kv_role: kv_both
              kv_connector_extra_config:
                cpu_bytes_to_use: 1073741824
                lazy_offload: false

The pinned vLLM image must support the supplied connectors and their fields. Apply this template to the prefill role of examples/vllm/vllm-router-moriio-disagg.yaml, the existing colocated discovery recipe. As with all discovery recipes, listener blocks remain allocator-owned.

Multiple Router processes

With enable_multiple_frontends: true, srtctl starts nginx on the public port and multiple identical Router processes on the internal frontend port. Each Router receives the same backend topology. Use nginx session affinity when a client session must remain on one Router process.