Skip to content

TokenSpeed

Use engine: tokenspeed with frontend.type: dynamo or frontend.type: smg.

  • Dynamo. Every worker runs Dynamo's TokenSpeed backend, python3 -m dynamo.tokenspeed, which registers with the Dynamo frontend like the other Dynamo engines.
  • SMG. Every worker runs python3 -m tokenspeed.cli serve --headless. SMG binds a TCP ZMQ handshake for each logical worker on the frontend node; the engine ranks connect to it. The gateway tokenizes and serves the OpenAI API. This supports aggregated and prefill/decode serving, including multi-node workers.

Recipes:

Configuration loading rejects the engine-specific routers (sglang-router, vllm-router, ...), dynamo.sidecar, and roles.<role>.kv_events.

The Dynamo integration follows TokenSpeed, Dynamo's dynamo.tokenspeed with the matching TokenSpeed version in its image. The SMG integration requires TokenSpeed's native headless ZMQ serving and its matching tokenspeed-smg dependency.

Image

Behind Dynamo, the worker image must contain TokenSpeed and an ai-dynamo build with dynamo.tokenspeed. Aggregated serving needs Dynamo 1.2.0 or newer; prefill/decode needs a build that includes ai-dynamo/dynamo#9237, which Dynamo 1.5.0 does not; the PyPI nightlies from 1.6.0.dev20260922 on do. Either let srtctl install one at job start (dynamo.source.pypi: "1.6.0.dev20260922") into an image that ships TokenSpeed, or build an image with Dynamo's TokenSpeed Dockerfile from a TokenSpeed runner and the Dynamo checkout. The examples set dynamo.install: false because their image already ships Dynamo.

Behind SMG, install the published TokenSpeed daily wheel and its dependencies, including the matching kernel wheel for the image's CUDA version and architecture. TokenSpeed pins tokenspeed-smg, which provides the smg command. The same image runs the workers and SMG unless frontend.container_image selects another image with that matching gateway build. No gRPC servicer is launched.

Arguments

roles.<role>.args are TokenSpeed server flags, such as tensor-parallel-size, data-parallel-size, enable-expert-parallel, max-model-len and gpu-memory-utilization. Set the parallelism to match the role's gpus. A role's served-model-name is the name every worker serves; without one, workers serve the model directory name.

srtctl sets these flags on every worker, under either frontend:

Flag Value
--model /model, the staged model path, or the Hugging Face ID
--served-model-name the role's served-model-name, else the model directory name
--host the worker's own IP on network_interface
--port an allocated engine port-scan base, 10000 and up, 1024 apart on a node
--dist-init-addr the endpoint leader's IP and an allocated port
--nnodes, --node-rank the endpoint's node count and this node's rank
--disaggregation-mode prefill or decode, on prefill/decode workers
--disaggregation-bootstrap-port an allocated port, on prefill workers

TokenSpeed derives several listeners from two of these. Its PortArgs.init_new picks the NCCL port 100 to 1000 above --port, and binds a control-plane cluster on the leader that starts at the --dist-init-addr port: six ports, plus one per attention-DP rank (data_parallel_controller.py). srtctl reserves a block of six ports plus the endpoint's GPU count for that cluster, and spaces --port 1024 apart, so workers sharing a node do not collide. Under the Dynamo integration, a follower node serves its health check on --host and --port (engine.py). Each node runs its own single-node srun step, so srtctl passes --nnodes, --node-rank and --dist-init-addr instead of relying on TokenSpeed's Slurm step detection.

Parallelism

Parallelism is in the role's args and needs nothing else from srtctl. Attention DP with expert parallelism (DEP) is data-parallel-size: <gpus> with enable-expert-parallel: true; tensor parallelism with expert parallelism (TEP) is tensor-parallel-size: <gpus> with enable-expert-parallel: true. A worker that spans nodes sizes these over all of its GPUs: srtctl passes its node count and rank, and TokenSpeed runs world size / nnodes ranks on each node (parallelism.md). Behind Dynamo, prefill and decode workers need attention DP 1 (see below); behind SMG a decode worker can be DEP.

KV cache offload

Unless a role sets disable-kvstore, every TokenSpeed rank keeps evicted KV blocks in a pinned host-memory KVStore (L2). Each rank allocates its own: kvstore-size sets it in GB, otherwise it is kvstore-ratio (default 2.0) times the rank's GPU KV pool, so size it to the node's memory.

kvstore-storage-backend: mooncake adds Mooncake Store under it as L3 (server.md). Declare a mooncake-master service: srtctl starts the master on the infra node and sets MOONCAKE_MASTER, MOONCAKE_TE_META_DATA_SERVER and MOONCAKE_LOCAL_HOSTNAME on every worker, which TokenSpeed's Mooncake client reads (mooncake.py). Set MOONCAKE_GLOBAL_SEGMENT_SIZE, MOONCAKE_PROTOCOL or MOONCAKE_DEVICE in the role's env. A TokenSpeed rank stops when Mooncake rejects a write for lack of space, so size the segments for the workload and let the master evict before it fills (the service's args, for example --eviction_high_watermark_ratio=0.7). The service runs mooncake_master from TokenSpeed's tokenspeed-mooncake dependency in the model image; give it a preamble that installs TokenSpeed when the image gets it from setup_script, which services do not run.

services:
  - name: mooncake-master
    type: mooncake-master
roles:
  agg:
    args:
      kvstore-size: 64
      kvstore-storage-backend: mooncake

Prefill/decode

TokenSpeed moves the KV cache with Mooncake. A prefill worker advertises its bootstrap server at --host and the bootstrap port (runtime_disaggregated_endpoint); the Dynamo frontend passes it to the decode worker with each request. Prefill workers register as the prefill component and decode workers as backend (args.py).

Dynamo checks the remaining requirements when the worker starts (validate_disagg_compatibility): the transfer backend is Mooncake, attention DP is 1, and prefix-granularity is positive (TokenSpeed's default is 64). Scale prefill or decode with more workers. Set disaggregation-ib-device in the role's args when Mooncake does not pick the right RDMA device.

SMG

Set frontend.enable_multiple_frontends: false and leave frontend placement on head. A ZMQ engine connects to one gateway, so multiple frontend replicas and non-head frontend placement are rejected at configuration load.

For each endpoint, srtctl allocates one gateway handshake port (31000 and up, unique across the job). SMG receives tcp://<head-ip>:<handshake-port> in --worker-urls, or --prefill / --decode for disaggregation, together with --backend tokenspeed. Its input and output sockets use OS-assigned TCP ports on the same frontend host, returned to the engines during the handshake. Every node of the endpoint runs ts serve --headless with:

  • --data-parallel-address <head-ip>
  • --data-parallel-rpc-port <handshake-port>
  • --zmq-engine-index 0

Attention-DP ranks register their own identities and declare their group size; SMG learns that size from the handshake. Different endpoints may have different DP sizes. Each endpoint retains its own distributed rendezvous and each node its own engine port-scan base. See TokenSpeed's headless entrypoint and gateway socket contract.

SMG must bind the handshake before an engine can become ready. srtctl therefore does not probe ZMQ URLs as HTTP or gRPC worker listeners before starting SMG. The gateway's /workers readiness gate waits for the expected healthy workers before the benchmark. Pending registrations are excluded even when SMG's stats already includes them.

For prefill/decode, the engine advertises its own KV bootstrap host and allocated bootstrap port in its handshake metadata. No bootstrap port is appended to the router's --prefill tcp://... URL: that URL names the gateway, not the prefill node. The engine-side --host remains the worker's own network-interface IP.

SMG receives the model path/ID and a --model-alias for a differing served-model-name. A role's tokenizer supplies --tokenizer-path unless the frontend sets one explicitly. The gateway uses the runtime-resolved shared model mount, including /srtctl-model-cache/snapshots/<revision> for a local Hugging Face cache snapshot, even when weights are staged on the worker nodes. An explicit frontend.args.model takes precedence. Set model-specific tool-call-parser, reasoning-parser, and other gateway options in frontend.args.

For structured-output benchmarks, set grammar-backend: xgrammar in the worker role's args when the checkpoint does not enable it by default. TokenSpeed's grammar-backend: none rejects requests using json_schema, regex, ebnf or structural_tag; the default depends on the checkpoint (upstream arguments).

With roles.<role>.args.enable-metrics: true and the Prometheus reporter, every worker node gets a separate allocated --metrics-port (6800 and up on each node). metrics-host binds all interfaces unless explicitly supplied. Tachometer and the benchmark's AIPERF_SERVER_METRICS_URLS scrape these HTTP /metrics listeners, including multi-node followers. Disabled metrics and non-Prometheus reporters advertise no engine metrics URL. SMG's own metrics remain on its separate Prometheus listener. The headless engine exposes no HTTP profiling control API.

The engine's handshake address, engine index and metrics port are managed by srtctl; conflicting role arguments are rejected rather than silently overwritten.