TokenSpeed¶
Use engine: tokenspeed with frontend.type: dynamo or frontend.type: smg.
- Dynamo. Every worker runs Dynamo's TokenSpeed backend,
python3 -m dynamo.tokenspeed, which registers with the Dynamo frontend like the other Dynamo engines. - SMG. Every worker runs
python3 -m tokenspeed.cli serve --headless. SMG binds a TCP ZMQ handshake for each logical worker on the frontend node; the engine ranks connect to it. The gateway tokenizes and serves the OpenAI API. This supports aggregated and prefill/decode serving, including multi-node workers.
Recipes:
- Dynamo: aggregated, prefill/decode
- SMG: aggregated, prefill/decode
Configuration loading rejects the engine-specific routers (sglang-router,
vllm-router, ...), dynamo.sidecar, and roles.<role>.kv_events.
The Dynamo integration follows
TokenSpeed,
Dynamo's
dynamo.tokenspeed
with the matching TokenSpeed version in its image. The SMG integration requires
TokenSpeed's native headless ZMQ serving and its matching tokenspeed-smg dependency.
Image¶
Behind Dynamo, the worker image must contain TokenSpeed and an ai-dynamo build with
dynamo.tokenspeed. Aggregated serving needs Dynamo 1.2.0 or newer; prefill/decode
needs a build that includes ai-dynamo/dynamo#9237,
which Dynamo 1.5.0 does not; the PyPI nightlies from 1.6.0.dev20260922 on do. Either
let srtctl install one at job start (dynamo.source.pypi: "1.6.0.dev20260922") into an
image that ships TokenSpeed, or build an image with Dynamo's
TokenSpeed Dockerfile
from a TokenSpeed runner and the Dynamo checkout. The examples set
dynamo.install: false because their image already ships Dynamo.
Behind SMG, install the published TokenSpeed daily wheel and its dependencies,
including the matching kernel wheel for the image's CUDA version and architecture.
TokenSpeed pins tokenspeed-smg, which provides the smg command. The same image
runs the workers and SMG unless frontend.container_image selects another image
with that matching gateway build. No gRPC servicer is launched.
Arguments¶
roles.<role>.args are TokenSpeed
server flags,
such as tensor-parallel-size, data-parallel-size, enable-expert-parallel,
max-model-len and gpu-memory-utilization. Set the parallelism to match the
role's gpus. A role's served-model-name is the name every worker serves; without
one, workers serve the model directory name.
srtctl sets these flags on every worker, under either frontend:
| Flag | Value |
|---|---|
--model |
/model, the staged model path, or the Hugging Face ID |
--served-model-name |
the role's served-model-name, else the model directory name |
--host |
the worker's own IP on network_interface |
--port |
an allocated engine port-scan base, 10000 and up, 1024 apart on a node |
--dist-init-addr |
the endpoint leader's IP and an allocated port |
--nnodes, --node-rank |
the endpoint's node count and this node's rank |
--disaggregation-mode |
prefill or decode, on prefill/decode workers |
--disaggregation-bootstrap-port |
an allocated port, on prefill workers |
TokenSpeed derives several listeners from two of these. Its
PortArgs.init_new
picks the NCCL port 100 to 1000 above --port, and binds a control-plane cluster
on the leader that starts at the --dist-init-addr port: six ports, plus one per
attention-DP rank
(data_parallel_controller.py).
srtctl reserves a block of six ports plus the endpoint's GPU count for that cluster,
and spaces --port 1024 apart, so workers sharing a node do not collide. Under the Dynamo integration, a
follower node serves its health check on --host and --port
(engine.py).
Each node runs its own single-node srun step, so srtctl passes --nnodes,
--node-rank and --dist-init-addr instead of relying on TokenSpeed's
Slurm step detection.
Parallelism¶
Parallelism is in the role's args and needs nothing else from srtctl. Attention DP with
expert parallelism (DEP) is data-parallel-size: <gpus> with enable-expert-parallel: true;
tensor parallelism with expert parallelism (TEP) is tensor-parallel-size: <gpus> with
enable-expert-parallel: true. A worker that spans nodes sizes these over all of its GPUs:
srtctl passes its node count and rank, and TokenSpeed runs world size / nnodes ranks
on each node
(parallelism.md).
Behind Dynamo, prefill and decode workers need attention DP 1 (see below); behind SMG a
decode worker can be DEP.
KV cache offload¶
Unless a role sets disable-kvstore, every TokenSpeed rank keeps evicted KV blocks in a
pinned host-memory KVStore (L2). Each rank allocates its own: kvstore-size sets it in GB,
otherwise it is kvstore-ratio (default 2.0) times the rank's GPU KV pool, so size it to
the node's memory.
kvstore-storage-backend: mooncake adds Mooncake Store under it as L3
(server.md).
Declare a mooncake-master service: srtctl starts the master on the infra node and sets
MOONCAKE_MASTER, MOONCAKE_TE_META_DATA_SERVER and MOONCAKE_LOCAL_HOSTNAME on every
worker, which TokenSpeed's Mooncake client reads
(mooncake.py).
Set MOONCAKE_GLOBAL_SEGMENT_SIZE, MOONCAKE_PROTOCOL or MOONCAKE_DEVICE in the role's
env. A TokenSpeed rank stops when Mooncake rejects a write for lack of space, so size the
segments for the workload and let the master evict before it fills (the service's args,
for example --eviction_high_watermark_ratio=0.7). The service runs mooncake_master from
TokenSpeed's tokenspeed-mooncake dependency in the model image; give it a preamble that
installs TokenSpeed when the image gets it from setup_script, which services do not run.
services:
- name: mooncake-master
type: mooncake-master
roles:
agg:
args:
kvstore-size: 64
kvstore-storage-backend: mooncake
Prefill/decode¶
TokenSpeed moves the KV cache with Mooncake. A prefill worker advertises its
bootstrap server at --host and the bootstrap port
(runtime_disaggregated_endpoint);
the Dynamo frontend passes it to the decode worker with each request. Prefill
workers register as the prefill component and decode workers as backend
(args.py).
Dynamo checks the remaining requirements when the worker starts
(validate_disagg_compatibility):
the transfer backend is Mooncake, attention DP is 1, and prefix-granularity is
positive (TokenSpeed's default is 64). Scale prefill or decode with more workers.
Set disaggregation-ib-device in the role's args when Mooncake does not pick the
right RDMA device.
SMG¶
Set frontend.enable_multiple_frontends: false and leave frontend placement on
head. A ZMQ engine connects to one gateway, so multiple frontend replicas and
non-head frontend placement are rejected at configuration load.
For each endpoint, srtctl allocates one gateway handshake port (31000 and up,
unique across the job). SMG receives tcp://<head-ip>:<handshake-port> in
--worker-urls, or --prefill / --decode for disaggregation, together with
--backend tokenspeed. Its input and output sockets use OS-assigned TCP ports on
the same frontend host, returned to the engines during the handshake.
Every node of the endpoint runs ts serve --headless with:
--data-parallel-address <head-ip>--data-parallel-rpc-port <handshake-port>--zmq-engine-index 0
Attention-DP ranks register their own identities and declare their group size; SMG learns that size from the handshake. Different endpoints may have different DP sizes. Each endpoint retains its own distributed rendezvous and each node its own engine port-scan base. See TokenSpeed's headless entrypoint and gateway socket contract.
SMG must bind the handshake before an engine can become ready. srtctl therefore
does not probe ZMQ URLs as HTTP or gRPC worker listeners before starting SMG.
The gateway's /workers readiness gate waits for the expected healthy workers
before the benchmark. Pending registrations are excluded even when SMG's
stats already includes them.
For prefill/decode, the engine advertises its own KV bootstrap host and allocated
bootstrap port in its handshake metadata. No bootstrap port is appended to the
router's --prefill tcp://... URL: that URL names the gateway, not the prefill
node. The engine-side --host remains the worker's own network-interface IP.
SMG receives the model path/ID and a --model-alias for a differing
served-model-name. A role's tokenizer supplies --tokenizer-path unless the
frontend sets one explicitly. The gateway uses the runtime-resolved shared
model mount, including /srtctl-model-cache/snapshots/<revision> for a local
Hugging Face cache snapshot, even when weights are staged on the worker nodes.
An explicit frontend.args.model takes precedence. Set model-specific
tool-call-parser, reasoning-parser, and other gateway options in
frontend.args.
For structured-output benchmarks, set grammar-backend: xgrammar in the worker
role's args when the checkpoint does not enable it by default. TokenSpeed's
grammar-backend: none rejects requests using json_schema, regex, ebnf or
structural_tag; the default depends on the checkpoint
(upstream arguments).
With roles.<role>.args.enable-metrics: true and the Prometheus reporter,
every worker node gets a separate allocated --metrics-port (6800 and up on each
node). metrics-host binds all interfaces unless explicitly supplied. Tachometer
and the benchmark's AIPERF_SERVER_METRICS_URLS scrape these HTTP /metrics
listeners, including multi-node followers. Disabled metrics and non-Prometheus
reporters advertise no engine metrics URL. SMG's own metrics remain on its
separate Prometheus listener. The headless engine exposes no HTTP profiling
control API.
The engine's handshake address, engine index and metrics port are managed by srtctl; conflicting role arguments are rejected rather than silently overwritten.