Guided Decoding#
Overview#
Guided decoding constrains generation so the output is guaranteed to match a schema, pattern, or grammar. At every decoding step the runtime asks a grammar engine (XGrammar) which tokens are still legal and drives the logits of every other token to a large negative value before sampling. Illegal tokens therefore cannot be sampled at all — this is a hard constraint, not a prompt-level suggestion or a retry loop.
Key Points:
Six modes:
json_object,json_schema,regex,ebnf,structural_tag,choiceTwo request surfaces: the low-level
guided_decodingfield, and the OpenAI-compatibleresponse_formaton the serverPer-request and per-slot: constrained and unconstrained requests can share a batch
Optional per-request jump-forward appends grammar-forced tokens without forward passes
Compiled grammars are cached, so repeating a schema across requests costs one lookup
Build#
Guided decoding is built into every runtime; there is nothing to enable. The XGrammar backend is a Git submodule that CMake checks out on demand, so a normal build needs no extra steps. If your environment has no network access at configure time, check it out yourself first:
git submodule update --init --depth 1 --checkout 3rdParty/xgrammar
git -C 3rdParty/xgrammar submodule update --init --depth 1 --checkout 3rdparty/dlpack
Usage#
Request file#
Set exactly one mode under guided_decoding. It can be set per request, or at the top level of the file as a default for every request.
json_schema — the most common mode. Output is a JSON document matching the schema:
{
"requests": [
{
"messages": [{"role": "user", "content": "Give me a person record."}],
"guided_decoding": {
"json_schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"age": {"type": "integer"}
},
"required": ["name", "age"]
}
}
}
]
}
The schema may be given as an inline object (above) or as an escaped JSON string.
json_object — any JSON object, no schema:
{"guided_decoding": {"json_object": true}}
regex — the whole output matches the pattern:
{"guided_decoding": {"regex": "[a-z]+@[a-z]+\\.(com|org)"}}
ebnf — a GBNF-style grammar. It must define a rule named root:
{"guided_decoding": {"ebnf": "root ::= \"yes\" | \"no\""}}
choice — output is exactly one of the listed strings. A convenience form of ebnf; the alternatives are escaped for you:
{"guided_decoding": {"choice": ["yes", "no", "maybe"]}}
structural_tag — free text until a trigger appears, then the tagged region is constrained. Used for tool calling, where only the arguments need to be well-formed:
{
"guided_decoding": {
"structural_tag": {
"type": "structural_tag",
"format": {
"type": "triggered_tags",
"triggers": ["<function="],
"tags": [
{
"begin": "<function=get_weather>",
"content": {
"type": "json_schema",
"json_schema": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
},
"end": "</function>"
}
]
}
}
}
}
OpenAI-compatible server#
The server accepts both surfaces. Use response_format for portability with OpenAI clients:
{
"model": "my-model",
"messages": [{"role": "user", "content": "Give me a person record."}],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "person",
"schema": {
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"]
}
}
}
}
response_format covers only text (unconstrained), json_object, and json_schema — that is the whole of the OpenAI specification. For regex, ebnf, structural_tag, and choice, use the low-level field, which takes the same shape as in a request file:
{
"model": "my-model",
"messages": [{"role": "user", "content": "Answer yes or no."}],
"guided_decoding": {"choice": ["yes", "no"]}
}
Setting both surfaces on one request is an error rather than one silently winning. Streaming works normally.
Jump-forward#
When the grammar allows exactly one continuation, such as the {"name": " that opens a schema-constrained object, the runtime appends it without a forward pass and brings the KV cache up to date in one multi-token pass. Set jump_forward inside guided_decoding; it is off by default and works with every mode.
Request file:
{
"requests": [
{
"messages": [{"role": "user", "content": "Give me a person record."}],
"guided_decoding": {
"json_schema": {
"type": "object",
"properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
"required": ["name", "age"]
},
"jump_forward": true
}
}
]
}
llm_inference --jump-forward and llm_stream --jump-forward set the flag on every request in the file that has guided_decoding.
OpenAI-compatible server:
{
"model": "my-model",
"messages": [{"role": "user", "content": "Answer yes or no."}],
"guided_decoding": {"choice": ["yes", "no"], "jump_forward": true}
}
response_format has no equivalent flag; use guided_decoding.
How it runs#
Vanilla decoding appends the forced tokens and catches up on them in one multi-token step. A speculative decoder grafts them into its draft before the verify round, so they are verified together with the draft; a request that falls back to vanilla decoding (disable_spec_decode, or non-greedy sampling the decoder hands to vanilla decoding) still jumps through the catch-up step.
Build requirements#
Drafting |
Requirement |
|---|---|
Tree: EAGLE3; MTP, DFlash, DFlash2, JetSpec, Domino, DSpark with |
Base built with |
Chain: MTP, Gemma4 MTP, DFlash, DFlash2, Domino, DSpark with |
No verify headroom: grafted tokens replace draft tokens. An MTP draft needs |
When the flag is ignored#
The request runs with its guide but without jumps, and the runtime logs one warning per reason:
non-greedy sampling on a speculative decoder that keeps such requests (DSpark and DFlash2 chain drafting);
a speculative decoder under in-flight batching (the OpenAI-compatible server and
RuntimeSession); the blocking request path grafts;a deployment that misses a build requirement above;
block-diffusion engines, vision-block attention, and engines whose context chunk holds a single token.
Effects#
A request with the flag reserves KV cache for the widest grafting round, so its maximum generation length can be up to
max_spec_verify_tokens - verify tokensshorter; requests without the flag keep their KV budget. A speculative deployment that can jump sizes its verify buffers, grammar bitmask rows and CUDA graphs for the widest round at load time, whether or not any request sets the flag.Forced tokens get a logprob of 0 and are their only top-k candidate. A draft token the forced text would respell is left alone (
respell_refusedin the statistics), and a forced run shorter than the draft it would cut is not grafted (draft_longer).Free text after a jump, and the output of other requests in the same batch, can differ from an unconstrained run.
Schema Support#
XGrammar enforces most of JSON Schema, but a few keywords are accepted and then ignored. Since silently returning output that violates the schema is worse than refusing the request, these are rejected up front with a message naming the keyword:
multipleOf, uniqueItems, contains, minContains, maxContains, patternProperties
format is enforced only for the values XGrammar implements: date, date-time, duration, email, hostname, ipv4, ipv6, json-pointer, relative-json-pointer, time, uri, uri-reference, uri-template, uuid. Any other format value is rejected for the same reason.
The check only inspects schema keywords, so a property genuinely named contains is fine.
Speculative decoding#
Every speculative decoding mode is supported, with no request-side configuration. Jump-forward runs on greedy requests in every geometry below; its build requirements are listed there.
Mode |
Geometry |
|---|---|
MTP |
chain and tree |
EAGLE3 |
tree |
DFlash |
linear and DDTree |
DFlash2 |
chain, greedy or sampled; DDTree, greedy |
JetSpec |
DDTree |
Domino |
linear and DDTree |
DSpark |
chain and tree, greedy or sampled |
Gemma4 MTP |
chain |
A speculative step verifies several candidate tokens at once, and each verification row sits at a different point in the grammar, so the mask is built per row rather than per request: the decoder hands the shape of its draft tree to the guided decoder, which walks the grammar down the tree and masks every row from the state its own path reaches. A candidate the grammar refuses takes its subtree with it, leaving those rows unmasked; they are unreachable anyway, because acceptance follows a root-to-node path.
Constrained output stays valid whether or not drafting is on, but it is not always identical. Tree geometries verify each node under an attention mask covering only its own path, which differs numerically from the vanilla forward, so a near-tie can resolve differently and long unconstrained stretches may diverge. Chain geometries do not have this effect.
Limitations#
Not supported on block-diffusion engines, which denoise a whole canvas per step instead of appending one token at a time. Such requests are rejected rather than silently left unconstrained.
Thinking models: the constraint starts at the token immediately after the reasoning-end marker (
</think>or<channel|>), including when speculative decoding commits the marker and the tokens following it in the same step. The reasoning block itself is unconstrained. A request whose reasoning never ends is therefore never constrained.Whether a request starts inside a reasoning block is read from the rendered prompt. The most recent reasoning marker is authoritative. If the prompt has no marker,
enable_thinking=falsestarts the constraint immediately, whileenable_thinking=trueleaves room for the model to open a reasoning block. Models without reasoning-marker tokens are constrained immediately.Regex dialect: XGrammar’s regex, not PCRE. No lookaround and no backreferences.
EBNF entry rule must be named
root.Guide size is capped at 128 KB. Compilation is superlinear in guide size and runs before any GPU work.
structural_tagaccepts only the moderntriggered_tagsform, not the legacy{structures, triggers}shape.
Notes#
A guide that fails to compile fails only its own request; the rest of the batch continues.
If a grammar reaches a state where no token in the model’s vocabulary is legal, the request ends with
finish_reason: errorand keeps the tokens generated so far, which do not satisfy the guide. When streaming, those tokens have already been sent, so checkfinish_reasonbefore using the result. This normally means the guide cannot be expressed in this tokenizer’s vocabulary.Constrained requests are slightly slower per step (one mask fill plus one elementwise kernel), but usually finish sooner, because the model cannot wander outside the grammar.