In-Flight Batching#
Scope#
In-flight batching (IFB) lets the RequestEngine (cpp/scheduler/) admit a queued
request into a session that is already generating, instead of waiting for the running
requests to finish. It is opt-in (--enable-in-flight-batching on the server) and early-stage: this page states exactly what the control plane serves, what it
refuses, and where the rest of the runtime’s feature set runs in the meantime. Remaining
gaps are listed below in implementation order.
The engine ships one control plane, over one runtime session that lives as long as
the engine (RuntimeSession in cpp/runtime/, opened once by
LLMInferenceRuntime::openSession). The actor owns the loop: at every step boundary it
folds in its command queue, admits queued requests into the session, then schedules one
step and commits the result. What a step may contain depends on the execution the
deployment supports (see Scheduling): under single-cohort execution the
actor admits at most one request per boundary and a step is either the context step of
the request it just admitted or one decode step over every resident; under compact
execution it admits every queue head that can join and a step combines prompt chunks of
several residents with decode rows. A request enters through the same
admission whether the session is empty or full; no request opens the session and no
teardown between requests, so an empty session simply idles until the next submit. A
request the session cannot serve is refused at submit() with std::invalid_argument
and a message naming the alternative – the same permanent-refusal semantics as
per-request LoRA. There is deliberately no fallback plane: a workaround that keeps a
feature “working” removes the pressure that gets its real implementation scheduled.
Scheduling#
Execution and strategy#
Phase 3 and phase 8 below are rt::ExecutionPhase values, kAutoregressiveDecode and
kMixedPrefillDecode: the extents of the execution-phase marker that tells the plugins which kind
of step runs (see the phase table in
TensorRT Plugins).
Term |
Meaning |
|---|---|
Single-cohort execution |
A step is one admission’s whole-prompt context step or one decode step over every resident (phase 3). Every deployment supports it. |
Compact execution |
A step is any subset of residents: prompt chunks first, then decode rows, packed without padding (phase 8, |
|
Decode first: every decoding resident gets one token per step, and prompt chunks fill the rest of the step budget. Needs compact execution. |
|
While any resident has an unfinished prompt the step runs only prompt work and the decodes wait; otherwise every resident decodes. On a compact deployment the waiting prompts of several residents share one phase-8 step; otherwise it is single-cohort execution. |
|
|
|
Single-cohort execution on any deployment. |
Whether a deployment supports compact execution is decided once per runtime, when the first
session that allows compact execution opens (LLMRankRuntime::resolveCompactCapability). Compact
execution first requires these static preconditions (LLMRankRuntime::compactPreconditionFailure):
not a speculative (draft engine, MTP, EAGLE, DFlash) or diffusion deployment;
not a TensorRT-RTX engine;
the engine’s
maxBatchSizeis at least 2;no Gemma4 vision-bidirectional attention (image tokens attend bidirectionally within an image, which a chunk boundary cannot split);
the vision and audio encoders’ outputs, if any, are FP16
[rows, hidden size];the vision encoder, if any, produces as many deepstack feature layers as the decoder engine takes (
num_deepstack_features); a mispaired bundle would otherwise refuse every image request when its features are staged;no n-gram embedding preprocessor (its row map assumes entry-padded token rows; Gemma4 per-layer embeddings are supported);
under cached Gemma4 per-layer embedding storage (
--ple-residency cached), the cache maps at leastmin(PLE table rows, B x S_in)distinct rows, that iscache_slots >= min(PLE table rows, B x S_in) + 1(slot 0 is reserved). One compact step gathers the rows of all its tokens at once, up toB x S_inof them whenprefillFirstpacks several waiting prompts, and a step whose distinct rows exceed the cache would fail the session. A single-cohort step needs only one prompt’s rows or one row per resident. Resident storage always qualifies;no bounded sliding-window KV storage;
no layer debugger;
the engine’s prefill and decode profiles both admit execution phase 8 (steps with prompt rows run on the prefill profile, decode rows of only some residents on the decode profile).
Then the runtime decides from the engine’s config.json and the GPU’s SM (compactExecutionGap,
cpp/runtime/config/compactCapability.cpp). It sorts the engine’s layers into kinds: attention
(kv_layer_configs, num_attention_heads, kv_cache_dtype, builder_config.tp_size), QSA
attention (indexer_head_dim greater than 0 makes every attention layer QSA),
gated_delta_net or Mamba2 SSD (the recurrent-state fields; only Mamba2 exports
recurrent_state_num_groups), causal_conv1d (conv_dim, conv_kernel) and the n-gram conv. It
checks each kind with the predicate that kind’s plugin applies when it reserves the phase-8
workspace and when it accepts or refuses a phase-8 enqueue (cpp/kernels/compactPhase/):
Layer kind |
Predicate |
Phase 8 needs |
|---|---|---|
Attention |
|
no context mask selector, tree, vision-block or sink; CTM decode at TP 1 on SM100/101/103/110, CTM or XQA decode elsewhere; Blackwell or FMHA-v2 ragged paged prefill |
QSA attention |
none |
refused on every SM: |
|
|
the packed-varlen GDN kernel, in an artifact built for the SM |
Mamba2 SSD |
|
the packed SSD kernel; on SM120/121 the state-pool kernel |
|
|
a depthwise layer and the packed-varlen conv kernel at dilation 1, in an artifact built for the SM |
n-gram conv |
|
the same, at a dilation of the n-gram size |
The predicates are compiled into both the runtime and the plugin library, under the same CUDA
version and CuTe DSL kernel-group macros, and read EDGELLM_DISABLE_CTM_DECODE and the device’s SM
count in the calling process. Within one build the runtime therefore decides as the plugins do,
including CUDA 11 builds, builds without a CuTe DSL kernel group (ENABLE_CUTE_DSL=OFF) and the GQA
ratios XQA and CTM cover. In a build with every kernel group and the kernels for the target SM,
attention runs on SM80 to SM121 for the shapes its kernels cover, gated_delta_net (K = V = 128) on
SM80 to SM90 and on SM100, 101, 103, 110, 120 and 121, causal_conv1d (width 2 to 8) on SM80 to
SM90 and on SM100, 101, 103, 110, 120 and 121, Mamba2 SSD ((64, 64), (64, 128), (80, 128)) on
SM100, 101, 103 and 110 and (head dim 64, 80 or 128, state size 64 or 128) on SM120 and 121, and the
n-gram conv and context-mask-selector attention nowhere. A CuTe DSL artifact carries the kernels of
the SMs it was built for, its metadata.json gpu_arch list. The kernel-group macros
(CUTE_DSL_CAUSAL_CONV1D_ENABLED and the GDN group) are defined per artifact, not per SM, so
cmake/CuteDsl.cmake also passes that list (CUTE_DSL_ARTIFACT_SMS) and the causal_conv1d and
gated_delta_net predicates accept only an SM the artifact carries (cuteDslArtifactCarriesSm, an
exact match, as the multi-SM dispatcher of a combined artifact dispatches). An SM100 build run on an
SM86 or SM120 GPU therefore refuses those kinds, with the linked CuTe DSL artifact has no kernels for this SM, instead of aborting the process at the first phase-8 enqueue (failUnsupportedSm in
the multi-SM dispatcher that kernelSrcs/build_cutedsl.py generates).
A single-SM artifact of an earlier minor revision of the same major, whose kernels the newer SM
could load, is refused too. An unknown SM is refused.
The reason names every blocking kind once, with its shape, why and the SMs that support that shape
in this build, for example no compact-execution kernels on SM87 for Mamba2 SSD layers (head dim 64, state size 128: no packed SSD prefill kernel for this SM, head dim and state size in this build; supported on SM100/SM101/SM103/SM110). No forward runs to decide.
CompactCapabilityTest.EachKindFollowsItsPluginPredicateOnEverySm checks that every kind is checked
with its predicate on every SM.
What the engine config does not record is taken from what the export paths produce: no attention layer is a tree, vision-block or attention-sink layer (the preconditions above refuse speculative and vision-block deployments), and a KV-shared bounded-SWA-capable layer receives the donor’s current K/V.
The predicates say what this build’s plugins can run, not what an engine reserved: a plugin sizes
its phase-8 workspace in getWorkspaceSize() when the engine is built, and TensorRT serializes that
size with the engine. Both builders (LLMBuilder::build and the checkpoint-direct
build_with_compact_report) therefore open a compact build report of the plugin library around the
TensorRT build (edgellmBeginCompactBuildReport / edgellmEndCompactBuildReport in
cpp/plugins/utils/pluginUtils.h). Each attention, gated_delta_net, causal_conv1d and
update_ssm_state layer reports from getWorkspaceSize() whether it reserved the compact workspace;
a layer counts as reserved only if every query of it did. When at least one such layer was queried,
every one reserved it and no other build overlapped, the builder writes
builder_config.compact_execution: true; otherwise it omits the field, and the library logs why
as a WARNING, naming (by kind and layer name) the layers that did not reserve it. Only the LLM
engine’s build opens a report; visual, audio and other component engines run no compact steps.
Under tensor parallelism only rank 0 writes config.json, so its build’s report decides, and every
rank builds the same plugin layers with the same per-rank shapes. After the predicates pass, the
runtime requires that field to be true; any other engine, for example one built before the field
existed or one with a layer that has no compact path in the build that built it, runs without
compact steps with the reason the engine was not built for compact execution, which points to
that builder log. A change that enlarges a plugin’s compact workspace must, in the
same change, make engines built before it stop running compact steps, for example by renaming this
builder_config field. Each plugin keeps its enqueue-time check, described next.
A plugin that refuses a phase-8 enqueue logs an ERROR that identifies the layer by kind, name and
shape (AttentionPlugin '<layer>' (16 query heads, 8 KV heads, head dim 128): compact execution is not supported for this layer on this GPU.) and returns before it launches any work. TensorRT passes an
empty name when it deserializes an engine, so the plugins serialize the build-time layer name (for
an ONNX-path engine, the ONNX node name, which TensorRT’s own enqueue error also prints).
LLMRankRuntime::runCompactForward tells this refusal apart from the other failures of a
compact forward: a failed prepare() (input shapes, bindings) and an enqueue that leaves the stream
in error or a CUDA error pending fail the step and leave the capability as it is. A refused step
fails too (StepResult::ok false), with StepResult::compactRefused set and StepResult::failure
stating that compact execution is now disabled and that the request can be retried. The session’s
state is no longer trusted, because the layers before the refusing one already advanced recurrent
and convolution state in place. The runtime then records compact execution as unavailable for its life
(a plugin refused compact execution at runtime (see the plugin's ERROR log, which identifies the layer by kind and shape)), so every session opened afterwards reports compact
false and runs single-cohort steps. The runtime never falls back within a step, and the caller
closes the failed session, as for any step that is not ok. The runtime cannot see why a plugin
returned failure, so any enqueue failure that leaves no device error counts as a refusal: a shape
the predicate refuses, a kernel module that fails to load, or an error the plugin catches. Outside
an explicit mixed, such a failure therefore disables compact execution for the runtime’s life and
the engine keeps serving; it shows as compact_refusals, the plugin’s ERROR log and
compact_unavailable_reason in /health, not as a stopped engine. How RequestEngine continues is
described under Failure handling.
RuntimeSession::capabilities() carries the result (compact, reason) and the engine’s B, S_in
and P.
RequestEngine resolves the configured strategy against these capabilities on its actor
thread (resolveSchedulingPolicy) when its first session opens, and again against the reopened
session after a refused compact step, and logs each outcome:
RequestEngine scheduling: mode=mixed (auto: compact execution supported), execution=compact, schedulingPolicy=fcfs,
preemption=none, maxNumTokens=640, maxChunkTokens=512, priorityReservedSlots=0, maxPauseSteps=0; engine B=4,
S_in=1024, P=128; compact execution available.
ResolvedSchedulingPolicy::modeSource (nanobind mode_source) records how the mode was chosen:
kRequested keeps the configured mode, kAuto is auto’s choice, and only then reason says why;
the line prints requested or auto: <reason>. resolveSchedulingPolicy runs as named steps:
validateSchedulingOptions refuses an option that could never apply, computeStepBudgets range-checks the
budgets and derives the defaults, resolveMode picks the mode, and collectWarnings lists what the
options asked for that the resolved policy does not apply as given.
execution is compact when the session runs compact steps and single-cohort otherwise
(executionName, nanobind ResolvedSchedulingPolicy.execution): prefillFirst names a step
composition, not a backend, and runs either way. After a refused compact step the line is logged as
a WARNING that starts RequestEngine scheduling changed after a plugin refused a compact step:.
RequestEngine::resolvedPolicy() (nanobind resolved_policy) returns the policy in effect by
value, because that fallback can replace it once.
Construction fails with std::invalid_argument (Python ValueError, server startup error
invalid server configuration: ... with the options spelled as command-line flags) when mixed
is requested on a deployment without compact execution (the message gives the reason, which
names each blocking layer kind and the SMs that support it, and suggests prefillFirst), when preemption=pause is set
without schedulingPolicy=priority or with an explicit prefillFirst or singleCohort, when
priorityReservedSlots is set without schedulingPolicy=priority, when maxPauseSteps is set without
preemption=pause, or when a budget below
violates its constraint. Any other failure while the first session is set up is a
std::runtime_error, not a configuration error. What the engine accepts but does not apply as given is reported in
ResolvedSchedulingPolicy::warnings (nanobind warnings), each logged once at startup as a
RequestEngine configuration: warning:
preemption=pauseunderautothat resolved toprefillFirst: pause is dropped (preemption=none,maxPauseSteps=0), since a pending prompt already runs while decoding residents wait. The same flags therefore start on deployments with and without compact execution.An explicit
maxNumTokensormaxChunkTokenswhen the mode resolved toprefillFirstorsingleCohort, which do not use them. The range checks still apply.An explicit
maxChunkTokensabove S_in undermixed: one prompt chunk holds at most S_in tokens, so the budget resolves tomax(P, roundDown_P(S_in)), the value the startup line shows.schedulingPolicy=prioritywithpriorityReservedSlots=0andmaxBatchSize >= 2: an urgent request waits for a resident to finish while every slot is busy. The engine does not reserve a slot on its own; the recipe is an engine built forB + kandmaxBatchSize = B + k, priorityReservedSlots = k.
Configuration#
The same options exist on SchedulingOptions (EngineConfig::scheduling), the nanobind
RequestEngine and the server (the server flags require --enable-in-flight-batching). B is the engine’s built
batch size, S_in its maximum input length, T_max = B x S_in, P the KV page size (128).
|
Default |
Constraint and meaning |
|---|---|---|
|
|
|
|
|
In [B + P, T_max]. Token budget of a |
|
|
A multiple of P in [P, maxNumTokens - B], then capped at |
|
|
|
|
|
|
|
0 |
In [0, maxBatchSize - 1] and requires |
|
0 |
Non-negative; a positive value requires |
The server checks the options that need no engine fact before the engine is built
(validate_ifb_scheduling_options, also run by LLM.__init__ before prepare_model): scheduling
flags without --enable-in-flight-batching and the option types, then the C++
validateSchedulingOptions through the binding validate_scheduling_options, for the batch size
the engine is built with. That binding parses the names with the scheduler’s own name tables and
refuses with the scheduler’s own messages, so the server keeps no copy of either. The
step budgets depend on the engine’s page size and are checked once, by the engine when it starts;
the CLI reports its refusal as invalid server configuration: ... with the flag names. Under
--scheduling-policy priority its HTTP admission gate
(_IFBAdmissionGate) lets a request with priority < 0 hold up to --max-batch-size leases past
the bound at which other requests get 429, so an urgent request reaches the engine’s queue under
overload. --min-request-priority (ApiConfig.min_request_priority, default 0) is the most
urgent priority the server accepts: EngineClient._prepare_native_request answers a request below
it with HTTP 400 (param: "priority") before it takes an admission lease or builds a native
request. The check is a server option, not an EngineConfig field, and applies with and without
--enable-in-flight-batching. The minimum is at most 0 (ApiConfig refuses a positive one), so a
request without a priority, which includes every /v1/responses and /v1/messages request, is
always accepted. With the default, a client cannot use the urgent headroom above or the reserved
slots; EngineClient logs a startup WARNING when priority with --priority-reserved-slots > 0
meets a non-negative minimum. To start an urgent HTTP request at once while every slot is busy, an
operator sets all of --scheduling-policy priority, --priority-reserved-slots k on an engine built
for k extra slots, and --min-request-priority below 0; --preemption pause in addition keeps less
urgent decode rows out of its prompt’s steps. With the defaults a negative priority gets HTTP 400,
and with a full batch and no reserved slot an urgent request waits for a running request to
finish: no running request is evicted or swapped out.
The request field is LLMGenerationRequest::priority (nanobind priority, OpenAI request
priority).
Step planning#
StepPlanner::plan (cpp/scheduler/stepPlanner.cpp) composes each compact step from the
resident table’s ledger (prompt length, committed positions, whether the final chunk ran):
mixed: every decoding resident gets one decode row; with any decode row the step budget ismaxNumTokensand the per-request capmaxChunkTokens, otherwise T_max and S_in.prefillFirst: with any unfinished prompt only prompt chunks run, with budget T_max and cap S_in; underschedulingPolicy=priorityonly the unfinished prompts of the best (lowest) priority value run, and the others wait for a later prompt-only step.Prompt chunks are handed out in order (arrival, or
(priority, arrival)):n = min(remaining, cap, S_in, budget). A chunk that does not finish its prompt is cut back to end on a page boundary; a request left withn <= 0is skipped so a shorter one later in order may still fit. The planner accepts only the budgets the resolver produces, so the first prompt in order always gets a chunk: a plan over residents is never empty and never consists of the decode rows of only some residents.
Exclusive residents (a guided request, media under a visual-token pruner, media whose token
layout only its encoder determines) keep single-cohort steps inside a compact session, whose
context step accepts only the whole uncommitted prompt. The planner meets that contract without
knowing about them: an exclusive resident is alone in the session, so a step has no decode row,
its cap is S_in and its budget T_max, and admission already bounds the uncommitted prompt by S_in,
so the whole prompt is one chunk. A planner change that chunks prompts when no request is
decoding (for example to bound step latency) must leave exclusive residents out, or the session
fails the step. The guided row of
RequestEngineCompactTests.AdmissionAtAStepBoundaryFollowsTheModeThePolicyAndTheSlots pins this.
Admission (RequestEngine::tryAdmitHead): the queue head in policy order is admitted when a
slot is free (and the reserved-slot rule allows it), it is compatible with the residents and
not exclusive, and its prompt fits; otherwise it waits and nothing behind it overtakes it. KV is reserved per slot (B x the per-sequence capacity),
so an admitted request always runs to completion: there is no eviction or recomputation.
Under schedulingPolicy=priority, a head that outranks every resident but waits on batch
compatibility is logged once, with the first field that differs
(BatchCompatibility::firstDifference).
Each resident carries its own generate budget: maxGenerateLength is not a batch
compatibility field. The session bounds a request’s budget by the KV capacity behind its own
prompt at admission (RankRuntimeSession::generateBudgetFor), stores it on the slot
(DecodingInferenceContext::maxGenerateLengths, moved with the row by eviction and the
seated prefill’s row exchange), and every length stop check reads the row’s budget
(DecodingInferenceContext::generateBudget). A batch built at request start gives every slot
the batch’s budget, so the blocking path stops where it always did.
Chunked prefill#
Aspect |
Rule |
|---|---|
Boundaries |
The session accepts a chunk of any positive length up to the remaining prompt and S_in; it does not require page alignment. The step planner ends every non-final chunk on a multiple of P (see Step planning); the final chunk has any length. |
Attention |
The chunk attends to all KV the sequence already has (causal mask aligned bottom-right) and writes its own K/V. |
Recurrent state (GDN, conv1d, SSD) |
Continues from the slot’s state and writes the end state back. |
Output |
A non-final chunk only advances the committed length: it is not sampled and emits nothing. The final chunk samples the first token. The ledger marks the prompt complete only at the final chunk. |
Cancellation |
Takes effect at the next step boundary; a request cancelled between chunks is retired without completing its prompt and publishes nothing to the context cache. |
Numerics |
Chunking changes the accumulation segments of attention softmax and the recurrences, and decode rows inside a mixed step run the prefill profile’s GEMMs: outputs may differ from single-cohort execution by rounding (greedy flips only at near-ties). On SM120/121 the prompt rows of a compact step run Mamba2 SSD through the packed state-pool kernel, while single-cohort prefill runs the SM80 chunk-scan kernel, so Mamba2 hybrids differ more there: Nemotron-3-Nano-4B FP16 can flip greedy tokens where the reference top-2 logprob gap is up to about 0.16, against up to 0.1 for the other families and GPUs. Neither path is the outlier: measured against B100 single-cohort output, SM120 single-cohort and SM120 compact steps differ by a similar amount. |
Skip-softmax (BLASST) |
Compact prompt chunks of an engine exported with a prefill skip-softmax table run the exact (dense) attention kernel by design: the ragged phase-8 kernels take no skip table. Such a deployment therefore gets no skip-softmax prefill speedup under |
Priority and pause#
With schedulingPolicy=priority and preemption=pause, each step takes h, the first resident in
(priority, arrival) order with an unfinished prompt, and leaves out every resident whose priority
value is larger than h’s (its KV and recurrent state stay in its slot, untouched). A paused resident
therefore resumes in the first step in which no resident with a smaller priority value has an
unfinished prompt; a second urgent prompt admitted before h’s final chunk extends the pause. If no
decode row remains in the step, h’s budget widens to its whole prompt (budget T_max, cap S_in), so its
time to first token approaches that of a request running alone. That holds only when no resident of
the same or a smaller priority value is decoding (and none is exempt under maxPauseSteps): such
residents keep their decode rows, and h’s prompt is chunked beside them under maxNumTokens and
maxChunkTokens. Pause never
frees KV: requests that need a slot when all are busy use priorityReservedSlots. With
maxPauseSteps = n > 0, a resident paused n steps in a row (RequestRecord::consecutivePausedSteps)
is not paused in the next step, then may be paused again: a decoding resident gets its decode row,
while one with an unfinished prompt gets a chunk only if the budget left after the more urgent
prompts holds one (strict priority order would starve it without pause too). With 0, a steady
stream of more urgent prompts keeps a resident paused. Under
auto resolved to prefillFirst, pause is dropped with a warning (see above); there, and with
an explicit prefillFirst, a resident’s decode already waits while a prompt runs, and under the
priority policy a compact prompt-only step carries only the most urgent prompts. Under
single-cohort execution the ordering applies and each admission’s prompt runs alone.
An urgent request therefore starts at the next boundary when a slot (free, or reserved for
priority < 0) is available, its batch-wide parameters match the residents’ (the fields
BatchCompatibility compares, such as the LoRA adapter and thinking mode; sampling parameters,
seed, generate budget and logprobs width are each sequence’s own), and neither it nor a resident
is exclusive. Without a slot it waits for a resident to finish; with
differing batch-wide parameters or beside an exclusive resident it waits for every resident to
finish. The ordering is strict, without aging. tests/test_cases/ifb_priority.json covers the free-slot and
reserved-slot cases, with the urgent request’s budget equal to and different from the
residents’.
CUDA graphs#
Step |
Profile |
Execution |
|---|---|---|
Every resident decodes (phase 3) |
decode |
A CUDA graph captured by the application ( |
At least one prompt chunk (phase 8) |
prefill |
Eager. |
Decode rows of only some residents (phase 8, |
decode |
Eager. The planner never plans one, because a step with a pending prompt or a paused resident always carries a chunk; it runs only when every prompt chunk of the step is cancelled before the step runs while another prompt is still pending or residents are paused. |
Under mixed, whenever a prompt is unfinished every step carries a chunk of it
(maxNumTokens >= B + P), so a step without a chunk decodes every unpaused resident and
stays on the graph. When the residents whose chunks a step names are cancelled before it runs
and its decode rows cover every resident left, the cancelled residents are evicted first and the
step runs as phase 3 (counted in decodeSteps, on the graph when one matches). When paused
residents are left out or another prompt is still pending, the remaining decode rows run as the
Np = 0 phase-8 step above.
A compact session runs its steps with prompt chunks (phase 8 on the prefill profile) on a second
TensorRT execution context that stays on the prefill profile (EngineExecutor::dedicateCompactContext).
TensorRT updates layer resources on the first enqueue after a profile switch: on B100 with
Qwen3-0.6B, the first compact step after a decode step spent 4.1 to 5.9 ms in enqueueV3, against
2.1 ms for the next step of the same shape. Every other step keeps the original context: decode
steps and their graphs, Np = 0 steps on the decode profile, the blocking path, and every step of a
single-cohort session, which creates no second context. Captured graphs are keyed by context,
profile and bindings, and a step looks one up only when its context and profile own a captured
graph. When TensorRT cannot create the second context or select the prompt profile on it
(EngineExecutor::dedicateCompactContext), the executor logs a warning and compact execution stays
available: prompt steps run on the original context and pay the profile switch. When the second
context’s auxiliary streams cannot be created, the setup throws and compact execution is recorded as
unavailable, as for a failed media reservation (see the next section).
The dedicated context has its own auxiliary streams but shares the executor’s device memory with the original context and with the vision and audio encoders that run at admission. Sharing is sound only because every enqueue of the session is ordered on the session stream; a change that runs an encoder or a step on another stream must give it memory of its own.
What the dedicated context adds is TensorRT’s per-context state and the private buffers of its plugin
clones (TensorRT clones every plugin per execution context). Per attention layer, a clone holds
128 x B x Hkv x 8 bytes of CTM decode semaphores when the layer decodes with CTM, and 256 bytes of
XQA counters when it can decode with XQA; recurrent and convolution plugins allocate nothing per
clone. The cost therefore grows linearly with the number of attention layers, B and Hkv: when every
layer decodes with CTM, as on B100, the semaphores of Qwen3-0.6B at B = 4 (28 attention layers,
Hkv = 8) take 0.9 MiB. Measured as the process’s peak device memory with and without the dedicated
context, at B = 4 under a workload of prompts arriving while others decode, it added 2 to 6 MiB on B100 (nvidia-smi, 2 MiB granularity)
and 2 to 4 MiB on Thor, for Qwen3-0.6B and Qwen3.5-0.8B. Larger models and B = 8 have not been
measured.
A compact step synchronizes the host with the session stream once in the steady state, after
sampling (COMPACT_SAMPLE), to read the sampled tokens. A step in which a sequence finishes
synchronizes once more in its eviction (LLMRankRuntime::performBatchEvict, or
ContextCacheCoordinator batch compaction under a context cache), as a single-cohort step does. Under
a context cache, a hybrid deployment’s step that publishes decode-end recurrent snapshots synchronizes
once more (completeDecodeStep). The host-staging and page-table uploads wait on events of the
previous step, which have normally completed by then. Admission work (the deferred media preprocess,
image content digests, encoders) synchronizes between steps, not inside one.
Context cache and multimodal input#
Context cache: a request’s reused prefix is its starting committed length; only the uncached part is chunked. Cache pages and hybrid snapshots are published when the final chunk commits, never for a prompt that was cancelled or failed before it. The first chunk’s
prepareSequenceAdmissionrestores the sequence’s hybrid snapshot or zeroes its recurrent state; without a context cache,reserveAdmissionclears the slot’s recurrent and convolution state when the sequence is admitted.Multimodal input: media prompts run in compact steps like text prompts, and an image may span chunks. A media request’s vision encoder runs at admission only after its slot (and, under a context cache, its page lease) is reserved, and only when the encoder embedding cache does not hold all of its images. Audio is split differently, because an audio runner’s
preprocess(which lays out the prompt’s audio tokens) has no text-only mode: it runs at intent, before the reservation. For the Qwen3-Omni / Qwen3-ASR, Gemma4 and Nemotron-Omni audio runners that preprocess already runs the audio encoder, so their encoder runs before the reservation, again at each boundary that retries a request refused for capacity, and also on an encoder embedding cache hit (the hit only replaces the rows it produced). The Gemma4 unified audio runner extracts features inpreprocessand runs its encoder after the reservation. The rows of its media positions from the prefill start on, together with the deepstack features, are copied into the slot’s region of a per-slot reservation (CompactMediaReservation,cpp/runtime/state/), which later admissions cannot overwrite while the request still has prompt chunks to run; positions inside a reused prefix are not embedded again. Prompt chunks gather their media rows from the reservation by global row index (64-bit offsets in the embedding kernels). M-RoPE positions are written once per admission into the slot’s resident row, when the media is staged for a media prompt and at the first chunk for a text prompt, and are not rewritten by later steps. Encoders run insideRuntimeSession::admiton the session stream, so the next step waits for them.Reservation memory: allocated once, when the first session that allows compact execution opens on a deployment that supports it and has a vision or audio encoder (
prepareCompactExecution).RequestEngineallows compact execution in every mode exceptsingleCohort, soautoandprefillFirstallocate it too, whether or not their steps then mix prompts with decode rows. It takes FP16 buffers ofB x (E_image x (1 + deepstack layers) + E_audio) x hidden sizeelements, whereE_image = min(S_in, rows of the vision encoder's output buffer)(0 without a vision encoder) andE_audio = S_in(0 without an audio encoder, and reserved whenever the deployment has one). The runtime logs the layout and size at INFO when it allocates the reservation (`Compact multimodal reservation: slots x (<E_image> image rows x (1 +deepstack layers) <E_audio> audio rows) x
hidden FP16 = MiB. ). Validated vision-language and ASR deployments at B = 4 and S_in 1024 or 2048 log 8 to 64 MiB. The size grows with B, S_in and the hidden size:E_image = S_in = 4096`, hidden size 4096 and 3 deepstack layers reserve 512 MiB at B = 4. The reservation stays allocated for the runtime’s life, also after a refused step disables compact execution.
Failed compact setup: when setting up compact resources throws with the stream healthy (
LLMRankRuntime::prepareCompactExecution: the media reservation cannot be allocated, or the dedicated context’s auxiliary streams cannot be created), the reservation is released, compact execution is recorded as unavailable for the runtime’s life with a reason naming the failure (for the reservation, its layout and size), a warning is logged, and the session serves single-cohort steps:autoresolves toprefillFirstwithout compact execution andmixedfails at startup with that reason. Every later session of the runtime keeps that result. A setup that leaves the stream in error makesRankRuntimeSession::openthrow instead.Media under a visual-token pruner and grammar-constrained (guided) requests are exclusive residents that keep single-cohort steps on a compact deployment: the pruner gates on an empty base KV and runs a whole-prompt prefill, and the guided decoder’s matchers follow execution rows. A request with a token callback is also exclusive (see the request-level matrix).
Media whose token layout only its encoder determines (
MultimodalRunner::tokenLayoutNeedsEncoder) is an exclusive resident with single-cohort steps on a compact deployment too, its encoder run at admission as on a deployment without compact execution. A Nemotron-Omni video is such media: its tubelets’ token counts come from EVS pruning, which needs the encoder output, so an encoder embedding cache hit cannot lay it out either and the encoder runs again.
Observability#
Startup: the resolved strategy, its reason, the execution, the budgets, the engine limits and the compact capability with the reason it is unavailable, naming any rejecting layer (log line above), then one warning per entry of
ResolvedSchedulingPolicy::warnings.NVTX ranges:
IFB_PLAN,IFB_EXECUTE,IFB_COMMITper step on the actor;COMPACT_BUILD,COMPACT_ENQUEUE,COMPACT_SAMPLE,COMPACT_COMMITinside a compact step (built withENABLE_NVTX_PROFILING).RequestEngine::metrics()(EngineMetrics, nanobindmetrics(), server/healthscheduling): besides queue latency, admission stalls and outcomes, counted over steps that ran: steps by kind (decodeStepsand thedecodeGraphStepsamong them that replayed a CUDA graph,promptStepsand themixedStepsamong them,decodeSubsetSteps), running sums of N, Np (stepPromptChunks,partialPromptChunksof them non-final), Nd, T and L (stepSequences,stepDecodeRows,stepTokens,stepSampledRows), actor host step time (stepTimeTotalUs,stepTimeMaxUs, plan through commit),pausedResidentSteps, andcompactRefusals, the compact steps a plugin refused.RequestEngine::status()(EngineStatus, nanobindstatus()):serving, false once the engine opens no session again (see Failure handling),unavailableReason, andcompactUnavailableReason, the current session’s capability reason or the refusal that stopped the engine. The server’s/healthadds it, with the policy in effect (mode, execution, reason, scheduling policy, applied preemption, budgets, reserved slots, pause limit and warnings), as theengineblock (LLM.get_engine_status()), and answers HTTP 503 withstatus: "unhealthy"and a top-levelreasonwhileservingis false;/health/readyanswers 503 with the reason too, andEngineClient._prepare_native_requestrefuses every new request with 503engine_unavailable.RequestHandle::stats()(nanobindstats()), available once the request is ready: admission, queue time, time to first token measured at the end of the step that sampled it, prompt and reused prompt tokens, prompt chunks, and the steps and host time the request was paused.
Support matrix#
Legend: served – runs under IFB, may share a step; exclusive – runs under IFB
but alone: it is admitted only into an empty session and nothing joins it while it is
resident (the head waits, it is never rejected);
blocking path – the server keeps the deployment on
LLMInferenceRuntime::handleRequest, IFB is never engaged, and starting such a deployment
with --enable-in-flight-batching fails with a message naming the reason (one gate,
ifb_unsupported_reason, decided once at load time – never a silent fallback);
text-only – the deployment starts under IFB and serves text; its speech endpoints
refuse each request with a message naming the flag;
rejected – submit() throws std::invalid_argument.
Deployment level#
Deployment |
Status |
Notes / target |
|---|---|---|
Text LLM, single rank |
served |
|
Text LLM, tensor parallelism (thread- or MPI-launched) |
blocking path |
The session runs on one rank in this release; |
Vision / audio input (multimodal) |
served |
Encoders run at admission; see the visual-token-pruner row below. |
Speculative decoding (EAGLE, MTP, Gemma4 MTP, DFlash, JetSpec, Domino, DSpark) |
served |
Single-cohort steps only (see Speculative decoding), with or without a context cache. |
Diffusion backbone |
blocking path |
Cannot be seated mid-request; same construction-time refusal. |
Bounded sliding-window KV storage |
blocking path |
Its request backend has no live-slot admission; same construction-time refusal. Full KV storage ( |
Hybrid Mamba (Nemotron-H) |
served |
Admitted sequences prefill in place and no per-slot state is relocated. Validated in |
Speech output, Qwen3-Omni bundle |
text-only |
The Thinker is a text runtime the engine serves, so the deployment starts under IFB with the Talker/code2wav engines unloaded; |
Speech output, standalone Qwen3-TTS model |
blocking path |
No request engine exists for a TTS-only runtime; the flag is refused at startup. |
Context cache (prefix reuse) |
served |
The session opens one cache request for its life; every sequence, full-attention or attention-plus-recurrent hybrid, leases its pages at admission and returns them when it retires. A hybrid sequence restores only its resident recurrent slot, and cache visibility commits with the step that produced it. Under compact execution several sequences may be in admission at once and publish at their final chunk. |
Compact execution ( |
served where the static preconditions hold and every layer kind has phase-8 kernels on this GPU |
Text and multimodal deployments whose plugins run phase 8 on this GPU (for example Qwen3, Qwen3.5 and Gemma4 E2B with full sliding-window KV storage on SM100). TensorRT-RTX engines, engines with |
Request level (what submit() does with a request)#
Request option |
Status |
Notes / target |
|---|---|---|
Plain text generation, streaming, logprobs, stop strings |
served |
|
Sampling parameters, seed, |
served |
Each sequence carries its own settings ( |
Temperature, top-p or top-k that differ from the residents’ on a speculative deployment |
waits |
A speculative decoder takes one set of sampling parameters per batch, and the sampling parameters select the decoder (see Speculative decoding). The head waits until the session is empty; seeds, budgets and logprobs widths remain per sequence. |
|
waits |
The sequence runs on the default decoder with the blocking path’s sampling and never shares a step with speculative sequences: the head waits until the session is empty, and opt-out sequences batch with each other. A speculative head behind them waits in turn. |
Guided decoding (grammar / JSON) |
exclusive |
The |
|
served |
Ordering under |
Media under an active visual-token pruner |
exclusive |
Pruning gates on an empty base KV; a sequence seated beside residents would silently skip it. It runs a whole-prompt prefill with single-cohort steps even on a compact deployment. |
Nemotron-Omni video, compact deployment |
exclusive |
Its token layout comes from the encoder (EVS), so it cannot be laid out before its slot is reserved; it runs its encoder at admission and a whole-prompt prefill with single-cohort steps. Served like other media without compact execution. |
|
exclusive |
The context holds one callback for the whole batch, so a callback-bearing request runs as the session’s only resident; its first token reaches the callback from the context step like every later one. |
|
rejected |
Response state is assembled only by the blocking path. Use the blocking path. |
|
rejected |
Trajectory execution carries blocking-path response state. Use the blocking path. |
Per-request LoRA ( |
rejected |
Adapter weights are engine-wide today. |
More than one sequence per request |
rejected |
The engine batches across requests, not within one. |
The legacy saveSystemPromptKVCache request field is retired: it is superseded by
the context cache / pre-engine warmup. A shared prefix is warmed through the blocking
path before the runtime is handed to the engine, or routed to the context cache (prefix
reuse), which already covers the shared-prefix use case under IFB. No caller needs the
legacy capture on the engine path.
Where the boundaries are enforced#
Construction (
RequestEngine::RequestEngine): the runtime must run on a single rank (supportsSteppedExecution), must be able to seat sequences into a running session (supportsSeatedAdmission), andmaxBatchSizemay not exceed the engine’s built batch dimension (maxBatchSize()). The session is opened here, on the actor’s thread; a runtime that refuses to open one fails construction.Submission (
RequestEngine::submit): the rejected rows above.Admission (
RequestEngine::mayAdmit): the waits and exclusive rows above. These wait rather than reject, so FCFS order is preserved: the head is admitted once the session is empty, and while an exclusive sequence is resident nothing is admitted beside it.RuntimeSession::requiresExclusiveAdmissiontells the engine which requests are exclusive.Runtime (
RankRuntimeSession::admit): a request the session cannot take is refused as transient (kNoCapacity) when nothing was modified, or rejected with a reason when the intent itself is invalid. A request whose batch-wide parameters disagree with the residents’ fails loudly there – the gate above is what keeps them equal. A failure inside the reservation itself (the context-cache lease), media staging that fails on the device, or an admission preprocess that leaves the session’s stream in error is not a refusal: it is reported asSessionLostError, and the engine fails the head and the session together, as it does for a failed step. An encoder that refuses its input is a refusal.Steps (
RuntimeSession::execute): the engine’s plan (oneScheduledSequenceDescriptorper participant, built from the resident table’s committed lengths) is checked against the session’s ledger before it runs; a plan that disagrees is a contract violation and fails the step. The result’s execution record says what ran – how many positions each sequence gained, which a visual-token pruner makes fewer than the prompt range a context step was asked to cover – and the resident table advances from that record, never from its own plan. After every step the table and the session must name the same residents; a divergence fails the session rather than the step that would eventually trip over it.Failures: see the next section.
Failure handling#
A failure is answered at the narrowest scope whose state is still known to be sound.
The request’s own. Validation, templating, tokenization, the encoder preprocess and, under a context cache, the media-placeholder count run before anything is reserved (
GenerationSession::buildAdmissionIntent); a throw there refuses the head and modifies nothing, unless it leaves the session’s stream in error. The encoders run on that stream, so the next step would fail whichever request was blamed:RankRuntimeSession::admitthen throwsSessionLostErrorand the failure is the session’s. A seated prefill that fails – the engine reports failure, or throws – closes its admission (failAdmission: the row is terminal withkError, its lease aborted) and the context step’s eviction files it in that same step, while the residents beside it decode on. That eviction synchronizes the stream, so it is also the check that the device survived: a sticky device error surfaces there and fails the step. A response that cannot be materialized at commit fails that one sequence.The session’s. A throw inside the reservation (the context-cache lease, device work) or an admission preprocess that leaves the session’s stream in error leaves the session’s device state unknown. A decode step that is not
okor throws, a plan the session refuses, or a divergence between the resident table and the session leaves a step half-applied that every resident took part in. In either case the engine retires every resident with the reason, closes the session (RequestEngine::failSession) and opens a fresh one. Closing is a full reset: the context is new, the session’s one context-cache request is released (ContextCacheCoordinator::abandonsynchronizes its pending device work, or quarantines it and poisons the coordinator when that fails), and the next session opens with a cleared recurrent/convolution state pool and asserts that the coordinator holds no request. Hybrid checkpoints the failed step had staged are dropped with the request; the ones it had already published were backed by device work thatabandonsynchronized, and when that synchronization fails the coordinator is poisoned and no later session can look them up. There is no in-place recovery of a failed decode step: none of its residents has state known to be good, so evicting them in place reaches the same empty session at the cost of a compaction over a half-applied step. vLLM’sEngineCoreand TensorRT-LLM’s executor loop likewise fail every request on a step exception.A refused compact step. A step whose
StepResulthascompactRefusedset (see Execution and strategy) is a session failure: the session closes andEngineMetrics::compactRefusalscounts it. No resident continues in place, because the layers before the refusing one already advanced their recurrent and convolution state. What follows depends on the configured mode:autoor an explicitprefillFirst: a resident whose prompt’s final chunk has not committed has sampled no token, so its caller has seen nothing and it can run again from scratch on the next session.RequestEngine::requeuePromptIncompletetakes these off the ledger (RequestCoordinator::releasePromptIncomplete), has the session release each one’s stream channel unfinished and unattached (RuntimeSession::releaseForReadmission, so the stream finalizer skips it when the session closes; the context-cache lease returns with the session as for any resident), and queues the request again at its(priority, arrival)place, ahead of later arrivals of the same or a less urgent priority, with its handle, stream, cancel token and backlog reservation. Its stats restart and its queue latency is not counted twice. A cancelled one retires as cancelled, including one the session retired at the top of the refused step, which has nothing to release. The other residents, which have streamed tokens, fail withStepResult::failure, which says that the request can be retried. Their outcome carriesFailureKind::kCompactStepRefused(RequestHandle::get()throwsRequestFailedError, nanobindRequestFailedError.kind), which the server maps to HTTP 503compact_step_refusedwithRetry-After. The engine reopens the session without counting the failure towardmaxConsecutiveSessionFailures, since the runtime reports no compact execution for its remaining life and the refusal cannot repeat. It resolves the policy again against the new session (RequestEngine::adoptPolicy): both becomeprefillFirstover single-cohort steps, the planner is removed, andpreemption=pauseunderautois dropped with its startup warning. Queued requests are served by the new session. The residents’ failure is published after the reopen: when no session opens or the policy cannot be resolved again, the engine stops serving and they fail withFailureKind::kEngineStoppedand the reason, not as retryable.An explicit
mixed: it asks for compact steps, which the runtime no longer runs, so the engine does not reopen. Every resident fails, none is queued again, with a reason that names the refusal and states that the engine stops serving becauseifbSchedulingMode=mixedrequires compact execution. Every later request fails at admission withthe runtime session is unavailable: ifbSchedulingMode=mixed requires compact execution, and a plugin refused ..., which suggestsauto. All of them fail withFailureKind::kEngineStopped, which the server maps to HTTP 503engine_unavailable.resolvedPolicy()keeps the mixed policy, andstatus().servingis false.
The engine’s. Opening a session hardly touches the device, so a reopen succeeds even when the failure’s cause is sticky. After
EngineConfig::maxConsecutiveSessionFailuressession failures in a row with no step completed between them (default 2; a refused compact step does not count), or when no session can be opened at all (a poisoned coordinator refuses every request), the engine stops reopening: every later request fails at admission with the reason (FailureKind::kEngineStopped) rather than parking its caller, and the engine keeps running until it is recreated.RequestEngine::status()then reportsservingfalse with that reason, and the server’s/healthand/health/readyanswer HTTP 503, as does every new request (engine_unavailable).
Cancellation#
A request’s cancellation is one bit the request owns (RequestRecord::cancelToken): set
on the caller’s thread by RequestHandle::cancel() or RequestEngine::cancel(id), read by
the actor before every admission and at every drain, and polled by the runtime at the top
of every step the sequence takes part in (it travels into the slot at admission). The
token channel keeps token transport and finish; its own cancel flag remains for consumers
that cancel the channel directly, and counts as a cancel too. Under compact execution the
top of every step checks every resident, so a request cancelled between prompt chunks is
retired before its next chunk runs.
Speculative decoding#
A speculative deployment runs one speculative round per decode step over every resident. The
decoder is a batch-wide parameter: the first sequence into an empty session selects it as the
blocking path does (DecoderRegistry::select), from disableSpecDecode and the sampling
parameters. A speculative decoder also takes one set of sampling parameters per batch
(DecodingInferenceContext::uniformSamplingParams). So beside the BatchCompatibility gate, which
compares disableSpecDecode, RequestEngine::mayAdmit holds a head when it or the residents run
on such a decoder (RuntimeSession::requiresUniformSampling) and its temperature, top-p or top-k
differ from theirs (BatchCompatibility::samplingAgrees). Requests that sample alike select the
same decoder, so a head that would select a different decoder waits too, counted in
stallsIncompatible.
Seating. A sequence’s draft state is built in its own context step. Inside the batch-one
projection, after the base prefill, DecodingStrategy::seatRow runs the draft prefill over every
prompt position but the last and leaves the row as if its last verification had accepted one
token: the first sampled token, with the target hidden state at the last prompt position. The
next decode step treats the row like every resident. Gemma4 MTP has no draft KV and seats the
seed alone.
Carried state. Each decoder keeps the target hidden states of every row’s accepted span
itself, since a seated prefill overwrites the pipeline’s baseHiddenStates. Draft KV lengths
persist across rounds, so the projection swaps every cache manager’s row (base and draft) to row
zero and back.
Ledger. A speculative round reports each sequence’s accepted length as the step record’s
queryLength; the caller’s ledger advances by that.
Context cache. The cache batch is opened for the cohort’s decoder (a speculative one leases paired base and draft working sets) and reopened when an empty session latches the other kind. A seated draft holds the prompt but its last position, which the next decode step materializes.
An EAGLE-flavor endpoint publishes at the row’s first decode completion, capped at the common state, or at its eviction if it leaves before decoding.
A Hybrid+spec sequence runs the blocking path’s endpoint-reuse prefill as a batch of one (
runHybridSpecPrefill): the boundary fold on a hit, the volatile-tail split, and the checkpoint publication after the seat. DFlash and DSpark write the last prompt position for the checkpoint and roll the draft length back by one. Such a request must usePREFILL_STATE_ONLYwith endpoint-only capture, as on the blocking path.
Not yet: compact steps (a verify segment beside a prompt chunk), per-sequence sampling and per-step draft length. DSpark’s probabilistic acceptance draws its uniforms by execution row and decode round, so a sampled DSpark request is not reproducible across batch compositions.
Batch bound. The session admits at most the smaller of the base and draft engines’ built
batch (LLMRankRuntime::maxBatchSize).
Follow-ups#
The remaining work is ordered by runtime dependency on the session:
Encoder batching and encoder/decoder overlap for media admissions.
Compact execution where kernels are missing: Mamba2 SSD outside Blackwell GPUs, and vision- block bidirectional and attention-sink ragged FMHA. Also per-SM kernel groups in the artifact metadata, so that a combined multi-SM artifact defines
CUTE_DSL_CAUSAL_CONV1D_ENABLEDand the GDN group only for the SMs that carry them.CUDA graphs for steps with prompt chunks (piecewise capture), if the measured eager host overhead of compact steps justifies it.
Speech output: assemble the Talker/TTS response inside the session so
generateAudiorequests can be admitted.Trajectory execution as session state.
Speculative decoding under IFB, beyond single-cohort steps: speculative rows in compact steps, chunked draft prefill, per-sequence sampling with rejection sampling, and a per-step draft length chosen by batch size.
Guided decoding beside other residents: matchers keyed by resident handle.
Tensor parallelism under IFB: broadcast each step to thread-launched ranks, then carry the same stream over MPI.
Recovery by recomputation after other step failures: a refused compact step already requeues its residents that sampled no token; a failed step of any other kind still fails every resident, and the ones that streamed tokens could be recomputed in a fresh session too.