In-Flight Batching#
In-flight batching lets the server and Python API serve several requests at once. Requests share the running batch and a new request joins at the next generation boundary rather than waiting for the batch to finish, so concurrent clients see their first token while earlier requests are still streaming. It is opt-in and, in this release, runs on a single device.
Enable It#
Server:
trtedge serve Qwen/Qwen3-8B-FP8 \
--max-batch-size 4 \
--enable-in-flight-batching \
--port 8000
Python API:
from tensorrt_edgellm.server import LLM
llm = LLM(model="Qwen/Qwen3-8B-FP8", max_batch_size=4,
enable_in_flight_batching=True)
--max-batch-size (or max_batch_size) is the number of requests that decode
together; it must not exceed the batch size the engine was built with. Without
the flag both entry points serve one request at a time, which remains the
default.
What Changes#
Default (one at a time) |
In-flight batching |
|
|---|---|---|
Concurrent requests |
Queued, served in order |
Up to |
Admission |
Bounded queue with |
Limit of |
Client disconnect |
Cancels the native channel |
Cancels the request inside the engine; its seat is freed |
|
|
|
Direct Python API |
One call at a time |
Calls from several threads overlap, up to the same batch-plus-queue limit; extra callers block |
Each request decodes with its own sampling parameters (temperature, top-k,
top-p, seed), max_tokens and logprobs setting, so requests that differ in
these join a running batch directly. A request whose thinking mode differs from
the running requests’ waits until they have finished and is then admitted on its
own terms. Guided decoding is stricter: a guided request waits the same way but
then runs alone, and nothing joins it until it finishes. The scheduling
counters in /health show how often a request had to wait and why
(stalls_incompatible, stalls_guided, stalls_no_capacity).
Scheduling Modes#
How a new request’s prompt shares the GPU with requests that are already
generating is set by --ifb-scheduling-mode (Python: ifb_scheduling_mode):
Mode |
Behavior |
|---|---|
|
Running requests keep producing one token per step; a new prompt is processed in chunks inside the same steps, so a long prompt does not stall token streams, and several prompts can be processed together. |
|
While a new prompt is pending, the running requests wait and the prompt is processed (in one step when it fits the engine’s maximum input length); then everyone decodes again. |
|
|
|
One new request is admitted per generation boundary and its whole prompt is processed alone while the running requests wait; then everyone decodes again. Compact execution is never used, so the outputs match in-flight batching without compact execution bit for bit. Use it to reproduce such outputs, to debug, or as a fallback. |
mixed needs kernels that run prefill and decode rows in one step (compact
execution). The runtime decides at startup, from the engine’s layer kinds and
shapes and the GPU, whether every layer has them. The startup log
states the execution next to the mode, execution=compact or
execution=single-cohort (Python: LLM.get_engine_status()["execution"]),
because the same mode name can run either way (see below). Compact execution is
available for text and multimodal models whose layers support it on the GPU in
use (for example Qwen3 and Qwen3.5 on Blackwell data-center GPUs, and Qwen3.5 on
SM80 to SM90 GPUs such as Orin when the build’s CuTe DSL artifact carries the
gated_delta_net and causal_conv1d kernels for that SM); image and
audio prompts are processed in chunks like text. It needs an engine built for
a batch size of at least 2. Deployments without it use prefillFirst, for
example TensorRT-RTX engines, n-gram embeddings and Gemma4
vision-bidirectional attention. Gemma4 per-layer embeddings stored with
--ple-residency cached need at least min(PLE table rows, batch size x max input length) + 1 --ple-cache-slots, because one compact step looks up the
rows of all its tokens at once; with fewer slots the deployment uses
prefillFirst without compact execution. The
developer guide
lists every condition. Asking for mixed where it is not available fails at
startup with the reason. When a kind of layer has no such kernel on the GPU in
use, the reason names it with the GPUs that have one in the build in use. For
example, a build with the Mamba2 SSD kernels for Blackwell and for SM120/SM121
reports on SM87 no compact-execution kernels on SM87 for Mamba2 SSD layers (head dim 64, state size 128: no packed SSD prefill kernel for this SM, head dim and state size in this build; supported on SM100/SM101/SM103/SM110/SM120/SM121). An engine whose config.json lacks
builder_config.compact_execution: true runs without compact steps; rebuild it
with the current builder. The builder writes the field when every layer that
has a compact path reserved its compact workspace, and otherwise logs a warning
that names the layers without one. The server rebuilds its cached engines
automatically.
Compact execution belongs to the deployment, not to the mode. Where it is
available, prefillFirst uses it too: several waiting prompts are processed
together in prompt-only steps, and decode steps still replay the captured CUDA
graphs. Only singleCohort turns compact execution off on such a deployment. Guided requests,
media requests under a visual-token pruner and, with compact execution, Nemotron-Omni video
requests run alone, as described below.
Compact execution gives the same task accuracy as singleCohort, but not
bit-identical outputs: a step’s kernels and floating-point rounding depend on
which requests share it. Under mixed, which requests share a step depends on
when they arrive, so repeating the same requests can produce slightly different
text near ties. Use singleCohort when you need outputs that are reproducible
bit for bit.
With vision or audio input, a deployment with compact execution reserves device
memory for the encoder outputs of every slot in every mode except singleCohort
(auto, mixed and prefillFirst), so
that a prompt’s media rows stay available across its chunks. The reservation holds
FP16 values and takes
B x (E_image x (1 + D) + E_audio) x H x 2 bytes
where B is the batch size the engine was built with, H the model’s hidden size,
D the number of deepstack feature layers (3 for Qwen3-VL, 0 for most models),
E_image the smaller of the engine’s maximum input length and the rows of the
vision encoder’s output buffer (0 without a vision encoder), and E_audio the
engine’s maximum input length (0 without an audio encoder). For example, Qwen3-VL-8B
(H = 4096, D = 3) built with batch size 4 and a maximum input length of 4096, with
an encoder buffer of at least 4096 rows, reserves 4 x 4096 x 4 x 4096 x 2 bytes =
512 MiB. The startup log reports the size (Compact multimodal reservation). If
the reservation cannot be allocated, the startup log warns with the reason and
size, auto serves the deployment without compact execution (as prefillFirst
on a deployment without it), and mixed fails at startup with that reason.
singleCohort never allocates it. Compact steps with prompt rows also run on a
second TensorRT execution context; if it cannot be created, the startup log
warns and compact execution stays available, with those steps running on the
original context.
The startup check is meant to keep a plugin from refusing a compact step at runtime. If one refuses anyway, it logs an error that names the layer, and compact execution stays off until the server restarts. What happens next depends on the configured mode:
autoor an explicitprefillFirst: the server keeps serving inprefillFirstmode without compact execution and without--preemption pause, and logs the new mode in a warning. A running request that has not produced a token yet goes back to the queue at its original place, ahead of later arrivals except, under--scheduling-policy priority, more urgent ones, and runs again from the start of its prompt; its client sees no error. A request that has already produced tokens fails with error typecompact_step_refused(see Errors): HTTP 503, or on a stream, which has already started, an SSE error event of that type. It can be retried.scheduling.compact_refusalsand theengineblock of/healthshow the change. If the engine cannot serve again (no new runtime session opens, or the scheduling cannot be resolved again), it stops instead: these requests fail withengine_unavailable, as every later request does.An explicit
mixed: the server stops serving. Every running request fails with an error that names the mode, every later request fails with the reason, and/healthreturns HTTP 503.
Budgets of a mixed step (0 selects the default). Batch size and max input
length here are the limits the engine is built with, --max-batch-size and
--max-input-len (the server reuses a cached engine only when it was built with
them). A budget outside its range stops the server when the engine starts, in
every mode, with an error that names the flag:
Flag |
Default |
Range |
Meaning |
|---|---|---|---|
|
about 512 + batch size, rounded up to 128 |
batch size + 128 .. batch size x max input length |
Tokens one step processes while requests are decoding |
|
up to 512: |
multiple of 128, up to |
Prompt tokens of one request per step while requests are decoding |
One prompt chunk holds at most the max input length (rounded down to 128), so
a larger --max-chunk-tokens, or a larger default, is capped there; for an
explicit value the startup log says so in a warning. The startup log and
/health show the budgets in effect.
A prompt chunk that does not finish the prompt ends on a multiple of 128 tokens
and produces no output; the first token comes from the last chunk. With no
request decoding, a prompt runs in one step. When the mode resolves to
prefillFirst or singleCohort, an explicit --max-num-tokens or
--max-chunk-tokens is not used, and the startup log says so in a warning.
When the engine refuses a scheduling configuration at startup, the server exits
with invalid server configuration: and the reason, naming the options by their
command-line flags (for example --ifb-scheduling-mode mixed is not supported by this deployment: ...). Combinations that do not depend on the engine, such as
--preemption pause with --ifb-scheduling-mode prefillFirst, are refused
before the engine loads.
Priorities#
--scheduling-policy priority orders waiting requests, and the prompt budget
of each step, by the request’s priority field (lower is more urgent, default
0; the OpenAI-compatible priority field is forwarded). Requests of equal
priority keep arrival order. The order is strict and does not age: while more
urgent requests keep arriving, less urgent ones keep waiting. The server
refuses a request whose priority is below --min-request-priority with HTTP
400. The minimum is at most 0, so a request without a priority is always
accepted. The default 0 refuses urgent (negative) priorities; to accept them,
set the minimum to the most urgent priority your clients may send, for example
--min-request-priority -1. Only /v1/chat/completions and /v1/completions
carry a priority; /v1/responses and /v1/messages requests run at priority
0. These options build on the priority policy:
--preemption pause(requires--scheduling-policy priority): while a more urgent request’s prompt is processed, less urgent running requests are paused. They keep their state and resume with their next token once no prompt more urgent than theirs is unfinished, so a second urgent request that arrives in the meantime extends the pause. When no running request of the same or a more urgent priority is generating beside it, the urgent prompt runs in one step, so its time to first token approaches that of a request running alone; otherwise it shares each step with them and can take several. Pausing needs themixedmode. Under the defaultauto, a deployment without compact execution resolves toprefillFirstand starts without pausing, with a warning: there, running requests already wait while a prompt is processed. With an explicitprefillFirstorsingleCohort, the server refuses--preemption pauseat startup.--max-pause-steps N(requires--preemption pause): a request paused for N steps in a row is not paused in the next step, then may be paused again. A request that is generating then produces its next token; one whose prompt is still being processed competes for that step’s prompt budget in priority order and may still get no part of it. The default 0 sets no limit: sustained urgent traffic keeps less urgent requests paused for as long as it lasts. Set a limit when such traffic is possible. A request that runs because of the limit generates beside the urgent prompt, which can then take more than one step.--priority-reserved-slots N(requires--scheduling-policy priority): requests with priority 0 or higher are admitted only while more than N slots are free, so an urgent request (priority below 0) finds a free slot even under load. HTTP clients reach these slots only with a negative--min-request-priority; with the default 0 the server warns at startup that no HTTP request can use them.
Under prefillFirst, a prompt-only step carries only the waiting prompts of the
most urgent priority; less urgent prompts wait for the next prompt-only step.
An urgent request starts at the next generation boundary, beside the running requests, when all of these hold:
A slot is free for it: a slot no request holds, or a reserved slot (
--priority-reserved-slots) for a request with priority below 0. Running requests are never evicted to make room.Its sampling parameters match the running requests’: temperature, top-p, top-k, the number of logprobs,
enable_thinking, whether a sampling seed is set,skip_special_tokensand the context-cache policies.max_tokensmay differ.It is not a request that runs alone (guided decoding, or media under a visual-token pruner), and no such request is running.
Without a slot, it waits for a running request to finish. With differing
sampling parameters, or beside a request that runs alone, it waits until every
running request has finished. Under --scheduling-policy priority, when such
a request is more urgent than every running request but waits for a parameter
mismatch, the server logs one warning for the request that names the parameter.
Running requests are never cancelled for it.
To serve urgent requests at once while all regular slots are busy, use an engine built for k more slots than the regular load needs and reserve them. For a regular load of 4 requests and one urgent request at a time (the engine is built for a batch size of 5):
tensorrt-edgellm-serve Qwen/Qwen3-8B-FP8 \
--max-batch-size 5 \
--enable-in-flight-batching \
--scheduling-policy priority \
--priority-reserved-slots 1 \
--preemption pause \
--min-request-priority -1 \
--port 8000
Regular requests then use at most 4 slots, and the engine pays the KV memory
of the reserved slot. With --scheduling-policy priority and no reserved slot
on an engine of two or more slots, the startup log warns that an urgent request
waits while all slots are busy. Under the priority policy the HTTP admission
limit also leaves room for urgent requests: a request with priority below 0 is
accepted up to --max-batch-size requests past the limit at which other
requests get 429, so it reaches the engine’s priority queue under overload.
All of these flags except --min-request-priority require
--enable-in-flight-batching.
Support Matrix#
Deployments the flag cannot serve refuse it at startup with the reason in the error message; they are never silently served one at a time.
Deployment |
Under |
|---|---|
Text LLM, single device |
Served |
Vision or audio input (VLM, ASR) |
Served; media requests join the running requests like any other, and under |
KV cache reuse ( |
Served; a joining request reuses a published prefix at admission |
Guided decoding |
Accepted, but never batched: a guided request waits for the running requests to finish, then runs alone, and no other request joins it while it runs |
Audio output (Qwen3-Omni speech, standalone Qwen3-TTS) |
Not available. An Omni bundle starts text-only: its speech engines are not loaded and |
Speculative decoding (MTP, EAGLE, DFlash, DSpark and the rest) |
Served, with or without |
Bounded sliding-window KV storage ( |
Refused at startup; use full KV storage |
Tensor parallelism |
Not available; the server and Python API run on a single device in this release |
Hybrid Mamba models (Nemotron-H) |
Starts and batches |
Requests that carry per-request LoRA or a trajectory are rejected with HTTP 400 under in-flight batching. Speech output is available on the default path only.
Speculative Decoding#
Speculative requests batch together. Two kinds of request wait for the running speculative requests to finish instead of joining them:
A request with
disable_spec_decode. It runs on the default decoder, as on the blocking path, and batches with other such requests.A speculative request whose temperature, top-p, or top-k differ from the running speculative requests.
Working With KV Cache Reuse#
Both flags can be enabled together. A request that arrives while another is
decoding is admitted mid-flight and still reuses the published prefix, so its
time to first token stays close to a sequential reuse hit. Its prompt plus
max_tokens should fit --max-kv-cache-capacity; otherwise it is admitted with
its generation length clamped to what fits behind its own prompt, as on the
default path.
Each response’s usage.prompt_tokens_details.cached_tokens reports how many
prompt tokens were served from reused state instead of being prefilled; on
streams it rides the usage chunk (stream_options.include_usage). The
context_cache block of /health carries the cumulative counters.
On a hybrid model such as Qwen3.5, a speculative request publishes only its
prompt: cache_generated_tokens defaults to false, and true is rejected
with HTTP 400.
Reading /health#
"scheduling": {
"submitted": 34, "completed": 33, "cancelled": 1, "failed": 0, "refused": 0,
"admitted_mid_flight": 18,
"stalls_incompatible": 152, "stalls_guided": 65, "stalls_exclusive": 0,
"stalls_no_capacity": 0,
"queue_latency_avg_us": 2669, "queue_latency_max_us": 42032,
"decode_steps": 2210, "decode_graph_steps": 2210, "prompt_steps": 41, "mixed_steps": 30,
"decode_subset_steps": 0, "step_sequences": 7921, "step_prompt_chunks": 52,
"partial_prompt_chunks": 19, "step_decode_rows": 7869, "step_tokens": 31570,
"step_sampled_rows": 7902, "step_time_total_us": 3605120, "step_time_max_us": 6210,
"step_time_avg_us": 1601, "paused_resident_steps": 0, "compact_refusals": 0,
"queued": 0, "resident": 0
},
"context_cache": {
"admitted_sequences": 33, "hit_sequences": 21, "matched_tokens": 5504,
"reused_tokens": 5376, "current_records": 12, "evicted_records": 0,
"base_kv_pages": {"free": 40, "capacity": 96}, ...
},
"engine": {
"serving": true, "unavailable_reason": null,
"scheduling_mode": "mixed", "scheduling_mode_source": "auto",
"scheduling_reason": "compact execution supported", "execution": "compact",
"scheduling_policy": "priority", "preemption": "pause",
"max_num_tokens": 640, "max_chunk_tokens": 512,
"priority_reserved_slots": 1, "max_pause_steps": 0, "warnings": [],
"compact_unavailable_reason": null
}
admitted_mid_flight counts requests admitted while other requests were
running, including a request admitted at the same step boundary as an earlier
one that has not run a step yet;
stalls_* count generation boundaries at which the queue head could not be
admitted and why; queued and resident are the live depths. refused is the
engine’s
own back-pressure and stays at zero while the HTTP gate returns 429 first.
context_cache is filled when --enable-context-reuse is on and null otherwise. Its counters
are cumulative since startup and current as of the engine’s last admission or
step; take differences between two reads to measure an interval.
The step counters cover every step that ran: decode_steps decoded every
running request (decode_graph_steps of them replayed a CUDA graph),
prompt_steps processed at least one prompt chunk (mixed_steps of them beside
decoding requests), decode_subset_steps decoded only some. step_sequences,
step_prompt_chunks, step_decode_rows, step_tokens and step_sampled_rows
are running sums; divide by the number of steps for the average step shape.
step_time_* is the scheduler’s host time per step,
paused_resident_steps sums the requests paused by --preemption pause over
steps, and compact_refusals counts the compact steps a plugin refused.
The engine block is null without in-flight batching. scheduling_mode and
the scheduling fields after it are the scheduling in effect, which a refused
compact step can change after startup (see Scheduling Modes):
scheduling_mode_source is requested when the configured mode is kept and
auto when --ifb-scheduling-mode auto chose it, in which case
scheduling_reason says why (it is null for a requested mode); preemption is the preemption
applied (none when auto dropped pause, and max_pause_steps is then 0), max_num_tokens and
max_chunk_tokens are the step budgets in effect (0 outside mixed), and
warnings lists the configuration the server accepted but does not apply as
given, as the startup log does. compact_unavailable_reason says why compact
execution is off. While the engine serves, /health returns HTTP 200 with
status: "healthy". Once it no longer serves and fails every request, it
returns HTTP 503 with status: "unhealthy", engine.serving: false and a
top-level reason. This happens after two session failures in a row with no
completed step between them (a failed step, or an admission that left the
device state unknown), when no new runtime session can be opened, when the
scheduling cannot be resolved again after a refused compact step, or after a
refused compact step under an explicit mixed; restart the server to
recover. A liveness probe on /health can therefore restart such a server.
/health/ready answers HTTP 200 with status: "ready" while the engine
serves, and HTTP 503 with status: "unavailable" and the same reason once it
does not, so a readiness probe takes the server out of rotation.
Errors#
Under in-flight batching, two failures return HTTP 503 instead of HTTP 500.
A client tells them apart by the error type in the OpenAI error body ({"error": {"type": ..., "message": ..., "code": 503}}), or in the SSE error event of a
stream that has already started:
|
When |
|
What to do |
|---|---|---|---|
|
A plugin refused a compact step after the request produced tokens; the server already serves again without compact steps |
|
Retry the request |
|
The engine no longer serves (see Reading |
none |
Restart the server; retrying does not help |
Any other engine failure stays HTTP 500 with type engine_error. On
/v1/messages the same statuses and Retry-After header apply, with the
Anthropic error body, whose type is api_error for both 503 responses; there
only Retry-After tells the retryable compact_step_refused failure apart.