Request Engine#
-
class SubmitError : public std::runtime_error#
Thrown by submit() when the request is refused outright rather than queued.
Refusal is not an error condition of the engine: a bounded queue is the backpressure signal, and the caller is expected to translate this into whatever its transport calls “busy”.
Public Functions
-
inline explicit SubmitError(std::string const &message)#
-
inline explicit SubmitError(std::string const &message)#
-
class RequestFailedError : public std::runtime_error#
Thrown by RequestHandle::get() for a request that ended in TerminalStatus::kExecutionError.
-
class RequestHandle#
A caller’s claim on one in-flight request. Move-only: a channel has exactly one consumer, and copying the handle would quietly create a second one.
Public Functions
-
RequestHandle() = default#
- std::shared_ptr<EngineChannel> channel,
- std::shared_ptr<RequestRecord> record
-
RequestHandle(RequestHandle&&) noexcept#
-
RequestHandle &operator=(RequestHandle&&) noexcept#
-
RequestHandle(RequestHandle const&) = delete#
-
RequestHandle &operator=(RequestHandle const&) = delete#
-
~RequestHandle() = default#
-
RequestId id() const noexcept#
-
StreamChannel &stream()#
The token channel for this request, as the runtime writes it. Valid while the handle is.
Tokens appear here as they are produced. To cancel, use RequestHandle::cancel() or RequestEngine::cancel(): cancelling the channel directly also stops generation at the next step boundary, but a get() waiting on this request is only released once the actor reports the outcome.
The same channel with shared ownership, for callers that outlive the handle.
The pybind layer holds StreamChannel through a shared_ptr holder, so handing it the reference above would manufacture a second owner of the same object. Anything else should prefer stream().
-
LLMGenerationResponse get()#
Block until the request reaches a terminal state, then hand back its response.
- Throws:
RequestFailedError – if the request ended in kExecutionError
std::runtime_error – if the request was cancelled
-
bool ready() const noexcept#
True once the request has an outcome, so get() would return without blocking.
Distinct from the channel being finished, which reports only that the token stream ended.
-
void cancel() noexcept#
Ask the engine to stop this request. Non-blocking and idempotent; a request that has already retired simply does not match anything the actor still holds.
Releases this caller immediately, but does not interrupt a step the runtime is already inside: the request stops at the next step boundary and is then reported as cancelled.
-
inline bool valid() const noexcept#
-
std::optional<RequestStats> stats() const#
The request’s scheduling statistics, once ready(); empty before that.
-
RequestHandle() = default#
-
class RequestEngine#
Owns the runtime and the one thread allowed to touch it.
submit() and cancel() are callable from any thread: each posts a command to the queue and returns without waiting for the actor. shutdown() is the exception – it joins, so it blocks and must not be called from the actor itself. The actor starts in the constructor and is joined by shutdown(), which the destructor implies, so there is no start() to forget.
The actor opens one RuntimeSession for its life and drives it one step at a time: at every step boundary it folds in the command queue, admits at most one queued request (queue head in the configured order, FCFS by default, gated by the residents’ compatibility, the session’s exclusivity rules and the reserved slots), then runs the pending admission’s context step or one decode step over every resident. Under compact execution (ResolvedSchedulingPolicy::compactExecution) it instead admits every queue head that can join, in the same order and under the same gates, and runs the step StepPlanner composes from the residents’ ledger: prompt chunks of several residents beside decode rows. Requests leave through the step that evicts them; an empty session idles until the next submit. Feeding the GPU has to be serial anyway, so that is what the actor serialises: callers never block on each other, and no caller ever holds a pointer to the runtime, so a data race on it cannot be written rather than being prevented by a lock the caller has to remember to take.
Public Types
-
using SessionFactory = std::function<std::unique_ptr<RuntimeSession>(cudaStream_t)>#
How the actor opens its runtime session. In production this is bound to LLMInferenceRuntime::openSession. It is a seam rather than a direct call so the actor’s admission, scheduling, cancellation and shutdown paths can be tested without a GPU: LLMInferenceRuntime is a concrete class and needs a real CUDA stream to assemble, which would put every one of those tests behind a device. A null return means the runtime refused; a throw carries the reason. Called on the actor’s thread, at start and after a failed step.
Public Functions
- RequestEngine(
- std::unique_ptr<LLMInferenceRuntime> runtime,
- cudaStream_t stream,
- EngineConfig config = {}
- Parameters:
runtime – Taken over outright. Nothing else may hold or call it afterwards.
stream – The CUDA stream the actor drives the runtime on.
- Throws:
std::runtime_error – if
runtimespans more than one rank (the session runs on a single rank in this release), cannot seat sequences one at a time (a diffusion backbone, bounded SWA storage, a context cache without live admission), or refuses to open the session.std::invalid_argument – if the scheduling policy cannot be served by this deployment (see resolveSchedulingPolicy).
-
RequestEngine(SessionFactory openSession, EngineConfig config)#
Test seam: an engine with no runtime, over whatever session
openSessionhands back.- Throws:
std::runtime_error – if the factory refuses the first session.
std::invalid_argument – if the scheduling policy cannot be served by the session.
-
~RequestEngine()#
-
RequestEngine(RequestEngine const&) = delete#
-
RequestEngine &operator=(RequestEngine const&) = delete#
-
RequestEngine(RequestEngine&&) = delete#
-
RequestEngine &operator=(RequestEngine&&) = delete#
-
RequestHandle submit(LLMGenerationRequest request)#
Any thread: hand a request to the actor. Does not wait for it to start.
- Throws:
SubmitError – if the engine is shutting down or the command queue is full
std::invalid_argument – for a request the engine never serves (see the IFB support matrix): per-request LoRA, speech output, the legacy system-prompt capture, trajectory execution, more than one sequence per request.
-
void cancel(RequestId id)#
Any thread: cancel a request by id. Same effect and same guarantees as RequestHandle::cancel(): the request’s cancel token is set immediately on the caller’s thread and nothing rides the command queue, so it cannot be lost under overload. Idempotent; unknown or retired ids are ignored (ids are never reused).
- bool handleRequest(
- LLMGenerationRequest const &request,
- LLMGenerationResponse &response
Submit one request and block until it finishes, in the shape of the old blocking call.
Exists so a caller written against LLMInferenceRuntime::handleRequest can move onto the engine without being restructured first: it reports failure by returning false rather than throwing, and leaves
responseuntouched on failure.This is a convenience over submit() plus RequestHandle::get(), not a second execution path. It gives up everything the engine exists to provide – the caller occupies a thread for the whole generation and cannot see tokens as they arrive – so new code should submit instead.
- Returns:
false if the request was refused, cancelled, or failed during execution.
-
void shutdown(ShutdownMode mode)#
Any thread except the actor’s: stop the actor and join it. Idempotent.
Unlike the other operations this one blocks, because it joins. kDrain lets every queued request run; kCancel abandons what is queued and what is resident. Neither can interrupt a step the runtime is inside – the actor is in it and cannot reach a boundary – so both wait at least that long. Both leave every live stream terminal, so no caller stays parked in waitPop().
The first caller’s mode wins; a later kCancel cannot escalate a drain already under way.
-
int32_t queued() const noexcept#
Requests the actor has taken off the queue but not yet admitted. Observability only.
Not a count of everything submitted: a request stays invisible here until the actor reaches a boundary and drains it, which it cannot do while it is inside a step. Do not use this to wait for a submission to become visible.
-
int32_t resident() const noexcept#
Requests currently resident in the session. Observability only.
-
bool running() const noexcept#
False once shutdown() has been called.
-
EngineMetrics metrics() const noexcept#
Counters for observability; see EngineMetrics for the consistency contract.
-
std::optional<ContextCacheMetrics> contextCacheMetrics() const#
The runtime’s context-cache counters as the actor last copied them: once when the session opens, then after every step, before the step’s outcomes are published, so a caller released by a request’s outcome already sees the step that finished it. Observability only; nullopt when the runtime has no context cache.
The coordinator behind them is the actor’s to touch, so the copy is all another thread can see.
-
ResolvedSchedulingPolicy resolvedPolicy() const#
Any thread: the scheduling policy in effect.
Resolved at construction. When a plugin refuses a compact step under any mode but an explicit mixed, a fresh session opens without compact execution, and the policy is resolved again against it: auto becomes prefillFirst over single-cohort steps, dropping pause with the startup’s warning. Residents that have sampled a token fail with a retryable reason; the others go back to the queue at their (priority, arrival) place, keeping their handles and streams, and run again from the start of their prompts. An explicit mixed instead fails every resident, stops the engine (see status()) and keeps the policy it had.
-
EngineStatus status() const#
Any thread: whether the engine still serves requests, and why compact execution is unavailable.
-
using SessionFactory = std::function<std::unique_ptr<RuntimeSession>(cudaStream_t)>#
-
struct EngineConfig#
Public Members
-
int32_t commandQueueCapacity = {64}#
Commands the queue holds before submit() starts refusing. Rounded up to a power of two and to CommandQueue::kMinCapacity.
-
int32_t maxPendingRequests = {256}#
Requests accepted but not yet terminal, across the queue and the actor’s own backlog. Bounding the command queue alone does not bound this: each boundary moves a queue’s worth of commands into actor-owned state, so without this the backlog grows without limit and the backpressure the bounded queue is supposed to provide never materialises.
-
int32_t maxBatchSize = {1}#
Requests allowed to be resident at once.
Arriving requests join the session at step boundaries (in-flight batching), up to this many; beyond it a request waits in the queue for a resident to leave. This is also the KV capacity bound: page tables partition their pool per slot, so a request beyond this count has no pages to run on – it waits, no eviction of others required. May not exceed the batch size the engine was built with.
-
int32_t maxConsecutiveSessionFailures = {2}#
Session failures in a row – each retiring every resident, closing the session and opening a fresh one – with no step completed between them, after which the engine stops reopening and every later request fails at admission with the last reason. Opening a session hardly touches the device, so a reopen succeeds even when the failure’s cause is sticky (a device error the runtime cannot clear); without this bound every later request would pay one failed step and a reopen, forever. A refused compact step (StepResult::compactRefused) is not counted: its cause cannot repeat, because every later session reports no compact execution.
-
std::chrono::milliseconds idleWait = {50}#
How long an idle actor sleeps before waking to look around again. It does not affect a busy engine, which never reaches the park at all.
Every operation that gives the actor something to do also wakes it, so this timeout is a backstop rather than the mechanism: it bounds the damage if a future caller changes actor state without waking it, turning a permanent hang into a delay of this length.
-
SchedulingOptions scheduling#
In-flight batching scheduling, resolved against the deployment when the session opens; see resolveSchedulingPolicy() for the defaults and constraints and RequestEngine::resolvedPolicy() for the outcome.
-
int32_t commandQueueCapacity = {64}#
-
struct EngineMetrics#
A point-in-time copy of the engine’s counters. Each field is read with relaxed ordering, so the snapshot is per-field accurate but not a single atomic cut – fine for observability, not for control flow.
Public Members
-
uint64_t submitted = {}#
Requests submit() accepted / refused (queue full, backlog full, or shutting down).
-
uint64_t refused = {}#
-
uint64_t stallsIncompatible = {}#
Step boundaries at which the queue head could not be admitted, by reason. High stall counts quantify the cost of FCFS head blocking under a mixed workload.
incompatible: its batch-wide parameters differ from the residents’ (BatchCompatibility).
guided: it is grammar-constrained, or a constrained resident holds the session.
exclusive: it needs the session to itself for another reason (media under a visual-token pruner), or such a resident holds the session.
noCapacity: the runtime refused the admission as transient pressure (no free pages).
-
uint64_t stallsGuided = {}#
-
uint64_t stallsExclusive = {}#
-
uint64_t stallsNoCapacity = {}#
-
uint64_t admittedMidFlight = {}#
Requests admitted while other residents were decoding (excludes those that found the session empty).
-
uint64_t completed = {}#
Running totals of terminal outcomes across all requests, one per TerminalStatus value: countOutcome() bumps exactly one of them each time a request retires, so they always sum to the outcomes callers saw. A single request’s own state is the TerminalStatus on its record, not anything here.
-
uint64_t cancelled = {}#
-
uint64_t failed = {}#
-
uint64_t queueLatencyTotalUs = {}#
Time from submit() to the actor admitting the request, once per request: a request queued again after a refused compact step is not counted at its second admission.
-
uint64_t queueLatencyMaxUs = {}#
-
uint64_t queueLatencyCount = {}#
-
uint64_t decodeSteps = {}#
Counted over steps that completed and ran at least one sequence. Steps by kind:
decodeSteps: every resident left after the step’s cancellations decoded one token (phase 3), of which decodeGraphSteps ran the base engine from a captured CUDA graph.
promptSteps: at least one prompt chunk ran (a single-cohort context step, or a compact step with Np >= 1), of which mixedSteps also ran decode rows.
decodeSubsetSteps: only decode rows, with some resident left out (compact, profile 1).
-
uint64_t decodeGraphSteps = {}#
-
uint64_t promptSteps = {}#
-
uint64_t mixedSteps = {}#
-
uint64_t decodeSubsetSteps = {}#
-
uint64_t stepSequences = {}#
Running sums over the same steps; divide by the step count for averages.
stepSequences: N, sequences the step ran.
stepPromptChunks: Np, prompt chunks; partialPromptChunks of them did not end their prompt.
stepDecodeRows: Nd, decode rows.
stepTokens: T, query tokens.
stepSampledRows: L, sequences whose last token was sampled.
-
uint64_t stepPromptChunks = {}#
-
uint64_t partialPromptChunks = {}#
-
uint64_t stepDecodeRows = {}#
-
uint64_t stepTokens = {}#
-
uint64_t stepSampledRows = {}#
-
uint64_t stepTimeTotalUs = {}#
Actor host time of those steps, from planning through the commit of their results.
-
uint64_t stepTimeMaxUs = {}#
-
uint64_t pausedResidentSteps = {}#
Residents withheld by preemption=pause, summed over completed steps.
-
uint64_t compactRefusals = {}#
Compact steps a plugin refused (StepResult::compactRefused); see RequestEngine::resolvedPolicy() for what became of the residents and RequestEngine::status() for what the engine did next.
-
uint64_t submitted = {}#
-
struct EngineStatus#
Whether the engine still serves requests, for health reporting. Read with RequestEngine::status().
Public Members
-
bool serving = {true}#
False once the engine has no session and opens none again: the failure limit was reached, a session could not be reopened, or a plugin refused a compact step under an explicit mixed mode. Every later request then fails at admission with unavailableReason until the engine is recreated.
Why the engine stopped serving; empty while it serves.
Why the current session cannot execute compact steps, as its capabilities state it, or the refusal that stopped the engine; empty when compact execution is available.
-
bool serving = {true}#