Runtime Session#

class SessionLostError : public std::runtime_error#: Thrown by RuntimeSession::admit when a failure inside the reservation – the context-cache lease, device work – leaves the session’s state unknown. The ordinary refusals (std::invalid_argument, std::runtime_error) modify nothing and the session goes on; after this one the session is not to be driven further, and every resident’s outcome is the caller’s to publish. The same contract a step’s ok == false carries, surfaced from admission.

class RuntimeSession#

What a scheduler drives: one runtime held open for the life of an engine, that sequences enter and leave one at a time while the ones already resident keep decoding.

The session owns what outlives any single request: the resident slot pool, the step counter, the ragged execution scratch, the context-cache lease container and the stream the work runs on. A request owns nothing here beyond its slot: it is admitted with admit(), executed by the steps the caller schedules with execute(), and leaves through a StepResult’s finished set when its slot is evicted. An empty session idles; it is not torn down and reopened between requests, so the first sequence into an empty session and the tenth into a full one take the same path.

What the session does not decide: which sequences take part in a step, and when. The caller emits the plan – one ScheduledSequenceDescriptor per participating resident – and the session checks it against the residents it holds, lowers it to the runtime’s execution batch, runs it and reports what happened. Under single-cohort execution a plan is either the context step of one just-admitted sequence or one decode step over every resident, in resident order; under compact execution a plan may combine prompt chunks of several sequences with decode rows (see execute()).

Abstract so the scheduler’s admission, cancellation and shutdown paths can be tested against a scripted session without a device; RankRuntimeSession is the one over an LLMRankRuntime.

Threading: one thread drives a session, the same thread that owns the runtime. Every method must be called from it.

Subclassed by trt_edgellm::rt::RankRuntimeSession

Public Functions

virtual ~RuntimeSession() = default#

virtual AdmissionResult admit( LLMGenerationRequest const &request, RequestId requestId, std::shared_ptr<std::atomic<bool> const> cancelToken = nullptr ) = 0#

Admit one request as a new resident.

Validation, chat templating / tokenization or the encoder preprocess, the generate-budget check, the context-cache lease (when reuse is on) and the seat commit. The sequence’s context step is not run here: the caller schedules it as the next execute() – it must be the next one, since the admission staging buffers stay valid only until the next admission’s preprocess. A second admit() before that step is a contract violation. Under compact execution (capabilities().compact) any admitted sequence may have an unfinished prompt, advanced by the prompt chunks the caller schedules in any later steps.

A request that must have the session to itself (see requiresExclusiveAdmission) is admitted only into an empty session and refuses company while resident; a caller admitting into or beside one violates the contract. Whether two requests may share the session at all – adapter, cache policies, and sampling parameters on a decoder that requires uniform sampling (requiresUniformSampling) – is the caller’s decision (BatchCompatibility), taken before this call; the session verifies the parameters agree and fails loudly when they do not.

Parameters:

cancelToken – The request’s cancellation bit, polled by the runtime before the sequence’s context step and at every step it takes part in; null when the caller cancels through the stream channel only.

Throws:

std::invalid_argument – for a request this session never serves (per-request LoRA, speech output, the legacy system-prompt capture, trajectory execution, more than one sequence). Nothing is modified.
std::runtime_error – when validation, templating, tokenization or the encoder preprocess fails with the session’s stream healthy, or when the prompt exceeds the KV capacity or its uncached part exceeds the engine’s input length (the EDGELLM_INPUT_TOO_LONG marker rides in the message). Nothing is modified.
SessionLostError – when the encoder preprocess fails leaving the session’s stream in error, when the reservation itself fails inside the runtime (a device failure in the context-cache lease), or when staging the reserved media fails on the device (a failed CUDA call or a stream in error). The session’s state is then unknown: the caller fails it, residents included, and does not drive it further.

virtual StepResult execute( std::vector<ScheduledSequenceDescriptor> const &plan ) = 0#

Run one step over the sequences named by plan.

The plan is checked against the residents this session holds – identities, work, committed lengths and query lengths must describe exactly what the runtime would execute – and then executed. A plan of every resident as kDecode with one query token each, in resident order, is a decode step. Otherwise, under single-cohort execution the plan is the one pending admission as kContext covering its whole not-yet-committed prompt; under compact execution it names any subset of residents in any order, each at most once, with pastLength equal to its committed length: a prompt-incomplete resident as kContext with 0 < queryLength <= its remaining prompt and selectLastTokenLogits exactly when the chunk ends the prompt, any other resident as kDecode with queryLength 1. Residents the plan does not name are neither read nor written. A plan naming a sequence the session does not hold, or describing lengths that disagree with the ledger, is a contract violation and leaves the session untouched.

An exclusive resident (see exclusiveResident()) keeps the single-cohort steps even under compact execution – a guided sequence, media under a visual-token pruner, media whose token layout only its encoder determines – so its prompt step must cover its whole uncommitted prompt in one plan of its own; admission bounds that prompt by S_in.

A result with ok == false means the runtime could not run the step; every resident’s outcome is then the caller’s to publish (it will not arrive through a later result), and the session is not to be driven further. compactRefused and failure tell a refused compact forward apart from the other failures.

Throws:: std::runtime_error – on a device failure inside the step, with the same meaning.

virtual LLMGenerationResponse materialize( BatchResult const &result, std::vector<std::string> const &stopStrings ) const = 0#: Convert a finished snapshot to a response: the tokenizer’s decode and the final stop-string trim, exactly as the blocking path assembles its own.

virtual std::vector<ResidentRef> residents() const = 0#: The refs of every live resident, in resident (execution) order.

inline virtual void copyResidents( std::vector<ResidentRef> &out ) const#: residents() written into out, reusing its capacity: the scheduler reads the set every step.

inline virtual void releaseForReadmission(ResidentRef ref)#

After a step failed with StepResult::compactRefused, before the session closes: release ref's stream channel unfinished, so its request can be admitted into another session.

Only for a resident whose prompt has not completed, which has pushed nothing to its channel. The resident stays until the session closes, which returns its context-cache lease like any other’s. A session that attaches no stream channel has nothing to release.

Throws:: std::runtime_error – when ref is not such a resident.

virtual bool idle() const noexcept = 0#: True when no sequence is resident. The session is not finished; it is waiting.

virtual bool exclusiveResident() const noexcept = 0#: True while a resident requires the session to itself (see admit()).

virtual bool requiresExclusiveAdmission( LLMGenerationRequest const &request ) const noexcept = 0#: Whether request would require the session to itself: a grammar-constrained request, one with a token callback, media under an active visual-token pruner, or, in a compact session, media whose token layout only its encoder determines. Pure; the caller asks before deciding to admit.

inline virtual bool requiresUniformSampling( LLMGenerationRequest const &request ) const noexcept#: Whether request would run on a decoder that takes one set of sampling parameters per batch – the deployment’s speculative decoder – so that it shares steps only with requests that sample alike (BatchCompatibility::samplingAgrees). Pure; the caller asks before deciding to admit. A session whose decoders all sample per sequence reports false.

inline virtual SessionCapabilities capabilities() const#: What this deployment can execute. Constant for the session’s life and across reopens until a step reports compactRefused. A session that does not override it reports single-cohort execution only.

inline virtual std::optional<ContextCacheMetrics> contextCacheMetrics( ) const noexcept#: The runtime’s context-cache counters, cumulative over the runtime’s life rather than this session’s; nullopt when the runtime has no context cache.

class RankRuntimeSession : public trt_edgellm::rt::RuntimeSession #

The RuntimeSession over one LLMRankRuntime. Opened by LLMRankRuntime::openSession(); holds the runtime’s one-request latch for its life, so the blocking handleRequest() path is refused until the session is destroyed.

Public Functions

~RankRuntimeSession() override#

RankRuntimeSession(RankRuntimeSession const&) = delete#

RankRuntimeSession &operator=(RankRuntimeSession const&) = delete#

virtual AdmissionResult admit( LLMGenerationRequest const &request, RequestId requestId, std::shared_ptr<std::atomic<bool> const> cancelToken = nullptr ) override#

Admit one request as a new resident.

Parameters:

Throws:

std::invalid_argument – for a request this session never serves (per-request LoRA, speech output, the legacy system-prompt capture, trajectory execution, more than one sequence). Nothing is modified.
std::runtime_error – when validation, templating, tokenization or the encoder preprocess fails with the session’s stream healthy, or when the prompt exceeds the KV capacity or its uncached part exceeds the engine’s input length (the EDGELLM_INPUT_TOO_LONG marker rides in the message). Nothing is modified.
SessionLostError – when the encoder preprocess fails leaving the session’s stream in error, when the reservation itself fails inside the runtime (a device failure in the context-cache lease), or when staging the reserved media fails on the device (a failed CUDA call or a stream in error). The session’s state is then unknown: the caller fails it, residents included, and does not drive it further.

virtual StepResult execute( std::vector<ScheduledSequenceDescriptor> const &plan ) override#

Run one step over the sequences named by plan.

Throws:: std::runtime_error – on a device failure inside the step, with the same meaning.

virtual LLMGenerationResponse materialize( BatchResult const &result, std::vector<std::string> const &stopStrings ) const override#: Convert a finished snapshot to a response: the tokenizer’s decode and the final stop-string trim, exactly as the blocking path assembles its own.

virtual std::vector<ResidentRef> residents() const override#: The refs of every live resident, in resident (execution) order.

virtual void copyResidents( std::vector<ResidentRef> &out ) const override#: residents() written into out, reusing its capacity: the scheduler reads the set every step.

virtual void releaseForReadmission(ResidentRef ref) override#

After a step failed with StepResult::compactRefused, before the session closes: release ref's stream channel unfinished, so its request can be admitted into another session.

Throws:: std::runtime_error – when ref is not such a resident.

virtual bool idle() const noexcept override#: True when no sequence is resident. The session is not finished; it is waiting.

virtual bool exclusiveResident() const noexcept override#: True while a resident requires the session to itself (see admit()).

virtual bool requiresExclusiveAdmission( LLMGenerationRequest const &request ) const noexcept override#: Whether request would require the session to itself: a grammar-constrained request, one with a token callback, media under an active visual-token pruner, or, in a compact session, media whose token layout only its encoder determines. Pure; the caller asks before deciding to admit.

virtual bool requiresUniformSampling( LLMGenerationRequest const &request ) const noexcept override#: Whether request would run on a decoder that takes one set of sampling parameters per batch – the deployment’s speculative decoder – so that it shares steps only with requests that sample alike (BatchCompatibility::samplingAgrees). Pure; the caller asks before deciding to admit. A session whose decoders all sample per sequence reports false.

virtual SessionCapabilities capabilities() const override#: What this deployment can execute. Constant for the session’s life and across reopens until a step reports compactRefused. A session that does not override it reports single-cohort execution only.

virtual std::optional<ContextCacheMetrics> contextCacheMetrics( ) const noexcept override#: The runtime’s context-cache counters, cumulative over the runtime’s life rather than this session’s; nullopt when the runtime has no context cache.

struct AdmissionResult#

What became of one admission.

Public Types

enum class Status : uint8_t#

Values:

enumerator kAdmitted#

enumerator kNoCapacity#: Transient pressure; nothing was modified, retry at a later step.

enumerator kRejected#: The session rejects the request outright; nothing was modified.

Public Members

Status status = {Status::kRejected}#

ResidentRef ref#: valid iff kAdmitted

std::string reason#: diagnostic for kRejected

int32_t promptLength = {0}#: The prompt as the runtime tokenized it (chat template applied, media expanded), and the prefix of it that already has state behind it (a context-cache hit; zero when cold). The sequence’s context step covers exactly [prefillStart, promptLength).

int32_t prefillStart = {0}#

struct TokenDelta#

Tokens one resident produced in one step.

Public Members

std::vector<int32_t> tokenIds#

struct StepResult#

What one executed step did. The single commit unit: everything in it happened together, and nothing outside it happened.

Public Members

bool ok = {false}#

SequenceWork work = {SequenceWork::kContext}#: The work of a single-cohort step. A compact step names each sequence’s work in execution[i].work and reports kContext here when any sequence ran a prompt chunk, kDecode otherwise.

StepId stepId = {0}#: Runtime-assigned identity of the step whose effects this result commits. Zero when no step ran: a seated prefill that fails before it reaches the engine files its sequence under finished and executes nothing.

std::vector<SequenceIdentity> participants#: The sequences the plan named, in plan order: what the caller asked this step to run.

std::vector<ScheduledSequenceDescriptor> execution#: What the runtime executed for them, in the same order, and empty when no step ran. A descriptor’s pastLength is the positions the sequence had state for before the step and its queryLength the positions the step added – which a visual-token pruner makes fewer than the prompt range the plan asked a context step to cover. The caller’s ledger advances from this, never from its own plan, so it mirrors what ran rather than what was requested.

std::vector<std::pair<ResidentRef, TokenDelta>> deltas#: Tokens produced this step, per participating resident (absent for a resident that produced none).

std::vector<std::pair<ResidentRef, BatchResult>> finished#: Residents that reached a terminal state and were evicted this step, with the finished sequence snapshot; their physical resident slots are released (and their refs invalidated) as part of this commit unit. A seat that failed its context step surfaces here on the step whose eviction files it, not before. Dense-row compaction is runtime-private and never reported: refs are stable resident identities, so the caller’s table needs no re-keying.

bool publishedPrefix = {false}#: True when this step published context-cache prefix blocks (cache visibility commits with the step that produced it, never ahead of it).

bool graphLaunched = {false}#: True when the base engine ran this step from a captured CUDA graph rather than an eager enqueue.

bool compact = {false}#: True when the step ran as one compact (phase-8) forward; false for a single-cohort context step and for the phase-3 decode step of every resident.

bool compactRefused = {false}#: True when ok is false because a plugin refused the compact forward with the device healthy. Compact execution is then unavailable for the runtime’s life: every session opened afterwards reports capabilities().compact false. The residents still fail – the layers before the refusing one advanced their recurrent state in place – but their requests can be retried.

std::string failure#: Why ok is false, complete for display; empty when the runtime reports no specific cause.

struct SessionCapabilities#

What the deployment behind a session can execute, and the engine limits a scheduler derives its step budgets from. Fixed for the life of the runtime, except that a refused compact step (StepResult::compactRefused) makes compact execution unavailable from then on.

Public Members

bool compact = {false}#: Whether a step may combine prefill chunks of several sequences with decode rows (phase 8).

std::string reason#: Why compact execution is unavailable, complete for display. Empty when compact execution is available.

int32_t maxBatchSize = {0}#: The batch size and input length the engine was built with, and the KV page size.

int32_t maxInputLength = {0}#

int32_t kvPageSize = {kTOKENS_PER_PAGE}#