LLM Request Context

class RequestOperations

Operations over Initialization-prepared bindings, staging buffers, sampler and contiguous KV storage. Implementations allocate nothing and finish fallible output transfers before returning success.

Subclassed by trt_edgellm::safety::ProcessingCapabilityCoordinator

Public Functions

virtual ~RequestOperations() = default

virtual SafetyStatus preparePrefillInputs(

LMStageContext const &context,

TokenIdView prompt

) noexcept = 0

virtual SafetyStatus prepareDecodeInputs(

LMStageContext const &context,

int32_t token,

int32_t kvLength

) noexcept = 0

virtual SafetyStatus selectGreedyTokenFromLogits(

safety::LMStageOutputs const &outputs,

int32_t &token

) noexcept = 0

virtual SafetyStatus commitKvCacheFromLMStage(

safety::LMStageOutputs const &outputs

) noexcept = 0

class RequestContext

Public Types

using LMStageOutputs = safety::LMStageOutputs

Public Functions

inline explicit RequestContext(

RuntimeData const &runtimeData

) noexcept

RequestContext(RequestContext const&) = delete

RequestContext &operator=(RequestContext const&) = delete

SafetyStatus beginRequest(TokenIdView prompt) noexcept

Prompt and RuntimeData remain valid and unchanged throughout this request.

SafetyStatus prepareLMStageInputs(

LMStageId stage,

LMStageContext const &context

) noexcept

SafetyStatus prepareLMStageOutputs(

LMStageId stage,

LMStageOutputs const &outputs,

PendingStep &step

) noexcept

void commitLMStageOutputs(PendingStep const &step) noexcept

Called once with the successful pending step; publishes host progress only.

int64_t generatedTokenCapacity() const noexcept

inline int32_t contextLength() const noexcept

struct PendingStep

Public Members

int32_t selectedTokenId = {-1}

int32_t kvLength = {0}

struct RuntimeData

Initialized after resource preparation; contexts and operation-owned resource views outlive requests. Capacities come from validated engine metadata. This view allocates and releases no resources.

Public Members

RequestOperations &operations

LMStageId prefillStage

LMStageId decodeStage

LMStageContext const &prefillContext

LMStageContext const &decodeContext

int32_t maxPrefillTokens

int32_t maxContextLength

int32_t vocabSize

struct PendingStep

Public Members

int32_t selectedTokenId = {-1}

int32_t kvLength = {0}