LLM Runner

class LlmRunner : public trt_edgellm::rt::multiStage::IStage

The backbone decode path, behind IStage.

An adapter: the decode loop, KV management and sampling stay in LLMInferenceRuntime. What lives here is the part the topology owns – admission, cancellation, and publishing the prefill window to whoever subscribed to it.

Nothing here is tied to one model family: whether a deployment runs this as the speaking topology’s thinker or as a text-only backbone is a stage id and a set of config values, never a branch. Whether hidden states are published at all follows from the ports the deployment declares.

Config keys (all optional; paths are relative to artifact). The deployment keys mirror llm_inference’s flags of the same meaning: multimodal_artifact perception engine directory checkpoint checkpoint directory for runtime-loaded weights draft_checkpoint separate checkpoint of a paired speculative draft accept_hidden_layer decoder layer this engine emits hidden states on; -1, the default, is the post-norm output max_generate_length decode cap; defaults to 256 temperature defaults to 1.0 top_p defaults to 0.8 top_k defaults to 50 spec_decode {draft_top_k, draft_step, proposal_tokens, verify_tokens, dspark_w2_backend, dspark_scheduler, dspark_confidence_threshold, dspark_min_proposal_len, dspark_max_proposal_len}; present only for a speculative build, which emits no hidden port, as in llm_inference. An absent count is resolved from the engines. lora_adapters {name: path} LoRA weights a request may select context_cache {enable_reuse, max_records, max_snapshots, encoder_cache_budget_bytes} visual_pruning {algorithm, reduction_ratio, min_visual_tokens, pivot_image_tokens, pivot_text_tokens}; present to enable ple {file, residency, cache_slots} runtime_cache TRT-RTX runtime cache file offload_embedding keep the embedding table in pinned host memory; defaults to false A request’s sampling input overrides the four sampling keys for that request.

Public Types

using RuntimeFactory = std::function<std::unique_ptr<LLMInferenceRuntime>(DeploymentOptions const &deployment, cudaStream_t stream)>

Builds the runtime preallocate() drives; the default loads the engines the config names.

Public Functions

explicit LlmRunner(multiStage::StageInitContext const &init)

LlmRunner(

multiStage::StageInitContext const &init,

RuntimeFactory makeRuntime

)

~LlmRunner() override

virtual void preallocate() override

Init-time allocation. Every device buffer this stage will ever use is claimed here, sized by maxBatch x maxSeq. Nothing on the step path may allocate.

virtual bool admit(

multiStage::StageExecContext const &ctx,

std::string &reason

) override

Per-request admission. Reject here – it costs nothing – rather than failing mid-step.

Returns:

false plus a reason when this stage cannot serve the request.

virtual void prepare(multiStage::StageExecContext &ctx) override

Per-request setup, once the inputs this stage reads are published. A throw fails only this request, so what prepare() writes must be this member’s own or scratch that the next member’s prepare() rewrites.

virtual multiStage::StageStatus step(

multiStage::StageExecContext &ctx

) override

void stepCohort(

std::vector<multiStage::StageExecContext*> const &members,

std::vector<multiStage::StageStatus> &statuses

) override

virtual void finish(

multiStage::StageExecContext &ctx,

multiStage::FinishReason reason

) override

Called for every member on every active stage when run() ends: success, failure and cancellation alike. Emits the terminating chunk and reports metrics. A stage that only cleans up on success is a leak. take() without run() skips it; see reset().

virtual void reset() override

Idempotent. Drops per-request state; anything longer-lived survives.

virtual nlohmann::json diagnostics(

multiStage::RequestSlot const &slot

) const override

What this stage did on slot's request, for that caller’s report.

Read after finish() and before reset(). Keys are the stage’s own; the runtime nests them under the stage id and never interprets them. An empty object means the stage has nothing to say, which is the default. Taking the slot is what keeps a cohort’s members from all reporting the same numbers. A throw fails only that request, which keeps the other stages’ reports.

This is deliberately not gated on profiling: a caller needs to know why generation stopped even when no timing was collected.

inline LLMInferenceRuntime const &runtime() const noexcept

The runtime behind this stage, for its aggregate prefill, generation and encoder metrics. Valid once preallocate() has run.

std::unique_ptr<multiStage::IStage> trt_edgellm::rt::createLlmRunner(

multiStage::StageInitContext const &init

)

multiStage::RunnerDecl trt_edgellm::rt::llmRunnerDeclaration()

What an llm.backbone stage offers a stage graph.