Engine Executor#

class EngineExecutor#

Engine execution interface with a prepare/execute split.

The production implementation owns a TRT runtime, engine, and execution context:

EngineExecutor knows nothing about models, phases, or features. It is abstract so callers can be exercised without a serialized engine on disk; the factories below are the only way to obtain the TRT-backed implementation.

Implementations divide into two roles. prepare/execute/captureGraph/setContextMemory/ getRequiredContextMemorySize run per request and a substitute must implement them. The introspection accessors (getEngine and the binding queries) run once, during startup validation, and a substitute may reject them.

Subclassed by trt_edgellm::rt::TrtEngineExecutor

Public Functions

virtual ~EngineExecutor() = default#
EngineExecutor(EngineExecutor const&) = delete#
EngineExecutor &operator=(EngineExecutor const&) = delete#
virtual bool prepare(
int32_t profileIndex,
InferenceDims const &dims,
TensorMap const &map,
cudaStream_t stream
) = 0#

Switch optimization profile, resolve shapes, bind all tensors.

Parameters:
  • profileIndex – TRT optimization profile index

  • dims – Symbolic dimension values for this step

  • map – Name-to-tensor mapping

  • stream – CUDA stream for the async profile switch

Returns:

True on success

virtual bool execute(cudaStream_t stream) = 0#

Execute inference.

Replays a cached CUDA graph if one matches the current bindings, otherwise falls back to enqueueV3.

Parameters:

stream – CUDA stream

Returns:

True on success

virtual bool captureGraph(cudaStream_t stream) = 0#

Capture a CUDA graph for the current binding state (after prepare()).

Performs a warmup enqueue, then captures via cudaStreamBeginCapture. The captured graph is keyed by a binding hash with full snapshot verification.

Parameters:

stream – CUDA stream (must not be the default stream)

Returns:

True if capture succeeded

virtual int64_t getRequiredContextMemorySize() const = 0#

Query required device memory for the execution context.

Returns:

Required memory size in bytes

virtual bool setContextMemory(Tensor &sharedMem) = 0#

Provide shared device memory for the execution context.

Parameters:

sharedMem – Tensor whose memory will back the TRT context

Returns:

True on success

virtual int32_t getNumIOTensors() const = 0#

Return the number of I/O tensors in the engine.

virtual char const *getIOTensorName(int32_t index) const = 0#

Return the name of the i-th I/O tensor.

virtual bool hasIOTensor(char const *name) const = 0#

Return whether the engine exposes a named I/O tensor.

virtual trt_edgellm::DataType getBindingDataType(
char const *name
) const = 0#

Return the data type of a named binding.

virtual nvinfer1::Dims getProfileShape(
char const *name,
int32_t profileIndex,
nvinfer1::OptProfileSelector selector
) const = 0#

Return a profile shape (min/opt/max) for a named binding.

virtual nvinfer1::Dims getTensorShape(char const *name) const = 0#

Return the engine’s build-time shape of a named input or output, with -1 for each dynamic dimension.

Unlike getProfileShape(), this also answers for outputs, whose static dimensions no profile records. An unknown name returns Dims{-1, {}} (nbDims == -1).

virtual nvinfer1::ICudaEngine const &getEngine() const noexcept = 0#

Access the underlying TRT engine for generic introspection.

inline virtual uint64_t graphLaunches() const noexcept#

execute() calls that replayed a captured CUDA graph, over the executor’s life.

inline virtual bool dedicateCompactContext(
int32_t profileIndex,
cudaStream_t stream
)#

Run every later phase-8 (compact) step of profileIndex on an execution context that never leaves that profile; every other step keeps the original context (see runsOnCompactContext).

TensorRT updates layer resources on the first enqueue after an optimization-profile switch, which costs milliseconds of host time on each switch when compact steps alternate with decode steps. The dedicated context shares this executor’s device memory, which is sound because every step runs in the order of one stream.

Throws:

std::runtime_error – when the dedicated context’s auxiliary streams cannot be created; the runtime then records compact execution as unavailable (LLMRankRuntime::prepareCompactExecution)

Returns:

True when phase-8 steps of profileIndex run on a dedicated context after the call; false when they keep the original context

inline virtual bool dedicateContextChunkContext(
int32_t profileIndex,
cudaStream_t stream
)#

Create the dedicated context on profileIndex (the compact one, when both exist) for the phase-2 context-chunk steps issued inside a ScopedContextChunkRouting; see runsOnContextChunkContext.

Jump-forward catch-up steps are context chunks that alternate with decode steps, so without this each one costs the original context two profile switches.

Throws:

std::runtime_error – when the dedicated context’s auxiliary streams cannot be created

Returns:

True when such steps run on a dedicated context after the call

inline virtual void setContextChunkRouting(bool enabled)#

Route the phase-2 context-chunk steps prepared while enabled to the dedicated context.

Public Static Functions

static std::unique_ptr<EngineExecutor> createWithRegistry(
std::filesystem::path const &enginePath,
TensorRegistry registry,
EngineRefitOptions refit = {}
)#

Build an executor for an internal engine with explicitly supplied tensor bindings. Optional refit ownership is consumed before context creation; its stream must be non-default.

static std::unique_ptr<EngineExecutor> createForLLM(
std::filesystem::path const &enginePath,
LLMEngineConfig const &cfg,
std::optional<int32_t> specDecodeBaseOutputHiddenDim = std::nullopt,
std::shared_ptr<RtxRuntimeCache> runtimeCache = nullptr,
EngineRefitOptions refit = {}
)#

Build an EngineExecutor for a vanilla single-engine LLM or a SpecDecode base engine. The factory builds the TensorRegistry internally via buildRegistryForLLM(cfg). Optional refit ownership is consumed before context creation; its stream must be non-default.

static std::unique_ptr<EngineExecutor> createForDraft(
std::filesystem::path const &enginePath,
DeploymentConfig const &bundle,
std::shared_ptr<RtxRuntimeCache> runtimeCache = nullptr,
EngineRefitOptions refit = {}
)#

Build an EngineExecutor for a speculative decoding draft engine. The factory chooses the draft binding registry from bundle.specDecodeMode().

struct BindingSnapshot#

Snapshot of binding addresses and shapes — used for graph-cache verification.

Public Functions

bool operator==(BindingSnapshot const &rhs) const noexcept#

Public Members

std::vector<std::pair<uintptr_t, nvinfer1::Dims>> bindings#
class ScopedContextChunkRouting#

Turns context-chunk routing on for its lifetime: one prepare()/execute() pair.

Public Functions

inline explicit ScopedContextChunkRouting(EngineExecutor &executor)#
inline ~ScopedContextChunkRouting()#
ScopedContextChunkRouting(ScopedContextChunkRouting const&) = delete#
ScopedContextChunkRouting &operator=(
ScopedContextChunkRouting const&
) = delete#
class ScopedContextChunkRouting

Turns context-chunk routing on for its lifetime: one prepare()/execute() pair.

Public Functions

inline explicit ScopedContextChunkRouting(EngineExecutor &executor)
inline ~ScopedContextChunkRouting()
ScopedContextChunkRouting(ScopedContextChunkRouting const&) = delete
ScopedContextChunkRouting &operator=(
ScopedContextChunkRouting const&
) = delete
struct BindingSnapshot

Snapshot of binding addresses and shapes — used for graph-cache verification.

Public Functions

bool operator==(BindingSnapshot const &rhs) const noexcept

Public Members

std::vector<std::pair<uintptr_t, nvinfer1::Dims>> bindings
bool trt_edgellm::rt::validateRaggedInferenceDims(
InferenceDims const &dims,
int32_t profileIndex,
bool allowSelectBeyondPhysicalTokens = false
) noexcept#

Validate cross-binding relations that TensorRT optimization profiles cannot express.

size_t trt_edgellm::rt::computeExecutionGraphKey(
uintptr_t engineIdentity,
int32_t profileIndex,
EngineExecutor::BindingSnapshot const &snapshot
)#