Engine Executor#
-
class EngineExecutor#
Engine execution interface with a prepare/execute split.
The production implementation owns a TRT runtime, engine, and execution context:
prepare()sets the optimization profile and delegates binding to TensorRegistry::bindAll().execute()replays a cached CUDA graph when the binding state matches, otherwise falls back to enqueueV3.captureGraph()captures a new CUDA graph for the current bindings.
EngineExecutor knows nothing about models, phases, or features. It is abstract so callers can be exercised without a serialized engine on disk; the factories below are the only way to obtain the TRT-backed implementation.
Implementations divide into two roles.
prepare/execute/captureGraph/setContextMemory/getRequiredContextMemorySizerun per request and a substitute must implement them. The introspection accessors (getEngineand the binding queries) run once, during startup validation, and a substitute may reject them.Subclassed by trt_edgellm::rt::TrtEngineExecutor
Public Functions
-
virtual ~EngineExecutor() = default#
-
EngineExecutor(EngineExecutor const&) = delete#
-
EngineExecutor &operator=(EngineExecutor const&) = delete#
- virtual bool prepare(
- int32_t profileIndex,
- InferenceDims const &dims,
- TensorMap const &map,
- cudaStream_t stream
Switch optimization profile, resolve shapes, bind all tensors.
- Parameters:
profileIndex – TRT optimization profile index
dims – Symbolic dimension values for this step
map – Name-to-tensor mapping
stream – CUDA stream for the async profile switch
- Returns:
True on success
-
virtual bool execute(cudaStream_t stream) = 0#
Execute inference.
Replays a cached CUDA graph if one matches the current bindings, otherwise falls back to enqueueV3.
- Parameters:
stream – CUDA stream
- Returns:
True on success
-
virtual bool captureGraph(cudaStream_t stream) = 0#
Capture a CUDA graph for the current binding state (after prepare()).
Performs a warmup enqueue, then captures via cudaStreamBeginCapture. The captured graph is keyed by a binding hash with full snapshot verification.
- Parameters:
stream – CUDA stream (must not be the default stream)
- Returns:
True if capture succeeded
-
virtual int64_t getRequiredContextMemorySize() const = 0#
Query required device memory for the execution context.
- Returns:
Required memory size in bytes
-
virtual bool setContextMemory(Tensor &sharedMem) = 0#
Provide shared device memory for the execution context.
- Parameters:
sharedMem – Tensor whose memory will back the TRT context
- Returns:
True on success
-
virtual int32_t getNumIOTensors() const = 0#
Return the number of I/O tensors in the engine.
-
virtual char const *getIOTensorName(int32_t index) const = 0#
Return the name of the i-th I/O tensor.
-
virtual bool hasIOTensor(char const *name) const = 0#
Return whether the engine exposes a named I/O tensor.
- virtual trt_edgellm::DataType getBindingDataType(
- char const *name
Return the data type of a named binding.
- virtual nvinfer1::Dims getProfileShape(
- char const *name,
- int32_t profileIndex,
- nvinfer1::OptProfileSelector selector
Return a profile shape (min/opt/max) for a named binding.
-
virtual nvinfer1::Dims getTensorShape(char const *name) const = 0#
Return the engine’s build-time shape of a named input or output, with -1 for each dynamic dimension.
Unlike getProfileShape(), this also answers for outputs, whose static dimensions no profile records. An unknown name returns
Dims{-1, {}}(nbDims == -1).
-
virtual nvinfer1::ICudaEngine const &getEngine() const noexcept = 0#
Access the underlying TRT engine for generic introspection.
-
inline virtual uint64_t graphLaunches() const noexcept#
execute() calls that replayed a captured CUDA graph, over the executor’s life.
- inline virtual bool dedicateCompactContext(
- int32_t profileIndex,
- cudaStream_t stream
Run every later phase-8 (compact) step of
profileIndexon an execution context that never leaves that profile; every other step keeps the original context (see runsOnCompactContext).TensorRT updates layer resources on the first enqueue after an optimization-profile switch, which costs milliseconds of host time on each switch when compact steps alternate with decode steps. The dedicated context shares this executor’s device memory, which is sound because every step runs in the order of one stream.
- Throws:
std::runtime_error – when the dedicated context’s auxiliary streams cannot be created; the runtime then records compact execution as unavailable (LLMRankRuntime::prepareCompactExecution)
- Returns:
True when phase-8 steps of
profileIndexrun on a dedicated context after the call; false when they keep the original context
- inline virtual bool dedicateContextChunkContext(
- int32_t profileIndex,
- cudaStream_t stream
Create the dedicated context on
profileIndex(the compact one, when both exist) for the phase-2 context-chunk steps issued inside a ScopedContextChunkRouting; see runsOnContextChunkContext.Jump-forward catch-up steps are context chunks that alternate with decode steps, so without this each one costs the original context two profile switches.
- Throws:
std::runtime_error – when the dedicated context’s auxiliary streams cannot be created
- Returns:
True when such steps run on a dedicated context after the call
-
inline virtual void setContextChunkRouting(bool enabled)#
Route the phase-2 context-chunk steps prepared while
enabledto the dedicated context.
Public Static Functions
- static std::unique_ptr<EngineExecutor> createWithRegistry(
- std::filesystem::path const &enginePath,
- TensorRegistry registry,
- EngineRefitOptions refit = {}
Build an executor for an internal engine with explicitly supplied tensor bindings. Optional refit ownership is consumed before context creation; its stream must be non-default.
- std::filesystem::path const &enginePath,
- LLMEngineConfig const &cfg,
- std::optional<int32_t> specDecodeBaseOutputHiddenDim = std::nullopt,
- std::shared_ptr<RtxRuntimeCache> runtimeCache = nullptr,
- EngineRefitOptions refit = {}
Build an EngineExecutor for a vanilla single-engine LLM or a SpecDecode base engine. The factory builds the TensorRegistry internally via
buildRegistryForLLM(cfg). Optional refit ownership is consumed before context creation; its stream must be non-default.
- std::filesystem::path const &enginePath,
- DeploymentConfig const &bundle,
- std::shared_ptr<RtxRuntimeCache> runtimeCache = nullptr,
- EngineRefitOptions refit = {}
Build an EngineExecutor for a speculative decoding draft engine. The factory chooses the draft binding registry from
bundle.specDecodeMode().
-
struct BindingSnapshot#
Snapshot of binding addresses and shapes — used for graph-cache verification.
Public Functions
-
bool operator==(BindingSnapshot const &rhs) const noexcept#
Public Members
-
std::vector<std::pair<uintptr_t, nvinfer1::Dims>> bindings#
-
bool operator==(BindingSnapshot const &rhs) const noexcept#
-
class ScopedContextChunkRouting#
Turns context-chunk routing on for its lifetime: one prepare()/execute() pair.
Public Functions
-
inline explicit ScopedContextChunkRouting(EngineExecutor &executor)#
-
inline ~ScopedContextChunkRouting()#
-
ScopedContextChunkRouting(ScopedContextChunkRouting const&) = delete#
- ScopedContextChunkRouting &operator=(
- ScopedContextChunkRouting const&
-
inline explicit ScopedContextChunkRouting(EngineExecutor &executor)#
-
class ScopedContextChunkRouting
Turns context-chunk routing on for its lifetime: one prepare()/execute() pair.
Public Functions
-
inline explicit ScopedContextChunkRouting(EngineExecutor &executor)
-
inline ~ScopedContextChunkRouting()
-
ScopedContextChunkRouting(ScopedContextChunkRouting const&) = delete
- ScopedContextChunkRouting &operator=(
- ScopedContextChunkRouting const&
-
inline explicit ScopedContextChunkRouting(EngineExecutor &executor)
-
struct BindingSnapshot
Snapshot of binding addresses and shapes — used for graph-cache verification.
Public Functions
-
bool operator==(BindingSnapshot const &rhs) const noexcept
Public Members
-
std::vector<std::pair<uintptr_t, nvinfer1::Dims>> bindings
-
bool operator==(BindingSnapshot const &rhs) const noexcept
- bool trt_edgellm::rt::validateRaggedInferenceDims(
- InferenceDims const &dims,
- int32_t profileIndex,
- bool allowSelectBeyondPhysicalTokens = false
Validate cross-binding relations that TensorRT optimization profiles cannot express.
- size_t trt_edgellm::rt::computeExecutionGraphKey(
- uintptr_t engineIdentity,
- int32_t profileIndex,
- EngineExecutor::BindingSnapshot const &snapshot