Engine Executor#

class EngineExecutor#

Engine execution interface with a prepare/execute split.

The production implementation owns a TRT runtime, engine, and execution context:

EngineExecutor knows nothing about models, phases, or features. It is abstract so callers can be exercised without a serialized engine on disk; the factories below are the only way to obtain the TRT-backed implementation.

Implementations divide into two roles. prepare/execute/captureGraph/setContextMemory/setProfiler/ getRequiredContextMemorySize run per request and a substitute must implement them. The introspection accessors (getEngine and the binding queries) run once, during startup validation, and a substitute may reject them.

Subclassed by trt_edgellm::rt::TrtEngineExecutor

Public Functions

virtual ~EngineExecutor() = default#
EngineExecutor(EngineExecutor const&) = delete#
EngineExecutor &operator=(EngineExecutor const&) = delete#
virtual bool prepare(
int32_t profileIndex,
InferenceDims const &dims,
TensorMap const &map,
cudaStream_t stream
) = 0#

Switch optimization profile, resolve shapes, bind all tensors.

Parameters:

  • profileIndex – TRT optimization profile index

  • dims – Symbolic dimension values for this step

  • map – Name-to-tensor mapping

  • stream – CUDA stream for the async profile switch

Returns:

True on success

virtual bool execute(cudaStream_t stream) = 0#

Execute inference.

Replays a cached CUDA graph if one matches the current bindings, otherwise falls back to enqueueV3.

Parameters:

stream – CUDA stream

Returns:

True on success

virtual bool captureGraph(cudaStream_t stream) = 0#

Capture a CUDA graph for the current binding state (after prepare()).

Performs a warmup enqueue, then captures via cudaStreamBeginCapture. The captured graph is keyed by a binding hash with full snapshot verification.

Parameters:

stream – CUDA stream (must not be the default stream)

Returns:

True if capture succeeded

virtual int64_t getRequiredContextMemorySize() const = 0#

Query required device memory for the execution context.

Returns:

Required memory size in bytes

virtual bool setContextMemory(Tensor &sharedMem) = 0#

Provide shared device memory for the execution context.

Parameters:

sharedMem – Tensor whose memory will back the TRT context

Returns:

True on success

virtual int32_t getNumIOTensors() const = 0#

Return the number of I/O tensors in the engine.

virtual char const *getIOTensorName(int32_t index) const = 0#

Return the name of the i-th I/O tensor.

virtual bool hasIOTensor(char const *name) const = 0#

Return whether the engine exposes a named I/O tensor.

virtual nvinfer1::DataType getBindingDataType(
char const *name
) const = 0#

Return the data type of a named binding.

virtual nvinfer1::Dims getProfileShape(
char const *name,
int32_t profileIndex,
nvinfer1::OptProfileSelector selector
) const = 0#

Return a profile shape (min/opt/max) for a named binding.

virtual void setProfiler(nvinfer1::IProfiler *profiler) noexcept = 0#

Attach a TRT profiler to the execution context.

The profiler receives per-layer timing callbacks during enqueueV3. Must be called before execute() for the profiler to receive data. Passing nullptr detaches any previously set profiler.

virtual nvinfer1::ICudaEngine const &getEngine() const noexcept = 0#

Access the underlying TRT engine for generic introspection.

Public Static Functions

static std::unique_ptr<EngineExecutor> createForLLM(
std::filesystem::path const &enginePath,
LLMEngineConfig const &cfg,
std::optional<int32_t> specDecodeBaseOutputHiddenDim = std::nullopt
)#

Build an EngineExecutor for a vanilla single-engine LLM or a SpecDecode base engine. The factory builds the TensorRegistry internally via buildRegistryForLLM(cfg).

static std::unique_ptr<EngineExecutor> createForDraft(
std::filesystem::path const &enginePath,
DeploymentConfig const &bundle
)#

Build an EngineExecutor for a speculative decoding draft engine. The factory chooses the draft binding registry from bundle.specDecodeMode().

struct BindingSnapshot#

Snapshot of binding addresses and shapes — used for graph-cache verification.

Public Functions

bool operator==(BindingSnapshot const &rhs) const noexcept#

Public Members

std::vector<std::pair<uintptr_t, nvinfer1::Dims>> bindings#
struct BindingSnapshot

Snapshot of binding addresses and shapes — used for graph-cache verification.

Public Functions

bool operator==(BindingSnapshot const &rhs) const noexcept

Public Members

std::vector<std::pair<uintptr_t, nvinfer1::Dims>> bindings