Multimodal Runner#

class MultimodalRunner#

Base class for multimodal vision-language model runners.

Provides interface for vision encoder processing in VLMs. Subclasses implement specific VLM architectures (Qwen-VL, InternVL, etc.).

Subclassed by trt_edgellm::rt::Gemma4AudioRunner, trt_edgellm::rt::Gemma4UnifiedAudioRunner, trt_edgellm::rt::Gemma4UnifiedVisionRunner, trt_edgellm::rt::Gemma4ViTRunner, trt_edgellm::rt::InternViTRunner, trt_edgellm::rt::NemotronOmniViTRunner, trt_edgellm::rt::Phi4MMViTRunner, trt_edgellm::rt::Qwen3OmniAudioRunner, trt_edgellm::rt::QwenViTRunner

Public Functions

MultimodalRunner() noexcept = default#

Default constructor.

MultimodalRunner(std::string const &engineDir, cudaStream_t stream)#

Construct multimodal runner.

Parameters:

  • engineDir – Directory containing engine files

  • stream – CUDA stream for operations

Throws:

std::runtime_error – If engine loading or initialization fails

virtual ~MultimodalRunner() noexcept = default#

Virtual destructor.

int64_t getRequiredContextMemorySize() const#

Get the required context memory size for this engine.

Note

Handles both visual and audio engines

Returns:

Required context memory size in bytes

bool setContextMemory(rt::Tensor &sharedContextMemory)#

Set shared context memory for the execution context.

Note

The tensor size must be >= getRequiredContextMemorySize(). Must be called before infer().

Note

Handles both visual and audio engines

Parameters:

sharedContextMemory – Tensor containing the shared device memory (must be on GPU)

Returns:

True on success, false if the tensor is too small

virtual bool preprocess(
rt::LLMGenerationRequest const &request,
std::vector<std::vector<int32_t>> &batchedInputIds,
tokenizer::Tokenizer const *tokenizer,
rt::OptionalOutputTensor mropeCosSinOut,
cudaStream_t stream,
bool imageOnly = false
) = 0#

Preprocess request with images and text.

Parameters:

  • request – Generation request with prompts and images

  • batchedInputIds – Output batched input token IDs

  • tokenizer – Tokenizer instance

  • mropeCosSinOut – Output MRope cos/sin cache. Required (has_value() == true) only for MRope-based multimodal runners (QwenViT, Qwen3OmniAudio), which write per-batch 3D position encodings into it. Pass std::nullopt when the base engine uses standard RoPE — runners with standard RoPE (InternViT, Phi4MMViT) do not read this parameter.

  • stream – CUDA stream

  • imageOnly – When true, only run image preprocessing (skip text tokenization and RoPE generation). Used for benchmarking where only the visual engine inputs need to be set up.

Returns:

True on success, false on failure

virtual bool preprocessSystemPrompt(
std::string const &systemPrompt,
tokenizer::Tokenizer const *tokenizer,
rt::OptionalOutputTensor mropeCosSinOut,
cudaStream_t stream
)#

Used for KVCache saving where we need to conduct the tokenization of the system prompt and generate ND-Rope parameters for the system prompt.

This function may be a no-op for some multimodal runners and only performs nontrivial work for some derived subclasses.

Parameters:

  • systemPrompt – System prompt text

  • tokenizer – Tokenizer instance

  • mropeCosSinOut – Output MRope cos/sin cache. Required only for MRope-based runners; otherwise pass std::nullopt. See preprocess for the full semantics.

  • stream – CUDA stream

Returns:

True on success, false on failure

virtual bool infer(cudaStream_t stream) = 0#

Run multimodal inference.

Parameters:

stream – CUDA stream

Returns:

True on success, false on failure

virtual rt::Tensor &getOutputEmbedding()#

Get output embeddings from vision encoder.

Returns:

Reference to output embedding tensor

virtual rt::OptionalInputTensors getDeepstackFeatures()#

Get deepstack features for Qwen3-VL models.

Returns:

Optional deepstack features vector (raw features before embedding lookup)

virtual bool validateAndFillConfig(std::string const &engineDir) = 0#

Validate and fill configuration from file.

Parameters:

engineDir – Path to engine directory

Returns:

True on success, false on failure

virtual bool allocateBuffer(cudaStream_t stream) = 0#

Allocate device buffers.

Returns:

True on success, false on failure

inline virtual multimodal::ModelType getModelType() const noexcept#

Get model type.

Returns:

Model type enum

inline metrics::MultimodalMetrics const &getMultimodalMetrics(
) const noexcept#

Get multimodal processing metrics.

Returns:

Multimodal metrics

void loadExternalWeights(
std::string const &engineDir,
std::string const &checkpointDir,
cudaStream_t stream
)#

Load this encoder’s externalized weights and bind them to its context.

A no-op when the encoder config lists neither external_weight_files nor checkpoint_weight_bindings, which is the case for engines that carry their weights inline. Call before the first infer(), and before initialize() on runners that have one, since it may enqueue the engine.

Parameters:

  • engineDir – Directory holding the encoder engine and its config.json

  • checkpointDir – HF checkpoint directory, empty when not supplied

  • stream – CUDA stream used during startup weight preparation

Throws:

std::runtime_error – If the weights cannot be loaded or do not match the engine’s input signature

Public Static Functions

static std::unique_ptr<MultimodalRunner> create(
std::string const &multimodalEngineDir,
int32_t llmMaxBatchSize,
int64_t llmMaxPositionEmbeddings,
cudaStream_t stream,
std::string const &checkpointDir = ""
)#

Create appropriate multimodal runner instance.

Factory method that detects model type and creates corresponding runner.

Parameters:

  • multimodalEngineDir – Directory containing multimodal engine files

  • llmMaxBatchSize – Maximum batch size from LLM engine

  • llmMaxPositionEmbeddings – Maximum position embeddings from LLM engine

  • stream – CUDA stream for operations

  • checkpointDir – HF checkpoint directory used by runtime weight loading

Throws:

std::runtime_error – If model type is unknown or runner creation fails

Returns:

Unique pointer to created runner