Gemma4 Unified Vision Runner

class Gemma4UnifiedVisionRunner : public trt_edgellm::rt::MultimodalRunner

Preprocess RGB images directly into 48x48 patches and run the lightweight Unified visual graph.

Public Functions

Gemma4UnifiedVisionRunner(

std::string const &engineDir,

cudaStream_t stream,

)

~Gemma4UnifiedVisionRunner() noexcept = default

virtual bool preprocess(

rt::LLMGenerationRequest const &request,

std::vector<std::vector<int32_t>> &batchedInputIds,

tokenizer::Tokenizer const *tokenizer,

rt::OptionalOutputTensor mropeCosSinOut,

cudaStream_t stream,

bool imageOnly = false,

) noexcept override

Preprocess request with images and text.

Parameters:

• request – Generation request with prompts and images

• batchedInputIds – Output batched input token IDs

• tokenizer – Tokenizer instance

• mropeCosSinOut – Output MRope cos/sin cache. Required (has_value() == true) only for MRope-based multimodal runners (QwenViT, Qwen3OmniAudio), which write per-batch 3D position encodings into it. Pass std::nullopt when the base engine uses standard RoPE — runners with standard RoPE (InternViT, Phi4MMViT) do not read this parameter.

• stream – CUDA stream

• imageOnly – When true, only run image preprocessing (skip text tokenization and RoPE generation). Used for benchmarking where only the visual engine inputs need to be set up.

Returns:

True on success, false on failure

virtual bool infer(cudaStream_t stream) noexcept override

Run multimodal inference.

Parameters:

stream – CUDA stream

Returns:

True on success, false on failure

virtual bool validateAndFillConfig(

std::string const &engineDir,

) override

Validate and fill configuration from file.

Parameters:

engineDir – Path to engine directory

Returns:

True on success, false on failure

virtual bool allocateBuffer(cudaStream_t stream) override

Allocate device buffers.

Returns:

True on success, false on failure

struct Gemma4UnifiedVisionConfig

Runtime configuration for the encoder-free Gemma4 Unified visual embedder.

Public Members

int64_t minPatches = {0}

int64_t maxPatches = {0}

int64_t maxPatchesPerImage = {0}

int64_t inputDim = {0}

int64_t outputHiddenSize = {0}

int64_t modelPatchSize = {48}

int64_t positionEmbeddingSize = {0}

int32_t imageTokenId = {0}

int32_t beginImageTokenId = {0}

int32_t endImageTokenId = {0}