Qwen3vl ViT Runner#

class Qwen3VLViTRunner : public trt_edgellm::rt::QwenViTRunner#

Qwen3-VL / Qwen3.5 vision encoder. Extends the Qwen2-VL base with FAST-POS-EMBED + DEEPSTACK, a 3D temporal resize budget, and per-frame timestamped video sub-spans. Mirror of HF modeling_qwen3_vl.py.

Subclassed by trt_edgellm::rt::Cosmos3EdgeViTRunner, trt_edgellm::rt::Qwen3OmniViTRunner

Public Functions

virtual std::vector<std::reference_wrapper<rt::Tensor>> getDeepstackFeatures(
) override#

Get deepstack features for Qwen3-VL models.

Returns:

Writable raw features before embedding lookup; empty when the model has no deepstack

std::vector<std::reference_wrapper<rt::Tensor>> const &getOutputTensors(
) noexcept#

Mutable encoder outputs in order: embedding, then deepstack features.

virtual bool infer(cudaStream_t stream) noexcept override#

Run multimodal inference.

Parameters:

stream – CUDA stream

Returns:

True on success, false on failure

bool infer(cudaStream_t stream, int64_t outputTokenOffset) noexcept#

Encode a local shard at a token offset in every output’s existing allocation. Tensor shapes remain local; ordinary infer(stream) binds the outputs at offset zero.

QwenViTRunner(
std::string const &engineDir,
int32_t llmMaxBatchSize,
int32_t llmMaxSequenceLength,
cudaStream_t stream,
std::string const &checkpointDir = ""
)#

Constructor for QwenViTRunner.

Parameters:
  • engineDir – [in] Directory containing the TensorRT engine files

  • llmMaxBatchSize – [in] Maximum batch size from LLM engine

  • llmMaxSequenceLength – [in] Maximum sequence length from LLM engine

  • stream – [in] CUDA stream for execution

  • checkpointDir – [in] HF checkpoint directory used by runtime weight loading

Throws:
  • std::runtime_error – if engine directory does not contain engine files, or if buffer allocation fails

  • json::type_error – if JSON configuration contains unexpected datatypes

std::vector<int32_t> trt_edgellm::rt::qwen3vl::tokenizeExpandedPrompt(
std::string const &formattedPrompt,
std::vector<rt::imageUtils::ImageData> const &media,
std::vector<VisionSpan> const &spans,
size_t spanBegin,
size_t spanEnd,
QwenViTConfig const &config,
trt_edgellm::tokenizer::Tokenizer const &tokenizer
)#

Expand media placeholders (video timestamps included) before tokenizing so text and timestamps share the Hugging Face processor’s BPE boundaries, then expand each visual pad id to its span’s token count.