Qwen3vl ViT Runner#
-
class Qwen3VLViTRunner : public trt_edgellm::rt::QwenViTRunner#
Qwen3-VL / Qwen3.5 vision encoder. Extends the Qwen2-VL base with FAST-POS-EMBED + DEEPSTACK, a 3D temporal resize budget, and per-frame timestamped video sub-spans. Mirror of HF modeling_qwen3_vl.py.
Subclassed by trt_edgellm::rt::Cosmos3EdgeViTRunner, trt_edgellm::rt::Qwen3OmniViTRunner
Public Functions
- virtual std::vector<std::reference_wrapper<rt::Tensor>> getDeepstackFeatures(
Get deepstack features for Qwen3-VL models.
- Returns:
Writable raw features before embedding lookup; empty when the model has no deepstack
- std::vector<std::reference_wrapper<rt::Tensor>> const &getOutputTensors(
Mutable encoder outputs in order: embedding, then deepstack features.
-
virtual bool infer(cudaStream_t stream) noexcept override#
Run multimodal inference.
- Parameters:
stream – CUDA stream
- Returns:
True on success, false on failure
-
bool infer(cudaStream_t stream, int64_t outputTokenOffset) noexcept#
Encode a local shard at a token offset in every output’s existing allocation. Tensor shapes remain local; ordinary infer(stream) binds the outputs at offset zero.
- QwenViTRunner(
- std::string const &engineDir,
- int32_t llmMaxBatchSize,
- int32_t llmMaxSequenceLength,
- cudaStream_t stream,
- std::string const &checkpointDir = ""
Constructor for QwenViTRunner.
- Parameters:
engineDir – [in] Directory containing the TensorRT engine files
llmMaxBatchSize – [in] Maximum batch size from LLM engine
llmMaxSequenceLength – [in] Maximum sequence length from LLM engine
stream – [in] CUDA stream for execution
checkpointDir – [in] HF checkpoint directory used by runtime weight loading
- Throws:
std::runtime_error – if engine directory does not contain engine files, or if buffer allocation fails
json::type_error – if JSON configuration contains unexpected datatypes
- std::vector<int32_t> trt_edgellm::rt::qwen3vl::tokenizeExpandedPrompt(
- std::string const &formattedPrompt,
- std::vector<rt::imageUtils::ImageData> const &media,
- std::vector<VisionSpan> const &spans,
- size_t spanBegin,
- size_t spanEnd,
- QwenViTConfig const &config,
- trt_edgellm::tokenizer::Tokenizer const &tokenizer
Expand media placeholders (video timestamps included) before tokenizing so text and timestamps share the Hugging Face processor’s BPE boundaries, then expand each visual pad id to its span’s token count.