MultimodalConfig#

class tensorrt_llm.llmapi.MultimodalConfig(
*,
encoder_cuda_graph: dict[str, MultimodalEncoderCudaGraphConfig] | None = None,
encoder_side_stream_max_ahead: Annotated[int, Ge(ge=0)] = 0,
encoder_cache_max_bytes: Annotated[int, Ge(ge=0)] = 134217728,
encoder_scheduling_policy: MultimodalEncoderSchedulingPolicy = MultimodalEncoderSchedulingPolicy.DEFAULT,
video_pruning_rate: Annotated[float | None, Ge(ge=0.0), Lt(lt=1.0)] = None,
)[source]#

Bases: StrictBaseModel

Multimodal model configuration.

field encoder_cache_max_bytes: Annotated[int, Ge(ge=0)] = 134217728#

Target capacity in bytes for reusable multimodal encoder embeddings; 0 disables cross-request reuse. The capacity applies per rank; ranks do not share a cache. With item scheduling, encoder outputs still use the model-owned cache and its capacity is at least one legal encoder iteration; this value can make that cache larger. String values such as ‘512MB’ and ‘1GiB’ use binary units. Both inline encoding and item scheduling cache individual items; inline caching supports only single-modality parameters. Compatible with side-stream prefetch.

Constraints:

  • ge = 0

field encoder_cuda_graph: dict[str, MultimodalEncoderCudaGraphConfig] | None = None#

CUDA graph capture for multimodal encoders, keyed by modality name. This config is not applied automatically - each model must read model_config.multimodal_config.encoder_cuda_graph and implement capture + replay via MultimodalEncoderCudaGraphRunner (see tensorrt_llm/_torch/models/multimodal_encoder_graph.py).

field encoder_scheduling_policy: MultimodalEncoderSchedulingPolicy = MultimodalEncoderSchedulingPolicy.DEFAULT#

MM encoder scheduling policy for models that support item-level encoder scheduling. DISABLED: legacy inline encode (item scheduling and its byte budget off). DEFAULT: item scheduling. EAGER: item scheduling that advances encoder work for active requests before LLM capacity filtering. Ignored for models that do not support item scheduling. Item scheduling does not support pipeline parallelism yet: with pipeline_parallel_size > 1, DEFAULT (explicit or unset) logs a warning and falls back to legacy inline encode, while EAGER raises an error.

field encoder_side_stream_max_ahead: Annotated[int, Ge(ge=0)] = 0#

Maximum number of pending multimodal requests whose encoder work can be prefetched on a side CUDA stream ahead of admission. 0 disables side-stream prefetch. Incompatible with encoder_cuda_graph because graph replay uses static buffers. Can be combined with encoder_cache_max_bytes; the two memory limits are additive.

Constraints:

  • ge = 0

field video_pruning_rate: float | None = None#

Pruning rate for video frames in multimodal models for Efficient Video Sampling (EVS). NOTE: this is currently only implemented in nemotron multimodal models. None (default) disables EVS, values in [0, 1) enable pruning.

Constraints:

  • ge = 0.0

  • lt = 1.0

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

validator parse_encoder_cache_max_bytes  »  encoder_cache_max_bytes[source]#
validator validate_encoder_optimization_compatibility  »  all fields[source]#