MultimodalEncoder

class tensorrt_llm.llmapi.MultimodalEncoder(

model: str | Path,

trust_remote_code: bool = False,

tensor_parallel_size: int = 1,

dtype: Literal['auto', 'float16', 'float32', 'bfloat16'] = 'auto',

**kwargs: Any,

)

Bases: _TorchLLM

MultimodalEncoder class is the main class for running a multimodal encoder model using PyTorch backend.

__init__(

model: str | Path,

trust_remote_code: bool = False,

tensor_parallel_size: int = 1,

dtype: Literal['auto', 'float16', 'float32', 'bfloat16'] = 'auto',

**kwargs: Any,

) → None

encode(

inputs: str | List[int] | TextPrompt | TokensPrompt | Sequence[str | List[int] | TextPrompt | TokensPrompt],

add_special_tokens: bool = True,

batch_indexed_model_output: bool = True,

copy_logits_to_host: bool = True,

return_raw_logits: bool = False,

**model_kwargs: Any,

) → EncoderOutput | List[EncoderOutput] | Tensor

prototype Encode inputs using an encoder-only model (PyTorch backend only).

Only available when encode_only=True is set in the LLM constructor.

Parameters:

• inputs (tensorrt_llm.inputs.data.PromptInputs, Sequence[tensorrt_llm.inputs.data.PromptInputs]) – The prompt text or token ids. It can be a single prompt or batched prompts.

• add_special_tokens (bool) – Whether to add special tokens (e.g., [CLS]/[SEP]) during tokenization. Defaults to True.

• batch_indexed_model_output (bool) – If specified, assume batched model output indexed by request index, as opposed to token index. Defaults to True.

• copy_logits_to_host (bool) – If set, copy logits from device to host. Otherwise, return a view into the on-device logits tensor. Defaults to True.

• return_raw_logits (bool) – Whether to return the raw CPU logits tensor for the whole input batch. Defaults to False.

• model_kwargs (Any) – Model-specific inputs passed through to the model’s forward(). Examples: token_type_ids (BERT), inputs_embeds (reward models).

Returns:

If return_raw_logits=True, returns the raw CPU logits tensor for the whole input batch. Otherwise, returns one EncoderOutput for a single input, or a list of EncoderOutput objects for batched inputs.

Return type:

Union[tensorrt_llm.llmapi.llm.EncoderOutput, List[tensorrt_llm.llmapi.llm.EncoderOutput], torch.Tensor]

Raises:

RuntimeError – If encode_only mode is not enabled.

generate(

inputs: str | List[int] | TextPrompt | TokensPrompt | Sequence[str | List[int] | TextPrompt | TokensPrompt],

use_tqdm: bool = True,

) → RequestOutput | List[RequestOutput]

Generate output for the given prompts in the synchronous mode. Synchronous generation accepts either single prompt or batched prompts.

Parameters:

inputs (tensorrt_llm.inputs.data.PromptInputs, Sequence[tensorrt_llm.inputs.data.PromptInputs]) – The prompt text or token ids. It can be single prompt or batched prompts.

Returns:

The output data of the completion request to the LLM.

Return type:

Union[tensorrt_llm.llmapi.RequestOutput, List[tensorrt_llm.llmapi.RequestOutput]]

generate_async(

inputs: str | List[int] | TextPrompt | TokensPrompt,

sampling_params: SamplingParams | None = None,

) → RequestOutput

Generate output for the given multimodal request in the asynchronous mode. Asynchronous generation accepts single multimodal request only.

Returns:

Future that resolves to tensorrt_llm.llmapi.RequestOutput containing mm_embeddings

get_data_transceiver_state() → bytes

prototype Get the serialized DataTransceiverState for arbitrary KV cache transfer.

Returns:

Serialized DataTransceiverState, or empty bytes if no transceiver is configured.

Return type:

bytes

get_kv_cache_capacity() → dict

beta Get the runtime’s static primary/GPU KV cache capacity.

Raises:

RuntimeError – If called when encode_only=True.

Returns:

KV cache capacity. The returned capacity covers the primary

GPU KV cache pool only; CPU/host offload capacity is not included. e.g., {“maxNumBlocks”: …, “tokensPerBlock”: …, “maxNumTokens”: …}

Return type:

dict

get_kv_cache_events(

timeout: float | None = 2,

) → List[dict]

beta Get iteration KV events from the runtime.

KV events are used to track changes and operations within the KV Cache. Types of events:

• KVCacheCreatedData: Indicates the creation of cache blocks.

• KVCacheStoredData: Represents a sequence of stored blocks.

• KVCacheRemovedData: Contains the hashes of blocks that are being removed from the cache.

• KVCacheUpdatedData: Captures updates to existing cache blocks.

To enable KV events:

• set event_buffer_max_size to a positive integer in the KvCacheConfig.

• set enable_block_reuse to True in the KvCacheConfig.

Parameters:

timeout (float, optional) – Max wait time in seconds when retrieving events from queue. Defaults to 2.

Returns:

A list of runtime events as dict.

Return type:

List[dict]

get_kv_cache_events_async(

timeout: float | None = 2,

) → IterationResult

beta Get iteration KV events from the runtime.

KV events are used to track changes and operations within the KV Cache. Types of events:

• KVCacheCreatedData: Indicates the creation of cache blocks.

• KVCacheStoredData: Represents a sequence of stored blocks.

• KVCacheRemovedData: Contains the hashes of blocks that are being removed from the cache.

• KVCacheUpdatedData: Captures updates to existing cache blocks.

To enable KV events:

• set event_buffer_max_size to a positive integer in the KvCacheConfig.

• set enable_block_reuse to True in the KvCacheConfig.

Parameters:

timeout (float, optional) – Max wait time in seconds when retrieving events from queue. Defaults to 2.

Returns:

An async iterable object containing runtime events.

Return type:

tensorrt_llm.executor.result.IterationResult

get_stats(

timeout: float | None = 2,

) → List[dict]

beta Get iteration statistics from the runtime. To collect statistics, call this function after prompts have been submitted with LLM().generate().

Parameters:

timeout (float, optional) – Max wait time in seconds when retrieving stats from queue. Defaults to 2.

Returns:

A list of runtime stats as dicts.

e.g., [{“cpuMemUsage”: …, “iter”: 0, …}, {“cpuMemUsage”: …, “iter”: 1, …}]

Return type:

List[dict]

get_stats_async(

timeout: float | None = 2,

) → IterationResult

beta Get iteration statistics from the runtime. To collect statistics, you can call this function in an async coroutine or the /metrics endpoint (if you’re using trtllm-serve) after prompts have been submitted.

Parameters:

timeout (float, optional) – Max wait time in seconds when retrieving stats from queue. Defaults to 2.

Returns:

An async iterable object containing runtime stats.

Return type:

tensorrt_llm.executor.result.IterationResult

preprocess(

inputs: str | List[int] | TextPrompt | TokensPrompt,

sampling_params: SamplingParams | None = None,

disaggregated_params: DisaggregatedParams | None = None,

) → PreprocessedInputs

prototype Preprocess raw prompts into token IDs and multimodal params.

Parameters:

• inputs (tensorrt_llm.inputs.data.PromptInputs) – The prompt text or token ids; it must be single prompt.

• sampling_params (tensorrt_llm.sampling_params.SamplingParams, optional) – The sampling params for the generation. Defaults to None. A default one will be used if not provided.

• disaggregated_params (tensorrt_llm.disaggregated_params.DisaggregatedParams, optional) – Disaggregated parameters. Defaults to None.

Returns:

A preprocessed-inputs object that can be

passed directly to generate_async() as inputs.

Return type:

tensorrt_llm.llmapi.llm.PreprocessedInputs

shutdown() → None

beta None

start_profile(

output_dir: str | None = None,

num_steps: int | None = None,

start_step: int = 0,

activities: List[str] | None = None,

) → None

prototype Start iteration-scoped profiling of the backend engine.

Mirrors the TLLM_PROFILE_START_STOP / TLLM_TORCH_PROFILE_TRACE environment-variable behaviour but can be triggered at runtime (for example via the trtllm-serve /start_profile HTTP endpoint).

See PyExecutor.start_profile for full argument semantics.

Parameters:

• output_dir (str, optional) – Directory where chrome traces are written. If None, falls back to the TLLM_TORCH_PROFILER_DIR environment variable and finally to /tmp. Defaults to None.

• num_steps (int, optional) – Number of engine iterations to capture. When set, the engine stops the window automatically and stop_profile does not need to be called. When None, profiling runs until stop_profile is called. Defaults to None.

• start_step (int) – Additional iterations to skip before profiling actually begins, relative to the current iteration counter. Defaults to 0.

• activities (List[str], optional) – Subset of ["CPU", "GPU", "CUDA_PROFILER"] selecting which profiler activities to record. When CUDA_PROFILER is the only entry, torch.profiler is not started so only cudaProfilerStart/cudaProfilerStop brackets run, which is suitable for nsys capture. Defaults to None, which resolves to ["CPU", "GPU"].

stop_profile() → None

prototype Stop any in-progress runtime profiling.

property disaggregated_params: dict

beta None

property llm_id: str

beta None

property startup_metrics: dict

beta Cache and return rank-0 startup metrics.

Returns:

The cached metrics, or an empty dict when metrics retrieval fails.

Return type:

dict

property tokenizer: TokenizerBase | None