Supported Models

The following is a table of supported models for the PyTorch backend:

Architecture	Model	HuggingFace Example
AfmoeForCausalLM	Arcee Foundation MoE (Trinity)	arcee-ai/Trinity-Mini
BartForConditionalGeneration	BART	facebook/bart-large-cnn
BertForSequenceClassification	BERT-based	textattack/bert-base-uncased-yelp-polarity
Cohere2ForCausalLM	Command A	CohereLabs/c4ai-command-a-03-2025
DeciLMForCausalLM	Nemotron	nvidia/Llama-3_1-Nemotron-51B-Instruct
DeepseekV3ForCausalLM	DeepSeek-V3, Kimi-K2	deepseek-ai/DeepSeek-V3
DeepseekV32ForCausalLM	DeepSeek-V3.2	deepseek-ai/DeepSeek-V3.2
DeepseekV4ForCausalLM [11]	DeepSeek-V4	deepseek-ai/DeepSeek-V4-Pro
DeepseekV41ForCausalLM	DeepSeek-V4.1 (text decoder)	—
DeepseekV41ForConditionalGeneration	DeepSeek-V4.1	—
Exaone4ForCausalLM	EXAONE 4.0	LGAI-EXAONE/EXAONE-4.0-32B
ExaoneMoeForCausalLM	K-EXAONE	LGAI-EXAONE/K-EXAONE-236B-A23B
ExaoneMoeForCausalLM	K-EXAONE 2.0	LGAI-EXAONE/K-EXAONE-2.0-750B-A37B
Gemma3ForCausalLM	Gemma 3	google/gemma-3-1b-it
Gemma4ForConditionalGeneration	Gemma 4	google/gemma-4-E2B-it, google/gemma-4-E4B-it, google/gemma-4-26B-A4B-it, google/gemma-4-31B-it
Gemma4UnifiedForConditionalGeneration	Gemma 4 12B Unified (encoder-free)	google/gemma-4-12B, google/gemma-4-12B-it
Gemma4AssistantForCausalLM	Gemma 4 MTP assistant	google/gemma-4-E2B-it-assistant, google/gemma-4-E4B-it-assistant, google/gemma-4-26B-A4B-it-assistant, google/gemma-4-31B-it-assistant
Glm4MoeForCausalLM	GLM-4.5, GLM-4.6, GLM-4.7	THUDM/GLM-4-100B-A10B
GlmMoeDsaForCausalLM	GLM-5, GLM-5.2, GLM-5.3	zai-org/GLM-5, zai-org/GLM-5.2, zai-org/GLM-5.3
Glm5NextForConditionalGeneration [20]	GLM-5.3-Flash	zai-org/GLM-5.3-Flash
GptOssForCausalLM	GPT-OSS	openai/gpt-oss-20b, openai/gpt-oss-120b
KimiK25ForConditionalGeneration	Kimi-K2.5	moonshotai/Kimi-K2.5
KimiK3ForConditionalGeneration [15]	Kimi-K3	moonshotai/Kimi-K3
KimiLinearForCausalLM [15]	Kimi-K3 (text decoder)	moonshotai/Kimi-K3
LagunaForCausalLM	Laguna-XS	poolside/laguna-XS.2
LiLiCorrDraftModel	LiLiCorr draft model	Compatible checkpoint via DFlashDecodingConfig.speculative_model
LlamaForCausalLM	Llama 3.1, Llama 3, Llama 2, LLaMA	meta-llama/Meta-Llama-3.1-70B
Llama4ForConditionalGeneration	Llama 4	meta-llama/Llama-4-Scout-17B-16E-Instruct
MBartForConditionalGeneration	mBART	facebook/mbart-large-50-many-to-one-mmt
MiniCPMV4_6ForConditionalGeneration [14]	MiniCPM-V 4.6	openbmb/MiniCPM-V-4.6
MiniMaxM3SparseForConditionalGeneration [12]	MiniMax-M3	MiniMaxAI/MiniMax-M3
MistralForCausalLM	Mistral	mistralai/Mistral-7B-v0.1
MixtralForCausalLM	Mixtral	mistralai/Mixtral-8x7B-v0.1
MllamaForConditionalGeneration	Llama 3.2	meta-llama/Llama-3.2-11B-Vision
NemotronForCausalLM	Nemotron-3, Nemotron-4, Minitron	nvidia/Minitron-8B-Base
NemotronHForCausalLM	Nemotron-3-Nano, Nemotron-3-Super, Nemotron-3-Ultra	nvidia/nvidia-nemotron-v3
NemotronNASForCausalLM	NemotronNAS	nvidia/Llama-3_3-Nemotron-Super-49B-v1
Phi3ForCausalLM	Phi-4	microsoft/Phi-4
Qwen2ForCausalLM	QwQ, Qwen2	Qwen/Qwen2-7B-Instruct
Qwen2ForProcessRewardModel	Qwen2-based	Qwen/Qwen2.5-Math-PRM-7B
Qwen2ForRewardModel	Qwen2-based	Qwen/Qwen2.5-Math-RM-72B
Qwen3ForCausalLM	Qwen3	Qwen/Qwen3-8B
Qwen3ForTextEmbedding	Qwen3-Embedding	Qwen/Qwen3-Embedding-8B
Qwen3MoeForCausalLM	Qwen3MoE	Qwen/Qwen3-30B-A3B
Qwen3NextForCausalLM	Qwen3Next	Qwen/Qwen3-Next-80B-A3B-Thinking
Qwen3_5MoeForCausalLM	Qwen3.8-MoE, Qwen3.5-MoE	Qwen/Qwen3.8-2.4T-A95B, Qwen/Qwen3.5-397B-A17B
Qwen4ExpForCausalLM	Qwen3.8-Flash-Next (text)	Qwen/Qwen3.8-Flash-Next
Qwen4ExpForConditionalGeneration	Qwen3.8-Flash-Next	Qwen/Qwen3.8-Flash-Next
Step3p7ForConditionalGeneration [8]	Step-3.7-Flash	stepfun-ai/Step-3.7-Flash
T5ForConditionalGeneration	T5, Flan-T5, ByT5	google/flan-t5-small
WhisperForConditionalGeneration	Whisper	openai/whisper-large-v3

Model-Feature Support Matrix (Key Models)

Note: Support for other models may vary. Features marked “N/A” are not applicable to the model architecture. This matrix covers language-only requests, including the text-decoder path of multimodal architectures. The multimodal matrix below independently covers requests with actual image, video, or audio input, so a feature can have different statuses for the same architecture in the two matrices.

Model Architecture/Feature	Overlap Scheduler	CUDA Graph	Attention Data Parallelism	Disaggregated Serving	Chunked Prefill	Speculative Decoding [18]	Torch Sampler	KV Cache Reuse	Sliding Window Attention	Logits Post Processor	Guided Decoding
DeepseekV3ForCausalLM	Yes	Yes	Yes	Yes	Yes [1]	MTP	Yes	Yes [2]	N/A	Yes	Yes
DeepseekV32ForCausalLM	Yes	Yes	Yes	Yes	Yes	MTP	Yes	Yes	N/A	Yes	Yes
GlmMoeDsaForCausalLM	Yes	Yes	Yes	Yes	Yes	MTP	Yes	Yes	N/A	Yes	Yes
DeepseekV4ForCausalLM [11]	Yes	Yes	Yes	Untested	Yes	MTP	Yes	Yes	Yes	Yes	Yes
Glm4MoeForCausalLM	Yes	Yes	Yes	Untested	Yes	MTP	Yes	Yes	N/A	Yes	Yes [16]
Qwen3MoeForCausalLM	Yes	Yes	Yes	Yes	Yes	EAGLE-3 (Linear, Dynamic)	Yes	Yes	N/A	Yes	Yes
Qwen3NextForCausalLM [3]	Yes	Yes	Yes	Untested	Yes	No	Yes	Yes [19]	No	Yes	Yes
Qwen3_5MoeForCausalLM	Yes	Yes	Yes	Yes	Yes	MTP	Yes	Yes	N/A	Yes	Yes
Qwen4ExpForCausalLM	Yes	Yes	Yes	Yes	Yes	MTP	Yes	Yes	N/A	Yes	Yes
Llama4ForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	EAGLE-3 (Linear, Dynamic)	Yes	Yes	N/A	Yes	Yes
GptOssForCausalLM	Yes	Yes	Yes	Yes	Yes	EAGLE-3 (Linear), DFlash	Yes	Yes	N/A	Yes	Yes
KimiK3ForConditionalGeneration [15] [17]	Yes	Yes	Yes	Yes	Yes	DSpark	Yes	Yes	N/A	Yes	Yes
NemotronHForCausalLM	Yes	Yes	Yes	Yes	Yes	MTP	Yes	Yes	N/A	Yes	Yes
Gemma4ForConditionalGeneration	Yes	Yes	Yes	No	Yes	MTP	Yes	No	Yes	Yes	Yes
Gemma4UnifiedForConditionalGeneration	Yes	Yes	Yes	No	Yes	No	Yes	No	Yes	Yes	Yes
Step3p7ForConditionalGeneration	Yes	Yes	Yes	Untested	Untested	MTP	Yes	No	Yes	Yes	Yes
MiniMaxM3SparseForConditionalGeneration [12]	Yes	Yes	Yes	Untested	Untested	EAGLE-3 (Linear)	Yes	No	N/A	Yes	Yes

Encoder-Decoder Feature Support Matrix (PyTorch Backend)

The following capabilities apply to the supported encoder-decoder architectures. For configuration guidance and limitations, see Use encoder-decoder models with the PyTorch backend.

Model Architecture/Feature	Overlap Scheduler	Decoder CUDA Graph	Encoder CUDA Graph	KV Cache Manager V1	KV Cache Manager V2	Beam Search	Tensor Parallelism	Pipeline Parallelism	Chunked Prefill
BartForConditionalGeneration	Yes	Yes (except FP32)	Yes	Yes	Yes (single beam)	Yes (V1 only)	Yes	No	No (encoder phase)
MBartForConditionalGeneration	Yes	Yes (except FP32)	Yes	Yes	Yes (single beam)	Yes (V1 only)	Yes	No	No (encoder phase)
T5ForConditionalGeneration	Yes	Yes (except FP32)	Yes	Yes	Yes (single beam)	Yes (V1 only)	Yes	No	No (encoder phase)
WhisperForConditionalGeneration	Yes	Yes (except FP32)	No (feature inputs)	Yes	Yes (single beam)	Yes (V1 only)	Yes	No	No (encoder phase)

Decoder CUDA graphs support greedy and beam-search decoding with KV cache manager V1 and single-beam decoding with V2. Encoder CUDA graphs support the token-input BART, mBART, and T5 families; Whisper’s feature-driven audio encoder runs eagerly. Use the TRTLLM attention backend for encoder-decoder models; tensor parallelism also requires attention head counts divisible by the tensor parallel size. Chunked prefill is not supported for the encoder phase, so the complete encoder input must fit in the iteration token budget.

Multimodal Feature Support Matrix (PyTorch Backend)

Entries in this matrix require a request with at least one listed non-language modality; language-only test evidence is not used to mark a feature as supported here.

Model Architecture/Feature	Overlap Scheduler	CUDA Graph	Chunked Prefill	Torch Sampler	KV Cache Reuse	Logits Post Processor	EPD Disaggregated Serving	Modality
Exaone4_5_ForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	No	L + I + V
Gemma3ForConditionalGeneration	Yes	Yes	N/A	Yes	N/A	Yes	No	L + I
Gemma4ForConditionalGeneration	Yes	Yes	Yes	Yes	No	Yes	No	L + I + V + A [9]
Gemma4UnifiedForConditionalGeneration	Yes	Yes	Untested	Yes	No	Yes	No	L + I + A
HCXVisionForCausalLM	Yes	Yes	No	Yes	Yes	Yes	No	L + I
LlavaLlamaModel (VILA)	Yes	Yes	No	Yes	No	Yes	No	L + I + V
LlavaNextForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	L + I
Llama4ForConditionalGeneration	Yes	Yes	No	Yes	No	Yes	No	L + I
MiniCPMV4_6ForConditionalGeneration [14]	Yes	Untested	Untested	Yes	Untested	Untested	No	L + I + V
Mistral3ForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	No	L + I
NemotronH_Nano_VL_V2	Yes	Yes	Yes	Yes	N/A	Yes	Yes	L + I + V + A [10]
Phi4MMForCausalLM	Yes	Yes	Yes	Yes	Yes	Yes	No	L + I + A
Qwen2VLForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	No	L + I + V
Qwen2_5_VLForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	L + I + V
Qwen3VLForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	L + I + V
Qwen3VLMoeForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	L + I + V
Step3p7ForConditionalGeneration	Yes	Yes	Untested	Yes	Untested	Yes	Untested	L + I
MiniMaxM3SparseForConditionalGeneration [12]	Yes	Yes	Untested	Yes	No	Yes	Untested	L + I + V
Cosmos3ForConditionalGeneration [13]	Yes	Yes	Yes	Yes	Yes	Untested	Untested	L + I + V
Qwen3_5ForConditionalGeneration	Yes	Yes	Untested	Yes	No	Untested	Yes	L + I + V
Qwen3_5MoeForConditionalGeneration	Yes	Yes	Untested	Yes	No	Untested	Yes	L + I + V
Qwen4ExpForConditionalGeneration	Untested	Untested	Untested	Yes	Untested	Yes	No	L + I

Note:

• L: Language

• I: Image

• V: Video

• A: Audio

Multimodal Encoder Optimizations

The following optimizations are available to models that implement MultimodalModelMixin.

Model Architecture	Multimodal Encoder Side Stream	Multimodal Embeddings Cache
Gemma4ForConditionalGeneration	Yes	Yes
Gemma4UnifiedForConditionalGeneration	Yes	Yes
Mistral3ForConditionalGeneration	Yes	Yes
Qwen3VLForConditionalGeneration	Yes	Yes
Qwen3VLMoeForConditionalGeneration	Yes	Yes
Qwen3_5ForConditionalGeneration	Yes	Yes
Qwen3_5MoeForConditionalGeneration	Yes	Yes

• Multimodal encoder side stream prefetches encoder work for pending requests on a separate CUDA stream, allowing it to overlap with work on the main stream. Set multimodal_config.encoder_side_stream_max_ahead to a positive value to enable it; the value limits the number of prefetched requests that can be ahead of admission. This option is mutually exclusive with multimodal_config.encoder_cuda_graph and can increase peak GPU memory use. It can be combined with the multimodal embeddings cache so side-stream cache hits skip encoder work and misses populate the cache.

• Multimodal embeddings cache is a per-model, cross-request LRU cache of encoder embeddings, stored per multimodal item. Set multimodal_config.encoder_cache_max_bytes to its capacity (for example, "512MiB"), or 0 to disable cross-request reuse. The capacity applies per rank; ranks do not share a cache.

  • On the inline encoder path, a request whose items all hit the cache skips the encoder, and a partially cached request encodes only its missing items. Inline caching supports only single-modality requests. When combined with side-stream prefetch, peak memory is the cache capacity plus any in-flight prefetched encoder inputs and outputs.

  • Models with item-level encoder scheduling use the item path unless multimodal_config.encoder_scheduling_policy is DISABLED. Cached items are reused before encoder work is selected, so they use none of the per-iteration encoder item or token budget. Only missing items are encoded, mixed-modality requests are cached per item, and newly encoded items populate the cache for later requests. The item path also keeps encoder outputs in this cache until the request’s prefill completes, so the cache is never smaller than the output of one encoder iteration, even when encoder_cache_max_bytes is 0. Item scheduling does not support side-stream prefetch or pipeline parallelism yet. With pipeline_parallel_size > 1, the DEFAULT policy, explicit or unset, logs a warning and falls back to the inline path, and the EAGER policy raises an error.

Visual Generation Models

TensorRT-LLM provides beta support for diffusion-based image and video generation. For full documentation, see the Visual Generation page.

Supported Models

HuggingFace Model ID	Tasks
black-forest-labs/FLUX.1-dev	Text-to-Image
black-forest-labs/FLUX.2-dev	Text-to-Image
Wan-AI/Wan2.1-T2V-1.3B-Diffusers	Text-to-Video
Wan-AI/Wan2.1-T2V-14B-Diffusers	Text-to-Video
Wan-AI/Wan2.1-I2V-14B-480P-Diffusers	Image-to-Video
Wan-AI/Wan2.1-I2V-14B-720P-Diffusers	Image-to-Video
Wan-AI/Wan2.2-T2V-A14B-Diffusers	Text-to-Video
Wan-AI/Wan2.2-I2V-A14B-Diffusers	Image-to-Video
Wan-AI/Wan2.2-TI2V-5B-Diffusers	Text-to-Video, Image-to-Video
Lightricks/LTX-2	Text-to-Video (with Audio), Image-to-Video (with Audio)
MiniMaxAI/MiniMax-H3	Text-to-Video (with Audio), First/Last-Frame-to-Video (with Audio)
Qwen/Qwen-Image	Text-to-Image
Qwen/Qwen-Image-2512	Text-to-Image
Qwen/Qwen-Image-Layered	Image-to-Image
Qwen/Qwen-Image-Edit-2511	Image Editing (text+images-to-image)
nvidia/Cosmos3-Nano	Text-to-Image, Text-to-Video, Image-to-Video
nvidia/Cosmos3-Super	Text-to-Image, Text-to-Video, Image-to-Video
nvidia/Cosmos3-Super-Text2Image-4Step	Text-to-Image (DMD2-distilled, fixed 4-step schedule)
nvidia/Cosmos3-Super-Image2Video-4Step	Image-to-Video (DMD2-distilled, fixed 4-step schedule)
nvidia/Cosmos3-Edge	Text-to-Image, Text-to-Video, Image-to-Video (Nemotron-dense backbone, 480p-native)

Feature Matrix

Model	FP8 blockwise	NVFP4	TeaCache	CFG Parallelism	Ulysses Parallelism	Parallel VAE	CUDA Graph	torch.compile	trtllm-serve	Attention2D	Ring Attention	Tensor Parallelism
FLUX.1	Yes	Yes	Yes	No	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes
FLUX.2	Yes	Yes	Yes	No	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes
Wan 2.1	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Wan 2.2	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
LTX-2	Yes	Yes	No	Yes	Yes	No	No	Yes	Yes	Yes	Yes	No
MiniMax-H3	Yes	Yes	No	No	Yes	Yes	No	Yes	Yes	No	No	No
Qwen-Image	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	No
Qwen-Image-Layered	Yes	No	No	No	No	No	Yes	Yes	Yes	No	No	No
Qwen-Image-Edit-2511	Yes	Yes	No	Yes	No	No	Yes	Yes	Yes	No	No	No
Cosmos3	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	No	No	Yes

---

[1]

Chunked Prefill for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120.

[2]

KV cache reuse for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120/SM121 and in BF16/FP8 KV cache dtype.

[3]

Qwen3-Next-80B-A3B exhibits relatively low accuracy on the SciCode-AA-v2 benchmark.

[8]

Supports text and image inputs. The vision tower runs in BF16 even when the text decoder is quantized (FP8 block-scale or NVFP4). The text decoder is also usable standalone (text-only) via the Step3p5ForCausalLM architecture.

[9]

Audio modality only supported on E2B/E4B variants.

[10]

Audio requires a checkpoint with a sound_config and is supported only on the full (non-disaggregated) model path, not the EPD disaggregated path.

[11] (1,2)

DeepSeek-V4 is only supported on Blackwell GPUs (SM100+). See the DeepSeek-V4 example README for setup and parallelism.

[12] (1,2,3)

Supports text, image, and video inputs over the block-sparse attention path. The published MXFP8 checkpoint is dequantized on load so the runtime sees an effectively BF16 model. The text decoder is also usable standalone (text-only) via the MiniMaxM3SparseForCausalLM architecture. KV cache reuse and MTP are not supported on the sparse-attention path in this release.

[13]

The Cosmos 3 family also supports visual generation through the VisualGen API. See Visual Generation Models.

[14] (1,2)

Requires transformers>=5.7.0: MiniCPM-V 4.6 was upstreamed into transformers as a native model type (minicpmv4_6) and the checkpoint ships no remote code (auto_map) to fall back on. The Qwen3.5-hybrid text tower runs in BF16. Image, video, and text inputs are supported in this release (video reuses the same NaViT-packed vision path as image via MiniCPMV4_6InputProcessor).

[15] (1,2,3)

Kimi K3 is only supported on NVIDIA Blackwell GPUs (SM100 family); see the Kimi K3 deployment guide. Which recipes fit is set by the attention layout rather than by the GPU count: DEP16 (enable_attention_dp: true) replicates the BF16 non-expert weights on every rank (114 GB) on top of the MXFP4 routed experts at 16-way expert parallelism (90 GB), needing 210 GB per rank, so it requires GB300-class per-GPU memory. TEP16 (enable_attention_dp: false) shards those non-expert weights instead and needs 115 GB per rank; it is validated end-to-end on GB200 (SM100) at 16 GPUs. On B200 (SM100), kernel and module support is functional and covered by CI. Note that the FP8 weight-read path (TRTLLM-14765) does not relax the DEP16 requirement: the conversion runs after the weights are already resident in BF16, so it lowers the steady-state footprint but not the load-time peak.

[16]

Guided decoding for Glm4MoeForCausalLM is currently supported only with the xgrammar backend; llguidance is not working.

[17]

Kimi K3 has no MTP or EAGLE-3 head, and its DSpark checkpoints are not compatible with plain DFlash.

[18]

NGram and standalone Suffix Automaton (SA) use model-free drafting on the PyTorch backend, so they are not listed in individual entries. This does not imply universal end-to-end support: compatibility depends on each model’s multi-token verification and cache-management paths and may be untested or explicitly restricted.

[19]

KV cache reuse for hybrid recurrent-attention models requires an explicit recurrent-state snapshot policy, such as kv_cache_config.mamba_state_config.periodic_snapshot_interval; the model default disables reuse when no snapshot policy is configured.

[20]

Supports text, image, and video inputs, MTP (including with attention data parallelism), and FP8 KV cache. Image and video inputs require transformers==5.17.0. Beam search is not supported. See the GLM-5.3-Flash deployment guide for setup and feature-specific requirements.