Supported Models

The following is a table of supported models for the PyTorch backend:

Architecture	Model	HuggingFace Example
AfmoeForCausalLM	Arcee Foundation MoE (Trinity)	arcee-ai/Trinity-Mini
BartForConditionalGeneration	BART	facebook/bart-large-cnn
BertForSequenceClassification	BERT-based	textattack/bert-base-uncased-yelp-polarity
Cohere2ForCausalLM	Command A	CohereLabs/c4ai-command-a-03-2025
DeciLMForCausalLM	Nemotron	nvidia/Llama-3_1-Nemotron-51B-Instruct
DeepSeekV2ForCausalLM [5]	DeepSeek V2	deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
DeepseekV3ForCausalLM	DeepSeek-V3, Kimi-K2	deepseek-ai/DeepSeek-V3
DeepseekV32ForCausalLM	DeepSeek-V3.2	deepseek-ai/DeepSeek-V3.2
DeepseekV4ForCausalLM [11]	DeepSeek-V4	deepseek-ai/DeepSeek-V4-Pro
ExaoneForCausalLM [5]	EXAONE 3.5	LGAI-EXAONE/EXAONE-3.5-32B-Instruct
Exaone4ForCausalLM	EXAONE 4.0	LGAI-EXAONE/EXAONE-4.0-32B
ExaoneMoEForCausalLM	K-EXAONE	LGAI-EXAONE/K-EXAONE-236B-A23B
Gemma3ForCausalLM	Gemma 3	google/gemma-3-1b-it
Gemma3nForConditionalGeneration [7]	Gemma 3n	google/gemma-3n-E2B-it, google/gemma-3n-E4B-it
Gemma4ForConditionalGeneration	Gemma 4	google/gemma-4-E2B-it, google/gemma-4-E4B-it, google/gemma-4-26B-A4B-it [6], google/gemma-4-31B-it [6]
Gemma4UnifiedForConditionalGeneration	Gemma 4 12B Unified (encoder-free)	google/gemma-4-12B, google/gemma-4-12B-it
Gemma4AssistantForCausalLM	Gemma 4 MTP assistant	google/gemma-4-E2B-it-assistant, google/gemma-4-E4B-it-assistant, google/gemma-4-26B-A4B-it-assistant, google/gemma-4-31B-it-assistant
Glm4MoeForCausalLM	GLM-4.5, GLM-4.6, GLM-4.7	THUDM/GLM-4-100B-A10B
Glm4MoeLiteForCausalLM [5]	GLM-4.7-Flash	zai-org/GLM-4.7-Flash
GlmMoeDsaForCausalLM	GLM-5	zai-org/GLM-5
GraniteForCausalLM [5]	Granite 3, Granite Guardian 3	ibm-granite/granite-3.1-8b-instruct, ibm-granite/granite-3.3-8b-instruct, ibm-granite/granite-guardian-3.2-5b
GraniteMoeHybridForCausalLM [5]	Granite 4.0 Hybrid MoE	ibm-granite/granite-4.0-h-small
GptOssForCausalLM	GPT-OSS	openai/gpt-oss-20b, openai/gpt-oss-120b
HunYuanDenseForCausalLM [5]	Hunyuan Dense	tencent/Hunyuan-7B-Instruct
HunYuanMoEForCausalLM [5]	Hunyuan MoE	tencent/Hunyuan-A13B-Instruct
InternLM3ForCausalLM [5]	InternLM3	internlm/internlm3-8b-instruct
KimiK25ForConditionalGeneration	Kimi-K2.5	moonshotai/Kimi-K2.5
LagunaForCausalLM	Laguna-XS	poolside/laguna-XS.2
LlamaForCausalLM	Llama 3.1, Llama 3, Llama 2, LLaMA	meta-llama/Meta-Llama-3.1-70B
Llama4ForConditionalGeneration	Llama 4	meta-llama/Llama-4-Scout-17B-16E-Instruct
MBartForConditionalGeneration	mBART	facebook/mbart-large-50-many-to-one-mmt
MiniCPMV4_6ForConditionalGeneration [14]	MiniCPM-V 4.6	openbmb/MiniCPM-V-4.6
MiniMaxM2ForCausalLM [5]	MiniMax M2/M2.1/M2.7	MiniMaxAI/MiniMax-M2.7
MiniMaxM3SparseForConditionalGeneration [12]	MiniMax-M3	MiniMaxAI/MiniMax-M3
MistralForCausalLM	Mistral	mistralai/Mistral-7B-v0.1
MixtralForCausalLM	Mixtral	mistralai/Mixtral-8x7B-v0.1
MllamaForConditionalGeneration	Llama 3.2	meta-llama/Llama-3.2-11B-Vision
NemotronForCausalLM	Nemotron-3, Nemotron-4, Minitron	nvidia/Minitron-8B-Base
NemotronHForCausalLM	Nemotron-3-Nano, Nemotron-3-Super, Nemotron-3-Ultra	nvidia/nvidia-nemotron-v3
NemotronNASForCausalLM	NemotronNAS	nvidia/Llama-3_3-Nemotron-Super-49B-v1
Olmo3ForCausalLM [5]	OLMo 3, OLMo 3.1	allenai/Olmo-3.1-32B-Instruct
OpenELMForCausalLM [5]	OpenELM	apple/OpenELM-270M-Instruct
Phi3ForCausalLM	Phi-4	microsoft/Phi-4
Qwen2ForCausalLM	QwQ, Qwen2	Qwen/Qwen2-7B-Instruct
Qwen2ForProcessRewardModel	Qwen2-based	Qwen/Qwen2.5-Math-PRM-7B
Qwen2ForRewardModel	Qwen2-based	Qwen/Qwen2.5-Math-RM-72B
Qwen3ForCausalLM	Qwen3	Qwen/Qwen3-8B
Qwen3ForTextEmbedding	Qwen3-Embedding	Qwen/Qwen3-Embedding-8B
Qwen3MoeForCausalLM	Qwen3MoE	Qwen/Qwen3-30B-A3B
Qwen3NextForCausalLM	Qwen3Next	Qwen/Qwen3-Next-80B-A3B-Thinking
Qwen3_5MoeForCausalLM	Qwen3.5-MoE	Qwen/Qwen3.5-397B-A17B
SeedOssForCausalLM [5]	Seed OSS, Seed-Coder	ByteDance-Seed/Seed-OSS-36B-Instruct
SkyworkR1V2ForConditionalGeneration [5]	Skywork R1V2, Skywork SWE	Skywork/Skywork-R1V2-38B
SmolLM3ForCausalLM [5]	SmolLM3	HuggingFaceTB/SmolLM3-3B
Step3p7ForConditionalGeneration [8]	Step-3.7-Flash	stepfun-ai/Step-3.7-Flash
T5ForConditionalGeneration	T5, Flan-T5, ByT5	google/flan-t5-small
WhisperForConditionalGeneration	Whisper	openai/whisper-large-v3

Model-Feature Support Matrix (Key Models)

Note: Support for other models may vary. Features marked “N/A” are not applicable to the model architecture.

Model Architecture/Feature	Overlap Scheduler	CUDA Graph	Attention Data Parallelism	Disaggregated Serving	Chunked Prefill	MTP	EAGLE-3 — Linear	EAGLE-3 — Dynamic	DFlash	Torch Sampler	TLLM C++ Sampler	KV Cache Reuse	Sliding Window Attention	Logits Post Processor	Guided Decoding
DeepseekV3ForCausalLM	Yes	Yes	Yes	Yes	Yes [1]	Yes	No	No	No	Yes	Yes	Yes [2]	N/A	Yes	Yes
DeepseekV32ForCausalLM	Yes	Yes	Yes	Yes	Yes	Yes	No	No	No	Yes	Yes	Yes	N/A	Yes	Yes
DeepseekV4ForCausalLM [11]	Yes	Yes	Yes	Untested	Yes	Yes	No	No	No	Yes	Yes	Untested	Yes	Untested	Untested
Glm4MoeForCausalLM	Yes	Yes	Yes	Untested	Yes	Yes	No	No	No	Yes	Yes	Untested	N/A	Yes	Yes
Qwen3MoeForCausalLM	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	No	Yes	Yes	Yes	N/A	Yes	Yes
Qwen3NextForCausalLM [3]	Yes	Yes	Yes	Untested	Yes	No	No	No	No	Yes	Yes	No	No	Untested	Untested
Qwen3_5MoeForCausalLM	Yes	Yes	Yes	Yes	Yes	Yes	No	No	No	Yes	Untested	Yes	N/A	Untested	Untested
Llama4ForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	No	Yes	Yes	Untested	N/A	Yes	Yes
GptOssForCausalLM	Yes	Yes	Yes	Yes	Yes	No	Yes	No	Yes	Yes	Yes	Yes	N/A	Yes	Yes
Glm4MoeLiteForCausalLM [5]	Yes	Yes	Untested	Untested	Yes	No	No	No	No	Yes	Untested	Untested	N/A	Untested	Untested
NemotronHForCausalLM	Yes	Yes	Yes	Yes	Yes	Yes	No	No	No	Yes	Yes	Yes	N/A	Untested	Untested
Gemma4ForConditionalGeneration	Untested	Yes	Untested	No	Yes	Yes	No	No	No	Yes	Untested	No	Yes	Untested	Untested
Gemma4UnifiedForConditionalGeneration	Untested	Untested	Untested	No	Yes	No	No	No	No	Yes	Untested	No	Yes	Untested	Untested
Step3p7ForConditionalGeneration	Yes	Yes	Yes	Untested	Untested	Yes	No	No	No	Yes	Untested	Untested	Yes	Untested	Untested
MiniMaxM3SparseForConditionalGeneration [12]	Yes	Yes	Yes	Untested	Untested	No	No	No	No	Yes	Untested	No	N/A	Untested	Untested

Encoder-Decoder Feature Support Matrix (PyTorch Backend)

The following capabilities apply to the supported encoder-decoder architectures. For configuration guidance and limitations, see Use encoder-decoder models with the PyTorch backend.

Model Architecture/Feature	Overlap Scheduler	Decoder CUDA Graph	Encoder CUDA Graph	KV Cache Manager V1	KV Cache Manager V2	Beam Search	Tensor Parallelism	Pipeline Parallelism	Chunked Prefill
BartForConditionalGeneration	Yes	Yes (except FP32)	Yes	Yes	Yes (single beam)	Yes (V1 only)	Yes	No	No (encoder phase)
MBartForConditionalGeneration	Yes	Yes (except FP32)	Yes	Yes	Yes (single beam)	Yes (V1 only)	Yes	No	No (encoder phase)
T5ForConditionalGeneration	Yes	Yes (except FP32)	Yes	Yes	Yes (single beam)	Yes (V1 only)	Yes	No	No (encoder phase)
WhisperForConditionalGeneration	Yes	Yes (except FP32)	No (feature inputs)	Yes	Yes (single beam)	Yes (V1 only)	Yes	No	No (encoder phase)

Decoder CUDA graphs support greedy and beam-search decoding with KV cache manager V1 and single-beam decoding with V2. Encoder CUDA graphs support the token-input BART, mBART, and T5 families; Whisper’s feature-driven audio encoder runs eagerly. Use the TRTLLM attention backend for encoder-decoder models; tensor parallelism also requires attention head counts divisible by the tensor parallel size. Chunked prefill is not supported for the encoder phase, so the complete encoder input must fit in the iteration token budget.

Multimodal Feature Support Matrix (PyTorch Backend)

Model Architecture/Feature	Overlap Scheduler	CUDA Graph	Chunked Prefill	Torch Sampler	TLLM C++ Sampler	KV Cache Reuse	Logits Post Processor	EPD Disaggregated Serving	Modality
Exaone4_5_ForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	L + I + V
Gemma3ForConditionalGeneration	Yes	Yes	N/A	Yes	Yes	N/A	Yes	No	L + I
Gemma4ForConditionalGeneration	Untested	Yes	Yes	Yes	Untested	No	Untested	No	L + I + V + A [9]
Gemma4UnifiedForConditionalGeneration	Untested	Untested	Untested	Yes	Untested	No	Untested	No	L + I + A
HCXVisionForCausalLM	Yes	Yes	No	Yes	Yes	Yes	Yes	No	L + I
LlavaLlamaModel (VILA)	Yes	Yes	No	Yes	Yes	No	Yes	No	L + I + V
LlavaNextForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	L + I
Llama4ForConditionalGeneration	Yes	Yes	No	Yes	Yes	No	Yes	No	L + I
MiniCPMV4_6ForConditionalGeneration [14]	Yes	Untested	Untested	Yes	Untested	Untested	Untested	No	L + I + V
Mistral3ForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	L + I
NemotronH_Nano_VL_V2	Yes	Yes	Yes	Yes	Yes	N/A	Yes	Yes	L + I + V + A [10]
Phi4MMForCausalLM	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	L + I + A
Qwen2VLForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	L + I + V
Qwen2_5_VLForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	L + I + V
Qwen3VLForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	L + I + V
Qwen3VLMoeForConditionalGeneration	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	L + I + V
Step3p7ForConditionalGeneration	Yes	Yes	Untested	Yes	Untested	Untested	Untested	Untested	L + I
MiniMaxM3SparseForConditionalGeneration [12]	Yes	Yes	Untested	Yes	Untested	No	Untested	Untested	L + I + V
Cosmos3ForConditionalGeneration [13]	Yes	Yes	Yes	Yes	Yes	Yes	Untested	Untested	L + I + V
Qwen3_5ForConditionalGeneration	Yes	Yes	Untested	Yes	Yes	No	Untested	Yes	L + I + V
Qwen3_5MoeForConditionalGeneration	Yes	Yes	Untested	Yes	Yes	No	Untested	Yes	L + I + V

Note:

• L: Language

• I: Image

• V: Video

• A: Audio

Multimodal Encoder Optimizations

The following optimizations are available to models that implement MultimodalModelMixin.

Model Architecture	Multimodal Encoder Side Stream	Multimodal Embeddings Cache
Gemma4ForConditionalGeneration	Yes	Yes
Gemma4UnifiedForConditionalGeneration	Yes	Yes
Mistral3ForConditionalGeneration	Yes	Yes
Qwen3VLForConditionalGeneration	Yes	Yes
Qwen3VLMoeForConditionalGeneration	Yes	Yes
Qwen3_5ForConditionalGeneration	Yes	Yes
Qwen3_5MoeForConditionalGeneration	Yes	Yes

• Multimodal encoder side stream prefetches encoder work for pending requests on a separate CUDA stream, allowing it to overlap with work on the main stream. Set multimodal_config.encoder_side_stream_max_ahead to a positive value to enable it; the value limits the number of prefetched requests that can be ahead of admission. This option is mutually exclusive with multimodal_config.encoder_cuda_graph and can increase peak GPU memory use. It can be combined with the multimodal embeddings cache so side-stream cache hits skip encoder work and misses populate the cache.

• Multimodal embeddings cache is a per-model, cross-request LRU cache of encoder embeddings. Set multimodal_config.encoder_cache_max_bytes to its capacity (for example, "512MiB"), or 0 to disable it. Entries are cached per multimodal item, but a request reuses cached embeddings only when all of its items hit the cache. At present, only single-modality requests are cacheable; mixed-modality requests bypass the cache. When combined with side-stream prefetch, peak memory is the cache capacity plus any in-flight prefetched encoder inputs and outputs.

Visual Generation Models

TensorRT-LLM provides beta support for diffusion-based image and video generation. For full documentation, see the Visual Generation page.

Supported Models

HuggingFace Model ID	Tasks
black-forest-labs/FLUX.1-dev	Text-to-Image
black-forest-labs/FLUX.2-dev	Text-to-Image
Wan-AI/Wan2.1-T2V-1.3B-Diffusers	Text-to-Video
Wan-AI/Wan2.1-T2V-14B-Diffusers	Text-to-Video
Wan-AI/Wan2.1-I2V-14B-480P-Diffusers	Image-to-Video
Wan-AI/Wan2.1-I2V-14B-720P-Diffusers	Image-to-Video
Wan-AI/Wan2.2-T2V-A14B-Diffusers	Text-to-Video
Wan-AI/Wan2.2-I2V-A14B-Diffusers	Image-to-Video
Wan-AI/Wan2.2-TI2V-5B-Diffusers	Text-to-Video, Image-to-Video
Lightricks/LTX-2	Text-to-Video (with Audio), Image-to-Video (with Audio)
Qwen/Qwen-Image	Text-to-Image
Qwen/Qwen-Image-2512	Text-to-Image
Qwen/Qwen-Image-Layered	Image-to-Image
Qwen/Qwen-Image-Edit-2511	Image Editing (text+images-to-image)
nvidia/Cosmos3-Nano	Text-to-Image, Text-to-Video, Image-to-Video
nvidia/Cosmos3-Super	Text-to-Image, Text-to-Video, Image-to-Video
nvidia/Cosmos3-Super-Text2Image-4Step	Text-to-Image (DMD2-distilled, fixed 4-step schedule)
nvidia/Cosmos3-Super-Image2Video-4Step	Image-to-Video (DMD2-distilled, fixed 4-step schedule)

Feature Matrix

Model	FP8 blockwise	NVFP4	TeaCache	CFG Parallelism	Ulysses Parallelism	Parallel VAE	CUDA Graph	torch.compile	trtllm-serve	Attention2D	Ring Attention	Tensor Parallelism
FLUX.1	Yes	Yes	Yes	No [1]	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes
FLUX.2	Yes	Yes	Yes	No [1]	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes
Wan 2.1	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Wan 2.2	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
LTX-2	Yes	Yes	No	Yes	Yes	No	No	Yes	Yes	Yes	Yes	No
Qwen-Image	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	No
Qwen-Image-Layered [3]	No	No	No	No	No	No	Yes	Yes	No	No	No	No
Qwen-Image-Edit-2511	Yes	Yes	No	Yes	No	No	Yes	Yes	No	No	No	No
Cosmos3	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	No	No	Yes

---

[1] (1,2,3)

Chunked Prefill for MLA can only be enabled on SM90/SM100/SM103/SM120.

[2]

KV cache reuse for MLA can only be enabled on SM90/SM100/SM103/SM120/SM121 and in BF16/FP8 KV cache dtype.

[3] (1,2)

Qwen3-Next-80B-A3B exhibits relatively low accuracy on the SciCode-AA-v2 benchmark.

[4]

FLUX models use embedded guidance and do not have a separate negative prompt path, so CFG parallelism is not applicable.

[5] (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15)

Supported via the AutoDeploy backend. See AD Configs.

[6] (1,2)

Also supports text-only inference via the AutoDeploy backend.

[7]

Text-only support via the AutoDeploy backend.

[8]

Supports text and image inputs. The vision tower runs in BF16 even when the text decoder is quantized (FP8 block-scale or NVFP4). The text decoder is also usable standalone (text-only) via the Step3p5ForCausalLM architecture.

[9]

Audio modality only supported on E2B/E4B variants.

[10]

Audio requires a checkpoint with a sound_config and is supported only on the full (non-disaggregated) model path, not the EPD disaggregated path.

[11] (1,2)

DeepSeek-V4 is only supported on Blackwell GPUs (SM100+). See the DeepSeek-V4 example README for setup and parallelism.

[12] (1,2,3)

Supports text, image, and video inputs over the block-sparse attention path. The published MXFP8 checkpoint is dequantized on load so the runtime sees an effectively BF16 model. The text decoder is also usable standalone (text-only) via the MiniMaxM3SparseForCausalLM architecture. KV cache reuse and MTP are not supported on the sparse-attention path in this release.

[13]

The Cosmos 3 family also supports visual generation through the VisualGen API. See Visual Generation Models.

[14] (1,2)

Requires transformers>=5.7.0: MiniCPM-V 4.6 was upstreamed into transformers as a native model type (minicpmv4_6) and the checkpoint ships no remote code (auto_map) to fall back on. The Qwen3.5-hybrid text tower runs in BF16. Image, video, and text inputs are supported in this release (video reuses the same NaViT-packed vision path as image via MiniCPMV4_6InputProcessor).