Supported Models

Code Location: tensorrt_edgellm/ (checkpoint export), tensorrt_edgellm/quantization/ (checkpoint quantization), experimental/server/ (Python API/server), cpp/ (runtime)

Pre-Quantized Checkpoints: When a supported pre-quantized checkpoint is available, the checkpoint exporter can export it directly without a separate quantization step.

Support Policy

TensorRT Edge-LLM supports the checkpoint IDs listed below. Dense LLM families include official dense checkpoints below 30B parameters. Larger dense checkpoints and non-dense variants require case-by-case validation. MoE, multimodal, audio, TTS, omni, EAGLE3, and DFlash support is limited to the listed rows.

The model coverage list is not comprehensive, and not every listed checkpoint has been fully verified on every supported platform and precision. If a listed model does not export, build, or run correctly, please report an issue with the checkpoint ID, precision, platform, and command line used.

The model class names were checked against the installed transformers==5.9.0 package and the upstream Transformers model source tree. Checkpoint IDs are linked to their Hugging Face pages and grouped into original checkpoints and quantized checkpoints.

Precision Notes

• Dense precision set: FP16/BF16 checkpoints, ModelOpt FP8/MXFP8/FP4/NVFP4/INT4 AWQ/INT8 SmoothQuant checkpoints, and INT4 GPTQ checkpoints. INT8 GPTQ is not supported.

• Jetson Orin supports FP16, INT8, and INT4 runtime precision in the supported JetPack configurations. Do not select FP8, MXFP8, FP4, or NVFP4 checkpoints for Orin.

• For INT4 engine builds on Jetson Orin devices with less system memory, such as Jetson Orin Nano, pass --externalize-weights int4_ffn for dense checkpoints or --externalize-weights int4_ffn int4_moe for MoE checkpoints to reduce engine build memory.

• For FP16/BF16 source checkpoints, use the Quantization script to create a unified quantized checkpoint for tensorrt_edgellm, then export the generated checkpoint.

• FP8 KV cache is detected automatically from checkpoint metadata by tensorrt_edgellm.

• tensorrt-edgellm-export exports visual encoders. Use tensorrt-edgellm-quantize llm --visual_quantization fp8 before export when FP8 visual weights are required.

• MXFP8 and FP4/NVFP4 require Blackwell-class hardware for runtime execution.

Support Matrix

Dense LLM

Model Series	Transformers Class	tensorrt_edgellm Handling	Supported Precisions
Llama 3.x Instruct	LlamaForCausalLM	llama -> default CausalLM	Dense precision set
Qwen2/Qwen2.5 dense	Qwen2ForCausalLM	qwen2 -> default CausalLM	Dense precision set
Qwen3 dense	Qwen3ForCausalLM	qwen3 -> default CausalLM	Dense precision set
Qwen3.5/3.6 text	Qwen3_5ForCausalLM	qwen3_5_text -> Qwen3_5CausalLM	Dense precision set
Gemma4 E2B/E4B text	Gemma4ForCausalLM	gemma4 / gemma4_text -> text decoder with PLE and dual-RoPE support	BF16/FP16 source checkpoints; no image/audio/video/MTP
Nemotron Nano dense	NemotronHForCausalLM	nemotron_h -> NemotronHCausalLM	BF16, FP8, NVFP4

Llama 3.x Instruct checkpoints

Original:

• meta-llama/Meta-Llama-3-8B-Instruct

• meta-llama/Llama-3.1-8B-Instruct

• meta-llama/Llama-3.2-1B-Instruct

• meta-llama/Llama-3.2-3B-Instruct

Quantized:

• nvidia/Llama-3.1-8B-Instruct-FP8

• nvidia/Llama-3.1-8B-Instruct-NVFP4

Qwen2/Qwen2.5 dense and Qwen-derived dense checkpoints

Original:

• Qwen/Qwen2-0.5B

• Qwen/Qwen2-0.5B-Instruct

• Qwen/Qwen2-1.5B

• Qwen/Qwen2-1.5B-Instruct

• Qwen/Qwen2-7B

• Qwen/Qwen2-7B-Instruct

• Qwen/Qwen2-Math-1.5B

• Qwen/Qwen2-Math-1.5B-Instruct

• Qwen/Qwen2-Math-7B

• Qwen/Qwen2-Math-7B-Instruct

• Qwen/Qwen2.5-0.5B

• Qwen/Qwen2.5-0.5B-Instruct

• Qwen/Qwen2.5-1.5B

• Qwen/Qwen2.5-1.5B-Instruct

• Qwen/Qwen2.5-3B

• Qwen/Qwen2.5-3B-Instruct

• Qwen/Qwen2.5-7B

• Qwen/Qwen2.5-7B-Instruct

• Qwen/Qwen2.5-14B

• Qwen/Qwen2.5-14B-Instruct

• Qwen/Qwen2.5-Coder-0.5B

• Qwen/Qwen2.5-Coder-0.5B-Instruct

• Qwen/Qwen2.5-Coder-1.5B

• Qwen/Qwen2.5-Coder-1.5B-Instruct

• Qwen/Qwen2.5-Coder-3B

• Qwen/Qwen2.5-Coder-3B-Instruct

• Qwen/Qwen2.5-Coder-7B

• Qwen/Qwen2.5-Coder-7B-Instruct

• Qwen/Qwen2.5-Coder-14B

• Qwen/Qwen2.5-Coder-14B-Instruct

• Qwen/Qwen2.5-Math-1.5B

• Qwen/Qwen2.5-Math-1.5B-Instruct

• Qwen/Qwen2.5-Math-7B

• Qwen/Qwen2.5-Math-7B-Instruct

• deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B

• deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

• deepseek-ai/DeepSeek-R1-Distill-Qwen-14B

Quantized:

• Qwen/Qwen2-0.5B-Instruct-AWQ

• Qwen/Qwen2-0.5B-Instruct-GPTQ-Int4

• Qwen/Qwen2-1.5B-Instruct-AWQ

• Qwen/Qwen2-1.5B-Instruct-GPTQ-Int4

• Qwen/Qwen2-7B-Instruct-AWQ

• Qwen/Qwen2-7B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-0.5B-Instruct-AWQ

• Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-1.5B-Instruct-AWQ

• Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-3B-Instruct-AWQ

• Qwen/Qwen2.5-3B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-7B-Instruct-AWQ

• Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-14B-Instruct-AWQ

• Qwen/Qwen2.5-14B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-Coder-0.5B-Instruct-AWQ

• Qwen/Qwen2.5-Coder-0.5B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-Coder-1.5B-Instruct-AWQ

• Qwen/Qwen2.5-Coder-1.5B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-Coder-3B-Instruct-AWQ

• Qwen/Qwen2.5-Coder-3B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-Coder-7B-Instruct-AWQ

• Qwen/Qwen2.5-Coder-7B-Instruct-GPTQ-Int4

• Qwen/Qwen2.5-Coder-14B-Instruct-AWQ

• Qwen/Qwen2.5-Coder-14B-Instruct-GPTQ-Int4

Qwen3 dense checkpoints

Original:

• Qwen/Qwen3-0.6B

• Qwen/Qwen3-0.6B-Base

• Qwen/Qwen3-1.7B

• Qwen/Qwen3-1.7B-Base

• Qwen/Qwen3-4B

• Qwen/Qwen3-4B-Base

• Qwen/Qwen3-4B-Instruct-2507

• Qwen/Qwen3-4B-Thinking-2507

• Qwen/Qwen3-8B

• Qwen/Qwen3-8B-Base

• Qwen/Qwen3-14B

• Qwen/Qwen3-14B-Base

Quantized:

• Qwen/Qwen3-4B-AWQ

• Qwen/Qwen3-8B-AWQ

• Qwen/Qwen3-14B-AWQ

• nvidia/Qwen3-8B-FP8

• nvidia/Qwen3-8B-NVFP4

• nvidia/Qwen3-14B-FP8

• nvidia/Qwen3-14B-NVFP4

Qwen3.5/3.6 text checkpoints

Qwen3.5:

• Qwen/Qwen3.5-0.8B

• Qwen/Qwen3.5-0.8B-Base

• Qwen/Qwen3.5-2B

• Qwen/Qwen3.5-2B-Base

• Qwen/Qwen3.5-4B

• Qwen/Qwen3.5-4B-Base

• Qwen/Qwen3.5-9B

• Qwen/Qwen3.5-9B-Base

• Qwen/Qwen3.5-27B

Qwen3.6 (same architecture as Qwen3.5):

• Qwen/Qwen3.6-27B

Nemotron Nano dense checkpoints

Original:

• nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

• nvidia/NVIDIA-Nemotron-Nano-9B-v2

Quantized:

• nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8

• nvidia/NVIDIA-Nemotron-3-Nano-4B-NVFP4

• nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8

• nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4

---

MoE

Model Series	Transformers Class	tensorrt_edgellm Handling	Supported Precisions
Qwen3-MoE	Qwen3MoeForCausalLM	qwen3_moe -> Qwen3MoeCausalLM	INT4, NVFP4
Qwen3.5/3.6-MoE	Qwen3_5MoeForConditionalGeneration	qwen3_5_moe -> Qwen3_5MoeCausalLM + Qwen3_5VLVisualModel	INT4 GPTQ, NVFP4
Nemotron3-MoE	NemotronHForCausalLM	nemotron_h -> NemotronHCausalLM	NVFP4 only
Nemotron3 Super 120B-A12B	NemotronHForCausalLM	nemotron_h -> NemotronHCausalLM with latent MoE routed path	NVFP4 only

NVFP4 MoE export picks the plugin and FC1 weight layout from EDGELLM_NVFP4_MOE_TARGET; see MoE Example.

Qwen3-MoE checkpoints

• Qwen/Qwen3-30B-A3B-GPTQ-Int4

• nvidia/Qwen3-30B-A3B-NVFP4

Qwen3.5/3.6-MoE checkpoints

• Qwen/Qwen3.5-35B-A3B-GPTQ-Int4

• nvidia/Qwen3.6-35B-A3B-NVFP4

Nemotron3-MoE checkpoints

• nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4

• nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

Nemotron3 Super uses latent MoE: routing is computed from the model hidden states, while the routed expert payload is projected to moe_latent_size before the NVFP4 MoE plugin path. The shared expert path remains separate.

---

VLM

Model Series	Transformers Class	tensorrt_edgellm Handling	Supported Precisions
Qwen2.5-VL	Qwen2_5_VLForConditionalGeneration	qwen2_5_vl + Qwen2_5VLVisualModel	Dense precision set for LLM backbone
Qwen3-VL / compatible	Qwen3VLForConditionalGeneration	qwen3_vl + Qwen3VLVisualModel	Dense precision set for LLM backbone
Qwen3.5/3.6 VLM	Qwen3_5ForConditionalGeneration	qwen3_5 -> Qwen3_5CausalLM + Qwen3_5VLVisualModel	VLM original checkpoints only
InternVL3 / InternVL3.5 HF format	InternVLForConditionalGeneration	internvl_chat / internvl + InternVL visual models	Dense precision set for LLM backbone
Phi-4-Multimodal	Phi4MultimodalForCausalLM	phi4mm / phi4_multimodal + Phi4MMVisualModel	Merge vision LoRA, then dense precision set for the LLM backbone

Qwen2.5-VL checkpoints

Original:

• Qwen/Qwen2.5-VL-3B-Instruct

• Qwen/Qwen2.5-VL-7B-Instruct

Quantized:

• Qwen/Qwen2.5-VL-3B-Instruct-AWQ

• Qwen/Qwen2.5-VL-7B-Instruct-AWQ

• nvidia/Qwen2.5-VL-7B-Instruct-FP8

• nvidia/Qwen2.5-VL-7B-Instruct-NVFP4

Qwen3-VL / compatible checkpoints

Original:

• Qwen/Qwen3-VL-2B-Instruct

• Qwen/Qwen3-VL-2B-Thinking

• Qwen/Qwen3-VL-4B-Instruct

• Qwen/Qwen3-VL-4B-Thinking

• Qwen/Qwen3-VL-8B-Instruct

• Qwen/Qwen3-VL-8B-Thinking

• nvidia/Cosmos-Reason2-2B

• nvidia/Cosmos-Reason2-8B

Quantized:

• nvidia/Cosmos-Reason2-2B-FP8

• nvidia/Cosmos-Reason2-2B-NVFP4

• nvidia/Cosmos-Reason2-8B-FP8

• nvidia/Cosmos-Reason2-8B-NVFP4

Qwen3.5/3.6 VLM — same checkpoints as Qwen3.5/3.6 text

Qwen3.5 and Qwen3.6 checkpoints are unified text+VLM models. The same checkpoints listed under Qwen3.5/3.6 text are used; tensorrt_edgellm selects the VLM path (qwen3_5 handler) when visual inputs are provided.

InternVL3 / InternVL3.5 HF format checkpoints

Original:

• OpenGVLab/InternVL3-1B-hf

• OpenGVLab/InternVL3-2B-hf

• OpenGVLab/InternVL3-8B-hf

• OpenGVLab/InternVL3-9B

• OpenGVLab/InternVL3-9B-Instruct

• OpenGVLab/InternVL3-14B-hf

• OpenGVLab/InternVL3_5-1B-HF

• OpenGVLab/InternVL3_5-2B-HF

• OpenGVLab/InternVL3_5-4B-HF

• OpenGVLab/InternVL3_5-8B-HF

• OpenGVLab/InternVL3_5-14B-HF

Quantized:

• OpenGVLab/InternVL3-1B-AWQ

• OpenGVLab/InternVL3-2B-AWQ

• OpenGVLab/InternVL3-8B-AWQ

• OpenGVLab/InternVL3-14B-AWQ

Phi-4-Multimodal checkpoints

• microsoft/Phi-4-multimodal-instruct

---

VLA

Model Series	Transformers Class	tensorrt_edgellm Handling	Supported Precisions
Alpamayo R1	Checkpoint architecture alpamayo_r1; VLM backbone compatible with Qwen3VLForConditionalGeneration	qwen3_vl + Qwen3VLVisualModel + AlpamayoAction	FP16

Alpamayo R1 checkpoints

• nvidia/Alpamayo-R1-10B

---

Audio / Speech

Model Series	Transformers Class	tensorrt_edgellm Handling	Supported Precisions
Qwen3-ASR	Checkpoint architecture Qwen3ASRForConditionalGeneration; text backbone compatible with Qwen3ForCausalLM	Qwen3ASRLanguageModel + QwenAudioEncoder	FP16; FP8 LLM (optional FP8 audio); NVFP4 LLM (optional FP8 audio; see ASR example)

Qwen3-ASR checkpoints

• Qwen/Qwen3-ASR-0.6B

• Qwen/Qwen3-ASR-1.7B

---

TTS

Model Series	Transformers Class	tensorrt_edgellm Handling	Supported Precisions
Qwen3-TTS	Checkpoint architecture Qwen3TTSForConditionalGeneration; talker/code-predictor decoders compatible with Qwen3ForCausalLM	TalkerCausalLM + CodePredictorCausalLM + Code2Wav from speech_tokenizer/	FP16

Qwen3-TTS checkpoints

• Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice

• Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice

---

Omni

Model Series	Transformers Class	tensorrt_edgellm Handling	Supported Precisions
Qwen3-Omni	Qwen3OmniMoeForConditionalGeneration	Qwen3OmniMoeThinkerCausalLM + Qwen3OmniMoeTalkerCausalLM + visual/audio/Code2Wav	NVFP4 only
Nemotron-Omni	Checkpoint architecture NemotronH_Nano_Omni_Reasoning_V3; LLM is Nemotron-H compatible with NemotronHForCausalLM	NemotronHCausalLM + NemotronOmniVisualModel + NemotronOmniAudioModel	NVFP4 only

Nemotron-Omni checkpoints

• nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4

---

Qwen3-ASR and Qwen3-TTS use checkpoint architecture names that are not present in the installed transformers==5.9.0 package, so TensorRT Edge-LLM handles their speech/audio/talker/Code2Wav components with local model implementations. Qwen3-TTS support is limited to the CustomVoice checkpoints listed above.

EAGLE3 Draft Models

EAGLE3 draft checkpoints are detected by draft_vocab_size in config.json and exported with Eagle3DraftModel. Draft checkpoints can be quantized with tensorrt-edgellm-quantize using the same ModelOpt methods exposed by the draft quantization CLI: fp8, int4_awq, nvfp4, mxfp8, and int8_sq for the backbone; fp8, int4_awq, nvfp4, and mxfp8 for the LM head; and fp8 for KV cache.

Draft checkpoint	Base model	Draft config class
yuhuili/EAGLE3-LLaMA3.1-Instruct-8B	meta-llama/Llama-3.1-8B-Instruct	LlamaForCausalLM-style draft
AngelSlim/Qwen3-1.7B_eagle3	Qwen/Qwen3-1.7B	LlamaForCausalLMEagle3-style draft
AngelSlim/Qwen3-4B_eagle3	Qwen/Qwen3-4B	Eagle3LlamaForCausalLM-style draft
Tengyunw/qwen3_8b_eagle3	Qwen/Qwen3-8B	LlamaForCausalLMEagle3-style draft
AngelSlim/Qwen3-8B_eagle3	Qwen/Qwen3-8B	LlamaForCausalLMEagle3-style draft
Rayzl/qwen2.5-vl-7b-eagle3-sgl	Qwen/Qwen2.5-VL-7B-Instruct	LlamaForCausalLMEagle3-style draft

DFlash Draft Models

DFlash draft checkpoints are detected by dflash_config in config.json and exported with DFlashDraftModel. DFlash base export uses --dflash-base --dflash-draft-dir <draft_checkpoint>, and draft export uses --dflash-draft --dflash-draft-dir <draft_checkpoint>. DFlash draft checkpoints can be quantized with tensorrt-edgellm-quantize draft; NVFP4 backbone quantization and optional NVFP4 LM-head quantization are validated.

So far DFlash support in TensorRT Edge-LLM is validated for Qwen3 and Qwen3.5 only. Other DFlash draft models in the z-lab collection are not tested for TensorRT Edge-LLM accuracy, acceptance rate, or runtime compatibility. For the listed pairs, match the paired HuggingFace generation behavior when evaluating performance: enable thinking for Qwen3.5 DFlash models and disable thinking for Qwen3 DFlash models.

Draft checkpoint	Base model	Draft config class
z-lab/Qwen3-4B-DFlash-b16	Qwen/Qwen3-4B-Instruct-2507	DFlashDraftModel
z-lab/Qwen3-8B-DFlash-b16	Qwen/Qwen3-8B	DFlashDraftModel
z-lab/Qwen3.5-4B-DFlash	Qwen/Qwen3.5-4B	DFlashDraftModel
z-lab/Qwen3.5-4B-DFlash (quantized checkpoint: Qwen3.5-4B-DFlash-NVFP4)	Qwen3.5-4B-NVFP4	DFlashDraftModel
z-lab/Qwen3.5-9B-DFlash	Qwen/Qwen3.5-9B	DFlashDraftModel
z-lab/Qwen3.5-27B-DFlash	Qwen/Qwen3.5-27B	DFlashDraftModel
z-lab/Qwen3.5-35B-A3B-DFlash	Qwen/Qwen3.5-35B-A3B-GPTQ-Int4	DFlashDraftModel