Supported Models#

Supported checkpoint IDs are listed below. See the support matrix for platform coverage and the examples for workflows.

Support Policy#

TensorRT Edge-LLM supports the checkpoint IDs listed below. Dense LLM families include official dense checkpoints below 30B parameters. Larger dense checkpoints and non-dense variants require case-by-case validation. MoE, multimodal, audio, TTS, omni, EAGLE3, EAGLE-4, DFlash, JetSpec, and Domino support is limited to the listed rows.

The model coverage list is not comprehensive, and not every listed checkpoint has been fully verified on every supported platform and precision. If a listed model does not export, build, or run correctly, please report an issue with the checkpoint ID, precision, platform, and command line used.

The model class names were checked against the upstream Transformers model source tree. Checkpoint IDs are linked to their Hugging Face pages and grouped into original checkpoints and quantized checkpoints. All links below are public or gated; run hf auth login after accepting the provider’s terms for a gated checkpoint.

Precision Notes#

  • Dense precision set: FP16/BF16; ModelOpt FP8/MXFP8/FP4/NVFP4/INT4 AWQ/INT8 SmoothQuant; AWQ/AutoAWQ W4A16; GPTQ/GPTQModel/AutoRound W4A16 and W8A16; and compressed-tensors W4A16, W8A16, W8A8, FP8-A16, block FP8, and NVFP4 checkpoints. Checkpoints marked (checkpoint-direct only) require the checkpoint-direct builder.

  • Jetson Orin supports FP16, INT8, and INT4 model precision in the supported JetPack configurations. Do not select FP8, MXFP8, FP4, or NVFP4 checkpoints for Orin. FP8 KV cache is supported with FP16 attention.

  • For INT4 engine builds on Jetson Orin devices with less system memory, such as Jetson Orin Nano, pass --externalize-weights int4_ffn for dense checkpoints or --externalize-weights int4_ffn int4_moe for MoE checkpoints to reduce engine build memory.

  • For FP16/BF16 source checkpoints, use the Quantization script to create a unified quantized checkpoint for tensorrt_edgellm, then export the generated checkpoint.

  • FP8 KV cache is detected automatically from checkpoint metadata by tensorrt_edgellm.

  • tensorrt-edgellm-export exports visual encoders. Use tensorrt-edgellm-quantize llm --visual-quantization fp8 before export when FP8 visual weights are required.

  • Native block FP8, MXFP8, and FP4/NVFP4 execution requires Blackwell-class hardware. Per-tensor FP8 requires Hopper-class hardware. On earlier GPUs, the checkpoint-direct builder lowers compatible checkpoints to W8A16 or W4A16 as described in Target Precision Fallback.

  • For platform-specific NVFP4 MoE layouts, follow Export NVFP4 MoE for SM12x before building the engine.

Text Generation#

Llama 3.x

model_type: llama; text input to text output.

Original:

Pre-quantized:

Mistral

model_type: mistral; text input to text output.

Pre-quantized:

Qwen2 / Qwen2.5

model_type: qwen2; text input to text output.

Original:

Pre-quantized:

Qwen3

model_type: qwen3 or qwen3_moe; text input to text output.

Original:

Pre-quantized:

HunYuan V1 Dense

model_type: hunyuan_v1_dense; text input to text output. These checkpoints apply per-head QK RMSNorm after RoPE and use the DynamicNTKAlpha RoPE variant (rope_scaling.type == "dynamic" with alpha); both are handled automatically by export and runtime.

Original:

Qwen vision-language families

model_type: qwen2_5_vl or qwen3_vl; text and image/video input to text output.

Qwen2.5-VL:

Qwen3-VL and compatible checkpoints:

Qwen3.5 / Qwen3.6 / Qwen3.8

model_type: qwen3_5 or qwen3_5_moe; accepts text and image/video input and produces text.

model_type: qwen4_exp (Qwen3.8-Flash-Next) accepts text and image/video input and produces text. Recommend to use the experimental builder rather than the ONNX export path: it is much faster and needs far less memory for this model. On Jetson Thor the 47.7 GiB n-gram table only fits with --ple-residency cached.

InternVL and Phi-4

InternVL uses model_type: internvl_chat or internvl; Phi-4 Multimodal uses phi4mm or phi4_multimodal. Both accept text and image input and produce text; Phi-4 also accepts audio.

InternVL:

Phi-4 Multimodal:

Nemotron-3 / 3.5

Nemotron-H language checkpoints use model_type: nemotron_h and accept text input. Nemotron Omni uses NemotronH_Nano_Omni_Reasoning_V3 and accepts text, image/video, and audio. Nemotron-3.5-Bolt-74B-A9B uses NemotronH_Omni_Reasoning_V3 with a Nemotron-H Puzzle language model (nemotron_h_puzzle) and accepts text and image/video. All produce text.

Nemotron-H:

Nemotron Omni:

Gemma4

model_type: gemma4 or gemma4_unified; accepted modalities depend on the checkpoint and all variants produce text. Paired gemma4_assistant or gemma4_unified_assistant checkpoints provide MTP.

Text, image, and audio input:

Text and image input:

MTP assistants:

DiffusionGemma

model_type: diffusion_gemma or diffusiongemma; text and image input to text output through block-diffusion decoding.

Muse-Glimmer

model_type: muse_glimmer; text, image, or video input to text output.

The FP16 checkpoint supports text, image, and video input; the NVFP4 checkpoint is text-only. Compatible DFlash and DFlash2 drafts are listed under Speculative Draft Checkpoints.

Speech Recognition#

Speech Generation#

Video Generation#

Action and Multimodal Reasoning#

Speculative Draft Checkpoints#

EAGLE3 Draft Models#

EAGLE-4 Draft Models#

EAGLE-4 drafters run in the EAGLE3 speculative decoding mode. Compared with an EAGLE3 draft, the draft normalizes each base hidden state before fusing them, normalizes its output state, uses sliding-window attention (4096 tokens for the listed drafter), and replaces the full-vocabulary output with a low-rank candidate head. Export them with tensorrt-edgellm-export, or build them with the ONNX-less builder, as shown in EAGLE-4.

The EAGLE-4 draft weights are distributed under the Eagle 4 Source-Available License, not under the TensorRT Edge-LLM license. Review the license on the checkpoint page before you download or deploy the weights.

MTP Draft Heads#

Qwen3.5 / Qwen3.6 / Qwen3.8 and Nemotron-H checkpoints embed their MTP head. Dense Qwen3 checkpoints need a separately trained head placed in the base checkpoint directory (mtp.* tensors, num_nextn_predict_layers: 1).

DFlash Draft Models#

DFlash2 Draft Models#

DFlash2 uses the public DFlash engine/runtime mode. The draft checkpoint selects the versioned contract, while the runtime may choose any block size covered by the engine profile. DFlash2 supports chain verification and greedy DDTree verification with --spec-draft-top-k up to the checkpoint selector_top_k.

Draft checkpoint

Base model

RedHatAI/Qwen3-8B-speculator.dflash2

Qwen/Qwen3-8B, including matched NVFP4 or INT4 quantized checkpoints

z-lab/Qwen3.8-27B-DFlash2

Qwen/Qwen3.8-27B, including matched NVFP4 or INT4 quantized checkpoints

incoai/Muse-Glimmer-30B-DFlash2

meta-models/Muse-Glimmer-30B or RadixArk/Muse-Glimmer-NVFP4

LiLiCorr Draft Models#

LiLiCorr drafts are DFlash V2 checkpoints whose dflash_config carries projector_type: "lilicorr". They are detected automatically by --dflash-draft and by the checkpoint-direct builder’s --spec-type dflash. The draft engine scores a lattice of candidate tokens instead of using the DFlash2 Candidate Selector, and the runtime verifies one linear chain. See LiLiCorr for the export, build, and run commands.

DSpark Draft Models#

The Nemotron-3.5-Lightning DSpark draft differs from the DeepSpec block7 drafts above: it uses a block size of 8, sliding-window attention (1024) with a learned per-head attention sink, and causal proposal attention. The sliding window is baked into the draft engine as the contiguous-query XQA variant. The Nemotron draft supports both chain decoding and greedy DSpark DDTree; tree mode reconstructs path-dependent recurrent state. The published checkpoint uses sample_from_anchor=false: slot 0 is anchor-only and proposals start at slot 1. The runtime also supports the older anchor-sampled layout. Tree mode uses --spec-draft-top-k > 1 with a tree-capable base.

JetSpec Draft Models#

JetSpec draft checkpoints are detected by jetspec_config in config.json and exported with DFlashDraftModel using causal proposal attention. The validated runtime path is branching tree verification: export the base with --jetspec-base --jetspec-draft-dir <draft_checkpoint>, export the draft with --jetspec-draft --jetspec-draft-dir <draft_checkpoint>, and run with --spec-draft-top-k > 1. Use --spec-proposal-tokens for the usable proposal depth and --spec-verify-tokens for the root-inclusive target node budget.

For the listed Qwen3 pairs, disable thinking mode in the input JSON when evaluating accuracy, acceptance rate, or throughput. The public drafts carry the JetSpec contract in dflash_config with causal_head: true; the exporter accepts that form as well as jetspec_config. On MoE bases a verification step touches most experts, so small trees are fastest: use --specVerifyTokens 16 to 32 with --specDraftTopK 7.

Domino Draft Models#

Domino draft checkpoints are DFlash checkpoints whose dflash_config carries projector_type: "domino" plus the GRU correction-head contract (gru_hidden_dim, emb_dim, pure_draft_prefix_len). They are detected automatically by --dflash-draft; --domino-base --domino-draft-dir <draft_checkpoint> and --domino-draft --domino-draft-dir <draft_checkpoint> are the explicit spellings. The draft export also writes domino_heads.safetensors, the runtime sidecar for the correction rollout. Domino supports greedy decoding only; --specDraftTopK 1 runs the block-16 chain and --specDraftTopK > 1 the DDTree on the same engines. Sampling is not supported. The base and draft each accept FP16, FP8, NVFP4, or INT4-AWQ checkpoints; quantize the BF16 public drafts with tensorrt-edgellm-quantize draft.