Quantization#
Use trtedge quantize when you start from an FP16/BF16 checkpoint and need to create a unified quantized checkpoint for tensorrt_edgellm.
The quantization CLI writes a unified HuggingFace-style checkpoint directory that tensorrt_edgellm can export directly.
Skip this step when you already have a supported pre-quantized HuggingFace checkpoint.
Select W4A4 or W4A16 for NVFP4 checkpoints#
ModelOpt W4A4 checkpoints can run the same packed NVFP4 weights with NVFP4 or FP16 activations. Export without an activation override for W4A4:
trtedge export /path/to/modelopt_nvfp4 /tmp/model_w4a4 \
--target-sm 110
trtedge export /path/to/modelopt_nvfp4 /tmp/model_w4a16 \
--no-quantize-activations --target-sm 110
The default uses the checkpoint’s activation scales and the W4A4 GEMM/MoE
path. --no-quantize-activations keeps activations in FP16 and selects the
W4A16 GEMM/MoE path. Both modes reuse the checkpoint’s packed weights.
The checkpoint-direct builder uses the same selection:
trtedge build --model-dir /path/to/modelopt_nvfp4 \
--engine-dir /tmp/engine_w4a16 --no-quantize-activations
Select the TensorRT-native MoE layer for Qwen3-family MoE#
Qwen3-family MoE experts run through the Edge-LLM MoE plugins by default.
--trt-native-moe selects TensorRT’s native MoE layer instead, on both
frontends:
trtedge export /path/to/qwen3_moe /tmp/model_native_moe \
--trt-native-moe --target-sm 100
trtedge build --model-dir /path/to/qwen3_moe \
--engine-dir /tmp/engine_native_moe --trt-native-moe
The precision follows the checkpoint: FP16/BF16 experts use the FP16 layer,
and ModelOpt NVFP4 experts use W4A8 (the checkpoint’s FP4 weights with static
FP8 activations from the calibrated scales). The switch is opt-in and covers
only these two forms: INT4, NVFP4-A16 (--no-quantize-activations) and
Nemotron-H experts are rejected rather than silently built with the plugins.
It needs TensorRT 11.3 or newer and an SM 10.x or 11.x device; performance
relative to the plugins is tracked separately.
--trt-native-moe-w4a4 (export only, on top of --trt-native-moe) switches
ModelOpt NVFP4 experts from W4A8 to W4A4, TensorRT’s MoE quantization mode 2:
the hidden states and the down-projection input are quantized dynamically to
NVFP4 (blocks of 16 with FP8 block scales; ModelOpt’s calibrated input_scale
is the second-level scale, used as is) and the FP4 weights keep the
checkpoint’s E4M3 block scales with a per-expert FP16 global. It needs
TensorRT 11.6 or newer (IMoELayer::setQuantizationDynamicDblQ) and is
rejected for unquantized experts and for --no-quantize-activations. It is
the OOTB counterpart of the plugins’ W4A4 recipe for like-for-like
comparisons.
Setup#
Complete Installation and add the
quantize and onnx extras described in
Optional Python dependencies.
This provides the dependencies for the quantization and ONNX export examples
below. Then verify the command:
trtedge quantize --help
Quantize An LLM#
trtedge quantize llm \
--model-dir Qwen/Qwen3-0.6B \
--output-dir /tmp/qwen3_0_6b_nvfp4 \
--quantization nvfp4 \
--lm-head-quantization nvfp4
Calibration uses the default cnn_dailymail text dataset. Pick another with
--text-dataset <name> (see Calibration Datasets).
The output directory is a HuggingFace-style checkpoint that tensorrt_edgellm can export directly. See Quick Start Guide for the full export-build-inference workflow.
Enable FP8 KV Cache#
trtedge quantize llm \
--model-dir Qwen/Qwen3-8B \
--output-dir /tmp/qwen3_nvfp4_fp8kv \
--quantization nvfp4 \
--kv-cache-quantization fp8
When this checkpoint is exported with tensorrt_edgellm, FP8 KV cache is detected from the checkpoint metadata automatically. See FP8 KV Cache for details on FP8 KV cache behavior and platform requirements.
Keep The Last Layer In FP16#
A 4-bit GEMM writes its output in FP16. When the last decoder layer produces
activations beyond the FP16 range (65504), those outputs saturate and accuracy
collapses. --keep-last-layer-fp16 leaves the last layer’s o_proj and MLP
projections (gate_proj, up_proj, down_proj) unquantized. It applies to
llm and to DFlash-family draft quantization; EAGLE drafts reject it.
trtedge quantize draft \
--base-model-dir /path/to/gemma4_target \
--draft-model-dir /path/to/qwen3_dflash_draft \
--output-dir /tmp/dflash_draft_nvfp4 \
--quantization nvfp4 \
--keep-last-layer-fp16
Quantize A Visual Tower To FP8#
For supported VLM checkpoints, the standalone quantizer can also calibrate the visual tower to FP8. Use a multimodal calibration dataset so the visual path sees real image activations:
trtedge quantize llm \
--model-dir Qwen/Qwen3-VL-2B-Instruct \
--output-dir /tmp/qwen3_vl_fp8_visual \
--quantization nvfp4 \
--lm-head-quantization nvfp4 \
--visual-quantization fp8 \
--image-dataset mmmu
--image-dataset selects the image-question calibration set (default mmmu);
the LLM path still uses --text-dataset.
Quantize Qwen3-ASR Audio To FP8#
Qwen3-ASR calibration streams audio-transcript pairs through the audio tower
and text decoder. Use an audio dataset with audio and text columns:
trtedge quantize llm \
--model-dir /path/to/qwen3_asr \
--output-dir /tmp/qwen3_asr_fp8_audio \
--quantization fp8 \
--audio-quantization fp8 \
--audio-dataset librispeech
Calibration Datasets#
Calibration data is selected by name, per modality — the CLI passes only a dataset name. Each name maps to a registered dataset implementation; an unknown name fails out with the list of available datasets and a pointer to the customization guide (it does not silently fall back).
Modality |
Flag (default) |
Built-in names |
Used for |
|---|---|---|---|
Text |
|
|
LLM weights, LM head, KV cache, CodePredictor, MTP, EAGLE3, EAGLE-4, DFlash |
Image |
|
|
Visual tower ( |
Audio |
|
|
Qwen3-ASR / audio tower |
To calibrate on your own data, write a generator for your modality and
register it under a name — see
Calibration Dataset Customization.
To run a built-in offline (e.g. CI), point it at a cached copy with the
EDGELLM_QUANT_DATASET_<NAME> environment variable; the CLI stays name-only.
Quantize An EAGLE3 Draft#
trtedge quantize draft \
--base-model-dir /path/to/base_model \
--draft-model-dir /path/to/eagle3_draft \
--output-dir /tmp/eagle3_draft_fp8 \
--quantization fp8
Quantize An EAGLE-4 Draft#
trtedge quantize draft \
--base-model-dir /path/to/bf16_base_model \
--draft-model-dir /path/to/eagle4_draft \
--output-dir /tmp/eagle4_draft_nvfp4 \
--quantization nvfp4
The draft command treats a draft whose configuration sets fc_norm,
norm_output, or a low-rank candidate head as an EAGLE-4 drafter. It
quantizes the decoder-layer linears and keeps fc in FP16, as for DFlash
drafts, and records it in exclude_modules. The KV cache and the candidate
head factors also stay FP16, so --kv-cache-quantization is rejected; with a
candidate head, lm_head stays FP16 and --lm-head-quantization is rejected
too. For NVFP4-A16, quantize with --quantization nvfp4 and export with
--no-quantize-activations; --quantization nvfp4_a16_gptq is rejected for
these drafts.
Calibration runs the base model only up to the deepest layer that the drafter
reads. For EAGLE drafts the command defaults to --quantization fp8
and for DFlash drafts to nvfp4, so name the precision explicitly.
Quantize A DFlash Draft#
trtedge quantize draft \
--base-model-dir /path/to/base_model \
--draft-model-dir /path/to/dflash_draft \
--output-dir /tmp/dflash_draft_nvfp4 \
--quantization nvfp4 \
--lm-head-quantization nvfp4
The draft command auto-detects DFlash from the draft checkpoint’s
dflash_config. The base model is used during calibration to produce target
hidden states and as an LM-head fallback for draft checkpoints that do not
store lm_head weights. DFlash draft quantization has been validated with
NVFP4. The target-hidden projector stays in dense precision; export will
reject checkpoints where that fc module is quantized.
Export NVFP4 MoE for SM12x#
NVFP4 MoE weights use different export layouts on SM120/SM121 and SM100/SM101/SM110. Select the SM12x layout before exporting for DGX Spark (SM121) or an SM120 GPU:
export EDGELLM_NVFP4_MOE_TARGET=sm12x
trtedge export \
/path/to/nvfp4_moe_checkpoint \
/tmp/nvfp4_moe_sm12x_onnx
Without this setting, export uses the SM100/SM101/SM110 layout. The selected weight layout is fixed in the exported artifacts and cannot be changed during engine build or inference. Use a separate export directory for each layout.
Quantize The Token Embedding Table#
Quantize the canonical token table independently of the backbone:
trtedge quantize llm \
--model_dir /path/to/model \
--output_dir /tmp/model_nvfp4_embedding \
--quantization nvfp4 \
--token_embedding_quantization nvfp4
The supported methods are fp8 and nvfp4. FP8 uses one FP32 scale for the
entire table, matching PLE and export-time FP8. NVFP4 uses a separate scale for
each token and 16-hidden block. Both layouts are consumed directly by the embedding lookup
kernel. Gemma4’s scaled token embedding is supported; its output multiplier is
preserved in the runtime artifact. If token embeddings and the LM head share
weights, quantization separates their storage so each can use its own method.
The LM head can remain in FP16.
For export-time token quantization, use trtedge export with
--fp8-embedding. This path uses the same per-tensor FP8 semantics and
preserves the source checkpoint. A checkpoint with quantized token embeddings
already supplies its runtime format; omit --fp8-embedding for that checkpoint.
Token and PLE FP8 lookup use the same dequantization semantics:
FP16(FP32(value) * scale). Model embedding multipliers are folded into the
FP32 scale. llm_inference --offloadEmbedding keeps the token artifact in
CUDA-pinned host memory; --pleResidency cached --pleCacheSlots 4096 keeps
PLE file-backed with a bounded GPU row cache. PLE uses the existing gather
plugin followed by standard dequantization. Offloading does not change these
operations or the engine contract.
The scalar scale is independent of row placement, so moving compressed rows
between storage and the cache does not change dequantization.
Gemma4 E2B/E4B PLE and Qwen4-Exp n-gram tables support per-tensor FP8 through either path:
Quantize time: Add
--ple_embedding_quantization fp8totrtedge quantize llm, then export the checkpoint normally.Export time: Add
--fp8-pletotrtedge export. The exporter writes a quantized table under the output directory’sple/subdirectory and records that directory incheckpoint_dir. Keep this directory accessible during inference, including when using a cached PLE pool.
For example, quantize both tables while exporting a dense Gemma4 checkpoint:
trtedge export /path/to/gemma4 /tmp/gemma4_onnx \
--skip-visual --skip-audio --fp8-embedding --fp8-ple
If you move the PLE artifact to another device, pass
--pleFile /path/to/ple/model.safetensors to llm_inference.
Both PLE paths preserve Gemma4’s embedding multiplier and use an FP8 pool input with dequantization inside the engine. Token and PLE options can be selected independently.
Supported Methods#
Component |
Methods |
|---|---|
Backbone |
|
LM head |
|
Token embedding |
|
Gemma4 PLE embedding |
|
KV cache |
|
Visual tower |
|
Audio tower |
|
CodePredictor |
|
DFlash draft |
validated with |
Notes#
The package writes unified checkpoints only. It does not export ONNX or build TensorRT engines.
GPTQ checkpoints are loaded as pre-quantized checkpoints; this package does not create GPTQ models.