TensorRT Edge-LLM Documentation#
TensorRT Edge-LLM provides optimized inference for text, vision, audio, speech, and action models on NVIDIA edge platforms.
Getting Started
Examples
Integration
Features
- LoRA (Low-Rank Adaptation)
- Quantization
- Select W4A4 or W4A16 for NVFP4 checkpoints
- Select the TensorRT-native MoE layer for Qwen3-family MoE
- Setup
- Quantize An LLM
- Enable FP8 KV Cache
- Keep The Last Layer In FP16
- Quantize A Visual Tower To FP8
- Quantize Qwen3-ASR Audio To FP8
- Calibration Datasets
- Quantize An EAGLE3 Draft
- Quantize An EAGLE-4 Draft
- Quantize A DFlash Draft
- Export NVFP4 MoE for SM12x
- Quantize The Token Embedding Table
- Supported Methods
- Notes
- Multi-Device Inference
- Vocabulary Reduction
- FP8 KV Cache
- FP8 Embedding
- Streaming Output
- Guided Decoding
- Visual-Token Pruning (DART)
- KV Cache Reuse
- In-Flight Batching
- CodePredictor Speculative Decoding
- Skip-Softmax Attention
Input & Chat Format
Performance
Software Design
Testing