TensorRT Edge-LLM Documentation#
TensorRT Edge-LLM provides optimized inference for text, vision, audio, speech, and action models on NVIDIA edge platforms.
Getting Started
Examples
Features
- LoRA (Low-Rank Adaptation)
- Quantization
- Select W4A4 or W4A16 for NVFP4 checkpoints
- Setup
- Quantize An LLM
- Enable FP8 KV Cache
- Quantize A Visual Tower To FP8
- Quantize Qwen3-ASR Audio To FP8
- Calibration Datasets
- Quantize An EAGLE3 Draft
- Quantize A DFlash Draft
- Export NVFP4 MoE for SM12x
- Quantize Embedding Table To FP8
- Supported Methods
- Notes
- Multi-Device Inference
- Vocabulary Reduction
- FP8 KV Cache
- FP8 Embedding
- Streaming Output
- Guided Decoding
- Visual-Token Pruning (DART)
- KV Cache Reuse
- In-Flight Batching
- CodePredictor Speculative Decoding
Input & Chat Format
Performance
Software Design
Testing
APIs