TensorRT Edge-LLM Documentation

TensorRT Edge-LLM provides optimized inference for text, vision, audio, speech, and action models on NVIDIA edge platforms.

Getting Started

• Overview
  • Deployment Workflows
  • Runtime Capabilities
  • Start Here
• Official Support Matrix
  • Platforms
  • KV Cache Reuse Support
• Supported Models
  • Text Generation
  • Speech Recognition
  • Speech Generation
  • Action and Multimodal Reasoning
  • Speculative Draft Checkpoints
• Installation
  • Part 1: Export and Quantization (x86 Host)
  • Part 2: C++ Runtime (Edge Device)
  • Next Steps
  • Troubleshooting
  • Uninstalling
• Quick Start
  • 1. Export the Checkpoint
  • 2. Build the Engines
  • 3. Run Vision-Language Inference
  • 4. Start the OpenAI-Compatible Server
• Experimental ONNX-less Engine Build
  • Prerequisites
  • Build All Components
  • Run The Engines
  • Speculative Decoding
  • Optional Features
  • Support And Validation Status
  • Build-Time Comparison
• Limitations and Known Issues
  • 0.10.0
  • 0.9.0
  • 0.7.0
  • 0.6.1
  • 0.6.0

Examples

• Examples
  • Text Output
  • Speech Output
  • Action Output
  • Serving and Evaluation
• Speculative Decoding
  • MTP
  • EAGLE3
  • DFlash
  • DSpark
  • ONNX-less Alternative
• Phi-4-Multimodal
  • Step 1: Merge, Quantize, and Export (x86 Host)
  • Step 2: Transfer to Device
  • Step 3: Build Engines (Thor Device)
  • Step 4: Run Inference (Thor Device)
• ASR (Automatic Speech Recognition)
  • Step 1: Export (x86 Host)
  • Step 2: Transfer to Device
  • Step 3: Build Engines (Thor Device)
  • Step 4: Run Inference (Thor Device)
  • Quantization (Optional)
• TTS (Text-to-Speech)
  • Precision & Quantization
  • Part 1: Export on x86 Host
  • Part 2: Build Engines
  • Part 3: Run Inference
• Vision-Language-Action
• Omni (Audio + Vision + Speech I/O)
  • Supported models
  • Part 1: Quantize (Thinker + Talker)
  • Part 2: Export ONNX
  • Part 3: Build Engines
  • Part 4: Preprocess Audio Input
  • Part 5: Run Inference
  • Notes
• Experimental High-Level Python API and Server
  • Prerequisites
  • Python API
  • OpenAI-Compatible Server
  • Common Inputs
  • Audio Input
  • Audio Output (Qwen3-Omni)
  • Text-to-Speech (/v1/audio/speech)
  • Video Input
  • Sampling Parameters
  • Server-Side Batching
  • Request Admission and Backpressure
  • Tool Calls
  • Anthropic Messages API
  • Connecting OpenClaw
  • EAGLE
  • Endpoints
  • Notes
• Evaluate with NeMo Evaluator
  • Install Dependencies
  • Evaluate a Running Server
  • Start the Server Automatically
  • Optional Named Cases
  • Scores and Overrides

Features

• LoRA (Low-Rank Adaptation)
  • Dynamic Runtime LoRA
  • Static LoRA Merge
  • Input Format
  • Script Reference
  • Build Parameters
  • Notes
• Quantization
  • Setup
  • Quantize An LLM
  • Enable FP8 KV Cache
  • Quantize A Visual Tower To FP8
  • Quantize Qwen3-ASR Audio To FP8
  • Calibration Datasets
  • Quantize An EAGLE3 Draft
  • Quantize A DFlash Draft
  • Quantize Embedding Table To FP8
  • Supported Methods
  • Notes
• Vocabulary Reduction
  • Overview
  • Quick Start
  • EAGLE Speculative Decoding Support
  • DFlash Speculative Decoding Support
  • Script Reference
  • Custom Vocabulary Mapping
  • Notes
• FP8 KV Cache
  • Overview
  • Workflow
  • Technical Details
  • Performance: FP8 vs FP16 FMHA on Thor
  • Warning
  • Notes
• FP8 Embedding
  • Overview
  • Workflow
  • Combining with Other Quantization
  • Technical Details
  • Notes
  • Limitations
• Streaming Output
  • Quick start
  • The StreamChunk payload
  • Batching with per-slot opt-in / opt-out
  • Throttling with streamInterval
  • Cancellation
  • Alternative delivery patterns
  • Channel lifetime rules
  • End-to-end demo: llm_stream
  • Common pitfalls
• KV Cache Reuse
  • Build Capacity
  • Enable Reuse
  • Request Policies

Input & Chat Format

• Input JSON Format
  • Basic Structure
  • Parameters
  • Examples
  • Notes
• Chat Template Format
  • Overview
  • File Structure
  • Thinking Mode
  • Custom Templates
  • Examples
  • System Prompt Priority

Performance

• Performance Benchmarks
  • Definitions
  • Reproducing Benchmark Runs
  • v0.9.0 Results
  • v0.8.0 Results
  • v0.7.1 Results
  • v0.7.0 Results
  • v0.4.0 Results
  • Key Observations

Software Design

• Checkpoint Exporter Design
  • Design Goals
  • Architecture
  • Data Flow
  • Checkpoint Metadata Contract
  • Quantization Integration
  • Model and Component Dispatch
  • ONNX and Runtime Artifact Contract
  • LoRA Integration
  • Limitations
• Experimental ONNX-less Builder Design
  • Design Goals
  • Package Boundaries
  • Build Lifecycle
  • Module And Network Ownership
  • Operation API And TensorRT Lowering
  • Adding An Operation
  • Checkpoint And Weight Ownership
  • Profiles And Runtime Artifacts
  • Adding A Model Family
  • Validation
• Quantization Package Design
  • Design Goals
  • Package Layout
  • Flow
  • Artifact Contract
  • Supported Methods
  • Limitations
• Engine Builder
  • Overview
  • Compatibility and Model Integrity
  • Build Process Workflow
  • Component Overview
  • Build Process Stages
  • LLM Builder
  • Visual Encoder Builder
  • Usage Examples
  • Best Practices
• C++ Runtime Overview
  • Overview
  • Runtime Architecture
• LLM Inference Runtime
  • Architecture
  • Inference Workflow
  • Usage Examples
• LLM Streaming — Design
  • Goals
  • Key decisions
  • Data types (cpp/runtime/streaming.h)
  • Runtime integration — handleRequest flow
  • Terminal-reason latching (race-free by construction)
  • Detokenization algorithm (Strategy A)
  • Non-Streaming Path
  • Non-goals (deferred)
  • Code layout
  • User guide

Models

• Cosmos3-Edge Design
  • Policy Components
  • Reasoner Components
  • RoboLab Integration
• Nemotron-3.5-ASR
  • Architecture
  • 1. Export (x86 host, CPU-only)
  • 2. Build engines
  • 3. Run
  • Runtime design notes

Customization

• Customization Guide
  • Export Customization Points
  • Quantization Customization Points
  • Adding A Text Model
  • Adding A Multimodal Or Action Component
  • Custom Operators
  • Runtime Customization
• Calibration Dataset Customization
  • Built-in datasets
  • The modality contracts
  • Adding a custom dataset
  • Pointing a built-in at a cached / offline copy
  • Python API
• TensorRT Plugins Guide
  • Overview
  • AttentionPlugin
  • Int4GroupwiseGemmPlugin
  • Int4GroupwiseGemmPluginV2

Testing

• Code Coverage with SonarQube
  • Prerequisites
  • Quick Start
  • Manual Workflow
  • SonarQube Integration
  • What Gets Instrumented
  • CI Integration Example
  • Troubleshooting
• Few-Layer Numeric Validation
  • Quick Start
  • What it does
  • Runtime hooks
  • Dump size
  • Scope

APIs

• Python API Reference
  • Experimental Server
  • Quantization
  • Checkpoint Exporter
• C++ API Reference
  • Action Module
  • Builder Module
  • Common Module
  • Kernels Module
  • Multimodal Module
  • Plugins Module
  • Profiling Module
  • Runtime Module
  • Sampler Module
  • Tokenizer Module

Quick Links

• Releases
• GitHub
• Roadmap

Indices and tables

• Index

• Module Index

• Search Page