TensorRT Edge-LLM Documentation

Welcome to the TensorRT Edge-LLM documentation. This library provides optimized inference capabilities for large language models and vision-language models on edge devices.

Getting Started

• Overview
  • What is TensorRT Edge-LLM?
  • Supported Platforms
  • Key Features
  • Key Components
  • Next Steps
• Supported Models
  • Support Policy
  • Precision Notes
  • Support Matrix
  • EAGLE3 Draft Models
• Installation
  • Part 1: Python Export Pipeline (x86 Host with GPU)
  • Part 2: C++ Runtime (Edge Device)
  • Next Steps
  • Troubleshooting
  • Uninstalling
• Quick Start Guide
  • Recommended: High-Level API or Server
  • Manual Export and C++ Runtime Path
  • Next Steps
• Limitations and Known Issues
  • 0.7.0
  • 0.6.1
  • 0.6.0

Examples

• Examples
  • Available Examples
• VLM (Vision-Language Model) Inference
  • Step 1: Quantize and Export (x86 Host)
  • Step 2: Transfer to Device
  • Step 3: Build Engines (Thor Device)
  • Step 4: Run Inference (Thor Device)
• Speculative Decoding
  • EAGLE3
• Phi-4-Multimodal
  • Step 1: Merge, Quantize, and Export (x86 Host)
  • Step 2: Transfer to Device
  • Step 3: Build Engines (Thor Device)
  • Step 4: Run Inference (Thor Device)
• ASR (Automatic Speech Recognition)
  • Part 0: Install ASR Dependency (x86 Host)
  • Step 1: Export (x86 Host)
  • Step 2: Transfer to Device
  • Step 3: Build Engines (Thor Device)
  • Step 4: Preprocess Audio Input (x86 Host or Thor Device)
  • Step 5: Run Inference (Thor Device)
• MoE (Mixture of Experts)
  • Step 1: Export (x86 Host, CPU-only)
  • Step 2: Transfer to Device
  • Step 3: Build Engine (Edge Device)
  • Step 4: Run Inference (Edge Device)
• TTS (Text-to-Speech)
  • Part 0: Install TTS Dependency (x86 Host)
  • Part 1: Export on x86 Host
  • Part 2: Build Engines
  • Part 3: Run Inference
  • Notes
• Experimental High-Level Python API and Server
  • Prerequisites
  • Python API
  • OpenAI-Compatible Server
  • Common Inputs
  • Sampling Parameters
  • EAGLE
  • Endpoints
  • Notes

Features

• LoRA (Low-Rank Adaptation)
  • Recommended: Checkpoint-Based Loader
  • Alternative: Legacy Export Tools
  • Approach 1: Static LoRA Merge
  • Approach 2: Dynamic Runtime LoRA
  • Input Format for Runtime LoRA
  • Script Reference
  • Build Parameters
  • Notes
• Quantization
  • Setup
  • Quantize An LLM
  • Enable FP8 KV Cache
  • Quantize An EAGLE3 Draft
  • Export The Quantized Checkpoint
  • Supported Methods
  • Notes
• Vocabulary Reduction
  • Overview
  • Quick Start
  • EAGLE Speculative Decoding Support
  • Script Reference
  • Custom Vocabulary Mapping
  • Notes
• FP8 KV Cache
  • Overview
  • Workflow
  • Technical Details
  • Performance: FP8 vs FP16 FMHA on Thor
  • Warning
  • Notes
• FP8 Embedding
  • Overview
  • Workflow
  • Combining with Other Quantization
  • Technical Details
  • Notes
  • Limitations
• Streaming Output
  • Quick start
  • The StreamChunk payload
  • Batching with per-slot opt-in / opt-out
  • Throttling with streamInterval
  • Cancellation
  • Alternative delivery patterns
  • Channel lifetime rules
  • End-to-end demo: llm_stream
  • Common pitfalls
• System Prompt Cache
  • Overview
  • Usage
  • Limitations
  • Best Practices
  • Notes

Input & Chat Format

• Input JSON Format
  • Basic Structure
  • Parameters
  • Examples
  • Notes
• Chat Template Format
  • Overview
  • File Structure
  • Thinking Mode
  • Custom Templates
  • Examples
  • System Prompt Priority

Performance

• Performance Benchmarks
  • Definitions
  • LLM — Vanilla Decoding
  • Vision Language Model — Vanilla Decoding
  • LLM — EAGLE Speculative Decoding
  • Vision Language Model — EAGLE Speculative Decoding
  • Key Observations

Software Design

• Checkpoint-Based Model Loader Design
  • Design Goals
  • Architecture
  • Data Flow
  • Checkpoint Metadata Contract
  • Quantization Integration
  • Model and Component Dispatch
  • ONNX and Runtime Artifact Contract
  • LoRA Integration
  • Limitations
  • Migrating From The Legacy Pipeline
• Experimental Quantization Package Design
  • Design Goals
  • Package Layout
  • Flow
  • Artifact Contract
  • Supported Methods
  • Limitations
• Python Export Pipeline (Legacy)
  • Overview
  • Export Tools
  • Quantization Methods
  • Usage Examples
  • Best Practices
  • Common Issues and Solutions
• Engine Builder
  • Overview
  • Compatibility and Model Integrity
  • Build Process Workflow
  • Component Overview
  • Build Process Stages
  • LLM Builder
  • Visual Encoder Builder
  • Usage Examples
  • Best Practices
• C++ Runtime Overview
  • Overview
  • Runtime Architecture
• LLM Inference Runtime
  • Architecture
  • Inference Workflow
  • Usage Examples
• LLM Inference SpecDecode Runtime
  • Architecture
  • Inference Workflow
  • Key Differences from LLM Inference Runtime
  • Usage Example
• LLM Streaming — Design
  • Goals
  • Key decisions
  • Data types (cpp/runtime/streaming.h)
  • Runtime integration — handleRequest flow
  • Terminal-reason latching (race-free by construction)
  • Detokenization algorithm (Strategy A)
  • Non-streaming back-compat
  • Non-goals (deferred)
  • Code layout
  • User guide

Customization

• Customization Guide
  • Recommended: Checkpoint-Based Loader
  • Customization Architecture
  • Layer 1: Models
  • Layer 2: Python Export Pipeline
  • Layer 3: Engine Builder
  • Layer 4: C++ Runtime
  • Layer 5: Application
• TensorRT Plugins Guide
  • Overview
  • AttentionPlugin
  • Int4GroupwiseGemmPlugin

Testing

• Code Coverage with SonarQube
  • Prerequisites
  • Quick Start
  • Manual Workflow
  • SonarQube Integration
  • What Gets Instrumented
  • CI Integration Example
  • Troubleshooting

APIs

• Python API Reference
  • Main Module
• C++ API Reference
  • Builder Module
  • Common Module
  • Kernels Module
  • Multimodal Module
  • Plugins Module
  • Profiling Module
  • Runtime Module
  • Sampler Module
  • Tokenizer Module

Quick Links

• Releases
• GitHub
• Roadmap

Indices and tables

• Index

• Module Index

• Search Page