TensorRT Edge-LLM Documentation

Welcome to the TensorRT Edge-LLM documentation. This library provides optimized inference capabilities for large language models and vision-language models on edge devices.

Getting Started

• Overview
  • What is TensorRT Edge-LLM?
  • Supported Platforms
  • Key Features
  • Key Components
  • Next Steps
• Supported Models
  • Large Language Models (LLMs)
  • Vision-Language Models (VLMs)
  • Audio Multimodal Models
  • Text-to-Speech (TTS) Models
  • Precision Support
  • Platform Compatibility
• Installation
  • Part 1: Python Export Pipeline (x86 Host with GPU)
  • Part 2: C++ Runtime (Edge Device)
  • Next Steps
  • Troubleshooting
  • Uninstalling
• Quick Start Guide
  • Part 1: Export and Quantize on x86 Host
  • Part 2: Build and Run on Edge Device
  • Next Steps
• Limitations and Known Issues
  • 0.6.0

Examples

• VLM (Vision-Language Model) Inference
  • Step 1: Quantize and Export (x86 Host)
  • Step 2: Transfer to Device
  • Step 3: Build Engines (Thor Device)
  • Step 4: Run Inference (Thor Device)
• Speculative Decoding
  • EAGLE3
• Phi-4-Multimodal
  • Step 1: Merge, Quantize, and Export (x86 Host)
  • Step 2: Transfer to Device
  • Step 3: Build Engines (Thor Device)
  • Step 4: Run Inference (Thor Device)
• ASR (Automatic Speech Recognition)
  • Step 1: Export (x86 Host)
  • Step 2: Transfer to Device
  • Step 3: Build Engines (Thor Device)
  • Step 4: Preprocess Audio Input (x86 Host or Thor Device)
  • Step 5: Run Inference (Thor Device)
• MoE (Mixture of Experts)
  • Step 1: Export (x86 Host, CPU-only)
  • Step 2: Transfer to Device
  • Step 3: Build Engine (Edge Device)
  • Step 4: Run Inference (Edge Device)
• TTS (Text-to-Speech)
  • Part 0: Install TTS Dependency (x86 Host)
  • Part 1: Export on x86 Host
  • Part 2: Build Engines
  • Part 3: Run Inference
  • Notes

Features

• LoRA (Low-Rank Adaptation)
  • Overview
  • Approach 1: Static LoRA Merge
  • Approach 2: Dynamic Runtime LoRA
  • Input Format for Runtime LoRA
  • Script Reference
  • Build Parameters
  • Notes
• Vocabulary Reduction
  • Overview
  • Quick Start
  • EAGLE Speculative Decoding Support
  • Script Reference
  • Custom Vocabulary Mapping
  • Notes
• FP8 KV Cache
  • Overview
  • Workflow
  • Technical Details
  • Warning
  • Notes
• System Prompt Cache
  • Overview
  • Usage
  • Limitations
  • Best Practices
  • Notes

Input & Chat Format

• Input JSON Format
  • Basic Structure
  • Parameters
  • Examples
  • Notes
• Chat Template Format
  • Overview
  • File Structure
  • Thinking Mode
  • Custom Templates
  • Examples
  • System Prompt Priority

Software Design

• Python Export Pipeline
  • Overview
  • Export Tools
  • Quantization Methods
  • Usage Examples
  • Best Practices
  • Common Issues and Solutions
• Engine Builder
  • Overview
  • Compatibility and Model Integrity
  • Build Process Workflow
  • Component Overview
  • Build Process Stages
  • LLM Builder
  • Visual Encoder Builder
  • Usage Examples
  • Best Practices
• C++ Runtime Overview
  • Overview
  • Runtime Architecture
• LLM Inference Runtime
  • Architecture
  • Inference Workflow
  • Usage Examples
• LLM Inference SpecDecode Runtime
  • Architecture
  • Inference Workflow
  • Key Differences from LLM Inference Runtime
  • Usage Example

Customization

• Customization Guide
  • Customization Architecture
  • Layer 1: Models
  • Layer 2: Python Export Pipeline
  • Layer 3: Engine Builder
  • Layer 4: C++ Runtime
  • Layer 5: Application
• TensorRT Plugins Guide
  • Overview
  • AttentionPlugin
  • Int4GroupwiseGemmPlugin

Testing

• Code Coverage with SonarQube
  • Prerequisites
  • Quick Start
  • Manual Workflow
  • SonarQube Integration
  • What Gets Instrumented
  • CI Integration Example
  • Troubleshooting

APIs

• Python API Reference
  • Main Module
• C++ API Reference
  • Builder Module
  • Common Module
  • Kernels Module
  • Multimodal Module
  • Plugins Module
  • Profiling Module
  • Runtime Module
  • Sampler Module
  • Tokenizer Module

Quick Links

• Releases
• GitHub
• Roadmap

Indices and tables

• Index

• Module Index

• Search Page