TensorRT Edge-LLM Documentation

Welcome to the TensorRT Edge-LLM documentation. This library provides optimized inference capabilities for large language models and vision-language models on edge devices.

Getting Started

• Overview
  • What is TensorRT Edge-LLM?
  • Supported Platforms
  • Key Features
  • Key Components
  • Next Steps
• Supported Models
  • Large Language Models (LLMs)
  • Vision-Language Models (VLMs)
  • Precision Support
  • Platform Compatibility
• Installation
  • Part 1: Python Export Pipeline (x86 Host with GPU)
  • Part 2: C++ Runtime (Edge Device)
  • Next Steps
  • Troubleshooting
  • Uninstalling
• Quick Start Guide
  • Part 1: Export and Quantize on x86 Host
  • Part 2: Build and Run on Edge Device
  • Next Steps
• Examples and Complete Workflows
  • Overview
  • Complete Workflow Summary
  • Example 1: VLM (Vision-Language Model) Inference
  • Example 2: LLM EAGLE Speculative Decoding
  • Example 3: VLM EAGLE Speculative Decoding
  • Example 4: LoRA-Enabled Models
  • Example 5: Phi-4-Multimodal with LoRA Merge
  • Input File Format Reference
  • Common Build Parameters
  • Profiling and Performance Analysis
• Input JSON Format
  • Basic Structure
  • Parameters
  • Examples
  • Notes
• Chat Template Format
  • Overview
  • File Structure
  • Thinking Mode
  • Custom Templates
  • Examples
  • System Prompt Priority

Software Design

• Python Export Pipeline
  • Overview
  • Export Tools
  • Quantization Methods
  • Usage Examples
  • Best Practices
  • Common Issues and Solutions
• Engine Builder
  • Overview
  • Compatibility and Model Integrity
  • Build Process Workflow
  • Component Overview
  • Build Process Stages
  • LLM Builder
  • Visual Encoder Builder
  • Usage Examples
  • Best Practices
• C++ Runtime Overview
  • Overview
  • Runtime Architecture
• LLM Inference Runtime
  • Architecture
  • Inference Workflow
  • Usage Examples
• LLM Inference SpecDecode Runtime
  • Architecture
  • Inference Workflow
  • Key Differences from LLM Inference Runtime
  • Usage Example

Advanced Features

• LoRA (Low-Rank Adaptation)
  • Overview
  • Approach 1: Static LoRA Merge
  • Approach 2: Dynamic Runtime LoRA
  • Input Format for Runtime LoRA
  • Script Reference
  • Build Parameters
  • Notes
• Vocabulary Reduction
  • Overview
  • Quick Start
  • EAGLE Speculative Decoding Support
  • Script Reference
  • Custom Vocabulary Mapping
  • Notes
• FP8 KV Cache
  • Overview
  • Workflow
  • Technical Details
  • Warning
  • Notes
• System Prompt Cache
  • Overview
  • Usage
  • Limitations
  • Best Practices
  • Notes

Customization

• Customization Guide
  • Customization Architecture
  • Layer 1: Models
  • Layer 2: Python Export Pipeline
  • Layer 3: Engine Builder
  • Layer 4: C++ Runtime
  • Layer 5: Application
• TensorRT Plugins Guide
  • Overview
  • AttentionPlugin
  • Int4GroupwiseGemmPlugin

APIs

• Python API Reference
  • Main Module
• C++ API Reference
  • Builder Module
  • Common Module
  • Kernels Module
  • Multimodal Module
  • Plugins Module
  • Profiling Module
  • Runtime Module
  • Sampler Module
  • Tokenizer Module

---

Need help? Visit our GitHub repository for issues and discussions.

Indices and tables

• Index

• Module Index

• Search Page