TensorRT Edge-LLM Documentation

Welcome to the TensorRT Edge-LLM documentation. This library provides optimized inference capabilities for large language models and vision-language models on edge devices.

Getting Started

Get up and running with TensorRT Edge-LLM. Learn about platform overview, key features, use cases, supported platforms, and complete installation instructions for Python and C++ components.

Getting Started

• Overview
  • What is TensorRT Edge-LLM?
  • Key Features
  • Key Components
  • Use Cases
  • Supported Platforms
  • Next Steps
• Quick Start Guide
  • Prerequisites
  • Step 1: Clone Repository (on both x86 host and device)
  • Step 2: Install Python Package (on x86 host)
  • Step 3: Build C++ Project (on device)
  • Step 4: Download and Export a Model (on x86 host)
  • Step 5: Build TensorRT Engine (on Thor device)
  • Step 6: Run Inference
  • Next Steps
• Installation
  • Part 1: Python Export Pipeline (x86 Host with GPU)
  • Part 2: C++ Runtime (Edge Device)
  • Complete Workflow Summary
  • Troubleshooting
  • Uninstalling
  • Next Steps

Models

Learn about supported model families and architectures.

Models

• Supported Models
  • Large Language Models (LLMs)
  • Vision-Language Models (VLMs)
  • Precision Support
  • Platform Compatibility
  • Additional Resources

Model Export & Engine Building

Convert and optimize your models for deployment. Learn how to convert HuggingFace models to ONNX with quantization and compile them into optimized TensorRT engines.

Model Export & Engine Building

• Python Export Pipeline
  • Overview
  • Export Tools
  • Quantization Methods
  • Security and Model Integrity
  • Usage Examples
  • Best Practices
  • Common Issues and Solutions
  • Next Steps
  • Additional Resources
• Engine Builder
  • Overview
  • Compatibility and Model Integrity
  • Build Process Workflow
  • Component Overview
  • Build Process Stages
  • LLM Builder
  • Visual Encoder Builder
  • Usage Examples
  • Best Practices
  • Next Steps
  • Additional Resources

Chat Template Configuration

Learn how to create and customize chat templates for formatting conversational messages for your models.

Chat Template Configuration

• Chat Template Format Guide
  • Overview
  • Chat Template File Format
  • Thinking Mode Support
  • Using Chat Templates During Export
  • Pre-built Templates
  • Examples
  • System Prompt Priority

C++ Runtime

Explore the C++ inference runtime and its capabilities, including runtime architecture, standard runtime for text and multimodal inference, EAGLE speculative decoding, CUDA graphs, LoRA, and batch processing.

C++ Runtime

• C++ Runtime Overview
  • Overview
  • Runtime Architecture
  • Next Steps
  • Additional Resources
• LLM Inference Runtime
  • Architecture
  • Inference Workflow
  • Usage Examples
  • Next Steps
  • Additional Resources
• LLM Inference SpecDecode Runtime
  • Architecture
  • Inference Workflow
  • Key Differences from LLM Inference Runtime
  • Usage Example
  • Next Steps
  • Additional Resources
• Advanced Runtime Features
  • Overview
  • CUDA Graph Optimization
  • LoRA (Low-Rank Adaptation) Support
  • Batch Processing
  • System Prompt KV-Cache Optimization
  • Vocabulary Reduction
  • Multimodal Processing
  • Usage Examples
  • Next Steps
  • Additional Resources

Examples

Reference implementations demonstrating LLM, multimodal, and utility use cases.

Examples

• Examples
  • Overview
  • Example Flow
  • Build Examples
  • Inference Examples
  • Complete Workflows
  • Input File Formats
  • Common Parameters
  • Next Steps
  • Additional Resources

Customization

Learn how to customize and extend TensorRT Edge-LLM for your specific needs.

Customization

• Customization Guide
  • Customization Architecture
  • Layer 1: Models
  • Layer 2: Python Export Pipeline
  • Layer 3: Engine Builder
  • Layer 4: C++ Runtime
  • Layer 5: Application

TensorRT Plugins

Learn about the usage of TensorRT plugins with TensorRT Edge-LLM and how to make further customizations.

TensorRT Plugins

• TensorRT Plugins Guide
  • Overview
  • AttentionPlugin
  • Int4GroupwiseGemmPlugin

APIs

API documentation for Python and C++ components.

APIs

• Python API Reference
  • Main Module
• C++ API Reference
  • Builder Module
  • Common Module
  • Kernels Module
  • Multimodal Module
  • Plugins Module
  • Profiling Module
  • Runtime Module
  • Sampler Module
  • Tokenizer Module

---

Need help? Visit our GitHub repository for issues and discussions.

Indices and tables

• Index

• Module Index

• Search Page