Welcome to TensorRT-LLM’s Documentation!

Getting Started

• Overview
  • About TensorRT-LLM
  • What Can You Do With TensorRT-LLM?
• Quick Start Guide
  • Prerequisites
  • Launch the Docker
  • Retrieve the Model Weights
  • Compile the Model into a TensorRT Engine
  • Run the Model
  • Deploy with Triton Inference Server
  • Send Requests
  • Next Steps
• Release Notes
  • TensorRT-LLM Release 0.9.0
  • TensorRT-LLM Release 0.8.0
  • TensorRT-LLM Release 0.7.1

Installation

• Installing on Linux
• Building from Source Code on Linux
  • Prerequisites
  • Building a TensorRT-LLM Docker Image
  • Building the Python Bindings for the C++ Runtime
  • Linking with the TensorRT-LLM C++ Runtime
• Installing on Windows
• Building from Source Code on Windows
  • Prerequisites
  • Building a TensorRT-LLM Docker Image
  • Building TensorRT-LLM on Bare Metal
  • Linking with the TensorRT-LLM C++ Runtime

Architecture

• TensorRT-LLM Architecture
  • Model Weights
• Model Definition
• Compilation
  • TensorRT Compiler
  • Model Engine
  • Weight Bindings
  • Pattern-Matching and Fusion
  • Plugins
• Runtime
• Multi-GPU and Multi-Node Support
• TensorRT-LLM Checkpoint
  • Overview
  • Prepare the TensorRT-LLM Checkpoint
  • Build Checkpoint into TensorRT Engine
  • Make Evaluation
• TensorRT-LLM Build Workflow
  • Overview
  • Conversion APIs
  • Quantization APIs
  • Build APIs
  • CLI Tools
• Adding a Model

Advanced

• Multi-Head, Multi-Query, and Group-Query Attention
  • Important Note
  • Padded and Packed Tensors
  • Context and Generation Phases
  • In-flight Batching
  • Chunked Context
  • KV Cache
  • INT8/FP8 KV Caches
  • Sliding Window Attention, Cyclic (Rolling Buffer) KV Cache
  • StreamingLLM
  • Beam-Search
  • Input QKV tensor
• C++ GPT Runtime
  • Overview
  • The Session
  • In-flight Batching Support
  • Know Issues and Future Changes
• Graph Rewriting Module
  • When to Use Graph Rewriting?
  • Graph Rewriting APIs
  • Classical Workflow
• The Batch Manager in TensorRT-LLM
  • The Batch Manager API
  • In-flight Batching with the Triton Inference Server
• Inference Request
• Run gpt-2b + LoRA using GptManager / cpp runtime
  • LoRA tensor format details
• Expert Parallelism in TensorRT-LLM
  • Mixture of Experts (MoE)
  • Tensor Parallel vs Expert Parallel
  • How to Enable

Performance

• Overview
  • Methodology
  • Peak Throughput
  • Low Latency^**
  • Known Issues
  • Reproducing Benchmarked Results
  • Methodology
  • Benchmarking per Model
• Best Practices for Tuning the Performance of TensorRT-LLM
  • How To Measure Performance?
  • Build Options to Optimize the Performance of TensorRT-LLM Models?
  • Runtime Options to Optimize the Performance of TensorRT-LLM Models?
• Performance Analysis
  • Feature Descriptions
  • Usage
  • Coordinating with NVIDIA Nsight Systems Launch
  • Examples
  • Profiling a single IFB iteration executing on a single rank of a multi-GPU model

Reference

• Troubleshooting
  • Debug on Unit Tests
  • Debug on E2E Models
  • Debug Execution Errors
  • Installation Errors
  • Tips
• Support Matrix
  • Hardware
  • Software
• Numerical Precision
  • FP32, FP16 and BF16
  • Quantization and Dequantization (Q/DQ)
  • INT8 SmoothQuant (W8A8)
  • INT4 and INT8 Weight-Only (W4A16 and W8A16)
  • GPTQ and AWQ (W4A16)
  • FP8 (Hopper)
  • Support matrix
  • Technical Detail: The QuantMode Flags
• Memory Usage of TensorRT-LLM
  • Understand inference time GPU memory usage
  • KV cache tensor
  • Memory pool
  • Known Issues
  • FAQ

Indices and tables

• Index

• Module Index

• Search Page