Welcome to TensorRT-LLM’s Documentation!

Getting Started

• Overview
  • About TensorRT-LLM
  • What Can You Do With TensorRT-LLM?
• Quick Start Guide
  • LLM API
  • Deploy with trtllm-serve
  • Model Definition API
  • Next Steps
  • Related Information
• Key Features
• PyTorch Backend
  • Quick Start
  • Quantization
  • Sampling
  • Developer Guide
  • Key Components
  • Known Issues
• Release Notes
  • TensorRT-LLM Release 0.19.0
  • TensorRT-LLM Release 0.18.2
  • TensorRT-LLM Release 0.18.1
  • TensorRT-LLM Release 0.18.0
  • TensorRT-LLM Release 0.17.0
  • TensorRT-LLM Release 0.16.0
  • TensorRT-LLM Release 0.15.0
  • TensorRT-LLM Release 0.14.0
  • TensorRT-LLM Release 0.13.0
  • TensorRT-LLM Release 0.12.0
  • TensorRT-LLM Release 0.11.0
  • TensorRT-LLM Release 0.10.0
  • TensorRT-LLM Release 0.9.0
  • TensorRT-LLM Release 0.8.0
  • TensorRT-LLM Release 0.7.1

Installation

• Installing on Linux
• Building from Source Code on Linux
  • Prerequisites
  • Building a TensorRT-LLM Docker Image
  • Build TensorRT-LLM
• Installing on Grace Hopper

Architecture

• TensorRT-LLM Architecture
  • Model Weights
• Model Definition
• Compilation
  • TensorRT Compiler
  • Model Engine
  • Weight Bindings
  • Pattern-Matching and Fusion
  • Plugins
• Runtime
• Multi-GPU and Multi-Node Support
  • Examples
• TensorRT-LLM Checkpoint
  • Overview
  • Prepare the TensorRT-LLM Checkpoint
  • Build Checkpoint into TensorRT Engine
  • Make Evaluation
• TensorRT-LLM Build Workflow
  • Overview
  • Conversion APIs
  • Quantization APIs
  • Build APIs
  • CLI Tools
• Adding a Model
  • Step 1. Write Modeling Part
  • Step 2. Implement Weight Conversion
  • Step 3. Register New Model
  • Step 4. Verify New Model
  • Reference

Advanced

• Multi-Head, Multi-Query, and Group-Query Attention
  • Important Note
  • Padded and Packed Tensors
  • Context and Generation Phases
  • In-flight Batching
  • Chunked Context
  • KV Cache
  • INT8/FP8 KV Caches
  • Sliding Window Attention, Cyclic (Rolling Buffer) KV Cache
  • StreamingLLM
  • Beam-Search
  • Input QKV tensor
• C++ GPT Runtime
  • Overview
• Executor API
  • API
  • C++ Executor API Example
  • Python Bindings for the Executor API
  • In-flight Batching with the Triton Inference Server
• Graph Rewriting Module
  • When to Use Graph Rewriting?
  • Graph Rewriting APIs
  • Classical Workflow
• Run gpt-2b + LoRA using Executor / cpp runtime
  • LoRA tensor format details
• Expert Parallelism in TensorRT-LLM
  • Mixture of Experts (MoE)
  • Tensor Parallel vs Expert Parallel
  • How to Enable
• KV Cache Management: Pools, Blocks, and Events
  • Hierarchy: Pool, Block, and Page
  • Events in KVCacheEventManager
• KV cache reuse
  • How to enable kv cache reuse
  • Performance expectations
  • Situations that can prevent kv cache reuse
  • Offloading to host memory
• Speculative Sampling
  • About Speculative Sampling
  • Performance Improvements
  • Draft-Target-Model
  • Prompt-Lookup-Decoding
  • Medusa
  • ReDrafter
  • EAGLE
  • Lookahead Decoding
• Disaggregated-Service (experimental)
  • Usage
  • Example
  • Benchmarks
  • Environment Variables
  • Troubleshooting and FAQ

Performance

• Overview
  • Throughput Measurements
  • Reproducing Benchmarked Results
• Benchmarking
  • Before Benchmarking
  • Throughput Benchmarking
  • Low Latency Benchmark
  • Summary
• Performance Tuning Guide
  • Prerequisite Knowledge
  • Table of Contents
  • Benchmarking Default Performance
  • Useful Build-Time Flags
  • Tuning Max Batch Size and Max Num Tokens
  • Deciding Model Sharding Strategy
  • FP8 Quantization
  • Useful Runtime Options
• Performance Analysis
  • Feature Descriptions
  • Coordinating with NVIDIA Nsight Systems Launch
  • Coordinating with PyTorch profiler (PyTorch workflow only)
  • Examples

Reference

• Troubleshooting
  • Installation Errors
  • Debug on Unit Tests
  • Debug on E2E Models
  • Debug Execution Errors
  • Tips
• Support Matrix
  • Models (PyTorch Backend)
  • Models (TensorRT Backend)
  • Hardware
  • Software
• Numerical Precision
  • FP32, FP16 and BF16
  • Quantization and Dequantization (Q/DQ)
  • INT8 SmoothQuant (W8A8)
  • INT4 and INT8 Weight-Only (W4A16 and W8A16)
  • GPTQ and AWQ (W4A16)
  • FP8 (Hopper)
  • NVFP4 (Blackwell)
  • Support matrix
  • Technical Detail: The QuantMode Flags
• Memory Usage of TensorRT-LLM
  • Understand inference time GPU memory usage
  • Memory pool
  • Known Issues
  • FAQ
• Continuous Integration Overview
  • Table of Contents
  • CI pipelines
  • Test definitions
  • Unit tests
  • Jenkins stage names
  • Finding the stage for a test
  • Waiving tests
  • Triggering CI Best Practices

Indices and tables

• Index

• Module Index

• Search Page