Skip to main content
Ctrl+K
TensorRT LLM - Home TensorRT LLM - Home

TensorRT LLM

TensorRT LLM - Home TensorRT LLM - Home

TensorRT LLM

Table of Contents

Getting Started

  • Overview
  • Quick Start Guide
  • Installation
    • Installation Guide
    • Build from Source
    • Container Images
  • Supported Hardware

Deployment Guide

  • LLM Examples
    • Generate text
    • Generate text asynchronously
    • Generate text in streaming
    • Distributed LLM Generation
    • Generate text with guided decoding
    • Control generated text using logits processor
    • Generate text with multiple LoRA adapters
    • Sparse Attention
    • Speculative Decoding
    • KV Cache Connector
    • KV Cache Offloading
    • Runtime Configuration Examples
    • Sampling Techniques Showcase
    • LMCache KV Cache Connector
    • Run LLM-API with pytorch backend on Slurm
    • Run trtllm-bench with pytorch backend on Slurm
    • Run trtllm-serve with pytorch backend on Slurm
  • VisualGen Examples
    • API walkthrough
  • Online Serving Examples
    • Aiperf Client
    • Aiperf Client For Multimodal
    • Curl Chat Client
    • Curl Chat Client For Multimodal
    • Curl Completion Client
    • Curl Responses Client
    • Deepseek R1 Reasoning Parser
    • OpenAI Chat Client
    • OpenAI Chat Client for Multimodal
    • OpenAI Completion Client
    • Openai Completion Client For Lora
    • OpenAI Completion Client with JSON Schema
    • OpenAI Responses Client
    • Prometheus Metrics
  • Dynamo K8s Example
  • Model Recipes
    • Deployment Guide for Nemotron v3 (Ultra & Super) on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for DeepSeek R1 on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for Llama3.3 70B on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for Llama4 Scout 17B on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for GPT-OSS on TensorRT-LLM - Blackwell Hardware
    • Deployment Guide for Qwen3 on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for Qwen3.5 on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for Kimi K2 Thinking on TensorRT LLM - Blackwell
    • Deployment Guide for GLM-5 on TensorRT LLM - Blackwell Hardware
    • Deployment Guide for MiniMax-M3 on TensorRT LLM
  • CPU Affinity configuration in TensorRT LLM

Models

  • Supported Models
  • Visual Generation (Beta)
  • Adding a New Model

CLI Reference

  • trtllm-bench
  • trtllm-eval
  • trtllm-serve
    • trtllm-serve
    • Run benchmarking with trtllm-serve

API Reference

  • LLM API Introduction
  • API Reference

Features

  • Feature Combination Matrix
  • Multi-Head, Multi-Query, and Group-Query Attention
  • Disaggregated Serving
  • Embeddings (Encoder-Only Models)
  • KV Cache System
  • Long Sequences
  • LoRA (Low-Rank Adaptation)
  • Multimodal Support in TensorRT LLM
  • Overlap Scheduler
  • Paged Attention, IFB, and Request Scheduling
  • Parallelism in TensorRT LLM
  • Quantization
  • Sampling
  • Additional Outputs
  • Post-Processing Hook
  • Guided Decoding
  • Speculative Decoding
  • Checkpoint Loading
  • AutoDeploy (Beta)
  • AutoDeploy Transforms
    • Core Transform APIs
    • Factory Stage
    • Export Stage
    • Post-Export Stage
    • Pattern Matching Stage
    • Sharding Stage
    • Weight Loading Stage
    • Post-Load Fusion Stage
    • Cache Initialization Stage
    • Visualization Stage
    • Compilation Stage
    • Additional Registered Transforms
  • Ray Orchestrator (Prototype)
  • Torch Compile & Piecewise CUDA Graph
  • Helix Parallelism
  • KV Cache Connector
  • Sparse Attention

Developer Guide

  • Architecture Overview
  • Performance Analysis
  • TensorRT LLM Benchmarking
  • Continuous Integration Overview
  • Using Dev Containers
  • LLM API Change Guide
  • Introduction to KV Cache Transmission
  • Telemetry
  • Sparse Attention Development Guide

Blogs

  • Scaling Video Generation Across NVL72 rack with TensorRT-LLM
  • MoE as Dense GEMM: Optimizing Low-Latency MoE Inference on NVIDIA Blackwell
  • Joint Optimization of Agent Applications and TensorRT-LLM
  • Helix Parallelism in TensorRT-LLM: Scaling Multi-Million-Token Decoding with KV Cache Sharding
  • Temporal Correlation Meets Sparse Attention: Guess-Verify-Refine Top-K for Blackwell
  • Tuning CUDA Graph Batch Sizes for Higher Output Throughput
  • DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
  • Optimizing MoE Communication with One-Sided AlltoAll Over NVLink
  • Sparse Attention in TensorRT LLM
  • Accelerating Long-Context Inference with Skip Softmax Attention
  • Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs
  • Scaling Expert Parallelism in TensorRT LLM (Part 3: Pushing the Performance Boundary)
  • Inference Time Compute Implementation in TensorRT LLM
  • Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly
  • Running GPT-OSS-120B with Eagle3 Speculative Decoding on GB200/B200 (TensorRT LLM)
  • ADP Balance Strategy
  • Running a High Performance GPT-OSS-120B Inference Server with TensorRT LLM
  • Scaling Expert Parallelism in TensorRT LLM (Part 2: Performance Status and Optimization)
  • N-Gram Speculative Decoding in TensorRT LLM
  • How to launch Llama4 Maverick + Eagle3 TensorRT LLM server
  • Disaggregated Serving in TensorRT LLM
  • Scaling Expert Parallelism in TensorRT LLM (Part 1: Design and Implementation of Large-scale EP)
  • Optimizing DeepSeek R1 Throughput on NVIDIA Blackwell GPUs: A Deep Dive for Developers
  • DeepSeek R1 MTP Implementation and Optimization
  • Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs
  • How to get best performance on DeepSeek-R1 in TensorRT LLM
  • H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT LLM
  • New XQA-kernel provides 2.4x more Llama-70B throughput within the same latency budget
  • H100 has 4.6x A100 Performance in TensorRT LLM, achieving 10,000 tok/s at 100ms to first token
  • Falcon-180B on a single H200 GPU with INT4 AWQ, and 6.7x faster Llama-70B over A100
  • Speed up inference with SOTA quantization techniques in TRT-LLM

Quick Links

  • Releases
  • Github Code
  • Roadmap

Migration

  • Migration Guide: TensorRT Backend Removed

Python Module Index

t
 
t
- tensorrt_llm
    tensorrt_llm._torch.auto_deploy.transform.graph_module_visualizer
    tensorrt_llm._torch.auto_deploy.transform.interface
    tensorrt_llm._torch.auto_deploy.transform.optimizer
NVIDIA NVIDIA
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, NVidia.

Last updated on July 14, 2026.

This page is generated by TensorRT-LLM commit 1662a87.