Skip to main content
Ctrl+K
TensorRT-LLM - Home TensorRT-LLM - Home

TensorRT-LLM

TensorRT-LLM - Home TensorRT-LLM - Home

TensorRT-LLM

Table of Contents

Getting Started

  • Overview
  • Quick Start Guide
  • Key Features
  • PyTorch Backend
  • Release Notes

Installation

  • Pre-built release container images on NGC
  • Installing on Linux via pip
  • Building from Source Code on Linux

LLM API

  • LLM API Introduction
  • API Reference

Examples

  • LLM Examples Introduction
  • LLM Common Customizations
  • LLM Examples
    • Generate text
    • Generate text asynchronously
    • Generate text in streaming
    • Distributed LLM Generation
    • Generate text with guided decoding
    • Control generated text using logits processor
    • Generate text with multiple LoRA adapters
    • Run LLM-API with pytorch backend on Slurm
    • Run trtllm-bench with pytorch backend on Slurm
    • Run trtllm-serve with pytorch backend on Slurm
  • Online Serving Examples
    • Curl Chat Client
    • Curl Chat Client For Multimodal
    • Curl Completion Client
    • Deepseek R1 Reasoning Parser
    • Genai Perf Client
    • Genai Perf Client For Multimodal
    • OpenAI Chat Client
    • OpenAI Chat Client for Multimodal
    • OpenAI Completion Client
    • Openai Completion Client For Lora

Model Definition API

  • Layers
  • Functionals
  • Models
  • Plugin
  • Quantization
  • Runtime

C++ API

  • Executor
  • Runtime

Command-Line Reference

  • trtllm-build
  • trtllm-serve

Architecture

  • TensorRT-LLM Architecture
  • Model Definition
  • TensorRT-LLM Checkpoint
  • TensorRT-LLM Build Workflow
  • Adding a Model

Advanced

  • Multi-Head, Multi-Query, and Group-Query Attention
  • C++ GPT Runtime
  • Executor API
  • Graph Rewriting Module
  • Run gpt-2b + LoRA using Executor / cpp runtime
  • Expert Parallelism in TensorRT-LLM
  • KV Cache Management: Pools, Blocks, and Events
  • KV cache reuse
  • Speculative Sampling
  • Disaggregated-Service (experimental)

Performance

  • Overview
  • Benchmarking
  • Performance Tuning Guide
    • Benchmarking Default Performance
    • Useful Build-Time Flags
    • Tuning Max Batch Size and Max Num Tokens
    • Deciding Model Sharding Strategy
    • FP8 Quantization
    • Useful Runtime Options
  • Performance Analysis

Reference

  • Troubleshooting
  • Support Matrix
  • Numerical Precision
  • Memory Usage of TensorRT-LLM
  • Continuous Integration Overview
  • Using Dev Containers

Blogs

  • H100 has 4.6x A100 Performance in TensorRT-LLM, achieving 10,000 tok/s at 100ms to first token
  • H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLM
  • Falcon-180B on a single H200 GPU with INT4 AWQ, and 6.7x faster Llama-70B over A100
  • Speed up inference with SOTA quantization techniques in TRT-LLM
  • New XQA-kernel provides 2.4x more Llama-70B throughput within the same latency budget
  • Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs
  • DeepSeek R1 MTP Implementation and Optimization
  • Online Serving Examples

Online Serving Examples#

  • Curl Chat Client
  • Curl Chat Client For Multimodal
  • Curl Completion Client
  • Deepseek R1 Reasoning Parser
  • Genai Perf Client
  • Genai Perf Client For Multimodal
  • OpenAI Chat Client
  • OpenAI Chat Client for Multimodal
  • OpenAI Completion Client
  • Openai Completion Client For Lora

previous

Run trtllm-serve with pytorch backend on Slurm

next

Curl Chat Client
NVIDIA NVIDIA
Privacy Policy | Manage My Privacy | Do Not Sell or Share My Data | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, NVidia.

Last updated on July 06, 2025.

This page is generated by TensorRT-LLM commit 66f299a.