Skip to main content
Ctrl+K
TensorRT LLM - Home TensorRT LLM - Home

TensorRT LLM

TensorRT LLM - Home TensorRT LLM - Home

TensorRT LLM

Table of Contents

Getting Started

  • Overview
  • Quick Start Guide
  • Installation
    • Pre-built release container images on NGC
    • Installing on Linux via pip
    • Building from Source Code on Linux

Deployment Guide

  • LLM Examples
    • Generate text
    • Generate text asynchronously
    • Generate text in streaming
    • Distributed LLM Generation
    • Generate text with guided decoding
    • Control generated text using logits processor
    • Generate text with multiple LoRA adapters
    • Sparse Attention
    • Speculative Decoding
    • KV Cache Connector
    • KV Cache Offloading
    • Runtime Configuration Examples
    • Sampling Techniques Showcase
    • Run LLM-API with pytorch backend on Slurm
    • Run trtllm-bench with pytorch backend on Slurm
    • Run trtllm-serve with pytorch backend on Slurm
  • Online Serving Examples
    • Aiperf Client
    • Aiperf Client For Multimodal
    • Curl Chat Client
    • Curl Chat Client For Multimodal
    • Curl Completion Client
    • Curl Responses Client
    • Deepseek R1 Reasoning Parser
    • OpenAI Chat Client
    • OpenAI Chat Client for Multimodal
    • OpenAI Completion Client
    • Openai Completion Client For Lora
    • OpenAI Completion Client with JSON Schema
    • OpenAI Responses Client
  • Dynamo K8s Example
  • Model Recipes
    • Deployment Guide for DeepSeek R1 on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for Llama3.3 70B on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for Llama4 Scout 17B on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for GPT-OSS on TensorRT-LLM - Blackwell Hardware
    • Deployment Guide for Qwen3 on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for Qwen3 Next on TensorRT LLM - Blackwell & Hopper Hardware
    • Deployment Guide for Kimi K2 Thinking on TensorRT LLM - Blackwell

Models

  • Supported Models
  • Adding a New Model

CLI Reference

  • trtllm-bench
  • trtllm-eval
  • trtllm-serve
    • trtllm-serve
    • Run benchmarking with trtllm-serve

API Reference

  • LLM API Introduction
  • API Reference

Features

  • Feature Combination Matrix
  • Multi-Head, Multi-Query, and Group-Query Attention
  • Disaggregated Serving
  • KV Cache System
  • Long Sequences
  • LoRA (Low-Rank Adaptation)
  • Multimodal Support in TensorRT LLM
  • Overlap Scheduler
  • Paged Attention, IFB, and Request Scheduling
  • Parallelism in TensorRT LLM
  • Quantization
  • Sampling
  • Additional Outputs
  • Guided Decoding
  • Speculative Decoding
  • Checkpoint Loading
  • AutoDeploy (Beta)
  • Ray Orchestrator (Prototype)
  • Torch Compile & Piecewise CUDA Graph
  • Helix Parallelism
  • KV Cache Connector

Developer Guide

  • Architecture Overview
  • Performance Analysis
  • TensorRT LLM Benchmarking
  • Continuous Integration Overview
  • Using Dev Containers
  • LLM API Change Guide
  • Introduction to KV Cache Transmission

Blogs

  • ADP Balance Strategy
  • Running GPT-OSS-120B with Eagle3 Speculative Decoding on GB200/B200 (TensorRT LLM)
  • Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly
  • Inference Time Compute Implementation in TensorRT LLM
  • Scaling Expert Parallelism in TensorRT LLM (Part 3: Pushing the Performance Boundary)
  • Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs
  • DeepSeek R1 MTP Implementation and Optimization
  • Optimizing DeepSeek R1 Throughput on NVIDIA Blackwell GPUs: A Deep Dive for Developers
  • Scaling Expert Parallelism in TensorRT LLM (Part 1: Design and Implementation of Large-scale EP)
  • Disaggregated Serving in TensorRT LLM
  • How to launch Llama4 Maverick + Eagle3 TensorRT LLM server
  • N-Gram Speculative Decoding in TensorRT LLM
  • Scaling Expert Parallelism in TensorRT LLM (Part 2: Performance Status and Optimization)
  • Running a High Performance GPT-OSS-120B Inference Server with TensorRT LLM
  • How to get best performance on DeepSeek-R1 in TensorRT LLM
  • H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT LLM
  • New XQA-kernel provides 2.4x more Llama-70B throughput within the same latency budget
  • H100 has 4.6x A100 Performance in TensorRT LLM, achieving 10,000 tok/s at 100ms to first token

Quick Links

  • Releases
  • Github Code
  • Roadmap

Use TensorRT Engine

  • LLM API with TensorRT Engine
  • Online Serving Examples

Online Serving Examples#

  • Aiperf Client
  • Aiperf Client For Multimodal
  • Curl Chat Client
  • Curl Chat Client For Multimodal
  • Curl Completion Client
  • Curl Responses Client
  • Deepseek R1 Reasoning Parser
  • OpenAI Chat Client
  • OpenAI Chat Client for Multimodal
  • OpenAI Completion Client
  • Openai Completion Client For Lora
  • OpenAI Completion Client with JSON Schema
  • OpenAI Responses Client

previous

Run trtllm-serve with pytorch backend on Slurm

next

Aiperf Client
NVIDIA NVIDIA
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, NVidia.

Last updated on January 04, 2026.

This page is generated by TensorRT-LLM commit a65b0d4.