Skip to main content

Ctrl+K

TensorRT LLM

TensorRT LLM

Table of Contents

Getting Started

Overview
Quick Start Guide
Installation
Supported Hardware

Deployment Guide

LLM Examples
VisualGen Examples
- API walkthrough
Online Serving Examples
Dynamo K8s Example
Model Recipes
CPU Affinity configuration in TensorRT LLM

Models

Supported Models
Visual Generation (Beta)
Adding a New Model

CLI Reference

trtllm-bench
trtllm-eval
trtllm-serve

API Reference

LLM API Introduction
API Reference

Features

Feature Combination Matrix
Multi-Head, Multi-Query, and Group-Query Attention
Disaggregated Serving
Sub-agent Routing
Embeddings (Encoder-Only Models)
KV Cache System
KV hint request metadata
KV Cache Compression
Long Sequences
LoRA (Low-Rank Adaptation)
Multimodal Support in TensorRT LLM
Overlap Scheduler
Paged Attention, IFB, and Request Scheduling
Parallelism in TensorRT LLM
Quantization
Sampling
Additional Outputs
Post-Processing Hook
Guided Decoding
Speculative Decoding
Checkpoint Loading
ModelExpress (MX) Checkpoint Loading
Runtime Memory Checkpointing (Prototype)
Ray Orchestrator (Prototype)
Torch Compile & Prefill CUDA Graph
Helix Parallelism
KV Cache Connector
Sparse Attention
Visual Generation (Beta)
Prefix-Tokenization Cache

Developer Guide

Architecture Overview
Multimodal Request Data
Performance Analysis
TensorRT LLM Benchmarking
MoE Performance Guide
- MoE Perf Dashboard
Continuous Integration Overview
Using Dev Containers
LLM API Change Guide
Introduction to KV Cache Transmission
KVCacheManagerV2 Cold-Page Codec Design
KV Cache Compression Development Guide
Sparse Attention Development Guide
Telemetry

Blogs

KV Cache Compression in TensorRT LLM
GVR V2: Self-Sampling and Multi-Thresholding for Faster Exact Top-K
Accelerating Video Generation with GEMM Quantization, Attention Quantization and Skip Softmax Attention in TensorRT-LLM
Evaluating Agentic Serving with Trace Replay and Job-Level Metrics
DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM
Scaling Video Generation Across NVL72 rack with TensorRT-LLM
MoE as Dense GEMM: Optimizing Low-Latency MoE Inference on NVIDIA Blackwell
Joint Optimization of Agent Applications and TensorRT-LLM
Helix Parallelism in TensorRT-LLM: Scaling Multi-Million-Token Decoding with KV Cache Sharding
Temporal Correlation Meets Sparse Attention: Guess-Verify-Refine Top-K for Blackwell
Tuning CUDA Graph Batch Sizes for Higher Output Throughput
DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
Optimizing MoE Communication with One-Sided AlltoAll Over NVLink
Sparse Attention in TensorRT LLM
Accelerating Long-Context Inference with Skip Softmax Attention
Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs
Scaling Expert Parallelism in TensorRT LLM (Part 3: Pushing the Performance Boundary)
Inference Time Compute Implementation in TensorRT LLM
Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly
Running GPT-OSS-120B with Eagle3 Speculative Decoding on GB200/B200 (TensorRT LLM)
ADP Balance Strategy
Running a High Performance GPT-OSS-120B Inference Server with TensorRT LLM
Scaling Expert Parallelism in TensorRT LLM (Part 2: Performance Status and Optimization)
N-Gram Speculative Decoding in TensorRT LLM
How to launch Llama4 Maverick + Eagle3 TensorRT LLM server
Disaggregated Serving in TensorRT LLM
Scaling Expert Parallelism in TensorRT LLM (Part 1: Design and Implementation of Large-scale EP)
Optimizing DeepSeek R1 Throughput on NVIDIA Blackwell GPUs: A Deep Dive for Developers
DeepSeek R1 MTP Implementation and Optimization
Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs
How to get best performance on DeepSeek-R1 in TensorRT LLM
H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT LLM
New XQA-kernel provides 2.4x more Llama-70B throughput within the same latency budget
H100 has 4.6x A100 Performance in TensorRT LLM, achieving 10,000 tok/s at 100ms to first token
Falcon-180B on a single H200 GPU with INT4 AWQ, and 6.7x faster Llama-70B over A100
Speed up inference with SOTA quantization techniques in TRT-LLM

Quick Links

Releases
Github Code
Roadmap

Migration

Migration Guide: TensorRT Backend Removed

API Reference
KvHintsInput

KvHintsInput#

tensorrt_llm.llmapi.KvHintsInput#: alias of KvHintsEnvelope | dict[str, JsonValue]

previous

KvHintsEnvelope

next

DisaggregatedParams

On this page

KvHintsInput

Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, NVidia.

Last updated on November 03, 2026.

This page is generated by TensorRT-LLM commit 9e334fb.