Skip to main content

Ctrl+K

TensorRT Edge-LLM

TensorRT Edge-LLM

Table of Contents

Getting Started

Overview
Supported Models
Installation
Quick Start Guide
Examples and Complete Workflows
Input JSON Format
Chat Template Format

Software Design

Python Export Pipeline
Engine Builder
C++ Runtime Overview
LLM Inference Runtime
LLM Inference SpecDecode Runtime

Advanced Features

LoRA (Low-Rank Adaptation)
Vocabulary Reduction
FP8 KV Cache
System Prompt Cache

Customization

Customization Guide
TensorRT Plugins Guide

APIs

Python API Reference
C++ API Reference

C++ API Reference
Kernels Module
Dequantize

Dequantize#

Warning

doxygenfile: Cannot find file “kernels/int4GroupwiseGemmKernels/dequantize.cuh

previous

Decoder XQA Runner

next

EAGLE Accept Kernels

Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, Nvidia.

Last updated on February 19, 2026.

This page is generated by TensorRT-Edge-LLM commit 53cd261.