Skip to main content

Ctrl+K

TensorRT Edge-LLM

TensorRT Edge-LLM

Table of Contents

Getting Started

Overview
Official Support Matrix
Supported Models
Installation
Quick Start
trtedge CLI
Experimental ONNX-less Engine Build
Limitations and Known Issues

Examples

Examples
Interactive Text Chat
Run inference with the Python runtime
Speculative Decoding
Phi-4-Multimodal
ASR (Automatic Speech Recognition)
TTS (Text-to-Speech)
Vision-Language-Action
Omni (Audio + Vision + Speech I/O)
Experimental Python API and Server
Evaluate with NeMo Evaluator

Features

LoRA (Low-Rank Adaptation)
Quantization
Multi-Device Inference
Vocabulary Reduction
FP8 KV Cache
FP8 Embedding
Streaming Output
Guided Decoding
Visual-Token Pruning (DART)
KV Cache Reuse
In-Flight Batching
CodePredictor Speculative Decoding

Input & Chat Format

Input JSON Format
Chat Template Format

Performance

Performance Benchmarks

Software Design

Checkpoint Exporter Design
Experimental ONNX-less Builder Design
Quantization Package Design
Engine Builder
C++ Runtime Overview
LLM Inference Runtime
LLM Streaming — Design
In-Flight Batching (V0)
Memory monitoring
HTTP Inference Server Architecture

Models

Cosmos3-Edge Design
Nemotron-3.5-ASR
pi0.5 Design

Customization

Customization Guide
Calibration Dataset Customization
TensorRT Plugins Guide

Testing

Unit-Testing the Runtime with a Substitute Engine
Code Coverage with SonarQube
Few-Layer Numeric Validation

APIs

Python API Reference
C++ API Reference

Quick Links

Releases
GitHub
Roadmap

C++ API Reference
Kernels Module
Marlin Mma

Marlin Mma#

Warning

doxygenfile: Cannot find file “kernels/moe/moe_marlin/marlin/marlin_mma.h

previous

Marlin Dtypes

next

Marlin Template

Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, Nvidia.

Last updated on October 28, 2026.

This page is generated by TensorRT-Edge-LLM commit 1231d64.