Skip to main content

Ctrl+K

TensorRT Edge-LLM

TensorRT Edge-LLM

Table of Contents

Getting Started

Overview
Official Support Matrix
Supported Models
Installation
trtedge CLI
Quick Start
Experimental ONNX-less Engine Build
Limitations and Known Issues

Examples

Examples
Interactive Text Chat
Run inference with the Python runtime
Speculative Decoding
Phi-4-Multimodal
ASR (Automatic Speech Recognition)
TTS (Text-to-Speech)
MiniMax-H3
Vision-Language-Action
Omni (Audio + Vision + Speech I/O)
Python API and OpenAI-Compatible Server

Integration

Integration

Features

LoRA (Low-Rank Adaptation)
Quantization
Multi-Device Inference
Vocabulary Reduction
FP8 KV Cache
FP8 Embedding
Streaming Output
Guided Decoding
Visual-Token Pruning (DART)
KV Cache Reuse
In-Flight Batching
CodePredictor Speculative Decoding
Skip-Softmax Attention

Input & Chat Format

Input JSON Format
Chat Template Format

Performance

Performance Benchmarks

Software Design

Checkpoint Exporter Design
Experimental ONNX-less Builder Design
Quantization Package Design
Engine Builder
C++ Runtime Overview
Multi-Stage Graph
LLM Inference Runtime
LLM Streaming — Design
In-Flight Batching
Memory monitoring
HTTP Inference Server Architecture

Models

Cosmos3-Edge Design
Nemotron-3.5-ASR
pi0.5 Design
GR00T N1.7

Customization

Customization Guide
Calibration Dataset Customization
TensorRT Plugins Guide

Testing

Unit-Testing the Runtime with a Substitute Engine
Code Coverage with SonarQube
Few-Layer Numeric Validation

APIs

Python API Reference
C++ API Reference

Quick Links

Releases
GitHub
Roadmap

C++ API Reference
Safety_Inference Module
Fp16 Embedding Kernels

Fp16 Embedding Kernels#

Warning

doxygenfile: Cannot find file “safety_inference/embedding/fp16EmbeddingKernels.cuh

previous

Fp16 Embedding

next

Greedy Argmax Kernel

Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, Nvidia.

Last updated on November 13, 2026.

This page is generated by TensorRT-Edge-LLM commit 513dcd1.