Skip to main content
Ctrl+K
TensorRT Edge-LLM - Home TensorRT Edge-LLM - Home

TensorRT Edge-LLM

TensorRT Edge-LLM - Home TensorRT Edge-LLM - Home

TensorRT Edge-LLM

Table of Contents

Getting Started

  • Overview
  • Supported Models
  • Installation
  • Quick Start Guide
  • Limitations and Known Issues

Examples

  • Examples
  • VLM (Vision-Language Model) Inference
  • Speculative Decoding
  • Phi-4-Multimodal
  • ASR (Automatic Speech Recognition)
  • MoE (Mixture of Experts)
  • TTS (Text-to-Speech)
  • Alpamayo-R1-10B (Vision-Language-Action)
  • Experimental High-Level Python API and Server

Features

  • LoRA (Low-Rank Adaptation)
  • Quantization
  • Vocabulary Reduction
  • FP8 KV Cache
  • FP8 Embedding
  • Streaming Output
  • System Prompt Cache

Input & Chat Format

  • Input JSON Format
  • Chat Template Format

Performance

  • Performance Benchmarks

Software Design

  • Checkpoint Exporter Design
  • Quantization Package Design
  • Engine Builder
  • C++ Runtime Overview
  • LLM Inference Runtime
  • LLM Streaming — Design

Customization

  • Customization Guide
  • Calibration Dataset Customization
  • TensorRT Plugins Guide

Testing

  • Code Coverage with SonarQube
  • Few-Layer Numeric Validation

APIs

  • Python API Reference
  • C++ API Reference
    • Action Module
      • Action Model Types
      • Action Utils
      • Alpamayo1 Action Runner
      • Alpamayo1 Runner Utils
    • Builder Module
      • Action Builder
      • Audio Builder
      • Builder Utils
      • LLM Builder
      • Visual Builder
    • Common Module
      • Binding Names
      • Check Macros
      • CUDA Macros
      • CUDA Utils
      • File Utils
      • Hash Utils
      • Input Limits
      • Logger
      • Math Utils
      • MMAP Reader
      • Rope Utils
      • Safetensors Utils
      • String Utils
      • Tensor
      • TRT Utils
      • Utf8
      • Version
    • Kernels Module
      • Nv Fp4 Mo E Utils
      • Apply Rope Write KV
      • Audio Fbank Kernels
      • Batch Evict Kernels
      • Build Layout
      • Causal Conv1d
      • Common
      • Context FMHA Runner
      • Conversion
      • Cute Dsl F16 Moe Runner
      • Cute Dsl Ffpa Runner
      • Cute Dsl FMHA Runner
      • Cute Dsl Gdn Runner
      • Cute Dsl GEMM Nv Fp4 Runner
      • Cute Dsl GEMM Runner
      • Cute Dsl Nvfp4 Moe Runner
      • Cute Dsl Nvfp4 Moe Sm110 Runner
      • Cute Dsl Ssd Runner
      • Ddtree Kernels
      • Decoder XQA Runner
      • Dequant
      • Dequantize
      • Dflash Runtime Kernels
      • EAGLE Accept Kernels
      • EAGLE Util Kernels
      • Embedding Kernels
      • F16 Moe Support Kernels
      • FMHA Params V2
      • Fp4 Quantize
      • Gdn Kernel Utils
      • Gemma4 Audio Attention
      • Gemma4 Mtp Runtime Kernels
      • Image Util Kernels
      • Initialize Cos Sin Cache
      • Int4 Groupwise GEMM
      • Kernel
      • Kernel Selector
      • KV Cache Utils Kernels
      • Marlin
      • Marlin Dtypes
      • Marlin Mma
      • Marlin Template
      • Moe Activation Kernels
      • Moe Align Sum Kernels
      • Moe Marlin
      • Moe Marlin Indices Kernels
      • Moe Per Expert Scale Kernels
      • Moe Sigmoid Group Topk Kernels
      • Moe Topk Softmax Kernels
      • Mtp State Scatter Kernels
      • Nvfp4 Moe Types
      • Nvfp4 Dequant
      • Nvfp4 Tensor
      • Scale Convert
      • Selective State Update
      • Speculative Kernels Utils
      • Ssd Varlen Metadata
      • Talker Mlp Kernels
      • Util Kernels
      • Vectorized Types
    • Multimodal Module
      • Audio Runner
      • Audio Utils
      • Code2 Wav Runner
      • Gemma4 Audio Runner
      • Gemma4 Unified Audio Runner
      • Gemma4 Unified Vision Runner
      • Gemma4 ViT Runner
      • Image Utils
      • Intern ViT Runner
      • Model Types
      • Multimodal Runner
      • Nemotron Omni Audio Runner
      • Nemotron Omni ViT Runner
      • Phi4mm ViT Runner
      • Qwen25vl ViT Runner
      • Qwen3omni ViT Runner
      • Qwen3vl ViT Runner
      • Qwen ViT Runner
    • Plugins Module
      • All Reduce Plugin
      • Attention Plugin
      • Causal Conv1d Plugin
      • Dflash Target KV Cache Update Plugin
      • Fp16 Moe Plugin
      • Fused Nvfp4 GEMM All Reduce Plugin
      • Gated Delta Net Plugin
      • Gemma4 Audio Attention Plugin
      • Int4 Groupwise GEMM Plugin
      • Int4 Moe Plugin
      • Mamba Plugin
      • Nvfp4 Moe Plugin
      • Nvfp4 Moe Plugin Geforce
      • Plugin Utils
      • VIT Attention Plugin
    • Profiling Module
      • Layer Profiler
      • Metrics
      • Nvtx Wrapper
      • Timer
    • Runtime Module
      • Audio Loader
      • Audio Utils
      • Decoder Registry
      • Decoder Utils
      • Decoding Inference Context
      • Decoding Strategy
      • Deepstack Binding
      • Deployment Config
      • Dflash Decode Utils
      • Dflash Decoder
      • EAGLE Decoder
      • Embedding Preprocessor
      • Engine Executor
      • External Weight Manager
      • Gemma4 Embedding Preprocessor
      • Gemma4 Mtp Decoder
      • Hybrid Cache Manager
      • Image Utils
      • Inference Dims
      • Inference Phase
      • KV Cache Manager
      • Layer Debugger
      • LLM Engine Config
      • LLM Inference Runtime
      • LLM Runtime Utils
      • Logit Bias
      • Lora Manager
      • Mamba Cache Manager
      • Mel Spectrogram
      • Mtp Decoder
      • Pipeline Io
      • Qwen3 Omni Tts Runtime
      • Registry Builder
      • Rope Cache
      • Shared Resources
      • Step Preparer
      • Streaming
      • System Prompt KV Cache
      • Tensor Map
      • Tensor Registry
      • Vanilla Decoder
    • Sampler Module
      • Sampling
    • Tokenizer Module
      • Pre Tokenizer
      • Token Encoder
      • Tokenizer
      • Tokenizer Utils
      • Unicode Data

Quick Links

  • Releases
  • GitHub
  • Roadmap
  • C++ API Reference
  • Multimodal Module

Multimodal Module#

API documentation for the multimodal module.

  • Audio Runner
  • Audio Utils
  • Code2 Wav Runner
  • Gemma4 Audio Runner
  • Gemma4 Unified Audio Runner
  • Gemma4 Unified Vision Runner
  • Gemma4 ViT Runner
  • Image Utils
  • Intern ViT Runner
  • Model Types
  • Multimodal Runner
  • Nemotron Omni Audio Runner
  • Nemotron Omni ViT Runner
  • Phi4mm ViT Runner
  • Qwen25vl ViT Runner
  • Qwen3omni ViT Runner
  • Qwen3vl ViT Runner
  • Qwen ViT Runner

previous

Vectorized Types

next

Audio Runner
NVIDIA NVIDIA
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, Nvidia.

Last updated on July 22, 2026.

This page is generated by TensorRT-Edge-LLM commit f98267f.