Skip to main content

Ctrl+K

TensorRT Edge-LLM

TensorRT Edge-LLM

Table of Contents

Getting Started

Overview
Official Support Matrix
Supported Models
Installation
Quick Start
Experimental ONNX-less Engine Build
Limitations and Known Issues

Examples

Examples
Speculative Decoding
Phi-4-Multimodal
ASR (Automatic Speech Recognition)
TTS (Text-to-Speech)
Vision-Language-Action
- Alpamayo-R1-10B (Vision-Language-Action)
- Cosmos3-Edge
Omni (Audio + Vision + Speech I/O)
Experimental Python API and Server
Evaluate with NeMo Evaluator

Features

LoRA (Low-Rank Adaptation)
Quantization
Multi-Device Inference
Vocabulary Reduction
FP8 KV Cache
FP8 Embedding
Streaming Output
Visual-Token Pruning (DART)
KV Cache Reuse

Input & Chat Format

Input JSON Format
Chat Template Format

Performance

Performance Benchmarks

Software Design

Checkpoint Exporter Design
Experimental ONNX-less Builder Design
Quantization Package Design
Engine Builder
C++ Runtime Overview
LLM Inference Runtime
LLM Streaming — Design
Memory monitoring
HTTP Inference Server Architecture

Models

Cosmos3-Edge Design
Nemotron-3.5-ASR

Customization

Customization Guide
Calibration Dataset Customization
TensorRT Plugins Guide

Testing

Unit-Testing the Runtime with a Substitute Engine
Code Coverage with SonarQube
Few-Layer Numeric Validation

APIs

Python API Reference
C++ API Reference

Quick Links

Releases
GitHub
Roadmap

C++ API Reference
Plugins Module
Nvfp4 A16 Blackwell Dispatch Policy

Nvfp4 A16 Blackwell Dispatch Policy#

struct Nvfp4A16BlackwellDispatch#

Public Members

Nvfp4A16BlackwellKernel kernel#

int32_t splitK#

previous

Mamba Plugin

next

Nvfp4 A16 GEMM Plugin

On this page

trt_edgellm::plugins::Nvfp4A16BlackwellDispatch
- kernel
- splitK

Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, Nvidia.

Last updated on September 03, 2026.

This page is generated by TensorRT-Edge-LLM commit e8b2952.