Skip to main content
Ctrl+K
TensorRT Edge-LLM - Home TensorRT Edge-LLM - Home

TensorRT Edge-LLM

TensorRT Edge-LLM - Home TensorRT Edge-LLM - Home

TensorRT Edge-LLM

Table of Contents

Getting Started

  • Overview
  • Official Support Matrix
  • Supported Models
  • Installation
  • trtedge CLI
  • Quick Start
  • Experimental ONNX-less Engine Build
  • Limitations and Known Issues

Examples

  • Examples
  • Interactive Text Chat
  • Run inference with the Python runtime
  • Speculative Decoding
  • Phi-4-Multimodal
  • ASR (Automatic Speech Recognition)
  • TTS (Text-to-Speech)
  • MiniMax-H3
  • Vision-Language-Action
    • Alpamayo-R1-10B (Vision-Language-Action)
    • Cosmos3-Edge
    • pi0.5
    • GR00T N1.7
  • Omni (Audio + Vision + Speech I/O)
  • Python API and OpenAI-Compatible Server

Integration

  • Integration
    • Benchmark the OpenAI Server with AIPerf
    • Evaluate with NeMo Evaluator
    • Evaluate with VLMEvalKit
    • Use Claude Code
    • Use OpenClaw
    • Use Codex

Features

  • LoRA (Low-Rank Adaptation)
  • Quantization
  • Multi-Device Inference
  • Vocabulary Reduction
  • FP8 KV Cache
  • FP8 Embedding
  • Streaming Output
  • Guided Decoding
  • Visual-Token Pruning (DART)
  • KV Cache Reuse
  • In-Flight Batching
  • CodePredictor Speculative Decoding
  • Skip-Softmax Attention

Input & Chat Format

  • Input JSON Format
  • Chat Template Format

Performance

  • Performance Benchmarks

Software Design

  • Checkpoint Exporter Design
  • Experimental ONNX-less Builder Design
  • Quantization Package Design
  • Engine Builder
  • C++ Runtime Overview
  • Multi-Stage Graph
  • LLM Inference Runtime
  • LLM Streaming — Design
  • In-Flight Batching
  • Memory monitoring
  • HTTP Inference Server Architecture

Models

  • Cosmos3-Edge Design
  • Nemotron-3.5-ASR
  • pi0.5 Design
  • GR00T N1.7

Customization

  • Customization Guide
  • Calibration Dataset Customization
  • TensorRT Plugins Guide

Testing

  • Unit-Testing the Runtime with a Substitute Engine
  • Code Coverage with SonarQube
  • Few-Layer Numeric Validation

APIs

  • Python API Reference
  • C++ API Reference
    • Action Module
      • Action KV Batch
      • Action Model Types
      • Action Utils
      • Alpamayo1 Action Runner
      • Alpamayo1 Runner Utils
    • Builder Module
      • Action Builder
      • Audio Builder
      • Builder Utils
      • Intern Vl Config
      • LLM Builder
      • Remote Auto Tuning Utils
      • Safety Builder Config
      • Safety Engine Serialization
      • Safety LLM Builder
      • Safety LLM Shape Utils
      • Safety Plugin Loader
      • Safety Visual Builder
      • Safety Visual Shape Utils
      • Visual Builder
    • Chattemplate Module
      • Chat Template
      • Inja Callbacks
      • Inja Json
      • Inja Renderer
    • Common Module
      • All Reduce Path
      • Binding Names
      • Check Macros
      • Counter Rng
      • CUDA Macros
      • CUDA Utils
      • Data Type
      • Execution Phase
      • File Utils
      • Hash Utils
      • Identity Hash
      • Input Limits
      • Logger
      • Math Utils
      • MMAP Reader
      • Paged KV Types
      • Parallel Artifact Names
      • Rope Utils
      • Safetensors Utils
      • Skip Softmax Scale Table
      • Spec Decode Config Utils
      • Spec Decode Limits
      • String Utils
      • Tensor
      • TRT Interop
      • TRT Utils
      • Utf8
      • Version
    • Kernels Module
      • Nv Fp4 Mo E Utils
      • Apply Rope Write KV
      • Argmax Kernel
      • Attention Scale Utils
      • Audio Fbank Kernels
      • Batch Evict Kernels
      • Build Layout
      • Causal Conv1d
      • Checkpoint Source Batch
      • Common
      • Compact Phase Support
      • Conversion
      • Cp Spec Kernels
      • Ctm Decode Attention Runner
      • Cute Dsl Causal Conv1d Runner
      • Cute Dsl Dual GEMM Runner
      • Cute Dsl F16 Moe Runner
      • Cute Dsl FMHA Runner
      • Cute Dsl Fmhav2 Runner
      • Cute Dsl FMHA Params
      • Cute Dsl FMHA Types
      • Cute Dsl Gdn Runner
      • Cute Dsl GEMM Nv Fp4 Runner
      • Cute Dsl GEMM Runner
      • Cute Dsl Layer Norm Runner
      • Cute Dsl Module Loader
      • Cute Dsl Nvfp4 Dual GEMM Runner
      • Cute Dsl Nvfp4 Moe Runner
      • Cute Dsl Nvfp4 Moe Sm110 Runner
      • Cute Dsl Qsa Sparse Runner
      • Cute Dsl Rms Norm Runner
      • Cute Dsl Ssd Runner
      • Cute Dsl Tensor Descriptors
      • Cute Dsl W2 A16 GEMM Runner
      • Cute Dsl W2 A16 Moe Runner
      • Dart Gather Kernels
      • Dart Selector Kernels
      • Ddtree Kernels
      • Decoder XQA Jit Compiler
      • Decoder XQA Runner
      • Dequant
      • Dequantize
      • Dflash2 Candidate Selector
      • Dflash2 Grouped Dynamic Conv
      • Dflash Runtime Kernels
      • Domino Rollout Kernels
      • Dspark Kernels
      • Embedding Kernels
      • F16 Moe Support Kernels
      • Fnv1a Hash Kernel
      • Force Acceptance Kernels
      • Fp16 Layout Convert
      • Fp16 Moe Assemble
      • Fp4 Quantize
      • Gdn Kernel Utils
      • Gdn Pdl Config
      • Gdn Tree Chunk Kernels
      • Gemma4 Audio Attention
      • Gemma4 Mtp Runtime Kernels
      • Gptq Activation Permutation
      • Gptq Marlin Repack
      • Grouped Lora GEMM
      • Image Util Kernels
      • Initialize Cos Sin Cache
      • Int4 Cute Dsl Repack
      • Int4 Groupwise GEMM
      • Int4 Moe Scale Repack
      • Int4 Plugin V1 Repack
      • Int4 Qkv Scale Concat
      • Int4 Source Accessors
      • Iq2 Xxs Repack
      • Kernel
      • Kernel Selector
      • KV Cache Utils Kernels
      • Lilicorr Chain Walk
      • Low Rank Candidate Head Kernels
      • Marlin
      • Marlin Repack Index
      • Marlin Dtypes
      • Marlin Mma
      • Marlin Template
      • Minimax H3 Kernels
      • Moe Activation Kernels
      • Moe Align Sum Kernels
      • Moe Marlin
      • Moe Marlin Indices Kernels
      • Moe Per Expert Scale Kernels
      • Moe Sigmoid Group Topk Kernels
      • Moe Topk Softmax Kernels
      • Mtp State Scatter Kernels
      • Nvfp4 A16 Blackwell GEMM Runner
      • Nvfp4 A16 Blackwell Gemv Jit Compiler
      • Nvfp4 A16 Blackwell Gemv Jit Runner
      • Nvfp4 A16 Blackwell Moe Dispatch Policy
      • Nvfp4 A16 Blackwell Moe Jit Compiler
      • Nvfp4 A16 Blackwell Moe Jit Runner
      • Nvfp4 A16 Blackwell Moe Runner
      • Nvfp4 A16 Blackwell Support
      • Nvfp4 Moe Layout
      • Nvfp4 Moe Normalize
      • Nvfp4 Moe Pdl Config
      • Nvfp4 Moe Plugin Geforce Transform
      • Nvfp4 Moe Plugin Transform
      • Nvfp4 Moe Scale Transform
      • Nvfp4 Moe Types
      • Nvfp4 Moe Weight Transform
      • Nvfp4 W4 A16 Marlin Repack
      • Nvfp4 Dequant
      • Nvfp4 Scale
      • Nvfp4 Tensor
      • Ple Gather
      • Ple Scatter
      • Plugin Jit Blob
      • Plugin Jit Compile Cache
      • Plugin Jit Compiler
      • Plugin Jit Cubin
      • Plugin Jit Digest
      • Plugin Jit Embedded Sources
      • Plugin Jit Key
      • Plugin Jit Module Registry
      • Q8 0 Kernels
      • Q8 0 Linear
      • Qsa Indexer Jit Compiler
      • Qsa Indexer Kernels
      • Qsa Indexer Runner
      • Request Stable Rng Kernels
      • Scale Convert
      • Selective State Update
      • Sliding Window Utils
      • Speculative Kernels
      • Speculative Kernels Utils
      • Speculative Sampling
      • Speculative Sampling Kernels
      • Ssd Varlen Metadata
      • Talker Mlp Kernels
      • Util Kernels
      • Vectorized Types
      • Video Sparse Attention Runner
      • W2a16 Small M GEMM
      • W2a16 Small M Moe
      • XQA Kernel Support
      • XQA Kernel Types
      • XQA Launch Config
    • Multimodal Module
      • Audio Runner
      • Audio Utils
      • Clone Encoder Runner
      • Code2 Wav Runner
      • Code2 Wav Stage
      • Code Predictor
      • Cosmos3 Edge ViT Runner
      • Gemma4 Audio Runner
      • Gemma4 Unified Audio Runner
      • Gemma4 Unified Vision Runner
      • Gemma4 ViT Runner
      • Image Utils
      • Intern ViT Runner
      • Minimax H3 Engine
      • Minimax H3 Stages
      • Model Types
      • Multimodal Runner
      • Muse Glimmer ViT Runner
      • Nemotron Omni Audio Runner
      • Nemotron Omni ViT Runner
      • Phi4mm ViT Runner
      • Qwen25vl ViT Runner
      • Qwen3omni ViT Runner
      • Qwen3vl ViT Runner
      • Qwen ViT Runner
      • Talker Runner
      • Talker Runner Keys
      • Talker Token Ids
    • Plugins Module
      • All Reduce Plugin
      • Attention Plugin
      • Causal Conv1d Plugin
      • Cute Dsl Int4 GEMM
      • Cute Dsl Int4 Gemv
      • Dflash2 Grouped Dynamic Conv Plugin
      • Dflash Target KV Cache Update Plugin
      • Dual GEMM Plugin
      • Fp16 Moe Plugin
      • Fused Nvfp4 GEMM All Reduce Plugin
      • Gated Delta Net Plugin
      • Gemma4 Audio Attention Plugin
      • Int4 Groupwise GEMM Plugin
      • Int4 Groupwise GEMM Plugin V2
      • Int4 Moe Plugin
      • Layer Norm Plugin
      • Low Rank Candidate Head Plugin
      • Mamba Plugin
      • Nvfp4 A16 Blackwell Dispatch Policy
      • Nvfp4 A16 Blackwell GEMM Plugin
      • Nvfp4 A16 Blackwell Moe Plugin
      • Nvfp4 A16 GEMM Plugin
      • Nvfp4 A16 Moe Plugin
      • Nvfp4 Dual GEMM Plugin
      • Nvfp4 Moe Plugin
      • Nvfp4 Moe Plugin Geforce
      • Ple Gather Plugin
      • Plugin Utils
      • Q8 0 Linear Plugin
      • Qkv Concat Plugin
      • Qsa Attention Plugin
      • Ragged Plugin Metadata
      • Rms Norm Plugin
      • Sparse Attention Plugin
      • VIT Attention Plugin
      • W2a16 GEMM Plugin
      • W2a16 Moe Plugin
    • Profiling Module
      • Metrics
      • Nvtx Wrapper
      • Request Profiler
      • Timer
    • Runtime Module
      • Audio Loader
      • Audio Utils
      • Block Diffusion Decoder
      • Block Hash
      • Block Index
      • Bounded Swa KV Page Manager
      • Cache Record
      • Checkpoint Reader
      • Checkpoint Weight Assemble
      • Collective Group
      • Compact Capability
      • Compact Media Reservation
      • Context Cache Config
      • Context Cache Coordinator
      • Context Cache Deployment
      • Context Cache Manager
      • Context Cache Metrics
      • Context Cache Request
      • Context Cache Types
      • Dart Pruner
      • Decoder Registry
      • Decoder Utils
      • Decoding Inference Context
      • Decoding Strategy
      • Deepstack Binding
      • Deployment Config
      • Deployment Options
      • Device Failure
      • Dflash Decode Utils
      • Dflash Decoder
      • Domino Decoder
      • Dspark Decoder
      • EAGLE Decoder
      • Embedding Preprocessor
      • Encoder Embedding Cache
      • Engine Executor
      • Eviction Planner
      • External Weight Manager
      • Gdn Conv Commit Policy
      • Gemma4 Embedding Preprocessor
      • Gemma4 Mtp Decoder
      • Guided Decoder
      • Hybrid Cache Manager
      • Hybrid Snapshot Storage
      • Image Utils
      • Inference Dims
      • Inference Phase
      • Jump Forward
      • KV Cache Manager
      • KV Page Table
      • Layer Debugger
      • LLM Engine Config
      • LLM Inference Runtime
      • LLM Rank Runtime
      • LLM Runner
      • LLM Runtime Utils
      • Logit Bias
      • Lora Manager
      • Mamba2 Commit Policy
      • Mamba Cache Manager
      • Managed KV Cache Request
      • Mel Spectrogram
      • Model Artifacts
      • Mtp Decoder
      • Multi Device Plugin Comm Registry
      • Multi Device Plugin Resource Factory
      • Multi Device Plugin Resources
      • Multi Stage Runtime
      • Nccl Collective Backend
      • Ngram Row Ids
      • Parallel Config
      • Pipeline Io
      • Ple Cache
      • Ple File Utils
      • Ple Io Budget
      • Ple Read Executor
      • Ple Read Plan
      • Ple Row Store
      • Ple Runtime Config
      • Ple Table
      • Qwen3 Omni Tts Runtime
      • Qwen4 Exp Ngram Preprocessor
      • Ragged Batch Builder
      • Recurrent Commit Policy
      • Recurrent Commit Policy Factory
      • Recurrent State Manager
      • Recurrent State Types
      • Refit Weights
      • Registry Builder
      • Request Stable Rng
      • Resident Slot Pool
      • Resource Pools
      • Reuse Plan
      • Rope Cache
      • Route
      • Rtx Runtime Cache
      • Runtime Benchmark Options
      • Runtime Coordinator
      • Runtime Resources
      • Runtime Session
      • Scheduled Step
      • Session Options
      • Shared Resources
      • Spec Common State Tracker
      • Spec Decode Utils
      • Spec Ple Staging
      • Spec Reuse Contract
      • Spec State Plan
      • Stage
      • Stage Config
      • Stage Registry
      • Step Preparer
      • Streaming
      • Swa KV Cache State
      • Tensor Map
      • Tensor Parallel Nccl Resources
      • Tensor Parallel Plugin Resources
      • Tensor Registry
      • Vanilla Decoder
      • Vision Gather
      • Visual Token Pruner
    • Safety_Inference Module
      • Byte Level Bpe Tokenizer
      • Container Storage
      • Declared Resource Manager
      • Embedding Capability
      • Estd Allocator
      • Estd Vector
      • Fp16 Embedding
      • Fp16 Embedding Kernels
      • Greedy Argmax Kernel
      • Greedy Sampler Capability
      • Internvl3 Embedding Capability
      • Internvl3 Multimodal Preprocessor Capability
      • Internvl3 Preprocess Kernels
      • Internvl3 Request Context
      • Internvl3 Request Processing
      • Internvl3 Tokenizer Capability
      • Internvl3 Vanilla Inference
      • Internvl Image Kernels
      • Internvl Image Plan
      • Lifecycle State
      • LLM Model Local Data
      • LLM Request Context
      • Lm Stage Context
      • Model Inference
      • Model Inference Catalog
      • Model Inference Types
      • Multimodal Preprocessor Capability
      • Processing Capabilities
      • Processing Capability Coordinator
      • Rgb Resize Normalize Kernels
      • Rope Cache
      • Runner Resource Resolver
      • Runtime Resource
      • Safety Attention Contract
      • Safety Attention Kernels
      • Safety Attention Modules
      • Safety Attention Plugin
      • Safety Blackwell Launch
      • Safety Creator Registration
      • Safety Error Recorder
      • Safety Error State
      • Safety FMHA Launch
      • Safety Generation Types
      • Safety Graph Initialization
      • Safety Model Local Data
      • Safety Plugin Export
      • Safety Plugin Registry
      • Safety Request Processing
      • Safety Rope Kernel
      • Safety Runtime
      • Safety Status
      • Safety Status Utils
      • Safety Tensor View
      • Sampler Capability
      • Single Request Text Response
      • Tokenizer Capability
      • TRT Resource Manager
      • TRT Safety Graph Executor
      • TRT Safety Runner
      • Vanilla Autoregressive Strategy
      • Vlm Request Context
    • Sampler Module
      • Diffusion Gemma Sampling
      • Sampling
    • Scheduler Module
      • Batch Compatibility
      • Command Queue
      • Engine Channel
      • Engine Command
      • Request Coordinator
      • Request Engine
      • Request Identity
      • Request Record
      • Scheduling Policy
      • Step Planner
    • Tokenizer Module
      • Pre Tokenizer
      • Token Encoder
      • Tokenizer
      • Tokenizer Utils
      • Unicode Data

Quick Links

  • Releases
  • GitHub
  • Roadmap
  • C++ API Reference
  • Safety_Inference Module
  • Single Request Text Response

Single Request Text Response#

class SingleRequestTextResponse#

Public Functions

SafetyStatus formatResponse(
TokenizerCapability &tokenizer,
std::size_t maxResponseBytes,
ModelInferenceResult const &result,
LLMGenerationResponse &response
) noexcept#

previous

Sampler Capability

next

Tokenizer Capability
On this page
  • trt_edgellm::safety::SingleRequestTextResponse
    • formatResponse()
NVIDIA NVIDIA
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025, Nvidia.

Last updated on November 13, 2026.

This page is generated by TensorRT-Edge-LLM commit 513dcd1.