C++ API Reference#
This section provides documentation for the TensorRT Edge-LLM C++ API.
- Action Module
- Builder Module
- Common Module
- Kernels Module
- Nv Fp4 Mo E Utils
- Apply Rope Write KV
- Argmax Kernel
- Attention Scale Utils
- Audio Fbank Kernels
- Batch Evict Kernels
- Build Layout
- Causal Conv1d
- Checkpoint Source Batch
- Common
- Conversion
- Cute Dsl F16 Moe Runner
- Cute Dsl FMHA Runner
- Cute Dsl Fmhav2 Runner
- Cute Dsl FMHA Params
- Cute Dsl Gdn Runner
- Cute Dsl GEMM Nv Fp4 Runner
- Cute Dsl GEMM Runner
- Cute Dsl Module Loader
- Cute Dsl Nvfp4 Moe Runner
- Cute Dsl Nvfp4 Moe Sm110 Runner
- Cute Dsl Ssd Runner
- Cute Dsl Tensor Descriptors
- Ddtree Kernels
- Decoder XQA Runner
- Dequant
- Dequantize
- Dflash Runtime Kernels
- Dspark Kernels
- EAGLE Accept Kernels
- EAGLE Util Kernels
- Embedding Kernels
- F16 Moe Support Kernels
- Fp16 Layout Convert
- Fp16 Moe Assemble
- Fp4 Quantize
- Gdn Kernel Utils
- Gdn Tree Chunk Kernels
- Gemma4 Audio Attention
- Gemma4 Mtp Runtime Kernels
- Gptq Activation Permutation
- Gptq Marlin Repack
- Image Util Kernels
- Initialize Cos Sin Cache
- Int4 Cute Dsl Repack
- Int4 Groupwise GEMM
- Int4 Moe Scale Repack
- Int4 Plugin V1 Repack
- Int4 Qkv Scale Concat
- Int4 Source Accessors
- Kernel
- Kernel Selector
- KV Cache Utils Kernels
- Marlin
- Marlin Repack Index
- Marlin Dtypes
- Marlin Mma
- Marlin Template
- Moe Activation Kernels
- Moe Align Sum Kernels
- Moe Marlin
- Moe Marlin Indices Kernels
- Moe Per Expert Scale Kernels
- Moe Sigmoid Group Topk Kernels
- Moe Topk Softmax Kernels
- Mtp State Scatter Kernels
- Nvfp4 Moe Layout
- Nvfp4 Moe Normalize
- Nvfp4 Moe Plugin Geforce Transform
- Nvfp4 Moe Plugin Transform
- Nvfp4 Moe Scale Transform
- Nvfp4 Moe Types
- Nvfp4 Moe Weight Transform
- Nvfp4 Dequant
- Nvfp4 Scale
- Nvfp4 Tensor
- Scale Convert
- Selective State Update
- Speculative Kernels Utils
- Ssd Varlen Metadata
- Talker Mlp Kernels
- Util Kernels
- Vectorized Types
- Multimodal Module
- Audio Runner
- Audio Utils
- Clone Encoder Runner
- Code2 Wav Runner
- Cosmos3 Edge ViT Runner
- Gemma4 Audio Runner
- Gemma4 Unified Audio Runner
- Gemma4 Unified Vision Runner
- Gemma4 ViT Runner
- Image Utils
- Intern ViT Runner
- Model Types
- Multimodal Runner
- Nemotron Omni Audio Runner
- Nemotron Omni ViT Runner
- Phi4mm ViT Runner
- Qwen25vl ViT Runner
- Qwen3omni ViT Runner
- Qwen3vl ViT Runner
- Qwen ViT Runner
- Plugins Module
- All Reduce Plugin
- Attention Plugin
- Causal Conv1d Plugin
- Cute Dsl Int4 GEMM
- Cute Dsl Int4 Gemv
- Dflash Target KV Cache Update Plugin
- Fp16 Moe Plugin
- Fused Nvfp4 GEMM All Reduce Plugin
- Gated Delta Net Plugin
- Gemma4 Audio Attention Plugin
- Int4 Groupwise GEMM Plugin
- Int4 Groupwise GEMM Plugin V2
- Int4 Moe Plugin
- Mamba Plugin
- Nvfp4 A16 GEMM Plugin
- Nvfp4 A16 Moe Plugin
- Nvfp4 Moe Plugin
- Nvfp4 Moe Plugin Geforce
- Plugin Utils
- VIT Attention Plugin
- Profiling Module
- Runtime Module
- Audio Loader
- Audio Utils
- Block Diffusion Decoder
- Block Hash
- Block Index
- Cache Record
- Checkpoint Reader
- Checkpoint Weight Assemble
- Collective Group
- Context Cache Config
- Context Cache Coordinator
- Context Cache Deployment
- Context Cache Manager
- Context Cache Metrics
- Context Cache Request
- Context Cache Types
- Decoder Registry
- Decoder Utils
- Decoding Inference Context
- Decoding Strategy
- Deepstack Binding
- Deployment Config
- Dflash Decode Utils
- Dflash Decoder
- Dspark Decoder
- EAGLE Decoder
- Embedding Preprocessor
- Engine Executor
- Eviction Planner
- External Weight Manager
- Gemma4 Embedding Preprocessor
- Gemma4 Mtp Decoder
- Hybrid Cache Manager
- Hybrid Snapshot Storage
- Image Utils
- Inference Dims
- Inference Phase
- KV Cache Manager
- KV Page Table
- Layer Debugger
- LLM Engine Config
- LLM Inference Runtime
- LLM Runtime Utils
- Logit Bias
- Lora Manager
- Mamba Cache Manager
- Mel Spectrogram
- Mtp Decoder
- Multi Device Plugin Comm Registry
- Multi Device Plugin Resource Factory
- Multi Device Plugin Resources
- Nccl Collective Backend
- Parallel Config
- Pipeline Io
- Qwen3 Omni Tts Runtime
- Registry Builder
- Resource Pools
- Reuse Plan
- Rope Cache
- Shared Resources
- Step Preparer
- Streaming
- System Prompt KV Cache
- Tensor Map
- Tensor Parallel Nccl Resources
- Tensor Parallel Plugin Resources
- Tensor Registry
- Vanilla Decoder
- Sampler Module
- Tokenizer Module