TensorRT Plugins Guide#
This documentation explains the usage of TensorRT plugins with TensorRT Edge-LLM and guides users to make further customizations.
Overview#
TensorRT plugins are custom operations that extend the functionality of the TensorRT core library through user-defined layer implementations. Within the TensorRT Edge-LLM framework, plugins provide specialized implementations for key large language model (LLM) inference operations that require optimizations beyond those available through standard TensorRT library deliverables.
Plugin Architecture and Capabilities#
TensorRT plugins are user-defined layers that implement the IPluginV2DynamicExt or IPluginV3 interface. Note that TensorRT Edge-LLM is migrating all plugins to V3. TensorRT Plugins provide the following capabilities:
Feature Extension: Extend functionality of existing TensorRT versions with new runtime and kernel level optimizations.
Modular Encapsulation: Package complex computational logic into reusable components with configurable parameters.
Current plugins#
AttentionPlugin: Implements standard MHA (Multi-Head Attention) and GQA (Group Query Attention).
Int4GroupwiseGemmPlugin: Int4 weights-only groupwise GEMM and GEMV using the AWQ CUDA-C++ kernels (V1 plugin-packed weights).
Int4GroupwiseGemmPluginV2: Int4 weights-only groupwise GEMM and GEMV using the cuteDSL W4A16 kernels (fragment-layout weights). Default backend (see Backend Selection).
LowRankCandidateHeadPlugin: Output head of EAGLE drafts with a low-rank candidate head, such as EAGLE-4 drafters; emits the top candidate token IDs and log-probabilities per row instead of full-vocabulary logits.
AttentionPlugin#
Functional Description:
Handles Rotary positional encoding, KVCache I/O, and MHA/GQA attention computation.
Implements FP16 precision and covers all supported SMs of TensorRT Edge-LLM.
Supports FP8 KV cache on SM89, SM90, SM100, SM101, SM110, SM120, and SM121.
Supports prefill (normal and chunked) stage causal attention.
Supports vanilla decoding attention and tree decoding attention that is used by EAGLE and DFlash speculative decoding.
Supports linear KVCache with equal capacity within one batch.
Pads to maximum input sequence length within the batch for prefill execution.
Configuration Parameters:
num_q_heads: Integer specification of query attention head countnum_kv_heads: Integer specification of key-value head count (enables MQA/GQA configurations)head_size: Integer specification of per-head dimension sizeenable_tree_attention: Boolean flag to enable tree attention for speculative decoding implementationskv_cache_type: Data type for KV cache (FP16 or FP8)
Decoder tensor contract (ragged ABI v1):
Tensor |
Shape |
Meaning |
|---|---|---|
|
|
Token-major physical rows. QKV is packed on its last dimension. |
|
|
Absolute position for each physical row. Padding uses |
|
|
Physical row starts. Phases 1-7 use entry padding, so entry |
|
|
Logical per-entry lengths. Padding changes |
|
|
Maps current entries to resident KV/recurrent/conv pool rows. Current entry order is independent of resident slot order. |
|
|
Current-entry page-table view into the resident paged KV pools. |
|
token-aligned with |
RoPE, DeepStack/PLE, vision metadata, tree parent/depth, and similar decoder-side data follow the same physical rows and padding sentinels. |
|
|
Physical token rows selected for logits; this is not a per-batch last-token tensor. |
|
|
Shape-only carrier; the INT32 payload is ignored. |
N is the active entry count, S_pad is the homogeneous step’s physical
width, and T_exec = N * S_pad. T_valid = sum(query_lengths) can be smaller
than T_exec. Attention output is [T_exec, hidden]. Past/present KV and
recurrent/conv state use fixed resident pools and keep the same address; only
rows selected by state_indices participate in the step. Padded rows must not
read or update persistent state.
The phase marker extent has one shared meaning in Attention, GDN, causal Conv, Mamba, and the runtime:
Extent |
Phase |
|---|---|
1 |
context prefill |
2 |
context chunk |
3 |
autoregressive decode |
4 |
speculative draft proposal |
5 |
speculative target verify |
6 |
diffusion denoise |
7 |
diffusion commit |
8 |
compact mixed prefill and decode |
Phase 8 (compact) contract:
Phase 8 is implemented by Attention, GDN, causal Conv, and Mamba2 SSD
(update_ssm_state) layers. The runtime emits it for the compact steps of a
runtime session opened with compact execution; such a session still runs a step
in which every resident decodes as phase 3 (see
In-Flight Batching).
Rows: the first
Npentries are prefill sequences packed contiguously in rows[0, T_p); each of the remainingNd = N - Npentries owns one decode row, soT_exec = T_p + Nd.query_start_offsetsholds the prefix sums ofquery_lengths.Npis the extent of the shape-onlycontext_sequence_count_carrier, with0 <= Np <= N;Np = 0meansT_exec = N. Every prefill sequence has at least one row; on the host a plugin can check onlyT_p >= Np.Dispatch: decode the ragged metadata with
decodeRaggedPluginMetadata(..., RaggedLayoutSupport::kPaddedAndCompact)(the defaultkPaddedOnlyrejects phase 8) and select the kernels by phase. A plugin decides from its attributes, static shapes, and the GPU whether it can run phase 8.getWorkspaceSize()andenqueue()must call the same decision function, with inputs that do not depend on the step’s sequence count: the workspace reserved at build time covers phase 8 exactly when enqueue accepts it.getWorkspaceSize()also passes that decision toreportCompactWorkspace()on every call, because TensorRT may query a layer more than once; the builders writebuilder_config.compact_executiononly when every such layer reported a reservation, and the runtime runs compact steps only on engines with that field. The decision function lives incpp/kernels/compactPhase/, and the runtime’s compact capability check (compactLayerGaps) calls the same function with the shapes from the engine’sconfig.json; a new phase-8 plugin adds its function there and its layer kind to that check. An unsupported layer fails the phase 8 enqueue with<plugin> '<layer>' (<shape>): compact execution is not supported for this layer on this GPU.before it decodes metadata or writes any state. TensorRT passescreatePlugin()an empty name at deserialization, so a phase-8 plugin serializes its layer name in thelayer_namefield (layerNameField()), restores it withresolvePluginLayerName(), and formats the layer withdescribePluginLayer(). The runtime treats such a refusal as permanent: it fails the step and disables compact execution for the runtime’s life. TheRequestEnginethen continues with single-cohort steps or stops, depending on its scheduling mode (see In-Flight Batching).Decode rows: rows
[T_p, T_exec)must be bit-identical to phase 3. Emit them through the same decode function phase 3 uses (enqueueDecodeRows), offset byT_prows andNpsequences. Outputs and state are written in place; only metadata that a kernel needs aligned may be staged in workspace. WithNp = 0the kernel sequence equals phase 3.
Old batch-major engines are not supported by this ABI. Re-export the ONNX model and rebuild the engine, and use export, build, plugin, and runtime artifacts from a compatible revision. The plugin creator version is unchanged: compatibility is enforced by ragged engine metadata, bindings, and tensor descriptors, not by a dual-ABI plugin implementation.
The packaged experimental decoder forward() APIs are supported only through
their token-major RaggedDecoderInputs argument. Remaining batch-major rank-3
external APIs or plugin inputs in that package belong to standalone vision,
audio, or action encoder-domain components; they are not a legacy decoder
fallback. Decoder-internal [T, heads, channels] intermediates are valid: they
preserve token-major ownership and flatten to [T, hidden] before crossing the
decoder plugin boundary.
Application Domains:
Transformer-based autoregressive language models that adopt standard MHA/GQA.
Kernel Sources#
Attention kernels are compiled into CUDA binaries. We provide the methods to produce CUDA binaries in kernelSrcs/.
Kernel Libraries:
fmha: Canonical Context and ViT attention CuTe DSL AOT family built withkernelSrcs/build_cutedsl.py. It includes the FP16 FMHA-v2 kernels fromkernelSrcs/fmha_v2_cutedsl/fmha.pyon supported GPUs and the optimized Blackwell overlay fromkernelSrcs/fmha_cutedsl_blackwell/fmha.pyon SM100/SM101/SM110.xqa: Performant decoding attention kernels developed by NVIDIA. Implements normal decoding and tree-attention decoding.
Integration Workflow#
The AttentionPlugin integrates into the TensorRT Edge-LLM inference pipeline through the following stages:
Export Phase: During ONNX model export,
tensorrt_edgellmemits attention custom-op nodes through TensorRT Edge-LLM ONNX translations.Engine Construction: The TensorRT engine builder identifies plugin operations via registered plugin creators and integrates them into the optimized computation graph.
Runtime Execution: During inference, the AttentionPlugin executes as a node within the TensorRT engine’s execution graph, with memory management handled by the TensorRT runtime.
Int4GroupwiseGemmPlugin#
Functional Description
Implements A([M, K]) x B([K, N]) GEMM semantic where A is activation input, B is weights input.
Supports INT4 weights-only groupwise quantization GEMM.
Supports group size of 128.
Accumulation is performed in FP16 precision for both GEMM and GEMV kernels.
Implements symmetric quantization schema and zero-points is not supported.
Configuration Parameters:
N: Output feature dimensions of the GEMM operationK: Inner dimensions of the GEMM operationGroupSize: Number of INT4 weight items corresponding to one scaling factor (currently supports 128 only)
Input Tensors:
GEMMInput: Input activation tensor for the GEMM computation.Int4Weights: INT4 weights in the V1 plugin layout, packed into INT8 datatypes.ScalingFactors: Groupwise scaling factors.
Output Tensors:
GEMMOutput: Result of the INT4 groupwise GEMM computation.
Kernel Sources#
A simplified kernel implementation is provided for this plugin. Evaluation indicates that this INT4 GEMM kernel achieves performance comparable to CUTLASS implementations on target production platforms (primarily Orin SKUs) with input sequence lengths (ISLs) of 2K to 3K tokens. Note that the GEMM kernel may not deliver sufficient performance for speculative decoding use cases with draft tree sizes of 64 to 128 tokens.
Integration Workflow#
The Int4GroupwiseGemmPlugin integrates into the TensorRT Edge-LLM inference pipeline through the following stages:
Quantization Phase:
trtedge quantizeor a supported pre-quantized checkpoint stores linear layers in INT4 weights-only groupwise format with group size 128.Export Phase: During ONNX model export,
tensorrt_edgellmemits quantized matrix multiplication custom-op nodes for Int4GroupwiseGemmPlugin.Engine Construction: The TensorRT engine builder identifies Int4GroupwiseGemmPlugin operations via registered plugin creators and integrates them into the optimized computation graph.
Runtime Execution: During inference, the Int4GroupwiseGemmPlugin executes quantized GEMM/GEMV operations as nodes within the TensorRT engine’s execution graph.
Int4GroupwiseGemmPluginV2#
Int4GroupwiseGemmPluginV2 is the default INT4 groupwise GEMM backend. It shares the Int4GroupwiseGemmPlugin GEMM semantics — INT4 weights-only groupwise quantization, group size 128, and symmetric (zero-point-free) quantization — but dispatches to AOT-compiled CuTe DSL (cuteDSL) W4A16 kernels instead of the AWQ CUDA-C++ kernels. The GEMM accumulates in FP32; the GEMV keeps FP16 partial sums.
The fragment layout supports any positive output width. The final 128-channel fragment is bounds-predicated by both the GEMM and GEMV kernels, so narrow projections retain their native output shape.
Configuration Parameters:
N: Output feature dimensions of the GEMM operationK: Inner dimensions of the GEMM operationGroupSize: Number of INT4 weight items corresponding to one scaling factor. The V2 runtime supports multiples of the 64-wide K tile, including 64 and 128.
Input Tensors:
GEMMInput: Input activation tensor for the GEMM computation.Int4Weights: INT4 weights in the cuteDSL fragment layout, packed into INT8 datatypes.ScalingFactors: Groupwise scaling factors.
Output Tensors:
GEMMOutput: Result of the INT4 groupwise GEMM computation.
Kernel Sources#
The plugin dispatches to AOT-compiled cuteDSL W4A16 GEMM kernels. The candidate kernel variants are generated by kernelSrcs/build_cutedsl.py (the int4_fp16_gemm group) into a static library, and the TensorRT plugin-V3 autotuner selects the best variant per problem shape at engine build time. A CUDA-core GEMV kernel serves the small-M (decode) regime, consuming the same fragment-layout weight buffer as the GEMM. The ONNX path prepares this layout during export; the ONNX-less builder can prepare it once during runtime initialization from the original checkpoint. Neither path repacks weights during inference.
Backend Selection#
Both plugins are registered and coexist. The --int4-gemm-plugin-version export CLI argument selects which one an INT4 checkpoint is exported for (the quantized checkpoint itself is backend-agnostic):
2(default):Int4GroupwiseGemmPluginV2(cuteDSL fragment weights).1:Int4GroupwiseGemmPlugin(V1 plugin-packed weights) — legacy fallback.
The same selector drives both the weight repack (checkpoint/repacking.py) and the emitted custom op (models/linear.py), so the exported ONNX node and its weight layout always agree. The selected backend is logged once per export run.
LowRankCandidateHeadPlugin#
The ONNX export emits the trt_edgellm::LowRankCandidateHeadPlugin node as the
output head of an EAGLE draft that has a low-rank candidate head, such as the
EAGLE-4 drafters.
Functional Description:
Projects each hidden row to rank
r(s = down_proj h, accumulated in FP32 and stored as FP16) and scores every vocabulary token withup_proj s.Keeps the
num_candidates(Kc) highest-scoring tokens; equal scores keep the smaller token ID and NaN scores rank last.Rescores the
Kccandidates with exact FP32 dot products against theirlm_headrows and outputs the log-softmax over thoseKclogits. A non-finite logit sorts last with log-probability negative infinity.Results are deterministic and independent of the other rows in the batch.
Supports FP16 inputs only and requires SM80 or newer.
enqueue()allocates no device memory and keeps no state between calls, so the draft engine’s CUDA graphs can capture it.
Configuration Parameters:
num_candidates(Kc): ONNX attribute, the number of candidates per row, in[1, 256]and at most the vocabulary sizeV.hidden_size,rank,vocab_size: Read from the weight shapes at engine build time and serialized.max_rows: The largest row countRof the optimization profiles, serialized.workspace_bytes: The workspace the plugin needs for 64 rows on the build GPU, serialized. Every optimization profile reserves this size.
Input Tensors:
hidden:[R, H]FP16 final-norm hidden states.His a positive multiple of 8.down_proj:[r, H]FP16 low-rank factor.ris a multiple of 16 in[16, 256].up_proj:[V, r]FP16 low-rank factor, one row per vocabulary token.lm_head:[V, H]FP16 output projection.
The weight shapes must be the same in every optimization profile.
Output Tensors:
support_ids:[R, Kc]INT32 full-vocabulary token IDs, sorted by exact logit (descending), then token ID (ascending).support_log_probs:[R, Kc]FP32 log-probabilities ofsupport_ids.
Rows and Workspace:
Rrows run in passes of at most 64 rows; each pass readsup_projonce.The plugin’s first launch on a GPU picks its configuration for that GPU. If that GPU’s preferred configuration needs more workspace than
workspace_bytesbecause the build GPU had fewer SMs, less shared memory, or a smaller L2 cache (for example, a Jetson power mode with fewer GPU TPCs, or a smaller MIG instance), the plugin uses the first of a fixed sequence of smaller configurations that fits and logs the sizes at the INFO level. The outputs do not depend on the configuration; only the speed does.If
workspace_bytesis below the smallest of those configurations, everyenqueueV3call that runs the plugin fails before the plugin launches anything. The error states the size that the GPU needs. Rebuild the engine on that GPU, in the power mode it runs in.