Experimental ONNX-less Builder Design#
The experimental builder compiles TensorRT engines directly from checkpoint configuration and safetensors:
checkpoint
-> model-specific configuration and weight conversion
-> model-specific NetworkModule
-> symbolic Tensor and F.* operations
-> TensorRT INetworkDefinition
-> optimization profiles and serialized engine
-> model-specific runtime artifacts
There is no ONNX graph, ONNX parser, or dependency on the existing ONNX exporter in this path. The result uses the existing TensorRT Edge-LLM C++ runtime contract.
For installation, commands, runtime examples, component paths, supported features, and current limitations, see Experimental ONNX-less Engine Build.
Design Goals#
Keep model definitions close to Hugging Face Transformers structure: modules create named child modules in
__init__and implementforward.Give every engine component an explicit, model-specific input and output contract.
Present TensorRT-native layers and Edge-LLM extension operations through one
ops.functionalAPI.Keep raw TensorRT network access out of model definitions.
Load configuration and safetensors on CPU without constructing or tracing a provider
torch.nn.Module.Build every component selected for a checkpoint in one command while preserving one TensorRT engine per runtime component.
Reuse operations and genuinely architecture-neutral modules, but keep configuration, weight mapping, component graphs, and runtime artifacts owned by each model family.
Package Boundaries#
The implementation is under tensorrt_edgellm/builder:
Layer |
Main files |
Responsibility |
|---|---|---|
CLI orchestration |
|
Resolve components, expand speculative base/draft builds, load the plugin once, and build components in runtime order. |
Build driver |
|
Create a strongly typed TensorRT network, bind checkpoint metadata, configure profiles, serialize the engine, and publish runtime weight metadata. |
Checkpoint contract |
|
Parse model/quantization metadata and provide streaming safetensors access. |
Component contract |
|
Define component names, output paths, profile limits, and deterministic build order. |
Model registry |
|
Map an exact checkpoint |
Model families |
|
Own configuration selection, model modules, checkpoint key conversion, preprocessing, and runtime artifacts. |
Public modeling API |
|
Provide PyTorch-like symbolic tensors, |
TensorRT lowering |
|
Own all |
Runtime artifacts |
|
Write configs, tokenizers, chat templates, embeddings, preprocessing assets, and external-weight manifests. |
The boundary is enforced by convention and imports: model files receive a
BuildContext, create Module objects, and call Tensor methods or
ops.functional functions. They do not receive an
INetworkDefinition.
Build Lifecycle#
One command follows this sequence:
cli._resolve_build_selectionloadsBundleConfigand asksmodels/registry.pyfor the exact component set associated with the rootmodel_type.cli._build_plancreates one entry per component. A speculative request expands the LLM entry into target/base and draft entries while keeping non-LLM components in the same plan.cli._buildloadslibNvInfer_edgellm_plugin.soonce, then invokes_build_onefor each planned component.core.builder.build_enginecreates a strongly typedtrt.INetworkDefinition,trt.IBuilderConfig, checkpointWeights, and the internalNetlowering backend.models.build_modelasks the registry for one concreteNetworkModule, creates aBuildContext, and invokesNetworkModule.build.NetworkModule.builddeclares component inputs withinput_tensors, calls the model’sforward, and marks only its explicitly returned outputs.core.buildercreates the component-specific optimization profiles and callsbuilder.build_serialized_network.The model-owned artifact writer emits the runtime config. Checkpoint bindings or validated transformed-sidecar references are then added to that component’s manifest.
Each loop iteration creates one independent TensorRT network. A VLM therefore
has one NetworkModule and INetworkDefinition for its LLM and another for its
visual encoder. The one-command behavior is orchestration, not a single engine
that mixes incompatible component I/O contracts.
Module And Network Ownership#
The frontend has two module types.
Module#
Module represents a nested model layer such as attention, MLP, RMSNorm,
convolution, or projector.
Child modules are constructed in
__init__.__call__enters the active build scope and delegates toforward.A module owns checkpoint names and ordinary configuration.
A module cannot add network inputs or mark outputs.
This keeps model code structurally similar to Transformers:
class Qwen3DecoderLayer(DecoderLayer):
attention_class = Qwen3Attention
class Qwen3Model(DecoderModel):
layer_class = Qwen3DecoderLayer
NetworkModule#
NetworkModule represents exactly one runtime component and one TensorRT
network.
input_tensorsdeclares the engine inputs and their dynamic dimensions.forwardcomposes model-specific submodules and returns explicitly named tensors.mark_outputsis the only model-facing output-marking path.from_configis the extension point for components that need component metadata or build-profile limits during construction.
For example, Qwen3ForCausalLM owns paged KV-cache, RoPE, context-length,
page-table, token-selection, logits, and feedback-hidden-state bindings.
Visual, audio, action, TTS, and speculative classes define different contracts
instead of inheriting an assumed LLM interface.
BuildContext carries the active Net, normalized DeviceConfig, streaming
Weights, BuildOptions, checkpoint bundle, and build arguments. The
ContextVar in ops/scope.py makes that context available while a module is
executing, so every public operation can retain a tensor-first signature
without passing a network argument through every forward call.
Operation API And TensorRT Lowering#
Model code has one operation level:
Tensor syntax for familiar expressions:
x + residual,x.reshape(...),x @ weight,x.mean(...), andx.silu().from ...ops import functional as Ffor free or multi-output operations:F.linear_from_weights,F.rms_norm,F.attention,F.vit_attention,F.gated_delta_net, andF.nvfp4_moe.
Neither API exposes whether an operation lowers to native TensorRT layers or an
Edge-LLM extension. That distinction exists only in ops/backend.py.
TensorRT API ownership is split at one explicit boundary:
core/builder.pyowns builder, network, builder-config, optimization-profile, serialization, and timing-cache lifecycle calls.ops/backend.pyowns all layer-creation calls ontrt.INetworkDefinition, including native TensorRT layers, constants, network weight inputs, and Edge-LLM extension layers.
Model modules and ops/functional never call TensorRT directly. Functional
operations define frontend semantics; the active Net in ops/backend.py
lowers those semantics into one or more TensorRT layers.
Concrete call paths are:
Model expression |
Public implementation |
Backend method in |
TensorRT API |
|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
FP16 |
|
|
Weight input or |
|
|
|
|
Decoder attention |
|
|
plugin creator plus |
ViT attention |
|
|
plugin creator plus |
Native TensorRT Layer Example#
Dense projection shows how a semantic frontend operation maps to a native TensorRT layer composition. The builder creates a strongly typed network and represents an FP16 projection using either an external weight input or a constant, according to the weight policy:
checkpoint metadata -> external weight input (default for eligible FP16 weights)
checkpoint weight -> IConstantLayer (when baked)
activation, weight -> IMatrixMultiplyLayer
optional bias -> IElementWiseLayer
The implementation is Net.linear in ops/backend.py; the strongly typed
network does not use the legacy fully-connected layer API. Quantized
projections enter through the same Linear module and
F.linear_from_weights call, after which Net.linear_from_weights dispatches
according to checkpoint metadata:
FP8, MXFP8, NVFP4, and INT8 SmoothQuant use their TensorRT quantize, dequantize, cast, MatMul, or extension lowering.
INT4 AWQ/GPTQ uses the INT4 groupwise GEMM extension.
Excluded or mixed-precision modules fall back to the FP16 path.
The model does not select a backend layer directly.
RMSNorm is another native composition. It is decomposed instead of represented by a model-owned custom operation:
x
-> cast FP32
-> square
-> reduce mean over hidden dimension
-> add epsilon
-> square root
-> divide
-> cast FP16
-> multiply checkpoint scale
ops/normalization.py owns the checkpoint-facing RMSNorm module,
ops/functional/core.py owns the semantic function, and
Net.rmsnorm owns the TensorRT layer sequence.
Edge-LLM Extension Layer Example#
Semantic extension functions accept symbolic tensors and ordinary Python attributes. For decoder attention:
model
-> F.attention(qkv, past_kv, ..., num_q_heads=..., head_size=...)
-> functional._operation.operation("attention", tensors, attributes)
-> Net.operation
-> semantic name mapped to "AttentionPlugin"
-> PluginField encoding and creator lookup
-> INetworkDefinition.add_plugin_v3
_OPERATION_CREATORS in ops/backend.py is the only semantic-name to
creator-name map. Net._operation_field is the only generic attribute encoder.
Creator names, PluginField construction, and plugin layer creation must not
appear in model or semantic functional files.
Adding An Operation#
Add operations according to semantics, not provider:
Choose the appropriate file under
ops/functional/and define a typed, tensor-first function. Add it toops/functional/__init__.py.For a native composition, add the minimal lowering method to
Netand call it from the functional function or aTensormethod.For an extension operation, add one stable semantic name to
_OPERATION_CREATORSand call the privatefunctional._operation.operationbridge.Keep checkpoint lookup and model-specific defaults in a
Module, not inNet.Keep dynamic shape and output-count validation at the highest layer that owns that contract.
A caller should not need to know whether an implementation changed from a plugin to a native TensorRT layer. That change stays behind the functional operation boundary.
Checkpoint And Weight Ownership#
BundleConfig reads root and nested component configuration with plain JSON.
DeviceConfig normalizes the fields needed by text-like runtime components.
core/quantization.py reads hf_quant_config.json or an embedded
quantization_config and resolves a concrete precision per module.
Weights keeps safetensors files memory-mapped and requests tensors by
frontend name. A model family’s weights.py owns aliases, prefix changes,
special tensor combinations, and quantized layout conversion. This prevents a
generic loader from accumulating model-name conditionals.
Eligible dense FP16 weights and large quantized extension weights can be static TensorRT network inputs:
Dense INT4 GEMM qweights and scales
INT4 MoE qweights
NVFP4 MoE weight buffers
They remain parameters owned by Linear or the model-specific expert module;
they do not appear in model forward signatures. Net.weight_input records
only final tensor metadata and the provider-checkpoint conversion recipe.
core/artifacts/external_weights.py publishes those bindings in the component
config, and the C++ ExternalWeightManager maps, transforms, and binds the
original checkpoint once when loading the engine.
Experimental Weightless Constants#
BuildArgs.weightless_engine and the CLI --weightless-engine flag opt into
a separate weight transport contract; both default to disabled. This path
requires compatible TensorRT/Myelin null-weight placeholder support, not merely
availability of the refit flags. Dense FP16, native FP8/NVFP4, and supported
INT4 checkpoint layouts are eligible; parameter lowering requires reproducible
checkpoint recipes. Execution requires vanilla single-rank decoding; both
unquantized and FP8 KV caches are supported. Quantization overrides must also use supported layouts.
The builder rejects LoRA, reduced vocabulary,
speculative configurations, non-LLM components, and explicit external-weight selections.
The existing model graph and checkpoint conversion recipes are reused:
core.builder._configure_weightless_enginevalidates scope and selectsSTRIP_PLANandREFIT_INDIVIDUAL, preserving the normal optimization level.Net.parameterandNet.weight_inputlower reproducible checkpoint parameter slots, including normalization scales, to constants with a null data pointer and the final dtype/count. The canonical bindingengine_namebecomes the named TensorRT weight;REFIT_PROXYis not used. Native FP8 matrices and NVFP4 matrices/block scales use typed placeholders. For tied embeddings, an FP16 parameter with an identical materialization recipe remains an engine input so the runtime can share its storage with the embedding. Matching includes checkpoint source, locations, shape, dtype, layout and transforms; scaled embeddings cannot share storage. This avoids both a refitted copy of the embedding and the large refittable-head compiler temporaries without introducing model-family-specific lowering. Global quantization and KV-cache scales remain build-time constants. Packed INT4 and NVFP4-A16 plugin parameters retain their byte-addressable storage types.core/artifacts/external_weights.pyretains the ordinary checkpoint recipes and addsdestination: "refit"to those entries. Bindings without a destination retain their input behavior. There is no second manifest.The component config records
weightless_engine: true. A missing or false marker selects ordinary engines; backend compatibility requires qualification with the actual TensorRT package.
Runtime initialization follows this order:
checkpoint identity and binding validation
-> checkpoint conversion into final-layout refit device storage
-> engine deserialization
-> named device-weight refit on the initialization stream
-> stream completion and transient refit-storage release
-> execution-context creation
ExternalWeightManager separates transient RefitWeightSet ownership from
persistent runtime inputs and embeddings. The embedding table is not refitted;
a tied embedding/output head cannot alias storage across those lifetimes.
runtime/weight/refitWeights.cpp checks the exact named-weight set, dtype,
count, device storage, and missing weights before refit. The completion path
drains the explicit stream before releasing storage, including a false-return
failure. runtime/exec/engineExecutor.cpp performs refit before context creation.
A build without ENABLE_EXPERIMENTAL_WEIGHTLESS=ON rejects marked bundles;
TensorRT RTX is outside this path.
Profiles And Runtime Artifacts#
Text-like components receive separate context and generation optimization
profiles. Profile construction is centralized in core/builder.py, but the
shape set is driven by bindings declared by the concrete NetworkModule.
Visual, audio, Code2Wav, and action components receive component-specific
profiles derived from their own inputs and command limits. pi0.5 additionally
owns the cross-component prefix capacity, paged K/V pool, fixed action horizon,
and hoisted-condition profiles so its direct engines retain the same bindings
as the experimental C++ policy runtime.
Artifact ownership follows the model:
models/<family>/artifacts.py
-> optional family runtime_config.py / tokenizer.py / embeddings.py
-> reusable serializers in core/artifacts/
The reusable serializers implement file formats. The family decides which files and model-specific fields are required. This keeps sidecars aligned with the component that consumes them.
Adding A Model Family#
Do not route a new architecture through another family’s model definition
unless the provider architecture and runtime contract are genuinely identical.
Add a directory under models/<family>/ with the applicable files:
File |
Required responsibility |
|---|---|
|
Select and validate component configuration from the root checkpoint. |
|
Define one |
|
Resolve checkpoint names and perform family-specific conversions. |
|
Select runtime artifact writers for every component. |
|
Add model-specific runtime fields when generic fields are insufficient. |
|
Own optional family-specific assets. |
|
Expose only the family’s intended public definitions. |
Then:
Add one
ModelFamilyentry inmodels/registry.pywith exact rootmodel_typealiases and component classes.Make each component’s
NetworkModule.input_tensorsmatch the C++ runner bindings and profile requirements.Return every runtime-visible output by name from
forward.Verify one-command
--components alloutput paths and sidecars.Add a focused end-to-end test that compiles the engine, executes the corresponding C++ runtime, and checks output accuracy.
Update the supported model matrix only after the model’s complete runtime contract is validated.
Validation#
tests/defs/test_builder_pipeline.py exercises the public contract rather than
individual helper classes:
Resolve a real checkpoint through the standard test configuration.
Invoke one all-component direct build.
Check every expected engine path and non-empty artifact.
Execute the matching C++ runtime once.
Validate modality-specific output contracts.
Run the existing dataset accuracy check.
Keep frontend tests focused on complete engine-build and runtime paths. Unit tests for isolated helpers do not substitute for a model-level output check.