Skip to main content

Model Families

A model family is one complete vertical slice under families/<family>/:

families/<family>/
  support.py
  model.py
  requirements.txt          # optional
  runtime/CMakeLists.txt
  runtime/*.cpp
  tests/test_e2e.py
  tests/manifests/*.json
  tests/thresholds/*.json    # optional

Discovery and build​

support.py performs dependency-free exact matching of root model metadata and declares the supported tasks plus default. It must not import the family build, TensorRT, PyTorch, Transformers, or another family.

After exactly one match, the core imports only that family's model.py and calls its plain build(request, writer) function. The family owns config and weight interpretation, TensorRT graph topology, precision/quantization policy, engine construction, and bundle-section semantics.

K2-Horizon-Uno task contract​

IFM/K2-Horizon-7B-Uno is an adapter-only checkpoint. The independent k2_horizon_uno family resolves its pinned IFM/K2-Horizon-7B base and compiles the rank-128 conditional LoRA into one BF16 TensorRT engine. Build it directly from the adapter ID:

python -m tensorrt_model_connect build IFM/K2-Horizon-7B-Uno \
  --revision ec92bbd768f4a404319625204544782e3377bcd7 \
  --precision bf16 \
  -o k2-horizon-7b-uno.bundle

Run the qualified high-reasoning chat path with the family-owned linear mode:

trtmc run k2-horizon-7b-uno.bundle \
  --runtime-root /opt/trtmc/lib \
  --prompt "Reply with the word OK." \
  --max-new-tokens 26 \
  --temperature 0 \
  --top-k 1 \
  --generation-mode linear_spec_lora \
  --block-length 8 \
  --use-chat-template true \
  --enable-thinking true

The initial runtime supports batch-one deterministic greedy generation with linear Psi-Spec and block lengths from 1 through 8. Block 8 is the qualified default. It also supports the pinned single-user, high-reasoning chat template and an autoregressive control mode. String prompts are currently limited to ASCII and fail closed before tokenization otherwise. Stochastic sampling, tree verification, batching, tensor parallelism, quantization, the 0.9B adapter, and long-context qualification remain outside this contract. The committed-token-per-forward receipt is an algorithmic diagnostic, not a wall-clock speedup claim.

Gemma4 Edge ONNX execution​

Standalone Gemma4 text/media offload also maps the published 26B-A4B NVFP4 checkpoint on native SM12x, preserving its FP8 KV cache without calibration. Its build and text/image commands execute, but recorded MC/direct text/image divergence and an unavailable independent FP32 reference prevent an accuracy qualification claim. See the owning Gemma adapter notes for exact boundaries; quality failures do not disable executable requests.

Use trtmc gemma build MODEL -o model.bundle with the owning family's options. trtmc gemma build --help works offline without a checkpoint or GPU imports. This uses the existing family CLI protocol, not an extension to the shared parser.

The Gemma family also owns explicit Gemma4-12B target/assistant MTP and Gemma4-12B/DSpark block7 execution through the optional pinned native Edge-LLM SDK. These are text-only FP16 paired profiles, qualified on SM80; selecting a Gemma4 checkpoint alone does not enable them or claim standalone native support.

Provision the native SDK, then pass --execution-variant mtp or --execution-variant dspark with --companion draft=/path/to/checkpoint to the build CLI. MTP is greedy-only; DSpark preserves the supported sampling controls. Exact checkpoint revisions, capacity bounds, validation results, unsupported controls and source-faithful chat-template handling are documented in the owning Gemma recipe. These local qualifications are separate from the registered manifest inventory and do not imply that CI executes the paired cases.

Qwen3.6 / Qwen3.8 Edge-LLM execution​

The family owns ordinary original-weight FP16 TP2, ordinary mixed-NVFP4 TP1, and explicit DSpark/DFlash2 ONNX execution through the optional Edge-LLM 0.11.0 SDK. Use trtmc qwen3_8 build MODEL -o model.bundle; offline help and options use the existing family CLI protocol, not shared parser extensions. For a pair, add --execution-variant dspark or dflash and --companion draft=/path/to/draft.

An ordinary build (no --execution-variant) of a checkpoint that carries MTP weights builds only the base engine. Add --mtp-seq-len N (N >= 2: one real token plus N-1 drafted tokens verified per round) to also build the embedded MTP engines and add them to the bundle; this option is separate from --execution-variant mtp. Without it the bundle holds no MTP engines and is about half the size, because every engine embeds its own copy of the weights.

The adapter additionally maps uniform NVFP4/INT4-AWQ embedded MTP and INT4-AWQ DFlash2. These have Python/native contract-test coverage, not local model-engine or accuracy qualification. For MTP, use --execution-variant mtp without an external companion. Prepared checkpoint metadata must match the owning recipe; Model Connect does not calibrate or quantize weights during this integration.

Local text requests include exact independent FP32 matches, with separate quality failures retained. Qwen3.8 original single-image passed; ordered-image and video accuracy failed. Mixed-NVFP4 ordinary and both paired raw-text profiles build and run, matching direct Edge exactly, but fail the independent FP32 NED gate. Accuracy failures remain executable and are not described as fully qualified. Original-weight speculative execution and batch-8 MTP remain outside this adapter's mapped scope.

Provision the native SDK. See the owning recipe for exact checkpoints, commands, platform limits, and evidence scope. Local profile results do not imply registered CI or catalog-wide qualification.

Qwen3.6 / Qwen3.8 native fallback​

Without compatible Edge offload, the owner constructs TensorRT graphs directly and runs them through the Model Connect C++ runtime, without native ONNX export. Prepared source quantization and exported FP8 KV scales are preserved. Native paths cover ordinary text/media, embedded MTP, complete DSpark/DFlash2 companions, and original FP16 TP2; they do not silently substitute target-only decoding.

Exact full-checkpoint and small integration results are distinguished in the family recipe. Original Qwen3.6 and mixed-quantized accuracy failures remain runnable with the unchanged reference gates. Prepared MTP, AWQ-DFlash2, and exported FP8 KV have small integration evidence, not full-checkpoint qualification. Test GPU types are evidence scope, not a native family allowlist.

Qwen3.8 native multi-token prefill​

A compatible native Qwen3.8 TP1 text bundle ships a multi-token prefill engine (prefill_engine.plan) next to the one-token decode engine. The runtime feeds the prompt --prefill-chunk-len tokens per call (default 128; the last chunk may be partial) instead of running one decoder step per prompt token. Projections, norms and MLPs run as GEMMs over the chunk, Gated DeltaNet layers use the chunked delta rule, and full-attention layers use one masked attention over the chunk.

Explicit --prefill-chunk-len is native-owned and bypasses Edge offload. Native media, external drafts, TP2, exported FP8 KV, and prepared MTP projections use per-token prefill to retain their exact input/state contract. For these modes, omit the option or use zero; an explicit positive chunk size is rejected. Bundles with an embedded MTP head also ship mtp_prefill_engine.plan, which warms the MTP cache for a whole chunk in one call.

--prefill-chunk-len 0 builds the previous token-by-token prefill. Valid values are 0, a power of two below 64, or a multiple of 64. Each extra engine carries its own copy of the weights, so it adds to the bundle size and build time. Bundles built before this engine existed still load and prefill token by token.

Nemotron-H Edge-LLM execution​

Use trtmc nemotron_h build MODEL -o model.bundle with the owning family's options. trtmc nemotron_h build --help works offline without a checkpoint or GPU imports. This uses the existing family CLI protocol, not an extension to the shared parser.

The Nemotron-H family owns optional pinned Edge-LLM whole-network offload. Ordinary compatible text builds use the experimental builder; the explicit Lightning NVFP4/DFlash pair uses ONNX with --execution-variant dflash --companion draft=/path/to/draft. Native fallback is attempted only during ordinary preparation; it cannot interpret unsupported packed checkpoints or replace a requested pair.

See the owning Nemotron-H recipe for the six recorded ordinary profiles, the qualified greedy DFlash pair, immutable revisions and unchanged quality gates. These are bounded historical local qualifications, not catalog-wide or current-head CI proof. The separate direct-Edge 9B-NVFP4 quality failure and longer-context failures remain open. The original plain 9B case is registered in the owning E2E inventory; the other exact profiles are not implied to run in CI.

Runtime and validation​

The directory name is also the runtime DSO identity: libtrtmc_model_<family>.so. The family factory implements one or more abstract interfaces in trtmc/task.h and owns all preprocessing, postprocessing, dispatch, state, sampling, and engine binding.

Tests, exact checkpoint manifests, fixtures, thresholds, and reference/oracle logic live in the same directory. A family with extra build or reference dependencies declares them in its own plain requirements.txt.

Live inventory​

The website's Models & Recipes pages are generated from families/*/support.py and tests/manifests/*.json. Validate the physical inventory with:

python3 -m tools.model_ci validate

The project covers text decoders and encoders, MoE and recurrent networks, vision-language, speech/audio, diffusion, perception, time-series, robotics, and world-model tasks. Support remains exact and model-owned: a similar architecture, parser option, or sibling family's passing test is not evidence for another checkpoint.

Isolation rule​

Families never import, include, or link siblings. Similar model code is copied by design so a team can implement, validate, change, and revert one family without modifying another. Shared code is limited to model-agnostic contracts and mechanics described in the Architecture.

Optional InternVL Edge execution​

Use trtmc internvl build MODEL -o model.bundle with the owning family's options. trtmc internvl build --help works offline without a checkpoint or GPU imports. This uses the existing family CLI protocol, not an extension to the shared parser.

The InternVL family can offload the recorded original-source InternVL3 1B/2B/8B HF configurations on native x86 SM80 and 14B on native x86 SM120 to the pinned Edge-LLM 0.10.1 SDK. These FP16, batch-one, TP-one profiles have historical local public/direct/HF qualification; this is not catalog-wide or fresh-head CI qualification. The existing 2B/8B image-health test reads actual Edge visual features through a small family-owned test helper without changing its gates. Quantized sources, InternVL3.5, and multi-image public requests are excluded. See the family recipe for exact source revisions, evidence boundaries, and replay requirements.

Optional Qwen3.5 Edge execution​

Use trtmc qwen3_5 build MODEL -o model.bundle with the owning family's options. trtmc qwen3_5 build --help works offline without a checkpoint or GPU imports. This uses the existing family CLI protocol, not an extension to the shared parser.

The Qwen3.5 family maps the recorded original-source dense 0.8B/2B/4B/9B configurations (Instruct and Base), embedded MTP, and explicit 4B/9B DFlash pairs to the pinned Edge-LLM 0.11.0 SDK on native x86 SM80/SM120, FP16 compute, batch one. Dense 27B uses TP2 for its LLM and TP1 for its visual component. The SM120 route preserves admitted ModelOpt weight formats; the recorded 2B FP8-KV and NVFP4 LM-head component flows are checkpoint-driven. Compatible PyPI Python tools are discovered automatically, with the source-built SDK interpreter as fallback; a matching native C++ SDK is still required. Execution admission is not a blanket accuracy claim: recorded quality-only failures remain runnable and existing gates are unchanged. Qwen3 or older, MoE, Qwen3.6/3.8 and speculative TP2 are outside this owner. See the family recipe for commands, paired execution, validation boundaries and known quality issues.

Visual bundles expose the existing typed video and mixed image/video APIs; decoded frame order and optional timestamps are preserved. The recorded 2B two-frame appearance case passed Model Connect/direct parity and independent FP32 NED0.0; it is not a motion-quality or all-configuration qualification. The generic image CLI is unchanged. See the owning recipe for API boundaries.

When Edge offload is unavailable or fails, this family builds with the TensorRT Python API and executes with the Model Connect C++ runtime. Native execution owns the prepared projections, FP8 KV storage, visual processing, TP2 and speculative orchestration; it does not require an Edge installation or ONNX. The existing TP1 image premerge case accepts this native fallback without changing its reference or accuracy checks. Prepared and speculative mappings are not blanket accuracy qualifications; the family recipe separates local results from unexecuted combinations and known quality limitations.

Optional Qwen3.5/3.6 MoE Edge execution​

trtmc qwen3_5_moe build MODEL -o model.bundle owns the GPTQ Qwen3.5 35B-A3B and NVFP4 Qwen3.6 35B-A3B routes. Its family-local options select explicit DFlash companions, greedy DDTree, or the documented embedded NVFP4 MTP recipe. Visual bundles map the existing image and video Task interfaces directly to Edge preprocessing and runtime APIs. Shared model selection remains unchanged.

The native SDK is pinned to official Edge-LLM 0.11.0. Compatible PyPI Python tools are discovered automatically; a source-built installation also works, and matching native headers, libraries and plugins remain required. Builds preserve each source checkpoint's weight quantization recipe, with FP16 compute and single-device execution. The published NVFP4 checkpoint lacks exported K/V scales and therefore uses FP16 KV storage; prepared sources with actual exported scales retain FP8 KV. Larger/original checkpoints outside the measured memory admission are not advertised as qualified configurations.

Runtime availability is not an accuracy claim: speculative decoding and NVFP4 repeatability have recorded quality failures. Those routes remain executable with unchanged validation gates. See the family recipe for exact recipes, commands, Task boundaries and qualification limitations.

When no compatible Edge route exists or Edge preparation fails, the same family owns native TensorRT Python graph construction and Model Connect C++ execution, including prepared routed experts and media inputs. Publication errors, cancellation and runtime errors do not trigger this fallback. Native execution needs neither Edge nor ONNX. The existing validation checks seed rejection on Edge and seeded request-local repeatability on native, retaining capacity recovery, the independent checkpoint reference and the unchanged accuracy gate. Native speculative mappings and incomplete full-model builds are not presented as completed qualification; see the owning recipe for limitations.

Optional Llama Edge execution​

Use trtmc llama build MODEL -o model.bundle with the owning family's options. trtmc llama build --help works offline without a checkpoint or GPU imports. This uses the existing family CLI protocol, not an extension to the shared parser.

The Llama family owns an optional native Edge-LLM 0.10.1 route for original unquantized sources with FP16 compute. Recorded ordinary Llama 3.1 8B and 3.2 1B/3B profiles use SM80; the explicit Llama 3.1 8B + EAGLE3 pair uses SM120. Other requests retain native behavior, and a failed explicit pair never silently becomes base-only decoding. Quantized Llama routes are outside this publication.

These are bounded historical build/inference results, not catalog-wide support or an assertion that every profile is registered in CI. Passing engine payloads were retired; publication checks are reported separately. See the family-owned recipe for exact revisions, capacities, controls and validation boundaries.