TensorRT-Model-Connect
TensorRT-Model-Connect turns a Hugging Face or local checkpoint into a
deployable .bundle bundle, then runs that bundle through a native C++ task API.
If you are new to the project, your first goal is deliberately small:
Prepare one supported environment, build Qwen3-0.6B, inspect its bundle, and run one deterministic NLP/text-generation inference.
The Getting Started section is the only required entry path for that goal.
Choose your path
Get started Check prerequisites, prepare the environment, install the project, and complete one build-inspect-run loop.
Find a model Search exact checkpoint IDs, tasks, family ownership, precision, and topology configurations generated from manifests.
Use a feature Look up task commands, build behavior, runtime configuration, quantization, parallelism, validation, and benchmarking.
Follow a course Learn progressively through short modules, hands-on labs, milestones, self-check questions, and answer keys.
Look up a contract Open the CLI, Python, C++, bundle, configuration, testing, or performance reference without reading a tutorial first.
Develop or contribute Understand architecture and choose the owning model, runtime, config, validation, or optimized-provider boundary before editing.
| Goal | Start here |
|---|---|
| Understand the project boundary and intended users | Project Overview |
| First successful NLP/text inference | Getting Started |
| Exact model/checkpoint lookup | Models & Recipes |
| Task and feature lookup | User Guides |
| Beginner-to-advanced curriculum | Tutorials |
| Exact public interfaces and configuration | Reference |
| Architecture, extension, and contribution | Developer Guide |
| Compatibility and lifecycle policy | Release & Support |
| Machine-readable and agent safety guidance | AI & Agent Guide |
The minimum mental model
| Term | Meaning |
|---|---|
| Checkpoint | Model config, weights, tokenizer/processor assets, and related metadata released by a training ecosystem. |
| TensorRT engine | A target-specific compiled execution plan. It is not the original checkpoint. |
.bundle bundle | The artifact boundary between Python-first build logic and native runtime execution. |
| Family | The model-owned Python implementation that recognizes a checkpoint and builds its artifacts. |
| Pipeline | The native task implementation returned to an application after the bundle is loaded. |
Use the Glossary whenever a term is unfamiliar; you do not need to memorize it before starting.
Current declared inventory
The following facts are generated from model-owned metadata in the current checkout:
- 79 Python family plugins under
python/tensorrt_model_connect/families/. - 209 E2E model manifests and 79 family indexes under
tests/e2e/models/. - 80 unique C++ runtime strategy keys declared by model metadata under
src/runtime/models/.
These are discovery counts, not proof that every declared model passed on your hardware. Use Model Support for evidence levels and exact ownership.
What comes after the first run
After Getting Started:
- use Inference Fundamentals to explain what happened;
- use User Guides for the next task you need to do;
- follow the Tutorial Curriculum when you want progressive learning and self-checks; and
- use Developer Guide only when you need source-level ownership or contribution instructions.