Skip to main content

Build from Source

Use this path on Linux x86_64 or aarch64 for the first Qwen inference from source. Start at the repository root.

Automated environment preparation​

The repository-local apps/devtoolkit Python API selects Dockerfile.dev.x86 or Dockerfile.dev.aarch64 from the host architecture, runs the direct development-image build, starts a persistent container for the current checkout, and optionally installs one family's declared dependencies:

from pathlib import Path
import subprocess
import sys

repo = Path.cwd()
sys.path.insert(0, str(repo / "apps" / "devtoolkit"))

from trtmc_devtoolkit import DevToolkit, DockerTargetPolicy

gpu = "0"
sm = subprocess.run(
    [
        "nvidia-smi",
        "-i",
        gpu,
        "--query-gpu=compute_cap",
        "--format=csv,noheader,nounits",
    ],
    check=True,
    capture_output=True,
    text=True,
).stdout.strip().replace(".", "")

toolkit = DevToolkit.from_checkout(repo)
environment = toolkit.prepare_docker(
    family="qwen",
    gpu=gpu,
    environment={"TRTMC_SM": sm},
    policy=DockerTargetPolicy.ENSURE,
)
print(" ".join(environment.command("bash")))

Runnable end-to-end examples are available for both supported execution paths:

# Build in a checkout-owned development container.
python3 apps/devtoolkit/examples/docker_build.py --gpu 0

# Build directly on a prepared host interpreter/toolchain.
python3 apps/devtoolkit/examples/local_build.py \
  --python /path/to/python3.12 \
  --tensorrt 11.0.0.114

The local path expects the generic native build prerequisites documented in apps/devtoolkit/README.md; use its --cmake-python and --cmake-prefix-path options when those dependencies live in isolated prefixes.

The toolkit reuses a container only when its checkout-owned configuration still matches. A foreign name collision or configuration drift fails without removing or replacing the container. Unknown host architectures fail before Docker is invoked. This preparation call remains separate from the toolkit's optional resolve, provision, build, and run capabilities; use environment.execution_target() to pass the prepared container into that evidence-producing path. Each development Dockerfile's first FROM is its base-image pin; repository CI continues to use the root Dockerfile. Optional Python dependencies remain in families/<family>/requirements.txt. See apps/devtoolkit/README.md for lifecycle policies, immutable environment identity and receipts, managed toolchain catalogs, and the explicit existing-interpreter local path.

YOLOv5, YOLOv8, YOLOv10, and YOLO11 builds do not require the ultralytics package. Supply your own appropriately licensed checkpoint; Model Connect does not include model weights. The .pt readers require PyTorch and accept only trusted checkpoints, because the archive format uses pickle.

Their accuracy references live under each family's tests/reference/ and use ordinary PyTorch layers with torchvision suppression where applicable. Neither builds nor tests install or import the Ultralytics package. E2E references run with the normal test interpreter; standalone qualification declares its optional dependencies in tests/requirements.txt. The reference implementations and their dependency declarations are excluded from the wheel. Checkpoint identifiers and serialized class names still identify the upstream model format. The references use the same checkpoints and preserve the existing accuracy thresholds. Selected E2E tests also check both the original and mirrored image against numeric snapshots recorded before the upstream package was removed. The snapshots pin checkpoint and image hashes; they contain predictions, not model weights.

The manual commands below remain the direct source-build path and show the operations performed by development mode.

Optional Edge-LLM Python tools​

After preparing the source-build prerequisites below, use CPython 3.12 to install this checkout with the pinned Edge-LLM builder and Python runtime from PyPI through the normal package extra:

python -m pip install -e '.[edgellm]'
python -c 'import tensorrt; from tensorrt_edgellm import runtime; runtime.load()'

This editable source installation still builds Model Connect's native targets; the extra does not provide a Python-only build mode.

The Edge wheel does not install the platform CUDA or TensorRT libraries. Its native payload must match the executing platform, GPU, and CUDA/TensorRT stack; see the Edge-LLM 0.11.0 installation guide. A successful package installation alone is not a model qualification result.

This extra provides Python tools, not the C++ SDK linked by the existing native family adapters. Those adapters still use the separately pinned optional CMake SDK dependency. Installing this extra does not switch a family's builder or runtime.

The base wheel includes the direct builder. Use a separate environment for Edge-LLM's optional export or tools extras: their Transformers pin differs from Model Connect's pin. Do not override either project's requirements to force the environments together.

1. Select the GPU and start the container​

Change only GPU. The commands derive the SM used by CMake and select the matching development Dockerfile. Repository CI continues to use Dockerfile.

GPU=0
SM="$(
  nvidia-smi -i "$GPU" \
    --query-gpu=compute_cap \
    --format=csv,noheader,nounits |
  tr -d '.[:space:]'
)"
IMAGE="trtmc-quickstart"

case "$(uname -m)" in
  x86_64) DOCKERFILE=Dockerfile.dev.x86 ;;
  aarch64) DOCKERFILE=Dockerfile.dev.aarch64 ;;
  *) echo "Unsupported host architecture: $(uname -m)" >&2; exit 1 ;;
esac

docker build \
  -f "$DOCKERFILE" \
  -t "$IMAGE" requirements

SOURCE_DIR="$(git rev-parse --show-toplevel)"

docker run --rm -it \
  --gpus "device=${GPU}" \
  --ipc=host \
  --mount "type=bind,source=${SOURCE_DIR},target=/src" \
  --workdir /src \
  --env TRTMC_SM="$SM" \
  "$IMAGE" \
  bash

Run the remaining commands inside the container.

2. Build the native runtime​

python -m pip install --no-deps -e . -C py-only=true

TRTMC_BUILD_DIR="build-sm${TRTMC_SM}"

cmake -S . -B "$TRTMC_BUILD_DIR" -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES="${TRTMC_SM}-real" \
  -DTRTMC_BUILD_BACKEND_RTX=OFF \
  -DTRTMC_BUILD_TESTS=OFF \
  -DTRTMC_BUILD_EXAMPLES=OFF

cmake --build "$TRTMC_BUILD_DIR" --parallel "$(nproc)" --target \
  trtmc \
  trtmc-server \
  trtmc_backend_trt \
  trtmc_model_qwen

export PATH="$PWD/$TRTMC_BUILD_DIR:$PATH"

To run the optional text server from this build, install its Python control plane dependencies with python -m pip install -e '.[serve]' -C py-only=true.

TensorRT-RTX is an explicit optional build. When its SDK is installed, enable only its backend DSO with the exact include and library directories:

cmake -S . -B "$TRTMC_BUILD_DIR" \
  -DTRTMC_BUILD_BACKEND_RTX=ON \
  -DTRTMC_RTX_INCLUDE_DIR=/absolute/tensorrt-rtx/include \
  -DTRTMC_RTX_LIBRARY_DIR=/absolute/tensorrt-rtx/lib
cmake --build "$TRTMC_BUILD_DIR" --target trtmc_backend_rtx

This path skips CI-only Python profiles and unrelated model DSOs. Continue to Quick Start in the same container shell. Full-repository ownership and backend boundaries are documented in the AI-Native Horizontal Scaling Architecture.