Install ModelOpt-Windows as a Standalone Toolkit

The Model Optimizer - Windows (ModelOpt-Windows) can be installed as a standalone toolkit for quantizing ONNX models. Below are the setup steps:

1. Setup Prerequisites

Before using ModelOpt-Windows, the following components must be installed:

  • NVIDIA GPU and Graphics Driver

  • Python version >= 3.10 and < 3.13

  • Visual Studio 2022 / MSVC / C/C++ Build Tools

  • CUDA Toolkit and matching CuDNN for using CUDA path during calibration (e.g. for calibration of ONNX models using onnxruntime-gpu or CUDA EP)

Update PATH environment variable as needed for above prerequisites.

2. Setup Virtual Environment (Optional but Recommended)

It is recommended to use a virtual environment for managing Python dependencies. Tools such as conda or Python’s built-in venv module can help create and activate a virtual environment. Example steps for using Python’s venv module:

$ mkdir myEnv
$ python -m venv .\myEnv
$ .\myEnv\Scripts\activate

In the newly created virtual environment, none of the required packages (e.g., onnx, onnxruntime, onnxruntime-directml, onnxruntime-gpu, nvidia-modelopt etc.) will be pre-installed.

3. Install ModelOpt-Windows Wheel

To install the ONNX module of ModelOpt-Windows, run the following command:

pip install "nvidia-modelopt[onnx]"

If you install ModelOpt-Windows without the extra [onnx] option, only the minimal core dependencies and the PyTorch module (torch) will be installed. Support for ONNX model quantization requires installing with [onnx].

4. ONNX Model Quantization: Setup ONNX Runtime Execution Provider for Calibration

The Post-Training Quantization (PTQ) process for ONNX models usually involves running the base model with user-supplied inputs, a process called calibration. The user-supplied model inputs are referred to as calibration data. To perform calibration, the base model must be run using a suitable ONNX Execution Provider (EP), such as DmlExecutionProvider (DirectML EP) or CUDAExecutionProvider (CUDA EP). There are different ONNX Runtime packages for each EP:

  • onnxruntime-directml provides the DirectML EP.

  • onnxruntime-trt-rtx provides TensorRT-RTX EP.

  • onnxruntime-gpu provides the CUDA EP.

  • onnxruntime provides the CPU EP.

By default, ModelOpt-Windows installs onnxruntime-gpu. The default CUDA version needed for onnxruntime-gpu since v1.19.0 is 12.x. The onnxruntime-gpu package (i.e. CUDA EP) has CUDA and cuDNN dependencies:

  • Install CUDA and cuDNN:

    • For the ONNX Runtime GPU package, you need to install the appropriate version of CUDA and cuDNN. Refer to the CUDA Execution Provider requirements for compatible versions of CUDA and cuDNN.

If you need to use any other EP for calibration, you can uninstall the existing onnxruntime-gpu package and install the corresponding package. For example, to use the DirectML EP, you can uninstall the existing onnxruntime-gpu package and install the onnxruntime-directml package:

pip uninstall onnxruntime-gpu
pip install onnxruntime-directml

5. Setup GPU Acceleration Tool for Quantization

ModelOpt uses CuPy to accelerate INT4 ONNX quantization. The nvidia-modelopt[onnx] extra installs cupy-cuda12x and a CUDA 12-compatible onnxruntime-gpu version by default.

CUDA 13.x Setup

The steps below assume a CUDA 13.x Toolkit, compatible cudnn, and a compatible driver are already installed on the host.

Replace the CUDA-dependent packages installed by the default ONNX extra:

python -m pip uninstall -y cupy-cuda12x onnxruntime-gpu
python -m pip install cupy-cuda13x "onnxruntime-gpu>=1.27"

ONNX Runtime 1.27 and later GPU packages published on PyPI use CUDA 13.x by default. Refer to the ONNX Runtime CUDA Execution Provider requirements before selecting or pinning an ONNX Runtime version.

Note

Make sure a cuDNN 9 build for CUDA 13 is available on the host (for example, via the NVIDIA Windows installer/zip package, or the nvidia-cudnn-cu13 Python wheel), and that the relevant environment variables like CUDA_PATH, CUDA_HOME, and PATH point to the CUDA 13.x installation).

6. Verify Installation

Ensure the following steps are verified:

  • Task Manager: Check that the GPU appears in the Task Manager, indicating that the graphics driver is installed and functioning.

  • Python Interpreter: Open the command line and type python. The Python interpreter should start, displaying the Python version.

  • Onnxruntime Package: Ensure that exactly one of the following is installed:

    • onnxruntime-directml (DirectML EP)

    • onnxruntime-trt-rtx (TensorRT-RTX EP)

    • onnxruntime-gpu (CUDA EP)

    • onnxruntime (CPU EP)

  • CUDA Toolkit: For CUDA workflows, verify that the selected Toolkit is found first and that nvcc reports the expected major version:

    where nvcc
nvcc --version

  • ONNX and ONNX Runtime: Ensure that imports succeed and that CUDA EP is available for CUDA workflows:

    python -c "import onnx; import onnxruntime as ort; print(ort.__version__, ort.get_available_providers())"

  • CuPy: Verify that CuPy can allocate and execute on the GPU. For CUDA 13.x, runtimeGetVersion() should report a value beginning with 13:

    python -c "import cupy; print(cupy.__version__, cupy.cuda.runtime.runtimeGetVersion(), cupy.arange(3).sum())"

  • Environment Variables: For workflows using CUDA dependencies (e.g., CUDA EP-based calibration), ensure environment variables such as CUDA_PATH, CUDA_PATH_V12_x, or CUDA_PATH_V13_x point to the intended Toolkit. Reopen the command prompt after changing persistent environment variables.

  • ModelOpt-Windows Import Check: Run the following command to ensure the installation is successful:

    python -c "import modelopt.onnx.quantization"

  • If you encounter any difficulties during the installation process, please refer ModelOpt-Windows FAQs for potential solutions and additional guidance.