Model Optimizer

/

Announcements

Announcements

Getting Started

Overview
Installation
- Installation for Linux
- Installation for Windows
  - Install ModelOpt-Windows as a Standalone Toolkit
  - Install ModelOpt-Windows with Olive
Quick Start: PTQ - PyTorch
Quick Start: PTQ - ONNX
Quick Start: PTQ - PyTorch to ONNX
Quick Start: PTQ - Windows
Quick Start: QAT
Quick Start: Pruning
Quick Start: Distillation
Quick Start: Speculative Decoding
Quick Start: Sparsity

Guides

Support Matrix
Recipes
ModelOpt Config System
Quantization
Saving & Restoring
Pruning
Distillation
Speculative Decoding
Sparsity
NAS
AutoCast (ONNX)
Autotune (ONNX)

Deployment

TensorRT-LLM
Onnxruntime
Unified HuggingFace Checkpoint

Examples

All GitHub Examples

Reference

Changelog
modelopt API
- deploy
  - llm
    - generate
- onnx
- torch
  - distill
  - export
  - fastgen
    - config
    - discriminators
    - ema
    - factory
    - flow_matching
    - loader
    - losses
    - methods
    - pipeline
    - plugins
    - utils
  - kernels
  - nas
    - algorithms
    - autonas
    - conversion
    - hparams
    - modules
    - patch
    - plugins
    - registry
    - search_space
    - traced_hp
    - utils
  - opt
    - config
    - config_loader
    - conversion
    - dynamic
    - hparam
    - mode
    - plugins
    - searcher
    - utils
  - peft
    - config
    - conversion
    - convert
    - custom
    - lora
    - mode
  - prune
  - puzzletron
  - quantization
    - algorithms
    - backends
    - calib
    - modelopt.torch.quantization.compress
    - config
    - conversion
    - export_onnx
    - extensions
    - mode
    - model_calib
    - model_quant
    - nn
    - plugins
    - qtensor
    - tensor_quant
    - utils
  - sparsity
    - attention_sparsity
    - weight_sparsity
  - speculative
    - config
    - dflash
    - eagle
    - medusa
    - mode
    - plugins
    - speculative_decoding
    - utils
  - trace
    - analyzer
    - modules
    - symbols
    - tracer
  - utils
    - cpp_extension
    - dataset_utils
    - distributed
    - graph
    - import_utils
    - list
    - logging
    - loss_mask
    - memory_monitor
    - mlflow
    - nemotron_vlm_dataset_utils
    - network
    - perf
    - plugins
    - random
    - regex
    - robust_json
    - serialization
    - speech_dataset_utils
    - tensor
    - vlm_dataset_utils

Support

Contact us
FAQs

Model Optimizer /
modelopt API /
deploy /
llm

llm#

Modules

modelopt.deploy.llm.generate

A wrapper over the TensorRT-LLM high level API runner.

LLM deployment utils with tensorrt_llm.

2023-2025, NVIDIA Corporation

Made with Sphinx and Shibuya theme.