Quantization

Quantization stores model values in fewer bits at lower precision. The weights are the usual target, and activations can be quantized as well. Fewer bits use less memory and can make each inference step faster. Quantization can be applied while a model is training, after a checkpoint is finished, or as a request is served. This page covers the post-training recipes in the playbooks. The recipes write a lower-precision checkpoint, so the model can fit on a smaller GPU, and memory no longer spent on weights can hold a larger batch for inference. A larger batch raises throughput when a server handles many requests at once.

Post-training quantization

Post-training quantization (PTQ) runs after training. A calibration pass runs the model forward on a small set of samples, measures the range of those values, and stores the resulting scales in the exported checkpoint. The weights are not updated.

Quantization-aware training (QAT) happens while the model is still training or being fine-tuned. The training loop simulates low precision so the weights can adapt, which takes a training dataset and a full training run. Dynamic quantization waits until inference and computes some scales from each activation as it arrives, so it has no separate calibration step. The recipes here are static PTQ: calibration chooses the scales before the checkpoint is served.

Nemotron-3-Nano-Omni

The playbooks quantize Nemotron-3-Nano-Omni with NVIDIA Model Optimizer. Model Optimizer is a library for compressing a trained model, including quantization, distillation, pruning, neural architecture search, and speculative decoding, for inference frameworks such as vLLM and TensorRT. These recipes use its post-training quantization path and export Hugging Face FP8 or NVFP4 checkpoints.

Code and the step-by-step runbook live in the Sports Intelligence Playbooks repository under avlm/quantization/ptq/nemotron_3_nano_omni/modelopt/. The runbook is modelopt/README.md.

Run

Run from the repo root inside the NeMo container nvcr.io/nvidia/nemo:26.06.00. On a workstation, pull the image and open a shell with the repo mounted at /workspace. On Slurm, import that same image to nemo_26_06_00.sqsh and start slurm/launch_interactive_session.sh. The runbook has both command sequences and the recipe fields.

Inside the container, log in so the checkpoint and the public calibration set can download, then quantize:

hf auth login
bash avlm/quantization/ptq/nemotron_3_nano_omni/modelopt/quantize.sh

NVFP4 for the same recipe:

bash avlm/quantization/ptq/nemotron_3_nano_omni/modelopt/quantize.sh hf QUANT_FORMAT=nvfp4

The public reasoning checkpoint:

QUANTIZATION_PARAMS=avlm/quantization/ptq/nemotron_3_nano_omni/modelopt/configs/quantization_hf.yaml \
  bash avlm/quantization/ptq/nemotron_3_nano_omni/modelopt/quantize.sh

Serve the exported directory with vLLM. Server examples are under avlm/deployment/vLLM/. See vLLM serving.