Runtime Optimization

Runtime optimization is a set of methods that make inference faster, use less memory, or both. Each page below is one of those methods. They apply at different points: some change the checkpoint, and some change how the model is run.

Quantization stores weights, and sometimes activations, in fewer bits. That uses less GPU memory and can make inference faster.

• Quantization