Precision debug tools

Precision debug tools

• Getting started
  • Example training script
  • Config file
  • Adjusting Python file
  • Inspecting the logs
  • Logging using TensorBoard
• Config File Structure
  • General Format
  • Layer Specification
  • Names in Transformer Layers
  • Structured Configuration for GEMMs and Tensors
  • Enabling or Disabling Sections and Features
• API
  • Setup
    • initialize()
    • set_tensor_reduction_group()
    • set_weight_tensor_tp_group_reduce()
  • Debug features
    • LogTensorStats
    • LogFp8TensorStats
    • LogNvfp4TensorStats
    • DisableQuantizationGEMM
    • DisableQuantizationLayer
    • PerTensorScaling
    • FakeQuant
    • DisableFP8GEMM
    • DisableFP8Layer
    • DumpTensors
  • Calls to Nvidia-DL-Framework-Inspect
    • modify_tensor()
    • modify_tensor_enabled()
    • fp8_gemm_enabled()
    • inspect_tensor()
    • inspect_tensor_postquantize()
    • inspect_tensor_enabled()
    • inspect_tensor_postquantize_enabled()
• Distributed training
  • Behavior of the features
  • Reduction groups
  • Microbatching
  • Logging to files and TensorBoard
• Adding custom feature to precision debug tools