Examples

Choose an example by the recovery behavior you want to exercise. Monitoring and checkpoint storage are separate choices: the small DDP example keeps synchronous PyTorch saves, while GPT uses NVRx async saves through Megatron.

Recommended examples

• GPT checkpoint recovery with Megatron-LM
  • Requirements and validation status
  • Single-node run
  • Fault injection and repeated runs
  • Verify recovery
  • Multinode TP/PP example
  • Source
• Section monitoring and checkpoint recovery with DDP
  • What is monitored
  • Checkpoint and data progress
  • Exercise a failure
  • Optional configuration and experiments
• Singleton job-array deployment (SLURM)
  • Quick start

Legacy heartbeat examples

• Legacy heartbeat monitoring example
• Legacy heartbeat DDP training example