nvidia-resiliency-ext v0.2.0

nvidia-resiliency-ext is a set of tools developed by NVIDIA to improve large-scale distributed training resiliency.

Features

• Hang detection and automatic in-job restarting

• In-process restarting

• Async checkpointing

• Local checkpointing

• Straggler (slower ranks) detection

Documentation contents:

• Fault Tolerance
  • Usage guide
    • Terms
    • FT Package Design Overview
    • FT Integration Guide for PyTorch
    • FT Integration Guide for PyTorch Lightning
  • API documentation
    • Config
    • Client
    • Server
    • Callback
  • Examples
    • Basic usage example
    • DDP usage example
• Inprocess Restart
  • Usage Guide
    • Requirements
    • Restrictions
    • Functionality overview
    • Restart latency
    • Known issues
  • API documentation
    • Wrapper
    • Compose
    • Rank Assignment
    • Rank Filter
    • Initialize
    • Abort
    • Finalize
    • Health Check
    • Exceptions
  • Examples
    • Basic usage example
    • Optimal usage example
• Async Checkpointing
  • Usage guide
    • Requirements
    • Synchronization of Asynchronous Checkpoint Requests
  • API documentation
    • Asynchronous Checkpoint Core Utilities
    • Asynchronous PyTorch torch.save with the Core utility
  • Basic usage example
• Local Checkpointing
  • Usage guide
    • Requirements
    • Restrictions
    • Functionality Overview
  • API documentation
    • PTL Callback support
    • BaseCheckpointManager
    • LocalCheckpointManager
    • Replication
    • BaseTensorAwareStateDict
    • BasicTensorAwareStateDict
  • Examples
    • Basic usage example
• Straggler Detection
  • Usage guide
    • Straggler Detection Package Design Overview
    • Straggler Detection Usage Overview
    • Straggler Detection Integration Guide
  • API documentation
    • Straggler
    • Reporting
    • Statistics
    • Callback
  • Examples
    • Basic usage example