nvidia-resiliency-ext

nvidia-resiliency-ext is a set of tools developed by NVIDIA to improve large-scale distributed training resiliency.

Features

• Hang detection and automatic in-job restarting

• In-process restarting

• Async checkpointing

• Local checkpointing

• Straggler (slower ranks) detection

Documentation contents:

• Fault Tolerance
  • Usage guide
    • Terms
    • Design Overview
    • Usage Overview
  • Integration Guides
    • Heartbeats API Integration
    • Sections API Integration
    • PyTorch Lightning Integration
    • FT Launcher & Inprocess integration
  • API documentation
    • Config
    • Client
    • Server
    • Callback
  • Examples
    • Basic usage example
    • Heartbeat API usage example with DDP
    • Section API usage example with DDP
    • FT Launcher & Inprocess integration example
• Inprocess Restart
  • Usage Guide
    • Requirements
    • Restrictions
    • Functionality overview
    • Restart latency
    • Known issues
  • API documentation
    • Wrapper
    • Compose
    • State
    • Rank Assignment
    • Rank Filter
    • Initialize
    • Abort
    • Finalize
    • Health Check
    • Exceptions
  • Examples
    • Basic usage example
    • Optimal usage example
• Async Checkpointing
  • Usage guide
    • Implementation Changes and Evolution
    • Synchronization of Asynchronous Checkpoint Requests
    • Using Multi-Storage Client with Async Checkpointing
    • Best Practices
  • API documentation
    • Asynchronous Checkpoint Core Utilities
    • Asynchronous FileSystemWriter Implementation
    • Asynchronous Pytorch Distributed Checkpoint save with optimized FileSystemWriter
    • Asynchronous PyTorch torch.save with the Core utility
  • Examples
    • Basic usage example
    • FileSystemWriter example
• Local Checkpointing
  • Usage guide
    • Requirements
    • Restrictions
    • Functionality Overview
  • API documentation
    • PTL Callback support
    • BaseCheckpointManager
    • LocalCheckpointManager
    • Replication
    • BaseTensorAwareStateDict
    • BasicTensorAwareStateDict
  • Examples
    • Basic usage example
• Straggler Detection
  • Usage guide
    • Design Overview
    • Usage Overview
    • Integration Guide
  • API documentation
    • Straggler
    • Reporting
    • Statistics
    • Callback
  • Examples
    • Basic usage example