nvidia-resiliency-ext

nvidia-resiliency-ext is a set of tools developed by NVIDIA to improve large-scale distributed training resiliency.

Features

• Hang detection and automatic in-job restarting

• In-process restarting

• Async checkpointing

• Local checkpointing

• Straggler (slower ranks) detection

• Failure attribution

• Shared utilities and distributed logging

Documentation contents:

• Fault Tolerance
  • Usage guide
    • Terms
    • Design Overview
    • Usage Overview
  • Integration Guides
    • Heartbeats API Integration
    • Sections API Integration
    • PyTorch Lightning Integration
    • FT Launcher & Inprocess integration
  • API documentation
    • Config
    • Client
    • Server
    • Callback
  • Examples
    • Basic usage example
    • Heartbeat API usage example with DDP
    • Section API usage example with DDP
• Inprocess Restart
  • Usage Guide
    • Requirements
    • Running with SLURM
    • Restrictions
    • Functionality overview
    • Restart latency
    • Known issues
  • API documentation
    • Wrapper
    • Compose
    • State
    • Rank Assignment
    • Rank Filter
    • Initialize
    • Abort
    • Finalize
    • Health Check
    • Exceptions
  • Examples
    • Basic usage example
• Async Checkpointing
  • Usage guide
    • Implementation Changes and Evolution
    • Synchronization of Asynchronous Checkpoint Requests
    • Using Multi-Storage Client with Async Checkpointing
    • Best Practices
  • API documentation
    • Asynchronous Checkpoint Core Utilities
    • Asynchronous FileSystemWriter Implementation
    • Asynchronous Pytorch Distributed Checkpoint save with optimized FileSystemWriter
    • Asynchronous PyTorch torch.save with the Core utility
  • Examples
    • Basic usage example
    • FileSystemWriter example
• Local Checkpointing
  • Usage guide
    • Requirements
    • Restrictions
    • Functionality Overview
  • API documentation
    • PTL Callback support
    • BaseCheckpointManager
    • LocalCheckpointManager
    • Replication
    • BaseTensorAwareStateDict
    • BasicTensorAwareStateDict
  • Examples
    • Basic usage example
• Straggler Detection
  • Usage guide
    • Design Overview
    • Usage Overview
    • Integration Guide
  • API documentation
    • Straggler
    • Reporting
    • Statistics
    • Callback
  • Examples
    • Basic usage example
• Failure Attribution
  • Attribution Feature Requirements
    • Progressive FT Launcher Attribution
    • Attribution Feature Requirements Template
• Shared Utilities
  • NVRx Logger Guide
    • Key Features
    • Architecture
    • Configuration
    • Basic Usage
    • Node Local Temporary Logging Setup
    • Advanced Configuration
    • Example Output Format
    • Integration with Other NVRx Components
    • Best Practices
    • Troubleshooting
  • Configuration Reference
    • Environment Variables
    • Python API Parameters
    • LogManager Constructor Parameters
    • Configuration Examples
    • SLURM Integration
    • Docker Integration
    • Kubernetes Integration
    • Configuration Precedence
    • Best Practices
    • Filesystem Selection
    • Troubleshooting
    • Quick API Reference
  • API Reference
    • Log Manager
    • Log Aggregator
    • Log Configuration
    • Core Classes
    • Core Functions
    • Quick Reference