nvidia-resiliency-ext

nvidia-resiliency-ext is a set of tools developed by NVIDIA to improve large-scale distributed training resiliency.

Features

• InJob restart with section-based monitoring

• Async checkpointing

• Local checkpointing

• Straggler (slower ranks) detection

• Failure attribution

• Shared utilities and distributed logging

Documentation contents:

• Release Notes
  • Unreleased
    • Removals
  • NVIDIA Resiliency Extension v0.7.0
    • Highlights
    • Deprecations & Removals
    • Installation
    • Known Issues & Limitations
  • NVIDIA Resiliency Extension v0.6.0
    • Highlights
    • Deprecations & Removals
    • Installation
    • Known Issues & Limitations
  • NVIDIA Resiliency Extension v0.5.0
    • Highlights
    • Known Issues & Limitations
  • NVIDIA Resiliency Extension v0.4.1
    • Highlights
    • Known Issues & Limitations
  • NVIDIA Resiliency Extension v0.4.0
    • Highlights
    • Known Issues & Limitations
  • NVIDIA Resiliency Extension v0.3.0
    • Highlights
    • Known Issues & Limitations
  • NVIDIA Resiliency Extension v0.2.1
    • Highlights
  • NVIDIA Resiliency Extension v0.2.0
    • Highlights
    • Known Issues & Limitations
  • NVIDIA Resiliency Extension v0.1.3
    • Highlights
• InJob restart
  • InJob restart usage guide
    • Recovery responsibilities
    • Processes and terminology
    • Launch and monitor
    • Recovery limits
    • Where to go next
  • Integration guides
    • Megatron-LM integration
    • Sections API integration
    • Segment Health Check
    • Legacy heartbeats API integration
  • Examples
    • GPT checkpoint recovery with Megatron-LM
    • Section monitoring and checkpoint recovery with DDP
    • Singleton job-array deployment (SLURM)
    • Legacy heartbeat monitoring example
    • Legacy heartbeat DDP training example
  • Configuration and timeouts
    • Defaults and precedence
    • Optional YAML configuration
    • Warmup and unmonitored intervals
    • Experimental calibration and legacy heartbeats
  • Multinode deployment
    • Launch shape and fixed training world
    • Replacement groups and rank assignment
    • Hot spares and NVLink segments
    • Scheduler continuation and external control
    • NUMA binding
  • Operations and troubleshooting
    • Cycle records and checkpoint progress
    • No restart: telling “do not requeue” apart from a failure
    • GPU Memory Reclaim
    • Per-cycle logging and gRPC log aggregation
    • Workload control and diagnosis
  • Optional health checks and attribution
    • Node health check service
    • Distributed storage health check (Lustre + NFS)
    • Storage path health check
    • Segment Health Check
    • Attribution service integration
  • Timeout diagnostics
    • Enable capture
    • CUDA-GDB launch mode
    • Debugger command files
    • Capture and upload
    • Failures and retention
    • Configuration
  • API documentation
    • Config
    • Client
    • Server
• Async Checkpointing
  • Usage guide
    • Implementation Changes and Evolution
    • Synchronization of Asynchronous Checkpoint Requests
    • Using Multi-Storage Client with Async Checkpointing
    • Best Practices
  • API documentation
    • Asynchronous Checkpoint Core Utilities
    • Asynchronous FileSystemWriter Implementation
    • Asynchronous Pytorch Distributed Checkpoint save with optimized FileSystemWriter
    • Asynchronous PyTorch torch.save with the Core utility
  • Examples
    • Basic usage example
    • FileSystemWriter example
• Local Checkpointing
  • Usage guide
    • Requirements
    • Restrictions
    • Functionality Overview
  • API documentation
    • BaseCheckpointManager
    • LocalCheckpointManager
    • Replication
    • BaseTensorAwareStateDict
    • BasicTensorAwareStateDict
  • Examples
    • Basic usage example
• Straggler Detection
  • Usage guide
    • Design Overview
    • Usage Overview
    • Integration Guide
  • API documentation
    • Straggler
    • Reporting
    • Statistics
  • Examples
    • Basic usage example
• Failure Attribution
  • Attribution Feature Requirements
    • Progressive FT Launcher Attribution
    • Restart Agent
    • Attribution Feature Requirements Template
• Shared Utilities
  • NVRx Logger Guide
    • Key Features
    • Architecture
    • Configuration
    • Basic Usage
    • Node Local Temporary Logging Setup
    • Advanced Configuration
    • Example Output Format
    • Integration with Other NVRx Components
    • Best Practices
    • Troubleshooting
  • Configuration Reference
    • Environment Variables
    • Python API Parameters
    • LogManager Constructor Parameters
    • Configuration Examples
    • SLURM Integration
    • Docker Integration
    • Kubernetes Integration
    • Configuration Precedence
    • Best Practices
    • Filesystem Selection
    • Troubleshooting
    • Quick API Reference
  • API Reference
    • Log Manager
    • Log Aggregator
    • Log Configuration
    • Core Classes
    • Core Functions
    • Telemetry Constants
    • Quick Reference