Release Notes
NVIDIA Resiliency Extension is a Python package for framework developers and users to implement fault-tolerant features. It improves effective training time by minimizing downtime due to failures and interruptions.
Unreleased
InJob control-host processes
On the control host,
ft_launchernow supervises a separate job-controller, node-agent, and enabled service processes. Existing colocated launch commands remain unchanged. Control-process logs use the exact--ft-nvrx-logfilepath; node-agent and rank-monitor logs from every node use a derived*_agent.logthrough the existing gRPC aggregation path. Update tools that read agent logs from the configured path to use the derived agent file. See the deployment guide for ownership, shutdown, and accepted exit-handling differences.
Removals
Replaced the experimental
nvrx-controlcommand withft_launcher --ft-control-onlyfor external CPU control hosts. When cycle-info output is enabled, pass the compute job’s ID with--ft-cycle-info-job-id.Removed
--ft-log-server-log-prefixand its underscore alias. The control supervisor captures gRPC root/leaf process logs directly at the configured--ft-nvrx-logfilepath or original console; separate aggregator process log files are no longer supported.--standaloneuses local logging without gRPC aggregation, including when--ft-enable-log-serveris set.
NVIDIA Resiliency Extension v0.7.1 (Unreleased)
These notes describe the changes selected for the 0.7.1 release candidate relative to 0.7.0. Final release validation is ongoing.
Highlights
Checkpoint metadata and async-save finalization
Adds faster PyTorch distributed-checkpoint
.metadataserialization, including an optional native C++ writer with a Python fallback. Cached global metadata is represented as compact tables, and finalization gathers compact write-result tables instead of large Python object lists (#438, #446, #454).Reuses compatible metadata from a loaded checkpoint on the first save and reuses plans and metadata on subsequent saves when caching is enabled. Cached metadata no longer retains per-save
storage_data(#438, #445).
Timeout diagnostics
Adds opt-in Python, native CPU, and CUDA diagnostics through
py-spy, GDB, and CUDA-GDB when a selected workload section times out. Captures are staged locally and uploaded to the job’s diagnostic directory; tools and capture budgets are configurable (#451, #463, #464, #465).Supports CUDA-GDB command delivery through stdin as well as batch mode, and fixes diagnostics configuration and integration regressions (#459, #474). See the timeout dumps guide for prerequisites and limitations.
InJob restart and health checks
Node-health decisions use actionable
failed_checks, rather than the rawfail_count, so checks ignored by site policy do not exclude nodes (#432).The segment-health watcher supports a persistent serve loop and SIGUSR1-triggered immediate polling (#424).
Missing checkpoint progress markers now count as no progress instead of disabling the no-progress guard (#468).
Launcher errors remain visible when log forwarding is unavailable; TCPStore server startup errors are preserved instead of silently falling back to client mode; and shutdown signals propagate through optional health-check and attribution calls (#470, #471, #472).
Attribution and observability (experimental)
Adds category-driven STOP context and corrects CPU-OOM policy, unknown-progress accounting, cancellation classification, checkpoint-OOM classification, and transient JIT-cache attribution (#411, #430, #431, #439, #456, #477).
Improves external Attribution Service lifecycle handling, provides an attrsvc container build without workload dependencies, and adds Slack alerts and persisted attribution results (#441, #452, #449).
Adds optional NeMo Lens/OpenTelemetry instrumentation for launcher and async-checkpoint operations, including checkpoint-stage and finalization spans to distinguish snapshot, persistence, synchronization, and metadata costs. NVRx-owned processes read
NVRX_OTEL_settings with fallback toNEMO_LENS_; core training remains usable without telemetry dependencies (#406, #435, #437, #440).
Documentation and integration
Requirements & Integration Changes
Python >=3.11 and PyTorch >=2.9.0 are required. Python 3.10 is no longer supported. The PyTorch floor reflects the distributed-checkpoint storage interfaces used by the metadata writer and cached reader (#492).
The async-save planning API returns
PendingSave. Pass the value returned bysave_state_dict_async_plan(...)directly tosave_state_dict_async_finalize(pending). Tuple unpacking and the legacy three-argument finalize form remain supported but are deprecated.CheckpointMetadataCache.set_cache_metadata,prepare_save_state_dict_ret, andget_cache_metadata, along with the old public cache fields, have been removed; integrations using them must migrate.set_cached_global_metadata(...)remains the entry point for metadata from a loaded checkpoint (#438, #454).Metadata caching requires a stable checkpoint structure. With
enable_cache=True, later saves check local plans and raise if the structure changes; the coordinator must stay the same while reusing its cached metadata. Disable caching when those assumptions do not hold.Telemetry uses NeMo Lens. Install
nemo-lens[sdk]alongside NVRx to enable telemetry, as shown below. This release was tested with Lens at thev0.3.0Git tag (commit7bcc03ce3565d2be6858b819b049bcc4c3ec05b4). Core installation does not require Lens.
Deprecations & Removals
In-process restart has been removed, including the
nvidia_resiliency_ext.inprocesspackage, its documentation, examples, and tests (#453). Use InJob restart throughft_launcher; see the InJob restart usage guide.Removed the remaining deprecated
nvidia_resiliency_ext.ptl_resiliencysource package, Lightning callback documentation, and its unit/NeMo functional tests and Dockerfile. Integrate the underlying fault-tolerance, checkpointing, and straggler APIs directly; use the Megatron GPT walkthrough for InJob training. Documentation builds no longer require Lightning (#455).YAML configuration for fault tolerance has been removed.
ft_launcherandnvrx-controlno longer accept--ft-cfg-path, andFaultToleranceConfig.from_yaml_file/to_yaml_fileare gone, along with thepyyamldependency.ft_launcheralso no longer reads--ft-cfg-path’s environment form,PET_FT_CFG_PATH: a job that sets only that variable now runs with default settings, without an error. Pass the equivalent--ft-*options instead, or build the config in code withFaultToleranceConfig(...)orFaultToleranceConfig.from_kwargs(...). Workloads set warmup iterations throughRankMonitorClient.init_workload_monitoring(num_warmup_iters=...). The--ft-ignore-missing-cfgflag, which has had no effect since v0.4, is removed as well; drop it from launch scripts (#462).
Installation
After the final 0.7.1 release is published:
# Core resiliency and attribution service
pip install 'nvidia-resiliency-ext==0.7.1'
# Optional MCP and Slack attribution integrations
pip install 'nvidia-resiliency-ext[attribution]==0.7.1'
# Optional telemetry, once NeMo Lens 0.3.0 is published to PyPI
pip install 'nvidia-resiliency-ext==0.7.1' 'nemo-lens[sdk]==0.3.0'
# Optional telemetry, from the Lens v0.3.0 Git tag
pip install 'nvidia-resiliency-ext==0.7.1' \
'nemo-lens[sdk] @ git+https://github.com/NVIDIA-NeMo/Lens.git@v0.3.0'
These installation requirements are specific to the 0.7.1 release branch.
Known Issues & Limitations
CUDA-GDB may need more than the default 30-second capture budget. QA measurements on aarch64 GB300 with Megatron-LM 26.10 and Transformer Engine 2.21 showed approximately 41 seconds for library symbol loading alone. Start with
--ft-timeout-dump-capture-timeout=60and validate the workload, or omit CUDA-GDB. A larger budget can delay recovery and does not prevent interruption by launcher teardown; see the configuration guidance in the timeout dumps guide.Timeout capture and worker teardown are not coordinated. A launcher can terminate a worker while its rank monitor is collecting diagnostics, resulting in incomplete or incorrect captures. Debugger attachment and worker teardown can also stall until the capture deadline. GDB and CUDA-GDB are affected most often (#461); see the Known Issues section of the timeout dumps guide.
Timeout captures target the training worker, not its separate DataLoader or async-checkpoint worker processes. Tools, ptrace permissions, and debugger command files must be configured inside the deployed training environment.
Attribution and Restart Agent features remain experimental. Enabling diagnostics or telemetry does not establish the root cause of a failure or replace workload-specific recovery validation.
The optional native metadata writer requires a C++20 compiler when building from source; checkpointing retains a Python fallback if that extension cannot be built.
Per-cycle log aggregation remains best-effort. Retain launcher and rank-monitor evidence when investigating failure-adjacent gaps.
Deprecations
FileSystemWriterAsync(is_multiproc_io=True)is deprecated and ignored. Checkpoint files are always written by threads in the async worker, and the writer no longer forks child processes. PassingTrueemits aDeprecationWarning; the argument will be removed in a future release.
NVIDIA Resiliency Extension v0.7.0
Highlights
InJob restart and resilient deployment
Barrier rendezvous is now the only FT rendezvous implementation (#379). The compatibility spelling
--ft-rdzv-impl barrierremains accepted;legacyis rejected.Hot-spare restart rounds prefer returning active nodes over early standbys and exclude every participant from an unhealthy Slurm array task (#325, #326). Recovery is also hardened against transient over-subscription and stale participants from failed replacement groups (#357, #391).
Scheduler segment-health integration lets each allocation unit publish and consume shared exclusion decisions before rendezvous, so unhealthy Slurm array tasks are omitted when eligible spares are available (#398, #399). See the Segment Health Check guide.
A new singleton job-array deployment example combines InJob hot spares with scheduler-managed replacement of an unrecoverable rendezvous-host allocation (#383). See the deployment guide.
Successful node-health responses with unusable output now fail open; only an explicit service failure or a non-empty
failed_checkslist marks the node unhealthy. The rawfail_countmay include checks ignored by site policy (#420).Local launcher failures no longer permanently close the shared rendezvous. Terminal worker-group failures and genuine job-wide shutdown decisions continue to close it (#350, #421).
Progress-tracker values supplied through YAML are preserved when the corresponding CLI options are omitted (#329).
Attribution and restart decisions (experimental)
ft_launchercan run or contact the Attribution Service and submit per-cycle logs for restart analysis (#338).Attribution analysis is asynchronous and does not delay restart. Acting on a
STOPverdict is opt-in through--ft-attribution-stop-action no-restart; the defaultlogmode records verdicts without terminating the job (#378).The new Restart Agent provides deterministic failure evidence and retry policy, with optional model-assisted enrichment, bounded history, operational logging, and auditable result artifacts (#370, #400).
The core wheel includes
attrsvc, Flight Recorder analysis, and the direct Restart Agent backend. MCP and Slack integrations requirenvidia-resiliency-ext[attribution]; legacy LogSage remains source-checkout-only and is excluded from wheels (#404, #410).
Health checks
NVLinkWindowHealthCheckdetects persistent NVLink replay and recovery counter growth across a configurable sampling window while ignoring isolated transient changes (#302).The default
nvhcdnode-health request is limited to theprolog,epilog,logs, andgpuDCAHC groups; explicit caller arguments still override that default (#356).
Checkpointing
Logging, packaging, and security
gRPC log servers confine client-selected write paths to launcher-derived allowed roots and reject traversal, symlink escape, and other out-of-root destinations (#415).
Multi-node direct append to a shared per-cycle log is no longer recommended. Enable the gRPC log funnel for a single shared-filesystem writer; direct append can serialize on Lustre and can lose or interleave data on NFS-style storage (#418).
Release wheels and CI target Python 3.12 and 3.14, including an
ft_launchercompatibility fix for Python 3.14. Python 3.10 and 3.11 are no longer covered by the wheel/unit-test matrix (#342, #372).The optional MCP SDK is constrained to
>=1.28.1,<2.0.0, incorporating the fix for CVE-2026-59950 while retaining compatibility with the NVRx MCP integration (#388, #412).
Deprecations & Removals
Legacy dynamic rendezvous has been removed. Remove
--ft-rdzv-impl legacy; omit the option or use--ft-rdzv-impl barrier(#379).--ft-restart-policyremains deprecated. Onlyany-failedis supported and it is already the default, so remove the option from launch scripts.In-process restart remains deprecated. New deployments should use InJob restart through
ft_launcher.The
[dataflow]extra andnvdataflowdependency have been removed. Attribution export uses the configured direct HTTP endpoint instead (9182c06).
Installation
# Core resiliency, attrsvc, and the direct Restart Agent backend
pip install nvidia-resiliency-ext
# Optional MCP and Slack attribution integrations
pip install 'nvidia-resiliency-ext[attribution]'
Known Issues & Limitations
Linux wheel compatibility: published wheels target
manylinux_2_39; systems with glibc older than 2.39, including standard Ubuntu 22.04 installations, must build from source.Python: release wheels are published for Python 3.12 and 3.14. The package metadata still permits Python 3.10+, but Python 3.10 and 3.11 are not covered by the release wheel/CI matrix.
Attribution and Restart Agent features are experimental. APIs, service contracts, schemas, and model-route configuration may change.
External attribution endpoints: the InJob client currently submits logs over HTTP(S). Other endpoint schemes do not provide an additional transport implementation.
Segment-health decisions fail open when their artifact is missing or unusable. Slurm does not evict an already running allocation merely because a node becomes drained; the decision is consumed at the next rendezvous health-check boundary.
Per-cycle log aggregation is best-effort. Failure-adjacent output may be incomplete; retain launcher/rank-monitor logs and core dumps for authoritative diagnostics.
NVIDIA Resiliency Extension v0.6.0
Highlights
InJob restart
Barrier-based rendezvous (v2) is now the default (#214). The legacy dynamic rendezvous (v1) is deprecated and will be removed in a future release (#282).
Rendezvous protocol hardening — round-scoped keys, round-fenced CAS to prevent stale slot writes, and cleaner handling of participants exiting mid-rendezvous (#262, #263, #300).
Robust startup and shutdown — wait for TCPStore on initial connection (#264), handle signals during rendezvous (#246), notify peers to abort current workers on failure (#228), fix
terminate_mp_processesto cover failed workers (#270).Hot-spare node support — closes the v0.5 spare-node gap. Hot-spare is always-on and works with
--max-restarts(#226, #250, #266):Simple mode (default,
--ft-segment=None) for H100 / non-NVSwitch systems — firstmin_nodesfrom--nnodes=min:maxbecome active, the rest become standbys with reserved ranks. No GPU ClusterUUID required.Segment-aware mode (
--ft-segment=N) for NVSwitch systems (DGX H200, HGX B200) — uses GPU ClusterUUID to identify NVLink domains; nodes in the same segment get contiguous group ranks for NVLink locality. Requiresmin_nodes % segment == 0.Block-aware rank assignment (#250) and hot-spare exit-handling fix (#266).
Progress-based early termination for InJob restarts and progress-tracker enhancements (#218, #255).
External InJob control-plane (experimental) — embed
ft_launcherorchestration in a host control plane (#321). Not yet QA-validated; APIs may change.Section-timeout fixes — out-of-section timeout now fires for section-less workloads, baseline iteration tracking corrected (#261, #299).
--max-restartsnow reflects job-level restart attempts (#211);ft_launcherruns with sensible defaults out of the box (#205, #271).NUMA binding support in
ft_launcherfor optimized memory affinity (#209).
Health checks
Checkpointing
CPU shared-memory D2H path (experimental) in
FileSystemWriterAsyncremoves a redundant H2H copy and resolves the prior shm D2H race (#298).PersistentAsyncCaller upgrades: QoS control, worker data cache, warmup, IPC-handle caching via
ConsistentDataIdentifier, and class-level metadata cache inCachedMetadataFileSystemReader(#273, #274, #275).Reliability fixes: SIGSEGV on SIGKILL with dangling CUDA IPC handles (#284), CUDA IPC handle errors in persistent worker (#288), premature GC of preloaded pinned host tensors in
TemporalAsyncCaller(#291), MXFP8/TE quantized tensor handling in IPC cache (#276), spawned persistent worker CUDA-device init (#238).
Fault attribution — productized as standalone services (experimental)
The attribution module — including the Attribution Service, Flight Recorder integration, LogSage, and MCP integration — remains experimental in v0.6. APIs, CLI flags, and service contracts may change in subsequent releases.
NVRx Attribution Service (
attrsvc) and NVRx Slurm Monitor Service (smonsvc) introduced as FastAPI-based standalone services (#242, #248).ft_launcher-managedattrsvcfor co-located deployment (#318); UDS endpoints forattrsvc/smonsvc(#315).Attribution is now an optional package — install with
pip install nvidia-resiliency-ext[attribution](#305). Attribution internals refactored under asvcsubpackage with a clear controller/runner boundary (#295, #313, #316).PyTorch Flight Recorder (experimental) support in
attrsvc(#283); FR ordering switched to window-based instead of PG description (#210, #216, #219).LogSage (experimental) integrated as an attribution module — direct in-process API, configurable LLM model via env, LogSage v0.1.7 (#224, #249, #267, #289, #297, #308).
Triggers and outputs: last-cycle attribution trigger (#247), job-completion handling (#251), Slack bot notifications (#253).
MCP (Model Context Protocol) integration (experimental) exposes the attribution module as a tool for the NVIDIA resiliency agent (#215);
nvrx-attrClaude skill bundle for the attribution workflow (#312).
Log aggregation & observability
Two-level gRPC log aggregation (N leaves + root) with auto-tier selection and end-to-end tests (#280, #307, #309).
Writer-thread + persistent-reader refactor (#254); pipe-based per-cycle log capture and split-log support (#225, #240).
NVRxCycleInfoprotobuf exposes per-cycle metadata over the new gRPC interface (#258, #292).GPU memory logger (#206).
Build, packaging & security
poetry-dynamic-versioning— wheel versions are derived from git tags (v0.6.0-rc1,v0.6.0, etc.) (#260).New optional extras:
[attribution]for the attribution service stack (LogSage, MCP, FastAPI, Slack),[dataflow]fornvdataflowintegration.Wheel security-scan cleanups and Bandit findings (
ionicesubprocess, FR module) (#219, #294, #306).Torch 2.10 and
langchainlock updates for CVE compliance.
Deprecations & Removals
Legacy dynamic rendezvous (v1) is deprecated; barrier-based rendezvous (v2) is the default. Plan to remove v1 in a future release (#282).
--ft-restart-policyis deprecated (#259). Themin-healthyvalue has been removed; onlyany-failedremains (which is also the default), so the flag is effectively a no-op in v0.6. Migration: remove the flag from your launch scripts. No replacement is required. The flag itself will be removed in a future release.ptl_resiliencypackage is deprecated and removed from the wheel (#282, #285). PyTorch Lightning users should pin v0.5.x or migrate to the underlyingfault_tolerance/checkpointing/attribution.stragglerAPIs directly.OneLogger integration removed (#257).
In-process restart is deprecated in v0.6 and will be removed in a future release. It is no longer the focus of the fast-restart solution. Migration: use InJob restart via
ft_launcher(see thefault_toleranceusage guide). The existingnvidia_resiliency_ext.inprocessAPIs continue to work in v0.6 but are no longer maintained for new feature development; bug fixes will be considered on a case-by-case basis.
Installation
# Core resiliency (fault tolerance, checkpointing, health checks)
pip install nvidia-resiliency-ext
# With attribution stack (Attribution Service, LogSage, MCP, Slack notifications)
pip install nvidia-resiliency-ext[attribution]
# With nvdataflow integration
pip install nvidia-resiliency-ext[dataflow]
Known Issues & Limitations
Ubuntu 22.04 / glibc < 2.39 users are advised to build from source — PyPI wheels target
manylinux_2_39and default to CUDA 13.Python: wheels are published for 3.10, 3.11, 3.12.
Attribution, Flight Recorder analysis, LogSage, and MCP integration are experimental. APIs, CLI flags, and service contracts may change in subsequent releases.
External InJob control-plane is experimental and not yet QA-validated; APIs may change in subsequent releases.
CPU shared-memory D2H path in
FileSystemWriterAsyncis experimental (#298); enable only in non-production runs while we collect soak feedback.
NVIDIA Resiliency Extension v0.5.0
Highlights
InJob restarts
PRs (185, 190, 201) improve the scalability, profiling, and performance of InJob restarts through improvements to the rendezvous operation
Key scaling and fault-tolerance improvements:
New barrier-based rendezvous operation introduces a substantial redesign that addresses several limitations of the previous dynamic rendezvous implementation. This provides more predictable, stable, and scalable InJob behavior
Faster termination path:
The worker termination timeout (–workers-stop-timeout) has been reduced from 30 seconds to 15 seconds, improving failure recovery latency and overall job responsiveness
New Flag for Infra-Aligned Rank Assignment:
A new flag, –ft-use-infra-group-rank, allows InJob scaling to follow the infrastructure scheduler’s rank assignment, preserving topology-aware placement decisions
Migration Guidance:
While the previous dynamic rendezvous-based implementation (v1) remains supported, users are strongly encouraged to adopt barrier-based rendezvous (v2) for improved reliability, stability, and performance
Enhanced GPU and NVLink health checks
PR 145 introduces several improvements to health check module including
Refactored
GPUHealthCheckto support device-specific monitoringNew
NVLHealthCheckclass for NVLink health validationAutomatic health check chaining in
WrapperclassChainedGPUHealthCheckandChainedNVLHealthCheckfor in-process useSingle GPU health check API for individual device validation and updated trace collector to use new GPU health check API
Node health check service integration
ft_launchercan query a node-local health check service before rendezvous by passing--ft-node-health-check-endpoint.Users can reuse NVIDIA Base Command Manager (BCM) Slurm prolog or epilog health checks behind the same service interface. For example, customers can deploy an
nvhcd-compatible daemon that invokes an existing BCM health check script and converts the result to the NVRx health check response format.The integration is service-compatible: users can provide any equivalent daemon that implements the NVRx
HealthCheckService.RunHealthCheckgRPC API over a Unix domain socket and returns JSON output with afailed_checksarray.Example:
--ft-node-health-check-endpoint unix:///var/run/nvhcd.sock. If the endpoint is not configured, or the optional service is unavailable, NVRx skips the external node health check. NVRx marks the node unhealthy when the service returnssuccess=falseor when a successful response contains a non-emptyfailed_checksarray; unusable successful responses are logged and ignored.
Checkpointing
PRs (108, 138, 154, 169, 170, 193, 197, 199) improve the stability of checkpointing by deprecating the use of fork in asynchronous checkpointing, simplifying error propagation and shutdown cleanup logic
Introduced the option to use Multithread File IO Instead of Multiprocess to simplify error propagation logic, improve shutdown cleanup and enhance overall stability
Made persistent async checkpoint worker default (except for local checkpointing) and fixed cross-call state pollution
Added ability to abort async checkpoint process
Fault attribution (new module introduced in v0.5)
PR 141 introduces the base attribution class which can be used to define any attribution module. This provides asynchronous combining multiple modules directly.
PR 172 improves error attribution by dumping NCCL traces from PyTorch for collective analysis on hang or watchdog timeout
It is an experimental module to identify ranks interrupting workload progress by analyzing Flight Recorder traces. It detects GPU errors, host issues, and GIL locks
PyT’s watchdog is currently configured to include the training process’s stack trace when generating Flight Recorder traces. However, this can lead to a deadlock if the trainer fails inside a routine that performs collectives while holding the GIL, since capturing the stack trace requires reacquiring the GIL. A new environment variable, TORCH_INCLUDE_STACK_TRACE=False (Default: True), has been added to PyTorch main to avoid this issue. This change will be included in the NGC PyT 25.11 container.
Known Issues & Limitations
Spare-Node Support
Spare nodes are not supported by either dynamic rendezvous or barrier-based rendezvous in the current release.
The earlier dynamic rendezvous technically supported spare nodes, but only when infra group rank assignment was not used. That mode isn’t viable in real deployments because bypassing the infrastructure topology-aware rank assignment leads to degraded performance and inconsistent scaling behavior. Because of this, spare-node support isn’t available in this release.
With barrier-based rendezvous, we’ve aligned fully with infra-assigned ranks to ensure correctness and performance. Spare-node support for barrier-based rendezvous is planned for a future update.
CUDA 12 and Ubuntu 22.04 users are advised to build from source, since PyPI wheel for v0.5 defaults to CUDA 13
In-process restart requires NCCL < v2.28.3 OR >= 2.28.9 due to a segmentation fault issue
NVIDIA Resiliency Extension v0.4.1
Highlights
This hotfix release includes important bug fixes, performance improvements, and minor updates to enhance stability.
Checkpointing
PR 104, PR 106, PR 108, PR 111 and PR 116 fix the asynchronous checkpointing module to switch from temporal to using the persistent worker that uses
spawninstead offork.The fix in this release is working toward an intermediate milestone of deprecating the use of
forkand instead using aspawnfor asynchronous checkpointing. The complete transition to usingspawnhas the following dependencies onforkthat will be eliminated in upcoming release:Local checkpointing must continue to use the
forkbased asynchronous checkpointing as clarified in the usage guide.File IO operations with multiprocessing can still trigger a
fork
In-process restart
PR 103 fixes a case where extra CUDA contexts were created on local rank 0 after restart, consuming extra GPU memory on local rank 0.
PR 112 fixes the workload state leaks across the restart boundary. The fix addresses a case where objects created in the wrapped function could not be garbage collected after a restart, manifesting as a memory leak.
Known Issues & Limitations
In a future release, we will add changes to automatically terminate the persistent process when the main process terminates.
Until this change is implemented, job schedulers must ensure proper termination of the persistent process and its child workers for a graceful shutdown.
NVIDIA Resiliency Extension v0.4.0
Highlights
Checkpointing
PR 29 - Support for storing checkpoints to cloud object stores
Leverage cloud storage provider’s multithreaded SDK for rapid loading and saving checkpoints to object stores such as AWS S3, Azure Blob Storage, Google Cloud Storage and more using NVIDIA Multi-storage Client
Provide scalable, reliable, cheaper, single source of truth across clouds/regions
Provide opt-out configuration when creating FileSystemWriterAsync class instance to allow users to passthrough to the filesystem
PR 36 - Critical bug fix to enable async checkpoint loading without errors
In-process & InJob restart
PR 35 - Nested restarter updates for in-process restart to align with InJob restart, so users have a consistent experience across in-process and InJob restarts
Updates to in-process nested restart functionality provided by Python Wrapper class and existing callback infrastructure with additional callbacks and logging
Known Issues & Limitations
Dependencies:
NVIDIA Resiliency Extension v0.3.0
Highlights
Support for Blackwell GPU
ARM based host CPU support
In-process & InJob restart
Hierarchical in-process and InJob restart support
Warm spare support
Health checks
GPU health check based on NVML
NIC
Checkpointing
Existing capabilities that used to be part of Megatron Core is refactored to be part of NVRx. The checkpointing feature will be maintained as part of NVRx, and Megatron Core and NeMo will use the code from NVRx in the future.
Added support for checkpoint metadata caching to improve performance for subsequent checkpoints.
Known Issues & Limitations
GPU health check requires NVML driver version >= 570
Current checkpointing implementation doesn’t support persistent queue with replication
NVIDIA Resiliency Extension v0.2.1
Highlights
This release includes important bug fixes, performance improvements, and minor updates to enhance stability.
Build Fixes & Code Improvements
Fixed missing #include to ensure proper compilationv in pytorch:24.12-py3 container.
Lazy loading of cupti_module.
Fixed ForkingPickler to ensure proper installation in pytorch:25.01-py3 container.
NVIDIA Resiliency Extension v0.2.0
Highlights
We excited to introduce many new features in NVIDIA Resiliency Extension v0.2.0.
In-process restart
Provides a mechanism to restart the training without killing the running process via a Python function wrapper.
Compared to a traditional scheduler-level restart, restarting within the same process removes overheads associated with launching a new scheduler job, starting a container, initializing a new Python interpreter, loading dependencies, and creating a new CUDA context.
Asynchronous checkpoint
Provides core utilities to make checkpointing routines run in the background.
It uses torch.multiprocessing to fork a temporary process to initiate asynchronous checkpointing routine.
Application can check this asynchronous checkpoint save in a non-blocking manner and specify a user-defined finalization step when all ranks finish their background checkpoint saving.
Local checkpoint
Provides a mechanism to create a checkpoint in local host memory.
The local checkpointing mechanism is implemented via the Python LocalCheckpointManager class, which operates on a TensorAwareStateDict wrapper.
This wrapper encapsulates the operations necessary for efficient replication and data transfers.
Known Issues & Limitations
For in-process restart
If there is hang, presence of SHARP raises an exception, which leads to triggering InJob restart and in-process restart.
Customer needs to disable SHARP for using in-process restart with current version.
Requires ENV VARs to be set as follows:
NCCL_NVLS_ENABLE=0 to disable SHARP.
NCCL_NET_PLUGIN=”none” if NCCL version < 2.24.1 to avoid duplicate NCCL net plugin init.
In-process and InJob restart
These work with PyTorch version 24.07, 24.08, 24.09, and 24.10 but not with 24.11 due to a known NCCL issue.
NVIDIA Resiliency Extension v0.1.3
Highlights
We are excited to announce the first release of NVIDIA Resiliency Extension v0.1.3!
Straggler Detection API
Provides tools for user to mark the section of code and configure the threshold to detect slow running GPU.
Fault Tolerance API
Provides the rank monitor server and client, and modified torchrun launcher based on TorchElastic to automatically detect hangs and restart training with InJob.