Asynchronous Pytorch Distributed Checkpoint save with optimized FileSystemWriter

State dict saver for PyT Distributed format allowing asynchronous save.

class nvidia_resiliency_ext.checkpointing.async_ckpt.state_dict_saver.CheckpointMetadataCache[source]

Bases: object

Global metadata and plans carried from one save to the next, to reuse the metadata.

Building the global metadata gathers every rank’s local plan on the coordinator. With enable_cache, save_state_dict_async_plan avoids that:

  • The first save in a process reuses the metadata of a loaded checkpoint (see set_cached_global_metadata) if all ranks’ local plans write exactly the chunks it lists, which one all_reduce checks.

  • Later saves reuse the plans and metadata of the previous save without communicating: enable_cache promises that the checkpoint structure does not change. Each rank still creates its local plan and raises if it differs from the previous one.

The metadata is kept without storage_data. With the fast .metadata writers it is kept as a few arrays, which the garbage collector doesn’t visit, and shares no objects with the Metadata it was made from but its fqn strings. The cache never modifies the Metadata it was given.

set_cached_global_metadata(cached_global_metadata)[source]

Sets the global metadata of a loaded checkpoint, to be reused if it still applies.

Every rank must set it. The next save checks whether all ranks’ local plans write exactly the chunks this metadata lists, and if so reuses it instead of gathering the plans and building the global metadata again. The metadata is not modified, and with the fast .metadata writers the cache keeps none of its objects, so the caller can free it.

Parameters:

cached_global_metadata (Metadata) – The global metadata from a previous checkpoint.

class nvidia_resiliency_ext.checkpointing.async_ckpt.state_dict_saver.PendingSave(storage_writer, dist_wrapper, *, _metadata_template=None, _fqn_list_slot=None, _delivers_fqn_list=False)[source]

Bases: object

A save planned and staged by save_state_dict_async_plan, to pass to save_state_dict_async_finalize.

Parameters:
  • storage_writer (FileSystemWriterAsync)

  • dist_wrapper (_DistWrapper)

  • _metadata_template (MetadataTemplate | None)

  • _fqn_list_slot (_FqnListSlot | None)

  • _delivers_fqn_list (bool)

dist_wrapper: _DistWrapper
storage_writer: FileSystemWriterAsync
nvidia_resiliency_ext.checkpointing.async_ckpt.state_dict_saver.init_checkpoint_metadata_cache(cached_global_metadata=None)[source]

Initializes the checkpoint metadata cache.

This function creates a new CheckpointMetadataCache instance and sets the cached global metadata from the previous checkpoint

Parameters:

cached_global_metadata (Metadata)

nvidia_resiliency_ext.checkpointing.async_ckpt.state_dict_saver.save_state_dict_async_finalize(pending: PendingSave) → None[source]
nvidia_resiliency_ext.checkpointing.async_ckpt.state_dict_saver.save_state_dict_async_finalize(storage_writer: FileSystemWriterAsync, global_metadata: Any, dist_wrapper: _DistWrapper, /) → None

Finalize the save that save_state_dict_async_plan planned and staged.

The write_results are retrieved from the pending save’s storage_writer. Each rank sends them to the coordinator as a table (see _metadata_pickler.write_result_table) if it can, else pickled.

Parameters:
  • pending (PendingSave) – the save planned and staged by save_state_dict_async_plan.

  • legacy_args – deprecated: the plan’s global metadata (a Metadata is accepted too) and distributed wrapper, after its storage writer as the first argument, as save_state_dict_async_plan used to return them in a tuple.

Return type:

None

Returns: None

nvidia_resiliency_ext.checkpointing.async_ckpt.state_dict_saver.save_state_dict_async_plan(state_dict, storage_writer, process_group=None, coordinator_rank=0, planner=None, enable_cache=False, metadata_cache=None)[source]

First stage of saving a state dict to storage.

This is an async adjustment of torch.distributed.checkpoint.state_dict_saver. In order to support async save, saving should be split into three parts:

  1. Planning

  2. Actual saving

  3. Finalization

Out of these, step (2) must happen asynchronously. The first step is realized with this function.

The planning part consists of several steps, described here: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.SavePlanner

Parameters:
  • state_dict (STATE_DICT_TYPE) – state dict to save

  • storage_writer (FileSystemWriterAsync) – in current version only an instance of FileSystemWriterAsync

  • process_group (dist.ProcessGroup, optional) – process group used for save planning

  • coordinator_rank (int) – coordinator rank for planning. Defaults to 0.

  • planner (SavePlanner, optional) – save planner for torch.distributed.checkpoint format

  • enable_cache (bool) – Reuse global metadata instead of building it: the first save reuses a loaded checkpoint’s metadata if it still applies, later saves reuse the previous save’s plans and metadata, as the checkpoint structure must not change between them (a rank whose local plan changed raises). See CheckpointMetadataCache. Defaults to False.

  • metadata_cache (CheckpointMetadataCache, optional) – Custom metadata cache instance to use for storing and retrieving checkpoint metadata. If not provided, the global cache will be used.

Returns:

the planned and staged save, to pass to save_state_dict_async_finalize.

Return type:

PendingSave