nvalchemi.data.datapipes.distributed_shard#

nvalchemi.data.datapipes.distributed_shard(indices, *, num_replicas, rank, drop_last, pad=True)[source]#

Return the subset of epoch items assigned to one distributed rank.

Parameters:
  • indices (list) – Sample indices in the order they would be retrieved for this epoch before splitting the work across data-parallel ranks. In a single-process run, this would be the sampler order. The list is not modified.

  • num_replicas (int) – Number of distributed ranks sharing the epoch.

  • rank (int) – Rank whose local shard should be returned.

  • drop_last (bool) – Whether to truncate the full epoch instead of padding it when the epoch length is not evenly divisible by num_replicas. Ignored when pad=False.

  • pad (bool, optional) – Whether to resize an uneven epoch so every rank receives the same number of items. The epoch is padded, or truncated when drop_last=True. False deals the items strided without resizing: the shards are then disjoint, cover indices exactly, and differ in length by at most one. Default True.

Returns:

Rank-local shard of indices.

Return type:

list

Notes

To make strided sharding produce the same number of items on each rank, the full epoch order is first resized to total_size. num_samples is the number of items one rank should emit, computed as ceil(len(indices) / num_replicas) unless drop_last=True requires truncating an uneven tail. total_size is the all-rank item count, num_samples * num_replicas.

With drop_last=True, the full list is truncated to total_size. Otherwise, items from the beginning of the epoch are repeated until the list is evenly divisible across ranks, matching PyTorch DistributedSampler behavior. After resizing, rank r receives every num_replicas-th item starting at offset r: indices[r:total_size:num_replicas]. With pad=False no resizing happens and rank r receives indices[r::num_replicas].