nvalchemi.data.datapipes.distributed_shard#
- nvalchemi.data.datapipes.distributed_shard(indices, *, num_replicas, rank, drop_last, pad=True)[source]#
Return the subset of epoch items assigned to one distributed rank.
- Parameters:
indices (list) – Sample indices in the order they would be retrieved for this epoch before splitting the work across data-parallel ranks. In a single-process run, this would be the sampler order. The list is not modified.
num_replicas (int) – Number of distributed ranks sharing the epoch.
rank (int) – Rank whose local shard should be returned.
drop_last (bool) – Whether to truncate the full epoch instead of padding it when the epoch length is not evenly divisible by
num_replicas. Ignored whenpad=False.pad (bool, optional) – Whether to resize an uneven epoch so every rank receives the same number of items. The epoch is padded, or truncated when
drop_last=True.Falsedeals the items strided without resizing: the shards are then disjoint, coverindicesexactly, and differ in length by at most one. DefaultTrue.
- Returns:
Rank-local shard of
indices.- Return type:
list
Notes
To make strided sharding produce the same number of items on each rank, the full epoch order is first resized to
total_size.num_samplesis the number of items one rank should emit, computed asceil(len(indices) / num_replicas)unlessdrop_last=Truerequires truncating an uneven tail.total_sizeis the all-rank item count,num_samples * num_replicas.With
drop_last=True, the full list is truncated tototal_size. Otherwise, items from the beginning of the epoch are repeated until the list is evenly divisible across ranks, matching PyTorchDistributedSamplerbehavior. After resizing, rankrreceives everynum_replicas-th item starting at offsetr:indices[r:total_size:num_replicas]. Withpad=Falseno resizing happens and rankrreceivesindices[r::num_replicas].