nvalchemi.training.distributed.all_reduce_flags#
- nvalchemi.training.distributed.all_reduce_flags(flag, manager=None)[source]#
Return every rank’s flag as one tensor, this rank’s at its own index.
Each rank raises its own entry of a zero vector and a
MAXall-reduce collects the vector, so the result reads the same on every rank and names the ranks whose flag was set. The vector lives on the devicecollective_device()picks for the backend in use, so callers never choose a device for a collective themselves.- Parameters:
flag (bool | int) – This rank’s verdict. A truthy value raises this rank’s entry.
manager (DistributedManager | None, optional) – Manager whose rank, world size, and
all_reduceare used when given. DefaultNonereadstorch.distributedor the launcher’s environment.
- Returns:
Integer tensor of shape
(world_size,)holding1where a rank raised its flag. A single process gets a length-one tensor holding its own flag, without a collective.- Return type:
Examples
>>> from nvalchemi.training.distributed import all_reduce_flags >>> flags = all_reduce_flags(shard_is_empty, manager) >>> if bool(flags.any()): ... raise ValueError(f"Ranks {flags.nonzero().flatten().tolist()} came up empty.")