nvalchemi.training.distributed.all_reduce_flags#

nvalchemi.training.distributed.all_reduce_flags(flag, manager=None)[source]#

Return every rank’s flag as one tensor, this rank’s at its own index.

Each rank raises its own entry of a zero vector and a MAX all-reduce collects the vector, so the result reads the same on every rank and names the ranks whose flag was set. The vector lives on the device collective_device() picks for the backend in use, so callers never choose a device for a collective themselves.

Parameters:
  • flag (bool | int) – This rank’s verdict. A truthy value raises this rank’s entry.

  • manager (DistributedManager | None, optional) – Manager whose rank, world size, and all_reduce are used when given. Default None reads torch.distributed or the launcher’s environment.

Returns:

Integer tensor of shape (world_size,) holding 1 where a rank raised its flag. A single process gets a length-one tensor holding its own flag, without a collective.

Return type:

torch.Tensor

Examples

>>> from nvalchemi.training.distributed import all_reduce_flags
>>> flags = all_reduce_flags(shard_is_empty, manager)
>>> if bool(flags.any()):
...     raise ValueError(f"Ranks {flags.nonzero().flatten().tolist()} came up empty.")