nvalchemi.training.distributed.all_gather_rows#

nvalchemi.training.distributed.all_gather_rows(tensor, manager=None, *, differentiable=True)[source]#

Return every rank’s tensor stacked along the leading dim, and this rank’s rows.

Shards may differ in their leading size. The sizes are gathered first, on the device collective_device() picks for the backend, then every shard is padded to the largest, gathered, and trimmed back, so the result holds each rank’s rows in rank order and nothing else. With differentiable, the gather goes through torch.distributed.nn.functional.all_gather(), so a gradient reaching the gathered tensor flows back to each rank’s own rows, summed over the ranks that used them. A single process gets its tensor back unchanged, with a slice over all of it, without a collective.

Parameters:
  • tensor (torch.Tensor) – This rank’s rows, of shape (n, ...). The trailing shape and the dtype must agree across ranks.

  • manager (DistributedManager | None, optional) – Manager whose rank and world size are read when given. Default None reads torch.distributed or the launcher’s environment.

  • differentiable (bool, optional) – Whether the gathered tensor carries an autograd graph back to tensor. Default True.

Returns:

The world tensor, of shape (sum of every rank's n, ...), and the slice of its rows that came from this rank.

Return type:

tuple[torch.Tensor, slice]

Raises:

RuntimeError – If more than one rank is reported but no process group is initialized.

Examples

>>> from nvalchemi.training.distributed import all_gather_rows
>>> world, rows = all_gather_rows(energies)
>>> torch.equal(world[rows], energies)
True