nvalchemi.distributed.ProcessGroupContext#

class nvalchemi.distributed.ProcessGroupContext(process_group, *, execution_device=None)[source]#

Read-only topology and communication-device details for a supplied group.

The context never creates, destroys, or otherwise owns the process group. Construct it after the caller has initialized the group and confirmed that this process is a member.

Parameters:
  • process_group (torch.distributed.ProcessGroup) – An initialized process group that includes this process.

  • execution_device (torch.device or str, optional) – Communication-device hint. Gloo always uses CPU. NCCL resolves a CUDA hint first, then an initialized DistributedManager CUDA device, then the caller’s current CUDA device.

Raises:
  • TypeError – If process_group is not a concrete PyTorch process group.

  • RuntimeError – If distributed is uninitialized, the group is unusable, or NCCL has no usable CUDA device.

  • ValueError – If the process is not a member or an explicit CUDA index is invalid.

  • NotImplementedError – If the group’s backend is neither Gloo nor NCCL.

communication_scope()[source]#

Select NCCL’s resolved device for scoped communication, then restore it.

Return type:

Iterator[None]

global_rank(group_rank)[source]#

Translate a group-local rank to its global rank.

Parameters:

group_rank (int) – Rank within this context’s process group.

Returns:

The corresponding global rank.

Return type:

int

Raises:
  • TypeError – If group_rank is not a non-boolean integral value.

  • ValueError – If group_rank is outside this group’s rank range.