Distributed runtime utilities#

General-purpose helpers for running any distributed nvalchemi workflow — DDP training, multi-GPU inference, or your own multi-process script. They are independent of the spatial domain-decomposition API and are re-exported from the package root (from nvalchemi.distributed import DistributedManager).

Process-group manager#

DistributedManager is the recommended way to initialise and query the process group. It is the single object to construct once per process; the rest of the toolkit (for example DDPHook) reads rank, world size, and the rank-local device from it. Use it whenever a workflow needs a coordinated group of processes, not just for domain decomposition.

`DistributedManager`	Distributed Manager for setting up distributed training environment.
`PhysicsNeMoUninitializedDistributedManagerWarning`	Warning to indicate usage of an uninitialized DistributedManager

Parameter resolvers#

Rather than reading environment variables or torch.distributed state by hand, use these best-practice resolvers. Each returns a sensible value whether the run is launched under DistributedManager, plain torch.distributed, torchrun environment variables, or single-process — so the same code path works in every launch mode.

resolve_world_size() — the number of processes.
resolve_global_rank() — this process’s global rank (accepts an explicit override).
collective_device() — the device to place tensors on for collectives (CPU for the Gloo backend, the rank-local CUDA device for NCCL).

`resolve_world_size`	Resolve world size from PhysicsNeMo, torch.distributed, or environment.
`resolve_global_rank`	Resolve global rank from an explicit value, distributed state, or env.
`collective_device`	Resolve the rank-local device for distributed tensor collectives.