nvalchemi.training.rehome_optimizer_state#

nvalchemi.training.rehome_optimizer_state(optimizer)[source]#

Move an optimizer’s per-parameter state onto its parameters’ devices.

torch.optim.Optimizer.load_state_dict() places state on the parameter devices as they stand at load time and never revisits them, so any move afterwards strands the state: resuming a checkpoint and then calling run(), or letting a DDPHook re-pin a rank to its local GPU, leaves exp_avg on the old device and the first step raises Expected all tensors to be on the same device. Call this after the parameters have reached their final devices and before the first step.

A top-level step entry is left on the CPU when it is already there, which is the placement PyTorch uses for optimizers that are neither capturable nor fused; every other tensor, step included, follows its parameter.

Each state entry is walked recursively, so a custom optimizer that keeps its moments inside a dict, list, or tuple is rehomed as thoroughly as Adam, and the containers it chose are preserved.

Parameters:

optimizer (torch.optim.Optimizer) – Optimizer whose state is rehomed in place. Parameters without state (never stepped) are skipped.

Return type:

None

Examples

>>> from nvalchemi.training.runtime import rehome_optimizer_state
>>> model.to("cuda:1")
>>> rehome_optimizer_state(optimizer)