nvalchemi.training.rehome_optimizer_state#
- nvalchemi.training.rehome_optimizer_state(optimizer)[source]#
Move an optimizer’s per-parameter state onto its parameters’ devices.
torch.optim.Optimizer.load_state_dict()places state on the parameter devices as they stand at load time and never revisits them, so any move afterwards strands the state: resuming a checkpoint and then callingrun(), or letting aDDPHookre-pin a rank to its local GPU, leavesexp_avgon the old device and the first step raisesExpected all tensors to be on the same device. Call this after the parameters have reached their final devices and before the first step.A top-level
stepentry is left on the CPU when it is already there, which is the placement PyTorch uses for optimizers that are neither capturable nor fused; every other tensor,stepincluded, follows its parameter.Each state entry is walked recursively, so a custom optimizer that keeps its moments inside a dict, list, or tuple is rehomed as thoroughly as
Adam, and the containers it chose are preserved.- Parameters:
optimizer (torch.optim.Optimizer) – Optimizer whose state is rehomed in place. Parameters without state (never stepped) are skipped.
- Return type:
None
Examples
>>> from nvalchemi.training.runtime import rehome_optimizer_state >>> model.to("cuda:1") >>> rehome_optimizer_state(optimizer)