Rank Filter

class nvidia_resiliency_ext.inprocess.rank_assignment.ActivateAllRanks

Activates all distributed ranks.

All healthy distributed ranks will call the provided wrapped function in the next iteration of inprocess.Wrapper.

ActivateAllRanks unconditionally activates all ranks, and cannot be combined with any other RankAssignment performing rank activation.

class nvidia_resiliency_ext.inprocess.rank_assignment.ActiveWorldSizeDivisibleBy(divisor=1)

ActiveWorldSizeDivisibleBy ensures that the active world size is divisible by a given number. Ranks within the adjusted world size are marked as active and are calling the wrapped function, while ranks outside this range are marked as inactive.

Parameters:

divisor (int) – the divisor to adjust the active world size by

class nvidia_resiliency_ext.inprocess.rank_assignment.MaxActiveWorldSize(max_active_world_size=None)

MaxActiveWorldSize ensures that the active world size is no greater than the specified max_active_world_size. Ranks with indices less than the active world size are active and calling the wrapped function, while ranks outside this range are inactive.

Parameters:

max_active_world_size (int | None) – maximum active world size, no limit if None

class nvidia_resiliency_ext.inprocess.rank_assignment.RankFilter

RankFilter is a subclass of RankAssignment which selects which ranks are active in the current restart iteration of inprocess.Wrapper.

Active ranks call the wrapped function. Inactive ranks are waiting idle, and could serve as a pool of static, preallocated and preinitialized reserve ranks. Reserve ranks would be activated in a subsequent restart iteration if one of the active ranks is terminated or becomes unhealthy.

Multiple RankFilter or RankAssignment instances can be composed using inprocess.Compose to achieve the desired behavior. Typically, all RankFilter instances should follow any RankAssignment steps that recalculate rank indices or adjust the world size.