cuda.bindings._v2.driver.checkpoint_process_restore#

cuda.bindings._v2.driver.checkpoint_process_restore(int pid, args)#

Restore a CUDA process’s GPU memory contents from its last checkpoint.

Restores a CUDA process specified by pid from its last checkpoint. Process must be in the CU_PROCESS_STATE_CHECKPOINTED state to restore. GPU UUID pairs can be specified in args to remap the process old GPUs onto new GPUs. The GPU to restore onto needs to have enough memory and be of the same chip type as the old GPU. If an array of GPU UUID pairs is specified, it must contain every checkpointed GPU. CUDA process restore requires persistence mode to be enabled or cuInit to have been called before execution. When CUcheckpointRestoreArgs.customStorageInfo_out is not NULL, all the GPU memory to be restored for the process with pid will be mapped onto the calling process’s GPUs so the application can copy the memory back to the GPU. Upon return of this call, the pointer referenced by CUcheckpointRestoreArgs.customStorageInfo_out will point to a CUcheckpointCustomStorageInfo struct allocated and populated by the driver. For each GPU which contains data from the restored process, there is a corresponding entry in the CUcheckpointCustomStoragePerDeviceData array.

The device pointers and sizes for the contiguously mapped memory on a particular GPU are set in the CUcheckpointCustomStoragePerDeviceData.devPtr and CUcheckpointCustomStoragePerDeviceData.size fields respectively.
A stream belonging to the primary context of the GPU is set in CUcheckpointCustomStoragePerDeviceData.stream. The application can use the stream to find out which GPU the memory is mapped on and to enqueue the copies from custom storage back to the GPU.

The application is responsible for copying the data from custom storage to the GPU at the specified device address. When restoring from custom storage, the application is expected to call cuCheckpointOperationComplete. The CUcheckpointCustomStorageInfo.handle set by this call should be passed to cuCheckpointOperationComplete in handle. The driver synchronizes all the streams in cuCheckpointOperationComplete, at which point the copy from custom storage is considered complete. When restoring from custom storage, the application is expected to retain primary contexts of all the devices used by the process to be restored. Otherwise cuCheckpointProcessRestore will return CUDA_ERROR_INVALID_CONTEXT. Restoring the calling process’s GPU memory from custom storage is not supported. Upon successful return, the target process will be in the CU_PROCESS_STATE_RESTORING state. When CUcheckpointRestoreArgs.customStorageInfo_out is NULL, the GPU memory contents will be restored from host memory by cuCheckpointProcessRestore. The application is not expected to copy back the GPU memory contents. The application is also not expected to call cuCheckpointOperationComplete. Upon successful return, the target process will be in the CU_PROCESS_STATE_LOCKED state.

Parameters:

pid (int) – The process ID of the CUDA process.
args (intptr_t) – Optional restore operation arguments.