cuda.bindings._v2.driver.checkpoint_process_checkpoint#

cuda.bindings._v2.driver.checkpoint_process_checkpoint(int pid, args)#

Checkpoint a CUDA process’s GPU memory contents.

Checkpoints a CUDA process specified by pid. The GPU memory contents will be brought into host memory or mapped onto the calling process’s GPUs for user- defined behavior. Underlying GPU references are released when checkpointing completes. The process must be in the CU_PROCESS_STATE_LOCKED state to checkpoint. When CUcheckpointCheckpointArgs.customStorageInfo_out is not NULL, all the GPU memory allocated by the process with pid will be mapped onto the calling process’s GPUs so the application can copy the memory to custom storage. Upon return of this call, the pointer referenced by CUcheckpointCheckpointArgs.customStorageInfo_out will point to a CUcheckpointCustomStorageInfo struct allocated and populated by the driver. For each GPU which contains data from the checkpointed process, there is a corresponding entry in the CUcheckpointCustomStoragePerDeviceData array.

The device pointers and sizes for the contiguously mapped memory on a particular GPU are set in the CUcheckpointCustomStoragePerDeviceData.devPtr and CUcheckpointCustomStoragePerDeviceData.size fields respectively.
A stream belonging to the primary context of a particular GPU is set in CUcheckpointCustomStoragePerDeviceData.stream. The application can use the stream to find out which GPU the memory is mapped on and to enqueue the copies to custom storage.

The application is responsible for copying the mapped data from the GPU at the specified device address to custom storage. When checkpointing to custom storage, the application is expected to call cuCheckpointOperationComplete. The CUcheckpointCustomStorageInfo.handle set by this call should be passed to cuCheckpointOperationComplete in handle. The driver synchronizes all the streams in cuCheckpointOperationComplete, at which point the copy to custom storage is considered complete. When checkpointing to custom storage, the application is expected to retain primary contexts of all the devices used by the process to be checkpointed. Otherwise cuCheckpointProcessCheckpoint will return CUDA_ERROR_INVALID_CONTEXT. Unlike other checkpoint operations, checkpointing to custom storage requires cuInit to have been called before execution. Checkpointing the calling process’s GPU memory to custom storage is not supported. Upon successful return, the target process will be in the CU_PROCESS_STATE_CHECKPOINTING state. When CUcheckpointCheckpointArgs.customStorageInfo_out is NULL, the GPU memory contents will be brought into host memory by cuCheckpointProcessCheckpoint. The application is not expected to copy the GPU memory contents. The application is also not expected to call cuCheckpointOperationComplete. Upon successful return, the target process will be in the CU_PROCESS_STATE_CHECKPOINTED state.

Parameters:

pid (int) – The process ID of the CUDA process.
args (intptr_t) – Optional checkpoint operation arguments.