memory_monitor

GPU/CPU memory and utilization monitoring utilities.

Provides GPUMemoryMonitor, which samples GPU memory/utilization (device level, via NVML) and CPU memory/utilization (system-wide plus this process tree, via psutil) at a fixed interval from a background thread, and prints a peak/mean summary on stop. When an output_dir is given it also writes a per-sample CSV timeseries (mem_trace.csv) and continuously refreshes the summary (mem_peak.txt) there, so a peak report survives even if the process is killed (SIGTERM/OOM) before stop() runs.

GPU monitoring is scoped to CUDA_VISIBLE_DEVICES when set (integer indices), so the report reflects the workload’s GPUs rather than every device on a shared node. If NVML is unavailable the monitor degrades to CPU-only rather than failing.

Example

>>> monitor = launch_memory_monitor(monitor_interval=1.0)
>>> # ... run your GPU workload ...
>>> monitor.stop()  # prints peak GPU/CPU memory + mean utilization

Classes

GPUMemoryMonitor

Monitor GPU and CPU memory + utilization in a background thread.

Functions

launch_memory_monitor

Launch a GPU/CPU memory monitor in a background thread.
class GPUMemoryMonitor

Bases: object

Monitor GPU and CPU memory + utilization in a background thread.

Tracks peak memory and mean utilization for the monitored GPUs (device level, i.e. all processes on the GPU) and for the CPU (system-wide plus this process tree). GPUs are scoped to CUDA_VISIBLE_DEVICES when set. If output_dir is set, a CSV timeseries and a continuously-refreshed peak/mean summary are written there.

__init__(monitor_interval=10.0, output_dir=None)

Initialize the monitor.

Parameters:

  • monitor_interval (float) – Seconds between samples.

  • output_dir (str | None) – If set, write mem_trace.csv and mem_peak.txt here.

start()

Start monitoring in a background daemon thread.

stop()

Stop monitoring, print the peak/mean summary, and shut down NVML (idempotent).

launch_memory_monitor(monitor_interval=1.0, output_dir=None)

Launch a GPU/CPU memory monitor in a background thread.

Never raises: any failure is logged and None is returned so an opt-in diagnostic cannot take down the host workload.

Parameters:

  • monitor_interval (float) – Seconds between samples.

  • output_dir (str | None) – If set, write mem_trace.csv and mem_peak.txt here.

Returns:

The running monitor, or None if it could not be launched.

Return type:

GPUMemoryMonitor | None