NCCL#

Measure collective communication across your selected GPUs and nodes.

Start with the Slurm setup. The starter uses two nodes for all-reduce.

uv run cloudai run --system-config slurm.toml \
    --test-scenario conf/experimental/quickstart/nccl.toml

Inspect execution status and measurements using Read results and generate reports.

Run on Kubernetes#

Prepare kubernetes.toml using Run on Kubernetes and install the Kubeflow MPI Operator with the kubeflow.org/v2beta1 API. The namespace must permit the privileged launcher and worker pods used by this integration.

Reuse the two-worker all-reduce scenario with the Kubernetes system:

uv run cloudai run --system-config kubernetes.toml \
    --test-scenario conf/experimental/quickstart/nccl.toml

num_nodes = 2 creates two MPI worker pods. Set gpus_per_node in the system config to the GPUs per worker; CloudAI launches one MPI process per GPU. Kubernetes controls pod placement.

The example uses the public NVIDIA PyTorch image, which includes all_reduce_perf_mpi. Workers install an SSH server if needed, so they require package repository access in that case. For RDMA device requests and network attachments, follow Use Network Operator attachments.

Inspect deployment.yaml for the MPIJob and stdout.txt for the NCCL measurements in the case output directory.

Adapt the workload#

This workload (test_template_name is NcclTest) allows users to execute NCCL benchmarks within the CloudAI framework.

Usage Example#

Test TOML example:

name = "my_nccl_test"
description = "Example NCCL test"
test_template_name = "NcclTest"

[cmd_args]
docker_image_url = "nvcr.io#nvidia/pytorch:25.06-py3"

Test Scenario example:

name = "nccl-test"

[[Tests]]
id = "nccl.1"
num_nodes = 1
time_limit = "00:05:00"

test_name = "my_nccl_test"

Test-in-Scenario example:

name = "nccl-test"

[[Tests]]
id = "nccl.1"
num_nodes = 1
time_limit = "00:05:00"

name = "my_nccl_test"
description = "Example NCCL test"
test_template_name = "NcclTest"

  [Tests.cmd_args]
  docker_image_url = "nvcr.io#nvidia/pytorch:25.06-py3"
  subtest_name = "all_reduce_perf_mpi"
  iters = 100

Configuration and API reference#

Command Arguments#

class cloudai.workloads.nccl_test.nccl.NCCLCmdArgs(

*,

docker_image_url: str,

subtest_name: Literal['all_reduce_perf_mpi', 'all_gather_perf_mpi', 'alltoall_perf_mpi', 'alltoallv_perf_mpi', 'broadcast_perf_mpi', 'gather_perf_mpi', 'hypercube_perf_mpi', 'reduce_perf_mpi', 'reduce_scatter_perf_mpi', 'scatter_perf_mpi', 'sendrecv_perf_mpi', 'bisection_perf_mpi', 'all_reduce_perf', 'all_gather_perf', 'alltoall_perf', 'alltoallv_perf', 'broadcast_perf', 'gather_perf', 'hypercube_perf', 'reduce_perf', 'reduce_scatter_perf', 'scatter_perf', 'sendrecv_perf', 'bisection_perf'] | list[Literal['all_reduce_perf_mpi', 'all_gather_perf_mpi', 'alltoall_perf_mpi', 'alltoallv_perf_mpi', 'broadcast_perf_mpi', 'gather_perf_mpi', 'hypercube_perf_mpi', 'reduce_perf_mpi', 'reduce_scatter_perf_mpi', 'scatter_perf_mpi', 'sendrecv_perf_mpi', 'bisection_perf_mpi', 'all_reduce_perf', 'all_gather_perf', 'alltoall_perf', 'alltoallv_perf', 'broadcast_perf', 'gather_perf', 'hypercube_perf', 'reduce_perf', 'reduce_scatter_perf', 'scatter_perf', 'sendrecv_perf', 'bisection_perf']] = 'all_reduce_perf_mpi',

nthreads: int | list[int] = 1,

ngpus: int | list[int] = 1,

minbytes: str | list[str] = '32M',

maxbytes: str | list[str] = '32M',

stepbytes: str | list[str] = '1M',

op: Literal['sum', 'prod', 'min', 'max', 'avg', 'all'] | list[Literal['sum', 'prod', 'min', 'max', 'avg', 'all']] = 'sum',

datatype: Literal['uint8', 'float'] | list[Literal['uint8', 'float']] = 'float',

root: int | list[int] = 0,

iters: int | list[int] = 20,

warmup_iters: int | list[int] = 5,

agg_iters: int | list[int] = 1,

average: int | list[int] = 1,

parallel_init: int | list[int] = 0,

check: int | list[int] = 1,

blocking: int | list[int] = 0,

cudagraph: int | list[int] = 0,

stepfactor: int | list[int] | None = None,

use_deepep_matrix: bool = False,

alltoallv_matrix_container_path: str = '/tmp/traffic_matrix.txt',

min_busbw: float | None = None,

**extra_data: Any,

)[source]#

Bases: CmdArgs

NCCL test command arguments.

Test Definition#

class cloudai.workloads.nccl_test.nccl.NCCLTestDefinition(*, name: str, description: str, test_template_name: str, cmd_args: ~cloudai.workloads.nccl_test.nccl.NCCLCmdArgs, dse_excluded_args: list[str] = <factory>, extra_env_vars: dict[str, str | ~typing.List[str]] = {}, extra_cmd_args: dict[str, str] = {}, extra_container_mounts: list[str] = [], git_repos: list[~cloudai._core.installables.git_repo.GitRepo] = [], nsys: ~cloudai.models.workload.NsysConfiguration | None = None, predictor: ~cloudai.models.workload.PredictorConfig | None = None, training_report: ~cloudai.models.workload.TrainingReportConfig | None = None, agent: str = 'grid_search', agent_steps: int = 1, agent_metrics: list[str] = ['default'], agent_reward_function: str = 'inverse', agent_config: dict[str, ~typing.Any] | None = None, env_params: dict[str, ~cloudai.configurator.env_params.EnvParamSpec] = <factory>)[source]#

Bases: TestDefinition

Test object for NCCL.

metric_observations( system: System, tr: TestRun, ) → list[MetricObservation][source]#: Return canonical metric observations produced by this test run, if supported.

property is_domain_randomization_enabled: bool#

at least one env_params annotation.

Type:: Whether the config declares domain randomization

is_dse_excluded_arg(path: str) → bool#: Return whether a dot-separated cmd_args path should be ignored by DSE.

is_env_sampled(cmd_args_path: str) → bool#: Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).

validator validate_env_params » all fields#

Validate env_params annotations against cmd_args.

env_params is an annotation: each key names a cmd_args field whose value is the candidate set (the single source of truth), and the entry carries only how to sample. So each key must name a real cmd_args field whose value is a candidate list; a scalar is already fixed, so annotating it is a meaningless label and is rejected here. When weights are declared, the list needs >= 2 values and the weights must align 1:1 with it. Sampling, persistence, the per-trial cmd_args overlay, and the cache key all live in CloudAIGymEnv; keeping this shape check in core lets the overlay stay agent- and workload-agnostic rather than re-implemented per workload.