NIXL Bench#

This workload (test_template_name is NIXLBench) runs NIXL benchmarking suite for network and interconnect performance testing.

Usage Examples#

Test TOML example:

name = "my_nixl_bench_test"
description = "Example NIXL Bench test"
test_template_name = "NIXLBench"

[cmd_args]
docker_image_url = "<docker container url here>"
path_to_benchmark = "/workspace/nixlbench/build/nixlbench"
backend = "UCX"
initiator_seg_type = "VRAM"
target_seg_type = "VRAM"
op_type = "READ"
filepath = "/data"
device_list = "11:F:/store0.bin"
# one could also use <num>kb, <num>mb, <num>gb shortcuts
total_buffer_size = 8000000000

Test Scenario example:

name = "nixl-bench-test"

[[Tests]]
id = "bench.1"
num_nodes = 1
time_limit = "00:10:00"

test_name = "my_nixl_bench_test"

Test-in-Scenario example:

name = "nixl-bench-test"

[[Tests]]
id = "bench.1"
num_nodes = 1
time_limit = "00:10:00"

name = "my_nixl_bench_test"
description = "Example NIXL Bench test"
test_template_name = "NIXLBench"

  [Tests.cmd_args]
  docker_image_url = "<docker container url here>"
  path_to_benchmark = "/workspace/nixlbench/build/nixlbench"
  backend = "UCX"
  initiator_seg_type = "DRAM"
  target_seg_type = "DRAM"
  op_type = "WRITE"

Runtime Coordination#

NIXLBench uses ETCD by default. CloudAI starts ETCD from the benchmark image when etcd_image_url is omitted, or from the configured image.

To use NIXLBench’s direct two-process ASIO runtime instead, set:

runtime_type = "ASIO"

CloudAI resolves asio_address to the first allocated node by default and does not install or start ETCD. asio_address and asio_port can be overridden explicitly. ASIO requires exactly two NIXLBench processes. For UCX, a one-node test runs both processes locally, while a two-node test runs one process on each node.

Storage backends can run without either runtime by using an empty ETCD endpoint:

backend = "POSIX"
etcd_endpoints = ""

This null-runtime mode is limited to storage backends and launches one NIXLBench process by default.

Independent Storage Processes#

To run independent storage benchmarks in one allocation, set launch_mode = "independent" and etcd_endpoints = "". CloudAI launches one process per node by default; use num_nodes to choose the number of nodes:

name = "storage-scaling"

[[Tests]]
id = "storage.4nodes"
test_name = "my_storage_test"
num_nodes = 4

  [Tests.cmd_args]
  launch_mode = "independent"
  etcd_endpoints = ""
  num_threads = [4, 8]

Here, my_storage_test is a NIXLBench test defining the storage backend, container image, benchmark path and workload arguments. CloudAI runs the 4-thread and 8-thread configurations as separate tests.

launch_mode selects CloudAI’s launch strategy and is not passed to NIXLBench. Its default value, "default", preserves existing behavior, including when task placement options are present. Placement options alone do not enable independent runs.

Independent mode requires etcd_endpoints = "" and the default runtime_type = "ETCD" setting, which together select NIXLBench’s null runtime. Managed/external ETCD and ASIO cannot be used with this mode. CloudAI launches one Slurm step across the requested nodes. Use extra_srun_args only when overriding the default task placement. System-level options precede test-level options.

Each task executes in its own shell. Node-specific environment values and command expansions are evaluated there. CloudAI does not discover backend devices.

Results and Reporting#

Independent runs retain nixlbench/<task-id>.stdout, .stderr, .status and .hostname files. The nixlbench/ntasks file records the actual Slurm task count. A failed process, missing task output or inconsistent block-size/batch-size coverage makes the run unsuccessful; incomplete results are not aggregated.

nixlbench_per_task.csv preserves individual measurements with task IDs and hostnames. nixlbench.csv summarizes matching block sizes and batch sizes: avg_lat and bw_gb_sec are arithmetic means across tasks, bw_min_gb_sec is the minimum task bandwidth, bw_sum_gb_sec is the sum, and task_count records the contributors. The native HTML report shows mean latency and minimum/mean/summed bandwidth, with separate plots for each batch size. Comparison reports and metric observations use the mean per-task bandwidth and latency. Existing single-process reports are unchanged.

API Documentation#

Command Arguments#

pydantic model cloudai.workloads.nixl_bench.nixl_bench.NIXLBenchCmdArgs[source]#

Command line arguments for a NIXL Bench test.

field path_to_benchmark: str [Required]#

field launch_mode: Literal['default', 'independent'] = 'default'#: CloudAI launch mode: ‘default’ preserves existing orchestration; ‘independent’ launches one null-runtime task per node in one Slurm step by default. Use num_nodes to select the node count and extra_srun_args to override task placement. Not passed to NIXLBench.

field etcd_endpoints: str = 'http://$NIXL_ETCD_ENDPOINTS'#

field runtime_type: Literal['ETCD', 'ASIO'] = 'ETCD'#

field asio_address: str = '$NIXL_ASIO_ADDRESS'#

field asio_port: int = 12345#

Constraints:

ge = 1
le = 65535

field docker_image_url: str [Required]#: URL of the Docker image to use for the benchmark.

field etcd_path: str = 'etcd'#: Path to the etcd executable.

field wait_etcd_for: int = 60#: Number of seconds to wait for etcd to become healthy.

field etcd_image_url: str | None = None#: Optional URL of the Docker image to use for etcd, by default etcd will be run from the same image as the benchmark.

field filepath: str | None = None#: Directory path (in container) for storage operations. Example: /data

field total_buffer_size: str | list[str] | None = None#: Total buffer size in bytes. Examples: 1024, 1kb, 1mb, 1gb. Use with device_list. The size will be passed into NIXL as integer (bytes)

field device_list: str | list[str] | None = None#: Device specs in format ‘id:type:path’ (e.g., ‘11:F:/store0.bin,27:K:/dev/nvme0n1’)

Test Definition#

class cloudai.workloads.nixl_bench.nixl_bench.NIXLBenchTestDefinition(*, name: str, description: str, test_template_name: str, cmd_args: ~cloudai.workloads.nixl_bench.nixl_bench.NIXLBenchCmdArgs, dse_excluded_args: list[str] = <factory>, extra_env_vars: dict[str, str | ~typing.List[str]] = {}, extra_cmd_args: dict[str, str] = {}, extra_container_mounts: list[str] = [], git_repos: list[~cloudai._core.installables.git_repo.GitRepo] = [], nsys: ~cloudai.models.workload.NsysConfiguration | None = None, predictor: ~cloudai.models.workload.PredictorConfig | None = None, training_report: ~cloudai.models.workload.TrainingReportConfig | None = None, agent: str = 'grid_search', agent_steps: int = 1, agent_metrics: list[str] = ['default'], agent_reward_function: str = 'inverse', agent_config: dict[str, ~typing.Any] | None = None, env_params: dict[str, ~cloudai.configurator.env_params.EnvParamSpec] = <factory>)[source]#

Bases: NIXLBaseTestDefinition[NIXLBenchCmdArgs]

Test definition for a NIXL Bench test.

property uses_etcd: bool#: Return whether CloudAI should launch ETCD for this benchmark.

property uses_asio: bool#: Return whether this benchmark uses NIXLBench’s ASIO runtime.

metric_observations( system: System, tr: TestRun, ) → list[MetricObservation][source]#: Return canonical metric observations produced by this test run, if supported.

property is_domain_randomization_enabled: bool#

at least one env_params annotation.

Type:: Whether the config declares domain randomization

is_dse_excluded_arg(path: str) → bool#: Return whether a dot-separated cmd_args path should be ignored by DSE.

is_env_sampled(cmd_args_path: str) → bool#: Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).

validator validate_env_params » all fields#

Validate env_params annotations against cmd_args.

env_params is an annotation: each key names a cmd_args field whose value is the candidate set (the single source of truth), and the entry carries only how to sample. So each key must name a real cmd_args field whose value is a candidate list; a scalar is already fixed, so annotating it is a meaningless label and is rejected here. When weights are declared, the list needs >= 2 values and the weights must align 1:1 with it. Sampling, persistence, the per-trial cmd_args overlay, and the cache key all live in CloudAIGymEnv; keeping this shape check in core lets the overlay stay agent- and workload-agnostic rather than re-implemented per workload.