Read results and generate reports#

Open the results directory printed by CloudAI. Start with execution status, then inspect the measurements needed for your decision.

Find the evidence#

  • experiment.json records cases, execution status, metrics, and selected search results.

  • stdout.txt and stderr.txt explain what happened within a workload execution.

  • Workload and scenario reports present measurements for inspection and comparison.

Compare only results with compatible model, workload, resources, input sizes, units, and measurement conditions. For a failed or incomplete run, follow Troubleshooting before interpreting its measurements. Use CLI reference to regenerate reports from saved output, Pareto Frontiers for trade-offs, and Publish results and integrate with CI to publish artifacts.

Overview

CloudAI has two reporting levels:

  • per-test (per each case in a test scenario)

  • per-scenario (per each test scenario)

All reports are generated after the test scenario is completed as part of the main CloudAI process. For Slurm, this means that the login node is used to generate reports.

Per-test reports are linked to a particular workload type (e.g. NcclTest). All per-test reports are implemented as part of the per_test scenario report and can be enabled or disabled via a single configuration option; see Enabling, Disabling and Configuring Reports.

To list all available reports, users can use cloudai list-reports. Use verbose output to also print report configurations.

Unified Experiment Output#

CloudAI writes experiment.json in each scenario’s results directory.

Record

Contents

Scenario

Scenario details, system name, submitting username (user), and tests.

Test case

Execution records in runs and recorded metrics.

Run

Iteration or DSE-step number, process/job ID, status, timing, result path, and workload metrics.

  • Slurm uses the cluster-reported system name.

  • Older files or failed username lookups have an empty user; lookup failures log a warning.

  • Timestamps use UTC; durations use seconds. Unknown timestamps are null.

  • unknown status means the outcome could not be determined.

  • Dry runs record scenario and case details without launching workloads.

Live runs status tracking:

  • Standalone jobs are running from submission.

  • Slurm jobs (excluding single-sbatch) start as pending while waiting for allocation, then transition to running once allocated and executing.

  • Single-sbatch Slurm run records remain pending until the allocation finishes and their final statuses are recorded.

Metrics are sourced from TestDefinition.metric_observations(), and are not affected by reporter settings.

MegatronRun metric

Unit

Direction

iteration_time

Milliseconds

Lower is better.

tflops_per_gpu

TFLOP/s/GPU

Higher is better.

Measurements use the mean of the last ten iteration metric lines in stdout.txt, or all lines if fewer exist. Missing iteration lines produce no observations. This matches the per-test CSV report.

Result field

Regular execution

Configuration search

tests[].metrics

Metrics from the successful execution.

Metrics from the successful step with the highest valid reward.

tests[].dse

Not a configuration search.

Search space, selected step, and configuration.

NCCL DSE Example#

This example comes from a Slurm NCCL all-reduce run on one node with eight H100 GPUs. DSE tried Ring and Tree in two steps. Ring (step 1) won using inverse latency as the reward, so its measurements appear in tests[].metrics.

Identifiers and paths are anonymized. To keep the example small, each metrics list includes only out-of-place latency and bus bandwidth for 1 MiB messages. Measurement values, timing, and DSE selection are unchanged.

Download experiment.json.

{
  "id": "nccl-dse_2026-09-21_14-55-45",
  "name": "nccl-dse",
  "system_name": "example-cluster",
  "user": "example-user",
  "description": null,
  "status": "completed",
  "path": "/results/nccl-dse_2026-09-21_14-55-45",
  "start": "2026-09-21T14:55:45.275325Z",
  "finish": "2026-09-21T15:01:50.432934Z",
  "duration": 365.157609,
  "tests": [
    {
      "id": "nccl",
      "name": "nccl_base_test",
      "description": "NCCL base test configuration",
      "status": "completed",
      "path": "/results/nccl-dse_2026-09-21_14-55-45/nccl",
      "metrics": [
        {
          "name": "Latency",
          "value": 24.76,
          "unit": "us",
          "dimensions": [
            {"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
            {"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
            {"name": "Size", "value": "1048576", "unit": "", "is_x": true}
          ]
        },
        {
          "name": "Bandwidth",
          "value": 74.12,
          "unit": "GB/s",
          "dimensions": [
            {"name": "Bandwidth basis", "value": "bus", "unit": "", "is_x": false},
            {"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
            {"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
            {"name": "Size", "value": "1048576", "unit": "", "is_x": true}
          ]
        }
      ],
      "runs": [
        {
          "path": "/results/nccl-dse_2026-09-21_14-55-45/nccl/0/1",
          "jobid": "1001",
          "status": "completed",
          "metrics": [
            {
              "name": "Latency",
              "value": 24.76,
              "unit": "us",
              "dimensions": [
                {"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
                {"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
                {"name": "Size", "value": "1048576", "unit": "", "is_x": true}
              ]
            },
            {
              "name": "Bandwidth",
              "value": 74.12,
              "unit": "GB/s",
              "dimensions": [
                {"name": "Bandwidth basis", "value": "bus", "unit": "", "is_x": false},
                {"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
                {"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
                {"name": "Size", "value": "1048576", "unit": "", "is_x": true}
              ]
            }
          ],
          "start": "2026-09-21T14:55:46Z",
          "finish": "2026-09-21T14:57:28Z",
          "duration": 102.0,
          "iteration": 0,
          "step": 1
        },
        {
          "path": "/results/nccl-dse_2026-09-21_14-55-45/nccl/0/2",
          "jobid": "1002",
          "status": "completed",
          "metrics": [
            {
              "name": "Latency",
              "value": 37.36,
              "unit": "us",
              "dimensions": [
                {"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
                {"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
                {"name": "Size", "value": "1048576", "unit": "", "is_x": true}
              ]
            },
            {
              "name": "Bandwidth",
              "value": 49.11,
              "unit": "GB/s",
              "dimensions": [
                {"name": "Bandwidth basis", "value": "bus", "unit": "", "is_x": false},
                {"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
                {"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
                {"name": "Size", "value": "1048576", "unit": "", "is_x": true}
              ]
            }
          ],
          "start": "2026-09-21T14:58:47Z",
          "finish": "2026-09-21T15:00:28Z",
          "duration": 101.0,
          "iteration": 0,
          "step": 2
        }
      ],
      "dse": {
        "space": {
          "extra_env_vars.NCCL_ALGO": ["Ring", "Tree"]
        },
        "best_config": {"extra_env_vars.NCCL_ALGO": "Ring"},
        "best_step": 1
      }
    }
  ]
}

Each file update replaces the previous snapshot atomically. Metric extraction or write errors produce warnings without affecting execution.

General Flow#

  • All reports should be registered via Registry() (.add_report() or .add_scenario_report())

  • Scenario reports are configurable via system config (Slurm-only for now) and scenario config

  • Configuration in a scenario config has the highest priority. Next, system config is checked. Then it defaults to report config from the registry

  • Finally, the report is generated (or not) according to this final config

Enabling, Disabling and Configuring Reports#

Note

Only scenario-level reports can be configured.

Enabling or disabling a report needs to be done in the system configuration:

[reports]
per_test = { enable = false }
status = { enable = true }
junit = { enable = true }

The junit reporter is disabled by default. When enabled:

  • Writes junit.xml in the scenario results directory.

  • Emits a test case for every regular iteration and DSE step.

  • Includes pass/fail status, failure details, available scheduler duration, and stdout/stderr.

Use the artifact with CI systems that accept JUnit XML, following Publish results and integrate with CI.

Speed-of-Light comparisons#

CloudAI can compare NCCL and NIXL Bench measurements with configured Speed-of-Light (SOL) targets. Targets are validated while the system or scenario TOML is parsed. A target without match is the default; a matching target with more dimensions takes precedence:

[[sol.bandwidth]]
value = 100.0 # GB/s

[[sol.bandwidth]]
value = 120.0 # GB/s
match = { operation = "write", size_bytes = 1048576 }

[[sol.latency]]
value = 8.0 # us

NCCL targets can distinguish the collective operation and placement:

[[sol.bandwidth]]
value = 250.0 # GB/s
match = { operation = "all_reduce", placement = "out_of_place", bandwidth_basis = "bus" }

[[sol.bandwidth]]
value = 300.0 # GB/s
match = { operation = "all_reduce", placement = "in_place", bandwidth_basis = "bus" }

A test case can replace the targets for a metric inherited from the scenario or system:

[[Tests]]
id = "nixl-case"
test_name = "nixl-bench"

[[Tests.sol.bandwidth]]
value = 120.0

The precedence is test case, then scenario, then system. NCCL and NIXL comparison v2 reports include measured, SOL, and percentage-of-SOL columns and draw a shared SOL curve when every compared run resolves the same targets.