AI Dynamo#

Measure serving performance with distributed prefill and decode workers using test_template_name = "AIDynamo". Choose Kubernetes or Slurm below, then inspect measurements using Read results and generate reports.

Run on Kubernetes#

Prepare kubernetes.toml using Run on Kubernetes, then install the Dynamo operator and platform services for your runtime version. The operator must serve nvidia.com/v1alpha1 DynamoGraphDeployment resources.

The supplied example uses:

The public nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.1 image and Qwen/Qwen3-0.6B model.
One prefill worker and one decode worker, each with eight GPUs; set gpus_per_node = 8 in the system config.
GenAI-Perf to send requests and collect inference measurements.

Use a dedicated namespace for the experiment. CloudAI names the deployment after default_namespace; run one Dynamo experiment at a time in that namespace. For accelerated transfers between workers, configure Network Operator attachments.

uv run cloudai run --system-config kubernetes.toml \
    --tests-dir conf/experimental/ai_dynamo/test \
    --test-scenario conf/experimental/ai_dynamo/test_scenario/vllm_k8s.toml

To change worker sizing, copy the scenario and edit each worker’s num-nodes and tensor-parallel-size. Keep the tensor parallel size consistent with the GPUs available to that worker. Workers spanning multiple nodes also need Dynamo’s multinode orchestration components on the cluster.

The Kubernetes path uses cmd_args.genai_perf settings from the vLLM test definition. Inspect deployment.yaml, genai_perf.log, and the collected GenAI-Perf artifacts in the case output directory.

Run Using Slurm#

Node Configuration for AI Dynamo#

AI Dynamo jobs use three distinct types of nodes:

Frontend node: Hosts the coordination services (etcd, nats), the frontend server, the request generator (aiperf by default, configurable via workloads in the test TOML), and the first decode worker
Prefill node(s): Handle the prefill stage of inference
Decode node(s): Handle the decode stage of inference (optional, depending on model and setup)

By default, when num_nodes is omitted, CloudAI allocates separate nodes for prefill and decode workers:

num_prefill_nodes + num_decode_nodes

Set top-level num_nodes explicitly to control the Slurm allocation. A value lower than num_prefill_nodes + num_decode_nodes enables shared-node disaggregated inference, where prefill and decode roles run on the same allocated node(s) with separate GPU slices.

All node role assignments and orchestration are automatically managed by CloudAI.

Multinode Backend Workers on Slurm#

prefill_worker.num-nodes and decode_worker.num-nodes specify the total nodes assigned to each role. Set nodes-per-worker to group those nodes into logical backend workers. For example, the following configuration creates one prefill worker spanning two nodes and one decode worker spanning two other nodes:

[cmd_args.dynamo.prefill_worker]
num-nodes = 2
nodes-per-worker = 2
  [cmd_args.dynamo.prefill_worker.args]
  tensor-parallel-size = 8
  pipeline-parallel-size = 1

[cmd_args.dynamo.decode_worker]
num-nodes = 2
nodes-per-worker = 2
  [cmd_args.dynamo.decode_worker.args]
  tensor-parallel-size = 8
  pipeline-parallel-size = 1

When nodes-per-worker is omitted, it defaults to 1 and preserves the previous one-worker-per-node behavior.

For vLLM TP/PP groups, CloudAI launches rank 0 as the Dynamo-facing process and the remaining ranks with --headless using Dynamo’s multiprocessing executor.
When data-parallel-size is greater than one, CloudAI instead launches a full Dynamo vLLM process on every node and derives data-parallel-size-local, data-parallel-start-rank, the coordinator address, and the RPC port from the worker topology.
SGLang receives --dist-init-addr, --nnodes, and --node-rank on every participating node; its configured data-parallel-size is forwarded as dp-size.
The launcher derives prefill and decode disaggregation modes from the configured roles, so worker commands do not need role flags.

For TP/PP-spanning workers, tensor-parallel-size * pipeline-parallel-size must be divisible by nodes-per-worker.

For vLLM multinode data parallelism, data-parallel-size must instead be divisible by nodes-per-worker, and each DP rank’s TP/PP group must fit within one node.
SGLang multinode data parallelism requires --enable-dp-attention.
The resulting per-node GPU footprint must fit on each node.
Ray-backed Dynamo vLLM groups are not currently supported.
The Slurm launcher explicitly selects etcd discovery and the NATS event plane.

No separate WideEP mode is required.

Multinode data-parallel orchestration is inferred from nodes-per-worker > 1 and data-parallel-size > 1.
Backend expert-parallel parameters such as vLLM’s --enable-expert-parallel or SGLang’s ep-size and --enable-dp-attention remain explicit backend configuration.
CloudAI uses the allocated Slurm node names for multinode rendezvous.
The cluster must make those names consistently resolvable and mutually reachable from every allocated node and workload container.
Multi-homed systems should also configure backend network selection, such as GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME, at the system level.
Explicit backend rendezvous addresses take precedence over addresses derived from the Slurm node names.
Clusters that cannot provide this hostname contract must supply backend-native per-node address overrides, such as SGLANG_HOST_IP or VLLM_NIXL_SIDE_CHANNEL_HOST, together with explicit worker rendezvous arguments.

Dedicated examples are available in test_scenario/vllm_multinode_worker_slurm.toml and test_scenario/sglang_multinode_worker_slurm.toml. The existing vllm_slurm.toml and sglang_slurm.toml scenarios retain their previous configurations. The vLLM example inherits tensor-parallel-size = 8 from test/vllm.toml; the SGLang example sets it explicitly because test/sglang.toml defaults to 1. WideEP examples are provided in test_scenario/vllm_wideep_slurm.toml and test_scenario/sglang_wideep_slurm.toml.

Launch and Monitor the Job#

To run the job:

uv run cloudai run --system-config <slurm system toml> \
   --tests-dir conf/experimental/ai_dynamo/test \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/vllm_slurm.toml

The job progress monitoring can be done using either of the following options:

watch squeue --me

watch tail -n 4 ./results/<scenario name>/*.txt

The frontend node will initially wait to allow weight loading on all nodes. Once ready, it will launch the configured benchmark tool (aiperf by default), which begins generating requests to the frontend server. All servers cooperate to complete inference, and the output will appear in stdout.txt.

Choosing a Benchmark Tool#

The benchmark tool is controlled by the workloads field in the test TOML. Set aiperf.sh to use AIPerf:

[cmd_args]
workloads = "aiperf.sh"   # uses aiperf, writes aiperf_report.csv

To use genai-perf, set:

[cmd_args]
workloads = "genai_perf.sh"   # uses genai-perf, writes genai_perf_report.csv

[cmd_args.genai_perf]
cmd = "genai-perf profile"
extra-args = "--streaming --verbose -- -v --async"

  [cmd_args.genai_perf.args]
  endpoint-type = "chat"
  output-tokens-mean = 500
  request-count = 50

AIPerf Multi-Phase Runs#

cmd_args.aiperf is the base AIPerf config. cmd_args.aiperf_phases can run several AIPerf rounds against the same live Dynamo stack. By default, CloudAI does not restart prefill, decode, or router processes between phases:

dse_excluded_args = ["cmd_args.aiperf_phases"]

[cmd_args.aiperf]
health-check-between-phases = true
between-phase-cmd = "true"  # default no-op

  [cmd_args.aiperf.args]
  request-count = 50
  server-metrics = "auto"

[[cmd_args.aiperf_phases]]
name = "round_1"
  [cmd_args.aiperf_phases.args]
  concurrency = 2

[[cmd_args.aiperf_phases]]
name = "round_2"
  [cmd_args.aiperf_phases.args]
  concurrency = 4

Single-phase runs keep the old artifact layout: aiperf_artifacts/, aiperf.log, and aiperf_report.csv. Multi-phase runs write per-phase artifacts/logs/reports and copy the last phase report to aiperf_report.csv for existing report generation.

between-phase-cmd is a bash command run after each non-final phase. The default is a no-op. Set it explicitly for backend-specific cache cleanup, for example /cloudai_run_results/routerctl.sh restart if a test needs to restart the Dynamo router between phases. health-check-between-phases probes the frontend after the command.

Comparison Reports#

CloudAI provides the scenario-level ai_dynamo_comparison report. It compares AI Dynamo runs using the standard LLM serving tables and charts for TTFT, TPOT, successful prompts, output-token throughput, TPS per user, and TPS per GPU. When aiperf_accuracy is configured and its result is available, the report also compares model accuracy. Both AIPerf and GenAI-Perf CSV output are supported.

The scenario-level Pareto report uses each configured AIPerf phase as a separate point, with that phase’s concurrency and benchmark settings. It also supports GenAI-Perf results.

AIPerf args are rendered as normal CLI flags. Multi-value AIPerf options should be passed with AIPerf CLI syntax, such as server-metrics-formats = "csv,json,jsonl" or gpu-telemetry = "node1:9401,node2:9401". server-metrics = "auto" expands to the frontend metrics endpoint, Dynamo worker metrics endpoints, and any CloudAI-started DCGM exporters.

Propagating LMCache Configuration#

AIDynamo can pass an LMCache YAML config to the worker processes by setting LMCACHE_CONFIG_FILE inside the container. This only propagates the LMCache configuration; the vLLM/SGLang runtime still needs to be launched with the appropriate LMCache or KV-transfer connector for that image/version.

The preferred form is structured TOML under [cmd_args.lmcache]. CloudAI converts that object to YAML in the run output directory, mounts that directory as /cloudai_run_results, and exports the generated file path as LMCACHE_CONFIG_FILE:

[cmd_args]
  [cmd_args.lmcache]
  chunk_size = 256
  local_cpu = true
  controller_pull_url = "{frontend_node}:8300"
  controller_reply_url = "{frontend_node}:8400"
  lmcache_worker_ports = [8788, 8789, 8790, 8791]
  max_local_cpu_size = 6.0
  nixl_buffer_size = 2079377920
  nixl_buffer_device = "cpu"

    [cmd_args.lmcache.extra_config]
    enable_nixl_storage = false
    nixl_backend = "POSIX"
    nixl_path = "{storage_cache_dir}"
    nixl_pool_size = 2048

For an example that uses test-in-scenario mode, see conf/experimental/ai_dynamo/test_scenario/vllm_lmcache.toml. Because the test is fully defined inside the scenario, --tests-dir is not required when running that example:

uv run cloudai run --system-config <slurm system toml> \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/vllm_lmcache.toml

The example sets dse_excluded_args = ["cmd_args.lmcache.lmcache_worker_ports"] because lmcache_worker_ports is a list-valued LMCache setting, not a DSE sweep dimension. Other list-valued LMCache fields can still be swept unless their cmd_args. path is also excluded.

Alternatively, mount your own LMCache YAML file with extra_container_mounts and set LMCACHE_CONFIG_FILE through extra_env_vars:

extra_container_mounts = ["/host/lmcache:/lmcache"]
extra_env_vars = { LMCACHE_CONFIG_FILE = "/lmcache/config.yaml" }

For multi-node LMCache storage tests, any path referenced by the LMCache YAML, such as nixl_path for POSIX-backed storage, must be visible and writable from every node that is expected to share cached data. A node-local path such as /tmp is suitable only for single-node smoke tests or configuration propagation checks.

LMCache YAML values can use runtime placeholders. CloudAI renders them inside the Slurm job before launching workers: {frontend_node}, {frontend_ip}, {results_dir}, and {storage_cache_dir}. Unknown placeholders fail the run before worker processes start.

Propagating HiCache Configuration#

Define a HiCache storage-backend configuration under [cmd_args.hicache]. CloudAI serializes the object as hicache-config.toml in the run output directory, exposes its container path through HICACHE_CONFIG_FILE, and passes the resolved path to both workers as --hicache-storage-backend-extra-config:

[cmd_args]
  [cmd_args.hicache.plugin.posix]
  active = true
  use_uring = "false"

  [cmd_args.dynamo.prefill_worker]
  extra-args = "--enable-hierarchical-cache"

  [cmd_args.dynamo.prefill_worker.args]
  hicache-size = 8
  hicache-ratio = 0
  hicache-write-policy = "write_through"
  hicache-mem-layout = "page_first"
  hicache-storage-prefetch-policy = "wait_complete"
  hicache-storage-backend = "nixl"

  [cmd_args.dynamo.decode_worker]
  extra-args = "--enable-hierarchical-cache"

  [cmd_args.dynamo.decode_worker.args]
  hicache-size = 8
  hicache-ratio = 0
  hicache-write-policy = "write_through"
  hicache-mem-layout = "page_first"
  hicache-storage-prefetch-policy = "wait_complete"
  hicache-storage-backend = "nixl"

The remaining HiCache engine options stay in the worker args sections so they can be configured or swept through DSE normally.

The runtime replacement mechanism below supports per-node HiCache values. For example, change the POSIX setting above to use_uring = "${use_uring}" and have the startup manifest replace ${use_uring} with false or true. The worker receives the resulting per-node TOML path automatically.

For a self-contained single-node smoke test using NIXL’s POSIX plugin, run the following on a node with at least two GPUs (one each for the prefill and decode workers):

uv run cloudai run --system-config <slurm system toml> \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/sglang_hicache.toml

Per-node Startup Commands and Runtime Replacements#

On Slurm, startup_cmd runs once per allocated node before the workload. It runs on the host by default. Set startup_cmd_docker_image to run it in a container using the workload’s mounts:

[cmd_args]
startup_cmd = "/mnt/discovery/discover-device.sh"
startup_cmd_docker_image = "<image-containing-your-startup-tools>"

CloudAI exports CLOUDAI_RUNTIME_FILE to the command. It points to runtime/<node>.json in the test output directory, or /cloudai_run_results/runtime/<node>.json inside a startup container. Write replacements there:

{
  "LMCACHE_CONFIG_FILE": {
    "${device}": "/dev/ng4n1",
    "${namespace_id}": "1"
  },
  "HICACHE_CONFIG_FILE": {
    "${use_uring}": "false"
  }
}

Each top-level key names an environment variable containing a config path. AIDynamo creates a per-node copy, applies literal replacements, then updates the variable to point to that copy.

For lmcache-config.yaml on node-01, the result is lmcache-config.node-01.yaml. The shared source remains unchanged. LMCache placeholders are rendered afterward in the per-node copy. Missing manifest means no replacements.

Startup runs once per test case in normal and --single-sbatch modes.

If the selected LMCache mode needs a controller, CloudAI can start one on the frontend node:

[cmd_args.lmcache_controller]
cmd = "lmcache_controller --host 0.0.0.0 --port 9000 --monitor-ports {\"pull\":8300,\"reply\":8400}"

This only launches the process. For disaggregated or multi-node runs, the LMCache YAML still needs controller addresses that resolve to the frontend node from every worker. With the default controller monitor ports, use controller_pull_url = "{frontend_node}:8300" and controller_reply_url = "{frontend_node}:8400". The lmcache_worker_ports list must match the number of worker ranks.

Semantic Degradation With AIPerf Accuracy#

AIDynamo uses AIPerf accuracy mode as its semantic degradation signal. Enable it with [cmd_args.aiperf_accuracy]. This runs after the configured performance workload, so it can be used with either aiperf.sh or genai_perf.sh:

[cmd_args]
workloads = "aiperf.sh"

[cmd_args.aiperf]
  [cmd_args.aiperf.args]
  request-count = 50
  synthetic-input-tokens-mean = 300
  output-tokens-mean = 500
  concurrency = 2

[cmd_args.aiperf_accuracy]
entrypoint = "aiperf profile"
setup-cmd = "python -m pip install --break-system-packages --upgrade aiperf==0.8.0"
cli = '''
--model {model}
--url {url}
--endpoint-type chat
--streaming
--artifact-dir {artifact_dir}
--no-server-metrics
--accuracy-benchmark mmlu
--accuracy-n-shots 5
--accuracy-tasks abstract_algebra
--concurrency 10
--extra-inputs '{"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}'
--num-requests 100
'''

When cmd_args.aiperf_accuracy is configured, CloudAI expects AIPerf to produce accuracy_results.csv and exposes the accuracy metric from its OVERALL row. The metric is reported as a 0.0-1.0 fraction. Keep synthetic prompt and token-length flags out of this mode; the benchmark dataset should come from AIPerf’s accuracy benchmark.

The entrypoint and cli fields form the accuracy command.

CloudAI expands {model}, {url}, {endpoint}, {result_dir}, and {artifact_dir} in cli before launching it.
The setup-cmd field is optional.
It is useful for Dynamo images that include an older system aiperf build without the accuracy benchmark plugins.
The example upgrades the image-level aiperf before launching aiperf profile.
MMLU is loaded from lighteval/mmlu, so either allow Hugging Face dataset access or pre-cache that dataset before running with HF_HUB_OFFLINE/HF_DATASETS_OFFLINE enabled.
For Qwen3 models, the example disables thinking mode so short MMLU answers can be parsed as choices.

Custom Accuracy Scripts#

cmd_args.aiperf_accuracy can also launch a custom mounted script instead of AIPerf. Mount the script or its parent directory with extra_container_mounts and set entrypoint to the in-container command:

extra_container_mounts = ["/host/custom_accuracy:/custom_accuracy"]

[cmd_args.aiperf_accuracy]
entrypoint = "python /custom_accuracy/dummy_accuracy.py"
cli = "--model {model} --url {url} --endpoint {endpoint} --artifact-dir {artifact_dir} --prompt ping"

CloudAI expands placeholders in cli and runs entrypoint with that CLI string. The custom command must write accuracy_results.csv inside {artifact_dir} with an OVERALL row. CloudAI copies that file to the run output directory and exposes the same accuracy metric as AIPerf accuracy mode.

Review Benchmark Results#

After job completion, CloudAI places output logs and result files in the designated results directory. The result file name depends on the configured workloads field:

aiperf.sh → aiperf_report.csv
genai_perf.sh → genai_perf_report.csv
cmd_args.aiperf_accuracy → accuracy_results.csv

If AIPerf accuracy mode is enabled, CloudAI copies aiperf_accuracy_artifacts/accuracy_results.csv to accuracy_results.csv in the run output directory and marks the run failed if that file is not produced.

Navigate to ./results/<scenario>/<test-id>/0/ and open the CSV to examine performance metrics.

Shared-Node Disaggregated Runs#

For Slurm, set top-level num_nodes lower than the sum of prefill_worker.num-nodes and decode_worker.num-nodes to run both roles on the same allocated node(s).

For example, num_nodes = 1 with prefill_worker.num-nodes = 1 and decode_worker.num-nodes = 1 runs one prefill worker and one decode worker on the same node.
CloudAI assigns decode GPUs first and prefill GPUs after that based on each role’s per-node GPU footprint, including local data-parallel ranks where applicable.
The combined role GPU count must fit on one node.

Example aiperf_report.csv:

Metric,avg,min,max,p25,p50,p75,p99,std
Inter Token Latency (ms),2.81,2.66,2.88,2.79,2.83,2.84,2.87,0.04
Time to First Token (ms),49.87,17.15,99.91,49.35,49.87,50.52,92.31,9.20
Time to Second Token (ms),0.50,0.03,4.05,0.03,0.04,0.04,3.47,1.08
Request Latency (ms),1652.30,1203.61,6433.87,1453.19,1462.99,1466.72,6431.16,976.18
Output Sequence Length (tokens),498.06,410.00,501.00,500.00,500.00,500.00,501.00,12.62
Input Sequence Length (tokens),300.00,300.00,300.00,300.00,300.00,300.00,300.00,0.00

Metric,Value
Output Token Throughput (tokens/sec),598.78
Total Token Throughput (tokens/sec),962.32
Request Throughput (requests/sec),1.20
Request Count,50.00

Supported Backends#

The following backends are available via the conf/experimental/ai_dynamo/test/ directory:

vLLM (vllm.toml) — use with test_scenario/vllm_slurm.toml
vLLM multinode worker — use test_scenario/vllm_multinode_worker_slurm.toml
vLLM WideEP — use test_scenario/vllm_wideep_slurm.toml
vLLM with LMCache config propagation — use self-contained scenario test_scenario/vllm_lmcache.toml
sglang (sglang.toml) — use with test_scenario/sglang_slurm.toml
SGLang multinode worker — use test_scenario/sglang_multinode_worker_slurm.toml
SGLang WideEP — use test_scenario/sglang_wideep_slurm.toml

Both backends use aiperf as the default benchmark tool and support disaggregated prefill/decode.

API Documentation#

Command Arguments#

class cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoCmdArgs( *, dynamo_version: str = 'f7e468c7e8ff0d1426db987564e60572167e8464', docker_image_url: str, startup_cmd: str | None = None, startup_cmd_docker_image: str | None = None, storage_cache_dir: str | list[str] | None = '/tmp', dynamo: ~cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoArgs, hicache: dict | None = None, lmcache: dict | None = None, lmcache_controller: ~cloudai.workloads.ai_dynamo.ai_dynamo.LMCacheController | None = None, genai_perf: ~cloudai.workloads.ai_dynamo.ai_dynamo.GenAIPerf = <factory>, aiperf: ~cloudai.workloads.ai_dynamo.ai_dynamo.AIPerf = <factory>, aiperf_phases: list[~cloudai.workloads.ai_dynamo.ai_dynamo.AIPerfPhase] | None = None, aiperf_accuracy: ~cloudai.workloads.ai_dynamo.ai_dynamo.AIPerfAccuracy | None = None, workloads: str = 'genai_perf.sh', )[source]#

Bases: CmdArgs

Arguments for AI Dynamo.

resolve_aiperf_phase( phase: AIPerfPhase, ) → AIPerf[source]#: Apply phase overrides using the execution artifact paths.

validator validate_aiperf_phases » all fields[source]#: Validate AIPerf phases.

Test Definition#

class cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoTestDefinition(*, name: str, description: str, test_template_name: str, cmd_args: ~cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoCmdArgs, dse_excluded_args: list[str] = <factory>, extra_env_vars: dict[str, str | ~typing.List[str]] = {}, extra_cmd_args: dict[str, str] = {}, extra_container_mounts: list[str] = [], git_repos: list[~cloudai._core.installables.git_repo.GitRepo] = [], nsys: ~cloudai.models.workload.NsysConfiguration | None = None, predictor: ~cloudai.models.workload.PredictorConfig | None = None, training_report: ~cloudai.models.workload.TrainingReportConfig | None = None, agent: str = 'grid_search', agent_steps: int = 1, agent_metrics: list[str] = ['default'], agent_reward_function: str = 'inverse', agent_config: dict[str, ~typing.Any] | None = None, env_params: dict[str, ~cloudai.configurator.env_params.EnvParamSpec] = <factory>, script: ~cloudai._core.installables.file.File = File(src=PosixPath('/home/runner/work/cloudai/cloudai/src/cloudai/workloads/ai_dynamo/ai_dynamo.sh')), constraints: ~cloudai.workloads.ai_dynamo.ai_dynamo.Constraints = Constraints(prefill_tp_le_decode_tp=True, tp_times_pp_le_gpus_per_node=True), success_marker: str = 'success-marker.txt', failure_marker: str = 'failure-marker.txt')[source]#

Bases: TestDefinition, InferenceReportProvider

Test definition for AI Dynamo.

classmethod parse_aiperf_json( path: Path, max_concurrency: int = 1, ) → AIDynamoBenchReport | None[source]#: Read one phase’s native export; absent statistics remain unavailable.

classmethod parse_benchmark_csv( csv_file: Path, max_concurrency: int = 1, ) → AIDynamoBenchReport | None[source]#: Parse one benchmark or phase CSV without substituting another phase’s output.

validator restore_script_paths » all fields[source]#: Unwrap script paths in saved TOMLs until the shared serialization fix (#5006455).

validator ignore_legacy_repository » all fields[source]#: Allow historical dumps containing the removed repository field.

metric_observations( system: System, tr: TestRun, ) → list[MetricObservation][source]#: Return canonical metric observations produced by this test run, if supported.

serving_gpu_count( roles: tuple[InferenceRoleConfig | None, ...], ) → int | None[source]#: Count configured engine devices, including replicas; attention DP shares TP GPUs.

inference_reports( system: System, tr: TestRun, ) → list[InferenceReport][source]#: Return one report per measurement, or an empty list when no results are available.

validator workload_scripts » all fields[source]#: Populate prefill/decode args.

get_workload_map() → dict[str, Workload][source]#: Get a map of workload scripts to workload objects.

property repo: GitRepo#: Return the AI Dynamo repository selected by cmd_args.dynamo_version.

property installables: list[Installable]#: Get all installables for this test definition.

property is_domain_randomization_enabled: bool#

at least one env_params annotation.

Type:: Whether the config declares domain randomization

is_dse_excluded_arg(path: str) → bool#: Return whether a dot-separated cmd_args path should be ignored by DSE.

is_env_sampled(cmd_args_path: str) → bool#: Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).

validator validate_env_params » all fields#

Validate env_params annotations against cmd_args.

env_params is an annotation: each key names a cmd_args field whose value is the candidate set (the single source of truth), and the entry carries only how to sample. So each key must name a real cmd_args field whose value is a candidate list; a scalar is already fixed, so annotating it is a meaningless label and is rejected here. When weights are declared, the list needs >= 2 values and the weights must align 1:1 with it. Sampling, persistence, the per-trial cmd_args overlay, and the cache key all live in CloudAIGymEnv; keeping this shape check in core lets the overlay stay agent- and workload-agnostic rather than re-implemented per workload.