Troubleshooting

Find the failed stage before changing the experiment. Start with the error message and the configuration used for that run.

Symptom	First check	Continue
Configuration fails to load	File paths, workload name, field types, and referenced test definitions.	Understand experiment configuration and Inspect commands before execution
Prerequisite installation fails	Image or package availability, authentication, disk space, and cache path.	Prepare workload prerequisites
Job remains queued	Scheduler allocation, requested resources, and partition access.	Set up your execution environment
Workload exits unsuccessfully	Generated command, stdout.txt, and stderr.txt for that execution.	Workload guide in Workloads
Metrics or reports are missing	Workload completion, expected output, parser inputs, and report configuration.	Read results and generate reports
Search selects an unexpected result	Objective direction, valid rewards, missing metrics, and compared conditions.	Choose an objective and search space

Inspect the execution

CloudAI writes debug.log in the invocation directory unless --log-file selects another path. Find the generated command and the execution’s output directory there. On Slurm, inspect the generated sbatch script and scheduler state. A pending allocation is different from a workload failure.

Reproduce narrowly

Use the same configuration with dry-run to inspect command generation. If you rerun the generated workload command, it executes real work and may request resources. Reduce the scenario to the failing case while retaining its required inputs and environment. Change one suspected cause at a time.

Report an issue

Use the issue template with the CloudAI revision, invocation, minimal configuration, expected result, and relevant logs. Remove credentials and internal hostnames before posting.