Version: main

sflow Error Analysis

Diagnose a failure in a fixed order: capture → classify → locate → fix → re-verify. The first move depends on when it broke:

Launch / validation failure — nothing ran (bad config, YAML, expression, artifact, merge, CSV). Caught by --dry-run; the fix is in the YAML.
Runtime failure — a task started and died (backend launch, probe timeout, crash, upload). You have a run dir; the fix is in the logs.

Figure out which you have, then follow that path below.

Step 1 — Capture the failure

A task ran and failed (you have a run dir) → start with the one-shot triage. It walks per-task status → orchestrator sflow.log → each failed <task>/<task>.log and prints a single root cause + fix. Drill in from its output:
```
python scripts/triage.py <output_dir>/<run_id>/
```

It failed at launch (no run dir) → reproduce with a dry-run and parse it:

sflow run -f config.yaml --dry-run
python scripts/parse_sflow_errors.py - < <(sflow run -f config.yaml --dry-run 2>&1)

Only have a pasted error? Match its marker in Step 2 to pick the path.

Step 2 — Classify (marker → where to look)

Category	Marker in the output	Path	Where to look
Config loading	`Configuration error:`, `File not found:`	launch	CLI output / sflow.log
YAML syntax	`Error parsing YAML`	launch	CLI output
Expression	`Undefined variable`, `Invalid expression syntax`	launch	CLI output
Artifact	`Artifact path validation failed`	launch	CLI output / dry-run
Schema	`type 'docker' is not valid` (use `docker_run`), `mutually exclusive`	launch	CLI output (validation)
Merge conflict	`Version conflict` (name/dup-task now warn+merge)	launch	CLI output (multi-file)
Batch / CSV	`CSV file`, `sflow_config_file column`	launch	CLI output
SLURM	`salloc failed`, `sbatch failed`	runtime	CLI / sbatch stderr
Kubernetes	`kubectl`, RBAC/preflight, pod `Pending`/`ImagePullBackOff`	runtime	CLI / sflow.log / `kubectl describe`
Docker	`docker run` errors, remote host, NVIDIA toolkit	runtime	`<task>/<task>.log`
Storage / uploads	`requires boto3`, `No AWS credentials`, upload failures	runtime	CLI / dry-run / sflow.log
Probe timeout	Task stuck waiting	runtime	`<task>/<task>.log`
Task failure	`Traceback`, non-zero exit	runtime	`<task>/<task>.log`

Step 3 — Locate the evidence

<output_dir>/<run_id>/
  sflow.log                    # orchestrator log (launch + status lines)
  sflow_summary.log            # live per-task status + failure hints
  *_cmds.log                   # command-only launch logs, by command family
  results.json                 # workflow-level parsed metrics (aggregated from task result: blocks)
  sflow_monitor.log            # hardware monitor overview (if monitor: enabled — local/slurm/docker only, NOT k8s)
  sflow_monitor/               # detailed monitor reports (csv/svg/png) + raw/ samples
  <task_name>/<task_name>.log  # task stdout/stderr — the crash lives here
  <task_name>/result.json      # that task's parsed metrics (if it declares a result: block)
  <task_name>_0/               # replica 0

Parsed results: a task with a result: block writes <task>/result.json (ok/values/errors) and updates the run-level results.json. Result parsing is best-effort — a missed regex or failed required: spec logs a warning and sets ok=false, but does not by itself fail the task; check errors there if a metric is empty. Parsing runs at task finalize, so a task shows COMPLETED only after it (and any uploads:) finish.
No monitor output? Hardware monitoring is unsupported on kubernetes (supports_host_monitoring=False) — sflow logs "node-level hardware monitoring is not implemented for this backend yet" and produces no sflow_monitor*. On local/slurm/docker, an empty monitor usually means the used_by_tasks target resolved to no nodes.
Batch jobs: also read the generated .sh and sbatch stdout/stderr (sflow_output/%j-sflow-submit.out / .err).
Kubernetes: kubectl describe pod <pod> and kubectl get events for scheduling/image issues the pod log can't show.

Step 4 — Root-cause & fix

4a. Launch / validation — fix in the YAML, re-run `--dry-run`

Error text	Fix
`Configuration file not found: <path>`	Check the `-f` path
`Error parsing YAML configuration: <detail>`	Fix YAML syntax at the indicated line
`Configuration validation failed: <detail>`	Check field types/values (schema-reference.md)
`Operator type 'docker' is not valid`	Use `type: docker_run` for the operator (backend is `type: docker`)
`Variable '<key>' ... is not defined`	Declare the variable in YAML before `--set`-ing it
`Undefined variable in expression`	Fix spelling; ensure the variable is declared
`Invalid expression syntax`	Fix the `${{ }}` Jinja2 syntax
`Artifact path validation failed`	Fix the `fs://` path or create it
`Version conflict`	Every file must use `version: "0.1"`
`CSV file must contain a 'sflow_config_file' column`	Add the required column to the CSV

4b. Runtime — read the logs (backend launch, then task crash)

Symptom	Fix
`salloc failed` / `sbatch failed`	Check partition/account/availability (`sinfo`, `sacctmgr`)
kube preflight: missing permission / namespace	Fix RBAC or `--kube-namespace`; `SFLOW_SKIP_K8S_PREFLIGHT=1` to bypass
pod stuck `Pending`	`kubectl describe pod`: no node has the GPUs — check `gpus_per_node`, `scheduling` mode, node-selector/tolerations, and that `gpus.count` is a multiple of the node count
pod `ImagePullBackOff` / container not found	Wrong `image` or missing `image_pull_secrets`; verify registry access
Docker GPU task fails	Install the NVIDIA container toolkit on the host; set `gpus_per_node`
Probe timeout (task hangs waiting)	Read the Probe Traces (last attempt) section of `sflow_summary.log` — it shows what each probe last saw (the last log line for `log_watch`, the endpoint result for `tcp_port`/`http`). Then compare server output to `log_watch` — it matches literally unless prefixed `re:`/`regex:`; adjust the pattern or `timeout`
`Traceback` in `<task>.log`	Read the traceback for the real cause (below are the usual ones)
CUDA OOM	Reduce batch size, TP/DP, or model size
`Address already in use` / port conflict	Kill stale processes or offset the port
NCCL / communication errors	Set `NCCL_SOCKET_IFNAME`; check the inter-node network
`S3 storage requires boto3` / `no AWS credentials` / upload fails	`pip install 'sflow[s3]'`; provide creds via the boto3 chain (`AWS_*` / `~/.aws/credentials` / IAM — never in YAML); verify bucket/region

For a pattern-by-pattern catalog (exact message, cause, and fix per error), see error-catalog.md.

Step 5 — Re-verify

Re-run the relevant capture command — clean exit means fixed:

sflow run -f config.yaml --dry-run          # launch fixes: exit 0 = ready
python scripts/triage.py <run_dir>/          # runtime fixes: no failed tasks

Diagnostic commands (reference)

sflow run -f config.yaml --dry-run                       # validate + resolve without executing
sflow compose f1.yaml f2.yaml --resolve -o merged.yaml   # see the merged + resolved config
python scripts/triage.py <output_dir>/<run_id>/          # one-shot: summary + logs -> root cause
python scripts/summarize_run.py <output_dir>/<run_id>/   #   per-task status + tracebacks only
python scripts/parse_sflow_errors.py <run_id>/sflow.log  #   categorize an orchestrator log
sinfo -p <partition>; sacctmgr show account <name>       # Slurm environment
kubectl -n <ns> get pods && kubectl -n <ns> describe pod <pod>   # Kubernetes scheduling/image
kubectl -n <ns> get events --sort-by=.lastTimestamp

Additional resources

Exhaustive error patterns → error-catalog.md
Docs → faq, cli, configuration, backends

Step 1 — Capture the failure​

Step 2 — Classify (marker → where to look)​

Step 3 — Locate the evidence​

Step 4 — Root-cause & fix​

4a. Launch / validation — fix in the YAML, re-run --dry-run​

4b. Runtime — read the logs (backend launch, then task crash)​

Step 5 — Re-verify​

Diagnostic commands (reference)​

Additional resources​