Troubleshooting

Organized by symptom, because that is what you arrive with. Each entry gives the cause, the fix, and where to read more.

The job hangs

Every GPU drops to 0% but holds its memory, and nothing logs an error

The classic disaggregated-run failure at the first weight sync. It is not a deadlock — it is a segfault that never gets to die.

Look in the Ray worker .err file (not the actor log) for the canary:

W ucx_utils.cpp:628] 8 NVIDIA GPU(s) were detected, but UCX CUDA support was not found!
E ucx_utils.cpp:582] VRAM memory is detected as host by UCX...

followed by nixlAgent::registerMem -> ... -> ucp_mem_unmap -> SIGSEGV.

Cause: two UCX copies in one process. The NIXL wheel vendors its own UCX; NCCL's HPC-X plugin loads the container's. The wheel's copy loses CUDA support, treats device memory as host memory, and faults — and the other UCX's error handler catches the SIGSEGV and freezes the process instead of aborting.

Fix: the framework calls warm_nixl() before the NCCL rendezvous so the NIXL agent initializes first. This wins the race rather than removing it, so it is order-fragile: any code path that initializes NCCL earlier — a library that builds a process group at import, for instance — brings the hang back silently and in the same shape. Full analysis, the evidence, and what to re-check whenever the image or the NIXL pin moves: the internal environment analysis.

UCX_NET_DEVICES=lo is not the fix. It was introduced before the ordering bug was understood and merely masked it; it also cannot work cross-node.

The run stops mid weight sync after a bucket fails

Known limitation. Streaming weight sync has no abort path for a mid-stream failure (comm/weight_transfer.py, marked FIXME ... HANG RISK). If applying a bucket fails on one peer, the others keep waiting for buckets that will never arrive, so a recoverable error presents as a hang.

Workaround: none automatic. Kill the job and inspect the actor logs for the first bucket-apply exception; that peer is the origin.

Prolog hung on node ...

Normal on some clusters — container or NFS setup exceeding the prolog timeout. The job proceeds.

Out of memory

The policy OOMs in log_softmax on a colocated run

The logprob computation is deliberately done in fp32 over the full vocabulary (bf16 was measured to drop the GRPO clip rate 17x, from 0.01493 to 0.00089 at n_mini=8), and at Qwen's 151k vocabulary that tensor is roughly 1 GB per chunk. The colocated vLLM engine's reservation is what leaves too little room.

Fix, in order — change one item, rerun a short smoke, and stop as soon as it fits. This order preserves training semantics for as long as possible:

1. Lower rollout.gpu_memory_utilization (e.g. 0.40 → 0.25 → 0.20). Reduces vLLM KV capacity, so throughput may drop, but training semantics are unchanged.
2. Add or lower rl.micro_batch_per_replica. A pure memory split: same trajectories, same single optimizer step, smaller activation footprint. See howto/batching.md.
3. Reduce rollout.max_tokens and any validation max_tokens together, then rollout.max_model_len — only if a shorter reasoning and truncation contract is acceptable. Record every changed value with the run.
4. Reduce train and held-out image ceilings together for VLM runs (e.g. rl.data.factory_kwargs.max_image_side and the matching validation key). This changes visual resolution, so keep the two identical and record it as a distinct variant.
5. Stop and qualify a different model or topology rather than changing several memory axes at once.

micro_batch_per_replica rejected as an OOM fix

rl.micro_batch_per_replica must be divisible by the policy's dp_world. With one eight-rank policy replica, 8 is already the minimum legal value — 4, 2, and 1 fail the batch-divisibility contract and are not OOM recovery options. Likewise rl.batch_per_replica must assign whole prompt groups to each rollout data replica.

Placement and process-group errors

Message	Cause	Fix
ROLLOUT_RANKS=N must equal rollout.tensor_parallel_size=M	grid.shapes.rollout disagrees with the engine's TP width	Set grid.ranks / grid.shapes.rollout so the rank width equals tensor_parallel_size
POLICY_REPLICAS=N is unsupported	Policy placement has no replica axis	Use grid.shapes.policy=[1,POLICY_RANKS] and configure HSDP as dp_replicate * dp_shard inside those ranks — see glossary.md
grid.shapes requires grid.backend='ray'	Explicit shapes under the local backend	Either drop the shapes or switch to ray
per-replica rendezvous not run (need ranks=TP>1 on GPU)	tensor_parallel_size > 1 under the local backend	vLLM TP>1 needs Ray; local is the TP=1 path
policy cells disagree on needs_old_logprobs()	Replicas were not built alike	A topology fault — do not retry; the run refuses rather than training some cells clipped and others not

Two jobs on one node collide. They share the --ray-port (Ray GCS / torchrun c10d rendezvous) and contend for GPUs. Run them sequentially, or give each a distinct --ray-port and non-overlapping nodes.

Wrong or missing files at runtime

dataset has 4 item(s) but world_size=8, or workers read stale data

IMAGINAIRE_OUTPUT_ROOT is not on a shared filesystem. The default /tmp/imaginaire-output is container-local: the driver on the head node writes run artifacts there, and actors on other nodes resolve the same path inside their containers, reading absent or leftover files. Prefix the launcher's --run with a shared path.

The same applies to --log-dir: the default /tmp is node-local to the batch host and disappears with the allocation.

Intermittent pyxis: ... File exists during import

A multi-minute squashfs extraction can fail this way when Slurm re-fires a task launch. The generated sbatch script retries the import up to 3 times after removing the partial rootfs; only if all attempts fail does the job abort.

The run aborts

dynamic sampling batch_step=N: replay cells [...] hold fewer informative trajectories than their quotas [...]

rl.min_informative_groups > 0 (or rl.min_informative_groups_frac set) with the default rl.dynamic_sampling_on_budget_exhausted=raise: the extra-round budget (rl.dynamic_sampling_max_extra_rollouts) was spent and at least one replay cell still lacks its quota of groups whose rewards disagree (the global count split exactly across the data cells). Every listed cell saw fewer informative groups than its quota across 1 + dynamic_sampling_max_extra_rollouts rounds (the base draw plus the extra rounds), so the step did not train: the reward is not discriminating at the current policy — too hard (all zero), too easy (all one), or rl.group_size too small to see disagreement.

Fix: look at train/grpo_all_zero_group_count / grpo_all_one_group_count on the preceding steps and adjust the data mix, reward, or group size. Raising the budget only defers the abort when no prompt in the source produces spread. Set rl.dynamic_sampling_on_budget_exhausted=proceed only when a fixed batch shape matters more than the informative-only guarantee; the step then trains on tied groups as zero-advantage filler and logs the shortfall. Details: guides/grpo.md.

The π_old ahead pass refuses to start

π_old ahead-pass determinism probe failed on a sparse-MoE policy

The message reports the share of tokens whose logprob moved between two forwards at unchanged weights and says the forward routing was NATURAL. Cause: the MoE router's top-k over near-tied scores flips with kernel jitter alone (about half the tokens change experts between two identical forwards of Qwen3-VL-MoE-30B-A3B), so the recorded π_old does not describe the training forward. Fix: set rl.routing_replay=rollout, which pins every training forward to the routing the vLLM engine used, or run with rl.mini_batch_per_replica unset (n_mini == 1), where π_old comes from the same forward but the train/infer routing gap remains. See the routing-replay section in reference/config.md. If the message says the probe was PINNED, the movement is a genuinely stochastic layer, not the router.

Import and construction errors

Message	Cause	Fix
Unable to import DataPacker factory '...'	The dotted path is unavailable on the driver or a worker PYTHONPATH	Install or mount the extension module identically on every node
DataPacker factory '...' must return a DataPacker	The factory returned the wrong type	Return a fresh DataPacker subclass instance from the zero-argument factory
model_type='x' declares vision_config but has no native packer	An unregistered VLM with the i4-native backend	Set model_backend="hf", register a native packer, or pass an explicit one
... requires a DataPacker at construction	packer=None passed explicitly	Build one with init_data_packer(name_or_path)
packer tokenizer adds special tokens	A BOS-adding tokenizer on the vLLM path	See the guard section in howto/custom-datapacker.md
DataPacker.setup: empty name_or_path	Packer built without a model id	Pass the same name_or_path the model or engine was built from

The full table, including the objective-side symptoms, is in howto/custom-datapacker.md.

MissingConfigException: Could not load 'experiment/<name>'

Preceded by a warning like skipped N experiment module(s) needing absent optional dependencies ['lerobot', 'psycopg2', 'torchcodec', 'zarr'].

Cause. experiment= selection requires the experiment catalogs to be imported. Those catalogs live in another project and carry its optional dependencies — robotics datasets, a database driver, video and array codecs — which the posttrain runtime image does not install. Modules that cannot be imported are skipped with that warning, so the experiments they define cannot be selected. If the experiment you asked for, or one it inherits through defaults, lives in a skipped module, config resolution fails with this error.

The public DPO and NFT recipes are not affected: they are self-contained and do not import the inherited experiment catalog.

Fix: install the named dependencies into the environment, or run from an image that carries them. The warning lists exactly which distributions are missing.

ModuleNotFoundError: No module named 'apex' at optimizer construction

Raised by utils/fused_adam.py when a recipe sets rl.optimizer.name="fusedadam". The run reaches this only after the models have loaded, so it looks like a late failure rather than a missing prerequisite.

Why a recipe would need apex at all. fusedadam exists to rebuild an fp32 master copy of the parameters inside the optimizer. That is only necessary when the parameters are stored in bf16: plain torch.optim.AdamW keeps its moments in the parameter dtype, and at posttrain learning rates a bf16 update falls below bf16 resolution and rounds away.

The usual answer is not to install apex — it is to store the parameters in fp32. Set rl.master_dtype="float32" with rl.param_dtype="bfloat16": storage is fp32, compute is bf16, the optimizer gets fp32 moments for free, and name="adamw" (optionally impl="fused" for the fused torch kernel) is then exactly right. The public GSM8K recipe uses this configuration, and it matches cosmos-rl, whose LLM path never touches apex — apex lives only on its WFM/diffusion path.

So if you hit this on an LLM recipe, check rl.master_dtype before reaching for apex.

When apex is genuinely required. Two cases, and the second is not optional:

• A recipe that deliberately keeps bf16 parameter storage — for example to fit a tight single-GPU budget — has no fp32 parameters for the optimizer to read.
• The DPO diffusion path cannot use fp32 storage at all. master_dtype is a field on HFRunConfig; DPOConfig extends RunConfig and does not have it. The two reproduction experiments therefore require fusedadam, and their docstrings record why: the first attempt ran plain AdamW on the bf16 net and was killed at step ~30 with roughly 7x slower loser-side drift and the margin plateaued at ~3 against the reference's climb to 48 — same data, same objective, same trainable set. Optimizer numerics alone.

So apex remains a real dependency of this package. What is avoidable is reaching for it on an LLM recipe that could simply store its parameters in fp32.

rl.optimizer.name=adamw alone, on a recipe that stores parameters in bf16, is a diagnostic and not a fix: it will run, but its updates round away. config/rl.py::OptimizerConfig says do not use that combination for real bf16 posttraining. Change the storage dtype too.

Weight sync into a vLLM-Omni engine

cannot export the live parameters ... when a rollout body starts or rebuilds its engine

The body maps the worker's live parameters over CUDA IPC once per worker (roles/rollout/vllm_omni_worker.py::OmniIpcWorkerExtension.export_live_parameters) and refuses to start when they could not stay in place behind the handles. The engine is closed before the error propagates, so no worker process is left behind.

Message	Cause	Fix
... with enable_sleep_mode=True / enable_cpu_offload=True / enable_layerwise_offload=True: they would not stay in place behind the handle	the engine moves or re-allocates its weights	Serve a weight-synced engine without sleep mode and offload (the GR00T body's engine kwargs do)
... with custom_pipeline_args: re_init_pipeline re-allocates them	vLLM-Omni rebuilds the pipeline per request	Do not pass custom_pipeline_args to a weight-synced engine
cannot export the live parameters from the 'cudaMallocAsync' allocator	PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync in the worker's environment	CUDA IPC needs torch's native caching allocator; unset it
cannot export live parameter '...' over CUDA IPC: ... expandable_segments	PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True in the worker's environment	Unset it; the weight plane needs the default allocator
direct parameter writes need one worker per engine; this engine runs N	a tensor-parallel Omni engine	The direct write addresses one worker; TP > 1 Omni engines are outside the weight plane
the live parameters were already exported by this worker; rebuild the engine for a fresh mapping	a second export on one worker (each export consumes IPC event slots)	rebuild_engine()

begin_weight_update / finish_weight_update refused a stream and the lane is invalidated

Message	Cause	Fix
N live parameter(s) were re-allocated since the export / the live parameter names changed since the export	the worker's parameters moved after the body mapped them	rebuild_engine() on that lane, then a full sync of the trainer's current publication version
finish_weight_update(e): the open update is epoch k / no update is open on this worker	body and worker disagree on the stream: a failed stream left another epoch open, or a stale body	the next set_bucket_plan supersedes an open epoch; if finish keeps failing, rebuild_engine()
begin_weight_update(e) does not advance past epoch k	the body's epoch counter went backwards: two bodies on one worker, or a body replaced without rebuilding the engine	one body per engine; rebuild_engine()
N live parameter(s) do not hold the bytes just copied into them	the byte-wise compare after the copies failed: the mapping does not alias the worker's memory	a fault, not a knob: the lane is invalidated; check the device identity (index and UUID) in the actor log, then rebuild_engine() and a full sync. Only publications with verify_weight_values on run this compare

This path requires an operator-provisioned GR00T runtime and transport.