Skip to content

Config system

One posttrain experiment = an inherited i4 model/data tree (defaults) + a typed rl section, in ONE OmegaConf tree. The design serves three stakeholders at once: researchers keep their existing i4 experiments (inherit them verbatim), infra gets full CLI override, and every run dumps a resolved YAML that agents/repro tooling can reload. It is deliberately NOT Hydra-_target_-instantiate-everything — typed dataclasses + i4 LazyConfig inheritance + dotlist override (DESIGN §11.3 / appendix B.4).

Selecting and overriding

PYTHONPATH=. torchrun --nproc_per_node=4 --standalone -m cosmos_framework.posttrain.entrypoints.dpo -- \
  experiment=cosmos3_ga_16bm8b_dpo_v1 \
  rl.beta=1000 rl.optimizer.lr=5e-7 rl.max_steps=20 \
  model.config.parallelism.data_parallel_shard_degree=4
  • experiment= selects a registered experiment (Hydra ConfigStore).
  • ONE dotlist (after --) covers both halves: RL hypers (rl.*, grid.*) and the inherited model tree (model.*) alike.
  • Every run writes <out>/config_resolved.yaml — fully resolved, reload-able. GRPO attempts add a top-level provenance mapping to it and a provenance.<attempt_ts>.json beside it: the tree hash, git commit, interpreter, and job that produced the attempt (../guides/grpo.md §7).

Where fields live

subtree what schema
model.* an inherited i4 LazyConfig tree, or NativePolicyBuildSpec for an online native policy the i4 experiment or config/model.py
rl.* the RL half: shared lifecycle + optimizer, then algorithm knobs (objective/ref/data, group/sampling, actor-critic updates, teacher credit, or online WFM) config/rl.py::RunConfig → DPOConfig/GRPOConfig/PPOConfig/OPDConfig/NFTConfig
grid.* orchestration: backend (local|ray); DPO/PPO/NFT grouping; GRPO/PPO/OPD gpu_placement (colocated|disaggregated); plus ranks, num_replicas, and per-role shapes config/rl.py::GridConfig and algorithm-specific extensions
model_backend / model config model sourcing: i4 (model, the default) | hf (model_hf, explicit); an unsupported native VLM can select hf for image-conditioned clipped GRPO, which dispatches configs with vision_config to the generic HF VLM packer and propagates the policy's trust_remote_code decision to its model, rollout/vLLM, and packer consumers; legacy qwen3_vl normalizes to i4 config/model.py::NativePolicyBuildSpec / HFModelConfig

config/build.py keeps lazy imports, general parallelism resolution, build_optimizer, and a thin explicit backend dispatch. Model-family construction belongs to each model/*.py; each algorithm entry assembles its own stack. config/config.py holds the shared PosttrainConfig(cosmos3.Config) base + build_root (registration + job/trainer defaults); the runnable roots are config/config_dpo.py, config/config_grpo.py, config/config_ppo.py, config/config_opd.py, and config/config_nft.py. PosttrainOPDConfig inherits PosttrainGRPOConfig, preserves its established HF default, and overrides algorithm plus the typed rl: OPDConfig field; shared model specs remain on PosttrainConfig. PosttrainPPOConfig adds separate policy/reference/critic model configuration and a PPOGridConfig; the synchronous entry requires Ray, per-role grouping, and a local multi-role checkpoint store.

An online NativePolicyBuildSpec names a dotted builder, an explicit model_id shared with rollout and tokenizer shaping, an opaque builder-owned model_config, and its data-parallel mesh. Generic entrypoints read model_id but do not require the opaque config to imitate Hugging Face fields or instantiate it on the driver.

Adding your own experiment

Drop a file in config/experiment/dpo/ (auto-registered):

from cosmos_framework.utils.lazy_config import LazyDict
from hydra.core.config_store import ConfigStore
from cosmos_framework.posttrain.config.rl import DPOConfig, OptimizerConfig, PreferenceEnvConfig, RefConfig

my_dpo_exp = LazyDict(dict(
    defaults=["/experiment/cosmos3_ga_16bm8b_v2_midtrain", "_self_"],  # inherit a real i4 experiment
    rl=DPOConfig(
        beta=500.0, max_steps=2000, save_every=250,
        # freeze backbone, train only the OmniMoT gen-pathway params
        optimizer=OptimizerConfig(lr=1e-6, train_keys=["moe_gen", "time_embedder", "vae2llm", "llm2vae"]),
        ref=RefConfig(update_every=0),                       # 0 = frozen DPO anchor
        data=PreferenceEnvConfig(jsonl_path="/path/pref.jsonl", media="video", num_frames=121),
    ),
    job=dict(group="posttrain_dpo", name="my_dpo_exp"),
), flags={"allow_objects": True})
ConfigStore.instance().store(group="experiment", package="_global_", name="my_dpo_exp", node=my_dpo_exp)

Then ... entrypoints.dpo -- experiment=my_dpo_exp. The shipped experiments (cosmos3_ga_16bm8b_dpo_v1, image variant cosmos3_ga_16bm8b_t2i_dpo_v1) inherit the 8B test experiment and use in-memory synthetic pairs (rl.data.synthetic_pairs), so they run with no external checkpoint/data.

Parallelism — two layers, don't mix them

  • Engine mesh (FSDP/HSDP/CP/CFG-parallel) lives in the inherited model.config.parallelism.* — how ONE model instance shards.
  • Orchestration (grid.*) is the RL-framework layer above it — who hosts the role bodies (backend), how roles become ActorGroups (grouping), how those groups share or split DeviceGroups (gpu_placement), and each WG's execution shape.

Use the same names for every role's orchestration shape:

R = *_REPLICAS  # independent WG execution replicas
K = *_RANKS     # worker/process ranks inside each execution replica

For policy and rollout this means:

policy  = (POLICY_REPLICAS,  POLICY_RANKS)
rollout = (ROLLOUT_REPLICAS, ROLLOUT_RANKS)

POLICY_REPLICAS = 1
POLICY_RANKS    = dp_replicate * dp_shard  # implemented HSDP-only topology
ROLLOUT_RANKS   = rollout tensor-parallel ranks per engine

PPO additionally declares one frozen reference role of width grid.reference_ranks and one trainable critic role of width grid.critic_ranks. Their model parallelism is independent of physical placement; reference and critic retain separate GPU DeviceGroups in both PPO placement modes.

For the current HSDP-only policy topology, dp_replicate and dp_shard are the complete Torch DeviceMesh dimensions inside the policy's K ranks. In the Ray backend, each (R,K) cell is one _Host actor; local execution has no Ray actor layer. They are not placement replicas: do not call dp_replicate simply "replica", map it to grid.num_replicas, or add framework-owned gradient reduction between policy execution replicas. FSDP owns gradient synchronization across the policy mesh. Rollout replicas, by contrast, are independent inference engines, each made of ROLLOUT_RANKS ranks.

HF policies configure these dimensions as model_hf.parallelism.dp_replicate and model_hf.parallelism.dp_shard. A NativePolicyBuildSpec uses the same two field names under model.parallelism; a legacy inherited i4 tree continues to use model.config.parallelism.data_parallel_{replicate,shard}_degree. A concrete native builder may enforce a narrower contract; Qwen3-VL, for example, accepts pure FSDP only and rejects policy TP/EP/PP/CP and a replicate axis.

Per-role shapes (grid.shapes). The fallback policy shape is (1, ranks) and the rollout shape is (num_replicas, ranks). Set, for example, grid.shapes = {"policy": [1, 64], "rollout": [16, 4]} (dotlist: grid.shapes.policy=[1,64] grid.shapes.rollout=[16,4]) to give policy and rollout different execution shapes. The experience roles (data/rm/replay) derive at (R_rollout, 1). Every axis must be >= 1, and policy R must be 1 under the normative contract. Shapes do not allocate GPUs or select a transport. For online GRPO and OPD, grid.gpu_placement=colocated (the default) places equal-product rollout/experience and policy ActorGroups in one GPU DeviceGroup; grid.gpu_placement=disaggregated assigns separate GPU DeviceGroups. OPD's teacher remains in its own pool in either case. Both entrypoints use the same lowering and derive same_gpu or cross_device from that structure before checking the independent rl.weight_sync_transport choice. By default, unequal policy/rollout products require the Ray disaggregated placement with weight_sync_transport="nixl". Every policy actor materializes the full bucket; asymmetric routing assigns each rollout flat actor q directly to policy flat actor q % POLICY_RANKS. weight_sync_r2r_broadcast=true instead keeps that existing P2R transfer for rollout replica 0 and then broadcasts the engines' live parameters and persistent buffers between corresponding rollout ranks over NCCL. The GRPO entry enables exact-byte value verification on its initial sync only; steady-state sync skips that full-state device-to-host proof. The R2R phase is also independently callable, with optional verification, after replica 0 loads an eval checkpoint. The OPD entry currently requires one rollout replica and rejects this option. Wider policy pools retain explicit idle actors. Ray GPU fractions are private binder details; there is no fractional topology knob. grid.gpus_per_role remains a reserved, unconsumed field.

For an eval-only rollout, bind with bind_device_groups(..., weight_sync_r2r_broadcast=True) without a trainer weight edge, load the desired checkpoint into replica 0, then call control.loop.weight_sync.r2r_weight_sync(rollout, model_step=version). The same R2R operation and two-phase version publication used by online relay mode run without a policy role.

Online GRPO/OPD requires explicit grid.shapes when grid.num_replicas > 1; the legacy scalar no longer implicitly creates policy execution replicas. This forces policy and rollout widths to be visible rather than silently choosing an unrunnable same-GPU unequal-product topology.

PPO fields and launch constraints are summarized in ../guides/ppo.md; the typed source of truth is config/rl.py::PPOConfig plus config/config_ppo.py validation.

Shared checkpoint component controls

All training entries use the logical component names model, optimizer, and training_state. rl.checkpoint_components controls cadence and normal-final saves and defaults to the full [model, optimizer, training_state] plan. rl.validation.checkpoint_components has the same full default, but validation does not request a save unless rl.validation.save_checkpoint=true; that switch defaults to false. If regular saving and validation coincide, their plans are unioned and written once.

Only the full plan is exact-resume capable. When a proper subset becomes active as the regular run plan or for a validation save, the checkpoint manager emits UserWarning; a resulting artifact is committed under the sibling checkpoint_artifacts/iter_* namespace without advancing latest_checkpoint.txt. An inactive validation plan does not warn. Empty, duplicate, non-string, and unknown names are rejected. Emergency saves always override both selections with the full plan.

rl.keep_optim_last=null is backend-aware automatic policy: a local primary resolves it to 1, while a direct object-store primary resolves it to 0 because remote pruning is unsupported. Explicit 0 keeps all local optimizer state; another non-negative integer selects that many newest full iterations. See checkpoint.md for physical component mapping, artifact layout, and resume semantics. No knob controls the .uploaded archive marker that archived checkpoints and artifacts carry; its semantics are described in checkpoint.md.

RL knobs reference (rl.*, DPO)

field default meaning
beta 500.0 DPO temperature (validated > 0)
max_steps / pairs_per_step 1000 / 1 loop length / pairs per step
grad_accum 1 optimizer-step cadence
deterministic false bit-reproducible runs: cuDNN deterministic, use_deterministic_algorithms(warn_only=True), FLASH_ATTENTION_DETERMINISTIC=1. Costs throughput, so it is opt-in and meant for equivalence gates, not production. Export CUBLAS_WORKSPACE_CONFIG=:4096:8 and PYTHONHASHSEED=42 in the LAUNCH environment — cuBLAS reads its variable when it creates its handle, so setting it from Python is silently too late, and Ray actors are separate processes that inherit only what the launch gave them. The helper raises rather than warns if CUDA is already up (utils/determinism.py). Reproducibility also needs deterministic prompt ids, which roles/env/dataset.py::_stamp_prompt_id now provides unconditionally.
optimizer.* AdamW lr=1e-6, train_keys=[] (train all) RL-owned; a non-empty train_keys freezes everything not matching it (fails loud if nothing matches). The cosmos3 recipes set the OmniMoT gen-pathway subset (moe_gen/time_embedder/vae2llm/llm2vae).
optimizer.name adamw fusedadam = apex-fused AdamW, fp32 master weights + fp32 moments — REQUIRED for real bf16-stored nets (see utils/fused_adam.py). muon reuses cosmos3.utils.MuonWithAuxAdamW: hidden nn.Linear weights and explicitly declared stacked MoE experts take Muon; embeddings, output heads, routers, norms, biases, convolutions, coupled 2-D gate/up and qkv fusions, and other parameters take the auxiliary AdamW route. Native expert stacks are processed from complete local expert-axis shards without a full-stack all-gather, and fused expert gate/up halves are orthogonalised separately. Requires master_dtype=float32, BF16 FSDP compute, Transformer Engine and GPU
optimizer.muon_momentum / optimizer.muon_ns_steps / optimizer.muon_rms_match 0.95 / 5 / 0.2 name=muon only: the momentum β of the Muon buffer (m ← β·m + g, Nesterov direction g + β·m), the positive-integer Newton-Schulz iteration count, and the finite-positive Moonlight LR scale (× sqrt(max(rows, cols))). The inherited AdamW LR is only an initialization heuristic: Muon is scale-free while Adam can be eps-dominated, so a Muon run must re-tune LR. Strictly validated at both config and optimizer construction; inert for other optimizer names
optimizer.eps 1e-8 Adam's denominator floor, update = lr · m / (√v + eps) (torch's default). A gradient-SCALE knob, not a stability nicety: once √v for a parameter falls below eps, that parameter's update degenerates to (lr/eps) · m, an SGD-like step whose size follows lr/eps rather than lr. The cosmos-rl-derived GRPO recipes carry 1e-6 (100× the default), sized for token-mean gradients; under loss_type=seq-mean-token-sum-fixed the gradients fall below it and GRPOConfig warns when eps > 1e-7 (refuses the pairing under recipe_guard_severity=error). The recipe layer that selects the fixed reduction together with 1e-8 is the Dr-GRPO recipe layer — see the Dr-GRPO paragraph in the GRPO section
optimizer.grad_clip_norm 0.0 clip the global gradient norm before an applied update; 0 disables clipping
optimizer.force_finite_grads true require rank-consistent finite gradients and reject the entire accumulation window otherwise; false is an explicit legacy opt-out for this gradient check only
ref.build_ref / ref.update_every true / 0 frozen anchor (0) vs moving ref (>0); moving mode requires an adapter with refresh_ref(), and current Cosmos3 DPO recipes use 0. The shared RefConfig has no anchor-source field: DPO anchors its reference through init_from, and a ref.model_dir override (GRPO's GRPORefConfig knob) is refused here as an unknown key
init_keys_to_skip_loading empty exact source-only init_from keys the DPO backbone does not build; any other source-only key fails the copy-mode load, and exact resume never consults the list
data.* — preference JSONL path (or tar_path offsets, or synthetic_pairs>0 for in-memory toy data), media/resolution/frames, cfg-dropout, shuffle (the run seed is rl.seed)
seed 0 the run seed (RunConfig): the process RNGs (set_random_seed) and the preference-source shuffle derive from it
save_every, resume, run_ts, checkpoint_format, checkpoint_components, checkpoint_store.*, save_async, keep_optim_last, keep_model_last, upload.* shared RunConfig defaults checkpoint_components defaults full; keep_optim_last=null resolves to local 1 or direct 0; checkpoint_store selects the primary resumable store, while upload is only an optional archive of a local primary. See checkpoint.md

Rollout engine knobs (rollout.*, vLLM)

field default meaning
limit_mm_per_prompt None Per-REQUEST multimodal item budget forwarded to vllm.LLM(limit_mm_per_prompt=...) only when set, e.g. {"image": 40}; None leaves the key out, so the engine construction identity of every other run is unchanged and the engine's own default applies (verified in the pinned container: vLLM 0.24.0, BaseDummyOptions.count 999 and get_limit_per_prompt() 999 for an unspecified modality, the 0.23.0 wheel pin carries the same code — so the engine's own default is 999 items per supported modality and setting a value LOWERS the budget; it is the lever for bounding the encoder memory the engine profiles for and for refusing over-budget prompts at admission, At most N image(s) may be provided in one request). 0 disables the modality (a prompt carrying it is refused). Keys must be bare lower-case vLLM modality names from {image, video, audio}: surrounding whitespace, punctuation or an unknown name (images) is refused at config time, and the built engine's model-specific supported set is checked again at VLLMRollout construction, because vLLM itself converts an unknown key into a generic dummy option and silently ignores it. Raising the limit makes vLLM profile more encoder memory at start-up (fewer KV blocks under the same gpu_memory_utilization); lowering it caps what one prompt may carry. Validation: non-empty mapping, non-bool integers >= 0.
compilation_config {} vLLM CompilationConfig as a mapping, forwarded as vllm.LLM(compilation_config=...) only when non-empty. On the command line use Hydra's add-or-override form, ++rollout.compilation_config={cudagraph_mode:FULL_DECODE_ONLY,cudagraph_capture_sizes:[1,2,4,8,16]} (or +rollout.compilation_config.<key>=... per key): the composed config is in struct mode, so the plain key={...} form cannot add keys. It configures torch.compile and CUDA-graph capture, so a non-empty mapping requires enforce_eager=false (refused otherwise). Empty means unset because Hydra applies a dict override by merging, which a None default cannot take. Hydra mappings are materialized to plain dicts and lists; the engine receives a deep copy.
async_scheduling None vLLM async_scheduling, forwarded only when set: true overlaps scheduling of the next engine step with the current step's GPU work.
max_num_seqs None vLLM max_num_seqs, forwarded only when set: the cap on sequences one engine runs at once, which also bounds the CUDA-graph capture sizes vLLM derives. Positive non-bool integer.
kv_cache_memory_bytes None vLLM kv_cache_memory_bytes, forwarded only when set: a fixed per-GPU KV-cache size in bytes instead of the size derived from gpu_memory_utilization after the memory profile. Positive non-bool integer.
load_format auto vLLM model-weight loader: auto or dummy. dummy skips checkpoint tensor I/O and starts from random weights; the rollout refuses to generate until a weight publication commits.
custom_all_reduce false Full-node tensor-parallel engines only, with vLLM-supported world size 2, 4, 6 or 8: true forwards disable_custom_all_reduce=false, so vLLM all-reduces with its custom kernel instead of NCCL. The TP replica must consume exactly the node's GPU capacity; partial-node/multi-replica and host-affinity placements are refused because the kernel maps every visible peer's buffers over CUDA IPC. GRPO shows the full-node experience replica its GPUs with each actor's own first (cuda:0 stays its device; the colocated trainer keeps one GPU). In this layout every engine rank reports device 0, which PyTorch symmetric memory refuses, so FlashInfer's all-reduce + RMSNorm fusion stays off; the custom all-reduce itself runs. PPO rejects it during preflight because its rollout actors keep one visible GPU.

RL knobs reference (rl.*, GRPO)

field default meaning
grpo_surrogate clipped GRPO policy-gradient surrogate: production frozen-π_old clipped loss, or legacy_pg for the minimal toy/parity loss. The default arithmetic root pins legacy_pg explicitly.
temperature 1.0 divides policy logits for both legacy_pg and clipped scoring and must equal top-level rollout.temperature. Grouped vLLM rollout rejects zero and rewrites nonzero values below 0.01, so GPU-less preflight rejects both ranges rather than accepting a different effective behavior distribution.
loss_type token-mean clipped-loss normalization: token-mean, seq-mean-token-sum, seq-mean-token-mean, or Dr-GRPO's seq-mean-token-sum-fixed, which divides by fixed rollout.max_tokens. The fixed divisor shrinks the gradients Adam sees, so the config warns when it pairs with optimizer.eps > 1e-7 — refuses it under recipe_guard_severity=error (Dr-GRPO paragraph below).
logprob_dtype float32 dtype used for clipped-surrogate log_softmax, independent of model compute dtype
param_dtype / master_dtype / reduce_dtype / transfer_dtype "" (inherit: model_hf.dtype / the resolved param_dtype / model_hf.fsdp_reduce_dtype / the resolved master_dtype) The FSDP2 precision block (config/rl.py::HFRunConfig): compute dtype, storage dtype (also the optimizer-moment dtype), gradient all-reduce dtype, and the dtype weights are cast to on the policy → rollout hop. master_dtype="float32" + param_dtype="bfloat16" is the fp32-master/bf16-compute recipe.
mixed_precision_mode "" (⇒ fsdp) How param_dtype reaches the forward. fsdp: the MixedPrecisionPolicy parameter cast, which needs an FSDP wrap whenever master_dtype != param_dtype (refused unwrapped). autocast: parameters stay in master_dtype and the forward runs under torch.autocast(param_dtype) at every world size, wrapped or not — numerically different from fsdp because autocast is per-op (what HF TrainingArguments(bf16=True) does). Refused with param_dtype="float16" (no GradScaler) and on model_backend="i4".
epsilon_low / epsilon_high 0.2 / 0.2 clipped surrogate ratio bounds; a negative value disables that side of the clip. Under importance_sampling_level=sequence they bound a response-mean log-ratio, which sits far below a token's, so the config warns when the largest non-negative epsilon exceeds 0.05 (the GSPO recipe uses 3e-4 / 4e-4).
lower_bound_ratio 3.0 dual-clip lower bound for negative-advantage tokens; zero disables it. Must be 0 under importance_sampling_level=sequence, which applies no dual clip to the response ratio and rejects any other value.
importance_sampling_level token granularity of the clipped surrogate's importance ratio. token scores one ratio per token (GRPO/PPO). sequence scores one ratio per response, exp of the mean token log-ratio over that response's loss-masked tokens, shared by all of them so the clip admits or drops whole responses (GSPO). Consumed only by clipped; legacy_pg and OPD reject sequence. See the GSPO paragraph below.
kl_beta 0.0 reference-policy KL scale for the clipped surrogate. Positive values add kl_beta · k3(logπ_ref − logπ_θ) to every loss-masked completion token's loss (k3 = exp(z) − z − 1 ≥ 0, the DeepSeekMath GRPO estimator of KL(π_θ‖π_ref); the log-ratio is clamped at ±20 nats like PPO's sampled_k3_kl), pulling the policy toward a frozen reference. Requires ref.build_ref=true; refused with legacy_pg and by OPD. Logged as train/kl_mean (token-weighted mean of the unscaled k3), train/kl_max (the largest per-token k3 in the batch, MAX-reduced across ranks and chunks) and train/kl_beta. Value and gradient name two divergences: the logged k3 estimates the forward KL(π_θ‖π_ref), while the gradient the sampled loss applies is that of KL(π_ref‖π_θ), the mode-covering pull toward the reference (the DeepSeekMath convention). Under routing_replay=rollout the reference forward prepares the same batch as the policy forward and so pins to the same recorded expert routing: z then measures weight drift, not routing churn (see that row). Without kl_coeff_clamp the config warns: the exact k3 gradient is unbounded (next row)
kl_coeff_clamp null bound, in nats, on the KL term's per-token gradient coefficient; null keeps the exact k3 gradient 1 − exp(z), byte-identical to before the knob existed. That coefficient is unbounded below inside the ±20 nat log-ratio clamp (−4.85e8 just inside it), so one token whose reference forward disagrees with the policy forward by a few nats owns the whole step — routing nondeterminism can make a small number of coefficients dominate a step. With kl_coeff_clamp=c the penalty's VALUE (and every train/kl_* statistic) stays the exact k3 while its gradient is that of k3's linear extension beyond \|z\| = c: coefficient 1 − exp(clamp(z, −c, c)), at most exp(c) − 1 in magnitude (5.0 recommended: ≤ 147) and still pulling toward the reference past the band. Must be positive and below 20 (the ±20 clamp already zeroes the gradient beyond it, so a larger value is inert and refused); requires kl_beta>0; recorded in the objective identity only when set. Implementation: algorithm/primitives/policy_gradient.py::sampled_k3_penalty
ref.build_ref / ref.update_every false / 0 frozen KL reference for kl_beta>0: a second FSDP-sharded, gradient-free instance of the policy built by the same builder from the same base checkpoint on every start (so it is always the run's step-0 weights; a resume restores the policy only), scored under no_grad on every training chunk (not on the π_old ahead pass or the determinism probe at n_mini > 1, whose losses are discarded) and offloaded with the policy under colocation. Costs one extra copy of the network per policy rank. build_trainer refuses a policy with live dropout (an nn.Dropout with p > 0 or a functional attention_dropout-style config value) before building the twin: the reference scores under eval() while the policy trains under train(), so the log-ratio would be a dropout mask at any n_mini. build_ref=true requires kl_beta>0 and vice versa; update_every must be 0 (no moving reference for the LLM trainer)
ref.model_dir null GRPO-only field of GRPORefConfig (the type of GRPO's ref; a plain RefConfig is adopted with model_dir=null). Warm restart for model-only saves: build the frozen KL reference from THIS checkpoint directory instead of the policy's own (model_qwen3_vl.name_or_path/checkpoint_path, model.model_id, or model_hf.name_or_path), so a run whose policy starts from its latest save keeps its anchor at the ORIGINAL base. Requires build_ref=true (checked at config load). Must be a directory when the trainer is built: build_trainer checks on every rank before any model is built, not the dataclass, so a saved config reloads on a host without the checkpoint mount; the trainer logs built frozen KL reference from <dir> (rl.ref.model_dir). The directory's exact CONTENT identity (utils/artifact_identity.py: full SHA-256 of every metadata and weight file, read once on rank 0) joins the objective's exact-resume identity as reference_artifact, so an exact resume that names another checkpoint, or the same path re-populated with other weights, is refused rather than silently anchored elsewhere; a directory without weight files is refused at build. The full read can add startup latency for large references, but it does not recur per step. Consumed by the GRPO entry only; the shared RefConfig DPO and NFT use has no such field, so rl.ref.model_dir on those configs is an unknown key
unbiased_advantage false false uses (reward - group mean) / group std; true mean-centres without the std division. Dr-GRPO pins true.
use_rollout_logprobs_for_loss false false recomputes/fixes π_old in the policy path; true consumes the rollout engine's raw (temperature-1, unwarped) logprobs as π_old and disables the ahead-pass cache, so preflight additionally requires rollout.temperature == rl.temperature == 1.0 (top-p/top-k/repetition-penalty warping is already rejected for every on-policy GRPO run). train/logprob_gap_* reports the train/infer gap in either setting
min_filter_prefix_tokens null minimum shared-prefix span to mask when members of one group share that prefix but receive different rewards; the complete group is annotated before dispatch
mask_truncated_completions false DAPO overlong filtering (arXiv 2503.14476 §3.3). true zeroes the loss mask of every completion the rollout engine cut at its token budget (policy_info["finish_reason"] == "length", stamped by the vLLM rollout from CompletionOutput.finish_reason; the HF and local rollouts stamp none, so nothing is masked there), so it enters neither the clipped surrogate nor the KL term nor their token/sequence divisors — the same exit an empty completion takes. false trains on it like any other completion. The mask removes the completion from the loss only (DAPO's definition): its reward stayed in the group mean/std the reward stage computed before dispatch, so the kept members train on their full-group advantages, and the group still counts toward the informative-group gate. The rate diagnostics train/rollout_truncated_frac, train/response_len_mean and train/response_len_max report either way. Consumed by grpo_surrogate=clipped only; legacy_pg and OPD refuse true. Joins the objective identity only when true, so existing clipped checkpoints resume unchanged
recipe_guard_severity warn how the two recipe-hygiene checks report. Both pairings — a literal min_informative_groups below half of groups_per_step (dynamic sampling in name only on a wider layout) and loss_type=seq-mean-token-sum-fixed with optimizer.eps > 1e-7 (the reported lr is not the step size) — are honored as written, so warn (default) WARNS at construction; existing fields and default runtime behavior remain unchanged. error collects every hazard found and refuses the config in ONE ValueError that lists them all (each hazard's text as the warning has it, joined by a blank line, then one sentence naming the knob), for a recipe that wants a mis-pairing caught at preflight: --validate-manifest re-instantiates rl after the dotted overrides, so rl.recipe_guard_severity=error on a command line is enough, and the Dr-GRPO Qwen3-VL RLVR layer (the Dr-GRPO recipe layer) sets it in the recipe. Thresholds and messages are the same in both modes; any other value is refused; keyword-only; inherited by OPD, which refuses both pairings outright anyway
score_centering.enabled / top_k / compose / tis_clip / mis_low / mis_high false / 128 / none / 2.0 / 0.5 / 5.0 score centering of the policy gradient under the SAMPLER's distribution (algorithm/primitives/score_centering.py). enabled=true builds the vLLM engine with logprobs_mode=processed_logprobs, max_logprobs=top_k, requests SamplingParams(logprobs=top_k), stamps policy_info["rollout_topk_ids"] / ["rollout_topk_logprobs"] ([T, k], int32 / float32 — never float16: a rounded head gives q != p for an identical sampler and trainer) on every training completion, and the clipped objective adds A · Σ_{v∈H} sg[q_v w_v − α p_v] log p_v per token (head H = the sampler's top-k, ρ = q_tail/p_tail, α = ρ f(1/ρ)) and scales the surrogate by sg[f(p_{y_t}/q_{y_t})], so the expected gradient under the sampler is drift-free. compose=none is pure centering (f = 1, weight exactly 1); tis composes truncated IS f(r) = min(r, tis_clip); mis masked IS f(r) = r · 1[mis_low ≤ r ≤ mis_high]. The KL term, the clip diagnostics and the loss reduction are untouched; with q == p the correction is exactly 0 and the loss bit-identical to enabled=false. Requires grpo_surrogate=clipped, importance_sampling_level=token, n_mini == 1, use_rollout_logprobs_for_loss=false (refused otherwise; OPD refuses it). Under it train/logprob_gap_* compares TEMPERED logprobs on both sides. Diagnostics: train/sc_* (metrics reference). Joins the objective identity when on.
mu_iterations 1 optimizer passes per collected batch in the upstream vocabulary; values other than one are rejected because the local loop expresses reuse through n_mini
certified_deterministic_forward false required when a clipped/advantage objective uses a cached π_old across n_mini > 1; attests that the ahead and training forwards are equivalent. The trainer still runs a one-shot eval-vs-train probe on the first ahead chunk, gated by the tolerance the model declares (ForwardNoisePolicy; strict 0.05 nats on any token by default). See the routing-replay paragraph below for why a sparse-MoE policy fails it at n_mini > 1.
routing_replay off sparse-MoE routing replay (Rollout Routing Replay, Ma et al., arXiv 2510.11370; DeepSeek's variant is "Keep Routing"): pin each forward's top-k expert dispatch to the recorded selection the policy batch carries, through the model's ForwardReplayPolicy capability, so the training forward routes as the rollout did and the importance ratio measures policy change instead of routing churn. rollout replays the vLLM engine's own routing: the engine is built with enable_return_routed_experts, the rollout records policy_info["rollout_routed_experts"], the packer aligns it to the policy batch and the MoE adapter pins every forward (ahead pass, probe, training) from it. With kl_beta > 0 the frozen reference prepares the same batch through its own adapter, so its forward is pinned to the same recorded routing: the KL anchor is π_ref under the rollout's expert dispatch, and z measures weight drift rather than routing churn (an unpinned reference would put the routing flip back into z). Clipped surrogate only (grpo_surrogate=clipped, the default; legacy_pg refuses it because that objective never prepares its batch through the model, so nothing would pin); valid at n_mini == 1 and with use_rollout_logprobs_for_loss=true. Any other value is refused at construction, OPD refuses non-off, and assembly refuses a dense or HF policy. A non-off value joins the exact-resume identity together with the model's replay_identity(), the engine flag joins the rollout identity, and the trainer refuses a forward whose pinned state contradicts the knob. See the section below.
max_steps 1000 collected batches. Each batch commits n_mini optimizer/model steps; validation, reports, and iter_* names use that model-step clock.
batch_per_replica / group_size 32 / 8 trajectories the policy trains on per policy execution replica per loop step / completions sampled per prompt. It is not per dp_replicate slice. batch_per_replica is the PRIMARY sizing knob; it must be a positive multiple of group_size.
prompts_per_data_replica DERIVED (batch_per_replica // group_size) — read-only property compatibility convenience for exactly one data execution replica. With multiple rollout/data replicas, control.loop.batching.prompts_per_data_cell divides this total prompt count evenly across them.
mini_batch_per_replica 0 ⇒ the whole batch trajectories per optimizer step in one policy execution replica; must evenly divide batch_per_replica (no ragged last mini-batch). Derives n_mini = batch_per_replica // mini_batch_per_replica
micro_batch_per_replica 0 ⇒ one chunk trajectories per forward/backward in one policy execution replica, not per HSDP dp_replicate; must evenly divide the mini-batch span (mini_batch_per_replica or, if unset, batch_per_replica). Derives n_micro = <span> // micro_batch_per_replica, which is also the trainer's grad_accum (see checkpoint.md). The trainer accumulates chunk sums and applies the divisor once per optimizer window, so changing this memory knob does not change the update; test_roles_trainer_deferred_norm.py::test_chunked_gradient_equals_unchunked pins the real trainer invariant.
min_informative_groups 0 ⇒ dynamic sampling OFF DAPO-style uniform-group filtering (arXiv 2503.14476 §3.2). When > 0, the replay buffer gates on the reward stage's stamped group reward spread — the only owner of "informative" — so whole groups whose rewards tie (zero advantage) are dropped before the optimizer step, and collection keeps drawing until the replay cells together hold this many informative groups (a global count that control.loop.batching.informative_quota_per_cell splits exactly across the data cells, so it does not grow with the replica count). The batch is assembled informative groups first at batch_per_replica; tied groups fill the remainder whenever this is below batch_per_replica // group_size (full DAPO sets it equal). Must satisfy min_informative_groups * group_size <= batch_per_replica. GRPO-only: OPDConfig rejects it. A literal count below half of groups_per_step (= batch_per_replica // group_size) warns at config time — the message names groups_per_step, the count and min_informative_groups_frac — because the count is global and does not scale with the batch: the 8 that is every group of b32g4 is 1/8 of b256g4's 64 groups, i.e. dynamic sampling in name only. Prefer min_informative_groups_frac for a recipe that moves between batch layouts
min_informative_groups_frac null ⇒ the literal count above is the quota the same quota spelled as a SHARE of the groups the batch holds per step: quota = max(1, ceil(frac × groups_per_step)), 0 < frac ≤ 1 (1.0 = full DAPO on any layout; never 0, so a set fraction is always ON; the fraction is read as the nearest rational with denominator ≤ 10⁶ before the ceil, so a share that names a whole number of groups is exactly that number: 0.14 of 50 groups is 7, where the float product 7.000000000000001 would ceil to 8). groups_per_step = batch_per_replica // group_size is placement-independent — the data cells split that one batch — so the share survives a change of batch layout where a literal count does not. Setting both this and min_informative_groups > 0 is rejected; leave the literal at 0. A literal count below half of groups_per_step warns at construction (the message names groups_per_step, the count and this key); recipe_guard_severity=error refuses it. GRPO-only: OPDConfig rejects it
groups_per_step / effective_min_informative_groups DERIVED — read-only properties the group capacity of one step (batch_per_replica // group_size) and the informative-group quota the collection loop enforces: the literal count, or max(1, ceil(min_informative_groups_frac × groups_per_step)) when the fraction is set. entrypoints/grpo.py reads only the latter and logs both, with the two knobs, at start-up when dynamic sampling is on
dynamic_sampling_max_extra_rollouts 4 top-up budget in extra ROUNDS per loop step (one more draw of prompts_per_data_cell prompts on each participating cell); how many rollout.generate calls those rounds take is set by dynamic_sampling_refill_dispatch and counted by train/grpo_dynamic_sampling_refill_dispatches; only consulted when the resolved quota (effective_min_informative_groups) is > 0
dynamic_sampling_refill_mode all all refills every cell; opt-in short_cells refills only cells still below their informative quota. The initial draw stays full everywhere, preserving batch capacity even for zero-quota cells. This changes subsequent source/engine histories and can retain tied filler at fractional quotas; it does not promise identical batches or learning curves.
dynamic_sampling_quota_scope cell cell requires every replay cell to buffer its exact share of the quota; opt-in pooled stops once the batch assembled informative-first holds the quota (each cell still contributes at most its batch share) and treats only cells whose share is not yet informative as short. Both enforce the same minimum quota on the trained batch, but the retained informative-group count and the batch composition can differ: cell keeps refilling owing cells after the batch already holds the quota, while pooled does not fix which cells supply it and avoids spending the whole budget when every cell draws one prompt.
dynamic_sampling_refill_dispatch per_round number of top-up rounds in one rollout.generate call, orthogonal to dynamic_sampling_refill_mode: per_round issues one round per call and re-counts short cells; whole_budget issues all remaining rounds in one call, so each participating cell serves remaining_rounds × prompts_per_data_cell prompts while cells idled by short_cells serve none. The wider call can advance the cursor farther and replace tied filler below full DAPO when per_round would have stopped early. train/grpo_dynamic_sampling_refill_dispatches counts calls, while train/grpo_dynamic_sampling_extra_rollouts retains the round unit. The widened request (per-engine width rounds per call × prompts_per_data_cell × group_size sequences when data cells map 1:1 to engines; the start-up line prints it) must stay within the engine's max_num_seqs: the framework passes none, so vLLM's usage-context default applies — 1024 for the LLM class on ≥ 70 GiB GPUs in vLLM 0.23.0. The data plane carries a budget × larger payload per publish: bulk_store has no page pool to exhaust, while ray_nixl_pool has a byte capacity to check against the widened payload before the first run.
dynamic_sampling_on_budget_exhausted raise what happens when the budget is spent and a replay cell is still short: raise aborts the run (the reward regime cannot supply informative groups — a recipe fault); proceed trains on tied groups as zero-advantage filler and logs the shortfall
grad_accum 1, and REJECTED if set to anything else retired as a GRPO knob — it meant "that many fresh rollouts per step", which the single-draw batch above replaces; use micro_batch_per_replica instead. Still a real DPO knob (grad_accum above)
save_every, resume, run_ts, checkpoint_format, checkpoint_components, checkpoint_store.*, save_async, keep_optim_last, keep_model_last, upload.* shared RunConfig defaults same backend-neutral CheckpointLifecycle as DPO; cadence/final component selection defaults full, keep_optim_last=null is backend-aware, and upload remains a local-after-save archive. Cadence is in collected batches while checkpoint names and validation use committed optimizer/model steps. See checkpoint.md
data.name / data.subset / data.split empty / empty / train Hugging Face dataset address. subset is the HF config name (e.g. main) and is required with name — empty reaches load_dataset as "" and fails at actor setup
data.path empty a local JSONL; with neither name nor path, the built-in arithmetic source is used
data.prompt_column empty required with data.path or data.name; renamed to the observation's prompt field
data.response_column empty ground-truth column, renamed to reference; required with data.name, optional for JSONL
data.items_fn empty dotted path to a deterministic CPU row map, fn(row: dict) -> dict; applied before column rename and requires data.name. No I/O, CUDA, or external mutation. See configs/grpo_qwen3b_gsm8k.py::gsm8k_items
data.shuffle false reshuffle every epoch with <data seed> + epoch, where the data seed is rl.seed (or top-level rollout.seed under data_seed_follows_rollout_seed). The whole corpus is permuted and THEN strided across data replicas, so rows migrate between replicas across epochs (matching DistributedSampler)
seed 0 the run seed (RunConfig). For GRPO it orders the PROMPT SOURCE only: the shuffle permutation above and the arithmetic generator. It does not seed sampling — vLLM samples under top-level rollout.seed (default null, the engine default) — so an arm set that gives each arm its own rollout.seed and leaves this at 0 replicates the sampling, not the prompt order; preflight warns about that combination when data.shuffle is on
data_seed_follows_rollout_seed false order the prompt source with top-level rollout.seed instead of rl.seed, so one per-arm seed varies the prompt order and the sampling together. Requires an integer rollout.seed (refused otherwise). Resolved once (config/rl.py::resolve_data_seed): preflight and the data role read the same value, and the cursor checkpoint records it, so an exact resume refuses a run that flipped it — set it from the first step
reward arithmetic provider key selected from the run-bound built-in + custom catalog
reward_provider_catalog empty dotted path to a zero-argument custom catalog factory; entries add group-native online providers and immutable semantic identities but cannot override built-ins. Pointwise scorers use map_candidates (guide).
weight_sync_transport null weight mover, independent of placement: unset serves colocated IPC/local host copy; nixl is required for grid.gpu_placement=disaggregated
weight_sync_r2r_broadcast false optional relay mode: unchanged P2R into rollout replica 0, followed by corresponding-rank NCCL broadcast of live engine tensors across rollout replicas
weight_sync_bucket_bytes null optional positive streaming bucket cap; the same value sizes the NIXL staging buffer/plan and, when relay mode is enabled, the R2R non-contiguous packing target
weight_sync_probe_bytes 268435456 pre-model NIXL bind probe payload bytes; the probe also times a pass at 1/8 of this size and gates on the slope between the two, so keep it large enough that the payload, not the per-transfer fixed cost, dominates (256 MiB is the minimum that works on InfiniBand)
weight_sync_min_gbps 20.0 non-negative minimum fabric throughput from the bind probe (net of the per-transfer fixed cost; the log line reports both)
colocated_offload true under grid.gpu_placement=colocated, sleep the vLLM engine while the trainer updates and park the trainer on the host while the engine generates, every step (the hand-off cycle); false builds the engine without sleep mode and skips the cycle, trading GPU memory for step time, so it is only for runs where engine and trainer fit side by side (bind-time memory checks and CUDA OOM surface one that does not); inert under disaggregated
data_pool.data_plane bulk_store bulk_store or the Ray/RDT-backed ray_nixl_pool; independent of weight placement and transport
data_pool.page_bytes 33554432 source allocation and receive-transfer granularity
data_pool.cuda_capacity_bytes / cpu_capacity_bytes 1073741824 / 536870912 persistent producer capacity per actor; each must be a page-size multiple
data_pool.cuda_cache_bytes / cpu_cache_bytes 536870912 / 536870912 persistent consumer receive capacity per actor; each must be a page-size multiple
data_pool.cpu_min_bytes 33554432 CPU tensors below this size bypass the page pool

Legacy selector migration is explicit: replace rl.objective=degenerate|clipped with rl.grpo_surrogate=legacy_pg|clipped, and replace OPD's rl.objective_mode=direct|advantage with rl.distillation_mode=direct|advantage. The old keys are not aliases and fail as unknown keys. legacy_pg rejects non-default clipped-only settings rather than recording knobs that its loss ignores.

Dr-GRPO is a resolved GRPO recipe, not another entry or objective wrapper. Launch cosmos_framework.posttrain.entrypoints.grpo with configs/dr_grpo_qwen3b_gsm8k.py. The GSM8K recipe records a distinct algorithm and job name, then selects rl.grpo_surrogate=clipped, rl.unbiased_advantage=true, and rl.loss_type=seq-mean-token-sum-fixed. The shared builder passes the single generation bound from rollout.max_tokens into that reduction and records it in the objective identity; no second max-response-length field, trainer, placement, or control loop is introduced. See objective_customization.md for the general objective contract, entry integration rules, and the Dr-GRPO ownership decision.

The fixed normaliser reduces the gradient scale relative to token-mean reduction. Choose an optimizer epsilon compatible with that scale and re-tune the learning rate when changing either the normalizer bound or epsilon.

rl.optimizer.name=muon reuses the model stack's MuonWithAuxAdamW and shared split_orthogonalizable_params classifier instead of maintaining a posttrain optimizer fork. Hidden nn.Linear weights take the scale-free Newton-Schulz route; embeddings, heads, routers, norms, biases, convolutions and ambiguous coupled 2-D fusions take auxiliary AdamW. Native [experts, rows, cols] parameters opt into the shared stacked-expert path: an expert-axis FSDP shard contains complete local matrices, so it avoids gathering every expert on every rank, and gate/up halves are split before orthogonalisation. Dense sharded matrices still require one BF16 all-gather each. Parameters must be stored in FP32 (master_dtype=float32) while FSDP supplies BF16 compute. muon_rms_match sets the initial Moonlight LR scale, not equivalence to an eps-dominated Adam step; re-tune LR for the objective and response-length regime.

GSPO (Group Sequence Policy Optimization) is a resolved GRPO recipe in the same sense. Launch cosmos_framework.posttrain.entrypoints.grpo with configs/gspo_qwen3b_gsm8k.py:

PYTHONPATH=. python -m cosmos_framework.posttrain.entrypoints.grpo \
  --config cosmos_framework/posttrain/configs/gspo_qwen3b_gsm8k.py

The recipe records algorithm=gspo with its own job name for provenance and calls apply_gspo_knobs, the one place that turns a clipped rl section into the paper's objective: rl.grpo_surrogate=clipped, rl.importance_sampling_level=sequence, rl.lower_bound_ratio=0, rl.epsilon_low=3e-4, rl.epsilon_high=4e-4, rl.loss_type=seq-mean-token-mean, rl.mini_batch_per_replica=32 (four optimizer steps per 128-trajectory batch), and rl.certified_deterministic_forward=true.

The knob changes one thing inside the clipped surrogate. Instead of one ratio per token, every loss-masked token of response i carries the response ratio s_i = exp(z_i), where z_i is the mean over those tokens of log π_θ − log π_old, so the epsilon clip admits or drops whole responses. algorithm/primitives/policy_gradient.py::sequence_log_ratio implements the GSPO-token form sg[z_i] + (c_t − sg[c_t]), whose value, clip decision, and parameter gradient equal the paper's objective for the one scalar advantage per response that GRPO produces; the log-ratio is clamped from above at 10 and not from below. Frozen π_old, temperature, logprob_dtype, loss_type, and the ahead-pass attestation are inherited unchanged. Because the per-token loss is constant within a response, seq-mean-token-mean is the paper's equal-weight mean over responses, while token-mean weights each response by its length.

The response ratio differs from one only when π_old is not the current policy, that is at n_mini > 1 (the cached ahead pass) or with use_rollout_logprobs_for_loss=true. At n_mini == 1 without rollout logprobs every s_i is exactly 1, and the run trains as token-level GRPO in value and gradient while its identity records GSPO; the config warns. It also warns when the largest non-negative epsilon exceeds 0.05 and when logprob_dtype is not float32. These knobs are honored but leave the sequence clip inert or noisy, so they warn; knobs the sequence-level loss never reads are rejected instead: lower_bound_ratio != 0, grpo_surrogate=legacy_pg, OPD, and per-token (tensor) advantages, which the objective refuses inside its rank-converged prepare step because one ratio per response means one advantage per response. min_filter_prefix_tokens still applies (the base GSM8K recipe keeps 1); the response mean runs over exactly the tokens that stay in the loss.

The clipped objective emits two response-level diagnostics at both levels, weighted by train/valid_sequence_count: train/seq_clip_frac, the fraction of responses with at least one token whose loss the clip changed (all or none of a response at sequence level), and train/seq_log_ratio_abs, the mean over responses of |z_i|, the magnitude the sequence clip compares against ln(1 ± ε). Under sequence the token-weighted train/clip_frac and train/ratio_absdev become length-weighted response statistics, since a clipped response clips all of its tokens. The GSPO paper (sec. 5.2) reports a response clip fraction about two orders of magnitude above GRPO's token clip fraction, so a high train/seq_clip_frac is the expected reading rather than a defect. The objective identity records importance_sampling_level only when it is sequence, so existing clipped checkpoints resume unchanged and a level change refuses exact resume in either direction.

Columns are RENAMED, not projected: prompt_column becomes prompt and response_column becomes reference, and every other column passes through to obs.representations for a reward or callback to read. Those two names are the wiring vocabulary — DataPackers read prompt, while a selected provider declares whether it needs prompt and/or reference — which keeps rewards and packers reusable across datasets while config decides which column plays each role.

data.path wraps at epoch boundaries. Preflight opens the path only for readability. Full parsing and materialization happen when the data role is constructed—in its Ray actor, or in-process for the local backend. Every nonblank JSONL line must be an object. The configured prompt is either a string or a nonempty conversation list whose items have a string role and a content field; a configured response must be present and non-null. Conversation prompts are rendered through the selected model's chat template as described in datapacker.md. Hub-backed sources are loaded lazily by the data role; name/subset/split and column mappings apply to training and validation. Source/reward field compatibility and the limits of that check are documented in validation.md. Train-time held-out source, cadence, and sampling settings are documented separately in model_validation.md.

Two seeds reach a GRPO run. rl.seed orders the prompt source: data.shuffle permutes the corpus with rl.seed + epoch, and the arithmetic source draws from rl.seed. Top-level rollout.seed is the vLLM engine seed and decides the sampled completions. A "seed replicate" that changes only rollout.seed therefore reads the identical prompt sequence and varies the sampling alone, and a continuation pool built by removing a file prefix from such a run's data does not remove the rows the parent consumed, which under a shuffle are a seeded random subset. Preflight warns when data.shuffle=true, rl.seed=0 and a non-zero rollout.seed meet with the flag off ("all arms share one prompt order"). rl.data_seed_follows_rollout_seed=true keys the prompt source on rollout.seed, so one per-arm seed varies both; a per-arm rl.seed is the other correct spelling. config/rl.py::resolve_data_seed resolves the value once for preflight and the data role, and the cursor checkpoint (roles/env/cursor.py) records it, so an exact resume refuses a run whose data seed changed.

Non-finite loss rejection is mandatory and is not disabled by optimizer.force_finite_grads=false. Disabling the gradient check emits an operator warning and optim/finite_gradient_check_enabled=0. Both DiffusionTrainer (DPO/NFT) and LLMTrainer (GRPO/OPD plus PPO's policy and critic adapters) use the same reject-without-step meaning. The shipped DPO, GRPO, PPO, OPD, and NFT control loops fail on the first rejected update; there is currently no retry or bounded-continuation config.

Routing replay on a sparse-MoE policy

A sparse MoE's top-k selection over near-tied router scores can vary across otherwise identical forwards. Routing replay pins the recorded dispatch so importance ratios and reference scores measure policy changes instead of routing churn.

rl.routing_replay=rollout (Rollout Routing Replay) replays the engine's routing end to end:

  • Rollout. The vLLM engine is built with enable_return_routed_experts (part of the engine construction identity) and VLLMRollout records policy_info["rollout_routed_experts"] next to rollout_logprobs: the top-k expert ids of every input position of prompt + completion, [P+N-1, layers, top_k] (the last sampled token is never fed back, so it has no row), kept as uint8 up to 256 experts and narrowed to int16 beyond. The record is prompt-inclusive on purpose: the training forward recomputes the prompt positions and their KV feeds every completion logit, so a completion-only pin would leave the prompt's routing churn in place. The layer axis is vLLM's text-decoder layer index; the adapter checks its length against its own sparse-block count (equal on 30B-A3B, where every layer is sparse) and refuses anything else. An empty completion has no record and fails the rollout step loud under rollout, where the natural path skips it. The codec carries the record as bulk like every large per-token tensor; only its lane-agreement digest is inline.
  • Packer. shape_policy_input checks the record against the shaped input_ids (the stored prompt ids are asserted against the packer's own prefix, so a shifted prefix fails there and a wrong row count fails here) and every packer collates it token-aligned into the policy batch as [B, T, layers, top_k]; the unrecorded last token and the padding hold experts 0..top_k-1, a valid selection whose combine weight is zero on padded rows (data/packer/forward_routing.py).
  • Model. The Qwen3-VL-MoE adapter (model/protocol.py::ForwardReplayPolicy) consumes the batch key in prepare_policy_batch and pins every adapted block; the key never reaches the network. This batch-driven pin is the single writer: the ahead pass, the probe and the training forwards of a chunk are all pinned because they all prepare the same batch, and no trainer code pins.
  • Trainer. After every forward it checks that the pinned state matches the knob (refusing a replay run whose batches carry no routing and a natural run whose batches pin), applies the MoE adapter's relaxed probe tolerance (0.05 nats on at most 5% of tokens, mesh-wide) only while pinned, and reports train/routing_replay_pinned_tokens (the rows dispatched through the pin: prompt + completion positions, padding excluded when the packer supplies attention_mask) and train/routing_replay_pinned_blocks.

Constraints: the native Qwen3-VL-MoE policy with the vLLM rollout and the clipped surrogate only (assembly refuses dense and HF backends; legacy_pg and OPD refuse the knob at construction, since neither prepares its batch through the model capability). rollout is valid at n_mini == 1, where it is the only mechanism that closes the train/infer routing gap, and pairs naturally with use_rollout_logprobs_for_loss=true, which keeps its own contract: rollout.temperature == rl.temperature == 1.0, rollout.top_p=1.0, rollout.top_k disabled and rollout.repetition_penalty=1.0 (the engine returns raw, untempered logprobs). The engine-side capture is incompatible with vLLM pipeline parallelism and KV connectors, neither of which posttrain uses. An unpinned MoE run (off) is held to the same strict probe gate as a dense one.

OPD additions (rl.*)

OPDConfig inherits the GRPO fields above and adds one frozen teacher plus the distillation objective. See opd.md for the complete topology and algorithms.

field default meaning
distillation_mode advantage direct minimizes same-token divergence; advantage uses the policy-gradient surrogate and is required by DPCA
jsd_beta 1.0 same-token direction (0 forward KL, 1 reverse KL); set null for DPCA because it has no shared support
kl_penalty_coef 1.0 non-negative scale on the legacy same-token penalty or additive DPCA teacher credit
teacher.name_or_path Qwen2.5-7B M1 default frozen teacher artifact
teacher.tensor_parallel_size 1 ranked teacher WorkerGroup width; values greater than one require grid.backend=ray
teacher.moe_backend auto vLLM MoE kernel selection for the teacher; set an explicit supported backend such as triton when the architecture-specific automatic choice is incompatible with the teacher's shard shape or platform
teacher.input_mode shared_vocab_multimodal shared_vocab_multimodal, cross_token_multimodal, or cross_token_text; production DPCA sends original images through the teacher's own processor, while the text control pulls no media
teacher.signal_mode topk topk or dpca; controls the schema and objective interface
teacher.logprobs_mode processed_logprobs fixed vLLM likelihood convention. Generated-token logprobs follow processed semantics, while forced-sequence prompt logprobs remain raw, so teacher/policy scoring temperature is pinned to one.
teacher.top_k 32 support width for topk; must be exactly zero for DPCA
teacher.scoring_temperature 1.0 top-k teacher and policy scoring must both remain one because the forced-sequence vLLM path returns raw prompt logprobs. DPCA also requires one.
teacher.max_attempts / retry_backoff_s 2 / 0 bounded, replica-wide retries of one identical teacher request; backoff must be in [0, 60] seconds
teacher.conditional_think_prefix empty closed-think prefix applied exactly once after the teacher's assistant header; a template-provided equivalent is detected rather than duplicated
teacher.max_alignment_chunk_tokens 64 DPCA rejection guard; reaching it without exact decoded-text alignment fails the batch
teacher.min_abs_student_chunk_logprob 1e-8 DPCA numerical floor for abs(L_S); exact zero follows the unit-ratio/zero-credit convention, while nonzero values at or below the floor fail

Both modes require kl_beta=0 and use_rollout_logprobs_for_loss=false; advantage mode freezes π_old through its policy-side cache, while direct mode forms no importance ratio. Top-k direct OPD treats nonzero rollout sampling controls as data-generation choices, but its teacher and policy scoring temperatures must both remain one. Top-k advantage OPD additionally requires rollout temperature one, top-p one, top-k disabled, and neutral repetition penalty. DPCA requires all three temperatures and all rollout controls to be neutral. Every grouped rollout rejects temperature zero and the vLLM-rewritten (0, 0.01) range. The OPD entrypoint checks these cross-tree couplings before Ray, placement, or model loading. OPD fixes inherited grpo_surrogate=clipped so it cannot become a second distillation selector, and rejects importance_sampling_level other than token in both modes: its advantage objective scores a token-level ratio and direct mode forms none. direct rejects non-default logprob_dtype, clip, dual-clip, kl_penalty_coef, and advantage-normalization settings, and warns that task reward and group advantage do not enter its training loss.

RL knobs reference (rl.*, Cosmos3 online NFT)

It uses the native 93-decoded-frame / 24-latent-frame T2V contract. The reduced 29-frame comparison geometry belongs only to the optional research harness.

field v1 default meaning
max_steps / group_size 1000 / 4 applied-update target / global terminal-sample group; v1 requires one group member per rank
sample_steps / sampler_shift / guidance 20 / 5.0 / 3.0 native UniPC rollout settings; the sampler base shift remains 1.0
beta / kl_beta 1.0 / 1.0 DiffusionNFT direction strength / velocity-KL coefficient
policy_loss_scale 5.0 explicit policy-term normalization, independent of optimality-map bounds
modality video decoded media type (video or image); recipe qualification is selected separately by profile
profile empty explicit qualified-recipe contract; t2i_hpsv3_lora_720p_v1 pins the production T2I coordinates while the native T2V path remains unprofiled
advantage_low / advantage_high / advantage_eps -5 / 5 / 1e-4 optimality bounds and sample-standardization epsilon
old.build_old / old.update_every true / 10 moving behavior snapshot; refreshes after update 10, so update 11 consumes version 10
ref.build_ref / ref.update_every true / 0 frozen KL anchor; kl_beta=0 instead requires build_ref=false. The shared RefConfig has no model_dir (that is GRPO's GRPORefConfig knob), so an override of it here is an unknown key
scheduler.* WFM lambda-linear schedule warmup and decay advance only after applied updates
grad_accum 1 source rows staged per optimizer window; the whole window commits or rewinds together
rollout_seed_clock source_row stochastic-identity clock; the T2I predecessor contract selects optimizer_step so all accumulation rows reuse one rank-local seed set
init_keys_to_skip_loading empty exact source-only rl.init_from keys; valid only for the image copy policy and never applied to exact resume
reward.provider_revision required override immutable deployed scorer identity; the recipe sentinel is rejected before model allocation

Normal launch and manifest preflight validate the selected config and required reward environment-variable names before model allocation. Explicit preflight also instantiates the model-free dataloader and prepares its first T2V row, catching empty selection, caption-schema, conditioned-frame, and multi-row-packing failures on that selected path. It does not scan every row or probe the live reward service. The native video profile rejects conditioned I2V/V2V inputs and remains T2V-only.

Production v1 requires grid.backend=ray, grid.grouping=single, grid.gpu_placement=colocated, and uniform (1, group_size) shapes for data, rollout, reward, replay, and policy. The nft dispatch profile keeps the control loop free of topology markers while preserving each exact-rank WFM lane. Future disaggregation must declare new placement and routing contracts; it does not require a topology branch in online_nft_step.

The release-facing image profile is cosmos3_nft_t2i_public; its external artifact and validation contract is documented in the public NFT run guide.

The objective publishes merge-safe train/optimality_mean and train/optimality_second_moment. After the trainer's data-parallel reduction, the entrypoint reconstructs the population standard deviation as train/optimality_std. This avoids averaging a rank-local standard deviation, which would be zero when each rank owns one sample. Provider diagnostics use train/reward_provider/<provider>/<metric> and carry the configured provider revision as provenance. Accumulation-window source metrics use rollout/source_window_rows and rollout/source_committed_rows.

The experimental 8B recipe bounds its local working set to two iteration directories, with optimizer state kept only for the newest. The older directory remains model-loadable but is not a training-resume point. This lower recipe does not configure an archive destination; production recipes must supply one before treating local retention as durable publication.

Bounded video-reward settings

VideoRewardConfig binds the rank-local decoded-video adapter described in reward-providers.md. The model-owned decoder attests finite [-1,1] [1,3,T,H,W] output before the adapter sends one uint8 video, FPS, and one sample identity to the service. Raw scores become clamp(fp32(score), reward_clip_min, reward_clip_max) * scale exactly once.

Service URLs and the bearer token remain actor-environment inputs and never enter resolved config. Enqueue is not retried because the service exposes no idempotency key; only correlated result pulls are retryable. Connect/operation timeouts and one absolute acceptance deadline bound the call. The positional singleton result must be finite. These contracts make transport failure bounded and attributable; they do not prove live service availability, cancellation, or reward quality.

ImageRewardConfig is an additive specialization of the same bounded transport and provider-result contract. It requires image conditioning and decoder output [1,3,1,H,W], serializes CPU uint8 [1,H,W,3] with media_type=image, and omits video FPS metadata. The T2V request shape and default remain unchanged.