# Config system One posttrain experiment = an **inherited i4 model/data tree** (`defaults`) + a **typed `rl` section**, in ONE OmegaConf tree. The design serves three stakeholders at once: researchers keep their existing i4 experiments (inherit them verbatim), infra gets full CLI override, and every run dumps a resolved YAML that agents/repro tooling can reload. It is deliberately NOT Hydra-`_target_`-instantiate-everything — typed dataclasses + i4 LazyConfig inheritance + dotlist override (DESIGN §11.3 / appendix B.4). ## Selecting and overriding ```bash PYTHONPATH=. torchrun --nproc_per_node=4 --standalone -m cosmos_framework.posttrain.entrypoints.dpo -- \ experiment=cosmos3_ga_16bm8b_dpo_v1 \ rl.beta=1000 rl.optimizer.lr=5e-7 rl.max_steps=20 \ model.config.parallelism.data_parallel_shard_degree=4 ``` - `experiment=` selects a registered experiment (Hydra `ConfigStore`). - ONE dotlist (after `--`) covers **both** halves: RL hypers (`rl.*`, `grid.*`) and the inherited model tree (`model.*`) alike. - Every run writes `/config_resolved.yaml` — fully resolved, reload-able. GRPO attempts add a top-level `provenance` mapping to it and a `provenance..json` beside it: the tree hash, git commit, interpreter, and job that produced the attempt ([`../guides/grpo.md`](../guides/grpo.md) §7). ## Where fields live | subtree | what | schema | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | `model.*` | an inherited i4 LazyConfig tree, or `NativePolicyBuildSpec` for an online native policy | the i4 experiment or `config/model.py` | | `rl.*` | the RL half: shared lifecycle + optimizer, then algorithm knobs (objective/ref/data, group/sampling, actor-critic updates, teacher credit, or online WFM) | `config/rl.py::RunConfig` → `DPOConfig`/`GRPOConfig`/`PPOConfig`/`OPDConfig`/`NFTConfig` | | `grid.*` | orchestration: `backend` (local\|ray); DPO/PPO/NFT grouping; GRPO/PPO/OPD `gpu_placement` (colocated\|disaggregated); plus `ranks`, `num_replicas`, and per-role shapes | `config/rl.py::GridConfig` and algorithm-specific extensions | | `model_backend` / model config | model sourcing: `i4` (`model`, the default) \| `hf` (`model_hf`, explicit); an unsupported native VLM can select `hf` for image-conditioned clipped GRPO, which dispatches configs with `vision_config` to the generic HF VLM packer and propagates the policy's `trust_remote_code` decision to its model, rollout/vLLM, and packer consumers; legacy `qwen3_vl` normalizes to `i4` | `config/model.py::NativePolicyBuildSpec` / `HFModelConfig` | `config/build.py` keeps lazy imports, general parallelism resolution, `build_optimizer`, and a thin explicit backend dispatch. Model-family construction belongs to each `model/*.py`; each algorithm entry assembles its own stack. `config/config.py` holds the shared `PosttrainConfig(cosmos3.Config)` base + `build_root` (registration + job/trainer defaults); the runnable roots are `config/config_dpo.py`, `config/config_grpo.py`, `config/config_ppo.py`, `config/config_opd.py`, and `config/config_nft.py`. `PosttrainOPDConfig` inherits `PosttrainGRPOConfig`, preserves its established HF default, and overrides `algorithm` plus the typed `rl: OPDConfig` field; shared model specs remain on `PosttrainConfig`. `PosttrainPPOConfig` adds separate policy/reference/critic model configuration and a `PPOGridConfig`; the synchronous entry requires Ray, per-role grouping, and a local multi-role checkpoint store. An online `NativePolicyBuildSpec` names a dotted builder, an explicit `model_id` shared with rollout and tokenizer shaping, an opaque builder-owned `model_config`, and its data-parallel mesh. Generic entrypoints read `model_id` but do not require the opaque config to imitate Hugging Face fields or instantiate it on the driver. ## Adding your own experiment Drop a file in `config/experiment/dpo/` (auto-registered): ```python from cosmos_framework.utils.lazy_config import LazyDict from hydra.core.config_store import ConfigStore from cosmos_framework.posttrain.config.rl import DPOConfig, OptimizerConfig, PreferenceEnvConfig, RefConfig my_dpo_exp = LazyDict(dict( defaults=["/experiment/cosmos3_ga_16bm8b_v2_midtrain", "_self_"], # inherit a real i4 experiment rl=DPOConfig( beta=500.0, max_steps=2000, save_every=250, # freeze backbone, train only the OmniMoT gen-pathway params optimizer=OptimizerConfig(lr=1e-6, train_keys=["moe_gen", "time_embedder", "vae2llm", "llm2vae"]), ref=RefConfig(update_every=0), # 0 = frozen DPO anchor data=PreferenceEnvConfig(jsonl_path="/path/pref.jsonl", media="video", num_frames=121), ), job=dict(group="posttrain_dpo", name="my_dpo_exp"), ), flags={"allow_objects": True}) ConfigStore.instance().store(group="experiment", package="_global_", name="my_dpo_exp", node=my_dpo_exp) ``` Then `... entrypoints.dpo -- experiment=my_dpo_exp`. The shipped experiments (`cosmos3_ga_16bm8b_dpo_v1`, image variant `cosmos3_ga_16bm8b_t2i_dpo_v1`) inherit the 8B *test* experiment and use in-memory synthetic pairs (`rl.data.synthetic_pairs`), so they run with no external checkpoint/data. ## Parallelism — two layers, don't mix them - **Engine mesh** (FSDP/HSDP/CP/CFG-parallel) lives in the inherited `model.config.parallelism.*` — how ONE model instance shards. - **Orchestration** (`grid.*`) is the RL-framework layer above it — who hosts the role bodies (`backend`), how roles become `ActorGroup`s (`grouping`), how those groups share or split `DeviceGroup`s (`gpu_placement`), and each WG's execution shape. Use the same names for every role's orchestration shape: ```text R = *_REPLICAS # independent WG execution replicas K = *_RANKS # worker/process ranks inside each execution replica ``` For policy and rollout this means: ```text policy = (POLICY_REPLICAS, POLICY_RANKS) rollout = (ROLLOUT_REPLICAS, ROLLOUT_RANKS) POLICY_REPLICAS = 1 POLICY_RANKS = dp_replicate * dp_shard # implemented HSDP-only topology ROLLOUT_RANKS = rollout tensor-parallel ranks per engine ``` PPO additionally declares one frozen reference role of width `grid.reference_ranks` and one trainable critic role of width `grid.critic_ranks`. Their model parallelism is independent of physical placement; reference and critic retain separate GPU DeviceGroups in both PPO placement modes. For the current HSDP-only policy topology, `dp_replicate` and `dp_shard` are the complete Torch DeviceMesh dimensions inside the policy's `K` ranks. In the Ray backend, each `(R,K)` cell is one `_Host` actor; local execution has no Ray actor layer. They are not placement replicas: do not call `dp_replicate` simply "replica", map it to `grid.num_replicas`, or add framework-owned gradient reduction between policy execution replicas. FSDP owns gradient synchronization across the policy mesh. Rollout replicas, by contrast, are independent inference engines, each made of `ROLLOUT_RANKS` ranks. HF policies configure these dimensions as `model_hf.parallelism.dp_replicate` and `model_hf.parallelism.dp_shard`. A `NativePolicyBuildSpec` uses the same two field names under `model.parallelism`; a legacy inherited i4 tree continues to use `model.config.parallelism.data_parallel_{replicate,shard}_degree`. A concrete native builder may enforce a narrower contract; Qwen3-VL, for example, accepts pure FSDP only and rejects policy TP/EP/PP/CP and a replicate axis. **Per-role shapes (`grid.shapes`).** The fallback policy shape is `(1, ranks)` and the rollout shape is `(num_replicas, ranks)`. Set, for example, `grid.shapes = {"policy": [1, 64], "rollout": [16, 4]}` (dotlist: `grid.shapes.policy=[1,64] grid.shapes.rollout=[16,4]`) to give policy and rollout different execution shapes. The experience roles (data/rm/replay) derive at `(R_rollout, 1)`. Every axis must be `>= 1`, and policy `R` must be `1` under the normative contract. Shapes do not allocate GPUs or select a transport. For online GRPO and OPD, `grid.gpu_placement=colocated` (the default) places equal-product rollout/experience and policy ActorGroups in one GPU DeviceGroup; `grid.gpu_placement=disaggregated` assigns separate GPU DeviceGroups. OPD's teacher remains in its own pool in either case. Both entrypoints use the same lowering and derive `same_gpu` or `cross_device` from that structure before checking the independent `rl.weight_sync_transport` choice. By default, unequal policy/rollout products require the Ray disaggregated placement with `weight_sync_transport="nixl"`. Every policy actor materializes the full bucket; asymmetric routing assigns each rollout flat actor `q` directly to policy flat actor `q % POLICY_RANKS`. `weight_sync_r2r_broadcast=true` instead keeps that existing P2R transfer for rollout replica 0 and then broadcasts the engines' live parameters and persistent buffers between corresponding rollout ranks over NCCL. The GRPO entry enables exact-byte value verification on its initial sync only; steady-state sync skips that full-state device-to-host proof. The R2R phase is also independently callable, with optional verification, after replica 0 loads an eval checkpoint. The OPD entry currently requires one rollout replica and rejects this option. Wider policy pools retain explicit idle actors. Ray GPU fractions are private binder details; there is no fractional topology knob. `grid.gpus_per_role` remains a reserved, unconsumed field. For an eval-only rollout, bind with `bind_device_groups(..., weight_sync_r2r_broadcast=True)` without a trainer weight edge, load the desired checkpoint into replica 0, then call `control.loop.weight_sync.r2r_weight_sync(rollout, model_step=version)`. The same R2R operation and two-phase version publication used by online relay mode run without a policy role. Online GRPO/OPD requires explicit `grid.shapes` when `grid.num_replicas > 1`; the legacy scalar no longer implicitly creates policy execution replicas. This forces policy and rollout widths to be visible rather than silently choosing an unrunnable same-GPU unequal-product topology. PPO fields and launch constraints are summarized in [`../guides/ppo.md`](../guides/ppo.md); the typed source of truth is `config/rl.py::PPOConfig` plus `config/config_ppo.py` validation. ### Shared checkpoint component controls All training entries use the logical component names `model`, `optimizer`, and `training_state`. `rl.checkpoint_components` controls cadence and normal-final saves and defaults to the full `[model, optimizer, training_state]` plan. `rl.validation.checkpoint_components` has the same full default, but validation does not request a save unless `rl.validation.save_checkpoint=true`; that switch defaults to `false`. If regular saving and validation coincide, their plans are unioned and written once. Only the full plan is exact-resume capable. When a proper subset becomes active as the regular run plan or for a validation save, the checkpoint manager emits `UserWarning`; a resulting artifact is committed under the sibling `checkpoint_artifacts/iter_*` namespace without advancing `latest_checkpoint.txt`. An inactive validation plan does not warn. Empty, duplicate, non-string, and unknown names are rejected. Emergency saves always override both selections with the full plan. `rl.keep_optim_last=null` is backend-aware automatic policy: a local primary resolves it to `1`, while a direct object-store primary resolves it to `0` because remote pruning is unsupported. Explicit `0` keeps all local optimizer state; another non-negative integer selects that many newest full iterations. See [`checkpoint.md`](checkpoint.md#logical-component-selection) for physical component mapping, artifact layout, and resume semantics. No knob controls the `.uploaded` archive marker that archived checkpoints and artifacts carry; its semantics are described in [`checkpoint.md`](checkpoint.md#local-checkpoint-archive-upload-rlupload). ## RL knobs reference (`rl.*`, DPO) | field | default | meaning | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `beta` | 500.0 | DPO temperature (validated > 0) | | `max_steps` / `pairs_per_step` | 1000 / 1 | loop length / pairs per step | | `grad_accum` | 1 | optimizer-step cadence | | `deterministic` | false | bit-reproducible runs: cuDNN deterministic, `use_deterministic_algorithms(warn_only=True)`, `FLASH_ATTENTION_DETERMINISTIC=1`. Costs throughput, so it is opt-in and meant for equivalence gates, not production. **Export `CUBLAS_WORKSPACE_CONFIG=:4096:8` and `PYTHONHASHSEED=42` in the LAUNCH environment** — cuBLAS reads its variable when it creates its handle, so setting it from Python is silently too late, and Ray actors are separate processes that inherit only what the launch gave them. The helper raises rather than warns if CUDA is already up (`utils/determinism.py`). Reproducibility also needs deterministic prompt ids, which `roles/env/dataset.py::_stamp_prompt_id` now provides unconditionally. | | `optimizer.*` | AdamW lr=1e-6, `train_keys=[]` (train all) | RL-owned; a non-empty `train_keys` freezes everything not matching it (fails loud if nothing matches). The cosmos3 recipes set the OmniMoT gen-pathway subset (`moe_gen`/`time_embedder`/`vae2llm`/`llm2vae`). | | `optimizer.name` | `adamw` | `fusedadam` = apex-fused AdamW, fp32 master weights + fp32 moments — REQUIRED for real bf16-stored nets (see `utils/fused_adam.py`). `muon` reuses `cosmos3.utils.MuonWithAuxAdamW`: hidden `nn.Linear` weights and explicitly declared stacked MoE experts take Muon; embeddings, output heads, routers, norms, biases, convolutions, coupled 2-D gate/up and qkv fusions, and other parameters take the auxiliary AdamW route. Native expert stacks are processed from complete local expert-axis shards without a full-stack all-gather, and fused expert gate/up halves are orthogonalised separately. Requires `master_dtype=float32`, BF16 FSDP compute, Transformer Engine and GPU | | `optimizer.muon_momentum` / `optimizer.muon_ns_steps` / `optimizer.muon_rms_match` | `0.95` / `5` / `0.2` | `name=muon` only: the momentum β of the Muon buffer (`m ← β·m + g`, Nesterov direction `g + β·m`), the positive-integer Newton-Schulz iteration count, and the finite-positive Moonlight LR scale (`× sqrt(max(rows, cols))`). The inherited AdamW LR is only an initialization heuristic: Muon is scale-free while Adam can be eps-dominated, so a Muon run must re-tune LR. Strictly validated at both config and optimizer construction; inert for other optimizer names | | `optimizer.eps` | `1e-8` | Adam's denominator floor, `update = lr · m / (√v + eps)` (torch's default). A gradient-SCALE knob, not a stability nicety: once `√v` for a parameter falls below `eps`, that parameter's update degenerates to `(lr/eps) · m`, an SGD-like step whose size follows `lr/eps` rather than `lr`. The cosmos-rl-derived GRPO recipes carry `1e-6` (100× the default), sized for `token-mean` gradients; under `loss_type=seq-mean-token-sum-fixed` the gradients fall below it and `GRPOConfig` warns when `eps > 1e-7` (refuses the pairing under `recipe_guard_severity=error`). The recipe layer that selects the fixed reduction together with `1e-8` is `the Dr-GRPO recipe layer` — see the Dr-GRPO paragraph in the GRPO section | | `optimizer.grad_clip_norm` | 0.0 | clip the global gradient norm before an applied update; 0 disables clipping | | `optimizer.force_finite_grads` | true | require rank-consistent finite gradients and reject the entire accumulation window otherwise; false is an explicit legacy opt-out for this gradient check only | | `ref.build_ref` / `ref.update_every` | true / 0 | frozen anchor (0) vs moving ref (>0); moving mode requires an adapter with `refresh_ref()`, and current Cosmos3 DPO recipes use 0. The shared `RefConfig` has no anchor-source field: DPO anchors its reference through `init_from`, and a `ref.model_dir` override (GRPO's `GRPORefConfig` knob) is refused here as an unknown key | | `init_keys_to_skip_loading` | empty | exact source-only `init_from` keys the DPO backbone does not build; any other source-only key fails the copy-mode load, and exact resume never consults the list | | `data.*` | — | preference JSONL path (or `tar_path` offsets, or `synthetic_pairs>0` for in-memory toy data), media/resolution/frames, cfg-dropout, shuffle (the run seed is `rl.seed`) | | `seed` | 0 | the run seed (`RunConfig`): the process RNGs (`set_random_seed`) and the preference-source shuffle derive from it | | `save_every`, `resume`, `run_ts`, `checkpoint_format`, `checkpoint_components`, `checkpoint_store.*`, `save_async`, `keep_optim_last`, `keep_model_last`, `upload.*` | shared `RunConfig` defaults | `checkpoint_components` defaults full; `keep_optim_last=null` resolves to local `1` or direct `0`; `checkpoint_store` selects the primary resumable store, while `upload` is only an optional archive of a local primary. See [`checkpoint.md`](checkpoint.md) | ## Rollout engine knobs (`rollout.*`, vLLM) | field | default | meaning | | ----------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `limit_mm_per_prompt` | `None` | Per-REQUEST multimodal item budget forwarded to `vllm.LLM(limit_mm_per_prompt=...)` only when set, e.g. `{"image": 40}`; `None` leaves the key out, so the engine construction identity of every other run is unchanged and the engine's own default applies (verified in the pinned container: vLLM 0.24.0, `BaseDummyOptions.count` 999 and `get_limit_per_prompt()` 999 for an unspecified modality, the 0.23.0 wheel pin carries the same code — so the engine's own default is 999 items per supported modality and setting a value LOWERS the budget; it is the lever for bounding the encoder memory the engine profiles for and for refusing over-budget prompts at admission, `At most N image(s) may be provided in one request`). `0` disables the modality (a prompt carrying it is refused). Keys must be bare lower-case vLLM modality names from `{image, video, audio}`: surrounding whitespace, punctuation or an unknown name (`images`) is refused at config time, and the built engine's model-specific supported set is checked again at `VLLMRollout` construction, because vLLM itself converts an unknown key into a generic dummy option and silently ignores it. Raising the limit makes vLLM profile more encoder memory at start-up (fewer KV blocks under the same `gpu_memory_utilization`); lowering it caps what one prompt may carry. Validation: non-empty mapping, non-bool integers >= 0. | | `compilation_config` | `{}` | vLLM `CompilationConfig` as a mapping, forwarded as `vllm.LLM(compilation_config=...)` only when non-empty. On the command line use Hydra's add-or-override form, `++rollout.compilation_config={cudagraph_mode:FULL_DECODE_ONLY,cudagraph_capture_sizes:[1,2,4,8,16]}` (or `+rollout.compilation_config.=...` per key): the composed config is in struct mode, so the plain `key={...}` form cannot add keys. It configures torch.compile and CUDA-graph capture, so a non-empty mapping requires `enforce_eager=false` (refused otherwise). Empty means unset because Hydra applies a dict override by merging, which a `None` default cannot take. Hydra mappings are materialized to plain dicts and lists; the engine receives a deep copy. | | `async_scheduling` | `None` | vLLM `async_scheduling`, forwarded only when set: `true` overlaps scheduling of the next engine step with the current step's GPU work. | | `max_num_seqs` | `None` | vLLM `max_num_seqs`, forwarded only when set: the cap on sequences one engine runs at once, which also bounds the CUDA-graph capture sizes vLLM derives. Positive non-bool integer. | | `kv_cache_memory_bytes` | `None` | vLLM `kv_cache_memory_bytes`, forwarded only when set: a fixed per-GPU KV-cache size in bytes instead of the size derived from `gpu_memory_utilization` after the memory profile. Positive non-bool integer. | | `load_format` | `auto` | vLLM model-weight loader: `auto` or `dummy`. `dummy` skips checkpoint tensor I/O and starts from random weights; the rollout refuses to generate until a weight publication commits. | | `custom_all_reduce` | `false` | Full-node tensor-parallel engines only, with vLLM-supported world size 2, 4, 6 or 8: `true` forwards `disable_custom_all_reduce=false`, so vLLM all-reduces with its custom kernel instead of NCCL. The TP replica must consume exactly the node's GPU capacity; partial-node/multi-replica and host-affinity placements are refused because the kernel maps every visible peer's buffers over CUDA IPC. GRPO shows the full-node experience replica its GPUs with each actor's own first (`cuda:0` stays its device; the colocated trainer keeps one GPU). In this layout every engine rank reports device 0, which PyTorch symmetric memory refuses, so FlashInfer's all-reduce + RMSNorm fusion stays off; the custom all-reduce itself runs. PPO rejects it during preflight because its rollout actors keep one visible GPU. | ## RL knobs reference (`rl.*`, GRPO) | field | default | meaning | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `grpo_surrogate` | `clipped` | GRPO policy-gradient surrogate: production frozen-π_old clipped loss, or `legacy_pg` for the minimal toy/parity loss. The default arithmetic root pins `legacy_pg` explicitly. | | `temperature` | `1.0` | divides policy logits for both `legacy_pg` and clipped scoring and must equal top-level `rollout.temperature`. Grouped vLLM rollout rejects zero and rewrites nonzero values below `0.01`, so GPU-less preflight rejects both ranges rather than accepting a different effective behavior distribution. | | `loss_type` | `token-mean` | clipped-loss normalization: `token-mean`, `seq-mean-token-sum`, `seq-mean-token-mean`, or Dr-GRPO's `seq-mean-token-sum-fixed`, which divides by fixed `rollout.max_tokens`. The fixed divisor shrinks the gradients Adam sees, so the config warns when it pairs with `optimizer.eps > 1e-7` — refuses it under `recipe_guard_severity=error` (Dr-GRPO paragraph below). | | `logprob_dtype` | `float32` | dtype used for clipped-surrogate `log_softmax`, independent of model compute dtype | | `param_dtype` / `master_dtype` / `reduce_dtype` / `transfer_dtype` | `""` (inherit: `model_hf.dtype` / the resolved `param_dtype` / `model_hf.fsdp_reduce_dtype` / the resolved `master_dtype`) | The FSDP2 precision block (`config/rl.py::HFRunConfig`): compute dtype, storage dtype (also the optimizer-moment dtype), gradient all-reduce dtype, and the dtype weights are cast to on the policy → rollout hop. `master_dtype="float32"` + `param_dtype="bfloat16"` is the fp32-master/bf16-compute recipe. | | `mixed_precision_mode` | `""` (⇒ `fsdp`) | How `param_dtype` reaches the forward. `fsdp`: the `MixedPrecisionPolicy` parameter cast, which needs an FSDP wrap whenever `master_dtype != param_dtype` (refused unwrapped). `autocast`: parameters stay in `master_dtype` and the forward runs under `torch.autocast(param_dtype)` at every world size, wrapped or not — numerically different from `fsdp` because autocast is per-op (what HF `TrainingArguments(bf16=True)` does). Refused with `param_dtype="float16"` (no GradScaler) and on `model_backend="i4"`. | | `epsilon_low` / `epsilon_high` | `0.2` / `0.2` | clipped surrogate ratio bounds; a negative value disables that side of the clip. Under `importance_sampling_level=sequence` they bound a response-mean log-ratio, which sits far below a token's, so the config warns when the largest non-negative epsilon exceeds `0.05` (the GSPO recipe uses `3e-4` / `4e-4`). | | `lower_bound_ratio` | `3.0` | dual-clip lower bound for negative-advantage tokens; zero disables it. Must be `0` under `importance_sampling_level=sequence`, which applies no dual clip to the response ratio and rejects any other value. | | `importance_sampling_level` | `token` | granularity of the clipped surrogate's importance ratio. `token` scores one ratio per token (GRPO/PPO). `sequence` scores one ratio per response, `exp` of the mean token log-ratio over that response's loss-masked tokens, shared by all of them so the clip admits or drops whole responses (GSPO). Consumed only by `clipped`; `legacy_pg` and OPD reject `sequence`. See the GSPO paragraph below. | | `kl_beta` | `0.0` | reference-policy KL scale for the clipped surrogate. Positive values add `kl_beta · k3(logπ_ref − logπ_θ)` to every loss-masked completion token's loss (`k3 = exp(z) − z − 1 ≥ 0`, the DeepSeekMath GRPO estimator of `KL(π_θ‖π_ref)`; the log-ratio is clamped at ±20 nats like PPO's `sampled_k3_kl`), pulling the policy toward a frozen reference. Requires `ref.build_ref=true`; refused with `legacy_pg` and by OPD. Logged as `train/kl_mean` (token-weighted mean of the unscaled `k3`), `train/kl_max` (the largest per-token `k3` in the batch, MAX-reduced across ranks and chunks) and `train/kl_beta`. Value and gradient name two divergences: the logged `k3` estimates the forward `KL(π_θ‖π_ref)`, while the gradient the sampled loss applies is that of `KL(π_ref‖π_θ)`, the mode-covering pull toward the reference (the DeepSeekMath convention). Under `routing_replay=rollout` the reference forward prepares the same batch as the policy forward and so pins to the same recorded expert routing: `z` then measures weight drift, not routing churn (see that row). Without `kl_coeff_clamp` the config warns: the exact k3 gradient is unbounded (next row) | | `kl_coeff_clamp` | `null` | bound, in nats, on the KL term's per-token **gradient** coefficient; `null` keeps the exact k3 gradient `1 − exp(z)`, byte-identical to before the knob existed. That coefficient is unbounded below inside the ±20 nat log-ratio clamp (−4.85e8 just inside it), so one token whose reference forward disagrees with the policy forward by a few nats owns the whole step — routing nondeterminism can make a small number of coefficients dominate a step. With `kl_coeff_clamp=c` the penalty's VALUE (and every `train/kl_*` statistic) stays the exact k3 while its gradient is that of k3's linear extension beyond `\|z\| = c`: coefficient `1 − exp(clamp(z, −c, c))`, at most `exp(c) − 1` in magnitude (`5.0` recommended: ≤ 147) and still pulling toward the reference past the band. Must be positive and below 20 (the ±20 clamp already zeroes the gradient beyond it, so a larger value is inert and refused); requires `kl_beta>0`; recorded in the objective identity only when set. Implementation: `algorithm/primitives/policy_gradient.py::sampled_k3_penalty` | | `ref.build_ref` / `ref.update_every` | false / 0 | frozen KL reference for `kl_beta>0`: a second FSDP-sharded, gradient-free instance of the policy built by the same builder from the same base checkpoint on every start (so it is always the run's step-0 weights; a resume restores the policy only), scored under `no_grad` on every training chunk (not on the π_old ahead pass or the determinism probe at `n_mini > 1`, whose losses are discarded) and offloaded with the policy under colocation. Costs one extra copy of the network per policy rank. `build_trainer` refuses a policy with live dropout (an `nn.Dropout` with `p > 0` or a functional `attention_dropout`-style config value) before building the twin: the reference scores under eval() while the policy trains under train(), so the log-ratio would be a dropout mask at any `n_mini`. `build_ref=true` requires `kl_beta>0` and vice versa; `update_every` must be 0 (no moving reference for the LLM trainer) | | `ref.model_dir` | `null` | GRPO-only field of `GRPORefConfig` (the type of GRPO's `ref`; a plain `RefConfig` is adopted with `model_dir=null`). Warm restart for model-only saves: build the frozen KL reference from THIS checkpoint directory instead of the policy's own (`model_qwen3_vl.name_or_path`/`checkpoint_path`, `model.model_id`, or `model_hf.name_or_path`), so a run whose policy starts from its latest save keeps its anchor at the ORIGINAL base. Requires `build_ref=true` (checked at config load). Must be a directory when the trainer is built: `build_trainer` checks on every rank before any model is built, not the dataclass, so a saved config reloads on a host without the checkpoint mount; the trainer logs `built frozen KL reference from (rl.ref.model_dir)`. The directory's exact CONTENT identity (`utils/artifact_identity.py`: full SHA-256 of every metadata and weight file, read once on rank 0) joins the objective's exact-resume identity as `reference_artifact`, so an exact resume that names another checkpoint, or the same path re-populated with other weights, is refused rather than silently anchored elsewhere; a directory without weight files is refused at build. The full read can add startup latency for large references, but it does not recur per step. Consumed by the GRPO entry only; the shared `RefConfig` DPO and NFT use has no such field, so `rl.ref.model_dir` on those configs is an unknown key | | `unbiased_advantage` | false | false uses `(reward - group mean) / group std`; true mean-centres without the std division. Dr-GRPO pins true. | | `use_rollout_logprobs_for_loss` | false | false recomputes/fixes π_old in the policy path; true consumes the rollout engine's raw (temperature-1, unwarped) logprobs as π_old and disables the ahead-pass cache, so preflight additionally requires `rollout.temperature == rl.temperature == 1.0` (top-p/top-k/repetition-penalty warping is already rejected for every on-policy GRPO run). `train/logprob_gap_*` reports the train/infer gap in either setting | | `min_filter_prefix_tokens` | `null` | minimum shared-prefix span to mask when members of one group share that prefix but receive different rewards; the complete group is annotated before dispatch | | `mask_truncated_completions` | false | DAPO overlong filtering (arXiv 2503.14476 §3.3). `true` zeroes the loss mask of every completion the rollout engine cut at its token budget (`policy_info["finish_reason"] == "length"`, stamped by the vLLM rollout from `CompletionOutput.finish_reason`; the HF and local rollouts stamp none, so nothing is masked there), so it enters neither the clipped surrogate nor the KL term nor their token/sequence divisors — the same exit an empty completion takes. `false` trains on it like any other completion. The mask removes the completion from the loss only (DAPO's definition): its reward stayed in the group mean/std the reward stage computed before dispatch, so the kept members train on their full-group advantages, and the group still counts toward the informative-group gate. The rate diagnostics `train/rollout_truncated_frac`, `train/response_len_mean` and `train/response_len_max` report either way. Consumed by `grpo_surrogate=clipped` only; `legacy_pg` and OPD refuse `true`. Joins the objective identity only when `true`, so existing clipped checkpoints resume unchanged | | `recipe_guard_severity` | `warn` | how the two recipe-hygiene checks report. Both pairings — a literal `min_informative_groups` below half of `groups_per_step` (dynamic sampling in name only on a wider layout) and `loss_type=seq-mean-token-sum-fixed` with `optimizer.eps > 1e-7` (the reported lr is not the step size) — are honored as written, so `warn` (default) WARNS at construction; existing fields and default runtime behavior remain unchanged. `error` collects every hazard found and refuses the config in ONE `ValueError` that lists them all (each hazard's text as the warning has it, joined by a blank line, then one sentence naming the knob), for a recipe that wants a mis-pairing caught at preflight: `--validate-manifest` re-instantiates `rl` after the dotted overrides, so `rl.recipe_guard_severity=error` on a command line is enough, and the Dr-GRPO Qwen3-VL RLVR layer (`the Dr-GRPO recipe layer`) sets it in the recipe. Thresholds and messages are the same in both modes; any other value is refused; keyword-only; inherited by OPD, which refuses both pairings outright anyway | | `score_centering.enabled` / `top_k` / `compose` / `tis_clip` / `mis_low` / `mis_high` | false / 128 / `none` / 2.0 / 0.5 / 5.0 | score centering of the policy gradient under the SAMPLER's distribution (`algorithm/primitives/score_centering.py`). `enabled=true` builds the vLLM engine with `logprobs_mode=processed_logprobs`, `max_logprobs=top_k`, requests `SamplingParams(logprobs=top_k)`, stamps `policy_info["rollout_topk_ids"]` / `["rollout_topk_logprobs"]` (`[T, k]`, int32 / float32 — never float16: a rounded head gives q != p for an identical sampler and trainer) on every training completion, and the clipped objective adds `A · Σ_{v∈H} sg[q_v w_v − α p_v] log p_v` per token (head `H` = the sampler's top-k, `ρ = q_tail/p_tail`, `α = ρ f(1/ρ)`) and scales the surrogate by `sg[f(p_{y_t}/q_{y_t})]`, so the expected gradient under the sampler is drift-free. `compose=none` is pure centering (`f = 1`, weight exactly 1); `tis` composes truncated IS `f(r) = min(r, tis_clip)`; `mis` masked IS `f(r) = r · 1[mis_low ≤ r ≤ mis_high]`. The KL term, the clip diagnostics and the loss reduction are untouched; with `q == p` the correction is exactly 0 and the loss bit-identical to `enabled=false`. Requires `grpo_surrogate=clipped`, `importance_sampling_level=token`, `n_mini == 1`, `use_rollout_logprobs_for_loss=false` (refused otherwise; OPD refuses it). Under it `train/logprob_gap_*` compares TEMPERED logprobs on both sides. Diagnostics: `train/sc_*` (metrics reference). Joins the objective identity when on. | | `mu_iterations` | `1` | optimizer passes per collected batch in the upstream vocabulary; values other than one are rejected because the local loop expresses reuse through `n_mini` | | `certified_deterministic_forward` | false | required when a clipped/advantage objective uses a cached π_old across `n_mini > 1`; attests that the ahead and training forwards are equivalent. The trainer still runs a one-shot eval-vs-train probe on the first ahead chunk, gated by the tolerance the model declares (`ForwardNoisePolicy`; strict `0.05` nats on any token by default). See the routing-replay paragraph below for why a sparse-MoE policy fails it at `n_mini > 1`. | | `routing_replay` | `off` | sparse-MoE routing replay (Rollout Routing Replay, Ma et al., arXiv 2510.11370; DeepSeek's variant is "Keep Routing"): pin each forward's top-k expert dispatch to the recorded selection the policy batch carries, through the model's `ForwardReplayPolicy` capability, so the training forward routes as the rollout did and the importance ratio measures policy change instead of routing churn. `rollout` replays the vLLM engine's own routing: the engine is built with `enable_return_routed_experts`, the rollout records `policy_info["rollout_routed_experts"]`, the packer aligns it to the policy batch and the MoE adapter pins every forward (ahead pass, probe, training) from it. With `kl_beta > 0` the frozen reference prepares the same batch through its own adapter, so its forward is pinned to the same recorded routing: the KL anchor is π_ref under the rollout's expert dispatch, and `z` measures weight drift rather than routing churn (an unpinned reference would put the routing flip back into `z`). Clipped surrogate only (`grpo_surrogate=clipped`, the default; `legacy_pg` refuses it because that objective never prepares its batch through the model, so nothing would pin); valid at `n_mini == 1` and with `use_rollout_logprobs_for_loss=true`. Any other value is refused at construction, OPD refuses non-`off`, and assembly refuses a dense or HF policy. A non-`off` value joins the exact-resume identity together with the model's `replay_identity()`, the engine flag joins the rollout identity, and the trainer refuses a forward whose pinned state contradicts the knob. See the section below. | | `max_steps` | 1000 | collected batches. Each batch commits `n_mini` optimizer/model steps; validation, reports, and `iter_*` names use that model-step clock. | | `batch_per_replica` / `group_size` | 32 / 8 | trajectories the policy trains on per policy execution replica per loop step / completions sampled per prompt. It is not per `dp_replicate` slice. `batch_per_replica` is the PRIMARY sizing knob; it must be a positive multiple of `group_size`. | | `prompts_per_data_replica` | DERIVED (`batch_per_replica // group_size`) — read-only property | compatibility convenience for exactly one data execution replica. With multiple rollout/data replicas, `control.loop.batching.prompts_per_data_cell` divides this total prompt count evenly across them. | | `mini_batch_per_replica` | 0 ⇒ the whole batch | trajectories per optimizer step in one policy execution replica; must evenly divide `batch_per_replica` (no ragged last mini-batch). Derives `n_mini = batch_per_replica // mini_batch_per_replica` | | `micro_batch_per_replica` | 0 ⇒ one chunk | trajectories per forward/backward in one policy execution replica, not per HSDP `dp_replicate`; must evenly divide the mini-batch span (`mini_batch_per_replica` or, if unset, `batch_per_replica`). Derives `n_micro = // micro_batch_per_replica`, which is also the trainer's `grad_accum` (see `checkpoint.md`). The trainer accumulates chunk sums and applies the divisor once per optimizer window, so changing this memory knob does not change the update; `test_roles_trainer_deferred_norm.py::test_chunked_gradient_equals_unchunked` pins the real trainer invariant. | | `min_informative_groups` | 0 ⇒ dynamic sampling OFF | DAPO-style uniform-group filtering (arXiv 2503.14476 §3.2). When > 0, the replay buffer gates on the reward stage's stamped group reward spread — the only owner of "informative" — so whole groups whose rewards tie (zero advantage) are dropped before the optimizer step, and collection keeps drawing until the replay cells together hold this many informative groups (a global count that `control.loop.batching.informative_quota_per_cell` splits exactly across the data cells, so it does not grow with the replica count). The batch is assembled informative groups first at `batch_per_replica`; tied groups fill the remainder whenever this is below `batch_per_replica // group_size` (full DAPO sets it equal). Must satisfy `min_informative_groups * group_size <= batch_per_replica`. GRPO-only: `OPDConfig` rejects it. A literal count below half of `groups_per_step` (= `batch_per_replica // group_size`) warns at config time — the message names `groups_per_step`, the count and `min_informative_groups_frac` — because the count is global and does not scale with the batch: the `8` that is every group of `b32g4` is 1/8 of `b256g4`'s 64 groups, i.e. dynamic sampling in name only. Prefer `min_informative_groups_frac` for a recipe that moves between batch layouts | | `min_informative_groups_frac` | `null` ⇒ the literal count above is the quota | the same quota spelled as a SHARE of the groups the batch holds per step: quota = `max(1, ceil(frac × groups_per_step))`, `0 < frac ≤ 1` (`1.0` = full DAPO on any layout; never 0, so a set fraction is always ON; the fraction is read as the nearest rational with denominator ≤ 10⁶ before the ceil, so a share that names a whole number of groups is exactly that number: `0.14` of 50 groups is 7, where the float product `7.000000000000001` would ceil to 8). `groups_per_step = batch_per_replica // group_size` is placement-independent — the data cells split that one batch — so the share survives a change of batch layout where a literal count does not. Setting both this and `min_informative_groups > 0` is rejected; leave the literal at 0. A literal count below half of `groups_per_step` warns at construction (the message names `groups_per_step`, the count and this key); `recipe_guard_severity=error` refuses it. GRPO-only: `OPDConfig` rejects it | | `groups_per_step` / `effective_min_informative_groups` | DERIVED — read-only properties | the group capacity of one step (`batch_per_replica // group_size`) and the informative-group quota the collection loop enforces: the literal count, or `max(1, ceil(min_informative_groups_frac × groups_per_step))` when the fraction is set. `entrypoints/grpo.py` reads only the latter and logs both, with the two knobs, at start-up when dynamic sampling is on | | `dynamic_sampling_max_extra_rollouts` | 4 | top-up budget in extra ROUNDS per loop step (one more draw of `prompts_per_data_cell` prompts on each participating cell); how many `rollout.generate` calls those rounds take is set by `dynamic_sampling_refill_dispatch` and counted by `train/grpo_dynamic_sampling_refill_dispatches`; only consulted when the resolved quota (`effective_min_informative_groups`) is > 0 | | `dynamic_sampling_refill_mode` | `all` | `all` refills every cell; opt-in `short_cells` refills only cells still below their informative quota. The initial draw stays full everywhere, preserving batch capacity even for zero-quota cells. This changes subsequent source/engine histories and can retain tied filler at fractional quotas; it does not promise identical batches or learning curves. | | `dynamic_sampling_quota_scope` | `cell` | `cell` requires every replay cell to buffer its exact share of the quota; opt-in `pooled` stops once the batch assembled informative-first holds the quota (each cell still contributes at most its batch share) and treats only cells whose share is not yet informative as short. Both enforce the same minimum quota on the trained batch, but the retained informative-group count and the batch composition can differ: `cell` keeps refilling owing cells after the batch already holds the quota, while `pooled` does not fix which cells supply it and avoids spending the whole budget when every cell draws one prompt. | | `dynamic_sampling_refill_dispatch` | `per_round` | number of top-up rounds in one `rollout.generate` call, orthogonal to `dynamic_sampling_refill_mode`: `per_round` issues one round per call and re-counts short cells; `whole_budget` issues all remaining rounds in one call, so each participating cell serves `remaining_rounds × prompts_per_data_cell` prompts while cells idled by `short_cells` serve none. The wider call can advance the cursor farther and replace tied filler below full DAPO when `per_round` would have stopped early. `train/grpo_dynamic_sampling_refill_dispatches` counts calls, while `train/grpo_dynamic_sampling_extra_rollouts` retains the round unit. The widened request (per-engine width `rounds per call × prompts_per_data_cell × group_size` sequences when data cells map 1:1 to engines; the start-up line prints it) must stay within the engine's `max_num_seqs`: the framework passes none, so vLLM's usage-context default applies — 1024 for the `LLM` class on ≥ 70 GiB GPUs in vLLM 0.23.0. The data plane carries a `budget ×` larger payload per publish: `bulk_store` has no page pool to exhaust, while `ray_nixl_pool` has a byte capacity to check against the widened payload before the first run. | | `dynamic_sampling_on_budget_exhausted` | `raise` | what happens when the budget is spent and a replay cell is still short: `raise` aborts the run (the reward regime cannot supply informative groups — a recipe fault); `proceed` trains on tied groups as zero-advantage filler and logs the shortfall | | `grad_accum` | 1, and REJECTED if set to anything else | retired as a GRPO knob — it meant "that many fresh rollouts per step", which the single-draw batch above replaces; use `micro_batch_per_replica` instead. Still a real DPO knob (`grad_accum` above) | | `save_every`, `resume`, `run_ts`, `checkpoint_format`, `checkpoint_components`, `checkpoint_store.*`, `save_async`, `keep_optim_last`, `keep_model_last`, `upload.*` | shared `RunConfig` defaults | same backend-neutral `CheckpointLifecycle` as DPO; cadence/final component selection defaults full, `keep_optim_last=null` is backend-aware, and `upload` remains a local-after-save archive. Cadence is in collected batches while checkpoint names and validation use committed optimizer/model steps. See [`checkpoint.md`](checkpoint.md) | | `data.name` / `data.subset` / `data.split` | empty / empty / `train` | Hugging Face dataset address. `subset` is the HF *config* name (e.g. `main`) and is required with `name` — empty reaches `load_dataset` as `""` and fails at actor setup | | `data.path` | empty | a local JSONL; with neither `name` nor `path`, the built-in arithmetic source is used | | `data.prompt_column` | empty | required with `data.path` or `data.name`; renamed to the observation's `prompt` field | | `data.response_column` | empty | ground-truth column, renamed to `reference`; required with `data.name`, optional for JSONL | | `data.items_fn` | empty | dotted path to a deterministic CPU row map, `fn(row: dict) -> dict`; applied before column rename and requires `data.name`. No I/O, CUDA, or external mutation. See `configs/grpo_qwen3b_gsm8k.py::gsm8k_items` | | `data.shuffle` | false | reshuffle every epoch with ` + epoch`, where the data seed is `rl.seed` (or top-level `rollout.seed` under `data_seed_follows_rollout_seed`). The whole corpus is permuted and THEN strided across data replicas, so rows migrate between replicas across epochs (matching `DistributedSampler`) | | `seed` | 0 | the run seed (`RunConfig`). For GRPO it orders the PROMPT SOURCE only: the shuffle permutation above and the arithmetic generator. It does not seed sampling — vLLM samples under top-level `rollout.seed` (default `null`, the engine default) — so an arm set that gives each arm its own `rollout.seed` and leaves this at 0 replicates the sampling, not the prompt order; preflight warns about that combination when `data.shuffle` is on | | `data_seed_follows_rollout_seed` | false | order the prompt source with top-level `rollout.seed` instead of `rl.seed`, so one per-arm seed varies the prompt order and the sampling together. Requires an integer `rollout.seed` (refused otherwise). Resolved once (`config/rl.py::resolve_data_seed`): preflight and the data role read the same value, and the cursor checkpoint records it, so an exact resume refuses a run that flipped it — set it from the first step | | `reward` | `arithmetic` | provider key selected from the run-bound built-in + custom catalog | | `reward_provider_catalog` | empty | dotted path to a zero-argument custom catalog factory; entries add group-native online providers and immutable semantic identities but cannot override built-ins. Pointwise scorers use `map_candidates` ([guide](../howto/custom-reward.md)). | | `weight_sync_transport` | `null` | weight mover, independent of placement: unset serves colocated IPC/local host copy; `nixl` is required for `grid.gpu_placement=disaggregated` | | `weight_sync_r2r_broadcast` | `false` | optional relay mode: unchanged P2R into rollout replica 0, followed by corresponding-rank NCCL broadcast of live engine tensors across rollout replicas | | `weight_sync_bucket_bytes` | `null` | optional positive streaming bucket cap; the same value sizes the NIXL staging buffer/plan and, when relay mode is enabled, the R2R non-contiguous packing target | | `weight_sync_probe_bytes` | `268435456` | pre-model NIXL bind probe payload bytes; the probe also times a pass at 1/8 of this size and gates on the slope between the two, so keep it large enough that the payload, not the per-transfer fixed cost, dominates (256 MiB is the minimum that works on InfiniBand) | | `weight_sync_min_gbps` | `20.0` | non-negative minimum fabric throughput from the bind probe (net of the per-transfer fixed cost; the log line reports both) | | `colocated_offload` | `true` | under `grid.gpu_placement=colocated`, sleep the vLLM engine while the trainer updates and park the trainer on the host while the engine generates, every step ([the hand-off cycle](../howto/batching.md#colocated-offload-what-the-trainer-holds-while-the-rollout-runs)); `false` builds the engine without sleep mode and skips the cycle, trading GPU memory for step time, so it is only for runs where engine and trainer fit side by side (bind-time memory checks and CUDA OOM surface one that does not); inert under `disaggregated` | | `data_pool.data_plane` | `bulk_store` | `bulk_store` or the Ray/RDT-backed `ray_nixl_pool`; independent of weight placement and transport | | `data_pool.page_bytes` | `33554432` | source allocation and receive-transfer granularity | | `data_pool.cuda_capacity_bytes` / `cpu_capacity_bytes` | `1073741824` / `536870912` | persistent producer capacity per actor; each must be a page-size multiple | | `data_pool.cuda_cache_bytes` / `cpu_cache_bytes` | `536870912` / `536870912` | persistent consumer receive capacity per actor; each must be a page-size multiple | | `data_pool.cpu_min_bytes` | `33554432` | CPU tensors below this size bypass the page pool | Legacy selector migration is explicit: replace `rl.objective=degenerate|clipped` with `rl.grpo_surrogate=legacy_pg|clipped`, and replace OPD's `rl.objective_mode=direct|advantage` with `rl.distillation_mode=direct|advantage`. The old keys are not aliases and fail as unknown keys. `legacy_pg` rejects non-default clipped-only settings rather than recording knobs that its loss ignores. Dr-GRPO is a resolved GRPO recipe, not another entry or objective wrapper. Launch `cosmos_framework.posttrain.entrypoints.grpo` with `configs/dr_grpo_qwen3b_gsm8k.py`. The GSM8K recipe records a distinct `algorithm` and job name, then selects `rl.grpo_surrogate=clipped`, `rl.unbiased_advantage=true`, and `rl.loss_type=seq-mean-token-sum-fixed`. The shared builder passes the single generation bound from `rollout.max_tokens` into that reduction and records it in the objective identity; no second max-response-length field, trainer, placement, or control loop is introduced. See [`objective_customization.md`](../howto/custom-objective.md) for the general objective contract, entry integration rules, and the Dr-GRPO ownership decision. The fixed normaliser reduces the gradient scale relative to token-mean reduction. Choose an optimizer epsilon compatible with that scale and re-tune the learning rate when changing either the normalizer bound or epsilon. `rl.optimizer.name=muon` reuses the model stack's `MuonWithAuxAdamW` and shared `split_orthogonalizable_params` classifier instead of maintaining a posttrain optimizer fork. Hidden `nn.Linear` weights take the scale-free Newton-Schulz route; embeddings, heads, routers, norms, biases, convolutions and ambiguous coupled 2-D fusions take auxiliary AdamW. Native `[experts, rows, cols]` parameters opt into the shared stacked-expert path: an expert-axis FSDP shard contains complete local matrices, so it avoids gathering every expert on every rank, and gate/up halves are split before orthogonalisation. Dense sharded matrices still require one BF16 all-gather each. Parameters must be stored in FP32 (`master_dtype=float32`) while FSDP supplies BF16 compute. `muon_rms_match` sets the initial Moonlight LR scale, not equivalence to an eps-dominated Adam step; re-tune LR for the objective and response-length regime. GSPO (Group Sequence Policy Optimization) is a resolved GRPO recipe in the same sense. Launch `cosmos_framework.posttrain.entrypoints.grpo` with `configs/gspo_qwen3b_gsm8k.py`: ```bash PYTHONPATH=. python -m cosmos_framework.posttrain.entrypoints.grpo \ --config cosmos_framework/posttrain/configs/gspo_qwen3b_gsm8k.py ``` The recipe records `algorithm=gspo` with its own job name for provenance and calls `apply_gspo_knobs`, the one place that turns a clipped `rl` section into the paper's objective: `rl.grpo_surrogate=clipped`, `rl.importance_sampling_level=sequence`, `rl.lower_bound_ratio=0`, `rl.epsilon_low=3e-4`, `rl.epsilon_high=4e-4`, `rl.loss_type=seq-mean-token-mean`, `rl.mini_batch_per_replica=32` (four optimizer steps per 128-trajectory batch), and `rl.certified_deterministic_forward=true`. The knob changes one thing inside the clipped surrogate. Instead of one ratio per token, every loss-masked token of response `i` carries the response ratio `s_i = exp(z_i)`, where `z_i` is the mean over those tokens of `log π_θ − log π_old`, so the epsilon clip admits or drops whole responses. `algorithm/primitives/policy_gradient.py::sequence_log_ratio` implements the GSPO-token form `sg[z_i] + (c_t − sg[c_t])`, whose value, clip decision, and parameter gradient equal the paper's objective for the one scalar advantage per response that GRPO produces; the log-ratio is clamped from above at `10` and not from below. Frozen π_old, `temperature`, `logprob_dtype`, `loss_type`, and the ahead-pass attestation are inherited unchanged. Because the per-token loss is constant within a response, `seq-mean-token-mean` is the paper's equal-weight mean over responses, while `token-mean` weights each response by its length. The response ratio differs from one only when π_old is not the current policy, that is at `n_mini > 1` (the cached ahead pass) or with `use_rollout_logprobs_for_loss=true`. At `n_mini == 1` without rollout logprobs every `s_i` is exactly `1`, and the run trains as token-level GRPO in value and gradient while its identity records GSPO; the config warns. It also warns when the largest non-negative epsilon exceeds `0.05` and when `logprob_dtype` is not `float32`. These knobs are honored but leave the sequence clip inert or noisy, so they warn; knobs the sequence-level loss never reads are rejected instead: `lower_bound_ratio != 0`, `grpo_surrogate=legacy_pg`, OPD, and per-token (tensor) advantages, which the objective refuses inside its rank-converged prepare step because one ratio per response means one advantage per response. `min_filter_prefix_tokens` still applies (the base GSM8K recipe keeps `1`); the response mean runs over exactly the tokens that stay in the loss. The clipped objective emits two response-level diagnostics at both levels, weighted by `train/valid_sequence_count`: `train/seq_clip_frac`, the fraction of responses with at least one token whose loss the clip changed (all or none of a response at sequence level), and `train/seq_log_ratio_abs`, the mean over responses of `|z_i|`, the magnitude the sequence clip compares against `ln(1 ± ε)`. Under `sequence` the token-weighted `train/clip_frac` and `train/ratio_absdev` become length-weighted response statistics, since a clipped response clips all of its tokens. The GSPO paper (sec. 5.2) reports a response clip fraction about two orders of magnitude above GRPO's token clip fraction, so a high `train/seq_clip_frac` is the expected reading rather than a defect. The objective identity records `importance_sampling_level` only when it is `sequence`, so existing clipped checkpoints resume unchanged and a level change refuses exact resume in either direction. Columns are RENAMED, not projected: `prompt_column` becomes `prompt` and `response_column` becomes `reference`, and every other column passes through to `obs.representations` for a reward or callback to read. Those two names are the wiring vocabulary — DataPackers read `prompt`, while a selected provider declares whether it needs `prompt` and/or `reference` — which keeps rewards and packers reusable across datasets while config decides which column plays each role. `data.path` wraps at epoch boundaries. Preflight opens the path only for readability. Full parsing and materialization happen when the data role is constructed—in its Ray actor, or in-process for the local backend. Every nonblank JSONL line must be an object. The configured prompt is either a string or a nonempty conversation list whose items have a string `role` and a `content` field; a configured response must be present and non-null. Conversation prompts are rendered through the selected model's chat template as described in [`datapacker.md`](../howto/custom-datapacker.md). Hub-backed sources are loaded lazily by the data role; `name`/`subset`/`split` and column mappings apply to training and validation. Source/reward field compatibility and the limits of that check are documented in [`validation.md`](../howto/preflight.md). Train-time held-out source, cadence, and sampling settings are documented separately in [`model_validation.md`](../howto/train-time-validation.md). Two seeds reach a GRPO run. `rl.seed` orders the prompt source: `data.shuffle` permutes the corpus with `rl.seed + epoch`, and the arithmetic source draws from `rl.seed`. Top-level `rollout.seed` is the vLLM engine seed and decides the sampled completions. A "seed replicate" that changes only `rollout.seed` therefore reads the identical prompt sequence and varies the sampling alone, and a continuation pool built by removing a file prefix from such a run's data does not remove the rows the parent consumed, which under a shuffle are a seeded random subset. Preflight warns when `data.shuffle=true`, `rl.seed=0` and a non-zero `rollout.seed` meet with the flag off ("all arms share one prompt order"). `rl.data_seed_follows_rollout_seed=true` keys the prompt source on `rollout.seed`, so one per-arm seed varies both; a per-arm `rl.seed` is the other correct spelling. `config/rl.py::resolve_data_seed` resolves the value once for preflight and the data role, and the cursor checkpoint (`roles/env/cursor.py`) records it, so an exact resume refuses a run whose data seed changed. Non-finite loss rejection is mandatory and is not disabled by `optimizer.force_finite_grads=false`. Disabling the gradient check emits an operator warning and `optim/finite_gradient_check_enabled=0`. Both `DiffusionTrainer` (DPO/NFT) and `LLMTrainer` (GRPO/OPD plus PPO's policy and critic adapters) use the same reject-without-step meaning. The shipped DPO, GRPO, PPO, OPD, and NFT control loops fail on the first rejected update; there is currently no retry or bounded-continuation config. ### Routing replay on a sparse-MoE policy A sparse MoE's top-k selection over near-tied router scores can vary across otherwise identical forwards. Routing replay pins the recorded dispatch so importance ratios and reference scores measure policy changes instead of routing churn. `rl.routing_replay=rollout` (Rollout Routing Replay) replays the engine's routing end to end: - **Rollout.** The vLLM engine is built with `enable_return_routed_experts` (part of the engine construction identity) and `VLLMRollout` records `policy_info["rollout_routed_experts"]` next to `rollout_logprobs`: the top-k expert ids of every input position of prompt + completion, `[P+N-1, layers, top_k]` (the last sampled token is never fed back, so it has no row), kept as `uint8` up to 256 experts and narrowed to `int16` beyond. The record is prompt-inclusive on purpose: the training forward recomputes the prompt positions and their KV feeds every completion logit, so a completion-only pin would leave the prompt's routing churn in place. The layer axis is vLLM's text-decoder layer index; the adapter checks its length against its own sparse-block count (equal on 30B-A3B, where every layer is sparse) and refuses anything else. An empty completion has no record and fails the rollout step loud under `rollout`, where the natural path skips it. The codec carries the record as bulk like every large per-token tensor; only its lane-agreement digest is inline. - **Packer.** `shape_policy_input` checks the record against the shaped `input_ids` (the stored prompt ids are asserted against the packer's own prefix, so a shifted prefix fails there and a wrong row count fails here) and every packer collates it token-aligned into the policy batch as `[B, T, layers, top_k]`; the unrecorded last token and the padding hold experts `0..top_k-1`, a valid selection whose combine weight is zero on padded rows (`data/packer/forward_routing.py`). - **Model.** The Qwen3-VL-MoE adapter (`model/protocol.py::ForwardReplayPolicy`) consumes the batch key in `prepare_policy_batch` and pins every adapted block; the key never reaches the network. This batch-driven pin is the single writer: the ahead pass, the probe and the training forwards of a chunk are all pinned because they all prepare the same batch, and no trainer code pins. - **Trainer.** After every forward it checks that the pinned state matches the knob (refusing a replay run whose batches carry no routing and a natural run whose batches pin), applies the MoE adapter's relaxed probe tolerance (`0.05` nats on at most 5% of tokens, mesh-wide) only while pinned, and reports `train/routing_replay_pinned_tokens` (the rows dispatched through the pin: prompt + completion positions, padding excluded when the packer supplies `attention_mask`) and `train/routing_replay_pinned_blocks`. Constraints: the native Qwen3-VL-MoE policy with the vLLM rollout and the clipped surrogate only (assembly refuses dense and HF backends; `legacy_pg` and OPD refuse the knob at construction, since neither prepares its batch through the model capability). `rollout` is valid at `n_mini == 1`, where it is the only mechanism that closes the train/infer routing gap, and pairs naturally with `use_rollout_logprobs_for_loss=true`, which keeps its own contract: `rollout.temperature == rl.temperature == 1.0`, `rollout.top_p=1.0`, `rollout.top_k` disabled and `rollout.repetition_penalty=1.0` (the engine returns raw, untempered logprobs). The engine-side capture is incompatible with vLLM pipeline parallelism and KV connectors, neither of which posttrain uses. An unpinned MoE run (`off`) is held to the same strict probe gate as a dense one. ## OPD additions (`rl.*`) `OPDConfig` inherits the GRPO fields above and adds one frozen teacher plus the distillation objective. See [`opd.md`](../guides/opd.md) for the complete topology and algorithms. | field | default | meaning | | ------------------------------------------ | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `distillation_mode` | `advantage` | `direct` minimizes same-token divergence; `advantage` uses the policy-gradient surrogate and is required by DPCA | | `jsd_beta` | `1.0` | same-token direction (`0` forward KL, `1` reverse KL); set `null` for DPCA because it has no shared support | | `kl_penalty_coef` | `1.0` | non-negative scale on the legacy same-token penalty or additive DPCA teacher credit | | `teacher.name_or_path` | Qwen2.5-7B M1 default | frozen teacher artifact | | `teacher.tensor_parallel_size` | `1` | ranked teacher WorkerGroup width; values greater than one require `grid.backend=ray` | | `teacher.moe_backend` | `auto` | vLLM MoE kernel selection for the teacher; set an explicit supported backend such as `triton` when the architecture-specific automatic choice is incompatible with the teacher's shard shape or platform | | `teacher.input_mode` | `shared_vocab_multimodal` | `shared_vocab_multimodal`, `cross_token_multimodal`, or `cross_token_text`; production DPCA sends original images through the teacher's own processor, while the text control pulls no media | | `teacher.signal_mode` | `topk` | `topk` or `dpca`; controls the schema and objective interface | | `teacher.logprobs_mode` | `processed_logprobs` | fixed vLLM likelihood convention. Generated-token logprobs follow processed semantics, while forced-sequence prompt logprobs remain raw, so teacher/policy scoring temperature is pinned to one. | | `teacher.top_k` | `32` | support width for `topk`; must be exactly zero for DPCA | | `teacher.scoring_temperature` | `1.0` | top-k teacher and policy scoring must both remain one because the forced-sequence vLLM path returns raw prompt logprobs. DPCA also requires one. | | `teacher.max_attempts` / `retry_backoff_s` | `2` / `0` | bounded, replica-wide retries of one identical teacher request; backoff must be in `[0, 60]` seconds | | `teacher.conditional_think_prefix` | empty closed-think prefix | applied exactly once after the teacher's assistant header; a template-provided equivalent is detected rather than duplicated | | `teacher.max_alignment_chunk_tokens` | `64` | DPCA rejection guard; reaching it without exact decoded-text alignment fails the batch | | `teacher.min_abs_student_chunk_logprob` | `1e-8` | DPCA numerical floor for `abs(L_S)`; exact zero follows the unit-ratio/zero-credit convention, while nonzero values at or below the floor fail | Both modes require `kl_beta=0` and `use_rollout_logprobs_for_loss=false`; advantage mode freezes π_old through its policy-side cache, while direct mode forms no importance ratio. Top-k direct OPD treats nonzero rollout sampling controls as data-generation choices, but its teacher and policy scoring temperatures must both remain one. Top-k advantage OPD additionally requires rollout temperature one, top-p one, top-k disabled, and neutral repetition penalty. DPCA requires all three temperatures and all rollout controls to be neutral. Every grouped rollout rejects temperature zero and the vLLM-rewritten `(0, 0.01)` range. The OPD entrypoint checks these cross-tree couplings before Ray, placement, or model loading. OPD fixes inherited `grpo_surrogate=clipped` so it cannot become a second distillation selector, and rejects `importance_sampling_level` other than `token` in both modes: its advantage objective scores a token-level ratio and direct mode forms none. `direct` rejects non-default `logprob_dtype`, clip, dual-clip, `kl_penalty_coef`, and advantage-normalization settings, and warns that task reward and group advantage do not enter its training loss. ## RL knobs reference (`rl.*`, Cosmos3 online NFT) It uses the native 93-decoded-frame / 24-latent-frame T2V contract. The reduced 29-frame comparison geometry belongs only to the optional research harness. | field | v1 default | meaning | | ---------------------------------------------------- | -------------------------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `max_steps` / `group_size` | 1000 / 4 | applied-update target / global terminal-sample group; v1 requires one group member per rank | | `sample_steps` / `sampler_shift` / `guidance` | 20 / 5.0 / 3.0 | native UniPC rollout settings; the sampler base shift remains 1.0 | | `beta` / `kl_beta` | 1.0 / 1.0 | DiffusionNFT direction strength / velocity-KL coefficient | | `policy_loss_scale` | 5.0 | explicit policy-term normalization, independent of optimality-map bounds | | `modality` | `video` | decoded media type (`video` or `image`); recipe qualification is selected separately by `profile` | | `profile` | empty | explicit qualified-recipe contract; `t2i_hpsv3_lora_720p_v1` pins the production T2I coordinates while the native T2V path remains unprofiled | | `advantage_low` / `advantage_high` / `advantage_eps` | -5 / 5 / 1e-4 | optimality bounds and sample-standardization epsilon | | `old.build_old` / `old.update_every` | true / 10 | moving behavior snapshot; refreshes after update 10, so update 11 consumes version 10 | | `ref.build_ref` / `ref.update_every` | true / 0 | frozen KL anchor; `kl_beta=0` instead requires `build_ref=false`. The shared `RefConfig` has no `model_dir` (that is GRPO's `GRPORefConfig` knob), so an override of it here is an unknown key | | `scheduler.*` | WFM lambda-linear schedule | warmup and decay advance only after applied updates | | `grad_accum` | 1 | source rows staged per optimizer window; the whole window commits or rewinds together | | `rollout_seed_clock` | `source_row` | stochastic-identity clock; the T2I predecessor contract selects `optimizer_step` so all accumulation rows reuse one rank-local seed set | | `init_keys_to_skip_loading` | empty | exact source-only `rl.init_from` keys; valid only for the image copy policy and never applied to exact resume | | `reward.provider_revision` | required override | immutable deployed scorer identity; the recipe sentinel is rejected before model allocation | Normal launch and manifest preflight validate the selected config and required reward environment-variable names before model allocation. Explicit preflight also instantiates the model-free dataloader and prepares its first T2V row, catching empty selection, caption-schema, conditioned-frame, and multi-row-packing failures on that selected path. It does not scan every row or probe the live reward service. The native video profile rejects conditioned I2V/V2V inputs and remains T2V-only. Production v1 requires `grid.backend=ray`, `grid.grouping=single`, `grid.gpu_placement=colocated`, and uniform `(1, group_size)` shapes for data, rollout, reward, replay, and policy. The `nft` dispatch profile keeps the control loop free of topology markers while preserving each exact-rank WFM lane. Future disaggregation must declare new placement and routing contracts; it does not require a topology branch in `online_nft_step`. The release-facing image profile is `cosmos3_nft_t2i_public`; its external artifact and validation contract is documented in the [public NFT run guide](../howto/run-posttrain.md#7-nft). The objective publishes merge-safe `train/optimality_mean` and `train/optimality_second_moment`. After the trainer's data-parallel reduction, the entrypoint reconstructs the population standard deviation as `train/optimality_std`. This avoids averaging a rank-local standard deviation, which would be zero when each rank owns one sample. Provider diagnostics use `train/reward_provider//` and carry the configured provider revision as provenance. Accumulation-window source metrics use `rollout/source_window_rows` and `rollout/source_committed_rows`. The experimental 8B recipe bounds its local working set to two iteration directories, with optimizer state kept only for the newest. The older directory remains model-loadable but is not a training-resume point. This lower recipe does not configure an archive destination; production recipes must supply one before treating local retention as durable publication. ## Bounded video-reward settings `VideoRewardConfig` binds the rank-local decoded-video adapter described in [`reward-providers.md`](../howto/custom-reward.md). The model-owned decoder attests finite `[-1,1]` `[1,3,T,H,W]` output before the adapter sends one uint8 video, FPS, and one sample identity to the service. Raw scores become `clamp(fp32(score), reward_clip_min, reward_clip_max) * scale` exactly once. Service URLs and the bearer token remain actor-environment inputs and never enter resolved config. Enqueue is not retried because the service exposes no idempotency key; only correlated result pulls are retryable. Connect/operation timeouts and one absolute acceptance deadline bound the call. The positional singleton result must be finite. These contracts make transport failure bounded and attributable; they do not prove live service availability, cancellation, or reward quality. `ImageRewardConfig` is an additive specialization of the same bounded transport and provider-result contract. It requires image conditioning and decoder output `[1,3,1,H,W]`, serializes CPU uint8 `[1,H,W,3]` with `media_type=image`, and omits video FPS metadata. The T2V request shape and default remain unchanged.