Posttrain documentation

Post-training for Cosmos models: offline SFT, DPO, GRPO, actor-critic PPO, on-policy distillation (OPD), and DiffusionNFT on one substrate that scales from a single GPU to a multinode Ray cluster.

New public users should start with the general run guide, then read the reference guide for the algorithm they need. The glossary.md is worth five minutes before anything else — replica, rank, dp_shard, ActorGroup, and DeviceGroup all mean specific things here.

Getting started

Offline supervised fine-tuning is covered in the SFT guide.

Document	What it covers
howto/run-posttrain.md	Step-by-step container recipes for GRPO, DPO, PPO, OPD, NFT, and Joint PPO, using a four-GPU showcase that can be adapted
howto/run-posttrain-qa.md	Run length and short runs, Joint PPO's silent pre-training evaluation, checkpoint intervals, and first-run errors
glossary.md	Terminology, including the three unrelated meanings of "validation"

Run a capability

Each guide follows the same skeleton, so a reader who has used one can navigate the others.

Guide	Use it for
guides/dpo.md	Offline preference optimization on preference pairs
guides/grpo.md	Online RL with a rule or model reward and a vLLM rollout
guides/ppo.md	Actor-critic PPO with frozen reference and critic roles
guides/opd.md	Distilling a frozen teacher into a policy on-policy
guides/nft.md	DiffusionNFT — online RL for diffusion world models
guides/joint-ppo-pi05-public.md	Public π0.5 checkpoint, LIBERO runtime, assets, and four-GPU Joint PPO launch
guides/joint-ppo-gr00t-public.md	Public GR00T-N1.7 checkpoint, pinned upstreams, and four-GPU Joint PPO launch

Customize

Document	Use it for
howto/custom-data.md	Pointing a run at your own dataset
howto/custom-datapacker.md	Supporting a new model architecture's tensor shaping
howto/custom-reward.md	Writing a reward provider
howto/custom-objective.md	Changing the training objective
howto/sim-integration.md	Putting your own simulator behind the Simulator/SimSession contract
howto/batching.md	Sizing mini/micro batches, fixing OOM, and what the colocated rollout/trainer offload holds on the card
howto/data-plane.md	The opt-in Ray/NIXL pooled data plane for large trajectory fields
howto/preflight.md	Checking wiring before allocating GPUs
howto/train-time-validation.md	Held-out evaluation during training
howto/checkpoint-artifacts.md	Shared archive publication and verified fetch/pruning; checkpoint.tools commands for archives, expert merge and HF teachers

Look something up

Document	Contents
reference/config.md	Every rl.* knob, per algorithm
reference/metrics.md	Every reported metric, and how it aggregates
reference/checkpoint.md	Formats, resume, preemption, retention
reference/compatibility.md	API maturity and migration windows
reference/embodied.md	The simulator-side contracts: Simulator/SimSession, EmbodimentMap, WorkUnit, and the plugin/import boundary
troubleshooting.md	Symptom-first failure catalogue

Qualified commands

Each command below was run end to end in the runtime image of the run guide and completed successfully. Rerun them after a dependency or image change.

Command	Result
--validate-manifest for GRPO, PPO, OPD	exit 0, bind_check: ok
GRPO GSM8K public recipe (run guide)	Completed optimizer updates with public Qwen and GSM8K artifacts
DPO public T2I recipe (run guide)	Four-rank FSDP initialized from the converted Cosmos3-Nano and completed optimizer updates over synthetic preference pairs
PPO Qwen3-0.6B public recipe (run guide)	Completed five policy/critic update iterations
OPD Countdown public recipe (run guide)	Completed five disaggregated policy/rollout/teacher updates over NIXL
NFT Cosmos3-Nano public recipe (run guide)	Completed five updates against the deterministic toy reward service
π0.5 Joint PPO (public guide)	Completed five LIBERO collection/update windows
GR00T-N1.7 Joint PPO (public guide)	Completed five LIBERO collection/update windows with exit code 0