# Cosmos-Framework

> One Python package to train, post-train, evaluate and serve Cosmos3 omnimodal world models.

Links to this site point to each page's Markdown; skill links point to the SKILL.md on GitHub.
Agent skills are step-by-step instructions for coding agents; they refer to files in the repository,
so use them from a clone of https://github.com/NVIDIA/cosmos-framework.

## Get Started

- [Home](https://nvidia.github.io/cosmos-framework/index.md)
- [Quickstart](https://nvidia.github.io/cosmos-framework/get_started/quickstart.md)
- [Install](https://nvidia.github.io/cosmos-framework/setup.md)
- [Agent skills](https://nvidia.github.io/cosmos-framework/get_started/agent_skills.md)

## Models

- [Models](https://nvidia.github.io/cosmos-framework/models/README.md)

## Inference & Deploy

- [Run inference](https://nvidia.github.io/cosmos-framework/inference.md)
- [Prompt upsampling](https://nvidia.github.io/cosmos-framework/prompt_upsampling.md)
- [Prompting the Reasoner](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/reasoner/reasoner_prompt_guide.md)
- [World simulation › Transfer-Auto](https://nvidia.github.io/cosmos-framework/inference_auto.md)
- [World simulation › Sim-Auto](https://nvidia.github.io/cosmos-framework/inference_sim_auto.md)
- [World simulation › Sim-Transfer](https://nvidia.github.io/cosmos-framework/inference_transfer.md)
- [World simulation › Sim-Bimanual](https://nvidia.github.io/cosmos-framework/inference_bimanual.md)
- [World simulation › Sim-Bimanual sessions](https://nvidia.github.io/cosmos-framework/bimanual_sessions.md)
- [Deploy › Choose a serving path](https://nvidia.github.io/cosmos-framework/deploy/serving.md)
- [Deploy › Serve DROID policies](https://nvidia.github.io/cosmos-framework/action_policy_droid_server.md)
- [Deploy › Backend environments](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/README.md)
- [Deploy › Deploy with NIM](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/README.md)
- [Notebooks › Generation](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/generator/audiovisual/README.md)
- [Notebooks › Action](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/generator/action/README.md)
- [Notebooks › Control-to-video transfer](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/generator/transfer/README.md)
- [Notebooks › Reasoner](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/reasoner/README.md)
- [Notebooks › Simulation](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/simulation/README.md)

## Post-Training

- [Choose a path](https://nvidia.github.io/cosmos-framework/post_training/choose_path.md)
- [SFT › SFT recipes](https://nvidia.github.io/cosmos-framework/training.md)
- [SFT › Datasets › JSONL datasets](https://nvidia.github.io/cosmos-framework/dataset_jsonl.md)
- [SFT › Datasets › Custom datasets](https://nvidia.github.io/cosmos-framework/custom_dataset.md)
- [SFT › Robot policies › DROID policy](https://nvidia.github.io/cosmos-framework/action_policy_droid_posttrain.md)
- [SFT › Robot policies › DROID forward dynamics](https://nvidia.github.io/cosmos-framework/action_fd_droid_posttrain.md)
- [SFT › Robot policies › LIBERO-10 policy](https://nvidia.github.io/cosmos-framework/action_policy_libero_posttrain.md)
- [SFT › Transfer-Auto](https://nvidia.github.io/cosmos-framework/training_transfer_auto.md)
- [Distillation › Sim-Auto](https://nvidia.github.io/cosmos-framework/training_sim_auto.md)
- [Distillation › Sim-Transfer › Stage 0: teacher](https://nvidia.github.io/cosmos-framework/training_sim_transfer_stage0.md)
- [Distillation › Sim-Transfer › Stages 1-3](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-transfer/post-training/SKILL.md)
- [Distillation › Sim-Bimanual › Check training data](https://nvidia.github.io/cosmos-framework/bimanual_data_preflight.md)
- [Distillation › Sim-Bimanual › Stages 0-3](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-bimanual/post-training/SKILL.md)
- [Distillation › Super four-step (DMD2)](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/generator/audiovisual/distill/README.md)
- [RL › Overview](https://nvidia.github.io/cosmos-framework/posttrain/index.md)
- [RL › Quickstart](https://nvidia.github.io/cosmos-framework/posttrain/howto/run-posttrain.md)
- [RL › Generator › DPO](https://nvidia.github.io/cosmos-framework/posttrain/guides/dpo.md)
- [RL › Generator › DiffusionNFT](https://nvidia.github.io/cosmos-framework/posttrain/guides/nft.md)
- [RL › Reasoner › GRPO](https://nvidia.github.io/cosmos-framework/posttrain/guides/grpo.md)
- [RL › Reasoner › PPO](https://nvidia.github.io/cosmos-framework/posttrain/guides/ppo.md)
- [RL › Reasoner › OPD](https://nvidia.github.io/cosmos-framework/posttrain/guides/opd.md)
- [RL › Customize › Custom data](https://nvidia.github.io/cosmos-framework/posttrain/howto/custom-data.md)
- [RL › Customize › DataPacker](https://nvidia.github.io/cosmos-framework/posttrain/howto/custom-datapacker.md)
- [RL › Customize › Reward](https://nvidia.github.io/cosmos-framework/posttrain/howto/custom-reward.md)
- [RL › Customize › Objective](https://nvidia.github.io/cosmos-framework/posttrain/howto/custom-objective.md)
- [RL › Operate › Preflight](https://nvidia.github.io/cosmos-framework/posttrain/howto/preflight.md)
- [RL › Operate › Batching and OOM](https://nvidia.github.io/cosmos-framework/posttrain/howto/batching.md)
- [RL › Operate › Data plane](https://nvidia.github.io/cosmos-framework/posttrain/howto/data-plane.md)
- [RL › Operate › Train-time validation](https://nvidia.github.io/cosmos-framework/posttrain/howto/train-time-validation.md)
- [RL › Operate › Checkpoint artifacts](https://nvidia.github.io/cosmos-framework/posttrain/howto/checkpoint-artifacts.md)
- [RL › Operate › Run guide FAQ](https://nvidia.github.io/cosmos-framework/posttrain/howto/run-posttrain-qa.md)
- [RL › Reference › Config](https://nvidia.github.io/cosmos-framework/posttrain/reference/config.md)
- [RL › Reference › Metrics](https://nvidia.github.io/cosmos-framework/posttrain/reference/metrics.md)
- [RL › Reference › Checkpointing](https://nvidia.github.io/cosmos-framework/posttrain/reference/checkpoint.md)
- [RL › Reference › Architecture](https://nvidia.github.io/cosmos-framework/posttrain/package-overview.md)
- [RL › Reference › Glossary](https://nvidia.github.io/cosmos-framework/posttrain/glossary.md)
- [RL › Reference › Troubleshooting](https://nvidia.github.io/cosmos-framework/posttrain/troubleshooting.md)

## Evaluate

- [Evaluate and score](https://nvidia.github.io/cosmos-framework/evaluation.md)
- [PAI-Bench-C](https://nvidia.github.io/cosmos-framework/skills/evaluation/cosmos3/generator/paibench_c/README.md)
- [PAI-Bench-G](https://nvidia.github.io/cosmos-framework/skills/evaluation/cosmos3/generator/paibench_g/README.md)
- [Physics-IQ](https://nvidia.github.io/cosmos-framework/skills/evaluation/cosmos3/generator/physics_iq/README.md)
- [R-Bench](https://nvidia.github.io/cosmos-framework/skills/evaluation/cosmos3/generator/rbench/README.md)

## Reference

- [SFT config](https://nvidia.github.io/cosmos-framework/sft_config.md)
- [Environment variables](https://nvidia.github.io/cosmos-framework/environment_variables.md)
- [FAQ](https://nvidia.github.io/cosmos-framework/faq.md)
- [Code structure](https://nvidia.github.io/cosmos-framework/code_structure.md)
- [Integrate into your framework](https://nvidia.github.io/cosmos-framework/examples/integration/README.md)
- [Contributing](https://nvidia.github.io/cosmos-framework/develop/contributing.md)
- [Security](https://nvidia.github.io/cosmos-framework/develop/security.md)

## Agent skills

- [cosmos3-nim-user](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cookbooks/cosmos3/nim/.agents/skills/cosmos3-nim-user/SKILL.md): Guide customers through Cosmos3 NIM deployment, endpoint verification, API example selection, hardware compatibility, and troubleshooting. Use for operating or calling the NIM from the public .agents/skills/cookbooks/cosmos3/nim directory; do not use for maintaining the documentation itself.
- [scenario-to-rds-hq](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cookbooks/cosmos3/simulation/auto/wsm/skills/scenario-to-rds-hq/SKILL.md): Convert a driving scenario or dataset in an unfamiliar format into RDS-HQ for WSM4 rendering, with final Hyperion camera retargeting by default. Use for source-schema discovery, coordinate and camera conversion, semantic class reduction, missing-field inference, and validation of converted scenarios.
- [cosmos-automl](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos-automl/SKILL.md): Optimize Cosmos Framework reasoner SFT hyperparameters or evaluation prompts with bounded AutoML. Use for numerical, multi-fidelity, population or LLM-guided search, GEPA Auto-Prompter, and monitoring or resuming those experiments.
- [cosmos-framework-docs](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos-framework-docs/SKILL.md): Edit the Cosmos-Framework documentation and its MkDocs site. Use when asked to fix, update, add, move or remove a docs page, find the source file behind a docs-site page or URL, change the site navigation, landing page, logo or styling, or preview the site and fix a failing docs build.
- [cosmos3-codebase-nav](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-codebase-nav/SKILL.md): Navigate the Cosmos3 package codebase to find where parameters, configs, defaults, scripts, and documentation live. Use when the user asks "where is X in cosmos3", "how do I find the config for Y", "where are the defaults", "where do I change a parameter", or any question about locating files, modules, or settings. Also use when the user opens or edits files and needs orientation.
- [cosmos3-env-troubleshoot](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-env-troubleshoot/SKILL.md): Diagnose and fix Cosmos3 environment, installation, and runtime errors. Use when the user encounters an ImportError, ModuleNotFoundError, CUDA error, Docker error, checkpoint download failure, or any traceback during setup or inference.
- [cosmos3-inference](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-inference/SKILL.md): Guide users through running Cosmos3 inference — offline batch generation, online serving with Ray and Gradio, parallelism options, input formats, sampling parameters, and prompt upsampling. Use when the user asks "how do I run inference", "how do I generate a video", "how do I serve the model", "what parameters should I use", or any question about running the model to produce outputs.
- [cosmos3-nano-sim-auto-inference](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-auto/inference/SKILL.md): Prepare inputs or run Cosmos3-Nano-Sim-Auto camera-only or joint camera/LiDAR generation with a released or exported student in Cosmos Framework. Check causal conditioning, per-camera captions, rolling metadata and saved outputs.
- [cosmos3-nano-sim-auto-post-training](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-auto/post-training/SKILL.md): Distill Cosmos3-Nano-Transfer-Auto into Cosmos3-Nano-Sim-Auto using customer RGB, WSM, captions and LiDAR in Cosmos Framework. Covers TF, TF-dCM, SF-DMD, checkpoint handoffs, resume and student export.
- [cosmos3-nano-sim-auto-troubleshooting](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-auto/troubleshooting/SKILL.md): Diagnose Cosmos3-Nano-Sim-Auto input, causal inference, distillation, checkpoint and export failures in Cosmos Framework. Check camera mapping, captions, rolling metadata and stage-state handoffs without silently changing the recipe.
- [cosmos3-nano-sim-bimanual-inference](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-bimanual/inference/SKILL.md): Generate action-conditioned video with Cosmos3 Nano Sim Bimanual in Cosmos Framework. Download the matched release, prepare RGB and action inputs, run inference, inspect outputs, or use an exported student from the user's Stage 3 training.
- [cosmos3-nano-sim-bimanual-post-training](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-bimanual/post-training/SKILL.md): Train Cosmos3 Nano Sim Bimanual from Cosmos3-Nano with Cosmos Framework's four native stage TOMLs and a local training bundle. Covers data preparation, initialization, training, checkpoint handoffs, resume and student export.
- [cosmos3-nano-sim-bimanual-troubleshooting](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-bimanual/troubleshooting/SKILL.md): Diagnose Cosmos3 Nano Sim Bimanual setup, data, training, checkpoint, inference, and evaluation failures by locating the first broken product contract. Use when a supported workflow fails, produces invalid media, or cannot be reproduced.
- [cosmos3-nano-sim-transfer-inference](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-transfer/inference/SKILL.md): Generate RGB video with a matched Cosmos3-Nano-Sim-Transfer checkpoint. Use HF example2 for ordinary spatial-control inference and example1 for long-rollout depth or segmentation inference.
- [cosmos3-nano-sim-transfer-post-training](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-transfer/post-training/SKILL.md): Prepare aligned spatial controls, RGB targets and captions for Cosmos3-Nano-Sim-Transfer; run native Cosmos Framework Stage 0–3 training with local assets; resume full-state checkpoints; pass selected EMA to inference.
- [cosmos3-nano-sim-transfer-troubleshooting](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-sim-transfer/troubleshooting/SKILL.md): Diagnose Cosmos3-Nano-Sim-Transfer setup, checkpoint loading, media alignment, training-state and output failures in an existing workflow.
- [cosmos3-nano-transfer-auto-inference](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-transfer-auto/inference/SKILL.md): Prepare or run Cosmos3-Nano-Transfer-Auto in Cosmos Framework on camera WSM controls and per-view captions, with optional initial RGB and joint LiDAR. Use for camera mapping, WSM/renderer and synchronization checks, supplied-caption review, av-per-view-caption generation or upsampling, targeted-KPI captions, native requests, sampling settings and output inspection. Input-checking and caption-only requests stop after the requested workflow.
- [cosmos3-nano-transfer-auto-post-training](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-transfer-auto/post-training/SKILL.md): Fine-tune the released Cosmos3-Nano-Transfer-Auto model on paired driving data in Cosmos Framework, or run the four-phase training curriculum from Cosmos3-Nano.
- [cosmos3-nano-transfer-auto-troubleshooting](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-nano-transfer-auto/troubleshooting/SKILL.md): Diagnose cosmos3-nano-transfer-auto failures, including distorted side/rear views, WSM alignment, caption mismatch, unexpected frame counts, missing LiDAR, out-of-memory errors, and checkpoint or resume failures. Use for focused checks and recovery, not automatic input remapping, retraining, causal Sim-Auto debugging, or quality certification.
- [cosmos3-post-training](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-post-training/SKILL.md): Guide users through Cosmos3 supervised fine-tuning (SFT) post-training: preparing the example dataset and Wan2.2 VAE, converting the base checkpoint to DCP, launching distributed training (paired launch shell recommended, raw `torchrun` as an alternative), running T2V/I2V/V2V inference with the trained DCP checkpoint, and optionally exporting it to Hugging Face safetensors. Use when the user asks how to post-train Cosmos3, fine-tune on a custom video dataset, export a trained checkpoint, or invoke one of the recipe launch shells (`launch_sft_vision_nano.sh`, `launch_sft_llava_ov.sh`, `launch_sft_videophy2_nano.sh`, plus the `_super` LoRA variant) — or any question about `cu130-train` / `cu128-train`, `convert_model_to_dcp` / `export_model` / `train`, or SFT output paths. For dataset captioning / JSONL assembly, see `docs/dataset_jsonl.md`.
- [cosmos3-setup](https://raw.githubusercontent.com/NVIDIA/cosmos-framework/main/.agents/skills/cosmos3-setup/SKILL.md): Guide users through Cosmos3 installation, environment setup, checkpoint downloading, and verification. Use when the user asks "how do I install cosmos3", "how do I set up the environment", "how do I download checkpoints", "how do I use Docker", or any question about getting the package running for the first time.

## Optional

- [Action](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/action.md)
- [API Reference](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/api-reference.md)
- [Bring Your Own Checkpoint](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/bring-your-own-checkpoint.md)
- [Configuration](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/configuration.md)
- [Deployment](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/deployment.md)
- [Generation](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/generation.md)
- [Helm Deployment](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/helm.md)
- [Operations and Troubleshooting](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/operations.md)
- [Prerequisites](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/prerequisites.md)
- [Reasoning](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/reasoning.md)
- [Release Notes](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/release-notes.md)
- [Support Matrix](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/support-matrix.md)
- [Transfer](https://nvidia.github.io/cosmos-framework/skills/cookbooks/cosmos3/nim/transfer.md)