Skip to content

Cosmos3 Simulation Cookbooks

Cosmos3 Simulation models generate what a robot or vehicle would observe, driven by a control signal: robot actions, per-camera driving controls, or a spatial control video.

Choose a model

Model Generates From Start here Notebooks
Cosmos3-Nano-Sim-Bimanual Video rollout of bimanual robot manipulation Initial RGB frame, task description, time-aligned action trajectory Bimanual guide Framework, vLLM-Omni
Cosmos3-Nano-Transfer-Auto Synchronized multi-camera driving video, joint mode adds LiDAR Per-camera WSM controls and captions, HD-map LiDAR controls for joint mode Auto overview, Transfer-Auto guide Framework, vLLM-Omni
Cosmos3-Nano-Sim-Auto Same outputs as Transfer-Auto, generated chunk by chunk Same inputs as Transfer-Auto Auto overview Framework, vLLM-Omni
Cosmos3-Nano-Sim-Transfer 480p video Caption and one edge, blur, depth, or segmentation control video Sim-Transfer guide Framework, vLLM-Omni

To drive the Auto models with your own scene, render WSM controls with generate_wsm.ipynb.

Sim-Bimanual renders the result of supplied actions. Pair it with a policy model for closed-loop simulation. The examples run open loop: they do not run a policy or feed generated frames back into one.

Transfer-Auto is driven like a transfer model (control video in, photoreal video out) but belongs to the Simulation family, so it lives here rather than under Generator Transfer. Sim-Auto is its causal, distilled student: a fixed four-step sampler with a rolling key/value cache.

Cosmos3-Nano-Sim-Bimanual and Cosmos3-Nano-Sim-Auto are supported by FlashDreams. Integrate with FlashDreams for accelerated inference (Sim-Bimanual, Sim-Auto)

Prerequisites

  • Linux with an NVIDIA GPU. GPU count and class differ by model, see each guide.
  • A Hugging Face token in HF_TOKEN with access to the model.
  • Cosmos Framework: a checkout set up with the Framework setup guide. The CUDA dependency groups (cu130-train on CUDA 13, cu128-train on CUDA 12.8) require Python 3.13 and include FlashAttention-4, which Sim-Auto needs. Bimanual and Sim-Transfer add model-specific setup steps in their guides.
  • vLLM-Omni: Docker with the NVIDIA Container Toolkit. All four notebooks use the vllm/vllm-omni:cosmos3 image.

If you pass --defaults-file to Framework inference, it replaces the built-in sampling defaults rather than merging with them. Start from the defaults file the model's guide provides.

Validate a rollout

For each sample, check that:

  1. sample_outputs.json reports status: success.
  2. Every output video decodes fully at the requested frame count and FPS.
  3. The motion follows the control signal (actions, WSM controls, or the spatial control). Visual quality alone does not show control adherence.
  4. Generated data is labeled as synthetic before it enters any dataset.

Use a fixed seed when comparing checkpoints or runtime changes. Then run the model-specific checks: Bimanual (drift, action response), Transfer-Auto (all views present, LiDAR sweep count, cross-view consistency), Sim-Auto, and Sim-Transfer (control adherence, temporal consistency).

Run and post-train with agent skills

Open your coding agent at the root of a Cosmos Framework checkout so it finds the skills under .agents/skills.

Model Skills
Sim-Bimanual Inference · Post-training · Troubleshooting
Transfer-Auto Inference · Post-training · Troubleshooting
Sim-Auto Inference · Post-training · Troubleshooting
Sim-Transfer Inference · Post-training · Troubleshooting · Setup guide · Caption guide

For environment problems shared by all models, use cosmos3-env-troubleshoot.