Skip to content

Cosmos3 RBench Reproduction

End-to-end recipe for generating the RBench robotics video-generation benchmark with Cosmos3-Nano or Cosmos3-Super using the native Cosmos Framework PyTorch entrypoint (python -m cosmos_framework.scripts.inference). The notebook defaults to Cosmos3-Nano; set RBENCH_MODEL_VARIANT=Super to run Cosmos3-Super instead.

RBench is an Image-to-Video (I2V) benchmark of 650 cases across 9 categories (common_manipulation, dual_arm, humanoid, long-horizon_planning, multi-entity_collaboration, quad, single_arm, spatial_relationship, visual_reasoning). Each case conditions on a single image and generates a clip.

  • Image-to-Video (I2V): condition on the per-case image (imgs/<image_path>); generate 121 frames (1 conditioning + 120 generated).

Generation is at 24 FPS, 720p, 16:9, and the raw output is kept (no staging). Generated inputs and videos are stored under outputs/<model_name>/ so runs from different model variants remain separate.

Files

  • run_with_cosmos_framework.ipynb — main notebook (demo case + full-sweep cell).
  • assets/prompts/*.json — 9 category files, 650 entries total, each with json_upsampled_prompt and negative_prompt.
  • setup_rbench_scorer.sh — isolated GPT/local-Qwen VQA environment.
  • setup_rbench_embodiment_scorer.sh — isolated motion-metric environment.

Dataset

The condition images come from the Hugging Face dataset DAGroup-PKU/RBench, cloned via git clone (Git LFS). Only the condition images are read from the dataset; the prompts come from the local assets/prompts/ files.

Sampling settings

Setting Value
num_frames 121
fps 24
resolution 720
num_steps 50
guidance 6.0
shift 10.0
seed 0

Requirements

  • 8-GPU Linux node (configurable via COSMOS3_NUM_GPUS, default 8)
  • uv == 0.11.29
  • git, git-lfs
  • Hugging Face access to the Cosmos3 model family

Scoring evaluators

The notebook uses GPT as the default VLM evaluator for both the five-task split and the three VQA metrics in the four-embodiment split. Configure an OpenAI-compatible endpoint before enabling scoring:

export OPENAI_BASE_URL=https://your-openai-compatible-endpoint/v1
export OPENAI_MODEL=your-endpoint-model-id
export RBENCH_VLM_API_KEY=your-api-key

OPENAI_API_KEY is accepted as a fallback key variable. If neither key variable is set, the notebook requests the key through a hidden input prompt; do not save a literal key in the notebook. The dated public ReVidgen model, gpt-5-2025-08-07, remains the default model when OPENAI_MODEL is unset.

To use local Qwen instead:

export RBENCH_USE_GPT=False
export QWEN_MODEL_PATH=Qwen/Qwen2.5-VL-72B-Instruct

GPT results are written under gpt and local-Qwen results under qwen_local; their summaries and overall-score files are also kept separate. GPT outputs are validated strictly against the upstream ReVidgen schema. Only local Qwen enables the narrow action_effectiveness compatibility mapping for the two task categories whose prompt terminology and example JSON field names differ.

VQA scoring uses .venv-rbench-scorer. The motion amplitude and smoothness metrics are evaluator-independent and continue to use .venv-rbench-ops. Changing the VLM evaluator does not affect generation or require regenerating videos.