Skip to content

Cosmos3 PAI-Bench (Generation) Reproduction

End-to-end recipe for generating the PAI-Bench (Physical AI Bench) generation set with Cosmos3-Nano or Cosmos3-Super using the native Cosmos Framework PyTorch entrypoint (python -m cosmos_framework.scripts.inference). The notebook defaults to Cosmos3-Nano; set PAIBENCH_MODEL_VARIANT=Super to run Cosmos3-Super instead.

PAI-Bench covers Physical AI domains (AV driving, robotics, industry, physics, human, common sense) across 1044 samples. The notebook runs both generation tasks:

  • Text-to-Video (T2V): generate from the prompt only (no condition image); generate 189 frames.
  • Image-to-Video (I2V): condition on the per-sample image (condition_image/<image_name>); generate 189 frames.

Both tasks generate at 24 FPS, 720p, 16:9, and keep the raw output (no staging).

The optional scoring section stores results under outputs/scores/<model_name>/<task>/. Most quality dimensions use the upstream distributed evaluator. Motion smoothness runs separately with the same AMT-S implementation, writes resumable per-worker checkpoints, and emits the standard PAI-Bench-G result schema used by the final aggregation cell.

Files

  • run_with_cosmos_framework.ipynb — main notebook (demos for both tasks + two full-sweep cells).
  • setup_paibench_scorer.sh — installs the isolated scoring environment and prepares scorer assets.
  • run_motion_smoothness_sharded.py — runs the official AMT-S metric with resumable workers across the visible GPUs.
  • assets/i2v_prompts.json — 1044 I2V entries with json_upsampled_prompt and negative_prompt.
  • assets/t2v_prompts.json — 1044 T2V entries with json_upsampled_prompt and negative_prompt.

Dataset

The condition images come from the Hugging Face dataset shi-labs/physical-ai-bench-generation, cloned via git clone (Git LFS). Only the condition images are read from the dataset; the prompts come from the local assets/ files.

Sampling settings

Setting Value
num_frames 189
fps 24
resolution 720
num_steps 50
guidance 6.0
shift 10.0
seed 0

Requirements

  • 8-GPU Linux node (configurable via COSMOS3_NUM_GPUS, default 8)
  • uv == 0.11.29
  • git, git-lfs
  • Hugging Face access to the Cosmos3 model family