Cosmos3 Generator Transfer Examples

Cosmos3 video transfer examples — Nano (single GPU) and Super (multi-GPU, 32B) — on the native PyTorch (Cosmos Framework) path, the Diffusers modular pipeline, and the OpenAI-compatible TensorRT-LLM and vLLM-Omni server paths. Sample assets under assets/ cover spatial control signals paired with prompt.json files:

• Edge (Canny) — edge map control plus caption.
• Blur — blurred-reference control plus caption.
• Depth — depth map control plus caption.
• Segmentation — segmentation map control plus caption.
• World scenario (WSM) — world-scenario map control plus caption.
• Multi-control — two or more hints; Cosmos Framework also supports per-hint weights.

Cosmos Framework, TensorRT-LLM, and vLLM-Omni support multi-control transfer. Per-hint weighting is supported only by Cosmos Framework; the serving backends accept multiple unweighted controls. Diffusers also accepts multiple precomputed controls without per-hint weights.

Environment setup is centralized in the shared Cosmos3 cookbooks environment setup guide.

Transfer Definition

Video transfer generates a target clip from a prompt.json caption and one or more spatial control signals. The Framework path uses model_mode video2video in a local JSON spec. TensorRT-LLM uses POST /v1/videos/sync. A raw source used to derive edge or blur is multipart video_reference; precomputed controls are carried directly under their extra_params hint and do not need a top-level reference. The vLLM-Omni path uses POST /v1/videos/sync and passes one or more hint keys (edge, blur, depth, seg, or wsm) inside extra_params. Cosmos Framework accepts pre-computed control videos (control_path) or derives active controls from a raw source video (vision_path). With vLLM-Omni, pass pre-computed controls through control_path; edge and blur controls can also be derived from an uploaded input_reference. Output frame count and geometry come from the control video; see the spec field reference for how fps and aspect_ratio are resolved. All examples share assets/negative_prompt.json for the negative caption.

Control	Asset folder	Inference input	Generation duration
Edge (Canny)	assets/edge/	control_edge.mp4 + prompt.json	121 frames @ 30 FPS
Blur	assets/blur/	control_blur.mp4 + prompt.json	121 frames @ 30 FPS
Depth	assets/depth/	control_depth.mp4 + prompt.json	121 frames @ 30 FPS
Segmentation	assets/seg/	control_seg.mp4 + prompt.json	121 frames @ 30 FPS
World scenario (WSM)	assets/wsm/	control_wsm.mp4 + prompt.json	100 frames @ 10 FPS
Multi-control	assets/multi_control/	vision_path + multiple hints (Framework example)	121 frames @ 30 FPS

Transfer inference is selected automatically when any hint key is present in the Framework spec or in vLLM-Omni extra_params. The same spec files are used for both Nano and Super — model selection is controlled entirely by --checkpoint-path.

Run with Cosmos Framework

Quickstart — Single-control transfer

Set up the environment: Cosmos Framework setup. Run the commands below inside the cosmos container (e.g. pytorch:25.09-py3) — the same environment used to install the venv and run the notebook. The commands mirror the notebook exactly: cd into the framework repo first, then invoke the venv's Python or torchrun (the system Python does not have cosmos_framework).

# Set once — the cosmos-framework repo root (contains .venv/ and pyproject.toml).
# In this cosmos checkout: packages/cosmos3 (or packages/cosmos-framework).
export COSMOS_FRAMEWORK=/path/to/cosmos-framework   # e.g. <cosmos_root>/packages/cosmos3
export TRANSFER_ROOT=$(pwd)/cookbooks/cosmos3/generator/transfer

# NGC containers bundle libtorch in LD_LIBRARY_PATH which conflicts with Triton/CUDA.
unset LD_LIBRARY_PATH

Cosmos3-Nano (single GPU)

cd "$COSMOS_FRAMEWORK"

# edge — replace edge.json with blur.json / depth.json / seg.json / wsm.json for other controls
CUDA_VISIBLE_DEVICES=0 \
.venv/bin/python -m cosmos_framework.scripts.inference \
  --parallelism-preset=latency \
  -i "$TRANSFER_ROOT/specs/edge.json" \
  -o "$TRANSFER_ROOT/outputs/Cosmos3-Nano/" \
  --checkpoint-path Cosmos3-Nano \
  --seed 2026

Cosmos3-Super (multi-GPU)

cd "$COSMOS_FRAMEWORK"

# edge — replace edge.json with other control specs as needed
CUDA_VISIBLE_DEVICES=0,1,2,3 \
.venv/bin/torchrun --nproc-per-node=4 \
  --master-addr=127.0.0.1 --master-port=29500 \
  -m cosmos_framework.scripts.inference \
  --parallelism-preset=latency \
  -i "$TRANSFER_ROOT/specs/edge.json" \
  -o "$TRANSFER_ROOT/outputs/Cosmos3-Super/" \
  --checkpoint-path Cosmos3-Super \
  --seed 2026

	Cosmos3-Nano	Cosmos3-Super
--checkpoint-path	Cosmos3-Nano	Cosmos3-Super
Launcher	.venv/bin/python (from framework root)	.venv/bin/torchrun --nproc-per-node=<N> (from framework root)
--parallelism-preset	latency	latency
GPUs	1	4+

The input spec sets prompt_path and a hint block with control_path pointing at the checked-in assets under assets/ via paths relative to specs/.

Outputs are written under the directory passed to -o, with one subdirectory per sample name, e.g. outputs/Cosmos3-Nano/transfer_edge/vision.mp4.

Notebook (self-contained)

run_video_transfer_with_cosmos_framework.ipynb is a self-contained tutorial: it installs all dependencies (system packages, framework clone, Python venv via uv), authenticates with Hugging Face, and runs all six controls with previews.

1. Open the notebook and edit §2 (Configure) — paste your HF_TOKEN and optionally set cache/output paths.
2. Run §9–§13 for Cosmos3-Nano single-control (single GPU), §14–§18 for Cosmos3-Super single-control (multi-GPU), or §19 for multi-control (Nano). No model flag needed — each section uses its matching checkpoint explicitly.

To execute headlessly:

cd .agents/skills/cookbooks/cosmos3/generator/transfer
jupyter execute run_video_transfer_with_cosmos_framework.ipynb

Outputs land under outputs/notebooks/<model>/transfer_<control>/vision.mp4.

Run with Diffusers

Quickstart

Set up the environment: Diffusers setup. Transfer is reachable only through the modular pipeline, which takes precomputed control videos as control_videos={hint: frames}:

import json
from pathlib import Path

import torch
from diffusers import Cosmos3OmniModularPipeline
from diffusers.utils import export_to_video, load_video

transfer_root = Path(".agents/skills/cookbooks/cosmos3/generator/transfer")
prompt = json.dumps(json.load(open(transfer_root / "assets/depth/prompt.json")))
negative = json.dumps(json.load(open(transfer_root / "assets/negative_prompt.json")))

pipe = Cosmos3OmniModularPipeline.from_pretrained("nvidia/Cosmos3-Nano", torch_dtype=torch.bfloat16)
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.enable_safety_checker()
pipe.to("cuda")

videos = pipe(
    prompt=prompt,
    negative_prompt=negative,
    control_videos={"depth": load_video(str(transfer_root / "assets/depth/control_depth.mp4"))},
    control_guidance=1.5,
    num_frames=121,
    num_video_frames_per_chunk=121,
    num_conditional_frames=1,
    height=720,
    width=1280,
    fps=30.0,
    num_inference_steps=50,
    guidance_scale=3.0,
    generator=torch.Generator(device="cuda").manual_seed(2026),
    output="videos",
)
export_to_video(videos, "/tmp/cosmos3_transfer_depth.mp4", fps=30, macro_block_size=1)

To run Cosmos3-Super instead, load the larger checkpoint: Cosmos3OmniModularPipeline.from_pretrained("nvidia/Cosmos3-Super", ...).

Diffusers notebook walkthrough

run_video_transfer_with_diffusers.ipynb is the full tutorial for the Diffusers backend: it provisions a dedicated venv, then runs edge, blur, depth, segmentation, and world-scenario-map transfer on Cosmos3-Nano followed by the same five controls on Cosmos3-Super. It reads the same specs/ files and reuses the previews from preview_helpers.py, writing outputs to outputs/notebooks/diffusers/<model>/transfer_<control>/vision.mp4.

Run with TensorRT-LLM

Quickstart

Set up and launch Nano or Super as described in the shared TensorRT-LLM setup, including the server-side NLTK data setup. Keep NLTK_DATA exported in the server's shell. TensorRT-LLM accepts a raw source video as multipart video_reference when edge or blur will be computed on the server. The checked-in assets are already precomputed controls, so this example base64-encodes the control inside its hint and does not send a top-level reference. The server decodes inline media at the HTTP boundary before validating and dispatching the request.

import base64
import json
from pathlib import Path

import requests

transfer_root = Path(".agents/skills/cookbooks/cosmos3/generator/transfer")
control_path = transfer_root / "assets/depth/control_depth.mp4"
prompt = json.dumps(json.load(open(transfer_root / "assets/depth/prompt.json")))
negative = json.dumps(json.load(open(transfer_root / "assets/negative_prompt.json")))
extra_params = {
    "use_resolution_template": False,
    "use_duration_template": False,
    "use_system_prompt": False,
    "use_guardrails": True,
    "flow_shift": 10.0,
    "depth": {
        "control": base64.b64encode(control_path.read_bytes()).decode("ascii")
    },
    "control_guidance": 1.5,
    "num_video_frames_per_chunk": 121,
    "num_conditional_frames": 1,
    "num_first_chunk_conditional_frames": 0,
    "share_vision_temporal_positions": True,
    "emphasize_control_in_prompt": False,
    "max_frames": 121,
}

response = requests.post(
    "http://localhost:8000/v1/videos/sync",
    json={
        "prompt": prompt,
        "negative_prompt": negative,
        "size": "1280x720",
        "num_frames": 121,
        "fps": 30,
        "num_inference_steps": 50,
        "guidance_scale": 3.0,
        "max_sequence_length": 4096,
        "seed": 2026,
        "format": "mp4",
        "response_format": "file",
        "extra_params": extra_params,
    },
    headers={"Accept": "video/mp4"},
)
response.raise_for_status()
if (
    "video/mp4" not in response.headers.get("content-type", "")
    or response.content[4:8] != b"ftyp"
):
    raise RuntimeError("TensorRT-LLM did not return browser-compatible MP4")
Path("/tmp/cosmos3_transfer_depth_trtllm.mp4").write_bytes(response.content)

Only edge and blur can be generated from a raw uploaded source ("edge": true or "blur": true); depth, segmentation, and WSM always require precomputed control media. Precomputed edge and blur are also accepted. Use use_guardrails, not vLLM-Omni's guardrails, and send encoded control bytes rather than a server-local control_path. When neither output dimension is specified, TensorRT-LLM chooses the nearest supported bucket from the source or first precomputed control's aspect ratio. The checked-in blur example explicitly requests its native 4:3 bucket, 1104×832; the other examples request 1280×720. All preserve their native frame count/fps, and WSM uses 100 frames at 10 fps. /v1/videos/generations remains only as a deprecated alias of the canonical blocking /v1/videos/sync route.

TensorRT-LLM notebook walkthrough

run_video_transfer_with_trt_llm.ipynb runs edge, blur, depth, segmentation, and WSM against an already-running Nano or Super server. It reuses the checked-in media and captions, shows the complete mode matrix, validates the JSON/base64 control request contract, and previews the synchronous encoded-video responses.

Run with vLLM-Omni

Quickstart

vLLM-Omni accepts multiple control hints in one request, but does not support the per-hint weight field available in Cosmos Framework.

Set up the environment and start the server: vLLM-Omni setup. Transfer controls are available from vLLM-Omni main and the released vllm/vllm-omni:cosmos3 container. Check the current Cosmos3-Nano recipe before selecting an image. For Docker, run the command from the cookbook root (.agents/skills) so it is mounted at /workspace and the server runs from that directory inside the container:

export COSMOS3_WORKDIR="$(pwd)"
export COSMOS3_HOST_PORT=8000

The transfer examples send repo-local control_path strings to the server. For Docker, those paths must be visible from the server working directory. With the shared Docker setup, the checked-in depth control video is:

cookbooks/cosmos3/generator/transfer/assets/depth/control_depth.mp4

If your server does not run from the cookbook root, start it from there or adjust control_path to a path the server process can read.

Transfer requests should also pass the spec resolution inside extra_params. Cosmos3 transfer bucket selection reads extra_params.resolution and the control/input aspect ratio; set the video API size field to the matching <width>x<height> value for a consistent request.

Send a depth-transfer request:

import json
from pathlib import Path

import requests

transfer_root = Path(".agents/skills/cookbooks/cosmos3/generator/transfer")
prompt = json.dumps(json.load(open(transfer_root / "assets/depth/prompt.json")))
negative = json.dumps(json.load(open(transfer_root / "assets/negative_prompt.json")))
control_path = transfer_root / "assets/depth/control_depth.mp4"

response = requests.post(
    "http://localhost:8000/v1/videos/sync",
    data={
        "prompt": prompt,
        "negative_prompt": negative,
        "size": "1280x720",
        "num_frames": "121",
        "fps": "30",
        "num_inference_steps": "50",
        "guidance_scale": "3.0",
        "flow_shift": "10.0",
        "seed": "2026",
        "extra_params": json.dumps(
            {
                "use_resolution_template": False,
                "use_duration_template": False,
                "guardrails": True,
                "depth": {"control_path": control_path.as_posix()},
                "resolution": "720",
                "control_guidance": 1.5,
                "num_video_frames_per_chunk": 121,
                "max_frames": 121,
            }
        ),
    },
    headers={"Accept": "video/mp4"},
)
response.raise_for_status()
Path("/tmp/cosmos3_transfer_depth.mp4").write_bytes(response.content)

Spec field reference

A representative spec (specs/edge.json):

{
  "name": "transfer_edge",
  "model_mode": "video2video",
  "resolution": "720",
  "aspect_ratio": "16,9",
  "num_frames": 121,
  "fps": 30,
  "num_video_frames_per_chunk": 121,
  "num_conditional_frames": 1,
  "num_first_chunk_conditional_frames": 0,
  "share_vision_temporal_positions": true,
  "guidance": 3.0,
  "control_guidance": 1.5,
  "negative_prompt_file": "../assets/negative_prompt.json",
  "prompt_path": "../assets/edge/prompt.json",
  "edge": {
    "control_path": "../assets/edge/control_edge.mp4",
    "preset_edge_threshold": "medium"
  }
}

Key fields:

• resolution — target resolution (e.g. 720 for 720p).

• aspect_ratio — aspect ratio of the control video; together with resolution determines the spatial dimensions (e.g. 720 + 16,9 → 1280 × 720).

• fps — model conditioning signal and playback rate of the saved output video. Should match the native fps of the control video.

• num_frames — number of video frames.

Cookbook entrypoints

• run_video_transfer_with_cosmos_framework.ipynb — self-contained notebook: §9–§13 Nano single-control, §14–§18 Super single-control, §19 multi-control (Nano). Edit §2, run top-to-bottom.
• run_video_transfer_with_diffusers.ipynb — full tutorial for the Diffusers modular pipeline: five single-control transfers on Nano, then the same five on Super, driven by the same specs.
• run_video_transfer_with_trt_llm.ipynb — five single-control transfers through TensorRT-LLM's synchronous VisualGen API, using base64-encoded precomputed controls in JSON extra_params.
• run_video_transfer_with_vllm_omni.ipynb — full tutorial against an already-running vLLM-Omni server: endpoint checks, repo-local control paths, five single-control transfer requests, and compact previews. The API also accepts unweighted multi-control requests by including multiple hint blocks in extra_params.
• specs/ — checked-in Framework input JSON per control (paths relative to specs/). Shared by both Nano and Super.

Troubleshooting

If inference fails inside attention with a NATTEN/libnatten error, verify that the active Python environment uses a matching Torch and NATTEN build. Avoid mixing a container-provided Torch/NATTEN stack with packages from ~/.local or another venv. In containerized environments, PYTHONNOUSERSITE=1 can help prevent user-site packages from shadowing the container stack.