Skip to content

Cosmos3 Reasoner Examples

Run the Cosmos3 Reasoner (vision-language reasoning over images and video) across multiple inference backends. Sample inputs live under assets/.

Environment setup for every backend is centralized in the shared Cosmos3 cookbooks environment setup guide; each backend below links to the section you need.

Choosing a backend (memory footprint)

Cosmos3 ships as a unified omni checkpoint (Reasoner + generator/diffusion), so how much GPU memory you pay depends on the backend:

  • The Cosmos Framework backend loads the full omni weights (Reasoner plus the generator: video VAE, audio tokenizer, and diffusion expert). It therefore uses the most memory — even for text-only reasoner inference. model_mode=reasoner only changes the execution path; it does not unload the generator weights.
  • The vLLM, TensorRT-LLM, Transformers, and NIM backends load only the Reasoner (VLM) weights, so they use much less memory.

If you need to save GPU memory for reasoner-only inference, use vLLM, TensorRT-LLM, Transformers, or NIM. Reach for the Cosmos Framework backend only when you also need generation.

For example, on a single GPU the Cosmos3-Nano Reasoner needs roughly ~34 GB with the Cosmos Framework backend versus ~16–17 GB with vLLM / TensorRT-LLM / Transformers / NIM — about half. Larger tiers (e.g. Cosmos3-Super) use more in absolute terms, but the same full-omni-vs-reasoner-only gap applies.

Reasoner Prompt Guide

See the Reasoner Prompt Guide.

Run with Cosmos Framework

Quickstart

Set up the environment: Cosmos Framework setup. This produces the framework venv at packages/cosmos3/.venv; run the commands below from that checkout (cd packages/cosmos3).

Create a Reasoner input file. The Reasoner does not currently support the audio modality by design, so the enable_sound=false field is intentional and required. The framework's strict validation rejects audio requests on a model that doesn't support audio, rather than ignoring the flag: 

mkdir -p outputs/cookbooks/cosmos3/reasoner/inputs
cat > outputs/cookbooks/cosmos3/reasoner/inputs/robot_image.json <<'JSON'
{
  "model_mode": "reasoner",
  "name": "robot_image",
  "prompt": "Describe what is happening in this image in one sentence.",
  "vision_path": "../../cookbooks/cosmos3/reasoner/assets/robot_153.jpg",
  "enable_sound": false
}
JSON

Run Nano on a single GPU:

COSMOS_TRAINING=false CUDA_VISIBLE_DEVICES=0 \
MASTER_ADDR=127.0.0.1 MASTER_PORT=29501 RANK=0 WORLD_SIZE=1 LOCAL_RANK=0 \
.venv/bin/python -m cosmos_framework.scripts.inference \
  --parallelism-preset=latency \
  -i outputs/cookbooks/cosmos3/reasoner/inputs/robot_image.json \
  -o outputs/cookbooks/cosmos3/reasoner/nano/cosmos_framework_image \
  --checkpoint-path Cosmos3-Nano \
  --seed=0 \
  --benchmark

The generated text is written to outputs/cookbooks/cosmos3/reasoner/nano/cosmos_framework_image/robot_image/reasoner_text.txt.

Notebook walkthrough

run_with_cosmos_framework.ipynb is the full tutorial cover Nano, Super, Edge. It writes text and image smoke tests, then walks through image capability sections — detailed captioning, robot task planning, 2D grounding, describe-anything, and action-trajectory prompts — rendering the prompt, media input, model output, and any parsed boxes or trajectories together for review. It also shows how to scale from Nano (single GPU, shown above) to Super (4 GPUs via .venv/bin/torchrun). Video examples live in the vLLM notebook, because the current Cosmos Framework Reasoner path expects image inputs through vision_path.

Run with vLLM

Quickstart

Set up the environment and start the server: vLLM setup (install) and Start the server (launch commands).

The quickstart below uses Cosmos3-Nano on port 8000. The run_with_vllm.ipynb notebook defaults to Cosmos3-Super on port 8001 — use that launch command from the env setup guide and point the client at http://localhost:8001/v1.

Once the server is ready, query it with the OpenAI client:

from pathlib import Path
import openai

image_path = Path("assets/robot_153.jpg").resolve()
image_url = image_path.as_uri()

client = openai.OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": image_url}},
                {"type": "text", "text": "Caption the image in detail."},
            ],
        }
    ],
    max_tokens=4096,
    seed=0,
)

print(response.choices[0].message.content)

Notebook walkthrough

run_with_vllm.ipynb uses the Cosmos3-Super launch from the environment setup guide and walks through many more image and video examples: detailed captioning, VQA, temporal localization, embodied reasoning, common-sense reasoning, 2D grounding, describe-anything, action CoT trajectories, driving scenes, physical-plausibility, and situation understanding. To run a different model, change only the server launch command and the client base_url; every example resolves the model dynamically (MODEL = client.models.list().data[0].id), so the prompts work unchanged.

Run with NIM

Quickstart

Set up the environment: NIM setup. The Cosmos3 NIM serves the Reasoner through the same OpenAI-compatible chat-completions API as the vLLM path on port 8000, without the vLLM/CUDA setup. You can also try this same NIM interactively in your browser on the cosmos3-nano-reasoner build page. Launch it with NIM_MODEL_TYPE=reasoner:

export NGC_API_KEY=<your_key>
export LOCAL_NIM_CACHE="${LOCAL_NIM_CACHE:-$HOME/.cache/nim}"
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R 777 "$LOCAL_NIM_CACHE" 2>/dev/null || true

docker run -d --name cosmos3-reasoner \
  --gpus '"device=0"' \
  --shm-size 16g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --ulimit nofile=65536:65536 \
  -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_MODEL_TYPE=reasoner \
  -e NIM_MODEL_VARIANT=nano \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/nvidia/cosmos3:2.1.0

For Super, Edge, and unified-memory systems such as DGX Spark, refer to Launch Reasoner and Deploy Edge Models. Once the server is up, query it with the OpenAI client and resolve the served model name dynamically:

import base64
import mimetypes
from pathlib import Path
import openai

image_path = Path("assets/robot_153.jpg").resolve()
mime = mimetypes.guess_type(image_path.name)[0] or "application/octet-stream"
image_url = f"data:{mime};base64,{base64.b64encode(image_path.read_bytes()).decode('ascii')}"

client = openai.OpenAI(api_key="not-used", base_url="http://127.0.0.1:8000/v1")
model = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": image_url}},
                {"type": "text", "text": "Caption the image in detail."},
            ],
        }
    ],
    max_tokens=4096,
)

print(response.choices[0].message.content)

Video inputs use the video_url content type and accept media_io_kwargs frame-sampling controls through extra_body ({"media_io_kwargs": {"video": {"fps": 4.0}}}). See the Cosmos3 NIM reasoning guide for the full request reference, the Responses API, structured output, and a runnable task catalog under nim/examples/.

Run with Transformers

Quickstart

Set up the environment: Transformers setup. This installs a Transformers release with the Cosmos3 Nano/Super integration.

Run Cosmos3-Nano Reasoner inference in process:

from pathlib import Path

import torch
from transformers import AutoProcessor, Cosmos3OmniForConditionalGeneration

model_id = "nvidia/Cosmos3-Nano"
image_path = Path("assets/robot_153.jpg").resolve()

processor = AutoProcessor.from_pretrained(model_id)
model = Cosmos3OmniForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "path": str(image_path)},
            {"type": "text", "text": "Caption the image in detail."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device, torch.bfloat16)

generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)
print(output[0])

The Transformers integration loads only the Reasoner tower from the unified checkpoint and drops Generator, audio, and action parameters. It returns text for text, image, and video understanding tasks; it does not generate images, videos, audio, or actions.

For video inputs, use a video content block and pass a sampling rate to apply_chat_template:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "path": "assets/video_caption.mp4"},
            {"type": "text", "text": "Describe the notable events in this video."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    fps=2,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device, torch.bfloat16)

Then reuse the model.generate and batch_decode block from the image example.

To run Cosmos3-Super, change model_id to nvidia/Cosmos3-Super. device_map="auto" can shard the model across multiple GPUs when Accelerate is installed. Cosmos3-Edge is covered in the notebook below. Use vLLM or NIM when you need an OpenAI-compatible server instead of local Python inference.

Notebook walkthrough

run_with_transformers.ipynb is the Python-first counterpart to the server notebooks: instead of launching a server, it installs an isolated venv (Transformers from main, so Edge is available), registers a Cosmos3 Transformers (Python 3.13) Jupyter kernel, and loads the Reasoner in process. A small run_reasoner helper wraps apply_chat_template + generate, and the notebook then runs the image and video examples shown above. Set model_id to nvidia/Cosmos3-Nano, nvidia/Cosmos3-Super, or nvidia/Cosmos3-Edge in the load cell and re-run; Nano/Super use Cosmos3OmniForConditionalGeneration, while Edge uses AutoModelForImageTextToText. device_map="auto" shards Super across multiple GPUs.

Run with TensorRT-LLM

Quickstart

Set up the environment and start the server with the shared TensorRT-LLM Reasoner setup. It covers the prebuilt release container or a local source installation, Hugging Face authentication, and launch commands for Cosmos3-Nano on one GPU or Cosmos3-Super on four GPUs.

The run_with_tensorrt_llm.ipynb notebook publishes the selected server on host port 8001 and waits for /health before running the request examples. Point standalone clients at http://localhost:8001/v1.

Once the server is ready, query it with the OpenAI client:

from pathlib import Path
import openai

image_path = Path("assets/robot_153.jpg").resolve()
image_url = image_path.as_uri()

client = openai.OpenAI(api_key="EMPTY", base_url="http://localhost:8001/v1")

response = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": image_url}},
                {"type": "text", "text": "Caption the image in detail."},
            ],
        }
    ],
    max_tokens=4096,
    seed=0,
)

print(response.choices[0].message.content)

Notebook walkthrough

run_with_tensorrt_llm.ipynb launches either Cosmos3-Nano or Cosmos3-Super, waits for the server to become healthy, and walks through image and video examples covering detailed captioning, temporal localization, embodied reasoning, common-sense reasoning, 2D grounding, describe-anything, action CoT trajectories, driving scenes, physical plausibility, and situation understanding.