# Cosmos3 Cookbooks: Environment Setup

Shared environment setup for every Cosmos3 cookbook (Reasoner, Generator, and
Simulation).
Each cookbook README links back here for the backend(s) it supports — pick the
backend you want to run and follow that one section.

| Backend | Use it for | Used by |
| --- | --- | --- |
| [Cosmos Framework](#cosmos-framework) | Native PyTorch inference, launched with `torchrun` | Reasoner, Generator (Audiovisual, Action, **Transfer**), Simulation (Bimanual, Auto, Sim-Auto, Sim-Transfer) |
| [Diffusers](#diffusers) | Direct generation with `Cosmos3OmniPipeline` | Generator (Audiovisual) |
| [TensorRT-LLM Generator](#tensorrt-llm-generator) | OpenAI-compatible VisualGen server (image/video/audio/action/transfer generation) | Generator (Audiovisual, Action, **Transfer**) |
| [TensorRT-LLM Reasoner](#tensorrt-llm-reasoner) | OpenAI-compatible image/video reasoning server | Reasoner |
| [Transformers](#transformers) | Hugging Face Transformers inference | Reasoner |
| [vLLM](#vllm) | OpenAI-compatible reasoning server (image/video understanding) | Reasoner |
| [vLLM-Omni](#vllm-omni) | OpenAI-compatible generation server (image/video/audio/action/transfer) | Generator (Audiovisual, Action, **Transfer**), Simulation (Transfer-Auto, [Sim-Auto](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/simulation/auto/run_sim_auto_with_vllm_omni.ipynb), Sim-Transfer) |
| [Cosmos3 NIM](#cosmos3-nim) | Prebuilt NGC container serving either the Generator (image/video/action/transfer) or the Reasoner from one image | Reasoner, Generator (Audiovisual, Action, Transfer) |

## Prerequisites

- Linux with NVIDIA GPU access.
- [`uv`](https://docs.astral.sh/uv/getting-started/installation/), `git`, and `git-lfs` installed.
- Hugging Face access to the gated Cosmos3 model repos. Generator also requires
  access to the gated
  [nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail)
  HF repository. Authenticate once before the first run:

  ```bash
  uvx hf@latest auth login
  # or:
  export HF_TOKEN=<your_token>
  ```

  To disable the guardrail, set `enable_safety_checker=False` (Diffusers),
  `TRTLLM_DISABLE_COSMOS3_GUARDRAILS=1` or `use_guardrails: false` through
  `extra_params` (TensorRT-LLM), `guardrails: false` (vLLM-Omni
  `extra_params`/`extra_args`), or `--no-guardrails` (Cosmos Framework). For the Cosmos3 NIM Generator, set `NIM_ENABLE_TEXT_GUARDRAILS=0 NIM_ENABLE_VIDEO_GUARDRAILS=0 NIM_ENABLE_SIGLIP_GUARDRAILS=0`.
- The NIM doesn't need Hugging Face access; instead, an NGC API key is required
  (used as `NGC_API_KEY`). You can generate one on [build.nvidia.com](https://build.nvidia.com/) or [NGC](https://catalog.ngc.nvidia.com/), then run `docker login nvcr.io` once (username `$oauthtoken`, password = your key). This repository uses the Cosmos3 NIM image `nvcr.io/nim/nvidia/cosmos3:2.1.0` for both the Generator and the Reasoner.
- For the Cosmos Framework backend: access to `git@github.com:NVIDIA/cosmos-framework.git`.
- Enough local disk for the venv/image, the uv cache, and the model cache. Nano
  downloads plus CUDA dependencies can take tens of GiB.

### CUDA driver and the `cuXXX` backend

Several backends pin a CUDA build of `torch`/`vllm` that **must match your NVIDIA
driver**. Pick the tag that matches the CUDA version your driver supports:

| Driver CUDA | Backend tag | Notes |
| --- | --- | --- |
| 13.x | `cu130` | Default in the notebooks. |
| 12.x | `cu128` | Use when a compatible wheel is available for the selected package version. |

vLLM does not publish a wheel for every CUDA minor version, so
`--torch-backend=auto` is not reliable here — choose the pair that matches your
driver.

## Cosmos Framework

Native PyTorch inference through the Cosmos Framework checkout. Used by the
`run_*_with_cosmos_framework.ipynb` notebooks and the Cosmos Framework
quickstarts.

From the cookbook root (`.agents/skills`), clone (or reuse) the framework checkout:

```bash
mkdir -p packages
git clone https://github.com/NVIDIA/cosmos-framework.git packages/cosmos3
cd packages/cosmos3
```

Install the framework dependencies into its venv. The inference path currently
imports modules from the training extras, so use the `*-train` dependency group
that matches your driver (see [CUDA driver and the `cuXXX` backend](#cuda-driver-and-the-cuxxx-backend)):

```bash
# lerobot tracks test artifacts with git-LFS that this cookbook does not need;
# skipping smudge avoids failures from missing LFS blobs in uv's git mirror.
export GIT_LFS_SKIP_SMUDGE=1

# CUDA 13 driver (default):
uv sync --all-extras --group=cu130-train

# CUDA 12.x driver:
# uv sync --all-extras --group=cu128-train
```

The notebooks honor `COSMOS3_UV_GROUP` (default `cu130-train`); set
`export COSMOS3_UV_GROUP=cu128-train` before launching them on CUDA 12.x systems.

This produces a venv at `packages/cosmos3/.venv`. Run framework commands either
by activating it (`source .venv/bin/activate`) or via its absolute interpreter
(`.venv/bin/python`, `.venv/bin/torchrun`).

### Recommended base image (optional)

For CUDA 13, NVIDIA documents the [NGC PyTorch container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)
`nvcr.io/nvidia/pytorch:25.09-py3` as the recommended starting point; for CUDA 12 use
`nvcr.io/nvidia/pytorch:25.06-py3`. See
[Cosmos Framework setup](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/setup.md#recommended-base-image).

Inside that image (or any minimal GPU host), install the system packages below **before**
your first `torchrun` inference — `uv sync --all-extras` alone is not enough for
guardrails.

### System packages (required for Framework guardrails)

Framework inference enables **guardrails by default**. The video guardrail path imports
OpenCV (via RetinaFace), which needs graphics libraries that are often missing on
headless servers and minimal containers.

From `packages/cosmos3` (or the framework repo root), with `apt-get` available.
NGC and many training containers run as **root** — use `apt-get` directly (no `sudo`).
On a normal host where you are not root, prefix with `sudo`.

```bash
apt-get update
apt-get install -y --no-install-recommends \
  curl ffmpeg git-lfs libgl1 libglib2.0-0 libx11-dev libxcb1 tree wget
```

Verify OpenCV imports after `source .venv/bin/activate`:

```bash
python -c "import cv2; print(cv2.__version__)"
```

If you see `libxcb.so.1: cannot open shared object file`, the `libxcb1` / `libgl1`
packages above were not installed.

Activate the environment while preserving `LD_LIBRARY_PATH`, including any CUDA
compatibility libraries required by the host driver. Verify CUDA initialization
before inference:

```bash
source .venv/bin/activate
python -c 'import torch; torch.cuda.init(); print(torch.cuda.get_device_name())'
```

If PyTorch fails to import because of conflicting container libraries, follow
[Framework library troubleshooting](../../../setup.md#pytorch-import-issue)
to remove the conflicting entries while retaining required CUDA compatibility paths.

Guardrails also require Hugging Face access to the gated safety models (accept the
license and set `HF_TOKEN` as in [Prerequisites](#prerequisites)). To disable guardrails
for a one-off run, pass `--no-guardrails` to `cosmos_framework.scripts.inference`.

## Diffusers

Direct generation with `Cosmos3OmniPipeline` (Generator · Audiovisual). Create a
venv and install the backend, choosing `--torch-backend` to match your driver
(see [CUDA driver and the `cuXXX` backend](#cuda-driver-and-the-cuxxx-backend)):

```bash
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate

uv pip install --torch-backend=cu130 \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate \
  av \
  cosmos_guardrail \
  huggingface_hub \
  imageio \
  imageio-ffmpeg \
  torch \
  torchvision \
  transformers
```

## TensorRT-LLM Generator

OpenAI-compatible **VisualGen** server for Generator audiovisual text-to-image,
text-to-video, image-to-video, video-to-video, synchronized audio, Transfer, and
Action examples.
Use a TensorRT-LLM build that includes the Cosmos3 VisualGen backend and the
modality required by your notebook.

Install TensorRT-LLM following its upstream documentation.

To build TensorRT-LLM from source, follow NVIDIA's
[Build from Source](https://nvidia.github.io/TensorRT-LLM/installation/build-from-source.html)
guide. This is the right path when you need a checkout that contains a recent
Cosmos3 VisualGen change before it is available in your installed package or
release image.

```bash
apt-get update && apt-get -y install git git-lfs
git lfs install

git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM
git checkout bca6761ab84fbcd58fc7f914eade7de48b32e35e
git submodule update --init --recursive
git lfs pull

# Pick a devel tag from the upstream build-from-source guide or NGC.
docker pull nvcr.io/nvidia/tensorrt-llm/devel:<tag>
docker run --rm -it \
  --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all \
  --volume "$PWD":"$PWD" \
  --workdir "$PWD" \
  nvcr.io/nvidia/tensorrt-llm/devel:<tag>

# Inside the container:
apt-get update && apt-get -y install ffmpeg
python3 scripts/build_wheel.py --use_ccache --skip_building_wheel --linking_install_binary
pip install -e .
```

For Python-only changes, the upstream guide also documents
`TRTLLM_USE_PRECOMPILED=1 pip install -e .` to reuse precompiled binaries while
installing the checkout in editable mode.

Then install the Cosmos3 guardrail package in the same environment unless you
explicitly disable guardrails before starting the server:

```bash
pip install cosmos_guardrail==0.3.0
# On headless servers without libGL.so.1, replace the OpenCV wheel pulled in by
# cosmos_guardrail with the matching headless build:
pip uninstall -y opencv-python
pip install opencv-python-headless==5.0.0.93
```

#### Server-side NLTK data setup

With `cosmos_guardrail==0.3.0`, NLTK's path checks can reject tokenizer or
dictionary files that are symlinks from a Hugging Face snapshot into its
`blobs/` directory. Prepare a separate copy of the NLTK data as regular files
**on the server, in the same container and shell used to launch TensorRT-LLM**:

```bash
export NLTK_DATA="$(mktemp -d "${TMPDIR:-/tmp}/cosmos3-nltk.XXXXXX")"
python3 - <<'PY'
import os
import shutil
from pathlib import Path

import nltk
from huggingface_hub import snapshot_download

snapshot = snapshot_download(
    "nvidia/Cosmos-1.0-Guardrail",
    revision="cf03c0395fac8c4de386c0bdab12cc4fc8d66362",
    allow_patterns=["blocklist/**"],
)
source = Path(snapshot) / "blocklist" / "nltk_data"
destination = Path(os.environ["NLTK_DATA"]).resolve()
shutil.copytree(source, destination, symlinks=False, dirs_exist_ok=True)
assert not any(path.is_symlink() for path in destination.rglob("*"))

# Check both resource lookups used by the text blocklist before starting a GPU server.
nltk.data.path[:] = [str(destination)]
tokens = nltk.word_tokenize("You are an autonomous vehicle planning system.")
assert nltk.WordNetLemmatizer().lemmatize("vehicles") == "vehicle"
print("Guardrail NLTK data ready:", destination, tokens)
PY
```

Keep `NLTK_DATA` exported when starting the server below. Repeat this setup
after recreating the container or removing the temporary directory. Running it
only in the client notebook's environment does not configure a remote server.
This workaround does not rewrite cached files or symlinks, and keeps NLTK
path security and `use_guardrails=True` enabled.

The separate `No safety models found, returning safe` warning in guardrail
0.3.0 refers to its intentionally empty video-content classifier list. Text
checks and face blurring remain configured; the NLTK workaround does not enable
video-content classification or suppress that warning.

Set the TensorRT-LLM source root for the shared VisualGen config YAMLs. Run this
from inside the TensorRT-LLM checkout — the directory the `git clone` above
created, which is where `examples/` lives — or point `TRTLLM_ROOT` at that
checkout explicitly. `trtllm-serve` only reports a bad `--visual_gen_args` path
after it has started, so check it here instead:

```bash
export TRTLLM_ROOT="${TRTLLM_ROOT:-$PWD}"
export COSMOS3_TRTLLM_PORT="${COSMOS3_TRTLLM_PORT:-8000}"

test -d "$TRTLLM_ROOT/examples/visual_gen/configs" \
  || echo "TRTLLM_ROOT=$TRTLLM_ROOT does not look like a TensorRT-LLM checkout"
```

**Cosmos3-Nano** (single GPU):

```bash
trtllm-serve nvidia/Cosmos3-Nano \
  --visual_gen_args "$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-nano-1gpu.yaml" \
  --port "$COSMOS3_TRTLLM_PORT"
```

**Cosmos3-Super** (four GPUs; CFG parallelism with Ulysses, plus parallel VAE):

```bash
torchrun --nproc_per_node=4 -m tensorrt_llm.commands.serve \
  nvidia/Cosmos3-Super \
  --visual_gen_args "$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-super-4gpu.yaml" \
  --port "$COSMOS3_TRTLLM_PORT"
```

**Cosmos3-Edge** (single GPU):

```bash
trtllm-serve nvidia/Cosmos3-Edge \
  --port "$COSMOS3_TRTLLM_PORT"
```

Edge is the compact 4B checkpoint. Its 480p-native generation defaults
(832x480 with 121 frames, 50 UniPC steps on the checkpoint-declared native flow
schedule, guidance 5.0, flow shift 3.0) are read from the checkpoint, so it takes
no `--visual_gen_args` override. Edge text-to-image goes to
`/v1/images/generations` with `"output_type": "image"` in `extra_params` (video
mode would otherwise apply Cosmos3's video negative prompt to a still); the two
video modes go to `/v1/videos/generations`. TensorRT-LLM serves Edge for
text-to-image, text-to-video, and image-to-video only: Edge has no audio tower, its action
weights are not served by this pipeline, and video-to-video is validated for Nano
and Super. Requests outside the model card's validated envelope (256p/480p,
50-150 frames, 12-30 FPS) still run and log an advisory line.

**Cosmos3-Super-Text2Image-4Step** (single GPU; DMD2-distilled text-to-image):

```bash
trtllm-serve nvidia/Cosmos3-Super-Text2Image-4Step \
  --visual_gen_args "$TRTLLM_ROOT/examples/visual_gen/configs/cosmos3-t2i-1gpu.yaml" \
  --port "$COSMOS3_TRTLLM_PORT"
```

**Cosmos3-Super-Image2Video-4Step** (single GPU; DMD2-distilled image-to-video):

```bash
trtllm-serve nvidia/Cosmos3-Super-Image2Video-4Step \
  --port "$COSMOS3_TRTLLM_PORT"
```

Both distilled students run a fixed four-step stochastic schedule read from the
checkpoint's scheduler config, with classifier-free guidance baked into the
weights. TensorRT-LLM supplies both values and rejects a request that sends a
different `num_inference_steps`, or a `guidance_scale` other than `1.0`, so leave
both out of the request. `Cosmos3-Super-Image2Video-4Step` also declares
`default_use_system_prompt: true`, which applies only while the request leaves
`use_system_prompt` unset. The text-to-image student deploys at 1024x1024, the
shape `cosmos3-t2i-1gpu.yaml` warms; the image-to-video student deploys at the
default 720p x 189-frame omni shape and needs no config file. The
[distilled 4-step notebook](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/generator/audiovisual/run_distilled_with_trt_llm.ipynb)
runs both against a running server. These students cover text-to-image and
image-to-video only; use the base checkpoints for text-to-video, video-to-video,
and synchronized audio.

The server exposes `/health`, the blocking `/v1/videos/sync`, the asynchronous
`/v1/videos`, and `/v1/images/generations`. The older
`/v1/videos/generations` spelling is a deprecated alias of `/v1/videos/sync`.
The base audiovisual notebook uses `/v1/images/generations` for text-to-image
and `/v1/videos/sync` for text-to-video, image-to-video, video-to-video, and
synchronized audio. Text-to-image sets `extra_params.output_type="image"` and
returns a base64-encoded PNG. Image-to-video uploads multipart
`image_reference`; video-to-video uses `video_reference`. Synchronized audio is
enabled with `enable_audio: true` in `extra_params` and is muxed into the output
video.
Every video request explicitly selects MP4. Keep `ffmpeg` on the server `PATH`:
without it, the request fails early instead of returning browser-incompatible
AVI or dropping generated audio. Requests send
Cosmos3 controls through `extra_params`, so use a TensorRT-LLM build that includes
the Cosmos3 VisualGen API schema. The notebook sets request-level
`max_sequence_length=4096` for longer structured JSON prompts.

Transfer uses the synchronous `/v1/videos/sync` route. For server-derived edge
or blur, upload the raw source video as multipart `video_reference` and set the
corresponding `extra_params` hint to `true`. For a precomputed edge, blur,
depth, segmentation, or WSM control, base64-encode the control inside its hint;
no top-level reference is needed. The server decodes inline media to bytes at the
HTTP boundary. TensorRT-LLM uses `use_guardrails` for its per-request safety
switch; `guardrails`, `control_path`, and other vLLM-Omni-only names are not
interchangeable.

Action requests use the same synchronous route and upload an image as
`image_reference` or a video as `video_reference`. For the checked-in AV
examples, use the Cosmos Framework reference prompt shown in the Action
cookbook; current TensorRT-LLM ignores the legacy `view_point` field.
Because an action trajectory cannot be represented in MP4 or
AVI, `format=auto` resolves to `safetensors`; the payload contains named `video`,
`action`, and `frame_rate` tensors. The asynchronous `/v1/videos` route also
supports this payload: poll `GET /v1/videos/{id}`, then download it from
`GET /v1/videos/{id}/content`.

## TensorRT-LLM Reasoner

OpenAI-compatible **reasoning** server for image and video understanding. Run
TensorRT-LLM in the prebuilt
[`nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`](https://catalog.ngc.nvidia.com/orgs/nvidia/tensorrt-llm/containers/release)
container, or follow the
[TensorRT-LLM Generator source-build instructions](#tensorrt-llm-generator) to
install TensorRT-LLM from source.
Authenticate with Hugging Face before loading gated checkpoints.

Install headless OpenCV in the environment that runs the server. Cosmos3
Reasoner uses it to process vision inputs:

```bash
python -m pip install opencv-python-headless
```

### Start the server

Run one of these commands inside the release container or activated local
environment.

**Cosmos3-Nano** (single GPU, port 8001):

```bash
CUDA_VISIBLE_DEVICES=0 \
trtllm-serve nvidia/Cosmos3-Nano \
  --host 0.0.0.0 \
  --port 8001 \
  --max_num_tokens 32768
```

**Cosmos3-Super** (four GPUs, port 8001):

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
trtllm-serve nvidia/Cosmos3-Super \
  --host 0.0.0.0 \
  --port 8001 \
  --tensor_parallel_size 4 \
  --max_num_tokens 32768
```

The server exposes `/health` and the OpenAI-compatible API at
`http://localhost:8001/v1`. See the
[Reasoner TensorRT-LLM notebook](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb) for image
and video requests.

## Transformers

Local Python inference for the Cosmos3 Reasoner. This backend uses the
Transformers Cosmos3 integration and loads only the Reasoner tower.

**Nano / Super** load the Reasoner tower from the unified
`nvidia/Cosmos3-Nano` or `nvidia/Cosmos3-Super` checkpoint with
`Cosmos3OmniForConditionalGeneration`. Support first appears in Transformers
`v5.11.0`. Create a venv and install Transformers `5.11.0` or newer:

```bash
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate

uv pip install --torch-backend=auto \
  accelerate \
  av \
  pillow \
  "safetensors>=0.8.0" \
  torch \
  "torchvision==0.25.0" \
  "transformers>=5.11.0"
```

`--torch-backend=auto` lets uv detect the CUDA build of `torch`/`torchvision`
that matches your NVIDIA driver. Pin a backend such as `cu128` or `cu130` if
your environment needs an explicit CUDA wheel.

**Cosmos3-Edge** uses a separate Transformers integration
(`AutoModelForImageTextToText` / `Cosmos3EdgeForConditionalGeneration`) with
`nvidia/Cosmos3-Edge`. Do not load Edge with `Cosmos3OmniForConditionalGeneration`.
Edge support is on Transformers `main` and is not yet in a stable PyPI release;
install from GitHub instead of the
`transformers>=5.11.0` pin above:

```bash
uv pip install "transformers @ git+https://github.com/huggingface/transformers.git"
```

See the
[Reasoner Transformers quickstart](reasoner/README.md#run-with-transformers)
for runnable Nano, Super, and Edge image examples and video input notes.

## vLLM

OpenAI-compatible **reasoning** server for the Reasoner cookbook (image/video
understanding). Native Cosmos3 Reasoner support first appears in the vLLM
`v0.23.0` stable release:

```bash
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate

# CUDA 13 driver:
uv pip install --torch-backend=cu130 "vllm>=0.23.0"

# CUDA 12.x driver:
# uv pip install --torch-backend=cu128 "vllm>=0.23.0"
```

The vLLM wheel and the torch backend must be compatible — see
[CUDA driver and the `cuXXX` backend](#cuda-driver-and-the-cuxxx-backend).

If your vLLM build reports that DeepGEMM is unavailable, disable it before
starting the server:

```bash
export VLLM_USE_DEEP_GEMM=0
```

> When launching with `.venv/bin/vllm` instead of activating the venv, make sure
> `.venv/bin` is on `PATH` (e.g. `source .venv/bin/activate`). FlashInfer's
> just-in-time kernel build shells out to `ninja`, which lives in the venv.

### Start the server

All Reasoner cookbooks talk to an OpenAI-compatible chat-completions API. After
[installing vLLM](#vllm), run the commands below from
`.agents/skills/cookbooks/cosmos3/reasoner` (same working directory as
[`run_with_vllm.ipynb`](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/reasoner/run_with_vllm.ipynb)). That sets
`$(dirname "$(pwd)")` to `<cosmos>/.agents/skills/cookbooks/cosmos3`, which matches the
notebook's `COSMOS3_MEDIA_ROOT`.

**Cosmos3-Nano** (single GPU, port 8000):

```bash
CUDA_VISIBLE_DEVICES=0 \
vllm serve nvidia/Cosmos3-Nano \
  --tensor-parallel-size 1 \
  --mm-encoder-tp-mode data \
  --async-scheduling \
  --allowed-local-media-path "$(dirname "$(pwd)")" \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --port 8000
```

**Cosmos3-Super** (four GPUs; default in [`run_with_vllm.ipynb`](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/reasoner/run_with_vllm.ipynb), port 8001):

```bash
export COSMOS3_MEDIA_ROOT="$(dirname "$(pwd)")"
export VLLM_PORT="${VLLM_PORT:-8001}"

CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve nvidia/Cosmos3-Super \
  --tensor-parallel-size 4 \
  --mm-encoder-tp-mode data \
  --async-scheduling \
  --allowed-local-media-path "$COSMOS3_MEDIA_ROOT" \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --port "$VLLM_PORT"
```

**Cosmos3-Edge** (single GPU, port 8000):

```bash
CUDA_VISIBLE_DEVICES=0 \
vllm serve nvidia/Cosmos3-Edge \
  --tensor-parallel-size 1 \
  --mm-encoder-tp-mode data \
  --async-scheduling \
  --allowed-local-media-path "$(dirname "$(pwd)")" \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --port 8000
```

The Super notebook polls `/health` for up to 1800 seconds on first start while CUDA
graphs compile.

| Option | Use |
| --- | --- |
| `--tensor-parallel-size` | Number of GPUs for tensor-parallel inference |
| `--mm-encoder-tp-mode data` | Data parallelism for the visual encoder |
| `--media-io-kwargs '{"video": {"num_frames": -1}}'` | Lets the processor see all frames before downstream sampling |
| `--allowed-local-media-path` | Must cover local `file://` media paths; defaults to `<cosmos>/.agents/skills/cookbooks/cosmos3` when run from `.agents/skills/cookbooks/cosmos3/reasoner` |

## vLLM-Omni

OpenAI-compatible **generation** server (image/video/audio/action/transfer) for the
Generator cookbooks and the Transfer-Auto,
[Sim-Auto](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/simulation/auto/run_sim_auto_with_vllm_omni.ipynb), and
Sim-Transfer Simulation cookbooks. For the
Sim-Transfer model, use its Docker setup and server instructions
or the [runnable notebook](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/simulation/transfer/run_transfer_with_vllm_omni.ipynb).

For Sim-Auto, use the [dedicated notebook](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/simulation/auto/run_sim_auto_with_vllm_omni.ipynb)
for its Docker runtime, HTTP server setup, the shared seven-camera daytime sample,
and output validation. It generates 297 frames per camera at 30 FPS with seed 0 and
tensor parallel size 1, alone or with joint LiDAR.

Cosmos3 checkpoints can exceed the default server init timeout — always pass
`--init-timeout 1800` on every `vllm serve` command below.

### Guardrails (gated dependency)

The vLLM-Omni server loads gated
[nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail)
at startup by default. Without Hugging Face access to that repo, the server
exits before serving requests. Per-request `guardrails: false` in `extra_params`
(see [Prerequisites](#prerequisites)) does not fix this — the guardrail models
must load at startup.

To disable guardrails server-wide (you are responsible for
[license compliance](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license)),
add `--no-guardrails` to any `vllm serve` command below:

```bash
vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --no-guardrails \
  --allowed-local-media-path / \
  --port 8000 \
  --init-timeout 1800
```

See also the
[vLLM-Omni Cosmos3-Nano recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md).

### Option 1: Docker (recommended)

The image `vllm/vllm-omni:cosmos3` is used by the Generator and all Simulation
vLLM-Omni notebooks, including Transfer-Auto, Sim-Auto, Sim-Transfer, and Bimanual.
Use each Simulation notebook's Docker command for its pipeline, deployment config,
timeouts, and media mounts. Pull once:

```bash
docker pull vllm/vllm-omni:cosmos3
```

Set paths once from the cookbook root (`.agents/skills`); adjust for your checkout and cache location:

```bash
export HF_HOME="${HF_HOME:-$HOME/.cache/huggingface}"
export COSMOS3_WORKDIR="${COSMOS3_WORKDIR:-$(pwd)}"
export COSMOS3_HOST_PORT="${COSMOS3_HOST_PORT:-8000}"
```

The container listens on port 8000; `-p "${COSMOS3_HOST_PORT}:8000"` publishes it
on the host. Generator notebooks often use `COSMOS3_HOST_PORT=8001` so port 8000
stays free for a Reasoner server. The Docker commands run from `/workspace`, so
repo-local paths such as `cookbooks/...` resolve inside the container.

**Cosmos3-Nano** (single GPU):

```bash
docker run --runtime nvidia --gpus '"device=0"' \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -v "${HF_HOME}:/root/.cache/huggingface" \
  -v "${COSMOS3_WORKDIR}:/workspace" \
  -p "${COSMOS3_HOST_PORT}:8000" --ipc=host \
  -w /workspace \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano \
    --omni \
    --model-class-name Cosmos3OmniDiffusersPipeline \
    --allowed-local-media-path / \
    --port 8000 \
    --init-timeout 1800
```

**Cosmos3-Super** (all GPUs; add tensor parallelism and layerwise offload):

```bash
docker run --runtime nvidia --gpus all \
  -v "${HF_HOME}:/root/.cache/huggingface" \
  -v "${COSMOS3_WORKDIR}:/workspace" \
  -p "${COSMOS3_HOST_PORT}:8000" --ipc=host \
  -w /workspace \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Super \
    --omni \
    --model-class-name Cosmos3OmniDiffusersPipeline \
    --allowed-local-media-path / \
    --tensor-parallel-size 4 \
    --enable-layerwise-offload \
    --port 8000 \
    --init-timeout 1800
```

**Cosmos3-Edge** (single GPU):

```bash
docker run --runtime nvidia --gpus all \
  -v "${HF_HOME}:/root/.cache/huggingface" \
  -v "${COSMOS3_WORKDIR}:/workspace" \
  -p "${COSMOS3_HOST_PORT}:8000" --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Edge \
    --omni \
    --model-class-name Cosmos3OmniDiffusersPipeline \
    --allowed-local-media-path / \
    --port 8000 \
    --init-timeout 1800
```

**Cosmos3-Super-Text2Image-4Step** (all GPUs; add tensor parallelism and layerwise offload):

```bash
docker run --runtime nvidia --gpus all \
  -v "${HF_HOME}:/root/.cache/huggingface" \
  -v "${COSMOS3_WORKDIR}:/workspace" \
  -p "${COSMOS3_HOST_PORT}:8000" --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Super-Text2Image-4Step \
    --omni \
    --model-class-name Cosmos3OmniDiffusersPipeline \
    --allowed-local-media-path / \
    --tensor-parallel-size 4 \
    --enable-layerwise-offload \
    --port 8000 \
    --init-timeout 1800
```

**Cosmos3-Super-Image2Video-4Step** (all GPUs; add tensor parallelism and layerwise offload):

```bash
docker run --runtime nvidia --gpus all \
  -v "${HF_HOME}:/root/.cache/huggingface" \
  -v "${COSMOS3_WORKDIR}:/workspace" \
  -p "${COSMOS3_HOST_PORT}:8000" --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Super-Image2Video-4Step \
    --omni \
    --model-class-name Cosmos3OmniDiffusersPipeline \
    --allowed-local-media-path / \
    --tensor-parallel-size 4 \
    --enable-layerwise-offload \
    --port 8000 \
    --init-timeout 1800
```

Mount any directory that holds local media or action JSON files referenced in
requests. Set `--allowed-local-media-path /` (as above) when the whole container
filesystem should be readable.

vLLM-Omni prints `Application startup complete.` when the API is ready.

### Option 2: Native venv

To install from `main` instead of Docker, create a venv and pick the CUDA build
that matches your driver (see
[CUDA driver and the `cuXXX` backend](#cuda-driver-and-the-cuxxx-backend)):

```bash
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate

# CUDA 13 driver:
uv pip install --torch-backend=cu130 \
  "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@main"

# CUDA 12.x driver:
# uv pip install --torch-backend=cu128 \
#   "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@main"
```

Run the same `vllm serve` arguments as in the Docker commands above, directly on
the host (no `docker run` wrapper):

```bash
vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --allowed-local-media-path / \
  --port 8000 \
  --init-timeout 1800
```

For Super, add `--tensor-parallel-size 4 --enable-layerwise-offload`.

Additional parallelism options (Docker or native):

| Option | Use |
| --- | --- |
| `--cfg-parallel-size 2` | Runs positive and negative CFG branches on two GPUs |
| `--ulysses-degree 2` | Ulysses sequence parallelism across GPUs |

Ensure the server has enough GPUs for the product of enabled degrees
(`tensor_parallel_size` × `cfg_parallel_size` × `ulysses_degree`).

## NIM

A prebuilt NGC container for Cosmos3. Like vLLM-Omni, NIM runs from Docker, so
there is no venv or `--torch-backend` to manage. Unlike the Hugging Face based
backends, NIM authenticates with an NGC API key instead of a Hugging Face token
(see [Prerequisites](#prerequisites)).

Authenticate Docker to NGC once:

```bash
export NGC_API_KEY=<your_key>
echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
```

The NIM exposes readiness at `GET /v1/health/ready` after model
download, engine initialization, and warmup complete.

### Cosmos3 NIM

A single prebuilt container that serves **either** runtime — one runtime per
container, chosen at launch with `NIM_MODEL_TYPE`:

- **Generator** (`POST /v1/infer`) — text-to-image, text-to-video,
  image-to-video, video-to-video, Action (forward dynamics, inverse dynamics,
  policy), and Transfer. Each request sets an explicit top-level `model_mode`.
- **Reasoner** — OpenAI-compatible image and video understanding through Chat
  Completions and the Responses API.

Start the Nano Generator runtime:

```bash
export NGC_API_KEY=<your_key>
export LOCAL_NIM_CACHE="${LOCAL_NIM_CACHE:-$HOME/.cache/nim}"
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R 777 "$LOCAL_NIM_CACHE" 2>/dev/null || true

docker run -d --name cosmos3-generator \
  --gpus '"device=0"' \
  --shm-size 16g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --ulimit nofile=65536:65536 \
  -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_MODEL_TYPE=generator \
  -e NIM_MODEL_VARIANT=nano \
  -e NIM_PERF_PROFILE=latency \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/nvidia/cosmos3:2.1.0
```

Set `NIM_MODEL_TYPE=reasoner` for the Reasoner runtime. `NIM_MODEL_VARIANT`
selects the checkpoint: `nano` or `super` for either runtime, `edge` for the
Reasoner, plus `edge-droid` (experimental), `nano-droid`, `super-t2i`,
`super-t2i-4step`, `super-i2v`, and `super-i2v-4step` for the Generator. On DGX
Spark/GB10 and Jetson AGX Thor, the Reasoner defaults to a 0.70
`NIM_GPU_MEMORY_UTILIZATION` share of unified memory.

The [Cosmos3 NIM cookbook](nim/README.md) is the full guide —
[deployment](nim/deployment.md), [support matrix](nim/support-matrix.md),
[configuration](nim/configuration.md), and [API reference](nim/api-reference.md),
with worked examples for [generation](nim/generation.md),
[action](nim/action.md), [transfer](nim/transfer.md), and
[reasoning](nim/reasoning.md).

## Verify the environment

For the Cosmos Framework / Diffusers / vLLM venvs, check that PyTorch sees the GPU:

```bash
.venv/bin/python - <<'PY'
import torch

print("torch:", torch.__version__)
print("torch cuda:", torch.version.cuda)
print("cuda available:", torch.cuda.is_available())
print("device count:", torch.cuda.device_count())
if torch.cuda.is_available():
    print("device 0:", torch.cuda.get_device_name(0))
PY
```

For a vLLM / vLLM-Omni / NIM server, confirm it is serving the model (use the host
port you set with `COSMOS3_HOST_PORT` or `VLLM_PORT`):

```bash
curl http://localhost:8000/v1/models
```

A NIM server also exposes a readiness endpoint that returns `200` once the model
is loaded:

```bash
curl http://localhost:8000/v1/health/ready
```