Skip to content

Cosmos3 NIM

Deploy Cosmos3 models for world generation and multimodal reasoning. One image contains two runtimes, but each container runs only one:

  • Generator serves image/video generation, Action, and Transfer through POST /v1/infer.
  • Reasoner serves OpenAI-compatible image/video understanding through Chat Completions and, when enabled, the Responses API.

Deployment commands pin the current release image, nvcr.io/nim/nvidia/cosmos3:2.1.0. Compatible configurations are in the Support Matrix.

Architecture

Cosmos3 is a Mixture-of-Transformers (MoT) world foundation model with two jointly trained towers that share latent representations. The Reasoner (vision-language) tower grounds the prompt. The Generator (world simulator) tower produces the output, such as video. The Generator runtime uses both towers, and the Reasoner runtime serves understanding. For details, refer to the Cosmos3 technical report.

Choose What to Run

Choose a runtime, a model variant (NIM_MODEL_VARIANT), optionally a precision, and for Generator a latency or throughput objective. The NIM selects a compatible profile, as described in How Selection Works.

Choose a Task

Task Runtime Input Output Guide
Text-to-image Generator Prompt JPEG image Generation
Text-to-video Generator Prompt MP4 video Generation
Image-to-video Generator Prompt + image MP4 video Generation
Video-to-video Generator Prompt + video MP4 video Generation
Forward dynamics Generator Image + action trajectory Rollout video Action
Policy Generator Image + task/state Video + predicted action Action
Nano-DROID policy Generator Image + task/current state Predicted action Action
Edge-DROID policy (experimental) Generator Composed camera views + task/current state Predicted action Action
Inverse dynamics Generator Video Video + predicted action Action
Video Transfer Generator Prompt + spatial control Controlled MP4 video Transfer
Image/video reasoning Reasoner Media + text Text or structured result Reasoning
Streaming video reasoning Reasoner Sequential image frames Per-frame text with retained context Streaming
Responses API Reasoner Responses input Response object/text Reasoning

Local examples publish either runtime at http://localhost:8000. Remove the active example container before launching the other runtime on the same port.

Get Started

Use an Existing NIM Endpoint

If an administrator already deployed the NIM, install and initialize the Client Tooling, then set the service URL:

cd .agents/skills/cookbooks/cosmos3/nim
export NIM_URL=${NIM_URL:-http://localhost:8000}
curl -fsS "$NIM_URL/v1/health/ready"
curl -fsS "$NIM_URL/v1/metadata" | python3 -m json.tool

Confirm that metadata identifies the intended runtime and primary endpoint. Generator uses /v1/infer, and Reasoner uses /v1/chat/completions.

Run an example that matches the active runtime and model. For a general-purpose Generator, start with an image request:

uv run python examples/t2i.py

For a Reasoner:

uv run python examples/reasoner.py --case image_caption

Deploy the NIM Yourself

  1. Verify the GPU host against Prerequisites.
  2. Choose a compatible model and hardware configuration from the Support Matrix.
  3. Follow Deployment to authenticate, pull the image, run the preflight, and launch Generator or Reasoner. On NVIDIA DGX Spark or Jetson AGX Thor, the NIM applies a 70% unified-memory share automatically.
  4. Return to Use an Existing NIM Endpoint to verify the service and run the first request.

Specialist variants accept only their task: run t2i_4step.py, i2v_4step.py, or policy_droid.py only against the matching variant. Examples save outputs under the ignored examples/outputs/ directory.

Validate the selected configuration with representative requests before production use.

AI-Assisted Usage

Open this directory in a compatible coding assistant to get guided help with host preparation, deployment, endpoint verification, request selection, and troubleshooting. The bundled customer instructions and skill use the public pages and examples as their source of truth and default to read-only assistance.

Documentation

Deploy and Configure

Use the APIs

Operate

Safety, License, and Notices

Generator guardrails are enabled by default. Disabling them can remove content-policy and face-privacy protections. Refer to Guardrails.

Refer to License and Notices for the terms. This cookbook is licensed under the repository LICENSE. The running NIM serves its license at /v1/license. The image also contains product terms, notices, and component licenses. Python package licenses are listed in Acknowledgements.

The NIM is distributed as a container image through the NVIDIA NGC Catalog. The catalog publishes a security scan report for each image, with a security rating, CVE severity by package, and links to CVE details.

NVIDIA Developer Program

Want to learn more about NIMs? Join the NVIDIA Developer Program to get free access to self-hosting NVIDIA NIMs and microservices on up to 16 GPUs on any infrastructure: cloud, data center, or personal workstation.

After you join the free NVIDIA Developer Program, you can access NIMs through the NVIDIA API Catalog at any time. For enterprise-grade security, support, and API stability, select the option to access NIM through our free 90-day NVIDIA AI Enterprise Trial with a business email address.

Refer to the NVIDIA NIM FAQ for additional information.