Choose a serving path¶
Use the framework CLI for batch jobs, Ray Serve for an online framework endpoint, a third-party backend for an OpenAI-compatible server or a Python library, NIM for a prebuilt container, and the DROID policy server to drive a robot.
Options¶
| Option | Use when | Entry point | Guide |
|---|---|---|---|
| Framework CLI | Batch generation or reasoning from JSON, JSONL or YAML sample files; one GPU with python -m, several with torchrun --nproc-per-node=N |
python -m cosmos_framework.scripts.inference |
Run inference |
| Ray Serve and Gradio | A long-running HTTP endpoint for framework inference, with an optional browser UI | python -m cosmos_framework.inference.ray.serve, python -m cosmos_framework.inference.ray.gradio |
Online inference with Ray |
| vLLM-Omni | An OpenAI-compatible Generator server for image, video, audio, action and transfer | vllm serve <model> --omni |
vLLM-Omni |
| vLLM | An OpenAI-compatible Reasoner server for image and video understanding | vllm serve <model> |
vLLM |
| TensorRT-LLM | An OpenAI-compatible Generator (VisualGen) or Reasoner server | trtllm-serve <model> |
Generator, Reasoner |
| Diffusers | Image and video generation, with optional audio, from Python | Cosmos3OmniPipeline |
Diffusers |
| Transformers | Reasoner inference from Python | Cosmos3OmniForConditionalGeneration (Nano, Super) |
Transformers |
| Cosmos3 NIM | A prebuilt NGC container with no Python environment to manage; each container runs the Generator or the Reasoner | nvcr.io/nim/nvidia/cosmos3:2.1.0 |
Deploy with NIM |
| DROID policy server | Streaming actions from Cosmos3-Nano-Policy-DROID or Cosmos3-Edge-Policy-DROID to a simulated or real robot | python -m cosmos_framework.scripts.action_policy_server_robolab |
Serve DROID policies |
How to choose¶
- The CLI and Ray Serve both take
--parallelism-preset.latency(the default) splits each sample across GPUs with context parallelism; on several GPUs also pass--dp-shard-size=1.throughputshards the model across all GPUs and runs a batch. - Generator modes differ by path. Cosmos3-Edge cannot generate audio, and TensorRT-LLM serves it for text-to-image, text-to-video and image-to-video only. The Cosmos3 NIM does not generate audio.
- Transfer-Auto, Sim-Auto, Sim-Transfer and Sim-Bimanual run on Cosmos Framework and vLLM-Omni; see the Simulation notebooks.
- NIM authenticates with an NGC API key instead of a Hugging Face token. In the Cosmos3 NIM,
NIM_MODEL_VARIANTselectsnanoorsuper,edgefor the Reasoner, or one of the Generator specialistsedge-droid(experimental),nano-droid,super-t2i,super-t2i-4step,super-i2vandsuper-i2v-4step.