# Choose a serving path

Use the framework CLI for batch jobs, Ray Serve for an online framework endpoint, a third-party backend for an OpenAI-compatible server or a Python library, NIM for a prebuilt container, and the DROID policy server to drive a robot.

## Options

| Option                | Use when                                                                                                                                  | Entry point                                                                                         | Guide                                                                                                                                                                    |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Framework CLI         | Batch generation or reasoning from JSON, JSONL or YAML sample files; one GPU with `python -m`, several with `torchrun --nproc-per-node=N` | `python -m cosmos_framework.scripts.inference`                                                      | [Run inference](../inference.md)                                                                                                                                         |
| Ray Serve and Gradio  | A long-running HTTP endpoint for framework inference, with an optional browser UI                                                         | `python -m cosmos_framework.inference.ray.serve`, `python -m cosmos_framework.inference.ray.gradio` | [Online inference with Ray](../faq.md#q-how-do-i-run-online-inference-with-ray)                                                                                          |
| vLLM-Omni             | An OpenAI-compatible Generator server for image, video, audio, action and transfer                                                        | `vllm serve <model> --omni`                                                                         | [vLLM-Omni](../skills/cookbooks/cosmos3/README.md#vllm-omni)                                                                                                  |
| vLLM                  | An OpenAI-compatible Reasoner server for image and video understanding                                                                    | `vllm serve <model>`                                                                                | [vLLM](../skills/cookbooks/cosmos3/README.md#vllm)                                                                                                            |
| TensorRT-LLM          | An OpenAI-compatible Generator (VisualGen) or Reasoner server                                                                             | `trtllm-serve <model>`                                                                              | [Generator](../skills/cookbooks/cosmos3/README.md#tensorrt-llm-generator), [Reasoner](../skills/cookbooks/cosmos3/README.md#tensorrt-llm-reasoner) |
| Diffusers             | Image and video generation, with optional audio, from Python                                                                              | `Cosmos3OmniPipeline`                                                                               | [Diffusers](../skills/cookbooks/cosmos3/README.md#diffusers)                                                                                                  |
| Transformers          | Reasoner inference from Python                                                                                                            | `Cosmos3OmniForConditionalGeneration` (Nano, Super)                                                 | [Transformers](../skills/cookbooks/cosmos3/README.md#transformers)                                                                                            |
| Cosmos3 NIM           | A prebuilt NGC container with no Python environment to manage; each container runs the Generator or the Reasoner                          | `nvcr.io/nim/nvidia/cosmos3:2.1.0`                                                                  | [Deploy with NIM](../skills/cookbooks/cosmos3/nim/README.md)                                                                                                  |
| DROID policy server   | Streaming actions from Cosmos3-Nano-Policy-DROID or Cosmos3-Edge-Policy-DROID to a simulated or real robot                                | `python -m cosmos_framework.scripts.action_policy_server_robolab`                                   | [Serve DROID policies](../action_policy_droid_server.md)                                                                                                                 |

## How to choose

- The CLI and Ray Serve both take `--parallelism-preset`. `latency` (the default) splits each sample across GPUs with context parallelism; on several GPUs also pass `--dp-shard-size=1`. `throughput` shards the model across all GPUs and runs a batch.
- Generator modes differ by path. Cosmos3-Edge cannot generate audio, and TensorRT-LLM serves it for text-to-image, text-to-video and image-to-video only. The Cosmos3 NIM does not generate audio.
- Transfer-Auto, Sim-Auto, Sim-Transfer and Sim-Bimanual run on Cosmos Framework and vLLM-Omni; see the [Simulation notebooks](../skills/cookbooks/cosmos3/simulation/README.md).
- NIM authenticates with an NGC API key instead of a Hugging Face token. In the Cosmos3 NIM, `NIM_MODEL_VARIANT` selects `nano` or `super`, `edge` for the Reasoner, or one of the Generator specialists `edge-droid` (experimental), `nano-droid`, `super-t2i`, `super-t2i-4step`, `super-i2v` and `super-i2v-4step`.