Customizing model servers#

Overview#

Change a sample’s model selection, GPU placement, or inference endpoints to suit your application and hardware. Keep the model-server settings and worker profile aligned, and stop the relevant services before applying deployment changes.

Existing services#

Each agent sample has a model-servers/ convenience launcher that brings up only its required local inference servers. It waits for readiness and returns; the servers remain running while you start, stop, or develop the agent sample. These launchers are editable development presets. Applications can also use compatible services launched individually or independently operated endpoints. Refer to Deploying samples with NIM endpoints for external NIM endpoints.

Inspect the selected services’ configured ports before invoking a model launcher. Server identities are shared across samples, but reuse checks differ:

vLLM Docker: reuse requires the expected container name and a matching launch-configuration fingerprint. On the selected ports, startup can evict other xr-ai containers and listeners, or recreate a changed container.
Pocket TTS: reuses a listener whose /health endpoint is healthy, without checking ownership or a launch-configuration fingerprint.
Speaker ASR: requires an owned listener and matching service fingerprint. An unmanaged listener or configuration mismatch stops startup; it does not evict or recreate the existing service.

Before launching against occupied ports, establish that services will be reused unchanged or obtain permission from their operator to replace them.

Required local servers#

Sample	Servers
Conversation example	Speaker ASR, Pocket TTS, Nemotron Omni, Cosmos VLM, Clef router
Simple VLM (legacy)	speaker ASR, Pocket TTS, Cosmos VLM
Lab instrument monitoring	Speaker ASR, TTS, Nemotron Omni, Cosmos VLM
Tea making	Speaker ASR, TTS, Nemotron Omni, Nemotron embedding
XR render demo	Speaker ASR, TTS, Nemotron Omni, Cosmos VLM
Procedure capture and guide	Batch STT, Pocket TTS, Nemotron Omni, Cosmos VLM

The conversation example uses Omni for language and Clef for turn routing. Tea making uses Omni for both language and vision; its RAG service needs embeddings. XR rendering uses one Omni server for its LLM roles and does not need embeddings. Hubs, workers, RAG, scene, and rendering processes remain in the agent launcher; the model launcher brings up inference servers only.

The conversation example, Simple VLM, lab monitoring, tea making, and XR render workers require speaker ASR on port 8102. They do not contain a batch-STT model role or automatically fall back to the batch server on 8103. Procedure capture and guide uses batch STT on port 8103 with speaker selection disabled. For explicit batch speech, replace speaker-stt with stt in the sample’s launcher service list and every selected hardware YAML, add the existing batch adapter under models.stt in the worker profile, and disable speaker.enabled in the selected voice-gate YAML. To use automatic selection, retain both servers and a real batch client. Refer to xr-ai-voice for backend policy.

Start, develop, and stop#

The following commands use deprecated Simple VLM as a legacy worked example. For new voice-and-vision applications, start with Conversation example. Run the example commands from agent-samples/simple-vlm-example/:

uv run --project model-servers simple_vlm_example_model_servers
uv run simple_vlm_example

The first command downloads any missing weights, waits for healthy servers, and returns. Configure HF_TOKEN first; refer to Credentials. Automatic GPU detection selects a bundled profile. To choose a reviewed profile explicitly, add --gpu-profile dual_48G_ada to the model command.

Stopping or restarting the agent leaves models warm. Stop its model stack when finished:

uv run --project model-servers simple_vlm_example_model_servers --stop

Stop does not detect GPUs or require download credentials. Without an explicit profile it uses the selected services’ shared default ports. If a hardware profile overrides ports, pass that same --gpu-profile NAME when stopping; this applies its port settings without inspecting the GPU. GPU placement and memory budgets do not affect cleanup. Refer to Command-line reference for each sample’s command. Each launcher has only --stop and --gpu-profile, besides help.

If a later model fails to start, or startup is cancelled, servers that had signaled readiness when cleanup began stay running. Fix the failure and rerun the same model command to reuse them. If the fix changes a server that already started, run the model command with --stop first so the new settings take effect. You can also use --stop to clear retained servers. Unfinished Docker backend launches are stopped through their managed wrappers. Persistent pip-backend vLLM runs in a separate session and can keep loading after its wrapper is stopped; use --stop once it is listening to clear it.

On startup failure, merged server YAML remains in a private model-configs-* directory under the existing run logs, at the paths printed in the failure summary. Successful startup removes those files. Credentials belong in the environment, not YAML; retained snapshots clear the supported inline hf_token field and never copy environment variables. Treat customized configurations and logs as private when sharing diagnostics.

Switch samples or update the checkout#

Use this flow whenever the set of models or server settings changes:

Stop the agent and run its model command with --stop, using the current checkout and the profile that launched it.
Switch samples, update the checkout, or edit the server configuration.
Run the destination sample’s model launcher, wait for readiness, and start the agent.

The downloaded weight cache is retained. You can restart the agent repeatedly without stopping models while its server configuration stays the same.

These stacks use common ports and server identities. They are intended for one local sample stack at a time. Stopping a stack also stops recognized services on its configured ports if another sample or manual invocation started them; --stop cannot selectively stop only services started by the latest invocation. Always stop the source stack before switching so unused models release memory. If any affected service predates your run, obtain its operator’s authorization before that whole-stack shutdown and switch. Starting a stack does not stop services outside its selection.

Launchers read the current configuration; they do not keep a launch history or reconcile every older checkout. If you already switched, use the previous checkout’s stop command and configuration before starting the new stack. This explicit workflow is the supported way to apply upgrades and config edits. Changes to a convenience preset do not require backward-compatible launch plans or migration machinery for old running servers. Existing service ownership checks still refuse to stop unknown listeners. Refer to AI inference servers for runtime persistence and cleanup details.

Developer sample API#

Model-server samples call the public xr_ai_model_server_stack.run_model_stack() entry point. Their code owns argument parsing, the base directory, and the list of required local servers. Configuration composition and process management remain inside the utility; sample code imports only public XR AI APIs.

Configuration layers#

Layer	Location	Responsibility
Service defaults	`services/<service>/<entry>.yaml`	Models, images, ports, cache paths, and common runtime settings
Sample GPU profile	`agent-samples/<sample>/model-servers/yaml/<gpu-profile>.yaml`	Placement, memory budgets, concurrency, and workload overrides for that sample
Model selection	`agent-samples/<sample>/model-servers/main.py`	Explicit list of required local servers
Consumer profile	`agent-samples/<sample>/yaml/models*.json`	Client adapters and endpoints used by workers and capability services

The launcher combines each server’s service defaults with its entry in the selected sample GPU profile. Sample values replace whole top-level fields; there is no recursive inheritance. For example, a hf_overrides mapping replaces the shared mapping entirely. Every selected server must have a mapping in the profile, including {} if no overrides are needed.

The resulting complete YAML files are temporary startup inputs to the existing service commands. Cache paths in service defaults resolve relative to the service YAML; a sample model_cache override resolves relative to that sample profile. The helper makes those paths absolute before launching services. Other relative path settings must be supplied as absolute paths when added to an override. Refer to Sample configuration reference for the checked-in YAML and comments.

Tune a sample and hardware combination#

Edit the sample’s GPU profile rather than copying complete server YAML files. For example, this entry places Cosmos on GPU 0 with an explicit cache:

vlm:
  cuda_visible_devices: "0"
  gpu_memory_utilization: 0.47
  kv_cache_memory_bytes: 1610612736
  max_model_len: 8192
  enforce_eager: true

gpu_memory_utilization controls vLLM’s startup admission threshold. With explicit kv_cache_memory_bytes, the byte value sizes the cache independently; neither is a cap on total server memory. Embedding models retain a fractional memory budget. Place STT and TTS alongside the larger models deliberately.

Context length, concurrency, image and video limits, and CUDA graph settings can also affect memory. Tune them with placement and cache size, accounting for all selected models and application processes; XR rendering also needs graphics memory. The initial per-sample overrides retain the former conservative stack budgets. Removing unused models leaves spare capacity rather than automatically increasing the remaining models’ allocations.

To support custom hardware, copy the closest profile YAML within that sample, edit its server entries, and select its filename stem with --gpu-profile NAME. An explicit profile bypasses automatic selection; it does not establish that the models fit. Validate a cold start of the whole sample stack on the target hardware and inspect vLLM’s reported token capacity and concurrency.

Change a model or endpoint#

Stop models before changing service defaults, sample overrides, or ports. A service default edit affects every sample and standalone launch that uses that server. Keep each consumer profile’s model adapters and endpoints synchronized with the service. Consumer profiles do not need deployment ownership objects; the convenience launcher’s explicit server list controls local startup and shutdown.

For an independently operated or hosted model, start no local replacement: configure the corresponding role in the sample’s consumer profile and launch the agent against that endpoint. If keeping some local models, edit the local launcher list and its GPU-profile entries to match. Refer to xr-ai-models for adapters and credentials.

Launch a service individually#

Service projects already expose commands accepting complete YAML via --config. For example, from the repository root:

uv run --project services/vlm-server vlm_server \
  --config services/vlm-server/vlm_server.yaml

Standalone YAML under services/ is intended for individual launch and can use more memory than a multi-model sample stack. A sample override file is partial and cannot be passed directly to a service command. For a manually assembled stack, provide complete service YAML and tune the aggregate GPU use. Refer to AI inference servers for the service entry points and lifecycle.

RAG embedding health#

Tea making’s RAG service reads the embedding role from the same yaml/models.local.json profile as its worker. It calls embedding health() before indexing and while reporting readiness. Preserve its health settings when changing the endpoint.

For a hosted provider without a health route, set readiness: none in the embedding role’s endpoint. Otherwise use readiness: health and the provider’s health_path. Omitting these settings probes /health, which can prevent RAG startup when that route is absent.

readiness: none disables only the health probe. The OpenAI-compatible embedding adapter still requires a successful GET /v1/models for model identity discovery during indexing, using the embedding role’s credentials. HTTP errors (including 401 and 404), transport failures, and invalid metadata abort startup; fix the endpoint or its access settings before retrying. A successful response with no matching backing root permits indexing without persistent caching, so documents are re-embedded on every startup. cache_key does not override missing metadata. Refer to Embedding SDK identity and RAG cache contract for the identity contract and Deploying samples with NIM endpoints for native Riva speech and NIM configuration.