xr-ai-models#

Overview#

The model that works for your prototype may change as you tune quality, hardware use, or deployment. Shared model interfaces let your application call the capabilities it needs while configuration selects the supported model adapter and endpoint.

Model interfaces#

The protocols cover LLM, VLM, STT, TTS, embedding, decision, and OCR services. Workers construct clients through the factories in xr-ai-models; model HTTP and vendor SDK calls stay inside the adapters. Refer to API Reference for exact classes, methods, fields, and defaults. Refer to AI inference servers for server operation.

Construct a model client#

from xr_ai_models import ChatMessage, load_models_config, make_llm

config = load_models_config("yaml/models.json")
async with make_llm(config, "llm") as llm:
    response = await llm.chat(
        [ChatMessage(role="user", content="hello")],
        max_tokens=128,
        enable_thinking=True,
    )
    print(response.content, response.reasoning)

Choice-only decision clients use make_decision and the SystemOne adapter:

from xr_ai_models import DecisionQuestion, load_models_config, make_decision

config = load_models_config("yaml/models.json")
async with make_decision(config, "triage") as decisions:
    result = await decisions.decide(
        "The participant asked to start the timer.",
        {
            "approval": DecisionQuestion(
                instructions="Did the participant explicitly approve the action?",
                criteria={"yes": "An explicit approval", "no": "No explicit approval"},
            )
        },
    )
    print(result.answers["approval"].choice)

The profile entry uses category: decision, kind: systemone, a model name, and an endpoint. A request the service rejects as more input than its model accepts (HTTP 413) raises DecisionInputTooLargeError. The limit belongs to the service, so a caller handles this by retrying with less input. The standalone Clef-Flash guide documents the local service and its pinned model release.

Model entry names are application-defined. Applications can configure multiple LLMs under distinct names with make_llm(config, name), using the same endpoint or different endpoints.

A client profile names logical roles and declares adapters and endpoints:

{
  "models": {
    "llm": {
      "category": "llm",
      "adapter": {"preset": "nemotron_omni"},
      "endpoint": {
        "base_url": "http://localhost:8108",
        "timeout": 60.0
      }
    }
  }
}

adapter owns the model name, wire quirks, capabilities, default request extras, and reasoning-field normalization.
endpoint owns connectivity, timeouts, and environment-variable credentials. Optional health settings control explicit SDK health() calls; they do not cause consumer workers to poll endpoints at startup.
Optional deployment metadata selects the processes a shared model-server orchestrator manages. Consumer profiles omit it: their endpoints are operated outside the sample, whether locally or remotely. Existing explicit reused and external entries remain supported for compatibility.

Annotated consumer configuration#

The package reference below shows local endpoint examples, not a default launch stack. Copy models.yaml.sample to your application’s models.yaml, keep only the roles it uses, and pass that file explicitly to load_models_config. Loading it does not connect to or launch a model server. The launcher requires JSON deployment profiles; this worker-side YAML reference does not replace a launcher’s models.json. Refer to Deploying samples with NIM endpoints for the separate annotated NIM profile and its transport-specific settings.

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Copy-only model consumer reference: copy to models.yaml, keeping only the
# roles your application uses, then select that working file explicitly.
# load_models_config accepts YAML or JSON; it does not auto-discover this file,
# start services, expand environment variables, or resolve URLs as file paths.
# These local endpoints are examples, not loader defaults or a launch profile.
# A sample's own models.json remains authoritative for that sample.
# YAML is supported by the worker-side SDK; launcher deployment profiles must
# remain JSON. Do not replace a launcher's models.json with this file.
# Speaker-conditioned ASR is configured separately in voice_gate.yaml; the stt
# role below is ordinary batch STT, needed only when the application uses it.

models:
  # Logical role names are application-defined; use the same names in make_*.
  llm:
    adapter:
      # Presets supply category, model identifier, capabilities, and wire quirks.
      # Explicit settings override the preset. Do not also set adapter.kind.
      preset: nemotron_omni
      # Optional overrides (must match the serving model's real contract):
      # model_name: llm
      # reasoning_field: reasoning_content
      # capabilities: {streaming: true, tool_calls: true}
      # thinking: {default: true}    # Reasoning when a call does not choose.
    endpoint:
      base_url: http://127.0.0.1:8108 # HTTP origin, without /v1 routes.
      timeout: 60.0                  # Positive seconds; speech defaults to 30.
      readiness: health             # Explicit health() calls only, not polling.
      health_path: /health          # HTTP health route; NIM: /v1/health/ready.
      # Name an environment variable, never put the actual token in this file.
      # api_key_env: MODEL_ENDPOINT_API_KEY
    # Omission means external ownership: consumers never stop this endpoint.
    # Managed JSON launch profiles additionally name the service and server-side
    # credentials (environment-variable names, not endpoint bearer tokens):
    # deployment:
    #   ownership: managed
    #   service: nemotron_omni_llm_server
    #   credentials: []
  vlm:
    adapter:
      preset: cosmos3_nano_reasoner
    endpoint:
      base_url: http://127.0.0.1:8100
  stt:
    adapter:
      preset: parakeet_stt
    endpoint:
      base_url: http://127.0.0.1:8103
  tts:
    adapter:
      preset: pocket_tts
    endpoint:
      base_url: http://127.0.0.1:8105
  embedding:
    adapter:
      preset: nemotron_embedding
    endpoint:
      base_url: http://127.0.0.1:8109
  # A decision role without a preset declares category and the SystemOne adapter.
  decision:
    category: decision
    adapter:
      kind: systemone
      model_name: Cloudflare/clef-flash
    endpoint:
      base_url: http://127.0.0.1:8120

# For Riva speech, including TLS, language, voice, sample rate, function IDs,
# endpoint credentials, and independently deployed NIM examples, refer to
# docs/source/reference/agent-sdk-models.md and deployment/nim/models.yaml.

Workers may load JSON or YAML. For compatibility, the loader accepts a direct role mapping, legacy flat entries, health_check: true or health_check: false, and kind: preset:<name>. The public role-spec classes also retain their legacy flat constructors and read-only flat properties. Profiles shared with the launcher must use the wrapped nested JSON form with adapter, endpoint, and deployment objects. Only the worker-side loader accepts omitted deployment metadata. Launcher credentials are explicit: endpoint credentials use api_key_env, while credentials needed by a managed service itself use deployment.credentials.

Request failures#

Applications handle errors from model requests. For worker startup behavior, refer to Consumer model readiness.

Built-in adapters#

Preset	Target	Important behavior
`cosmos3_nano_reasoner`	Cosmos3 Nano VLM	Image; video requires `max_videos_per_prompt >= 1`; no reasoning switch
`cosmos_vlm`	Cosmos-Reason1 compatibility	Image; video requires `max_videos_per_prompt >= 1`; thinking disabled by default
`nemotron3_nano`	Nemotron 3 Nano LLM	Normalizes the `reasoning` field; thinking disabled unless requested
`nemotron_omni`	Nemotron Omni	Tool calls, image and video, `reasoning_content` normalization; thinking disabled unless requested
`nemotron_embedding`	Embedding server	OpenAI-compatible dense vectors
`nemotron_ocr`	Hugging Face OCR server or OCR NIM 2.0	Multilingual text, confidence, normalized polygons; `/v1/ocr`
`parakeet_stt`	STT server	OpenAI-compatible transcription
`pocket_tts`	Pocket TTS	OpenAI-compatible synthesis plus native PCM streaming
`magpie_tts`	Magpie TTS	OpenAI-compatible speech synthesis

The Cosmos adapter capability describes the supported request shape, but the server controls whether video input is enabled. Every checked-in vlm-server profile sets max_videos_per_prompt: 0 to avoid reserving unused activation memory. Set it to at least 1 and restart the persistent VLM server before sending a video request.

Reasoning control#

Callers request reasoning per call with enable_thinking. True or False always applies to that call; omitting it (None) uses the endpoint default. Code written this way runs unchanged when a deployment swaps models, because the adapter’s thinking setting decides how the switch reaches the model:

`control`	Wire behavior	Used by
`chat_template`	`chat_template_kwargs.enable_thinking`; `thinking_budget` as vLLM’s `thinking_token_budget`	`nemotron3_nano`, `nemotron_omni`, `cosmos_vlm`
`system_prompt`	A `detailed thinking on` or `off` line at the start of the system message	Profiles for Llama-Nemotron endpoints
`none`	Nothing is sent; the model has no switch or rejects template arguments	`cosmos3_nano_reasoner`
unset	`chat_template` when `default` is set; otherwise only an explicit `enable_thinking=True` sends the switch. A supplied budget is sent unless reasoning is explicitly off.	Profiles without a preset

The per-call thinking_budget is sent as thinking_token_budget for chat_template and unset controls unless reasoning is off. Budget enforcement requires support in the deployed model server, independently of support for the thinking switch. It has been verified on Nemotron Omni served by vLLM with --reasoning-parser nemotron_v3; NIM budget support has not been qualified. Other endpoints may reject or ignore the field. The SDK does not detect budget support or retry without it. Omitting thinking_budget (or passing None) leaves the thinking switch available without requesting a reasoning cap. Sample requests that explicitly pass a budget continue to send it; selecting a different endpoint profile does not independently disable that budget.

default is the endpoint’s reasoning state when a call does not choose one; null sends no switch and leaves the model’s own default. With system_prompt, a complete detailed thinking on or off line already in the system message stands for the default, and an explicit enable_thinking replaces it. A profile overrides either key on top of its preset:

adapter:
  preset: nemotron3_nano
  thinking:
    default: true

Set control: none for a hosted endpoint that rejects unknown request fields; such a model ignores enable_thinking. capabilities.reasoning reports only whether responses carry a separate reasoning field, not whether the switch is honored. A legacy default_extras.chat_template_kwargs.enable_thinking boolean is still accepted: with control: chat_template or unset it becomes thinking.default, overriding the preset’s, and a call can override it. With another control it is sent unchanged. A profile that sets both keys to different values is rejected.

ChatResponse.reasoning is the canonical post-normalization field. Model adapters absorb whether the provider calls it reasoning or reasoning_content. LLM and VLM calls accept controlled per-request headers for Relay lineage, but callers cannot replace the profile’s Authorization header.

Single-image ask_image() and stream() calls are wrappers over the ordered multi-image methods. All images are placed in one user message in caller order.

Hosted endpoints#

A hosted OpenAI-compatible endpoint changes only the profile:

{
  "models": {
    "vlm": {
      "category": "vlm",
      "adapter": {
        "kind": "openai_compat",
        "model_name": "nvidia/cosmos3-nano-reasoner"
      },
      "endpoint": {
        "base_url": "https://integrate.api.nvidia.com",
        "api_key_env": "NGC_API_KEY"
      }
    }
  }
}

For code that calls health() explicitly, readiness: none makes that call succeed without a request; health_path selects the HTTP route when probing is enabled. Tea making’s RAG service requires this configuration for hosted embedding endpoints; refer to RAG embedding health.

Deployment profiles#

Use the same factories and typed interfaces for local development and native NIM deployment. The example in deployment/nim/models.yaml selects native HTTP chat and embedding endpoints and Riva gRPC speech, without compatibility servers. The embedding factory honors existing request defaults, including asymmetric input_type batches. Refer to Deploying samples with NIM endpoints for deployment ownership, sample configuration paths, and validation commands.

Riva speech over gRPC#

Riva speech NIMs use kind: riva_grpc, not OpenAI /v1/audio. Install the riva extra; its nvidia-riva-client import is deferred until make_stt() or make_tts() selects that kind.

stt:
  kind: riva_grpc
  category: stt
  base_url: localhost:50051
  language: en-US

STT accepts 16-bit PCM WAV or raw int16 PCM with an explicit sample rate. TTS also accepts voice and sample_rate. A hosted NVCF endpoint uses TLS, api_key_env, and its function_id. If custom code explicitly calls health(), set health_check: false for a hosted endpoint without a channel-ready health surface. Recognition timeout and cancellation cancel the native RPC and join its pending client read; synthesis uses the same cleanup rule for streamed audio.

Tests#

The CPU-only tests/test_models_*.py modules exercise model wire formats with the tests/_stub_openai.StubOpenAI httpx mock transport. Run them from the repository root:

uv run --project tests pytest -q tests/test_models_*.py