AI Factory Operations Agent Helm Deployment#

AI Factory Operations Agent is installed with Helm. This guide documents the shared upgrade, LLM, module, UI, and headless workflows.

Install AI Factory Operations Agent#

For a bare Kubernetes cluster, follow the complete Kind installation guide.

For an NVIDIA Mission Control managed cluster, follow the complete NMC installation guide.

Optionally, use the installation builder to select modules and generate your Helm command.

Custom Installation#

New installations use aifoa for both the Helm release and namespace. For existing installations, retain their current release name and namespace in all commands; changing either targets a different installation and does not migrate its data. Custom names remain supported.

1. Create The Namespace And Registry Access#

Ask your NVIDIA contact to invite the email address you will use with NGC to the AI Factory Operations Agent afoa-release team in the private registry. Accept the invitation while signed in with that account; use a private browser window if another NGC account is already signed in. Then create an NGC personal API key from NGC Setup > API Keys with the Private Registry service enabled.

MOSAIC_CHART=oci://nvcr.io/0948643769302270/afoa-release/mosaic-stack
# Remove --devel for the latest stable release, or replace it with --version 0.0.1 to pin that release.

read -rsp 'NGC API key: ' NGC_API_KEY; echo
printf '%s' "$NGC_API_KEY" | helm registry login nvcr.io \
  --username '$oauthtoken' \
  --password-stdin

kubectl create namespace aifoa --dry-run=client -o yaml | kubectl apply -f -
kubectl -n aifoa create secret docker-registry nvcr-image-pull-secret \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="$NGC_API_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

unset NGC_API_KEY

Persistent storage must be available through the cluster’s default StorageClass. The chart uses that default for the agent workspace and UI audit log; set openclaw.pvc.storageClassName or mosaicUi.auditPvc.storageClassName only when selecting another class.

2. Choose An LLM#

Choose one inference path below. Both paths install the same agent stack.

  • External inference: use an existing OpenAI-compatible endpoint, whether hosted by a provider or on your own infrastructure.

  • On-prem inference: deploy a model with vLLM through this chart on your cluster GPUs.

External Inference#

To install AI Factory Operations Agent with an external OpenAI-compatible LLM, set the provider URL and model, enter its API key, and run:

export EXTERNAL_LLM_BASE_URL='https://integrate.api.nvidia.com/v1'
export EXTERNAL_LLM_MODEL='nvidia/nemotron-3.5-super-vl-120b-a12b'
read -rsp 'External LLM API key: ' EXTERNAL_LLM_API_KEY; echo

kubectl -n aifoa create secret generic mosaic-external-llm \
  --from-literal=apiKey="$EXTERNAL_LLM_API_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

helm upgrade --install aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --set 'global.imagePullSecrets[0].name=nvcr-image-pull-secret' \
  --set llm.mode=external \
  --set-string llm.external.baseUrl="$EXTERNAL_LLM_BASE_URL" \
  --set-string llm.external.model="$EXTERNAL_LLM_MODEL" \
  --set llm.external.existingSecret=mosaic-external-llm \
  --set modules.ui.enabled=true \
  --set modules.execution.enabled=true \
  --set modules.kubernetes.enabled=false \
  --set modules.observability.enabled=false \
  --set modules.grafana.enabled=false \
  --set modules.bcm.enabled=false \
  --set modules.slurm.enabled=false \
  --set modules.diagnostics.enabled=false \
  --set modules.research.enabled=false \
  --set modules.terminal.enabled=false \
  --reset-values \
  --atomic \
  --wait \
  --timeout 12m

unset EXTERNAL_LLM_API_KEY

The NVIDIA API Catalog example uses the public API base URL above. build.nvidia.com hosts the catalog; it is not the inference API base URL. Hosted inference requires your explicitly approved NVIDIA API key. For OpenAI, use EXTERNAL_LLM_BASE_URL=https://api.openai.com/v1 and a model available to the account. Do not put provider keys in committed values files.

AI Factory Operations Agent requests reasoning_effort: none from external endpoints. For an independently managed vLLM server, also set its server default:

vllm serve MODEL --default-chat-template-kwargs '{"enable_thinking":false}'

After installing with your external endpoint, skip the on-prem inference instructions and continue to 3. Add Extensions.

On-Prem Inference#

This path deploys the inference server on your cluster. Choose one model profile below. If you already have an inference endpoint, use External inference instead.

To use the bundled on-prem inference profiles, the GPU nodes must belong to the same Kubernetes cluster as the agent’s Helm release. For NMC deployments with separate admin and GPU workload clusters, use the standalone inference example and connect the agent with external inference settings.

For chart-managed vLLM profiles, unpack the published chart once:

MOSAIC_CHART_WORKDIR=$(mktemp -d)
helm pull "$MOSAIC_CHART" \
  --devel \
  --untar \
  --untardir "$MOSAIC_CHART_WORKDIR"
MOSAIC_CHART_PATH="$MOSAIC_CHART_WORKDIR/mosaic-stack"

The chart defaults to Nemotron 3.5 Super VL NVFP4 on one compatible GPU, with 64 GiB of requested host memory and a 500 GiB model cache. GPU count alone does not establish compatibility: verify the GPU memory, architecture, driver, and pinned runtime against the model requirements.

Nemotron 3.5 Super VL#

To install AI Factory Operations Agent with Nemotron 3.5 Super VL running on 1 GPU in vLLM, run:

helm upgrade --install aifoa "$MOSAIC_CHART_PATH" \
  -n aifoa \
  -f "$MOSAIC_CHART_PATH/profiles/vllm-super-1gpu.yaml" \
  --set 'global.imagePullSecrets[0].name=nvcr-image-pull-secret' \
  --set modules.ui.enabled=true \
  --set modules.execution.enabled=true \
  --set modules.kubernetes.enabled=false \
  --set modules.observability.enabled=false \
  --set modules.grafana.enabled=false \
  --set modules.bcm.enabled=false \
  --set modules.slurm.enabled=false \
  --set modules.diagnostics.enabled=false \
  --set modules.research.enabled=false \
  --set modules.terminal.enabled=false \
  --reset-values \
  --atomic \
  --wait \
  --timeout 30m

The Super profile pins the Nemotron 3.5 Super VL checkpoint and compatible vLLM image. It was validated on one B200 for text generation and automatic tool calling with 262,144-token context and 16 maximum concurrent sequences. Image/video input and production concurrency were not tested; the agent retains its existing text-input configuration.

When upgrading from Nemotron 3 Super, preserve existing module and site values and the model-cache size, then apply profiles/vllm-super-1gpu.yaml last. The served model name changes to nemotron-3.5-super-vl; update external clients that used nemotron-super-120b. Restarting inference interrupts requests until the new checkpoint is ready. For shared endpoints, validate a separate inference release before switching clients.

Nemotron Ultra#

To install AI Factory Operations Agent with Nemotron Ultra running on four Blackwell GPUs on one node, run:

The Ultra model card specifies four B200/GB200/B300/GB300 GPUs or eight H100 GPUs as the minimum. For H100, override both llm.vllm.gpuCount=8 and llm.vllm.tensorParallelSize=8 on one node. Set llm.vllm.nodeSelector to keep inference on compatible hardware in mixed clusters. The distributed Ray option and vllm-ultra-16gpu.yaml profile have been removed; replace old values instead of reusing them during upgrades.

When upgrading an existing release, preserve its llm.vllm.modelCache.size override. Kubernetes does not allow changing the StatefulSet volume claim template in place, and the smaller Super default does not shrink an existing cache.

helm upgrade --install aifoa "$MOSAIC_CHART_PATH" \
  -n aifoa \
  -f "$MOSAIC_CHART_PATH/profiles/vllm-ultra-4gpu.yaml" \
  --set 'global.imagePullSecrets[0].name=nvcr-image-pull-secret' \
  --set modules.ui.enabled=true \
  --set modules.execution.enabled=true \
  --set modules.kubernetes.enabled=false \
  --set modules.observability.enabled=false \
  --set modules.grafana.enabled=false \
  --set modules.bcm.enabled=false \
  --set modules.slurm.enabled=false \
  --set modules.diagnostics.enabled=false \
  --set modules.research.enabled=false \
  --set modules.terminal.enabled=false \
  --reset-values \
  --atomic \
  --wait \
  --timeout 30m

Configure additional model options#

The chart configures the primary provider as llm from the selected inference path above. To make more models available to OpenClaw, add provider definitions under openclaw.additionalModelProviders. Each provider specifies an endpoint and a list of model IDs. Models on the same endpoint share one provider; use separate providers for different endpoints or credentials. The provider ID llm is reserved for the primary provider.

Create secondary-model-credentials in the release namespace through your normal Secret provisioning process, and store the API credential there. Keep the credential out of Helm values. The example below contains only the Secret name, the name of its credential field, and an environment-variable reference.

Save these settings in an additional values file:

openclaw:
  extraEnv:
    - name: SECONDARY_MODEL_API_KEY
      valueFrom:
        secretKeyRef:
          name: secondary-model-credentials
          key: apiKey # Name of the field inside the existing Secret.
  additionalModelProviders:
    secondary:
      baseUrl: https://inference.example.com/v1
      apiKey: ${SECONDARY_MODEL_API_KEY} # Loaded from the Secret at runtime.
      api: openai-completions
      models:
        - id: secondary-model
          name: Secondary model
          reasoning: false
          input: [text]
          contextWindow: 131072
          maxTokens: 8192

extraEnv loads the Secret field named apiKey into the gateway environment as SECONDARY_MODEL_API_KEY. OpenClaw resolves ${SECONDARY_MODEL_API_KEY} when it starts. For a private cluster endpoint, set request.allowPrivateNetwork: true on that provider. Each configured endpoint must already serve the listed model IDs; the chart does not deploy additional inference servers for these definitions.

Include the values file when installing or upgrading the release, along with your existing site values. Provider definitions are seeded into OpenClaw’s configuration on startup. Adding or changing endpoints, models, or Secret references requires a Helm upgrade. These settings preserve the configured primary model and existing provider credentials.

In frontend builds with a conversation model selector, these definitions become options such as secondary/secondary-model. Selecting an already configured model stores a per-conversation override for subsequent messages without a Helm upgrade. Other conversations keep their selections. The Helm fields supply configuration; they do not add the frontend selector themselves.

3. Add Extensions#

The minimal installation includes the UI, headless interfaces, OpenClaw, OpenShell, and key-free web search. Configure the modules needed for your cluster using the Extensions instructions, then continue to Usage.

Usage#

After installing with either inference path, open the UI by running:

kubectl -n aifoa port-forward svc/mosaic-ui 3000:3000

By default, the chart creates the mosaic-ui-auth Secret and generates both the UI password and an internal machine token. Leave mosaicUi.auth.password and mosaicUi.auth.machineToken empty; users do not need to provide either value. The generated credentials are preserved across upgrades.

Retrieve the browser login with:

kubectl -n aifoa get secret mosaic-ui-auth -o jsonpath='{.data.username}' | base64 --decode; echo
kubectl -n aifoa get secret mosaic-ui-auth -o jsonpath='{.data.password}' | base64 --decode; echo

username and password are browser credentials. machineToken is used automatically by OpenClaw when it calls protected AI Factory Operations Agent UI APIs; it is never entered in the browser.

mosaicUi.auth.existingSecret is only for installations that manage credentials outside this chart. When it is set, the chart does not create or modify the Secret, so the externally managed Secret must provide username, password, and machineToken.

Open http://localhost:3000 and sign in with the username and password above.

Extensions#

All module changes below update an existing AI Factory Operations Agent release and preserve its current site configuration.

Observability#

Connect to an existing Prometheus or VictoriaMetrics service with observability.prometheusUrl. Choose the query base URL for your backend:

Backend

Example base URL

Prometheus

http://kube-prometheus-stack-prometheus.prometheus.svc.cluster.local:9090

Single-node VictoriaMetrics

http://victoriametrics.monitoring.svc.cluster.local:8428

VictoriaMetrics cluster

http://vmselect.monitoring.svc.cluster.local:8481/select/0/prometheus

Replace 0 with your tenant ID for a VictoriaMetrics cluster. Include any proxy prefix and omit /api/v1. Set PROMETHEUS_URL to your endpoint:

export PROMETHEUS_URL='http://kube-prometheus-stack-prometheus.prometheus.svc.cluster.local:9090'
helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.observability.enabled=true \
  --set-string observability.prometheusUrl="$PROMETHEUS_URL" \
  --wait \
  --timeout 12m

Read-only Loki logs#

To connect Mosaic to an existing Loki service, run:

export LOKI_URL='https://logs.example.com'
helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.observability.enabled=true \
  --set-string observability.loki.url="$LOKI_URL" \
  --wait \
  --timeout 12m

Mosaic queries existing logs; it does not install Loki or a collector. For endpoints requiring authentication or a tenant ID, see Loki connection options. Verify access by asking Mosaic to show 10 log entries from the past 15 minutes.

Alertmanager#

AI Factory Operations Agent defaults to the Alertmanager service installed by NMC at http://kube-prometheus-stack-alertmanager.prometheus.svc.cluster.local:9093. To use another Alertmanager endpoint, run:

export ALERTMANAGER_URL='https://alerts.example.com/alertmanager'
helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set-string observability.alertmanagerUrl="$ALERTMANAGER_URL" \
  --wait \
  --timeout 12m

alertmanagerUrl is the Alertmanager API base URL. AI Factory Operations Agent reads active alerts from its /api/v2/alerts endpoint.

Grafana#

To connect AI Factory Operations Agent to an existing Grafana service, run:

export GRAFANA_URL='http://kube-prometheus-stack-grafana.prometheus.svc.cluster.local/grafana'
export GRAFANA_DATASOURCE_UID='prometheus'
export GRAFANA_DATASOURCE_TYPE='prometheus'
export GRAFANA_USERNAME='admin'
read -rsp 'Grafana password: ' GRAFANA_PASSWORD; echo

kubectl -n aifoa create secret generic mosaic-grafana-auth \
  --from-literal=username="$GRAFANA_USERNAME" \
  --from-literal=password="$GRAFANA_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.grafana.enabled=true \
  --set-string observability.grafanaUrl="$GRAFANA_URL" \
  --set-string observability.grafanaDatasourceUid="$GRAFANA_DATASOURCE_UID" \
  --set-string observability.grafanaDatasourceType="$GRAFANA_DATASOURCE_TYPE" \
  --set observability.grafanaAuth.existingSecret=mosaic-grafana-auth \
  --wait \
  --timeout 12m

unset GRAFANA_PASSWORD

grafanaUrl includes Grafana’s configured serving path. Use the service root for a root-served Grafana or /grafana for the NMC deployment.

For an installed native VictoriaMetrics datasource, set GRAFANA_DATASOURCE_UID to its UID and GRAFANA_DATASOURCE_TYPE=victoriametrics-metrics-datasource. For a Prometheus datasource, keep prometheus.

Terminal#

To enable the optional browser terminal service, run:

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.terminal.enabled=true \
  --wait \
  --timeout 12m

Access Modes#

Enable guarded write operations with the Edit module:

modules:
  edit:
    enabled: true
    hitl: true
    kubernetes:
      enabled: true

Every interactive conversation starts in View. Users can switch that conversation to Edit for approval-gated changes or Auto for validated changes without approval. Auto requires confirmation through a warning dialog. Other conversations remain in View, and automated alert and cluster-health sessions are always read-only. When the module is disabled, the access control is not shown.

Native Slack users remain read-only unless their verified Slack user ID is listed under openclaw.slack.editUserIds. Edit users approve their own mutations with Slack buttons. The broader openclaw.slack.allowedUserIds list controls who may use AI Factory Operations Agent through DMs, group DMs, and channel mentions.

Privileges are attached to separate tools and credentials. Read-only infrastructure tools and sandbox-local execution run automatically in View. OpenShell enforces local execution boundaries; the diagnostic command allowlist applies to remote SSH. Mutating Kubernetes, Base Command Manager, and SSH operations are blocked in View, approval-gated in Edit when hitl: true, and automatic in Auto. Kubernetes RBAC, request validation, configured SSH hosts, and audit logging apply in every mode.

Read the AI Factory Operations Agent security model before enabling Edit or Auto.

Managed MCP servers#

The chart can deploy private MCP services and register their cluster-local endpoints with OpenClaw. Each enabled server uses a digest-pinned image, ClusterIP Service, hardened pod security settings, and existing Kubernetes Secrets for credentials. An ingress policy limits the service to OpenClaw.

managedMcpServers:
  servers:
    example:
      enabled: true
      image:
        repository: registry.example.com/example-mcp
        digest: sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
      args: ["--read-only"]
      env:
        API_BASE_URL: https://api.example.com
      secretEnv:
        API_TOKEN:
          name: example-mcp-credentials
          key: token
      mcp:
        headers:
          X-Organization: operations
        secretHeaders:
          Authorization:
            name: example-mcp-caller
            key: token
            prefix: "Bearer "
      toolPolicy:
        defaultAccess: edit
        viewTools: [status, search]

The resulting endpoint is registered as example at http://mcp-example:8080/mcp. Use port, path, or transport to override those connection defaults. For a private CA, set tls.existingSecret; the Secret must contain ca.crt unless tls.key is overridden. Do not place credentials in env.

args are passed to the container entrypoint. mcp.headers use OpenClaw’s native HTTP MCP configuration. Put sensitive header values in mcp.secretHeaders; the chart injects the referenced Secret only into OpenClaw and substitutes it into the header at startup. Managed-server tools require Edit by default. Set toolPolicy.defaultAccess: view only when every tool exposed by the server is safe in View, or list individual safe tool names in viewTools. Tools from unconfigured servers remain fail-closed.

Externally managed MCP servers#

Installers can register an existing MCP endpoint without deploying it or changing the UI image. Add the connection to the Helm values file and keep credentials in an existing Kubernetes Secret:

externalMcpServers:
  inventory:
    url: https://mcp.example.com/mcp
    transport: streamable-http
    secretHeaders:
      Authorization:
        name: inventory-credentials
        key: token
        prefix: "Bearer "
    toolPolicy:
      defaultAccess: edit
      viewTools: [lookup]

The chart injects the Secret value only into OpenClaw and references it in the MCP header. External tools require Edit by default. List verified read-only tools in viewTools, or set defaultAccess: view only when every tool is safe in View. URLs must use HTTPS, except for cluster-local service URLs under svc.cluster.local; credentials and query strings are not permitted in URLs. The chart does not create, proxy, or authenticate the external server itself.

Run:ai#

Run:ai is included as a disabled managed MCP module. Provision a dedicated Viewer application’s clientId and clientSecret as a Kubernetes Secret, then enable it in the installer’s tracked values file:

managedMcpServers:
  servers:
    runai:
      enabled: true
      env:
        RUNAI_BASE_URL: https://runai.example.com
      oauth:
        tokenUrl: https://runai.example.com/auth/realms/runai/protocol/openid-connect/token
        clientIdSecret:
          name: runai-credentials
        clientSecretSecret:
          name: runai-credentials

The chart’s sidecar renews a Viewer application’s OAuth token; the Run:ai server exposes only read tools. For a private CA, set managedMcpServers.servers.runai.tls.existingSecret to a Secret containing ca.crt.

Base Command Manager#

To enable the Base Command Manager extension, provide the Base Command Manager head host and SSH key:

export CLUSTER_MANAGER_HEAD_HOST='bcm-head.example.com'
export CLUSTER_MANAGER_SSH_KEY_PATH='/root/.ssh/id_ecdsa'
test -r "$CLUSTER_MANAGER_SSH_KEY_PATH"

kubectl -n aifoa create secret generic bcm-host-ssh-key \
  --from-file=id_ecdsa="$CLUSTER_MANAGER_SSH_KEY_PATH" \
  --dry-run=client -o yaml | kubectl apply -f -

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.bcm.enabled=true \
  --set bcmMcp.enabled=true \
  --set bcmMcp.mode=ssh-adapter \
  --set-string bcmMcp.headHost="$CLUSTER_MANAGER_HEAD_HOST" \
  --set bcmMcp.hostSshKeySecretName=bcm-host-ssh-key \
  --wait \
  --timeout 12m

The SSH identity must be allowed to create or update the dedicated read-only CMSH user during startup.

Read-only Redfish#

Redfish lets the agent read a node’s BMC when the node is down or unreachable in-band: power state, hardware health, temperatures, fans, power supplies, BMC event logs, and firmware inventory. The agent passes only a Base Command Manager hostname. The Base Command Manager adapter looks up the BMC address and credentials through CMSH, the same lookup the Hardware Agent uses, and never returns them to the model.

Prerequisites:

  • The Base Command Manager extension is configured in ssh-adapter mode.

  • The adapter’s SSH identity can read bmcsettings and BMC interfaces through CMSH on the head node.

  • The head node can reach the BMC network over HTTPS and has curl. Check one BMC from the head node:

    curl -sk https://<bmc-address>/redfish/v1 | head -c 300; echo

Enable it:

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.bcm.redfish.enabled=true \
  --wait \
  --timeout 12m

By default each request runs curl on the head node over the adapter’s existing SSH session (modules.bcm.redfish.viaHeadNode=true), because the head node is routed to the BMC network. The BMC credential reaches curl on standard input, so it does not appear in process listings. Set viaHeadNode=false only when the adapter pod itself can reach the BMCs. BMCs usually present self-signed certificates, so modules.bcm.redfish.verifyTls defaults to false. modules.bcm.redfish.timeoutSeconds (default 60) bounds each request; some DGX BMCs take about 30 seconds to return the expanded firmware inventory.

The adapter enforces the following allowlist before it looks up any credential. Only GET is used, and the only query parameters accepted are $top, $skip, and $expand=. limited to one level, which is refused on the service root:

Resource

Paths under /redfish/v1

Service root and collections

/, Systems, Chassis, Managers

Systems

Systems/{id} and its Processors, Memory, Storage, EthernetInterfaces, and LogServices/{id}/Entries

Chassis

Chassis/{id} and its Thermal, ThermalSubsystem, Power, PowerSubsystem, and Sensors

Managers

Managers/{id} and its EthernetInterfaces and LogServices/{id}/Entries

Firmware and tasks

UpdateService/FirmwareInventory, TaskService/Tasks

Everything else is refused, including any path containing Actions, and AccountService, SessionService, CertificateService, EventService, SecureBoot, Bios/Settings, and VirtualMedia. Paths with .., //, %, or other query parameters are also refused. The adapter also checks every resource embedded in a response, such as $expand results and continuation pages, against the same allowlist. It reduces any embedded resource outside it to a bare link, so a BMC that expands more than requested cannot return denied content. Fields named like passwords, tokens, or secrets are redacted from responses. Resets and other BMC actions are not available in any session mode.

Research#

To deploy the Research Agent and OpenSearch, set the embedding API key, active model, and corpus path, then run:

read -rsp 'NVIDIA embedding API key: ' NVIDIA_API_KEY; echo
export IRAOP_API_KEY="$(openssl rand -hex 32)"
export MOSAIC_CHAT_MODEL='nvidia/nemotron-3.5-super-vl-120b-a12b'
export IRA_CORPUS_PATH='/cm/shared/iraop-corpus'

kubectl -n aifoa create secret generic iraop-secrets \
  --from-literal=NVIDIA_API_KEY="$NVIDIA_API_KEY" \
  --from-literal=NVIDIA_CHAT_API_KEY=EMPTY \
  --from-literal=IRAOP_API_KEY="$IRAOP_API_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.research.enabled=true \
  --set modules.research.secrets.create=false \
  --set-string researchAgent.iraop.config.NVIDIA_CHAT_MODEL="$MOSAIC_CHAT_MODEL" \
  --set-string researchAgent.iraop.corpus.hostPath="$IRA_CORPUS_PATH" \
  --wait \
  --timeout 12m

unset NVIDIA_API_KEY IRAOP_API_KEY

Hardware Agent#

The Hardware Agent reuses the Base Command Manager head address and bcm-host-ssh-key configured above. The chart exposes the required Base Command Manager lookup inside the cluster and generates its internal credentials. It uses NVDebug evidence and generic analysis.

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.diagnostics.enabled=true \
  --wait \
  --timeout 12m

Upgrade An Existing Installation#

Retain the working site configuration with:

MOSAIC_CHART=oci://nvcr.io/0948643769302270/afoa-release/mosaic-stack
# Remove --devel for the latest stable release, or replace it with --version 0.0.1 to pin that release.
helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --atomic \
  --wait \
  --timeout 12m

If Helm is not already authenticated to NVCR, run the registry login command from custom installation step 1 first.

Headless Mode#

The AI Factory Operations Agent UI service also exposes a headless API:

export MOSAIC_URL=http://127.0.0.1:3000
export MOSAIC_UI_MACHINE_TOKEN="$(kubectl -n aifoa get secret mosaic-ui-auth \
  -o jsonpath='{.data.machineToken}' | base64 --decode)"
curl -sS "$MOSAIC_URL/api/headless/chat" \
  --header @<(printf 'Authorization: Bearer %s\n' "$MOSAIC_UI_MACHINE_TOKEN") \
  -H 'content-type: application/json' \
  -d '{"prompt":"summarize whether the cluster is healthy","sessionKey":"headless"}'

Invoke the packaged CLI inside the mosaic-ui pod with:

kubectl -n aifoa exec deploy/mosaic-ui -- \
  mosaic --session headless "summarize whether the cluster is healthy"

The AI Factory Operations Agent HTTP service is not itself an MCP endpoint. Install the stdio MCP bridge on the client machine (Node.js 18 or newer is required):

git clone https://github.com/NVIDIA/AI-Factory-Operations-Agent.git
cd AI-Factory-Operations-Agent
install -d ~/.local/bin
install -m 0755 utils/mosaic-mcp.mjs ~/.local/bin/mosaic-mcp

Point the bridge at any reachable AI Factory Operations Agent UI URL. For Claude Code:

claude mcp add mosaic \
  -e MOSAIC_URL=http://localhost:3000 \
  -- ~/.local/bin/mosaic-mcp

For Codex, add the following to your user ~/.codex/config.toml, replacing the command with the absolute installed path:

[mcp_servers.mosaic]
command = "/home/USER/.local/bin/mosaic-mcp"
env_vars = ["MOSAIC_URL", "MOSAIC_UI_MACHINE_TOKEN"]

Start the MCP client from the shell where MOSAIC_URL and MOSAIC_UI_MACHINE_TOKEN are exported.

Replace http://localhost:3000 with the deployed AI Factory Operations Agent URL when it is reachable directly. The bridge exposes ai_factory_operations_agent_chat, ai_factory_operations_agent_history, ai_factory_operations_agent_commands, and the AI Factory Operations Agent/OpenClaw tools enabled by the chart over stdio MCP. See docs/skills/ai-factory-operations-agent-headless/SKILL.md for the complete agent workflow.

Vanilla Slurm RCA#

The public Slurm workflow is evidence based. By default, the chart deploys a read-only Slurm evidence collector. The collector mounts the host filesystem read-only inside the collector pod, exposes bounded HTTP tools to OpenClaw, and keeps broad host filesystem access out of the LLM sandbox.

Default collector roots:

  • /var/log

  • /cm/shared

  • /slurm

  • /etc/slurm

  • /cm/shared/apps/slurm/etc

  • /run/log/journal

  • /var/log/journal

The Slurm skill first calls slurm_job_evidence, then falls back to read-only sacct/scontrol and any mounted evidence available in the sandbox. Missing paths are expected on many clusters; the skill continues with whatever evidence is present.

To use the vanilla collector with a site-specific evidence root, run:

export SLURM_EVIDENCE_ROOT='/shared/slurm'
helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.slurm.enabled=true \
  --set modules.slurm.backend=vanilla \
  --set-json "slurmEvidenceCollector.roots=[\"$SLURM_EVIDENCE_ROOT\"]" \
  --wait \
  --timeout 12m

The Slurm backend defaults to auto. When Base Command Manager and Slurm are both enabled, AI Factory Operations Agent uses Base Command Manager WLM’s read-only job metadata, stdout, and stderr interface and does not deploy the node-local collector. Without Base Command Manager, it uses the vanilla collector. Set modules.slurm.backend to bcm or vanilla only to require one backend explicitly.

After enabling the Base Command Manager extension, enable Slurm through it with:

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.slurm.enabled=true \
  --set modules.slurm.backend=auto \
  --wait \
  --timeout 12m

Kubernetes Access#

AI Factory Operations Agent exposes one read-only OpenClaw tool, run_kubectl. The tool invokes a pinned kubectl binary with an argument array, never a shell command. It rejects mutation, pod execution, port forwarding, Secret reads, impersonation, raw kubeconfig output, and credential or API endpoint overrides before launching kubectl. Kubernetes RBAC independently denies those operations.

The local cluster is registered by default using the openclaw ServiceAccount. Its token and kubeconfig are mounted only in the OpenClaw pod; OpenShell sandboxes receive neither Kubernetes credentials nor kubectl.

Enable read-only access to the local cluster with:

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.kubernetes.enabled=true \
  --wait \
  --timeout 12m

To register another cluster, first create a kubeconfig for a read-only identity on that cluster. Store it as a Secret in the AI Factory Operations Agent namespace:

kubectl -n aifoa create secret generic production-west-kubeconfig \
  --from-file=config=/path/to/read-only-kubeconfig

Register that Secret with AI Factory Operations Agent by running:

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set modules.kubernetes.enabled=true \
  --set-json 'kubernetes.clusters=[{"name":"production-west","kubeconfigSecretRef":{"name":"production-west-kubeconfig","key":"config"}}]' \
  --wait \
  --timeout 12m

The model selects only the registered name. It cannot provide a kubeconfig path, context, token, or server address. Removing the list entry on upgrade removes the corresponding credential mount. A missing Secret leaves the OpenClaw pod unready with the Kubernetes volume error from the kubelet.

OpenShell Dependency#

The published chart includes the pinned official OpenShell OCI dependency from ghcr.io/nvidia/openshell. It uses the official gateway, supervisor, and unprivileged base sandbox images. The pinned kubernetes-sigs/agent-sandbox prerequisite required by OpenShell’s Kubernetes driver is included as well:

  • CustomResourceDefinition resources are placed under chart crds/ so Helm installs them before templates.

  • The controller namespace, RBAC, service, and StatefulSet are rendered as normal templates when agentSandbox.install=true.

If the cluster already provides a compatible agent-sandbox installation, run:

helm upgrade aifoa "$MOSAIC_CHART" \
  --devel \
  -n aifoa \
  --reuse-values \
  --set agentSandbox.install=false \
  --wait \
  --timeout 12m

The packaged chart includes the tracked agent-sandbox CRD and controller templates.