Deployment Mechanics

l8k deploy applies a generated bundle in dependency order and waits for controller reconciliation.

Before applying

Confirm the intended kubeconfig context, operator ownership, and the reviewed bundle. A generated bundle is tied to its effective configuration; keep deployment/.l8k/resolved-config.yaml with the manifests. If another system owns Helm, configure networkOperator.skipHelmChart: true before generation. If another system applies custom resources, use its artifact handoff rather than this command.

Preview the same directory that will be applied:

l8k deploy \
  --deployment-files ./deployment \
  --kubeconfig "$KUBECONFIG" \
  --dry-run

Dry run asks the API server and Helm to validate their proposed operations without persisting them. It reports preflight effects but cannot establish reconciliation or data-plane readiness. Review every conflict and intended resource identity before using the real command.

Apply the reviewed bundle

l8k deploy \
  --deployment-files ./deployment \
  --kubeconfig "$KUBECONFIG"

If <deployment-files>/network-operator/ exists, Launch Kit uses it automatically. Otherwise it reads YAML files directly from <deployment-files>.

Launch Kit loads and checks the complete flat artifact directory before Helm installation, preflight remediation, or resource apply. Invalid YAML, missing resource identity fields, and duplicate declared resources stop deployment with the file and document number in the error. Only values.yaml is treated as Helm values; rename values.yml or case variants to values.yaml. The expected preflight inventory and applied objects come from that same snapshot.

Secondary-network ownership guard

Before Helm, preflight remediation, or apply, an affected NicClusterPolicy bundle requires a live read of the policy. Removing an existing Multus or CNI plugin section is blocked because the operator can delete its managed children. If a rendered section requests installation, recognized installers are inspected across namespaces; foreign or ambiguous ownership, or failed inspection, blocks deployment. --overwrite-existing, external Helm, and dry-run cannot bypass this guard. Ownership is checked again immediately before every NCP apply, including a polling retry after the policy disappears. Retries use the retained desired artifact rather than component intent returned by the API server; installer health still uses the operation's configured workers. With overwrite enabled, the guard also inspects differently named NCPs that stray remediation would delete, even when their installer DaemonSets are missing. An NCP with a managed Multus or reference-plugin section requires a separately controlled migration, rather than automatic deletion. Ordinary NCP strays without either managed section remain eligible for removal. The exact policy deletion candidates are read again before any remediation delete; UID/resource-version preconditions reject a changed or replaced policy. The API does not provide a transaction excluding all concurrent platform changes.

For omitted external components, installer health inspection is optional. No recognized API installer or a denied health read produces an unverified warning, since platform binaries may be host-installed. A denied live-NCP read still blocks deployment. Recognized installers excluding required workers or still reconciling stop deployment until repaired/ready. Ready installer pods and current DaemonSet status establish API-level coverage only, not host binary/configuration correctness. Without configured worker names, coverage remains unverified. Non-overwrite bundles without an NCP add no networking-inspection reads. Overwrite adds mandatory NCP candidate-list reads only when the existing stray check has an actionable operator namespace; denied inspection blocks before Helm.

See component policy for automatic and explicit ownership. The guard does not change the documented broad stray-resource deletion scope below; review the complete inventory before authorizing overwrite.

Helm Install Or Upgrade

When the bundle contains values.yaml and the selected release supplies a Helm repository URL, Launch Kit uses the Helm Go SDK to manage the network-operator release:

• No release: install the selected chart in networkOperator.namespace.
• Matching chart and values: no-op.
• Different chart or values: fail and show the conflict.
• Different chart or values with --overwrite-existing: run Helm upgrade.
• Release stuck in a pending Helm state: fail with rollback/uninstall guidance.

The chart is downloaded to a temporary directory. Launch Kit does not add or modify a local Helm repository.

If Helm metadata is absent, phase 0 is skipped so environments that manage the chart out of band can still apply the generated CRs.

Set networkOperator.skipHelmChart: true or pass --skip-network-operator-helm to disable Helm management explicitly even if a bundle still contains values.yaml. Launch Kit skips phase 0 and the Helm chart-version/values preflight checks, but retains component-version and stray resource preflight checks plus every manifest apply/reconciliation phase.

Preflight

Before applying custom resources, Launch Kit compares the bundle with the cluster:

Check	Detects
Helm chart version	Installed chart differs from the selected release.
Helm values	Installed user values differ from generated values.yaml.
Component versions	Live NicClusterPolicy component versions differ from the release catalog.
Stray resources	Instances of the enumerated networking CR kinds exist in the checked scope but are absent from the generated bundle, regardless of who created them. Recognized Spectrum-X controller children are excluded.

The two Helm rows are reported as skipped when Network Operator Helm management is disabled. The remaining rows still gate deployment.

Without --overwrite-existing, any mismatch stops deployment and all detected drift is reported together.

With --overwrite-existing, Launch Kit authorizes the Helm upgrade, deletes every reported stray CR, and lets server-side apply converge policy fields. Stray detection does not require an l8k ownership annotation. Manually created resources and resources from another generated cohort can be deleted.

Stray Resource Deletion Boundary

Scope	Kinds checked
Cluster-wide	NicClusterPolicy, NicNodePolicy, NicConfigurationTemplate, NicInterfaceNameTemplate
Resolved Network Operator namespace	SriovNetworkNodePolicy, SriovNetwork, SriovIBNetwork, SriovNetworkPoolConfig, OVSNetwork, IPPool, CIDRPool, MacvlanNetwork, IPoIBNetwork, HostDeviceNetwork, SpectrumXRailPoolConfig (v1alpha1/v1alpha2)

The expected bundle identities are retained. NicDevice, SriovNetworkNodeState, and SriovOperatorConfig are outside this check. Spectrum-X children have the exception described below. This boundary differs from clean, which performs broader removal.

Review the complete conflict list before approving overwrite, including cluster-scoped resources. A bundle generated with --groups or --gpu-type does not create an independent ownership boundary: resources from omitted groups may be classified as strays. Use a combined intended inventory for convergence, or apply an explicitly reviewed subset through a separately managed deployment process without l8k stray remediation.

The Spectrum-X operator labels the SR-IOV pool configs, node policies, and OVS networks it derives from SpectrumXRailPoolConfig with spectrumx.nvidia.com/owner-name. Launch Kit leaves these controller-owned objects out of stray detection and remediation. This makes a restarted Spectrum-X deployment idempotent after the controller has created its child resources, while unlabelled resources of the same kinds remain protected by the normal conflict check.

Apply Order

After Helm and preflight, deployment follows this order:

1. Apply NicClusterPolicy and wait for a terminal state.
2. Apply each NicNodePolicy and wait for each policy.
3. For RA2.4/RA3.1 NIC templates, verify the live NCO CRD's served version includes the platformType string field. This runs after policy bootstrap installs the CRD and before applying any remaining manifests. Missing APIs or failed reads stop deployment, including dry-run, external Helm and overwrite modes.
4. Apply every remaining manifest without serial waits so controllers reconcile concurrently.
5. Poll each applied additional resource until it is READY or ERROR.

The RA2.4 capability check requires read access to the cluster-scoped nicconfigurationtemplates.configuration.net.nvidia.com CRD. Dry-run cannot install a missing CRD through policy reconciliation. A failed capability check can follow Helm changes, ordinary stray remediation and policy bootstrap; those effects are retained. See RA2.4 prerequisites.

Launch Kit gates changed resources on controller observation, avoiding a false success from stale status left by an earlier generation.

IPPool application follows the NicClusterPolicy readiness gate when the bundle contains that policy. On Kubernetes this is not a separate wait for the ippools.nv-ipam.nvidia.com CRD to become established: an unavailable API still fails the apply. A bundle without NicClusterPolicy relies on prior operator bootstrap, and dry-run cannot create missing APIs. External Helm installers must stage dependent IPPools after NV-IPAM bootstrap themselves; filename order alone does not establish readiness. Validation reports missing APIs and does not install them.

Example workload manifests are not part of deployment. Files with example in the name are reserved for l8k validate.

Hardware target scope

Before Helm, stray remediation or manifest writes, Launch Kit checks the desired selectors of SR-IOV node policies and NV-IPAM IPPools against the worker inventory referenced by their source-group annotations. It reads Nodes and requires exact membership, rejecting missing workers, extra CPU/DPU nodes and empty selectors. It repeats the check before affected resource apply/reapply, and checks live selectors during reconciliation. --overwrite-existing, external Helm and server dry-run do not bypass the check. Node-label changes are not transactional; recheck the resulting deployment after concurrent cluster changes.

During reconciliation polling, temporary API timeouts, throttling and connection failures retry within the existing deployment timeout. Scope remains unverified and readiness is not evaluated until a fresh inspection succeeds. Denied reads, invalid source metadata and selector mismatches still stop deployment; failures of a pre-write inspection still prevent the write.

Explicit user config takes precedence over the resolved sidecar. Annotated resources require all source IDs to resolve. [""] explicitly references an unnamed source only when it is the sole configured source; its worker inventory is checked normally and its configuration remains required. Mixed populated/empty worker lists, duplicate source IDs or worker memberships, and ambiguous legacy multi-source inventory are errors. Regenerate legacy manifests to add source metadata; only a single configured source has an unambiguous legacy association.

Preset/manual resources whose sources all omit worker inventory, and unannotated artifact-only workflows, retain selector-only operation with a visible target scope unverified diagnostic. They do not establish inventory membership. This warning does not weaken PF or controller readiness failures. An annotated resource with missing configuration fails rather than falling back to this mode.

Terminal State

The shared resource-state registry classifies generated objects as:

State	Meaning
READY	Controller reconciliation and kind-specific checks passed.
IN-PROGRESS	Controller has not reached the requested state.
ERROR	Controller or a kind-specific cross-check reported failure.
MISSING	Object does not exist; used by validation rather than immediately after apply.

Kind-specific checks include per-component Network Operator state, SR-IOV per-node sync, matched PF counts, NIC configuration templates, IP pools, and Spectrum-X rail configuration.

For NicConfigurationTemplate and NicFirmwareTemplate, Launch Kit first waits for the operator to publish matched device names in status.nicDevices and for that name set to reflect the current nodeSelector, NIC type, PCI-address, serial-number, and part-number selectors. It then evaluates only those NicDevice objects and waits for the corresponding spec.configuration or spec.firmware field to reflect the current template payload. A successful device condition is accepted only after its observedGeneration catches up with the NicDevice generation. A configuration template checks FirmwareUpdateInProgress only when the matched device carries spec.firmware; without a deployed firmware template, a stale firmware condition from an older device generation does not block configuration reconciliation. Other discovered NICs do not block on configuration or firmware state. Changed templates are also observation-gated before this status is accepted, so status left by an earlier generation cannot produce a false success.

For NicInterfaceNameTemplate, InterfaceNameMismatch is retryable because the NIC configuration daemon can publish it while newly-written udev rules are still taking effect. Launch Kit starts a five-minute retry window when it first observes that mismatch. Initial device discovery and other ordinary in-progress states remain unbounded. Since additional-resource verification is ordered, the template gates later checks during this window. A persistent mismatch fails deployment after the local timeout and retains the per-node and per-port mismatch details.

Timeout

The default deploy budget is unbounded because SR-IOV and driver reconciliation can exceed a small fixed timeout on large clusters. NicInterfaceNameTemplate is the only bounded exception: after the first InterfaceNameMismatch, its retry window is five minutes. A shorter deploy-wide timeout takes precedence.

Bound the entire Helm, apply, and reconciliation operation to a maintenance window:

l8k deploy \
  --deployment-files ./deployment \
  --deploy-timeout 90m

Other manifests have no independent per-manifest deadline.

Server-Side Dry Run

Use the preview before application. Dry run sends resources through Kubernetes server-side validation without persisting them, runs Helm in dry-run mode, reports preflight effects, and skips reconciliation polling.

Manual Inspection

kubectl get nicclusterpolicy -o yaml
kubectl get nicnodepolicy -o yaml
kubectl get pods -n nvidia-network-operator -o wide

Finish every applied deployment with Validation. Review the report's readiness and connectivity coverage using the acceptance criteria; exit success alone does not guarantee complete coverage.