Configuration File¶
cluster-config.yaml contains hardware inventory and deployment settings.
Discovery writes it; generation reads it without modifying it and fills omitted
values from the built-in defaults. Generation records the effective result in
<deployment-dir>/.l8k/resolved-config.yaml; deploy and validate use that
bundle metadata when no explicit --user-config is supplied.
The default release is 26.7. The 26.10 catalog entry selects
Network Operator v26.10.0-beta.1 when requested. A compatible Spectrum-X release
selected during fresh discovery takes precedence over that default; the saved
file contains the selected release and its matching catalog versions. An
explicit release in --user-config remains pinned unless a CLI release
override is supplied.
For generation, configuration source precedence is:
- Canonical defaults from the repository-root
cluster-config.yaml. - Resolved hardware defaults for missing profile fields and host driver ownership.
--config-dir/l8k-config.yamlor--user-configvalues.- Explicit CLI flags.
Presence is tracked before normalization. An explicit false, 0, empty
string, or empty list therefore overrides lower layers; YAML null means the
field is unset. A selected Network Operator release remains authoritative and
replaces the catalog-managed version and repository fields after all layers
are merged.
Fresh discovery treats the selected reference file as a user layer except for
its sample profile, which is discarded before hardware defaults are derived.
Explicit non-profile values such as docaDriver.enable: false are preserved.
Discovery with --user-config replaces clusterConfig, discoveredSecondaryNetwork,
and discoveredDocaHost, and clears any previously generated discoveredDocaHostScope.
It preserves every other loaded section, then applies explicit CLI flags. It does not fill
missing profile fields, recompute validation settings, or expand a release
selected only in YAML.
The refresh patches the source YAML, retaining explicit zero values, omitted
fields, and custom keys. YAML aliases and merge keys are expanded when saving
so a CLI override affects only its target field.
Standalone validation applies its validation CLI flags before checking the resolved configuration. A valid flag value can therefore replace an invalid value from the YAML file.
--config-dir/presets/ replaces the embedded preset catalog. It does not merge with it.
Invalid enum values are rejected when CLI inputs are bound. Cross-field requirements are checked after the effective configuration is resolved.
DPF discovery configuration¶
discover --target dpf uses a separate site configuration described in
DPF discovery. The Host configuration defaults above
do not apply to it. Select dpf.useCase: hbn-bgp or explicitly supply
--use-case hbn-bgp to evaluate that use case's prerequisites.
The release catalog declares the HBN-BGP default and supported release as
26.8.0; it also supplies that release's default HBN chart and image
coordinates. An explicit unsupported dpf.release is rejected during shared
configuration resolution. The profile supplies operator namespace
dpf-operator-system and networking/policy defaults. Override the namespace
with dpf.namespace and networking fields under dpf.hbn. Explicit
provisioning.hbn artifact values remain user-owned overrides and are
validated and recorded as supplied.
Omitted/null fields select defaults; explicit empty/zero values are validated.
General DPF inventory without a use case retains its 26.4.0 release default.
Switch endpoints, credentials, pinned SSH trust, and intended port bindings are
site inputs under discovery.switches; they are not inferred from profile defaults.
For BF4 provisioning, supply provisioning.software.osISO and
provisioning.software.pldm.<PSID>. Each artifact has url (HTTPS without
credentials, a query or a fragment) and sha256 (64 lowercase hex characters).
PLDM keys match the discovered PSIDs case-insensitively; duplicate or unused
entries are rejected when the full selected PSID set is known. These artifacts
have no public profile defaults. Missing inputs do not stop inventory collection,
but HBN readiness reports them as blockers. Supplying URLs and digests does not
prove that DPF can download the artifacts or that their bytes are compatible.
Bootstrap and HBN defaults¶
provisioning.bootstrap supplies the management-plane installation inputs.
Discovery reports missing inputs while continuing to collect hardware information:
| Field | Default or source | Site responsibility |
|---|---|---|
managementAPIIP, managementAPIPort |
Observed management API endpoint, when it contains a literal IP | Override when DPUs need a different reachable endpoint; a DNS name is not converted into an assumed IP |
tenant.vip |
Required | Reserve an unused IPv4 address for the tenant API, outside DHCP allocation |
tenant.interface |
Required | Identify the management interface on the selected control-plane nodes |
tenant.routerID |
126 |
Verify that this VRRP ID is unused on the selected control-plane nodes; override collisions |
tenant.allocation.vipReserved |
No default | Explicit site declaration that the VIP is reserved for this tenant; readiness requires true |
tenant.allocation.vipExcludedFromDHCP |
No default | Explicit site declaration that dynamic DHCP cannot allocate the VIP; readiness requires true |
tenant.allocation.routerIDUniqueOnSelectedNodes |
No default | Explicit site declaration that the selected VRRP ID is available on the selected control-plane nodes; readiness requires true |
tenant.name, tenant.namespace |
dpu-cplane-tenant1 |
Override when another tenant uses these names |
tenant.nodeSelector |
node-role.kubernetes.io/control-plane: "" |
Select the control-plane nodes that host the VIP |
tenant.maxNodes |
1000 |
Must accommodate all selected DPUs |
podCIDR |
10.244.0.0/14 |
Pod network for the DPU tenant cluster; verify separation from networks sharing its forwarding paths |
artifactPVC |
Omitted: DPF uses host storage | Optionally select an existing PVC in the operator namespace; storage health is checked separately |
operatorConfigName |
dpfoperatorconfig |
Identifies the configuration watched by the operator |
bmcFactoryResetPolicy |
Never |
The native HBN-BGP deployment path requires Never to preserve BMC network settings. The general schema may accept OnInitialization, but native deployment stops before enrollment when it is selected. |
The tenant VIP and interface are site allocations, not discoverable unused resources. A complete input check does not establish address reservation, reachable interfaces, healthy storage or installed controllers.
The selected release catalog records the fixed tenant service range used by
its Kamaji controller (10.96.0.0/16 for the current HBN-BGP release). This is
not a profile default or a service-CIDR override. Generation rejects
an overlap between tenant pod and service ranges or their control-plane/BMC
addresses. Matching ranges in independent clusters are not automatically a conflict;
verify which ranges share a forwarding path.
For generation, provisioning.flavor contains the concrete DPUFlavor spec
for the selected hardware and HBN datapath. It is required and checked against
the selected operator's schema. It is not inferred from a generic BlueField family.
provisioning.resources is a flat map of Kubernetes quantities. Defaults are
memory: 6Gi and nvidia.com/bf_sf: "4"; the HBN chart uses these values as both
requests and limits. Partial maps override the supplied quantities; omitted/null
uses defaults, and an explicitly empty map is invalid. See
HBN-BGP generation.
provisioning.hbn overrides HBN generation settings independently of the
networking values under dpf.hbn. The selected release row supplies default
chart 3.5.0, image 3.5.0-doca3.5.0 with a SHA-256 digest, and startup
format HBN 2.4.0. The profile supplies loopback pool 10.0.124.0/24,
workload gateway index 1, host address index 2,
OnDelete strategy, and
nodeEffect: hold. The flat profile disables the VXLAN startup probe.
Shared resolution requires nodeEffect: hold and chartName: doca-hbn for
this native workflow. Explicit noEffect or another chart name is unsupported
and rejected before credential resolution or artifact retrieval; these values
are never silently replaced by defaults. The DPF service identity remains
doca-hbn; the upstream Helm chart name is a separate release coordinate.
Chart repository/version/digest, image repository/tag/digest and startup version
overrides remain available. imageRepository must include an explicit registry
host and lowercase repository path, without a tag or digest; do not use a
familiar name such as team/hbn. Registry ports from 1 through 65535 are
supported (for example, registry.example:5000/team/hbn). Keep the image tag
and digest in their separate fields. Overrides are preserved, and matching
artifact bytes are verified; custom coordinates are not thereby certified
compatible. Repository syntax alone does not establish native deployment
support: the CLI must read manifest/config metadata over anonymous HTTPS, with
only same-origin anonymous Bearer token exchange. Docker/kubelet pull credentials
do not grant this CLI access. Deployment rejects unsupported metadata access
before discovery probes or cluster changes, and runtime reads it again; see
image metadata prerequisites.
Shared resolution also rejects enabling the VXLAN startup probe.
hold pauses provisioning at the node-effect boundary until the selected
maintenance request is released; it does not mean provisioning has completed.
Host and gateway indexes must be different usable addresses in each per-DPU
subnet. Generation reserves the host slot from ordinary dynamic IPAM allocation;
the actual address follows the DPU subnet allocated during deployment. At most
4096 subnets can be reserved. See host workload addressing.
An overridden loopback pool must not overlap the workload pool. Software
coordinate overrides require matching digests; their compatibility needs
separate validation.
Deploy And Validate Config Lookup¶
Standalone deploy and validate choose the first existing candidate in this order:
- Explicit
--user-config(selected even when unreadable; no lower path is tried). <deployment-files>/.l8k/resolved-config.yaml.<deployment-files>/../.l8k/resolved-config.yaml.<deployment-files>/../cluster-config.yaml.<deployment-files>/cluster-config.yaml../cluster-config.yaml.- With explicit
--config-dir, only itsl8k-config.yamlis considered as a default fallback; if absent, no local/share default file is selected. With no--config-dir, try./l8k-config.yaml, then the installation share-directoryl8k-config.yaml.
The two sidecar locations support passing either the bundle root or its
network-operator/ directory. Effective metadata is loaded without applying
defaults or expanding the release catalog again. Explicit deploy/validate CLI
overrides still apply. An explicit --user-config bypasses the sidecar and
resolves that file; use it only when intentionally changing the configuration
used to interpret the bundle. Connectivity validation requires a sidecar or
user-owned config from steps 1–6, not the installed/default fallback.
A configuration load error prevents connectivity validation. Static-only validation can continue with configuration-dependent version checks skipped; inspect the report before interpreting it as complete acceptance.
Keep .l8k/resolved-config.yaml in its reserved path. It is a versioned metadata
envelope, not a Kubernetes manifest or an ordinary flat cluster config.
Generation still takes a flat config; edit that source and regenerate when the
deployment intent changes.
Top-Level Sections¶
| Section | Purpose |
|---|---|
flavor |
k8s (default) or ocp; selects profile and operator integration. |
nfd |
OpenShift NFD configuration object namespace and name. |
networkOperator |
Release line, image repositories, Helm repository, namespace, image pull secrets, and secondary-network ownership. |
networkNamespaces |
Namespaces that receive secondary network CRs and example DaemonSets. |
workload |
Optional custom workload manifest. |
validation |
Connectivity mode, enabled checks, and RDMA parameters. |
docaDriver |
DOCA driver version and module unload behavior. |
maintenance |
Maintenance Operator and upgrade concurrency limits. |
nvIpam |
IPPool per-node block allocation, subnet generation, manual subnets, and exclusions. |
sriov, hostdev, rdmaShared, ipoib, macvlan |
Profile-specific resource and network naming. |
nicConfigurationOperator |
Optional NIC firmware/runtime tuning and interface naming templates. |
spectrumX |
Spectrum-X naming defaults. |
profile |
Selected fabric, deployment, routing, ARP, and Spectrum-X options. |
cluster |
Optional authoritative pod/service CIDR overrides for IPPool collision checks (YAML only). |
clusterConfig |
Discovered or preset-provided hardware groups. |
discoveredDocaHost |
Discovery-time per-node host doca-host package status and complete version; used for driver defaults and DOCA release compatibility. |
discoveredDocaHostScope |
Generated host-evidence membership (workerNodes, membershipKnown), preserved independently of render topology with host observations. Rediscovery clears it. |
discoveredSecondaryNetwork |
Optional read-only API observations of Multus and reference CNI plugin installers; separate from desired policy. |
Secondary-network ownership¶
Multus and the reference CNI plugin suite are independent components. A platform
Multus installation does not prove that tuning, sbr, host-device, or other
reference CNI binaries are installed. These policies control only the NCP
multus and cniPlugins sections; profile-required components such as
secondaryNetwork.ipoib remain configured. Configure each owner separately:
Each field accepts auto (default), managed, or external. Omitted, null,
and empty values mean auto for compatibility. Managed renders the component
in NicClusterPolicy; external omits it. Auto uses
discoveredSecondaryNetwork: recognized external installers cause omission,
and a verified NCP-owned installer preserves managed rendering. Unavailable
or ambiguous evidence stops automatic generation until the operator chooses
explicit ownership or obtains usable evidence. An offline config with no
observation preserves managed rendering and reports ownership as unverified;
deploy must inspect the live cluster before installing it.
Discovery records notDetected, operatorManaged, external, ambiguous,
or unavailable per component, with reasons and installer names/UIDs. It never
rewrites desired auto into external. Detection recognizes API-visible
Multus executable/image signatures and reference plugins/cni-plugins
images with writable host CNI-directory mounts. Names and the NAD CRD alone
are not proof. Custom host-only installers can be invisible: choose explicit
external ownership after checking the site's prerequisites. Generation does
not contact the cluster or prove installer health.
OpenShift always uses external Multus and reference plugins; explicit managed
ownership is rejected. Helm ownership (skipHelmChart) remains independent
of component ownership.
Deploy blocks collisions and implicit managed-to-external transitions before
Helm or resource mutations. Removing an existing NCP section can cause the
Network Operator to delete its networking children. An ownership migration
requires a separately reviewed operator-controlled procedure; changing this
field or using --overwrite-existing does not authorize that migration. This
also protects differently named NCPs selected for overwrite deletion, independently
of installer DaemonSet presence. Policies without managed Multus/CNI sections
retain ordinary stray-remediation behavior.
Release Catalog¶
Supported release lines are currently:
26.126.426.726.10(v26.10.0-beta.1)
The release line fills Network Operator versions, component image tags, DOCA driver version, full-runtime validation image, repositories, Helm repository URL, and version-gated template behavior. Spectrum-X releases also carry a manually maintained xPlane repository and version used in spectrumXOperator.xPlane instead of reusing the generic component tag.
Nightly synchronization follows each upstream release branch. Once a catalog line points to a GA Network Operator release, beta and release-candidate builds of a later patch do not replace its public artifact set. The line advances again when that patch is published as GA.
Network Operator¶
| Field | Meaning |
|---|---|
selectedRelease |
Catalog release line. Equivalent to --network-operator-release. |
version |
Network Operator chart/operator version. Filled by the selected release. |
componentVersion |
Tag used by managed component images. |
skipHelmChart |
When true, generation omits values.yaml, deploy skips Network Operator chart installation and Helm preflight checks, validate skips Helm release-version and values checks, and clean retains the externally owned Helm release. Network Operator CR generation, deployment, validation, and cleanup remain enabled. Equivalent to --skip-network-operator-helm for generate, deploy, and validate. |
repository |
Registry path for component images such as drivers, CNI, IPAM, and device plugins. |
operatorRepository |
Registry path for the Network Operator controller image. |
helmRepoURL |
Chart repository used by l8k deploy. Empty means Helm phase 0 is skipped. |
namespace |
Namespace for the Helm release and namespaced Network Operator resources. |
imagePullSecrets |
Secret names propagated into the discovery daemon, generated policies, and Helm values for the Network Operator and enabled subcharts. During deploy, matching credentials also authenticate the Helm chart download. |
When selectedRelease is set, catalog values replace explicit version and repository fields so the cohort remains consistent.
Use skipHelmChart: true when another system owns the Network Operator Helm
release:
The selected release still drives rendered component versions and their validation; only the Helm artifact/install/check/uninstall boundary is disabled.
For an authenticated Helm repository, each referenced Secret must already
exist in networkOperator.namespace before l8k deploy starts, and the
kubeconfig must allow get on Secrets there. l8k reads
kubernetes.io/dockerconfigjson and legacy kubernetes.io/dockercfg data in
memory and never logs or persists the credential. Credentials are sent only
when the Docker registry host exactly matches the Helm repository host. The
one explicit cross-host mapping is NGC: nvcr.io credentials use the same
$oauthtoken and API key required by helm.ngc.nvidia.com. Unrelated registry
credentials are never forwarded to the chart server.
Network And Workload Namespaces¶
Secondary-network CRs and default example workloads render once per network
namespace outside Spectrum-X, which uses the first namespace. Shared policy
and pool resources do not fan out. workload.manifest is equivalent to
--workload-manifest: it replaces the example with an operational
90-workload-*.yaml deployment input. Kubernetes custom workloads use the first
namespace only; OpenShift custom workloads fan out per network bucket and
namespace. See custom workload lifecycle
for the resulting connectivity-fixture requirement.
Validation¶
validation:
gpuDirect:
enabled: false
gpuResourceType: nvidia.com/gpu
connectivity: true
mode: strict
checks:
- gateway
- icmp
- rping
- ib_write_bw
rdma:
rpingIterations: 5
ibWriteSize: 65536
ibWriteMinBandwidthGbps: 100
| Field | Meaning |
|---|---|
gpuDirect.enabled |
Run a separate CUDA DMA-BUF ib_write_bw matrix when ib_write_bw is selected. Fresh discovery enables it only when every discovered worker can satisfy its render bucket's topology-derived gpuResourceType request. Discovery with --user-config preserves the supplied value. |
gpuDirect.gpuResourceType |
Qualified Kubernetes extended resource requested by the primary validation container. Defaults to nvidia.com/gpu. |
connectivity |
Enables the data-plane stage. Static acceptance checks always run. |
mode |
quick, full, or strict. |
checks |
Any combination of gateway, icmp, rping, and ib_write_bw (all enabled by default); an empty list disables all connectivity test families. Gateway runs first and resolves the NAD's NV-IPAM IPPool in networkOperator.namespace, falling back to runtime interface routes for other IPAM/CIDRPool or an unset pool gateway. Explicit no-gateway rows are skipped. |
rdma.rpingIterations |
Client iterations for each rping test. |
rdma.ibWriteSize |
Message size passed to ib_write_bw. |
rdma.ibWriteMinBandwidthGbps |
Minimum observed peak bandwidth. Set 0 to disable the bandwidth gate. |
GPU resource counts are not added to clusterConfig. Generation derives the
request needed to expose the discovered GPU<N> indices from the existing PF
topology. The DMA-BUF runner resolves source and destination indices separately
from connectedGPU (and reports connectedGPUPCIAddress when available); an
unresolved or ambiguous NIC/GPU association is a failed validation result.
networkOperator.imagePullSecrets is copied to the validation Pod spec, so
each named Secret must exist in every generated network namespace.
DOCA Driver¶
docaDriver:
enable: true
version: <catalog-version>
unloadStorageModules: true
enableNFSRDMA: false
unloadThirdPartyRDMAModules: true
skipPreflightChecks: false
# Advanced users only. Custom values override generated driver env by name.
# env:
# - name: THIRD_PARTY_RDMA_MODULES
# value: "nvidia_peermem"
| Field | Meaning |
|---|---|
enable |
Include DOCA/OFED container driver configuration. Defaults to true. After discovery, complete all-worker host DOCA-HOST evidence changes an omitted/null choice to false; explicit true/false wins. |
version |
Driver image tag; filled by the selected release catalog. |
unloadStorageModules |
Allow unload of storage-over-RDMA dependencies before driver replacement. |
unloadThirdPartyRDMAModules |
Allow unload of non-MLX RDMA dependencies before driver replacement. |
enableNFSRDMA |
Enable NFS-over-RDMA support. |
skipPreflightChecks |
Skip the init-container module dependency check. |
env |
Advanced literal name/value environment entries forwarded through the generated NicClusterPolicy or NicNodePolicy. Custom values override generated values by name; the last custom duplicate wins. |
Host observation scope is independent of topology. Generation retains original
worker names and membership completeness in discoveredDocaHostScope when host
observations exist. The optional scope contains unique, nonempty worker names;
membershipKnown: true also requires a nonempty list. Incomplete membership
remains unverified and cannot justify automatic container disablement. Without a
preserved scope, host checks use clusterConfig worker membership. --for
changes topology only; it does not replace this source of host evidence. The
scope and observations round-trip in the generated bundle, including when reused
without a preset. Rediscovery clears the old scope before replacing inventory.
Host package observations are independent of desired driver ownership. Mixed installed/absent or installed/unknown workers require an explicit enable choice before generation; automatic inference covers the entire source inventory, including filtered generation. Discovery refresh preserves omitted enable fields. See host-installed DOCA-HOST for observation status, release comparison granularity, and snapshot lifetime.
Both unload controls default to true. Discovery does not populate the holder-module lists automatically. Confirm that no active workload depends on a module that the driver flow will unload.
env is an advanced escape hatch for driver settings that Launch Kit does not
model. The generated policy contains no duplicate environment names. Incorrect
values can prevent MOFED from reloading and disrupt node networking.
Maintenance¶
The top-level maintenance fields control Maintenance Operator and legacy upgrade concurrency:
maintenance:
maxParallelOperations: 4
maxUnavailable: 4
maxNodeMaintenanceTimeSeconds: 3600
maxParallelUpgrades: 4
Values can be integers or percentages where supported. See Maintenance for release-specific requestor behavior.
NV-IPAM¶
Auto-generate per-group subnets:
nvIpam:
poolName: nv-ipam-pool
perNodeBlockSize: 10
startingSubnet: "192.168.0.0"
mask: 22
offset: 1
startingGateway: 1
reserveFirstIPs: 10
reserveLastIPs: 6
Or list subnets manually:
nvIpam:
perNodeBlockSize: 10
subnets:
- subnet: 192.168.2.0/24
gateway: 192.168.2.1
exclusions:
- {startIP: 192.168.2.2, endIP: 192.168.2.3}
Reserved first/last IPs are merged with explicit exclusions for every subnet.
| Field | Meaning |
|---|---|
poolName |
Base name for generated IPPool resources. |
perNodeBlockSize |
Non-negative number of addresses reserved per node in each generated IPPool; 0 or omission defaults to 10. SR-IOV generation warns when the effective value is smaller than sriov.numVfs. |
startingSubnet |
Aligned IPv4 network address for automatic allocation. |
mask |
Automatic subnet prefix length, from /1 through /30. |
offset |
Number of subnet-sized blocks between allocations; minimum 1. |
startingGateway |
Positive host offset from each automatic subnet network address; omitted/null defaults to 1. Must be below the broadcast offset (2^(32-mask)-1); explicit 0 and negative values are rejected. Manual subnet gateways remain authoritative. YAML only. |
reserveFirstIPs / reserveLastIPs |
Host addresses excluded from every automatic or manual subnet. |
subnets |
Explicit {subnet, gateway, exclusions} entries. A non-empty list takes precedence over automatic fields. |
For example, startingGateway: 254 gives 192.168.2.254 for
192.168.2.0/24. For 192.168.2.128/25, startingGateway: 126
gives 192.168.2.254.
Offsets can exceed 255: startingGateway: 300 gives 192.168.1.44 for
192.168.0.0/22. This is separate from offset, which spaces subnet networks.
The generated IPPool.spec.gateway carries this address; NV-IPAM excludes its
gateway from allocation independently of the optional reserved ranges.
Automatic subnet allocation is precomputed across all final heterogeneous render buckets to avoid overlap within the generated bundle.
These ranges are configurable defaults, not cluster-reserved address space.
l8k generate checks the rendered IPPools against any explicit cluster ranges
before replacing output. l8k deploy additionally reads live cluster addresses
before any writes. Choose subnets that do not overlap other site networks either.
Cluster address overrides¶
Each supplied list is authoritative for its own address class and replaces live discovery of that class. Include every applicable IPv4/IPv6 range. Omitted/null lists request discovery; explicit empty lists, invalid CIDRs and IPv4-mapped IPv6 CIDRs are rejected. Host bits are masked for overlap comparisons. These YAML-only fields survive resolved-config handoff and discovery refresh without being filled from partial observations.
Deployment inspects all Nodes, regardless of selected hardware groups:
spec.podCIDRs (falling back to spec.podCIDR) and each status.addresses
InternalIP. A Node's missing pod range or InternalIP produces a warning naming
that Node. Node-list failures and malformed addresses fail the check. Node pod
ranges are only best-effort evidence: CNIs managing their own IPAM may leave them
empty or populate ranges different from actual pod allocation. Use podCIDRs
for authoritative CNI ranges. Node InternalIPs are checked even with overrides;
they identify individual addresses, not the node network's full subnet.
On OpenShift, where OVN-Kubernetes leaves Node pod ranges empty, deployment
first reads spec.clusterNetwork from the config.openshift.io/v1 Network
cluster and uses those ranges instead of Node pod ranges. The lookup happens on
every run, so an expanded cluster network is checked without rediscovery. If the
API or object is absent, or it lists no ranges, deployment falls back to Node pod
ranges. Other read failures, including RBAC denial, and malformed ranges stop
deployment. A supplied podCIDRs skips this lookup.
Unless serviceCIDRs is supplied, deployment lists all ServiceCIDR objects,
using networking.k8s.io/v1 then v1beta1 only if the GA API is absent. Failed
reads (including RBAC denial), malformed ranges and an empty served API stop
deployment. If neither API is served, deployment warns that service ranges are
unverified; this supports Kubernetes 1.32 without MultiCIDRServiceAllocator.
This is capability-based, not a version-string exemption: supply an override
when the API is unavailable. See the upstream ServiceCIDR guide.
Any detected overlap fails with the IPPool file/document/name and conflicting
range or Node. Checks cover NV-IPAM IPPool resources, not Spectrum-X CIDRPool,
other IPAM providers, or site-wide address reservation. Offline generation can
check only explicit overrides; discover does not persist inferred cluster
ranges. Standalone validate is not a replacement for this deployment guard.
Profile Resource Settings¶
| Section | Fields |
|---|---|
sriov |
ethernetMtu, infinibandMtu, numVfs, priority, resourceName, networkName. |
hostdev |
resourceName, networkName. |
rdmaShared |
resourceName, hcaMax. |
ipoib |
networkName. |
macvlan |
networkName. |
When multirail is enabled, Launch Kit adds rail suffixes to generated resource and network names.
NIC Configuration And Naming¶
nicConfigurationOperator:
deployNicConfigurationTemplate: true
deployNicInterfaceNameTemplate: true
rdmaPrefix: "rdma_r%rail_id%"
netdevPrefix: "eth_r%rail_id%"
updateFW: false
spectrumX:
overlay: "none"
singlePlane:
netdevPrefix: "eth_r%rail_id%"
rdmaPrefix: "roce_r%rail_id%"
hwplb:
netdevPrefix: "eth_r%rail_id%_p%plane_id%"
rdmaPrefix: "roce_r%rail_id%"
swplb:
netdevPrefix: "eth_r%rail_id%_p%plane_id%"
rdmaPrefix: "roce_r%rail_id%_p%plane_id%"
| Field | Meaning |
|---|---|
deployNicConfigurationTemplate |
Default true; standard-profile NIC tuning independently of interface naming. Explicit false opts out. CLI: --deploy-nic-configuration-template[=false] on root, generate and discover. |
deployNicInterfaceNameTemplate |
Default true; standard multirail profiles skip naming for a single selected group unless RDMA-shared needs missing interface names. Multiple selected groups retain naming; Spectrum-X uses its required templates. Explicit false disables standard-profile naming. |
rdmaPrefix / netdevPrefix |
Standard-profile names. Multirail values require a rail placeholder. |
updateFW |
Enables firmware staging storage in generated Network Operator configuration. |
spectrumX.overlay |
Spectrum-X overlay mode. |
spectrumX.singlePlane |
Prefix block selected by none; defaults both device types to rail-only names. |
spectrumX.hwplb |
Prefix block selected by hwplb; defaults RDMA to rail-only and NET to rail-plane names. |
spectrumX.swplb |
Prefix block selected by swplb; defaults both device types to rail-plane names. |
For standard SR-IOV, RDMA-shared (Macvlan/IPoIB), and host-device profiles on
Kubernetes and OpenShift, deployNicConfigurationTemplate: true enables NCO
and emits one NicConfigurationTemplate per source hardware group. It uses
that group's node selector, east-west NIC type and exact PCI addresses. Missing
selectors or mixed east-west NIC types within one group are rejected. The
setting is independent of deployNicInterfaceNameTemplate and updateFW;
Spectrum-X owns its required NIC configuration templates independently of this
standard-profile toggle, so either value leaves those templates enabled.
For standard profiles, omitted tuning settings default to true; explicit YAML
false or --deploy-nic-configuration-template=false opts out.
Standard interface naming orders NICs by their configured rail numbers so generated names match policy selectors. Rail numbers must be contiguous from zero, with one east-west PF per rail; invalid mappings fail generation.
Standard interface naming rejects an emitted mapping that places PCI functions of one physical NIC in different rails. NCO stores one rail index per physical NIC; the resulting names would not match the generated policy selectors. This check applies only to selected groups whose naming template is emitted and does not silently turn off naming.
The template routes numVfs from sriov.numVfs for all these profiles and maps
profile.fabric to Ethernet or Infiniband. It enables PCI performance tuning
with maxReadRequest: 4096 and gpuDirectOptimized for Baremetal. Ethernet
also enables roceOptimized with DSCP trust and PFC "0,0,0,1,0,0,0,0";
InfiniBand omits that block. NCO settings operate on whole NICs, so confirm that
all ports of each selected NIC are within the intended configuration scope.
Generation rejects inventory that assigns east-west and excluded ports to the
same PCI device (domain:bus:device). This check cannot identify omitted ports or
NICs spanning different PCI devices; inventory and site validation must cover
those cases.
Only SR-IOV profiles disable the SR-IOV operator's mellanox plugin when this
option is enabled: through Helm values on Kubernetes, and through
SriovOperatorConfig on OpenShift. OCP adds the entry without removing other
disabled plugins. Turning the option off stops rendering its template and
plugin-disable request, but does not restore applied NIC settings. On Kubernetes,
deploy preflight treats omitted templates (including unselected groups in a
subset deployment) as stray resources: deployment blocks without
--overwrite-existing, and deletes them with that flag. Review the deletion
scope before an opt-out or subset deployment. OCP preserves existing templates
and disablePlugins values; cleanup requires an explicit action. With
skipHelmChart: true on Kubernetes, the externally managed SR-IOV operator must
be configured separately to disable mellanox before applying the template.
RDMA-shared and host-device profiles do not change SR-IOV operator settings.
Deploy and validate monitor all matched NicDevice objects and require current
ConfigUpdateInProgress conditions with reason: UpdateSuccessful and
status: "False". Missing, stale or partial status remains in progress; runtime
or firmware configuration failures are reported. Confirm NCO/DOCA/RHCOS runtime
RoCE support for the selected release before enabling this option on OCP,
then validate connectivity on the selected workers.
Spectrum-X NicConfigurationTemplate.spec.nicSelector is not a separate
configuration input. Launch Kit renders one template per source hardware group
and derives both nicType and pciAddresses from clusterConfig[].pfs[]
entries whose traffic is east-west. The NIC Configuration Operator matches
the intersection of those fields, so a north-south DPU with the same device ID
as an east-west SuperNIC is not selected. Every selected east-west PF must have
the same non-empty device ID and a non-empty PCI address or generation fails.
Profile¶
profile:
fabric: ethernet
deployment: sriov
multirail: true
routing: destination-based
ignoreARP: false
spectrumX:
enable: false
| Field | Meaning |
|---|---|
fabric |
ethernet or infiniband. |
deployment |
sriov, rdma_shared, or host_device. |
multirail |
Enable more than one east-west rail. An explicit false is preserved. |
routing |
destination-based or source-based; source-based adds the sbr CNI plugin outside Spectrum-X. |
ignoreARP |
Adds arp_ignore=1, arp_announce=2, and rp_filter=0 through the tuning CNI plugin outside Spectrum-X. Both Kubernetes and OpenShift render all and IFNAME scopes. OpenShift deploy adds the corresponding Multus sysctl allowlist entries while preserving existing entries. |
spectrumX.enable |
Select a Spectrum-X profile. |
spectrumX.spcxVersion |
RA2.1, RA2.2, RA2.3, RA2.4, or RA3.1; RA2.3 requires 26.7, RA2.4 and RA3.1 require 26.10 or a newer catalogued release. |
spectrumX.multiplaneMode |
none, swplb, or hwplb. When absent, H100/H200/B200/GB200 default to none; B300/GB300 default to the GA swplb path. B300/GB300 require explicit hwplb; VR with ConnectX-9 defaults to hwplb on RA2.4/RA3.1. |
spectrumX.numberOfPlanes |
1, 2, 4, or 8. Single-plane platforms default to 1; B300/GB300 default to 2; VR/CX9 HWPLB defaults to 8 on RA2.4/RA3.1. Pass 4 explicitly for quad-plane B300. |
spectrumX.topologyType |
2-tier or 3-tier. |
spectrumX.ipVersion |
ipv4 for per-node /31 allocation or ipv6 for per-node /64 allocation. |
spectrumX.hostFirstOctet |
Config-only first octet for generated IPv4 topology addressing. |
spectrumX.topologyFile |
Path to spcx-gen/reference-generator or contract-compliant NVIDIA AIR topology JSON. The format is detected from its structure; relative paths resolve from the config file. |
spectrumX.configMapName / profile |
RA2.3 legacy ConfigMap name/data, or RA2.4/RA3.1 full doSPCX ConfigMap (including binaryData). Rendering always uses the operator namespace. |
spectrumX.useDRA |
Render DRA ResourceClaimTemplate workload allocation; see prerequisites. |
The example above is a standard profile. When Spectrum-X is disabled, omit its RA, mode, plane, topology, and ConfigMap fields; setting them is an error. For an enabled RA2.3 profile, start with the full ConfigMap generation example, which supplies the required profile payload as well as the RA/release pair.
RA2.4 and RA3.1 additionally accept spectrumX.ovsConfig (a default-empty string map)
and per-group clusterConfig[].spectrumX.swPlaneByRail (a default-empty
integer map for existing HWPLB rails). Discovery and resolution write derived
clusterConfig[].spectrumX.platformType using the longest case-insensitive
doSPCX platform substring in gpuType; unresolved discovery warns, while
RA2.4/RA3.1 generation fails. See the RA2.4 contract
for release prerequisites, merging, defaults and supported layouts.
Discovery saves the effective RA naming in
clusterConfig[].spectrumX.interfaceNames (netdevPrefix and rdmaPrefix).
These are derived values: resolution refreshes them after mode or naming
changes, including for older configs without this block. Change the existing
spectrumX.singlePlane, hwplb, or swplb prefix settings for custom naming.
PF inventory names remain observed values. Naming templates and PF selectors
consume the same resolved per-group prefixes; standard profiles are unaffected.
VR HWPLB uses eth_r%rail_id%_p%plane_id% and
roce_r%rail_id%_n%nic_id%; traditional HWPLB retains roce_r%rail_id%.
VR generation supports IPv4/IPv6, eight-plane HWPLB, four rails and two
NICs per rail. It emits per-NIC pools/attachments and requires explicit host
indices in reference topology JSON. See VR addressing and naming
for field limits, custom prefix rules and migration requirements.
profile.spectrumX.ipv6SiteID is a YAML-only integer from 0 to 15, defaulting
to zero. Nonzero values apply only to VR IPv6; hostFirstOctet remains IPv4-only.
A machineType containing VR (case-insensitive) selects platform vr before
GPU-product matching, including when gpuType is empty. With ConnectX-9 and
RA2.4/RA3.1, omitted mode/planes default to hwplb/8. Explicit swplb with no
plane count retains the CX9 four-plane fallback. See VR discovery
for the NetworkBay-derived rail mapping and discovery prerequisites.
Hardware Groups¶
Each clusterConfig entry represents one source group:
clusterConfig:
- identifier: pe-xe9680-h200
machineType: PowerEdge-XE9680
gpuType: NVIDIA-H200
netplanManaged: true
# Discovery adds this when no match is found but at least one worker could
# not be verified
# netplanManagementUnknown: true
capabilities:
nodes:
sriov: true
rdma: true
ib: false
nodeSelector:
nvidia.kubernetes-launch-kit.machine: pe-xe9680-h200
pfs:
- deviceID: a2dc
pciAddress: 0000:1a:00.0
rdmaDevice: rocep26s0f0
networkInterface: eth2
traffic: east-west
rail: 0
Fresh discovery may merge compatible source groups during generation, but it keeps source groups explicit in the config for review and filtering.
| Field | Meaning |
|---|---|
identifier |
Saved group name used with --groups and the Launch Kit machine node label. Names are lowercase and may be shortened; copy the saved value rather than constructing it. Unresolved multiple groups use group-N. |
machineType / gpuType |
Discovered hardware identity. |
linkType |
Per-group Ethernet or InfiniBand result when fabric probes agree. |
netplanManaged |
true when any worker has an NVIDIA PF selected by a netplan match.macaddress stanza with a non-empty set-name. Blocks generation when a NicInterfaceNameTemplate would configure the group. |
netplanManagementUnknown |
true when no match was found but a worker's netplan observation was unavailable or incomplete, or a MAC-based set-name rule could not be checked because an expected PF MAC was unreadable. A readable /etc without a netplan directory, or a complete configuration without MAC-based set-name rules, is a verified negative. When a NicInterfaceNameTemplate would configure the group, generation warns and continues unless netplanManaged is true. Review host netplan configuration and re-run discovery to resolve the warning; existing configs should be refreshed with discovery to populate this field. |
presetApplied |
An exact topology preset was applied. |
presetDeviation |
PF count, PCI address, or device ID drift from a matched preset. |
capabilities.nodes |
Group-level SR-IOV, RDMA, and InfiniBand capability flags. |
workerNodes |
Kubernetes node names in the source group. |
nodeSelector |
Deployment selector, normally the Launch Kit machine label whose value matches identifier. |
storageModules / thirdPartyRDMAModules |
Optional site-supplied dependent module lists. |
pfs |
Physical-function inventory. |
Each PF can include deviceID, pciAddress, rdmaDevice, networkInterface,
traffic, rail, psid, partNumber, model, numaNode, connectedGPU,
connectedGPUPCIAddress, and gpuProximity. PF MAC addresses are
host-specific and are not part of the saved configuration.
OpenShift flavor¶
See OpenShift host deployments for operator prerequisites, separate namespace defaults, supported profiles, and validation behavior. sriov.operatorNamespace, nfd.operatorNamespace, nfd.configurationName, and maintenance.operatorNamespace can override their OpenShift defaults.