Validation¶
l8k validate checks generated manifests against the live cluster, runs
configured data-plane checks, and writes an acceptance report. Read the
acceptance outcomes together with the test coverage:
exit 0 alone does not prove that every requested stage ran.
When --user-config is omitted, deploy and validate prefer the bundle's
.l8k/resolved-config.yaml over conventional cluster config files. See the
exact lookup order.
The sidecar preserves the release, namespace, profile, groups, and validation
settings used during generation. Connectivity accepts this resolved metadata
or a user-owned configuration with the required explicit decisions; installed
and embedded defaults alone do not satisfy that requirement.
Host DOCA-HOST release compatibility¶
Before cluster checks, validation compares installed discoveredDocaHost
observations for discovery workers with networkOperator.selectedRelease.
When generation overrides topology with --for, the bundle retains these
workers in discoveredDocaHostScope; preset topology cannot bypass this check.
It checks the three-part DOCA release against the catalog synchronized from the
Network Operator release SBOM's Mofed.version. For example, host package
3.5.0-041000-26.07-ubuntu2404 matches a doca3.5.0-... driver coordinate;
package/OFED/container build suffixes are not equivalent and are not compared.
A different or unparseable DOCA release, missing release selection, or malformed
snapshot fails with exit 2 and a partial report. Explicitly enabling the
container driver does not bypass this gate; it also applies with
--connectivity=false and in connectivity-only validation.
These are saved discovery-time package observations, not a live host check. Rerun discovery after host package changes. Missing or unknown observations do not establish compatibility; configs without installed-host observations keep existing validation behavior. A match does not qualify a package build, firmware, loaded kernel modules, or data-plane functionality.
InfiniBand partition diagnostics¶
During connectivity setup, validation reads the P_Key table for each selected
IPoIB interface in the test pods, including SR-IOV VFs. It uses the interface's
unique direct RDMA device mapping and zero-based dev_port to select the port;
it does not substitute the host PF table or guess from rail ordering.
Ethernet interfaces are excluded. Static validation does not run this probe.
A table containing only limited membership keys (for example 0x7fff) produces
an advisory warning: two limited members cannot communicate within the same
partition (see NVIDIA partition membership semantics).
Ask the subnet manager administrator to grant appropriate full membership to the intended host/VF GUIDs. Empty slots are ignored; an entirely
empty table also warns. Missing or ambiguous mappings, unreadable sysfs data,
malformed tables and exec failures are reported as unverified, never healthy.
The read-only probe has a five-second budget per pod and requires no extra pods.
Warnings appear in terminal output, the JSON connectivity object's
PartitionWarnings, and the HTML warnings section, including when subsequent
connectivity tests fail. They do not change exit status: limited-to-full traffic
can be valid, and a full key in an unrelated partition does not prove connectivity.
No warning is a substitute for checking the intended partition and testing the
actual data path. Interfaces excluded from the matrix are not inspected.
Acceptance Outcomes¶
| Invocation | Exit and coverage behavior |
|---|---|
| Full Kubernetes validation | Failed static gates or gating connectivity tests return 4. In-progress manifests can return 0 with warnings and skip connectivity. A skipped matrix (for example, fewer than two usable pods) or missing selected-family coverage does not independently fail this mode. |
| Connectivity-only validation | Requires at least one gating test for every selected family and all gating tests to pass, except explicit no-configured-gateway applicability skips. Empty endpoints or incomplete coverage returns 4. |
| OpenShift with connectivity enabled | Enforces the same per-family coverage requirement; in-progress manifests that prevent connectivity return 4. |
--connectivity=false |
Runs static checks only. Success supplies no data-plane evidence. |
Missing/errored manifests, release or values/component mismatches, stray
resources, and certified-preset deviations are failure gates in full
validation. Missing or malformed test DaemonSets are input errors (exit 2)
when connectivity is enabled, before cluster checks. validation.checks: []
disables all test families; it does not demonstrate connectivity.
Use --wait <duration> to poll in-progress manifests. Kubernetes resources
still in progress after that wait remain warnings under the current behavior;
the wait deadline alone is not a failure gate. Before accepting a deployment,
require the report to show the intended manifests ready, the required static
checks completed, and gating tests for every required data-plane family.
Review skipped stages and non-gating cross-rail observations separately.
External networking prerequisites¶
For Kubernetes NCP bundles, validation checks live Multus/CNI ownership independently of catalog versions. An omitted section with a still-managed live section is an ownership conflict. Supported external installers are checked against configured worker names: incompatible node selectors are errors; missing Ready pods or stale DaemonSet rollout status are in-progress. Ready pods on all required workers plus current DaemonSet readiness establish API-level installer coverage.
No recognized API installer, failed optional health inspection, or unavailable
worker inventory is UNVERIFIED, with a reason in text, JSON (Skipped:
true), HTML, and JUnit. It is neither missing networking nor a successful
health check and does not alone fail static validation. Failed mandatory
ownership inspection is an error. The report lists external prerequisite rows
separately from NCP-managed catalog-version checks. Host CNI binaries,
configuration, and data-plane behavior still require runtime qualification.
On OpenShift, Multus and the reference CNI plugins are platform-managed. Validation reports their external prerequisite rows as UNVERIFIED and does not inspect installer DaemonSets or gate on their count, selectors, pods, or rollout status. It still checks NCP ownership: managed sections in OpenShift artifacts, implicit managed-to-external transitions, and failed mandatory ownership reads remain errors. NCP readiness and connectivity checks still apply; skipped installer checks do not establish networking health.
The same ownership boundary applies to NCP status.appliedStates: when both
desired and live specs omit Multus/CNI ownership, their controller states are
marked unverified and excluded from an otherwise non-ready NCP aggregate.
Every configured component in either spec must still report ready; missing
coverage and other component failures retain the aggregate error/progress state.
Unknown spec fields,
malformed or duplicate status entries, and policy-level reasons retain the
controller's aggregate verdict. The report's live YAML preserves the original
status. Connectivity runs only after the remaining managed resources are ready.
Worked acceptance decisions¶
The following examples are illustrative decisions, not records of a live qualification run. Compare the intended worker and rail set with the report's usable test pods and actual matrix coverage before accepting a deployment.
| Example | Process result and report | Decision |
|---|---|---|
| Complete two-worker, one-rail check | Intended manifests are READY; required static checks complete; both workers provide usable test pods; every required selected family has completed gating tests on the intended rail with no gating failures. |
Accept for those stated checks and topology, subject to the site's thresholds and exclusions. |
| Incomplete coverage | Kubernetes full validation exits 0, but a manifest is still in progress or only one of the two intended workers has a usable test pod. Connectivity is skipped or its matrix omits the second worker. |
Do not accept a deployment that requires two-worker data-plane evidence. Inspect the warning and missing endpoint, then rerun. |
| Failed gate | A release or manifest check fails, or a gating connectivity test fails; validation returns 4. |
Investigate the failing stage and preserve the report before retrying. |
Non-gating cross-rail observations, an empty selection such as
validation.checks: [], or a static-only success do not prove required
connectivity. --wait can give reconciliation more time; on Kubernetes,
exhausting that wait does not by itself change an in-progress warning into a
failure. Document any site-approved exclusions outside Launch Kit and assess
coverage against that intended scope.
Report¶
By default, validation writes a self-contained HTML report at the root of
--deployment-files (default ./deployment), even when generated manifests
are under its network-operator/ subdirectory:
Override or disable it:
The report includes release checks, component checks, manifest state, live YAML dropdowns, topology/preset comparison, connectivity matrices, and warnings.
| Report section | Content |
|---|---|
| Environment | Launch Kit version, timestamp, API server, operator namespace, and kubeconfig context. |
| Profile | Fabric, deployment type, multirail, routing, and Spectrum-X settings. |
| Node groups | Machine/GPU identity, labels, capabilities, worker nodes, PF inventory, and paired actual/expected preset topology. |
| Release and preflight | Selected and deployed chart, values, component versions, and stray-resource results. |
| Manifest state | State, reason, kind-specific details, and live YAML with managedFields removed. |
| Connectivity | Same-rail and cross-rail ICMP/RDMA matrices with bandwidth observations. |
| Warnings | Skipped stages, in-progress resources, and other non-success conditions. |
The report is one HTML file with inline styling and no external runtime dependency, so it can be opened offline or attached to an approved diagnostic record.
JUnit XML¶
Write a JUnit report alongside text, JSON and the HTML report:
--junit-path is opt-in and independent of --report-path. Parent directories
are created. The file is replaced atomically after validation, including early
errors and partial results. Failure to write a requested file fails an otherwise successful command; if
validation already failed, its original exit code is preserved and the write
error is logged separately.
Preserve the exit status as well as the report. The existing JSON stream is
unchanged; XML is written only to the requested file.
Connectivity reporting requires profile.fabric: ethernet or infiniband
in the resolved cluster config. No fabric is guessed from the probe command.
Existing families retain their catalog names from Network Test Discussion Notes;
gateway checks add a dedicated suite:
| Probe | Suite name (append -ethernet or -infiniband) |
|---|---|
| Gateway ICMP | K8sEastWestNetworkGatewayICMPPing |
| ICMP | K8sEastWestNetworkICMPPing |
| RDMA ping | K8sEastWestNetworkRDMAPing |
| Host-memory bandwidth | K8sEastWestNetworkIBWriteBandwidth |
| GPUDirect DMA-BUF bandwidth | K8sEastWestNetworkDMABufBandwidth |
The root is <testsuites name="l8k validation tests">. Each family is a direct
<testsuite> child with tests, failures, errors, skipped and time
attributes. Each directional pod/rail probe is a direct <testcase> child,
named FamilyName::source→destination; endpoint names include node, rail, pod
and IP to distinguish probes. Suite counters summarize these individual results.
Suite time is measured wall-clock seconds for the family stage, including route
checks. Individual probe times are omitted because batched RDMA execution does
not measure each probe separately. Unexecuted suites have time 0.000.
system-out carries structured probe evidence (including expectations,
observations, endpoints, bandwidth and command output); system-err carries
stderr. Failures use the existing expectation-aware verdict: expected isolation
can pass, and observe-only disconnections do not independently fail validation.
Disabled families and families without executed probes have an explicit skipped
case with a reason. A missing selected-family coverage gate is recorded as a
separate ConnectivityCoverage failure when the existing validation policy
requires coverage, even if another check also failed. Completed probes
survive execution errors. Other checks and execution errors are recorded in
network/validation; in-progress manifests are skipped, not passed. Each evaluated preset group has
a TopologyPresets::group case: matches pass, deviations fail, and missing or
skipped presets are skipped with their reason. The report
preserves the existing mode/flavor acceptance rules described below.
Check Stages¶
| Stage | What it verifies |
|---|---|
| Helm release | Network Operator chart appVersion and rendered user values. |
| Component versions | Version-bearing sections of NicClusterPolicy and NicNodePolicy match the selected release catalog. |
| Manifest state | Each generated CR is classified as READY, IN-PROGRESS, ERROR, or MISSING. |
| Preflight | Stray CRs and Helm value drift that can make the apply path ambiguous. |
| Connectivity | Per-pod/rail gateway ICMP reachability first, then ICMP, rping, host-memory ib_write_bw, and optional GPUDirect DMA-BUF bandwidth checks between generated test DaemonSet pods. |
The Helm version check prefers the exact release name network-operator and
reads its highest revision, so a sibling release such as network-operator-post
cannot override it. If the exact name is absent, the check retains compatibility
with a single release whose name contains network-operator (case-insensitive).
Multiple such fallback releases are ambiguous: the version check is reported as
skipped with a reason, not as a verified match. Revisions from different releases
are never compared. The check compares chart appVersion with the catalog;
Helm values validation continues to use the exact network-operator release.
When another system owns the Network Operator Helm release, set
networkOperator.skipHelmChart: true or pass
--skip-network-operator-helm. The Helm release/version and values checks are
then reported as skipped. Component versions, manifest state, stray-resource
preflight, connectivity, and the HTML report still run.
In full validation, connectivity is skipped when manifests are missing, errored, or still in progress.
Each generated example DaemonSet declares two validation containers: the DOCA
container runs rping, ib_write_bw, and DMA-BUF bandwidth, while the netshoot container runs
ICMP and route checks from the same pod network namespace. Validation applies
the generated DaemonSet as written; it does not inject a helper container at
runtime.
Connectivity discovery supports IPv4-only and IPv6-only secondary attachments.
On dual-stack attachments it retains IPv4 preference; it does not test both
families. IPv6 uses ping -6, ip -6 route get, and
ib_write_bw --ipv6 --ipv6-addr on both endpoints. rping accepts IPv6
addresses directly. Link-local and malformed source endpoint addresses are excluded;
IPv6 link-local gateways remain supported with interface scope.
Gateway checks run first after test pods are ready, before all pair-based
connectivity families, regardless of validation.checks order. Every mode
checks each pod/secondary rail once; --validation-checks gateway also works
with one pod. Validation follows the exact namespaced NetworkAttachmentDefinition
from Multus status and its single explicit NV-IPAM poolName to the live IPPool
in networkOperator.namespace. Its spec.gateway is probed even when the pod
has only a connected route and no via route. Source IP membership in that
pool's subnet must match. Standalone CNI and plugin chains are supported;
multiple pools/plugins or CNI args overrides fail as ambiguous. Validation also
checks the matching pod network-selection entry by NAD namespace/name and
requested interface. Nonempty per-pod cni-args on NV-IPAM attachments are
unsupported and fail before pool lookup; malformed or ambiguous selections
also fail gateway measurement. Args on unrelated attachments do not override
the selected gateway. The runner uses
runtime interface routes when no IPPool gateway is set, for other IPAM providers,
and for explicit poolType: cidrpool. CIDRPool per-node configured allocations
are not separately resolved by this check. It never guesses a .1 address or uses
the management default route. A missing, malformed, denied or ambiguous gateway
lookup fails measurement. When neither configuration nor interface routes has
a gateway, the row is explicitly SKIPPED, with zero contribution to passed
or failed counts. This applicability assessment can exit zero when gateway is
the only selected family; zero endpoints and missing coverage for other selected
families still fail connectivity-only acceptance.
A configured gateway requires a positive, source-IP-bound ICMP echo after route
qualification selects the intended interface. IPv6 link-local gateways are
scoped to that interface. No echo fails gateway ICMP reachability; it does not
prove that ARP/ND resolution failed because a gateway may filter ICMP. Other
selected families continue after a gateway failure. Text reports gateway endpoints and status. JSON, HTML and JUnit retain
gateway configuration, route and probe evidence. In HTML, gateway rows show a
compact result; expand the row using Evidence for the full error, commands
and output. Skipped rows remain skipped
in JUnit. The automatic deadline includes a shared 10-second NAD/IPPool lookup limit,
a 10-second runtime discovery limit when needed, a 10-second source-route limit
and a 5-second echo limit per endpoint. Direct Go callers must supply
connectivity.Options.IPAMNamespace for NV-IPAM IPPool attachments.
Before every same-rail and cross-rail ICMP probe in quick, full,
and strict modes, validation runs ip route get <dst> from <src>. A route
that does not select the requested source interface means that rail pair is not
connected; validation does not force the packet onto that interface. When the
source rail is selected, ICMP uses ping -I <src-ip> so source-based policy
rules participate naturally. Routing diagnostics compare the selected device
with the profile expectation: the source interface for source-based routing,
or the destination interface for destination-based routing.
NIC configuration success requires ConfigUpdateInProgress with reason: UpdateSuccessful and status: "False" on every matched device; a contradictory True/Unknown status remains IN-PROGRESS. This applies to both deployment waiting and standalone validation, including standard-profile templates, which are enabled by default.
Manifest-state checks for NicConfigurationTemplate and NicFirmwareTemplate use the operator-populated status.nicDevices list as the matched device set. An empty list, a list that does not yet reflect the current node, NIC type, PCI-address, serial-number, and part-number selectors, a missing named NicDevice, a device spec that does not yet reflect the current template payload, or a relevant device condition with a stale observedGeneration remains IN-PROGRESS. NicConfigurationTemplate considers FirmwareUpdateInProgress relevant only when the matched device has spec.firmware; a stale firmware condition cannot block a configuration-only deployment. Unrelated discovered devices are used only to verify selector freshness; their configuration and firmware state is ignored.
SR-IOV node-policy and NV-IPAM IPPool validation compares the live selector with the intended workers from the bundle's source-group annotation and effective configuration. Live annotation edits cannot redefine that membership. Missing or extra nodes are target-scope errors, even when controllers report ready. Missing live resources remain missing; denied reads remain inspection failures. Preset or legacy configurations with no worker inventory retain a visible target scope unverified warning alongside ordinary readiness. An intended worker missing a required PF still fails; empty SR-IOV interface status remains in progress. See hardware target scope for legacy compatibility and configuration requirements.
Preflight uses the same checks as deployment: Helm chart version, generated Helm values, component versions, and stray CRs from the enumerated networking kinds and scopes, without an l8k ownership requirement. SR-IOV pool configs, node policies, and OVS networks labeled with spectrumx.nvidia.com/owner-name are controller-owned outputs of SpectrumXRailPoolConfig, so they are not reported as strays. Validation never remediates drift.
The Helm chart/version checks are omitted when Helm management is disabled; component and stray-resource checks remain active.
Validation Modes¶
| Mode | Coverage | Cross-rail gating |
|---|---|---|
quick |
Same-rail all nodes plus one cross-rail canary per rail pair. | Non-gating. |
full |
Every source rail x destination rail pair. | Non-gating. |
strict |
Full matrix. | Gated by profile.routing. Source-based routing must pass; destination-based routing must stay isolated. |
l8k validate --validation-mode quick
l8k validate --validation-mode full
l8k validate --validation-mode strict
Check Selection¶
Fresh discovery writes:
validation:
gpuDirect:
enabled: false
gpuResourceType: nvidia.com/gpu
connectivity: true
mode: strict
checks:
- gateway
- icmp
- rping
- ib_write_bw
rdma:
rpingIterations: 5
ibWriteSize: 65536
ibWriteMinBandwidthGbps: 100
gpuDirect.enabled is never omitted. Discovery sets it to true only when
every worker in every discovered group can satisfy its render bucket's
topology-derived gpuResourceType request; otherwise it writes false. You may
change the value before generation. GPUDirect runs as a distinct result family whenever
it is enabled and ib_write_bw is selected.
The generated validation DaemonSet selects its full-runtime DOCA image from
the Network Operator release catalog and copies
networkOperator.imagePullSecrets into the Pod spec. Create those Secrets in
every network namespace used by validation. Only the DOCA container requests
the configured GPU resource. When GPUDirect is enabled, that container also
sets LD_LIBRARY_PATH to prefer the host-injected NVIDIA driver libraries and
exclude the CUDA compatibility-library directory. The override prevents a
bundled libcuda from taking precedence over the host driver and is omitted
when GPUDirect is disabled.
For each test, Launch Kit maps the source rail and destination rail to their
own connectedGPU value from discovery or the selected topology preset. It
then passes --use_cuda=<source-index> --use_cuda_dmabuf to the client and
--use_cuda=<destination-index> --use_cuda_dmabuf to the server. Missing or
ambiguous mappings fail; GPU 0 is never assumed. Text, JSON, and HTML output
keep DMA-BUF results separate and include indices, PCI addresses when known,
bandwidth, threshold, and errors.
Override per run:
l8k validate \
--validation-checks icmp,rping \
--connectivity-timeout 10m \
--rdma-rping-iterations 20 \
--rdma-ib-write-size 65536 \
--rdma-ib-write-min-bandwidth-gbps 100
Connectivity Timeout¶
By default, --connectivity-timeout=0 selects an automatic timeout. Launch Kit
first applies and discovers the generated validation workload under a bounded
setup allowance. Once the matrix is known, it calculates the total budget from
the selected tests, their individual command limits, ordered pod-pair batches,
cleanup allowances, and a safety margin. The log reports the calculated total
before connectivity test execution starts:
Set a positive duration to replace the automatic budget with an explicit hard deadline for connectivity workload setup and execution:
The explicit deadline is useful for fitting validation into an external maintenance or CI window. Test DaemonSet and RDMA-process cleanup use short, independent best-effort contexts so cleanup is still attempted after either an automatic or user-supplied deadline expires.
Debug and Trace Logs¶
Use debug logging for structured progress without command output:
Debug includes static-check durations, the complete endpoint inventory, matrix and timeout planning, source-route executions and cache hits, stage and RDMA-batch progress, pass/fail counts, cleanup, report writes, elapsed time, and remaining time.
Use trace when a connectivity failure needs command evidence:
Trace adds bounded route and ICMP commands, RDMA batch commands, per-test
client stdout/stderr, and RDMA server logs. Failed RDMA server logs are
collected before the temporary files and test workload are removed. Each log
field is capped to keep command output bounded. Add --keep only when the
validation pods must remain available for follow-up inspection.
Disable only the connectivity stage:
Connectivity-Only Validation¶
When a custom operational workload replaces the generated example, follow the application fixture recipe to retain matching test DaemonSets for a later full validation run.
The test DaemonSet does not need to be generated by Launch Kit. Put one or
more user-provided DaemonSets in *example*.yaml files under a directory and
pass that directory through the existing --deployment-files option:
l8k validate \
--deployment-files ./connectivity-test \
--user-config ./cluster-config.yaml \
--kubeconfig "$KUBECONFIG"
When that directory contains only example workloads, Launch Kit runs only the
connectivity stage. If it also contains values.yaml or any non-example YAML
manifest, Launch Kit runs the complete deployment-validation pipeline and then
connectivity. A cluster-config.yaml stored in the same directory is treated
as configuration, not as a deployment manifest.
Validation checks every artifact file before connecting to the cluster.
Malformed or nameless resources fail with a file and document number, even
when they are example support resources. values.yaml is the only Helm values
filename; values.yml and case variants need renaming. Desired manifests
stay fixed during --wait and connectivity checks, while live cluster state
continues to refresh. An explicit HTML report path still receives a partial
report when artifact loading fails.
Connectivity requires these explicit decisions even though the remaining validation settings have defaults:
The test workload must be an apps/v1 DaemonSet with metadata.name and
metadata.namespace. Selected checks require the netshoot route/ICMP helper;
rping, ib_write_bw, and GPUDirect also require the RDMA tools container.
Referenced namespaces, network attachments, credentials, and network resources
must already exist. Launch Kit applies the DaemonSet as provided and does not
inject or generate containers.
When validation.gpuDirect.enabled is true and ib_write_bw is selected,
clusterConfig must also identify every worker group and its east-west PFs.
Each group requires workerNodes; each east-west PF requires a non-negative
rail and connectedGPU: GPU<N>. This topology lets Launch Kit select the
source and destination GPU independently for every tested rail.
A connectivity-only invocation succeeds only when every selected check family
produces at least one gating test and all gating tests pass, except explicit
no-configured-gateway applicability skips. Non-gating
cross-rail observations do not satisfy this requirement. A skipped,
empty, or incomplete matrix exits with status 4 because there is no other
validation stage providing acceptance evidence.
When Validation Does Not Pass¶
Keep the test DaemonSet:
Wait for in-progress manifests:
Use the automation guide to capture JSON output and the CLI exit status without discarding diagnostics.
Continue with Troubleshooting to investigate the failed stage while preserving the report and test resources as evidence.