Health probe IDs

This page provides a list of health probes provided by NCX Infra Controller (NICo), along with their IDs. Health reports will contain these IDs in the alerts section in case the associated check or validation has failed.

Machine validation health probe identifiers

`FailedValidationTest`

Indicates that a certain host validation test failed. The alert will carry details about which test failed.

`FailedValidationTestCompletion`

Indicates that the host validation test framework failed to complete scheduling all specified tests on the host.

SKU validation health probe identifiers

`SkuValidation`

An alert with this ID is placed on a host in case the SKU validation workflow failed. The alert will make the host un-allocatable by tenants.

`TenantReportedIssue`

Indicates that a tenant reported an issue with the host while releasing the bare metal instance. The host won't be available for other tenants until the alert is cleared.

`RequestRepair`

Indicates that a tenant reported an issue with the host while releasing the bare metal instance and that repair by an external framework is required.

Site Explorer health probe identifiers

`BmcExplorationFailure`

Indicates that the hosts BMC endpoint could not be scraped. This can happen if the BMC is not reachable, but also in case the BMC response to any API call is malformed.

`PoweredOff`

Indicates that the power status of a host as reported by the BMC is not on.

`SerialNumberMismatch`

Indicates that the serial number on a host does not match the serial number in the Expected Machine manifest.

Hardware/BMC health probe identifiers

carbide-hardware-health currently reports sensor-based hardware health with a single probe ID:

`BmcSensor`

Indicates that a BMC sensor reported a warning/critical/failure condition.

Details:

target is set to the BMC sensor ID (for example, a fan/temperature/power sensor name).
The alert message contains the entity type, reading, unit, and threshold ranges used for evaluation.
Classifications are documented in Health alert classifications, including Hardware, SensorWarning, SensorCritical, and SensorFailure.

message format:

<entity_type> '<sensor_id>': <status> - reading <value><unit> (<reading_type>), valid range: <range>, caution: <range>, critical: <range>

Example:

power_supply 'PSU0_OutputPower': Critical - reading 1320.00W (power), valid range: 0.0 to 1500.0, caution: 1200.0 to 1300.0, critical: 0.0 to 1310.0

`BgpPeeringTor`

Indicates that a BGP session with a top-of-rack (TOR) switch could not be established by a host/DPU.

`BgpPeeringRouteServer`

Indicates that a BGP session with the route server that is part of the part of the Carbide control plane could not be established by a host/DPU.

`BgpStats`

Indicates that BGP statistics could not be extacted by dpu-agent

`BgpDaemonEnabled`

Indicates that the BGP daemon (FRR) is not running on the DPU

`DhcpRelay`

Indicates issues regarding the start of the DHCP relay on the DPU

`DhcpServer`

Indicates issues regarding the start of the DHCP server on the DPU

`HeartbeatTimeout`

Indicates that there was no communication between dpu-agent and carbide-core for a certain amount of time. This condition usually implies that the DPU won't be able to apply any configuration changes.

`StaleAgentVersion`

Indicates that dpu-agent has not been updated to the newest version, even though the newest release had been available for a certain amount of time.

`ContainerExists`

Indicates that a container that was expected to run on the DPU is not running

`SupervisorctlStatus`

Indicates an issue with retrieving the list of running services

`ServiceRunning`

Indicates that an expected service on the DPU is not runnning

`PostConfigCheckWait`

The alert is placed on a host for a few seconds after a configuration change by dpu-agent in order to allow the configuration changes to "settle" before doing the health assessment. That avoids the host to move between states even though the new configuration might be problematic.

`RestrictedMode`

Indicates that the DPU is not running in restricted mode

`DpuDiskUtilizationCheck`

Indicates that the dpu-agent failed to check disk utilization

`DpuDiskUtilizationCritical`

Indicates that the dpu-agent disk utilization on the DPU is above a critical threshold

Other health probe identifiers

`MissingReport`

The alert indicates that no health report was received, where health report was expected. It is different from HeartbeatTimeout in the following sense

HeartbeatTimeout alerts can be emitted if data is available, but stale. MissingReport is only emitted if data has never been received.
MissingReport is mainly used on the NICo client side. It has no impact on state changes.

`MalformedReport`

An alert which can be generated if a HealthReport can not be parsed This alert is only be used the NICo client side if failing to render the health report is preferrable to failing the workflow.

`Maintenance`

The alert is used by site admins to mark hosts that are under maintenance - e.g. for CPU or memory replacements.

`HostUpdateInProgress`

Indicates that an update for host firmware was scheduled on the host

`IbCleanupPending`

Indicates that the host was released back to the admin pool without the system being able to fully clean up all port to partition key associations for all InfiniBand interfaces. This means the host might still be bound to a tenants partition. Once the IB subsystem can communicate with UFM and detects that the port is not bound to a partition anymore, the alert will automatically clear.

Keyboard shortcuts

NCX Infra Controller Documentation