Skip to content

feat: add force-restart verification support for GB200s - #4376

Merged
krish-nvidia merged 2 commits into
NVIDIA:mainfrom
krish-nvidia:gb200-force-restart-verification
Jul 31, 2026
Merged

feat: add force-restart verification support for GB200s#4376
krish-nvidia merged 2 commits into
NVIDIA:mainfrom
krish-nvidia:gb200-force-restart-verification

Conversation

@krish-nvidia

Copy link
Copy Markdown
Contributor

This PR adds force-restart verification support for GB200s by looking for the following BMC log entry immediately after a force-restart is issued (manually verified):

    {
      "@odata.id": "/redfish/v1/Systems/System_0/LogServices/EventLog/Entries/3890",
      "@odata.type": "#LogEntry.v1_15_0.LogEntry",
      "Created": "2026-07-30T14:44:56+00:00",
      "EntryType": "Event",
      "Id": "3890",
      "Message": "The property ResetType was assigned the value 'ForceWarmReboot' due to modification by the service.",
      "MessageArgs": [
        "ResetType",
        "ForceWarmReboot"
      ],
      "MessageId": "Base.1.15.PropertyValueModified",
      "Name": "System Event Log Entry",
      "Resolution": "None.",
      "Resolved": false,
      "Severity": "OK"
    },

It also bumps libredfish to v0.46.1 which brings in the following changes:

Related issues

NVBug 6520998

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

Additional Notes

@krish-nvidia
krish-nvidia requested a review from poroh July 30, 2026 15:27
@krish-nvidia krish-nvidia self-assigned this Jul 30, 2026
@krish-nvidia
krish-nvidia requested a review from a team as a code owner July 30, 2026 15:27
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cf88a06f-24ed-430f-ab04-6f81310c4cd9

📥 Commits

Reviewing files that changed from the base of the PR and between 5af969e and 7894254.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • Cargo.toml
  • crates/bmc-mock/src/redfish/computer_system.rs
  • crates/machine-controller/src/handler.rs
🚧 Files skipped from review as they are similar to previous changes (3)
  • Cargo.toml
  • crates/machine-controller/src/handler.rs
  • crates/bmc-mock/src/redfish/computer_system.rs

Summary by CodeRabbit

  • Bug Fixes
    • Improved restart detection for GB200 systems by recognizing service-initiated warm reboot events in event logs.
  • New Features
    • Enhanced the Redfish mock to support per-boot-option settings overrides, including recording and applying PATCH updates per boot option.
  • Chores
    • Updated the underlying Redfish library to the latest tagged release.

Walkthrough

The workspace updates libredfish, restart-log verification recognizes GB200 ForceWarmReboot events, and the BMC mock persists and returns per-boot-option PATCH overrides.

Changes

GB200 restart verification and boot-option mock updates

Layer / File(s) Summary
Redfish dependency update
Cargo.toml
Updates the workspace libredfish dependency from tag v0.45.1 to v0.46.1.
GB200 restart log matching
crates/machine-controller/src/handler.rs
Adds the GB200 ResetType = 'ForceWarmReboot' message to restart verification.
Boot-option override flow
crates/bmc-mock/src/redfish/computer_system.rs
Adds PATCH routing, per-option JSON override storage, merge behavior, initialization, and GET responses that include stored overrides. Unknown systems or boot options return 404.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RedfishClient
  participant BMCMockRoute
  participant SingleSystemState
  RedfishClient->>BMCMockRoute: PATCH boot option settings
  BMCMockRoute->>SingleSystemState: Merge boot-option override
  SingleSystemState-->>BMCMockRoute: Return success or 404
  RedfishClient->>BMCMockRoute: GET boot option
  BMCMockRoute->>SingleSystemState: Render base option with overrides
  SingleSystemState-->>BMCMockRoute: Return overridden boot-option JSON
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 10.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: force-restart verification support for GB200 systems.
Description check ✅ Passed The description directly explains the GB200 verification logic, dependency update, related issue, and manual testing.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/machine-controller/src/handler.rs (1)

2299-2299: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Avoid a redundant exact-match entry.

The existing generic matcher already recognizes this message because ForceWarmReboot contains reboot after lowercasing. Remove this entry, or change the matching strategy and add a regression test if exact GB200 matching is intentionally required.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/machine-controller/src/handler.rs` at line 2299, Remove the redundant
exact-match message entry near the existing generic reboot matcher in the
relevant handler configuration. Preserve the generic case-insensitive matching
behavior for messages containing “reboot”; only retain a GB200-specific entry if
you also change the matching strategy and add regression coverage for exact
matching.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/machine-controller/src/handler.rs`:
- Line 2299: Remove the redundant exact-match message entry near the existing
generic reboot matcher in the relevant handler configuration. Preserve the
generic case-insensitive matching behavior for messages containing “reboot”;
only retain a GB200-specific entry if you also change the matching strategy and
add regression coverage for exact matching.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e3a58ccb-373a-47b2-a60c-274466be6c59

📥 Commits

Reviewing files that changed from the base of the PR and between 94c24d2 and 3443637.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (2)
  • Cargo.toml
  • crates/machine-controller/src/handler.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/bmc-mock/src/redfish/computer_system.rs (1)

308-332: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Avoid the redundant clone when merging boot-option overrides.

current.clone().patch(patch_request) clones the full stored override JSON right before overwriting it. Since current is a &mut serde_json::Value from Entry::or_insert_with, use serde_json::Value::take() to move the value out (leaving Null in its place) instead of cloning it.

♻️ Proposed fix
         let mut overrides = self.boot_option_overrides.lock().expect("mutex poisoned");
         let current = overrides
             .entry(option_id.to_string())
             .or_insert_with(|| json!({}));
-        *current = current.clone().patch(patch_request);
+        *current = current.take().patch(patch_request);
         true

As per coding guidelines, "Avoid needless .clone() calls; prefer borrowing, moving with into_iter, ordering struct fields to enable moves, or using Cow where values may be borrowed or owned."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/bmc-mock/src/redfish/computer_system.rs` around lines 308 - 332,
Update patch_boot_option to replace current.clone().patch(patch_request) with a
take-based move from the mutable entry, leaving Null in the entry before
assigning the patched value. Keep the existing entry initialization and boolean
return behavior unchanged.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/bmc-mock/src/redfish/computer_system.rs`:
- Around line 308-332: Update patch_boot_option to replace
current.clone().patch(patch_request) with a take-based move from the mutable
entry, leaving Null in the entry before assigning the patched value. Keep the
existing entry initialization and boolean return behavior unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ec3c9a34-76ef-4a57-b3c2-fa71efbb09f7

📥 Commits

Reviewing files that changed from the base of the PR and between 3443637 and 672e19b.

📒 Files selected for processing (1)
  • crates/bmc-mock/src/redfish/computer_system.rs

@github-actions

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (3)
crates/bmc-mock/src/redfish/computer_system.rs (3)

887-891: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Accept partial HPE settings payloads.

Json<HpeBootSettingsPatch> requires PersistentBootConfigOrder. If a client stages any other HPE boot property, axum rejects the request with 422 before the handler runs. The mock then diverges from the permissive behavior used elsewhere in crates/bmc-mock. Make the field optional and ignore the request when it is absent.

♻️ Proposed refactor
 struct HpeBootSettingsPatch {
-    persistent_boot_config_order: Vec<String>,
+    persistent_boot_config_order: Option<Vec<String>>,
 }
-    system_state.set_hpe_boot_order(request.persistent_boot_config_order);
+    if let Some(boot_order) = request.persistent_boot_config_order {
+        system_state.set_hpe_boot_order(boot_order);
+    }
     json!({}).into_ok_response()

Based on learnings: in the Redfish mock server used for test support, prefer permissive behavior over strict input validation for malformed or partial payloads.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/bmc-mock/src/redfish/computer_system.rs` around lines 887 - 891,
Update HpeBootSettingsPatch and patch_hpe_boot_settings so
PersistentBootConfigOrder is optional during JSON deserialization, allowing
partial HPE boot settings payloads to reach the handler. When the field is
absent, ignore the request while preserving existing behavior when it is
present.

Source: Learnings


1237-1303: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add coverage for the boot-option override round trip.

The test module covers only the HPE OEM ordering path. The primary change of this layer, patch_boot_option_settings plus the override-aware get_boot_option response, has no test. A short test that PATCHes a boot-option settings resource, then asserts the merged GET body, and asserts 404 for an unknown boot-option ID would lock in the new contract consumed by log_host_config in crates/machine-controller/src/handler.rs.

I can generate that test if you want.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/bmc-mock/src/redfish/computer_system.rs` around lines 1237 - 1303, The
tests module only covers HPE ordering; add coverage for the boot-option override
flow implemented by patch_boot_option_settings and get_boot_option. Create a
test that PATCHes a boot-option settings resource, verifies the subsequent GET
contains the merged override values, and verifies GET for an unknown boot-option
ID returns 404, preserving the contract used by log_host_config.

Source: Coding guidelines


358-369: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove the needless clone of the stored override.

current.clone() copies the whole persisted JSON object on every PATCH. serde_json::Value implements Default, so std::mem::take moves the value out in place and keeps the merge semantics identical.

Also note the lock style: the two new call sites use .expect("mutex poisoned"), while the adjacent methods in this file use .unwrap(). Align them for consistency.

♻️ Proposed refactor
-        let mut overrides = self.boot_option_overrides.lock().expect("mutex poisoned");
+        let mut overrides = self.boot_option_overrides.lock().unwrap();
         let current = overrides
             .entry(option_id.to_string())
             .or_insert_with(|| json!({}));
-        *current = current.clone().patch(patch_request);
+        *current = std::mem::take(current).patch(patch_request);
         true

As per coding guidelines: "Avoid needless .clone() calls; prefer borrowing, moving with into_iter, ordering struct fields to enable moves".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/bmc-mock/src/redfish/computer_system.rs` around lines 358 - 369,
Update patch_boot_option to replace current.clone() with std::mem::take(current)
before applying patch, preserving the existing merge behavior without cloning
the stored JSON value. Change the boot_option_overrides lock handling in this
method to use .unwrap() consistently with adjacent methods.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/bmc-mock/src/redfish/computer_system.rs`:
- Around line 887-891: Update HpeBootSettingsPatch and patch_hpe_boot_settings
so PersistentBootConfigOrder is optional during JSON deserialization, allowing
partial HPE boot settings payloads to reach the handler. When the field is
absent, ignore the request while preserving existing behavior when it is
present.
- Around line 1237-1303: The tests module only covers HPE ordering; add coverage
for the boot-option override flow implemented by patch_boot_option_settings and
get_boot_option. Create a test that PATCHes a boot-option settings resource,
verifies the subsequent GET contains the merged override values, and verifies
GET for an unknown boot-option ID returns 404, preserving the contract used by
log_host_config.
- Around line 358-369: Update patch_boot_option to replace current.clone() with
std::mem::take(current) before applying patch, preserving the existing merge
behavior without cloning the stored JSON value. Change the boot_option_overrides
lock handling in this method to use .unwrap() consistently with adjacent
methods.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0d7439f8-32cc-443f-8c81-e7ec5c1a0e86

📥 Commits

Reviewing files that changed from the base of the PR and between 672e19b and 5af969e.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • Cargo.toml
  • crates/bmc-mock/src/redfish/computer_system.rs
  • crates/machine-controller/src/handler.rs
🚧 Files skipped from review as they are similar to previous changes (2)
  • Cargo.toml
  • crates/machine-controller/src/handler.rs

@krish-nvidia
krish-nvidia force-pushed the gb200-force-restart-verification branch from 5af969e to 7894254 Compare July 31, 2026 17:45
@krish-nvidia
krish-nvidia merged commit 4c7d841 into NVIDIA:main Jul 31, 2026
64 checks passed
polarweasel pushed a commit to polarweasel/infra-controller that referenced this pull request Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants