fix(native-eval): rebootstrap replacement leases

[vincentkoc]

What does this PR do?

Rehydrates replacement native-eval leases before dispatch.

Why?

The controller previously treated an old bootstrap timestamp as proof that a
new lease contained the runner. Recovery could then fail before the harness
started, producing a false benchmark failure.

Fixes #55

Changes

• record the exact bootstrapped_lease_id
• reuse bootstrap state only when it matches the active lease
• add a regression test for recovery with an old timestamp and replacement lease

Live proof

Recovered a legacy OpenClaw run whose manifest had an old bootstrap timestamp
but no bootstrapped_lease_id.

• rebound the run to its active replacement lease
• performed the full native-runner bootstrap on that lease
• persisted the exact active lease as bootstrapped_lease_id
• dispatched the OpenClaw harness successfully
• exported a verified final artifact with 4/4 completed results
• run exit code: 0

This is the exact recovery state the regression test covers; the controller no
longer trusts timestamp-only bootstrap state.

Tests

• 40 focused fleet tests pass
• Python 3.11 and Python 3.12 CI pass
• Ruff passes on the touched files
• fresh Codex autoreview reports no actionable findings
• live replacement-lease recovery completes a four-task OpenClaw run

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 3, 2026, 12:09 AM ET / 04:09 UTC.

ClawSweeper review

What this changes

The branch records the lease ID used for native-evaluation bootstrap and rehydrates a recovery run when its active lease differs from that recorded lease.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep this PR open for maintainer review. The patch is a focused, source-backed fix for the timestamp-only recovery bug and adds a direct regression test, but it deliberately makes a missing remote run trigger full lease hydration, which replaces the remote runner and task directories; that recovery policy needs explicit maintainer confirmation before merge.

Priority: P2
Reviewed head: 8949dc8328e9b86a0cf1f80bcf3a976c67c4837a
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused, regression-tested repair with no discrete correctness finding; merge readiness is limited only by confirmation of the destructive recovery policy.
Proof confidence	🌊 off-meta tidepool	Not applicable: The author is a repository member, so the external-contributor proof gate does not apply; the PR body nevertheless describes a successful live replacement-lease recovery with a four-result artifact and exit code 0.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The author is a repository member, so the external-contributor proof gate does not apply; the PR body nevertheless describes a successful live replacement-lease recovery with a four-result artifact and exit code 0.
Evidence reviewed	6 items	Current-main defect path: Current main decides whether to hydrate a remotely missing run solely from bootstrapped_at_utc, so a timestamp retained from an expired lease suppresses hydration on a replacement lease. PR repair: The PR reloads the manifest after lease acquisition and hydrates unless bootstrapped_lease_id equals the active lease ID; hydration persists that identity for later recovery. Focused regression coverage: The added test creates a recovery_required manifest with an old bootstrap timestamp and no stored lease identity, then asserts hydration, dispatch, completion, and persistence of the replacement lease ID.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench's native-evaluation fleet controller restores benchmark runs on disposable Crabbox leases. It reads a persisted run manifest, obtains or replaces a lease, probes remote run state, then hydrates the remote runner and dispatches the benchmark when needed.

flowchart LR
  A[Persisted run manifest] --> B[Lease recovery]
  B --> C[Remote run probe]
  C -->|missing run state| D[Lease hydration]
  D --> E[Remote runner and tasks]
  E --> F[Benchmark dispatch]
  C -->|running or done| F
  F --> G[Verified result export]

Loading

Decision needed

Question	Recommendation
Should a recovery_required run whose active replacement lease has no matching bootstrapped_lease_id always perform full hydration before dispatch when the remote run probe reports missing?	Confirm full rehydration policy: Approve the stated recovery contract: a missing remote run on an unmatched replacement lease is rebuilt before dispatch.

Why: The code is mechanically narrow and matches the reported failure, but full hydration intentionally replaces remote runner and task directories and updates provider environment state; only maintainers can confirm that this is the desired recovery contract.

Before merge

• Resolve merge risk (P1) - Merging changes recovery from timestamp-based bootstrap reuse to full hydration whenever the active lease identity is absent or differs; that hydration replaces remote runner and public-task directories and refreshes the provider environment before dispatch.
• Complete next step (P2) - A maintainer must confirm the intended replacement-lease hydration contract; there is no narrow mechanical repair remaining for an automated worker.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	2 files affected; production +3/-1, tests +29	The change is tightly constrained to the recovery predicate, persisted lease identity, and one direct regression test.

Merge-risk options

Maintainer options:

1. Confirm replacement-lease rebuild semantics (recommended)
   Merge after a maintainer confirms that a missing run state on an unmatched lease may rebuild that lease's runner, tasks, and provider environment before dispatch.
2. Pause for a preservation contract
   Keep the PR open if recovery must retain any remote installation state beyond the run-state probe, then add that preflight and focused coverage.

Technical review

Best possible solution:

Confirm that a recovered run with no remote run state should always rebuild the active lease, then merge this narrow identity-based guard and regression test; if that overwrite policy is not intended, define a narrower remote-state preflight before changing recovery behavior.

Do we have a high-confidence way to reproduce the issue?

Yes, from source and the added focused regression: a recovery_required manifest retaining bootstrapped_at_utc but lacking a stored active lease identity reaches dispatch without hydration on current main. This read-only review did not execute the test or a live lease recovery.

Is this the best way to solve the issue?

Unclear pending maintainer intent. Tracking bootstrap reuse by exact lease identity is the narrowest repair for the established defect, but the resulting full-hydration policy must be confirmed because hydration replaces remote lease contents.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P2: This is a bounded native-evaluation recovery failure that can create a false benchmark failure but does not affect the general runtime.
• merge-risk: 🚨 compatibility: The PR changes upgrade and recovery behavior by replacing timestamp-only bootstrap reuse with lease-identity-aware full hydration.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The author is a repository member, so the external-contributor proof gate does not apply; the PR body nevertheless describes a successful live replacement-lease recovery with a four-result artifact and exit code 0.

Evidence

What I checked:

• Current-main defect path: Current main decides whether to hydrate a remotely missing run solely from bootstrapped_at_utc, so a timestamp retained from an expired lease suppresses hydration on a replacement lease. (scripts/native_eval/fleet.py:566, 884dd1bb5511)
• PR repair: The PR reloads the manifest after lease acquisition and hydrates unless bootstrapped_lease_id equals the active lease ID; hydration persists that identity for later recovery. (scripts/native_eval/fleet.py:566, 8949dc8328e9)
• Focused regression coverage: The added test creates a recovery_required manifest with an old bootstrap timestamp and no stored lease identity, then asserts hydration, dispatch, completion, and persistence of the replacement lease ID. (tests/test_native_eval_fleet.py:1276, 8949dc8328e9)
• Recovery-policy risk: Hydration validates and installs runner/task archives and the provider environment, then removes and replaces the remote runner and public-task directories. The branch invokes this when the remote run state is missing and the lease identity does not match. (scripts/native_eval/fleet.py:861, 884dd1bb5511)
• Feature provenance: The native-evaluation controller and its stabilization follow-up both appear in the available file history under Vincent Koc, establishing a direct ownership trail for the recovery path. (scripts/native_eval/fleet.py:541, 69f75c6629c4)
• Current-main and release check: The checked-out default branch remains at 884dd1b and does not contain this PR's lease-ID logic; no local release tag contains the current main commit, so there is no release provenance for the proposed fix. (scripts/native_eval/fleet.py:566, 884dd1bb5511)

Likely related people:

• Vincent Koc: Available history attributes the initial native matrix runner, its stabilization follow-up, and this focused recovery repair to Vincent Koc. (role: feature owner and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 8949dc8328e9; files: scripts/native_eval/fleet.py, tests/test_native_eval_fleet.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Obtain explicit maintainer confirmation that full hydration is the intended response to missing remote run state on a replacement lease.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (20 earlier review cycles; latest 8 shown)

• reviewed 2026-08-01T19:16:49.517Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T22:59:14.951Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T13:17:00.915Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T19:14:34.796Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T20:37:22.956Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T22:18:20.095Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T00:19:32.496Z sha 8949dc8 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T01:48:36.800Z sha 8949dc8 :: needs maintainer review before merge. :: none