Rebootstrap native evals when recovery replaces a lease

[vincentkoc]

What happened

When a native fleet run recovered onto a replacement Crabbox lease, the run
retained bootstrapped_at_utc from the expired lease. The controller therefore
skipped hydration on the fresh machine and attempted to dispatch a runner path
that did not exist.

Expected behaviour

Bootstrap reuse must be tied to the exact lease identity. A replacement lease
must be hydrated before dispatch even when the run manifest contains an older
bootstrap timestamp.

Steps to reproduce

# Resume a recovery_required run with:
# - requested_lease_slug present
# - no active stored lease
# - bootstrapped_at_utc retained from the previous lease
python -m scripts.native_eval.fleet ... 

Relevant output

dispatch openclaw-gpt55-high-exploratory-4-r1-20260729 failed with exit 1:
fleet-dispatch: line 32: kill: (...) - No such process

/opt/shellbench-native/runner/scripts/native_eval/remote_run.sh:
No such file or directory

Environment

• Python version: 3.12
• OS: macOS controller, Ubuntu AWS lease
• ClawBench version / commit: 98301b2405070630ec61a331b11937e886688f61

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 3, 2026, 12:09 AM ET / 04:09 UTC.

Summary
This issue remains necessary: current main still treats any retained bootstrap timestamp as sufficient after recovery, even when recovery selects a fresh lease. The explicitly linked open pull request is a narrow, proof-backed candidate fix, and this MEMBER-authored issue should remain open until that work is reviewed and merged.

Reproducibility: yes. Current main's recovery path selects a lease and dispatches without hydration whenever any old bootstrapped_at_utc exists; replacing the stored lease while retaining that field reaches the faulty decision. This is source-reproducible, although this read-only review did not run a live Crabbox recovery.

Next step
An explicit open pull request already owns the focused repair, and this MEMBER-authored issue must remain open until that candidate is reviewed and merged.

Review details

Best possible solution:

Persist the lease identity that was hydrated, require it to match the active lease before bootstrap reuse, and land focused regression coverage for both legacy manifests and same-lease recovery.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main's recovery path selects a lease and dispatches without hydration whenever any old bootstrapped_at_utc exists; replacing the stored lease while retaining that field reaches the faulty decision. This is source-reproducible, although this read-only review did not run a live Crabbox recovery.

Is this the best way to solve the issue?

Yes. Lease-identity-aware reuse is the narrowest repair: it retains efficient reuse on the machine actually hydrated while making replacement machines bootstrap before dispatch. The open candidate at #58 follows that boundary.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• Until the linked fix lands, a recovered run that retains an old bootstrapped_at_utc can dispatch against a replacement machine that has not received the remote runner.
• The linked fix changes persisted run-manifest state, so review should confirm that legacy timestamp-only manifests rebootstrap safely while same-lease resumes still avoid unnecessary hydration.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P2: A replacement lease can cause a bounded native benchmark recovery failure, but the report does not establish a broad user-facing outage.
• impact:other: The affected surface is remote benchmark-run orchestration and readiness, which does not fit the more specific owned impact categories.

Evidence reviewed

What I checked:

• Current timestamp-only reuse check: When the selected lease has no remote run state, current main hydrates it only if bootstrapped_at_utc is absent; it does not compare bootstrap state with the selected lease identity before dispatch. (scripts/native_eval/fleet.py:561, 884dd1bb5511)
• Recovery can replace the manifest lease: If no stored lease remains, recovery obtains or warms the requested lease and overwrites the manifest lease while leaving unrelated persisted fields intact, which allows an older bootstrap timestamp to survive onto the replacement lease. (scripts/native_eval/fleet.py:644, 884dd1bb5511)
• Current bootstrap persistence lacks lease identity: Hydration records only bootstrapped_at_utc; current main does not persist a lease identifier alongside that marker. (scripts/native_eval/fleet.py:893, 884dd1bb5511)
• Existing recovery coverage is adjacent, not replacement-specific: The current recovery test covers resuming an already-running remote job on its recorded lease, but not an old bootstrap marker paired with a replacement lease. (tests/test_native_eval_fleet.py:1238, 884dd1bb5511)
• Feature-history ownership: The native matrix runner and its fleet controller were introduced in the referenced commit; subsequent adjacent native-eval fixes and documentation changes are by the same contributor. (scripts/native_eval/fleet.py:541, 69f75c6629c4)
• Open implementation candidate: The provided issue timeline and linked pull-request context identify an open, cleanly mergeable candidate that records bootstrapped_lease_id, reuses bootstrap only for the matching lease, adds the matching regression test, and reports a successful live replacement-lease recovery. (8949dc8328e9)

Likely related people:

• vincentkoc: Git history attributes the native matrix runner, an adjacent native-harness stabilization fix, the campaign runbook, and current main's latest native-eval trace repair to Vincent Koc; the linked candidate fix is also in this established area. (role: native-eval feature introducer and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 569b5c39c783; files: scripts/native_eval/fleet.py, tests/test_native_eval_fleet.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.