fix(native-eval): reject invalid harness runs

[vincentkoc]

What does this PR do?

Records structured native-eval execution outcomes and rejects runs where every
trial failed at the harness, infrastructure, or verifier layer.

Why?

Verifier rewards are useful diagnostics, but they are not proof that the
benchmark harness executed correctly. Fully invalid runs were being presented
as completed score data, and fleet recovery could infer success without an
archived exit status.

Fixes #63

Changes

• classify clean, agent, harness, infrastructure, and verifier outcomes
• preserve per-trial rewards while recording run-level execution acceptance
• exclude rejected runs from score-eligible aggregates
• infer historical OpenClaw exit 70/71 outcomes correctly
• remove fleet success inference based only on result count and checkpoint text

Tests

• python -m pytest -q passes on Blacksmith Testbox
• python -m ruff check clawbench app.py scripts tests passes on Blacksmith Testbox

Focused proof: 117 tests passed; Ruff clean. Autoreview reported no actionable
findings.

[clawsweeper]

Codex review: found issues before merge. Reviewed August 3, 2026, 12:09 AM ET / 04:09 UTC.

ClawSweeper review

What this changes

The PR records native-evaluation execution outcomes, excludes fully invalid harness runs from score aggregates, and requires an archived exit status during fleet recovery.

Merge readiness

⚠️ Ready for maintainer review - 5 items remain

Keep open. This member-authored PR addresses the linked native-evaluation bug, but its unchanged fleet-recovery change turns verified legacy exports without an archived exit_status from completed runs into failed runs; a maintainer needs to choose a bounded compatibility path before merge.

Priority: P1
Reviewed head: 9f4490e8e4370cbc752404a1becce42d5a6a69f8
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The invalid-run classification work is focused and tested, but the unresolved P1 legacy-recovery regression prevents a higher readiness rating.
Proof confidence	🌊 off-meta tidepool	Not applicable: The PR author is a repository member, so the external-contributor real-behavior-proof gate does not apply; the PR body’s test claims remain supplemental validation.
Patch quality	🦐 gold shrimp (3/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The PR author is a repository member, so the external-contributor real-behavior-proof gate does not apply; the PR body’s test claims remain supplemental validation.
Evidence reviewed	6 items	Current main preserves a legacy recovery path: Current main accepts a verified full-coverage exported archive with a final checkpoint as exit code zero when older archives lack exit_status. The PR removes that compatibility path: The proposed branch removes both the final-checkpoint helper and the full-coverage recovery fallback, leaving only an explicit archived exit status to complete recovery. The regression is asserted by the changed test: The replacement test expects a legacy-style verified archive with full result coverage and a final checkpoint, but no archived exit status, to end as failed with an unknown exit.
Findings	1 actionable finding	[P1] Preserve recovery for legacy exported runs
Security	None	None.

How this fits together

Native evaluation runs agent trials, writes result archives, then aggregates scores and lets the fleet controller recover exported remote campaigns. This PR changes the boundary between diagnostic trial results, score eligibility, and whether an archived campaign is considered successfully recoverable.

flowchart LR
  A[Native harness trials] --> B[Trial execution outcomes]
  B --> C[Run acceptance decision]
  C --> D[Score aggregation]
  A --> E[Exported run archive]
  E --> F[Fleet recovery]
  F --> G[Completed or failed campaign]
  D --> H[Benchmark reporting]

Loading

Decision needed

Question	Recommendation
Should verified legacy exported runs that lack exit_status retain the bounded final-checkpoint recovery path, or should they be deliberately failed and rerun under the new strict evidence policy?	Preserve or migrate legacy recovery: Keep the existing bounded fallback for pre-change archives or add a migration/backfill, while requiring explicit exit status for newly created runs.

Why: Both choices are plausible, but they impose different guarantees for historical benchmark archives and cannot be resolved mechanically without maintainer intent about upgrade compatibility and evidence standards.

Before merge

• Preserve recovery for legacy exported runs (P1) - The removal of the fallback leaves a verified full-coverage archive without exit_status unable to complete recovery; the revised test now asserts that it fails. Existing archives can legitimately have that older shape, so this changes upgrade behavior and may force reruns. Retain a bounded legacy path or add a migration/backfill before enforcing the strict rule.
• Resolve merge risk (P1) - Merging as written changes upgrade recovery for verified legacy archives lacking exit_status: they become failed runs after cleanup, which can force campaign reruns or leave historical exports incorrectly marked failed.
• Resolve merge risk (P1) - The strict recovery policy is deliberate data-integrity behavior, but the repository needs an explicit compatibility decision or migration rather than silently changing legacy recovery outcomes.
• Complete next step (P2) - A maintainer must choose the upgrade contract for legacy exported archives before an automated repair can safely preserve or replace the removed recovery fallback.

Findings

• [P1] Preserve recovery for legacy exported runs — scripts/native_eval/fleet.py:1267-1274

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	7 files affected; production +186/-28, tests +93/-7	Most of the change is a new execution-outcome model, while the merge blocker is concentrated in the fleet recovery compatibility deletion.

Merge-risk options

Maintainer options:

1. Restore a bounded legacy path (recommended)
   Retain the full-coverage plus final-checkpoint fallback only for archives created before the explicit exit-status contract, and test both legacy and new archives.
2. Accept forced legacy reruns
   Land the strict rule only after maintainers explicitly accept that qualifying legacy archives will be marked failed and require reruns.
3. Pause the strict recovery change
   Keep the score-acceptance work separate and pause the fleet fallback removal until the archive-compatibility policy is settled.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Preserve a narrowly scoped recovery path or migration for legacy exported archives without `exit_status`, then add focused upgrade-compatibility tests.

Technical review

Best possible solution:

Preserve a narrowly validated legacy recovery path, or migrate/backfill legacy archives before requiring explicit exit-status evidence for all future and historical fleet recovery.

Do we have a high-confidence way to reproduce the issue?

Yes, from source and the focused changed test: a verified full-coverage archive with a final checkpoint but no archived exit_status completes on current main and is marked failed by this branch. The read-only review did not execute the test suite.

Is this the best way to solve the issue?

No. Rejecting invalid harness-only runs is a targeted fix, but removing the existing legacy recovery path without a bounded compatibility rule or migration changes established recovery behavior for old exports.

Full review comments:

• [P1] Preserve recovery for legacy exported runs — scripts/native_eval/fleet.py:1267-1274
  The removal of the fallback leaves a verified full-coverage archive without exit_status unable to complete recovery; the revised test now asserts that it fails. Existing archives can legitimately have that older shape, so this changes upgrade behavior and may force reruns. Retain a bounded legacy path or add a migration/backfill before enforcing the strict rule.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P1: The patch can change real native-evaluation campaign recovery from completed to failed for existing verified archives, requiring reruns or operator intervention.
• merge-risk: 🚨 compatibility: It removes recovery behavior that current main supports for legacy archives without an archived exit status.
• merge-risk: 🚨 availability: Legacy recovered campaigns may fail after cleanup and need to be rerun, interrupting benchmark completion workflows.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: The PR author is a repository member, so the external-contributor real-behavior-proof gate does not apply; the PR body’s test claims remain supplemental validation.

Evidence

What I checked:

• Current main preserves a legacy recovery path: Current main accepts a verified full-coverage exported archive with a final checkpoint as exit code zero when older archives lack exit_status. (scripts/native_eval/fleet.py:1285, 884dd1bb5511)
• The PR removes that compatibility path: The proposed branch removes both the final-checkpoint helper and the full-coverage recovery fallback, leaving only an explicit archived exit status to complete recovery. (scripts/native_eval/fleet.py:1267, 9f4490e8e437)
• The regression is asserted by the changed test: The replacement test expects a legacy-style verified archive with full result coverage and a final checkpoint, but no archived exit status, to end as failed with an unknown exit. (tests/test_native_eval_fleet.py:1312, 9f4490e8e437)
• Prior blocker remains at the same head: The latest completed ClawSweeper review identified the same legacy-export recovery blocker, and the current head is still 9f4490e8e4370cbc752404a1becce42d5a6a69f8. (scripts/native_eval/fleet.py:1267, 9f4490e8e437)
• Relevant native-evaluation history: The current main branch includes recent native-evaluation work by Vincent Koc, including the campaign-runbook change that provides the closest available provenance for this recovery surface. (scripts/native_eval/fleet.py:1249, 569b5c39c783)
• Current main does not contain the proposed acceptance model: Current main has the older aggregate and fleet behavior; it does not contain execution_acceptance, run_accepted, or execution_outcome, and the PR head is not an ancestor of main. (scripts/native_eval/aggregate.py:1, 884dd1bb5511)

Likely related people:

• vincentkoc: Vincent Koc authored the current main native-evaluation changes closest to fleet recovery and aggregation, in addition to this proposed fix. (role: recent native-evaluation contributor; confidence: high; commits: 884dd1bb5511, 569b5c39c783; files: scripts/native_eval/fleet.py, scripts/native_eval/aggregate.py, scripts/native_eval/runtime.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Choose and implement a bounded legacy recovery or migration path.
• Add focused coverage for both pre-change archives without exit_status and newly created strict archives.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (18 earlier review cycles; latest 8 shown)

• reviewed 2026-08-01T17:56:59.496Z sha 9f4490e :: found issues before merge. :: [P1] Preserve or migrate legacy exported runs
• reviewed 2026-08-01T19:16:54.084Z sha 9f4490e :: found issues before merge. :: [P1] Preserve or migrate legacy exported runs
• reviewed 2026-08-01T22:59:18.642Z sha 9f4490e :: found issues before merge. :: [P1] Preserve recovery for legacy exported runs
• reviewed 2026-08-02T13:16:46.448Z sha 9f4490e :: needs changes before merge. :: [P1] Preserve recovery for legacy exported runs
• reviewed 2026-08-02T19:15:22.778Z sha 9f4490e :: found issues before merge. :: [P1] Preserve recovery for legacy exported runs
• reviewed 2026-08-02T20:37:31.213Z sha 9f4490e :: needs changes before merge. :: [P1] Preserve recovery for legacy exported runs
• reviewed 2026-08-02T22:14:45.112Z sha 9f4490e :: found issues before merge. :: [P1] Preserve recovery for legacy exported runs
• reviewed 2026-08-03T01:48:46.759Z sha 9f4490e :: found issues before merge. :: [P1] Preserve recovery for legacy exported runs