Native eval accepts fully invalid harness runs

[vincentkoc]

What happened

Native eval runs can be reported as completed and included in aggregate scoring
when every trial failed at the harness, infrastructure, or verifier layer.
OpenClaw exit codes 70/71 are a concrete example: verifier rewards may still be
present, while the benchmark execution itself was invalid.

Fleet recovery also inferred a zero exit code from result-file count plus a
final checkpoint marker, even when the archived run exit status was unknown.

Expected behaviour

Preserve verifier rewards for diagnosis, but mark fully invalid runs as
rejected and exclude them from score-eligible aggregates. Fleet completion
must require an explicit archived run exit status.

Steps to reproduce

# Run a native OpenClaw campaign where every trial exits 71 after the verifier
# has produced reward files, then aggregate the exported run.
python -m scripts.native_eval.aggregate <jobs-root> <summary-dir>

Relevant output

NonZeroAgentExitCodeError: Agent exited with code 71

Before the fix, a full set of reward files could still produce a completed,
eligible run despite all trials ending with the harness-specific failure.

Environment

• Python version: 3.12
• OS: Linux
• ClawBench commit: 884dd1b

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 3, 2026, 5:04 AM ET / 09:04 UTC.

Summary
Keep this member-authored issue open: current main still lets NonZeroAgentExitCodeError fall into the score-eligible agent-exit path and still infers success during fleet recovery when an archived exit status is absent. The focused open repair at #64 explicitly owns both defects, so a second repair would duplicate active work. Likely related people: Vincent Koc is the high-confidence recent native-eval contributor.

Reproducibility: yes. Current source provides a high-confidence path: all trials with NonZeroAgentExitCodeError retain scorable agent-exit classification, and fleet recovery infers zero when coverage and a final checkpoint exist without an archived exit status.

Next step
The open linked pull request already owns the source-proven repair, and this member-authored issue must remain open until that implementation is reviewed and merged.

Review details

Best possible solution:

Review and land a focused repair that records explicit execution acceptance, rejects fully invalid runs from aggregate scoring, and treats a missing archived fleet exit status as unresolved rather than successful.

Do we have a high-confidence way to reproduce the issue?

Yes. Current source provides a high-confidence path: all trials with NonZeroAgentExitCodeError retain scorable agent-exit classification, and fleet recovery infers zero when coverage and a final checkpoint exist without an archived exit status.

Is this the best way to solve the issue?

Unclear. The linked PR addresses the two source-proven paths described here, but it remains open and requires normal PR-level compatibility and proof review before it can be accepted as the final solution.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• Landing the linked repair can change how historical native runs are classified and may mark previously accepted benchmark data ineligible; its compatibility and recovery behavior need PR-level review before merge.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P2: Invalid benchmark acceptance can misstate native-evaluation results, but the defect has a bounded scope and an active focused repair.
• impact:other: The issue affects benchmark result integrity and fleet completion accounting rather than a more specific owned impact category.

Evidence reviewed

What I checked:

• Current aggregation behavior remains vulnerable: NonZeroAgentExitCodeError is explicitly classified as agent_exit, while the run eligibility predicate excludes incomplete, infrastructure-dominated, harness-wide, identity, and trajectory failures but has no all-invalid-agent-exit rejection condition. (scripts/native_eval/aggregate.py:30, 884dd1bb5511)
• Current fleet recovery can infer an unrecorded success: When no saved exit code or archived exit-status file is found, recovery assigns exit code zero from full result coverage plus a final checkpoint log entry; the adjacent current test asserts this inferred-success behavior. (scripts/native_eval/fleet.py:1285, 884dd1bb5511)
• Focused linked implementation is still open: The supplied GitHub context shows fix(native-eval): reject invalid harness runs #64 is open, targets the current main SHA, uses closing syntax for this issue, and describes changes to both aggregation acceptance and fleet exit-status recovery. (9f4490e8e437)
• Feature-history provenance: Current eligibility logic traces through native-eval work by Vincent Koc, including fix(eval): separate parity from score eligibility (#47); fleet recovery traces through feat(eval): add reproducible research campaign runbook (#51). Both commits are ancestors of current main. (scripts/native_eval/aggregate.py:918, 9afe6a4f1746)
• Maturity label unavailable: The required maturity shortlist helper could not locate qa/maturity-scores.yaml in this checkout, so no stable-surface label is asserted.

Likely related people:

• Vincent Koc: Current main history attributes the central aggregation eligibility and fleet-recovery work to Vincent Koc, and both relevant commits are reachable from main. (role: recent area contributor; confidence: high; commits: 9afe6a4f1746, 569b5c39c783; files: scripts/native_eval/aggregate.py, scripts/native_eval/fleet.py, tests/test_native_eval_aggregate.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.