fix: accept Claude native trajectories

[vincentkoc]

What does this PR do?

Marks Claude Code native trajectories as real harness events so completed Claude runs remain eligible for native aggregation.

Why?

The native runner already converts Claude Code stream-json output into trajectory.json, but the harness allowlist omitted claude-code. Aggregation therefore classified otherwise valid Claude runs as trajectory_unavailable.

Changes

• add claude-code to the real-trajectory harness allowlist
• assert every pinned native harness reports real_harness_events

Live proof

A native Claude Code control run completed four tasks with four result files, four valid completed results, and no infrastructure failures. Aggregation classified every trajectory as real; the run remained ineligible only because it was intentionally marked as an exploratory four-task subset.

• coverage: 4/4
• score: 0.75
• exact passes: 3
• trajectory complete: true
• trajectory status: real for all four tasks
• infrastructure failures: 0
• exclusion reason: exploratory_subset

Tests

• python -m pytest -q passes locally (444 passed, 5 skipped)
• python -m ruff check clawbench app.py scripts tests passes locally
• live Claude Code native run aggregates as real harness events

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 3, 2026, 5:05 AM ET / 09:05 UTC.

ClawSweeper review

What this changes

Adds claude-code to ShellBench’s real-native-trajectory allowlist and tests that every pinned native harness declares real harness events.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

Current main still classifies the pinned claude-code harness as unsupported even though it converts its stream output into real trajectories, so this focused PR remains necessary. The patch is technically correct and well-scoped; a maintainer should confirm that reclassifying completed Claude Code runs as real native evidence is the intended benchmark-comparability policy before merge.

Priority: P2
Reviewed head: ac1c974c12749dddb5d49af734561cec59910c65
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused, source-consistent repair with a targeted contract test; merge readiness depends on maintainer approval of the benchmark-evidence classification.
Proof confidence	🌊 off-meta tidepool	Not applicable: This member-authored PR is not subject to the external-contributor proof gate; the PR body nevertheless reports a live Claude Code control run and the focused code path is source-verifiable.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This member-authored PR is not subject to the external-contributor proof gate; the PR body nevertheless reports a live Claude Code control run and the focused code path is source-verifiable.
Evidence reviewed	5 items	Current-main classification gap: Current main includes claude-code among pinned harnesses but excludes it from REAL_TRAJECTORY_HARNESSES; therefore its manifest receives unsupported rather than real_harness_events. Current-main converter already emits real Claude trajectories: The native runtime dispatches Claude Code runs to write_claude_code_trajectory, and the current regression test asserts its resulting trajectory status is real with canonical model identity preserved. Aggregate behavior makes the omission user-visible: Native aggregation requires manifest mode real_harness_events and real task trajectories; otherwise it excludes the run as trajectory_unavailable.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench’s native runner records each harness run in a manifest and converts agent output into a standardized trajectory. Aggregation consumes the manifest mode and per-task trajectory status to decide whether a completed run is eligible for native benchmark results.

flowchart LR
  A[Claude Code stream output] --> B[Native trajectory conversion]
  B --> C[Run manifest trajectory mode]
  C --> D[Aggregate eligibility check]
  D --> E[Native benchmark result]

Loading

Decision needed

Question	Recommendation
Should completed Claude Code stream trajectories be accepted as real_harness_events for native aggregation and longitudinal benchmark comparisons?	Accept Claude trajectories: Confirm the existing Claude converter meets the native evidence contract and merge the allowlist correction.

Why: The code path and test establish a narrow consistency repair, but accepting a harness into the real-evidence category changes benchmark eligibility policy rather than only correcting presentation.

Before merge

• Resolve merge risk (P1) - Merging changes completed Claude Code runs from unsupported evidence to eligible real native trajectory evidence, which can alter aggregate eligibility and longitudinal benchmark comparisons.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	2 files affected; production +1/-1, tests +12	The branch confines the behavioral change to the trajectory allowlist and adds a complete pinned-harness contract assertion.
Pinned harness coverage	4 native harnesses asserted	The added test prevents future drift between the harness registry and real-trajectory classification.

Merge-risk options

Maintainer options:

1. Confirm evidence equivalence before merge (recommended)
   Approve the change only after confirming Claude Code stream conversion is intentionally equivalent to the existing real native harness evidence contract.
2. Preserve current aggregation policy
   Leave Claude Code classified as unsupported until a separately approved benchmark-evidence policy defines its admission criteria.

Technical review

Best possible solution:

Confirm that Claude Code’s standardized stream events meet the same evidence contract as the other native harnesses, then merge this narrow allowlist repair with its contract test.

Do we have a high-confidence way to reproduce the issue?

Yes, from current source: a Claude Code run is converted to a real trajectory, but its manifest is marked unsupported and aggregation consequently selects trajectory_unavailable. This review did not execute a live native run in the read-only checkout.

Is this the best way to solve the issue?

Yes mechanically: adding the already-pinned harness to the existing allowlist is the narrowest repair and the new contract test covers the registry boundary. Maintainer confirmation is still needed because the classification controls benchmark eligibility.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P2: This is a bounded native-evaluation correctness repair with benchmark impact but no demonstrated production outage.
• merge-risk: 🚨 compatibility: Reclassification changes how existing completed Claude Code runs are aggregated and compared over time.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This member-authored PR is not subject to the external-contributor proof gate; the PR body nevertheless reports a live Claude Code control run and the focused code path is source-verifiable.

Evidence

What I checked:

• Current-main classification gap: Current main includes claude-code among pinned harnesses but excludes it from REAL_TRAJECTORY_HARNESSES; therefore its manifest receives unsupported rather than real_harness_events. (scripts/native_eval/models.py:104, 884dd1bb5511)
• Current-main converter already emits real Claude trajectories: The native runtime dispatches Claude Code runs to write_claude_code_trajectory, and the current regression test asserts its resulting trajectory status is real with canonical model identity preserved. (scripts/native_eval/runtime.py:1044, 884dd1bb5511)
• Aggregate behavior makes the omission user-visible: Native aggregation requires manifest mode real_harness_events and real task trajectories; otherwise it excludes the run as trajectory_unavailable. (scripts/native_eval/aggregate.py:906, 884dd1bb5511)
• Focused branch repair: The PR changes the allowlist by one harness and adds a four-harness contract test, directly covering the manifest classification boundary. (tests/test_native_eval_runner.py:60, ac1c974c1274)
• Feature-history routing: Path-specific history identifies Vincent Koc as the author of the available recent native-harness stabilization commit affecting this area. (scripts/native_eval/models.py:119, b9acd9f7a010)

Likely related people:

• vincentkoc: Recent path-specific history attributes native harness stabilization and the research campaign work around this trajectory surface to Vincent Koc. (role: recent native-evaluation contributor; confidence: high; commits: b9acd9f7a010, 569b5c39c783; files: scripts/native_eval/models.py, scripts/native_eval/runtime.py, scripts/native_eval/harness_trajectories.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Confirm that Claude Code stream trajectories should participate in the same native-evidence category as OpenClaw, Hermes, and Codex.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (19 earlier review cycles; latest 8 shown)

• reviewed 2026-08-01T17:56:41.628Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T19:16:43.925Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T13:16:25.271Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T19:14:46.312Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T20:37:25.866Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T22:18:39.060Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T01:48:10.144Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T04:09:32.756Z sha ac1c974 :: needs maintainer review before merge. :: none