fix: apply native reasoning effort to harnesses

[vincentkoc]

What does this PR do?

Propagates the native eval plan's reasoning effort into OpenClaw, Hermes,
Codex, and Claude Code client controls.

Why?

Fixes #53. Without this, a run can be labeled high while OpenClaw executes
with thinking disabled, invalidating cross-harness quality and efficiency
comparisons.

Changes

• carry reasoning_effort in RunSpec and the run manifest
• map effort to each native harness's supported control
• translate ShellBench's canonical xhigh to Claude Code's max
• preserve existing harness defaults when effort is unspecified
• test low, medium, high, and xhigh across all four harnesses

Tests

• 114 focused native runner, proxy, and fleet tests pass
• focused Ruff check passes
• fresh Codex autoreview reports no actionable findings
• all GitHub CI and CodeQL checks pass

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 3, 2026, 5:05 AM ET / 09:05 UTC.

ClawSweeper review

What this changes

The PR carries each planned run’s reasoning effort through RunSpec and the manifest, then emits the matching OpenClaw, Hermes, Codex, or Claude Code client setting while retaining existing defaults when no effort is planned.

Merge readiness

⚠️ Ready for maintainer review - 4 items remain

This member-authored PR addresses a current and material native-evaluation consistency defect: current main hard-codes OpenClaw thinking off while the evaluation plan carries a reasoning-effort setting. The branch is cleanly based on current main, has focused command-construction coverage across all four native harnesses, and contains no definite patch bug; before merge, a maintainer should decide whether to require a short smoke run against the pinned client versions because this changes live client controls.

Priority: P1
Reviewed head: 887c2597c1e6ef7ad369cd9a72c800b8fb52ffeb
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	The patch is focused, covers all claimed mappings and preserves defaults, with the remaining question limited to real pinned-client acceptance.
Proof confidence	🌊 off-meta tidepool	Not applicable: This member-authored PR is exempt from the external-contributor real-behavior-proof gate; its body supplies focused tests, but a pinned-toolchain smoke run remains the recommended merge-risk reducer.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This member-authored PR is exempt from the external-contributor real-behavior-proof gate; its body supplies focused tests, but a pinned-toolchain smoke run remains the recommended merge-risk reducer.
Evidence reviewed	5 items	Current defect remains on main: Current main constructs the OpenClaw command with the literal --thinking off, while the plan and proxy already carry canonical reasoning-effort values; the central problem is therefore not implemented on the default branch. Branch implements the intended propagation: The proposed branch adds reasoning_effort to RunSpec, sources it from the CLI/environment, records it in the manifest, and maps it into OpenClaw, Hermes, Codex, and Claude Code command/configuration generation; Claude Code’s canonical xhigh is translated to its max spelling. Fleet dispatch preserves planned effort: The existing fleet obtains reasoning_effort from the planned run entry and exports it as SHELLBENCH_REASONING_EFFORT for the remote runner, so the branch’s build_run_spec environment fallback reaches the generated harness command without requiring an additional fleet change.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench builds a native evaluation plan, dispatches each run through the fleet, and generates one command/configuration per harness. Those clients send benchmark prompts through the proxy, while the run manifest records the intended execution settings used to interpret quality, token, and latency comparisons.

flowchart LR
  A[Evaluation plan] --> B[Fleet dispatch]
  B --> C[Run environment]
  C --> D[Run specification]
  D --> E[Native harness command]
  E --> F[OpenClaw Hermes Codex Claude Code]
  F --> G[Proxy and benchmark results]
  D --> H[Run manifest]

Loading

Decision needed

Question	Recommendation
Should this PR require one short pinned-toolchain smoke run for each native harness before merging its live reasoning-control changes?	Require pinned-client smoke proof: Run a minimal redacted job for OpenClaw, Hermes, Codex, and Claude Code at a non-default effort and attach command/log or manifest evidence showing each client accepted the setting.

Why: Command-construction tests establish the repository wiring, but only a real run can confirm that each pinned external client accepts and honors the emitted control rather than rejecting it or silently using a different setting.

Before merge

• Resolve merge risk (P1) - This intentionally changes the effective reasoning mode of planned native runs; historical results generated with OpenClaw forced to off are not directly comparable to newly generated runs at the same label.
• Resolve merge risk (P1) - The new flags and configuration keys are asserted as generated strings but have not been demonstrated against the pinned OpenClaw 2026.7.1-2, Codex 0.145.0, Claude Code 2.1.220, and Hermes commit toolchain in the supplied evidence; an unsupported or differently interpreted control could fail or distort a harness run.
• Complete next step (P2) - No discrete source defect remains to hand to an automated repair worker; a maintainer must set the evidence threshold for pinned-client runtime validation before merge.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	production +29/-3, tests +92 across 4 files	The production change is bounded to run specification, manifest, and harness command construction, with broad parameterized coverage for all four supported clients.
Harness coverage	4 harnesses × 4 planned effort levels	The tests exercise generated mappings for OpenClaw, Hermes, Codex, and Claude Code at low, medium, high, and xhigh.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #53
Summary: This PR is the explicit candidate fix for the linked native-harness reasoning-effort inconsistency report.

Members:

• canonical: Native harnesses do not apply planned reasoning effort consistently #53 - The PR body explicitly targets the report’s expected per-harness effort mappings, and the issue is still open pending a merged implementation.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

1. Verify pinned client controls (recommended)
   Before merging, capture a small real run for each pinned harness showing that its effort option is accepted and produces the intended runtime or manifest evidence.
2. Accept controlled comparison reset
   Merge with an explicit maintainer acknowledgment that future planned runs intentionally differ from prior native harness defaults and should not be pooled without that distinction.

Technical review

Best possible solution:

Land the focused propagation after a redacted smoke run verifies that each pinned harness accepts its mapped effort setting and that the resulting metadata reflects the requested value; preserve the current no-effort defaults and clearly separate pre-fix benchmark data from newly controlled runs.

Do we have a high-confidence way to reproduce the issue?

Yes—source establishes the mismatch on current main: the plan/fleet carry reasoning effort while the OpenClaw command is built with --thinking off. The supplied regression tests provide a focused command-construction path, although this read-only review did not execute a live toolchain run.

Is this the best way to solve the issue?

Yes—the branch uses the existing plan, fleet environment, run specification, and harness adapters rather than introducing a parallel configuration path. A short pinned-client smoke run is the remaining prudent validation for the external command contracts.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P1: Without matching native client effort to the planned value, benchmark quality, latency, and efficiency comparisons can be materially confounded across all native harnesses.
• merge-risk: 🚨 compatibility: Merging changes existing planned runs from the clients’ prior effective defaults to explicit reasoning controls, affecting comparison continuity and pinned-client behavior.
• merge-risk: 🚨 availability: If any pinned external CLI rejects or mishandles its new control, a native harness job can fail before producing benchmark results.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This member-authored PR is exempt from the external-contributor real-behavior-proof gate; its body supplies focused tests, but a pinned-toolchain smoke run remains the recommended merge-risk reducer.

Evidence

What I checked:

• Current defect remains on main: Current main constructs the OpenClaw command with the literal --thinking off, while the plan and proxy already carry canonical reasoning-effort values; the central problem is therefore not implemented on the default branch. (scripts/native_eval/harnesses.py:619, 884dd1bb5511)
• Branch implements the intended propagation: The proposed branch adds reasoning_effort to RunSpec, sources it from the CLI/environment, records it in the manifest, and maps it into OpenClaw, Hermes, Codex, and Claude Code command/configuration generation; Claude Code’s canonical xhigh is translated to its max spelling. (scripts/native_eval/harnesses.py:825, 887c2597c1e6)
• Fleet dispatch preserves planned effort: The existing fleet obtains reasoning_effort from the planned run entry and exports it as SHELLBENCH_REASONING_EFFORT for the remote runner, so the branch’s build_run_spec environment fallback reaches the generated harness command without requiring an additional fleet change. (scripts/native_eval/fleet.py:1011, 884dd1bb5511)
• Focused regression coverage and clean diff: The PR changes three production files and one test file, adding parameterized coverage for low, medium, high, and xhigh on all four harnesses plus no-effort default behavior and environment-to-RunSpec propagation. git diff --check reported no whitespace errors. (tests/test_native_eval_runner.py:681, 887c2597c1e6)
• Feature-history ownership: Available history shows Vincent Koc introduced the native matrix runner and subsequently maintained native harness reliability and OpenClaw trace behavior. The PR’s two commits also build directly on current main rather than duplicating a merged implementation. (scripts/native_eval/harnesses.py:527, 69f75c6629c4)

Likely related people:

• vincentkoc: Available repository history attributes the native matrix runner, subsequent harness-stability work, current OpenClaw trace maintenance, and this focused repair to Vincent Koc. (role: primary native-evaluation contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 884dd1bb5511; files: scripts/native_eval/harnesses.py, scripts/native_eval/run_job.py, scripts/native_eval/models.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Attach a redacted smoke transcript or manifest excerpt for each pinned native client at a non-default effort level.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (25 earlier review cycles; latest 8 shown)

• reviewed 2026-08-02T13:16:46.864Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T15:52:39.949Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T17:05:47.258Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T19:14:59.352Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T20:38:06.057Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T22:18:33.173Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T01:48:53.091Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T04:09:25.942Z sha 887c259 :: needs maintainer review before merge. :: none