fix: apply native reasoning effort to harnesses

[vincentkoc]

What does this PR do?

Propagates the native eval plan's reasoning effort into OpenClaw, Hermes,
Codex, and Claude Code client controls.

Why?

Fixes #53. Without this, a run can be labeled high while OpenClaw executes
with thinking disabled, invalidating cross-harness quality and efficiency
comparisons.

Changes

• carry reasoning_effort in RunSpec and the run manifest
• map effort to each native harness's supported control
• translate ShellBench's canonical xhigh to Claude Code's max
• preserve existing harness defaults when effort is unspecified
• test low, medium, high, and xhigh across all four harnesses

Tests

• 114 focused native runner, proxy, and fleet tests pass
• focused Ruff check passes
• fresh Codex autoreview reports no actionable findings
• all GitHub CI and CodeQL checks pass

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 3, 2026, 12:09 AM ET / 04:09 UTC.

ClawSweeper review

What this changes

The PR carries each native evaluation plan’s reasoning-effort value through the run specification and manifest, then emits the matching OpenClaw, Hermes, Codex, or Claude Code client control while preserving defaults when no effort is specified.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep this member-authored PR open for normal maintainer review. The branch addresses the source-confirmed mismatch in the linked issue and has focused command-construction coverage, but it changes effective runtime controls across four pinned native clients, so a short real harness smoke should confirm those controls are accepted and applied before merge.

Priority: P1
Reviewed head: 887c2597c1e6ef7ad369cd9a72c800b8fb52ffeb
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The focused patch and source coverage are coherent, but no inspectable live evidence confirms that all four pinned native clients honor the mapped controls.
Proof confidence	🌊 off-meta tidepool	Not applicable: The PR author is a repository member, so the external-contributor proof gate does not apply; the stated focused tests and CI are useful supplemental validation, while a pinned-client smoke would still reduce compatibility risk.
Patch quality	🦐 gold shrimp (3/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The PR author is a repository member, so the external-contributor proof gate does not apply; the stated focused tests and CI are useful supplemental validation, while a pinned-client smoke would still reduce compatibility risk.
Evidence reviewed	4 items	Current-main source confirms the reported mismatch: Current main unconditionally emits openclaw agent ... --thinking off, so a planned high reasoning effort can be absent from the OpenClaw client invocation. The branch propagates the plan value through all native adapters: The PR adds reasoning_effort to RunSpec, derives it from the CLI or environment, records it in the manifest, and maps it to OpenClaw, Hermes, Codex, and Claude Code command/config surfaces. Focused coverage exercises all supported effort levels and default preservation: The added parameterized test checks low, medium, high, and xhigh mappings for all four harnesses, including Claude Code’s xhigh-to-max conversion; a separate test checks that unspecified effort retains each harness default.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench turns a native evaluation plan into a per-run specification, builds a command for one of four external agent clients, and captures results and metadata for comparison. The changed adapter layer determines whether the reasoning setting recorded for a benchmark is also applied by the client that executes it.

flowchart LR
  A[Native evaluation plan] --> B[Run specification]
  B --> C[Harness command builder]
  C --> D[OpenClaw Hermes Codex Claude Code]
  D --> E[Agent execution logs]
  B --> F[Run manifest]
  E --> G[Benchmark results]
  F --> G

Loading

Decision needed

Question	Recommendation
Should this PR merge after a pinned-client smoke validates the four native reasoning controls, or should ShellBench retain the previous client-default behavior until that compatibility evidence exists?	Validate then merge: Capture a short redacted smoke for the pinned OpenClaw, Hermes, Codex, and Claude Code clients, then merge if each accepts the mapped effort and the default path remains unchanged.

Why: The source establishes the benchmark mismatch and the adapter mappings are coherent, but only maintainers can accept the compatibility risk for the repository’s pinned external client toolchains.

Before merge

• Resolve merge risk (P1) - Merging changes existing planned runs from each client’s prior effective reasoning behavior to an explicit client control; command-construction tests do not prove that every pinned OpenClaw, Hermes, Codex, and Claude Code version accepts and honors the emitted setting at runtime.
• Complete next step (P2) - The patch has no discrete source-level defect to repair automatically, but a maintainer should review the pinned-client compatibility evidence before merging this member-authored runtime behavior change.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Implementation and coverage	production +29/-3, tests +92 across 4 files	The compact production change is backed by a matrix covering four harnesses and four planned effort levels.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #53
Summary: This PR is the explicit candidate fix for the source-confirmed native reasoning-effort mismatch tracked in the linked issue.

Members:

• canonical: Native harnesses do not apply planned reasoning effort consistently #53 - The issue documents the same OpenClaw --thinking off mismatch and specifies the four expected native control mappings implemented by this PR.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

1. Capture pinned-client compatibility proof (recommended)
   Before merge, run one redacted native smoke per harness that shows the installed client accepts the emitted setting and that an unspecified plan retains the existing default.
2. Accept the toolchain risk
   Merge based on command-construction coverage and explicitly accept that an unsupported pinned client may fail or silently ignore the requested reasoning level.

Technical review

Best possible solution:

Land the shared RunSpec propagation only after a small redacted pinned-client smoke demonstrates that each harness starts with its intended reasoning setting and that an unspecified setting still retains the prior default behavior.

Do we have a high-confidence way to reproduce the issue?

Yes—source inspection gives a high-confidence path: current main builds an OpenClaw command with --thinking off even when the native plan/environment carries a reasoning effort. This review did not execute the pinned client containers.

Is this the best way to solve the issue?

Yes in design: carrying the plan value through RunSpec gives all harnesses one source of truth and preserves defaults when omitted. A pinned-client smoke remains the safest final validation of the exact external CLI/config mappings.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The PR author is a repository member, so the external-contributor proof gate does not apply; the stated focused tests and CI are useful supplemental validation, while a pinned-client smoke would still reduce compatibility risk.
• remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.
• remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

• P1: A mismatch between recorded and effective reasoning modes invalidates native benchmark quality, latency, and efficiency comparisons.
• merge-risk: 🚨 compatibility: The PR introduces explicit reasoning controls for four externally versioned client CLIs whose runtime acceptance is not established by the source-only tests.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The PR author is a repository member, so the external-contributor proof gate does not apply; the stated focused tests and CI are useful supplemental validation, while a pinned-client smoke would still reduce compatibility risk.

Evidence

What I checked:

• Current-main source confirms the reported mismatch: Current main unconditionally emits openclaw agent ... --thinking off, so a planned high reasoning effort can be absent from the OpenClaw client invocation. (scripts/native_eval/harnesses.py:619, 884dd1bb5511)
• The branch propagates the plan value through all native adapters: The PR adds reasoning_effort to RunSpec, derives it from the CLI or environment, records it in the manifest, and maps it to OpenClaw, Hermes, Codex, and Claude Code command/config surfaces. (scripts/native_eval/harnesses.py:535, 887c2597c1e6)
• Focused coverage exercises all supported effort levels and default preservation: The added parameterized test checks low, medium, high, and xhigh mappings for all four harnesses, including Claude Code’s xhigh-to-max conversion; a separate test checks that unspecified effort retains each harness default. (tests/test_native_eval_runner.py:684, 887c2597c1e6)
• Feature-history provenance: The native evaluation runner and later OpenClaw trace maintenance in this scoped path are attributed to Vincent Koc, making him the strongest routing candidate for the harness-control contract. (scripts/native_eval/harnesses.py:520, 69f75c6629c4)

Likely related people:

• vincentkoc: Current-main history attributes the native matrix runner, follow-up stabilization, and OpenClaw trace work in the affected files to Vincent Koc; this is broader ownership evidence than the proposed PR alone. (role: native-evaluation feature owner and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 884dd1bb5511; files: scripts/native_eval/harnesses.py, scripts/native_eval/models.py, scripts/native_eval/run_job.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Add a redacted pinned-client smoke transcript or logs showing each harness accepts the selected effort and preserves its default when no effort is supplied.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (24 earlier review cycles; latest 8 shown)

• reviewed 2026-08-01T19:17:11.479Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T13:16:46.864Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T15:52:39.949Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T17:05:47.258Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T19:14:59.352Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T20:38:06.057Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T22:18:33.173Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T01:48:53.091Z sha 887c259 :: needs maintainer review before merge. :: none