fix(native-eval): make OpenClaw Tool Search controls explicit

[vincentkoc]

Summary

ShellBench's OpenClaw native harness does not currently encode the disabled
Tool Search control explicitly. When a run omits openclaw_tool_search_mode,
the generated OpenClaw config omits tools.toolSearch, so the installed
OpenClaw default decides the mode.

Impact

An intended Tool Search "off" ablation can silently execute with code mode
enabled. That contaminates score, token, tool-call, and latency comparisons
because the control and treatment are not actually distinct.

This occurred in an exact candidate run on July 29, 2026: the run index had no
mode, but the exported OpenClaw trace reported codeModeEngaged: true.

Expected behavior

• No mode means the harness writes tools.toolSearch: false.
• directory and code are explicit supported values.
• The selected value survives fleet dispatch, manifests, and automatic reruns.
• Empty fleet environment values normalize to unset metadata, not "".

Verification

Add focused runner and fleet tests covering explicit off, code mode, dispatch,
manifest normalization, and rerun preservation.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 3, 2026, 5:05 AM ET / 09:05 UTC.

Summary
Keep this issue open: current main still omits Tool Search from the planned run, fleet-dispatch contract, manifest, and rerun metadata, while the linked open PR is the active implementation candidate. Because this member-authored request introduces an explicit mode contract beyond the current boolean environment switch, maintainer scope approval is needed before deciding whether the broader PR is the right landing path.

Reproducibility: yes. —the current source provides a high-confidence reproduction path: plan an OpenClaw run without the boolean environment value and inspect the generated configuration, fleet command, manifest, and rerun entry. Those current-main paths contain no explicit Tool Search mode, while the config writer only conditionally emits the setting.

Maintainer decision needed

Question	Recommendation
Should the canonical fix be limited to explicit Tool Search mode planning, propagation, and audit metadata, or should the broader lifecycle and trajectory changes in #62 land as one benchmark-validity correction?	Keep the fix focused: Require a narrow change for explicit disabled, directory, and code mode propagation, manifest recording, rerun preservation, and focused regression coverage.

Why: Current source proves the missing data flow, but the only open candidate bundles materially broader behavior; selecting the permanent scope and public mode contract requires maintainer intent.

Next step
The issue is already paired with an open implementation PR and requires a maintainer decision about whether that PR’s broader scope is acceptable; a parallel repair would duplicate work.

Review details

Best possible solution:

Represent Tool Search mode as explicit planned-run data with a disabled default, propagate it unchanged through remote dispatch and reruns, write it into run manifests, and add focused tests for disabled, directory, code, empty-value normalization, and rerun preservation.

Do we have a high-confidence way to reproduce the issue?

Yes—the current source provides a high-confidence reproduction path: plan an OpenClaw run without the boolean environment value and inspect the generated configuration, fleet command, manifest, and rerun entry. Those current-main paths contain no explicit Tool Search mode, while the config writer only conditionally emits the setting.

Is this the best way to solve the issue?

Unclear—the proposed explicit data flow is the maintainable solution, but the linked open PR includes wider native-evaluation behavior changes. Maintainers should first choose whether the permanent fix is narrow control propagation or the combined correction.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• The linked PR combines the requested Tool Search-control contract with broader lifecycle and trajectory changes, so merging it without an explicit scope decision would make benchmark-validity review harder.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P2: An implicit benchmark control can invalidate comparison results, but the impact is limited to the native-evaluation research path and has an identified implementation candidate.
• impact:other: The affected outcome is benchmark validity and auditability rather than data loss, delivery, session state, security, or provider authentication.

Evidence reviewed

What I checked:

• Current config leaves the setting implicit: The current OpenClaw config writer parses CLAWBENCH_OPENCLAW_TOOL_SEARCH and writes tools.toolSearch only when the value is a boolean; when it is unset, no Tool Search setting is emitted. (scripts/container_lane_eval.sh:510, 884dd1bb5511)
• Fleet dispatch has no Tool Search field: The dispatch argument list passes run identity, task data, and model metadata to the remote runner, but no Tool Search mode; the remote script accepts the same fixed argument contract. (scripts/native_eval/fleet.py:991, 884dd1bb5511)
• Manifest and reruns omit the selected control: The run manifest records execution and reasoning metadata but no Tool Search value. Reruns rebuild only the fixed RUN_SPEC_FIELDS and copy a limited metadata list, neither of which carries this mode. (scripts/native_eval/run_job.py:219, 884dd1bb5511)
• Existing test gap confirms the boundary: The fleet rerun test proves reasoning-effort preservation but contains no Tool Search-mode assertion, matching the absent current data flow. (tests/test_native_eval_fleet.py:1148, 884dd1bb5511)
• Current-main history: The native-evaluation fleet and run-job path on current main was introduced and recently maintained through the July 29 native-harness work; no later local default-branch commit implements the requested mode propagation. (scripts/native_eval/fleet.py:936, 884dd1bb5511)
• Linked implementation candidate remains open: The linked PR explicitly states that it fixes this issue and describes explicit direct, directory, and code OpenClaw surfaces, but it is unmerged and therefore cannot establish an implemented-on-main close. (f5b31cb1260b)

Likely related people:

• vincentkoc: Current main records Vincent Koc as author of the July 29 native-harness implementation that owns the fleet and runner path, and they also authored the linked implementation candidate. (role: recent native-evaluation contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 884dd1bb5511; files: scripts/native_eval/fleet.py, scripts/native_eval/run_job.py, scripts/native_eval/models.py)
• scoootscooob: History for the current OpenClaw container configuration writer shows this contributor authored the relevant evaluation configuration path before native-eval orchestration was added. (role: earlier OpenClaw evaluation configuration contributor; confidence: medium; commits: 5dfa4c9280d3, e3ad7ac1733b, 0f1b45e4674b; files: scripts/container_lane_eval.sh)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.