Run Hermes adapter through local harness

[afrog33k]

Summary

• Wire registered non-OpenClaw adapters through the generic harness lifecycle instead of rejecting them as known-but-unwired.
• Add Hermes local adapter configuration from environment and a local verification shim for file/execution checks.
• Prepend explicit staged-workspace guidance to Hermes MiniSWERunner prompts so local runs inspect the task workspace first.

Validation

• /Users/reckon/miniconda3/bin/python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q -> 21 passed
• /Users/reckon/miniconda3/bin/python -m pytest -q -> 318 passed, 5 skipped
• Local Hermes rerun: t2-msg-summarize-thread improved from 0.63704 to 1.000 with the workspace-guidance patch.

Notes

• This keeps OpenClaw behavior intact and routes adapter != "openclaw" through the adapter lifecycle.
• Memory/session/cron/browser checks still need adapter-aware verification before broad non-file task sweeps.

[clawsweeper]

Codex review: needs changes before merge. Reviewed August 3, 2026, 5:05 AM ET / 09:05 UTC.

ClawSweeper review

What this changes

The branch routes registered local adapters through ShellBench’s generic execution lifecycle, configures Hermes from environment variables, and adds workspace guidance to Hermes prompts.

Merge readiness

⚠️ Needs maintainer review before merge - 4 items remain

Keep this PR open: current main still rejects Hermes as an unwired adapter, so the central capability is not implemented there. However, the PR’s local scoring path bypasses Hermes’s state verifier after tearing down the adapter, which makes supported memory, session, and cron assertions score against an unavailable gateway instead.

Priority: P2
Reviewed head: 67ff264005ab6b55679299ba4e7fcd7cbb1276bf

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The local-run proof is useful and the integration is focused, but the P1 state-scoring defect prevents confidence in the advertised broader adapter capability.
Proof confidence	🐚 platinum hermit (4/6)	Sufficient (live_output): The PR body provides after-fix local Hermes evidence: a real rerun of t2-msg-summarize-thread reportedly improved from 0.63704 to 1.000, alongside focused and full test results.
Patch quality	🦐 gold shrimp (3/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): The PR body provides after-fix local Hermes evidence: a real rerun of t2-msg-summarize-thread reportedly improved from 0.63704 to 1.000, alongside focused and full test results.
Evidence reviewed	5 items	Current main does not implement Hermes execution: The current harness explicitly rejects every registered adapter except OpenClaw and tells users that Hermes is not wired into end-to-end scoring; the adjacent test asserts that adapter="hermes" raises that error. PR bypasses the adapter state verifier: The branch leaves the adapter context and calls teardown before scoring, then supplies _LocalVerificationClient to score_task_run; that shim raises for gateway state calls instead of invoking adapter.verify_state_query. Hermes advertises state capabilities and implements verification: In ai_agent mode, Hermes declares memory and cron capabilities and implements verify_state_query for memory, session, and cron checks, so routing those checks through a gateway shim produces the wrong evaluation boundary.
Findings	1 actionable finding	[P1] Route state assertions through the live adapter
Security	None	None.

How this fits together

ShellBench’s harness selects an adapter, runs benchmark task phases in a task workspace, then scores files, execution results, and agent state. Hermes is a local agent adapter whose post-run state must be verified through its adapter-specific implementation rather than the OpenClaw gateway.

flowchart LR
  Tasks[Benchmark tasks] --> Harness[Benchmark harness]
  Harness --> Adapter[Hermes adapter]
  Adapter --> Workspace[Task workspace]
  Adapter --> State[Adapter state verifier]
  Workspace --> Scoring[Completion scorer]
  State --> Scoring
  Scoring --> Results[Benchmark results]

Loading

Before merge

• Route state assertions through the live adapter (P1) - The local path calls teardown before scoring and passes _LocalVerificationClient to score_task_run, so Hermes state assertions never reach HermesAdapter.verify_state_query. In ai_agent mode Hermes advertises memory, session, and cron support; those tasks can pass capability gating but be scored against an unavailable gateway. Score before teardown or thread the adapter/context into verification, with a regression test for a state assertion.
• Resolve merge risk (P1) - In Hermes ai_agent mode, memory, session, and cron tasks can be accepted by capability gating but then scored through _LocalVerificationClient, which cannot inspect Hermes state and can report incorrect benchmark results.
• Resolve merge risk (P1) - The new local-adapter path is based on an older main revision; the focused repair should be rebased and exercised against the current harness before merge.
• Complete next step (P2) - The remaining P1 is a concrete mechanical repair: preserve the live adapter context through state scoring and add a targeted regression test.

Findings

• [P1] Route state assertions through the live adapter — clawbench/harness.py:640-644

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	4 files changed, +350/-17	The main behavior change is concentrated in the harness and Hermes adapter, with focused tests for each surface.
Code and tests	production +247/-10, tests +103/-7	Most of the patch is production harness integration, so the missing state-verification regression is material before merge.

Merge-risk options

Maintainer options:

1. Route local state checks through Hermes (recommended)
   Keep the adapter and its context available during scoring, then add coverage for a local state assertion that proves the adapter verifier—not the gateway shim—decides the result.
2. Limit the PR to file and execution tasks
   Explicitly restrict the generic local path to file and execution verification until adapter-aware state scoring is implemented, accepting a narrower Hermes capability surface.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Keep the local adapter context available through scoring, route state assertions to its verifier, and add a regression test for an adapter-backed state assertion.

Technical review

Best possible solution:

Score local adapter runs while the live adapter context is available, passing state assertions through the adapter’s verifier and preserving OpenClaw’s existing gateway-based scoring path.

Do we have a high-confidence way to reproduce the issue?

Yes—source inspection gives a high-confidence path: select Hermes ai_agent mode for a task with a supported state assertion, then observe that scoring receives _LocalVerificationClient rather than the live Hermes adapter verifier.

Is this the best way to solve the issue?

No—the generic adapter lifecycle is the right direction, but scoring after teardown with a gateway-only client is not the narrowest correct integration; state verification must use the live adapter context.

Full review comments:

• [P1] Route state assertions through the live adapter — clawbench/harness.py:640-644
  The local path calls teardown before scoring and passes _LocalVerificationClient to score_task_run, so Hermes state assertions never reach HermesAdapter.verify_state_query. In ai_agent mode Hermes advertises memory, session, and cron support; those tasks can pass capability gating but be scored against an unavailable gateway. Score before teardown or thread the adapter/context into verification, with a regression test for a state assertion.
  Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.98

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
• remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P2: The patch enables a useful benchmark adapter path, but its remaining defect is bounded to correctness of local-adapter state scoring.
• merge-risk: 🚨 session-state: Merging as written can score Hermes memory, session, and cron assertions against an unavailable gateway instead of the agent state that ran the task.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body provides after-fix local Hermes evidence: a real rerun of t2-msg-summarize-thread reportedly improved from 0.63704 to 1.000, alongside focused and full test results.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides after-fix local Hermes evidence: a real rerun of t2-msg-summarize-thread reportedly improved from 0.63704 to 1.000, alongside focused and full test results.

Evidence

Acceptance criteria:

• [P1] python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q.
• [P1] python -m pytest -q.
• [P1] Run one redacted local Hermes ai_agent stateful task and capture terminal output showing the state assertion result.

What I checked:

• Current main does not implement Hermes execution: The current harness explicitly rejects every registered adapter except OpenClaw and tells users that Hermes is not wired into end-to-end scoring; the adjacent test asserts that adapter="hermes" raises that error. (clawbench/harness.py:124, 884dd1bb5511)
• PR bypasses the adapter state verifier: The branch leaves the adapter context and calls teardown before scoring, then supplies _LocalVerificationClient to score_task_run; that shim raises for gateway state calls instead of invoking adapter.verify_state_query. (clawbench/harness.py:640, 67ff264005ab)
• Hermes advertises state capabilities and implements verification: In ai_agent mode, Hermes declares memory and cron capabilities and implements verify_state_query for memory, session, and cron checks, so routing those checks through a gateway shim produces the wrong evaluation boundary. (clawbench/adapters/hermes.py:249, 884dd1bb5511)
• Adapter contract establishes the intended lifecycle: The adapter base contract says state queries are resolved with adapter.verify_state_query(query, ctx) before adapter teardown; the PR has not implemented that contract in the generic scorer path. (clawbench/adapters/base.py:20, 884dd1bb5511)
• Relevant feature history: The adapter canonicalization layer was introduced by 56531fbf432b97db1fc14b0254efb9d858eeb679, and recent current-main harness work includes workspace isolation in 9fe22f9233f723e5554e7bef01b75daf54cfcba5. (clawbench/harness.py:124, 56531fbf432b)

Likely related people:

• scoootscooob: Introduced the canonical adapter layer that defines the lifecycle and capability boundary this PR extends. (role: adapter-layer introducer; confidence: high; commits: 56531fbf432b, f09a9f4bf74b; files: clawbench/adapters/base.py, clawbench/adapters/hermes.py, clawbench/harness.py)
• Robin Narsingh Ranabhat: Most recent locally available current-main history touching the harness focused on per-run workspace behavior, which this local-adapter path also uses. (role: recent harness contributor; confidence: medium; commits: 9fe22f9233f7; files: clawbench/harness.py)
• Vincent Koc: Recent harness history includes scoring and runtime hardening changes relevant to preserving correct benchmark results. (role: recent scoring and runtime contributor; confidence: medium; commits: fb486a1ed307, ed9adf8d8451; files: clawbench/harness.py, clawbench/scorer.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Route state assertions through adapter.verify_state_query before teardown.
• Add a focused regression test for a supported local memory, session, or cron assertion.
• Rebase the repaired path against current main and post redacted live output for that stateful task.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (30 earlier review cycles; latest 8 shown)

• reviewed 2026-08-02T13:16:49.266Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-02T14:58:11.781Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-02T17:06:20.210Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-02T19:15:11.302Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-02T20:37:21.042Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-02T22:14:48.627Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-03T01:48:51.671Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-03T04:09:35.877Z sha 67ff264 :: needs changes before merge. :: [P1] Pass adapter state verification into scoring