From 8c4377ab86f5623dbed3c90a009621b29618d111 Mon Sep 17 00:00:00 2001 From: Lee Overy Date: Sun, 26 Jul 2026 20:03:18 +0100 Subject: [PATCH] feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both agents were put on Opus back when a walk was judged by the single message an agent returns. A compressed closing summary read as a skipped step, the walker looked unreliable, and the answer was to spend more. With the walk now captured turn by turn the reason is gone. Measured rather than assumed: a Sonnet walk of implementation-picks- first-task passes every path step with quoted evidence, an identical world, and no markers. An Opus walk of the same case made the same calls in the same order. Across both cases and both models the code-checked layer came out identical. Sonnet also declined a system-reminder pushing auto-mode bias against stopping, citing the prose line that names harness auto mode as the thing to ignore — the STOP-gate discipline the pipeline depends on. Judging stays on Opus. It is the cheaper half by far, one prompt in and a verdict out, and it is the half the result rests on. A FAIL is rerun once on Opus. A failure is where spending more is worth it, and a defect a stronger walker also hits is a defect. That escalation happens in the moment, never by editing the definition — so a routine run is always the model the definition names, and the model of every walk is recorded and reported either way, which keeps an escalated rerun from being read as a like-for-like one. Co-Authored-By: Claude Opus 5 --- .claude/agents/prose-orchestrator.md | 26 +++++++++++++++++++------- .claude/agents/prose-walker.md | 2 +- design/prose-tests.md | 22 +++++++++++++--------- 3 files changed, 33 insertions(+), 17 deletions(-) diff --git a/.claude/agents/prose-orchestrator.md b/.claude/agents/prose-orchestrator.md index 06564ca76..fee116010 100644 --- a/.claude/agents/prose-orchestrator.md +++ b/.claude/agents/prose-orchestrator.md @@ -63,17 +63,28 @@ a walk. `VERDICT: HARNESS ERROR` with the message it printed. Never fall back to judging a walk with no record of what it did. -Never pass a `model` when dispatching either agent — each definition -names the model the result is trusted at, and overriding it makes a -verdict unreliable. +Never pass a `model` on a first walk or on any assertion — each +definition names the model the result is trusted at. The single +exception is the confirmation rerun in step 4, below. 4. **Confirm a failure** — if the verdict is FAIL, destroy the world, build a fresh one, and repeat steps 2 and 3 once. A defect in the - prose reproduces; a one-off does not. Two outcomes: + prose reproduces; a one-off does not. + + Dispatch this second walk with `model: opus`. Walks run on the model + the definition names; a failure is where it is worth spending more, + and a defect a stronger walker also hits is a defect. The model each + walk ran on is recorded and reported, so an escalated rerun is never + mistaken for a like-for-like one. Escalate here and nowhere else — + never on a first walk, and never for the asserter. + + Three outcomes: - The second run also FAILs → a confirmed finding. Report the evidence from the second run. - - The second run PASSes → a non-reproducing failure. Report it as - `FLAKY`, quoting both, and resolve nothing yourself. + - The second run PASSes → report `FLAKY`, quoting both, and resolve + nothing yourself. Name both models: the same case passing on the + stronger walker is a fact about the walk, not about the prose. + - The second run returns `INVALID WALK` → follow step 5. 5. **Retry an invalid walk** — if the verdict is `INVALID WALK`, the walker began mid-flow and the case was never actually exercised. @@ -90,7 +101,8 @@ Return exactly this and nothing else: ``` CASE: -MODEL: +MODEL: VERDICT: PASS | FAIL | FLAKY | INVALID | HARNESS ERROR PATH: / steps passed WORLD: PASS | FAIL diff --git a/.claude/agents/prose-walker.md b/.claude/agents/prose-walker.md index bfb5ddf8d..758d17e69 100644 --- a/.claude/agents/prose-walker.md +++ b/.claude/agents/prose-walker.md @@ -2,7 +2,7 @@ name: prose-walker description: Executes workflow prose exactly as a live session would, against a disposable test world, and returns a transcript of what it did. Dispatched by prose-orchestrator during a prose-test run. tools: Read, Write, Edit, Bash, Glob, Grep -model: opus +model: sonnet hooks: PreToolUse: - matcher: "Bash|Write|Edit|Read|Glob|Grep" diff --git a/design/prose-tests.md b/design/prose-tests.md index dba57d890..1970bcd02 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -119,15 +119,19 @@ testing. computation powers the PR-end suggestion — "these N cases intersect this PR's prose changes"), by hand-picked ids, or `--all`. An engine-only PR intersects nothing and suggests nothing. -- **P6 — one model, chosen for trust.** Walker and asserter both run on - **Opus**. Measured, not assumed: across three runs a Sonnet walker - performed the walk correctly every time but narrated it in summary, - omitting the quoted evidence the asserter requires — even with a - worked transcript example in front of it. A tiered arrangement then - produces a disagreement on every case, which is noise, not a signal. - A result you cannot rely on is worth nothing, so the cheaper tier is - a false economy. The orchestrator never overrides the model: each - definition names the model its result is trusted at. +- **P6 — the walker walks on Sonnet, the asserter judges on Opus, and a + failure escalates.** Both ran on Opus while the harness was reading only + an agent's closing message and mistaking a compressed summary for a + skipped step; with the walk captured turn by turn (P6e) that gap closed, + and a Sonnet walk of implementation-picks-first-task passed every step + with quoted evidence, matching an Opus walk of the same case action for + action. Judging stays on Opus — it is the cheap half and the half a + verdict rests on. A FAIL is reran once on Opus by the orchestrator, + because a failure is where spending more is worth it and a defect a + stronger walker also hits is a defect; the model of every walk is + recorded and reported, so an escalated rerun is never mistaken for a + like-for-like one. Escalation happens in the moment, never by editing + the definition. - **P6a — a FAIL is confirmed by repetition, not by promotion.** A failing case re-runs once from a fresh world at the same models. A defect in the prose reproduces; a one-off does not, and is reported