diff --git a/.claude/agents/prose-orchestrator.md b/.claude/agents/prose-orchestrator.md index 06564ca76..fee116010 100644 --- a/.claude/agents/prose-orchestrator.md +++ b/.claude/agents/prose-orchestrator.md @@ -63,17 +63,28 @@ a walk. `VERDICT: HARNESS ERROR` with the message it printed. Never fall back to judging a walk with no record of what it did. -Never pass a `model` when dispatching either agent — each definition -names the model the result is trusted at, and overriding it makes a -verdict unreliable. +Never pass a `model` on a first walk or on any assertion — each +definition names the model the result is trusted at. The single +exception is the confirmation rerun in step 4, below. 4. **Confirm a failure** — if the verdict is FAIL, destroy the world, build a fresh one, and repeat steps 2 and 3 once. A defect in the - prose reproduces; a one-off does not. Two outcomes: + prose reproduces; a one-off does not. + + Dispatch this second walk with `model: opus`. Walks run on the model + the definition names; a failure is where it is worth spending more, + and a defect a stronger walker also hits is a defect. The model each + walk ran on is recorded and reported, so an escalated rerun is never + mistaken for a like-for-like one. Escalate here and nowhere else — + never on a first walk, and never for the asserter. + + Three outcomes: - The second run also FAILs → a confirmed finding. Report the evidence from the second run. - - The second run PASSes → a non-reproducing failure. Report it as - `FLAKY`, quoting both, and resolve nothing yourself. + - The second run PASSes → report `FLAKY`, quoting both, and resolve + nothing yourself. Name both models: the same case passing on the + stronger walker is a fact about the walk, not about the prose. + - The second run returns `INVALID WALK` → follow step 5. 5. **Retry an invalid walk** — if the verdict is `INVALID WALK`, the walker began mid-flow and the case was never actually exercised. @@ -90,7 +101,8 @@ Return exactly this and nothing else: ``` CASE: -MODEL: +MODEL: VERDICT: PASS | FAIL | FLAKY | INVALID | HARNESS ERROR PATH: / steps passed WORLD: PASS | FAIL diff --git a/.claude/agents/prose-walker.md b/.claude/agents/prose-walker.md index bfb5ddf8d..758d17e69 100644 --- a/.claude/agents/prose-walker.md +++ b/.claude/agents/prose-walker.md @@ -2,7 +2,7 @@ name: prose-walker description: Executes workflow prose exactly as a live session would, against a disposable test world, and returns a transcript of what it did. Dispatched by prose-orchestrator during a prose-test run. tools: Read, Write, Edit, Bash, Glob, Grep -model: opus +model: sonnet hooks: PreToolUse: - matcher: "Bash|Write|Edit|Read|Glob|Grep" diff --git a/design/prose-tests.md b/design/prose-tests.md index dba57d890..1970bcd02 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -119,15 +119,19 @@ testing. computation powers the PR-end suggestion — "these N cases intersect this PR's prose changes"), by hand-picked ids, or `--all`. An engine-only PR intersects nothing and suggests nothing. -- **P6 — one model, chosen for trust.** Walker and asserter both run on - **Opus**. Measured, not assumed: across three runs a Sonnet walker - performed the walk correctly every time but narrated it in summary, - omitting the quoted evidence the asserter requires — even with a - worked transcript example in front of it. A tiered arrangement then - produces a disagreement on every case, which is noise, not a signal. - A result you cannot rely on is worth nothing, so the cheaper tier is - a false economy. The orchestrator never overrides the model: each - definition names the model its result is trusted at. +- **P6 — the walker walks on Sonnet, the asserter judges on Opus, and a + failure escalates.** Both ran on Opus while the harness was reading only + an agent's closing message and mistaking a compressed summary for a + skipped step; with the walk captured turn by turn (P6e) that gap closed, + and a Sonnet walk of implementation-picks-first-task passed every step + with quoted evidence, matching an Opus walk of the same case action for + action. Judging stays on Opus — it is the cheap half and the half a + verdict rests on. A FAIL is reran once on Opus by the orchestrator, + because a failure is where spending more is worth it and a defect a + stronger walker also hits is a defect; the model of every walk is + recorded and reported, so an escalated rerun is never mistaken for a + like-for-like one. Escalation happens in the moment, never by editing + the definition. - **P6a — a FAIL is confirmed by repetition, not by promotion.** A failing case re-runs once from a fresh world at the same models. A defect in the prose reproduces; a one-off does not, and is reported