Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions .claude/agents/prose-orchestrator.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,17 +63,28 @@ a walk.
`VERDICT: HARNESS ERROR` with the message it printed. Never fall back
to judging a walk with no record of what it did.

Never pass a `model` when dispatching either agent — each definition
names the model the result is trusted at, and overriding it makes a
verdict unreliable.
Never pass a `model` on a first walk or on any assertion — each
definition names the model the result is trusted at. The single
exception is the confirmation rerun in step 4, below.

4. **Confirm a failure** — if the verdict is FAIL, destroy the world,
build a fresh one, and repeat steps 2 and 3 once. A defect in the
prose reproduces; a one-off does not. Two outcomes:
prose reproduces; a one-off does not.

Dispatch this second walk with `model: opus`. Walks run on the model
the definition names; a failure is where it is worth spending more,
and a defect a stronger walker also hits is a defect. The model each
walk ran on is recorded and reported, so an escalated rerun is never
mistaken for a like-for-like one. Escalate here and nowhere else —
never on a first walk, and never for the asserter.

Three outcomes:
- The second run also FAILs → a confirmed finding. Report the
evidence from the second run.
- The second run PASSes → a non-reproducing failure. Report it as
`FLAKY`, quoting both, and resolve nothing yourself.
- The second run PASSes → report `FLAKY`, quoting both, and resolve
nothing yourself. Name both models: the same case passing on the
stronger walker is a fact about the walk, not about the prose.
- The second run returns `INVALID WALK` → follow step 5.

5. **Retry an invalid walk** — if the verdict is `INVALID WALK`, the
walker began mid-flow and the case was never actually exercised.
Expand All @@ -90,7 +101,8 @@ Return exactly this and nothing else:

```
CASE: <case-id>
MODEL: <the model the asserter reported, or `unrecorded`>
MODEL: <the model the asserter reported, or `unrecorded`. On a confirmed or
flaky failure, both — the first walk's and the escalated rerun's>
VERDICT: PASS | FAIL | FLAKY | INVALID | HARNESS ERROR
PATH: <n>/<total> steps passed
WORLD: PASS | FAIL
Expand Down
2 changes: 1 addition & 1 deletion .claude/agents/prose-walker.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name: prose-walker
description: Executes workflow prose exactly as a live session would, against a disposable test world, and returns a transcript of what it did. Dispatched by prose-orchestrator during a prose-test run.
tools: Read, Write, Edit, Bash, Glob, Grep
model: opus
model: sonnet
hooks:
PreToolUse:
- matcher: "Bash|Write|Edit|Read|Glob|Grep"
Expand Down
22 changes: 13 additions & 9 deletions design/prose-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,15 +119,19 @@ testing.
computation powers the PR-end suggestion — "these N cases intersect
this PR's prose changes"), by hand-picked ids, or `--all`. An
engine-only PR intersects nothing and suggests nothing.
- **P6 — one model, chosen for trust.** Walker and asserter both run on
**Opus**. Measured, not assumed: across three runs a Sonnet walker
performed the walk correctly every time but narrated it in summary,
omitting the quoted evidence the asserter requires — even with a
worked transcript example in front of it. A tiered arrangement then
produces a disagreement on every case, which is noise, not a signal.
A result you cannot rely on is worth nothing, so the cheaper tier is
a false economy. The orchestrator never overrides the model: each
definition names the model its result is trusted at.
- **P6 — the walker walks on Sonnet, the asserter judges on Opus, and a
failure escalates.** Both ran on Opus while the harness was reading only
an agent's closing message and mistaking a compressed summary for a
skipped step; with the walk captured turn by turn (P6e) that gap closed,
and a Sonnet walk of implementation-picks-first-task passed every step
with quoted evidence, matching an Opus walk of the same case action for
action. Judging stays on Opus — it is the cheap half and the half a
verdict rests on. A FAIL is reran once on Opus by the orchestrator,
because a failure is where spending more is worth it and a defect a
stronger walker also hits is a defect; the model of every walk is
recorded and reported, so an escalated rerun is never mistaken for a
like-for-like one. Escalation happens in the moment, never by editing
the definition.
- **P6a — a FAIL is confirmed by repetition, not by promotion.** A
failing case re-runs once from a fresh world at the same models. A
defect in the prose reproduces; a one-off does not, and is reported
Expand Down