From 375330c84e0db9d5c31a9b6eef91bab6306902dd Mon Sep 17 00:00:00 2001 From: Lee Overy Date: Sun, 26 Jul 2026 21:58:22 +0100 Subject: [PATCH 1/2] fix(prose-tests): the asserter judges which of prose or walker was at fault MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An epic walk came back FLAKY over a step the prose got right. The walker read confirm-trigger's route to Step 7, talked itself into Step 6 on the grounds that the file numbers it earlier, read the reference, ran one manifest read, then caught itself: "my own misreading of the routing, not a prose deviation." It corrected and finished correctly. Both runs landed the right world and passed every deterministic check. The asserter had that sentence in front of it and marked the step failed anyway, because its rules told it to. "Never explain a failure" was written against asserters that diagnosed causes and proposed fixes, and it over-reached: it also forbade deciding whether the prose or the walker went wrong. That attribution is not an explanation of a failure — it is what makes something a failure in the first place, and it is the one judgement only this agent can make. The deterministic checks already catch everything a script can; a reasoning model that is barred from reasoning is just a slower script. So the rules now say what they should have said. A path step fails when the prose failed. A walker that wandered off its own initiative and came back has shown nothing about the prose, and the step passes on the corrected path — quoting the correction, since a pass still needs its evidence. The record settles that a call was made and never why: a call made during an abandoned detour looks identical to one the prose asked for, and only the walk says which it was. The guards that made the original rule necessary all stay: no invented reading that rescues a step, no assuming an unrecorded action happened, no correct end state excusing a wrong path, and genuine ambiguity fails. Judgement is not generosity. The wander is still reported, under markers. One walker wandering is noise; the same wander recurring on the same step is prose that reads misleadingly however correct it is, and that only becomes visible if the wobbles are recorded rather than swallowed. Co-Authored-By: Claude Opus 5 --- .claude/agents/prose-asserter.md | 40 ++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/.claude/agents/prose-asserter.md b/.claude/agents/prose-asserter.md index ad27af794..ca0785349 100644 --- a/.claude/agents/prose-asserter.md +++ b/.claude/agents/prose-asserter.md @@ -28,12 +28,30 @@ tools you appear to have or what mode you are running in. ## Rules -- **Never explain a failure.** Report what the evidence shows and stop. - Diagnosing why the prose behaved as it did, tracing an implementation - to account for it, or proposing what should change — all forbidden. - "The step is missing from the walk" is the finding, whole. +- **You are judging the prose, not the walker.** A path step fails when + the *prose* failed — it routed somewhere it should not have, skipped + something it prescribes, or produced the wrong thing. A walker that + took a wrong turn on its own initiative and then corrected itself has + demonstrated nothing about the prose, and the step it wandered past + passes on the corrected path. Deciding which of the two was at fault + is the judgement you exist to make. It is why a reasoning agent does + this and not a script: the deterministic checks above already caught + everything a script could. +- **Read the whole walk before ruling on any step.** A detour that is + taken back is not a failure, and a walk that reaches the right place by + the wrong route is not a pass. Neither is visible from a single line. +- **Never explain a failure.** Once the prose is what failed, report what + the evidence shows and stop. Diagnosing why the prose behaved as it + did, tracing an implementation to account for it, or proposing what + should change — all forbidden. "The step is missing from the walk" is + the finding, whole. +- **Judgement is not generosity.** Do not invent a reading that rescues a + step, do not assume an unrecorded action happened, and do not let a + correct end state excuse a wrong path. Where the evidence is genuinely + ambiguous, say so and fail the step. - A PASS on any path step **requires a quote** from the walk that - shows it. A PASS without a quote is invalid — mark it FAIL. + shows it. A PASS without a quote is invalid — mark it FAIL. Where a + step passes on a corrected path, quote the correction too. - Judge the path against what the walk records, and the world against the delta. Never infer one from the other. - A world can match while the walk still went wrong: engine calls that @@ -65,6 +83,12 @@ closes with an incorporate") is evidenced from the recorded actions. A step about reasoning or output is evidenced from the walk. Where the two disagree about what was done, the recorded actions win. +The record settles *that* a call was made, never *why*. A call the walker +made during a wrong turn of its own, and then abandoned, appears in the +record exactly like one the prose asked for — only the walk says which it +was. So a claim that something was not done is answered by both together: +the record for whether it happened, the walk for whose doing it was. + ## An invalid walk is not a failing walk Check the **recorded actions**, not the walk, for where the walk @@ -100,6 +124,12 @@ Return exactly this and nothing else: 3. **Markers** — list every `UNSCRIPTED QUESTION`, `AMBIGUOUS`, and `DEVIATION` in the walk. Each is a finding in its own right, even when everything else passed. + + List here too any wrong turn the walker took and corrected, naming + the step it wandered into and what brought it back. It is not a + failure and must not be scored as one, but it must not vanish either: + one walker wandering is noise, and the same wander recurring on the + same step is prose that reads misleadingly however correct it is. 4. **VERDICT** — PASS, FAIL, or INVALID WALK, then one sentence on what it means for the prose. Any FAILing deterministic check makes this FAIL. From 71c48d997664b2e4e6e1f6f299c6726fa6d5f2a3 Mon Sep 17 00:00:00 2001 From: Lee Overy Date: Sun, 26 Jul 2026 22:02:17 +0100 Subject: [PATCH 2/2] docs(prose-tests): a run reports, it never repairs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two rules the last run earned. Nothing is fixed during or after a run — prose, case, snapshot or harness — however small or obvious. Everything is surfaced and decided together. The reason is not caution for its own sake: on the run that prompted this, the first fix reached for was a prose change to correct prose, and the second was deleting a claim that was doing real work. The actual fault was a third thing, in the judging, and neither repair would have found it. A test that repairs what it finds also destroys the record of what it found. And a pass carrying notes is not a clean pass. Markers always warrant a look. A walker that wandered and corrected itself has not failed the prose, but the same wander recurring on the same step is prose that reads misleadingly however correct it is — visible only if every one is surfaced rather than shrugged off. Co-Authored-By: Claude Opus 5 --- .claude/skills/prose-test/SKILL.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/.claude/skills/prose-test/SKILL.md b/.claude/skills/prose-test/SKILL.md index bbc3178c6..7b6d891e3 100644 --- a/.claude/skills/prose-test/SKILL.md +++ b/.claude/skills/prose-test/SKILL.md @@ -61,6 +61,18 @@ passed. ## Rules -- Findings are findings. Never edit prose, cases, or snapshots to make a run pass — report and stop. +- **Never fix anything during or after a run.** Not the prose, not a + case, not a snapshot, not the harness — however small, however obvious, + however certain you are. Every failure, every flake, and every pass + carrying notes is surfaced to the user, and what to do about it is + decided together. A test that repairs what it finds destroys the record + of what it found, and the first fix reached for is regularly aimed at + the wrong layer: correct prose has been called defective, and a case + has been blamed for a fault in the judging. +- **A pass with notes is not a clean pass.** Markers are findings in + their own right and always warrant a look. A walker that wandered and + corrected itself did not fail the prose — but the same wander recurring + on the same step is prose that reads misleadingly however correct it + is, and that is only ever visible if every one is surfaced. - A failing case is a finding either way: broken prose, or a stale case. Say which is suspected; the user decides. - Never regenerate a snapshot as part of a run.