From 2dc30b2950b649b29d3896215f5c4806c211ed1c Mon Sep 17 00:00:00 2001 From: Lee Overy Date: Mon, 27 Jul 2026 15:05:45 +0100 Subject: [PATCH] fix(prose-tests): a confirmed failure reports both runs' checks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The quick-fix verdict showed one CHECKS block with two FAILs in it. What actually happened: the first walk's checks all passed and it failed only on a path step; the confirmation walk then skipped two prescribed calls and its checks caught that. The verdict's single block was the second run's, standing silently for both — reading as though the case's checks failed, when the check failures were one walk's behaviour. The orchestrator was following its own instruction: report the evidence from the second run. That instruction is right for the path evidence and wrong for the checks, because checks that pass on one walk and fail on the other are not a property of the prose at all. They are the two walks behaving differently — and that variance is itself a finding, the exact signal that separates a defect from an unreliable walk. A confirmed or flaky verdict now carries both runs' check blocks, labelled per run, with any divergence between them named as walk variance. Untangling this from the verdict took transcript archaeology that the report should have made unnecessary. Co-Authored-By: Claude Opus 5 --- .claude/agents/prose-orchestrator.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/.claude/agents/prose-orchestrator.md b/.claude/agents/prose-orchestrator.md index 04735107f..50d7c4333 100644 --- a/.claude/agents/prose-orchestrator.md +++ b/.claude/agents/prose-orchestrator.md @@ -79,6 +79,12 @@ exception is the confirmation rerun in step 4, below. stronger walker is a fact about the walk, not about the prose. - The second run returns `INVALID WALK` → follow step 5. + When two runs happened, report both runs' deterministic checks, + labelled per run — never one run's block standing for both. Checks + that pass on one walk and fail on the other are not a property of the + prose; they are the two walks behaving differently, and that variance + is a finding in its own right. Name it as one. + 5. **Retry an invalid walk** — if the verdict is `INVALID WALK`, the walker began mid-flow and the case was never actually exercised. Destroy the world, build a fresh one, and repeat steps 2 and 3 once. @@ -103,7 +109,9 @@ VERDICT: PASS | FAIL | FLAKY | INVALID | HARNESS ERROR CHECKS: + them turns a fact back into an assumption. When two runs happened, + both runs' blocks, labelled — `first walk:` / `confirmation:` — + and a line naming any divergence between them as walk variance.> PATH: / steps passed WORLD: PASS | FAIL MARKERS: