Skip to content

fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope#573

Open
leeovery wants to merge 1 commit into
fix/prose-closing-turn-and-known-differencefrom
fix/prose-prescribe-what-we-observed
Open

fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope#573
leeovery wants to merge 1 commit into
fix/prose-closing-turn-and-known-differencefrom
fix/prose-prescribe-what-we-observed

Conversation

@leeovery

@leeovery leeovery commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Three things the full run turned up, none of them failures.

The asserter's cross-check is now prescribed

It had begun comparing the action record against the walker's narrative unprompted, and reporting divergences. Right instinct — but nobody's instruction, so any run depending on it was depending on a coincidence.

Now required, both directions named:

  • Narrated but not recorded — the walker described something it didn't do. The step resting on that claim fails, and the discrepancy is a finding in itself.
  • Recorded but not narrated — usually orientation, fails nothing, but reported: a call that touched state and went unmentioned is the shape a silent repair takes.

The two streams exist so neither is taken on trust. That only holds if they're actually set against each other every time.

engine_before_write declared only where it can fire

It was on all ten cases and could only ever trip on the three that write state. Reporting N/A seven times is honest, but it's still a check dressed as coverage.

The check itself stays — it's the anti-fabrication guard for anything landing state, and state-landing cases are the direction we're heading.

New: the scope lint

A case's files[] isn't documentation — it decides whether editing a file selects that case for a run. Prose the walk goes through but the case omits can be changed without the test covering it ever running again. Every case has such gaps today.

It can't be built by following links: the reachable set is far wider than the walked one, and a case declaring everything reachable would be selected for branches it deliberately stops before (session-loop.md, task-loop.md, conclude-discovery.md…). So it's computed from the walk's own Read calls, and reported as a finding about the case — correcting nothing, since which files belong is an authoring judgement.

Test plan

  • 86/86 across the prose suites, including five new cases for the lint: it names undeclared prose, stays quiet when the list is complete, ignores world-state reads and shell cats, and reports each file once
  • Corpus valid

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571
  28. fix(prose-tests): keep the walk's closing turn, and name the one live difference #572
  29. fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope #573 👈 current
  30. test(prose): declare the prose each case actually walks #574
  31. fix(investigation): the symptom interview never ran for work shaped in discovery #575
  32. feat(prose-tests): conduct — playing a person where a script cannot reach #576
  33. fix(prose-tests): what a write put in a file is evidence, not incident #577
  34. fix(investigation): the interview starts from the carrier, not over it #578
  35. test(prose): quick-fix scoping, the whole definition stage in one walk #579
  36. fix(prose-tests): the world log is the walker's record #580
  37. fix(prose-tests): a confirmed failure reports both runs' checks #581
  38. fix(prose-tests): each prescribed command runs as written #582
  39. fix(scoping): context gathering reads the whole carrier, asks only the gaps #583

This was referenced Jul 27, 2026
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 7331b27 to 3f2fced Compare July 27, 2026 14:33
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from f141d41 to 57c2940 Compare July 27, 2026 14:33
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 3f2fced to 5a4fc5c Compare July 27, 2026 14:35
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 57c2940 to 334973a Compare July 27, 2026 14:35
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 5a4fc5c to 948daa9 Compare July 27, 2026 14:37
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 334973a to 995afd9 Compare July 27, 2026 14:37
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 948daa9 to f46d9e3 Compare July 27, 2026 14:39
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 995afd9 to 963d371 Compare July 27, 2026 14:40
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from f46d9e3 to 5b0673f Compare July 27, 2026 14:42
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 963d371 to fd67ef4 Compare July 27, 2026 14:42
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 5b0673f to c2c7abf Compare July 27, 2026 14:44
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from fd67ef4 to 8e11bc3 Compare July 27, 2026 14:44
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from c2c7abf to 4de7f8e Compare July 27, 2026 14:46
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 8e11bc3 to 85650d2 Compare July 27, 2026 14:46
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 4de7f8e to 6e95a39 Compare July 27, 2026 14:48
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 85650d2 to f7a5638 Compare July 27, 2026 14:48
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 6e95a39 to b12965b Compare July 27, 2026 14:50
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from f7a5638 to e990814 Compare July 27, 2026 14:50
… fire, lint the scope

Three things the full run turned up, none of them failures.

The asserter had begun comparing the action record against the walker's
narrative unprompted, and reporting where they diverged. It is the right
instinct and it was nobody's instruction, so a run that depended on it
was depending on a coincidence. It is now required, with both directions
named: narrated but not recorded means the walker described something it
did not do, and the step resting on that claim fails; recorded but not
narrated is usually orientation and fails nothing, but is reported,
because a call that touched state and went unmentioned is the shape a
silent repair takes. The two streams exist so neither is taken on trust,
which only holds if they are actually set against each other.

engine_before_write was declared on every case and could only ever fire
on the three that write state. Reporting N/A seven times is honest but it
is still a check dressed as coverage, so it is declared only where a walk
can trip it. The check itself stays: it is the anti-fabrication guard for
anything that lands state, and cases that land state are the direction we
are heading.

The scope lint is new. A case's file list is not documentation — it
decides whether editing a file selects that case for a run, so prose the
walk goes through but the case omits can be changed without the test that
covers it ever running again. Every case has such gaps. The list cannot
be built by following links: the reachable set is far wider than the
walked one, and a case that declared everything reachable would be
selected for branches it deliberately stops before. So it is computed
from the walk's own Reads and reported as a finding about the case,
correcting nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from b12965b to f1cbc1d Compare July 27, 2026 14:51
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from e990814 to 6325906 Compare July 27, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant