feat(prose-tests): decide in code what an agent should not be deciding

[leeovery]

Summary

• The world delta proves the outcome, not that it was earned. A walk that ignored every instruction and wrote the expected files directly lands the same state as one that followed the prose. Only the order of what it did separates them — recorded all along, never read.
• Cases now declare invariants checked in code before the asserter sees anything:
  • engine_before_write — no .workflows write without first consulting the engine. A skill learns what it knows through the engine, so a write preceding every call didn't come from following the prose.
  • calls_include / calls_exclude — commands the walk must, or must not, have run.
• A computed verdict can't drift, and the asserter is told it may not overturn one.
• Claims move to checks where they can. implementation-picks-first-task no longer asks an agent whether a task was started; it asserts task start and task next are absent from the record.
• Not a pinned call sequence — deliberately. Recording and replaying one would freeze whatever the walker happened to do, which is how a test starts certifying broken prose instead of catching it. Declarations are hand-written like the rest of a case.

Test plan

• node --test tests/scripts/test-prose-invariants.cjs — 20/20 covering each check, the declaration surface, and validation of malformed declarations
• Full prose gate 60/60
• Integration-checked against a fabricated walk: an action log containing a bare .workflows write and nothing else produces

  FAIL  engine_before_write — wrote ./.workflows/pay/discussion/pay.md having never called the engine
  FAIL  calls_include — never ran: task init
  PASS  calls_exclude — ran none of: task start, task next
  

  which is precisely the walk the world delta cannot distinguish from a good one.

🤖 Generated with Claude Code

Stack

1. docs(design): prose-tests programme design log #544
2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
3. test(prose): feature happy-path corpus — five worlds, seven cases #546
4. test(prose): bugfix corpus — the investigation-centric surfaces #548
5. test: retry recursive teardown removals — kill a class of phantom failures #549
6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
7. docs: a contributing page for working on the system #551
8. fix(entry-skills): every handoff arm says to invoke the skill #552
9. fix(implementation): environment setup belongs to the setup reference alone #553
10. fix(prose-tests): the asserter is told which substitutions were armed #554
11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
12. test(prose): claims assert consequences, not what was displayed #556
13. feat(prose-tests): record everything the agents do, results included #557
14. fix(discussion-entry): the handoff reports the source it actually had #558
15. fix(prose-tests): the stop hook records, and names the model that walked #559
16. fix(prose-tests): command output was never actually recorded #560
17. feat(prose-tests): judge the walk as told, not the summary returned #561
18. feat(prose-tests): decide in code what an agent should not be deciding #562 👈 current
19. test(prose): a case starts where a session starts #563
20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
21. test(prose): give the eight read-only cases something that can fail #565
22. test(prose): only walks that can be observed, and checks that survive the trip #566
23. fix(prose-tests): the verdict names only the model the record names #567
24. test(prose): discovery, walked to the point where work first exists #568
25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
26. docs(conventions): a step whose reference routes every exit still signposts #570
27. test(prose): discovery's epic arm, to the same durability boundary #571