Skip to content

test(prose): give the eight read-only cases something that can fail#565

Open
leeovery wants to merge 1 commit into
feat/prose-walker-on-sonnetfrom
test/prose-claims-that-can-fail
Open

test(prose): give the eight read-only cases something that can fail#565
leeovery wants to merge 1 commit into
feat/prose-walker-on-sonnetfrom
test/prose-claims-that-can-fail

Conversation

@leeovery

@leeovery leeovery commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • Eight cases were enforcing nothing, all showing green. Every entry case declared exactly one check — engine_before_write — and those walks write nothing, so it reported "no workflow state was written" and passed regardless of what the prose did.
  • A check with nothing to examine now reports N/A, not PASS. A green tick for absence of coverage is precisely how this went unnoticed.
  • The eight get real declarations, drawn from claims they already make. Their "further claims" turn out to be not-called assertions almost word for word — "the entry skill records nothing", "an entry that only validates writes nothing", "the knowledge base is queried, never written to" — so each becomes a calls_exclude over topic start / topic reopen / manifest set / knowledge index. What each says it reads becomes a calls_include over the gate it must render.
  • calls_in_order is new — the one thing presence could never settle. A gate read after the arm it was supposed to select proves the arm was chosen some other way. Declared commands must appear as a subsequence, anything permitted between them.
  • P9 corrected. It demanded every case run until state landed. Wrong twice: it treated file changes as the only hard evidence when recorded actions are equally deterministic, and it would have pushed short entry walks into needing conversation stubs — trading live prose for hand-written fiction. A handoff is a fine ending. The rule is that every claim must have a deterministic answer; the stop point follows.

Verified against live walks

These declarations were derived from each case's claims and its skill's actual calls, then confirmed by running the whole corpus: all eight passed on claude-sonnet-5, so the gates, orderings and not-called assertions were right as written. No corrections were needed.

Test plan

  • Prose suites 75/75, including new coverage for ordering (in-sequence, out-of-sequence, gaps tolerated, missing call, minimum length) and for N/A vs PASS
  • Corpus valid
  • Full run: 8/8 PASS

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565 👈 current
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571

Every entry case declared exactly one check, `engine_before_write`, and
those walks write nothing — so it reported "no workflow state was
written" and passed, every time, whatever the prose did. Eight cases with
nothing enforceable behind them, all showing green.

Two changes so that cannot recur quietly.

A check with nothing to examine now reports N/A rather than PASS. A green
tick for absence of coverage is how this went unnoticed in the first
place; the distinction is now on the page.

And the eight get real declarations, drawn from the claims they already
make. Their "further claims" turn out to be not-called assertions almost
word for word — "the entry skill records nothing", "an entry that only
validates writes nothing", "the knowledge base is queried, never written
to" — so each becomes a calls_exclude over `topic start`, `topic reopen`,
`manifest set`, `knowledge index`. What each case says it reads becomes a
calls_include over the gate it must render.

calls_in_order is new, and is the one thing presence could never settle:
a gate read *after* the arm it was supposed to select proves the arm was
chosen some other way. Declared commands must appear as a subsequence,
with anything permitted in between.

P9 is corrected too. It demanded every case run until state landed, which
was wrong twice: it treated file changes as the only hard evidence, when
recorded actions are equally deterministic, and it would have pushed
short entry walks into needing conversation stubs — trading live prose
for hand-written fiction. A handoff is a fine ending. The rule is that
every claim must have a deterministic answer; the stop point follows from
that rather than being legislated.

The declarations are derived from the prose and not yet from a live run.
Any that prove wrong get corrected on this branch before it lands.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the test/prose-claims-that-can-fail branch from fd3126f to c4b2589 Compare July 26, 2026 19:39
This was referenced Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant