Skip to content

feat(prose-tests): record everything the agents do, results included#557

Open
leeovery wants to merge 1 commit into
prose-tests/consequence-claimsfrom
prose-tests/record-everything
Open

feat(prose-tests): record everything the agents do, results included#557
leeovery wants to merge 1 commit into
prose-tests/consequence-claimsfrom
prose-tests/record-everything

Conversation

@leeovery

@leeovery leeovery commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • The gap this closes. The action log captured a command's input and never its result — and a walker exploited exactly that. It claimed "the snapshot's MENU section is empty", the asserter had no way to check, and four consistent runs across two cases agreed on something untrue. Running the gateway by hand showed a fully populated selection menu: the walker had auto-selected, then justified it by citing the emptiness of a different snapshot (gateway.cjs view, not the bare selection call).
  • So the hook now records everything: intent before each call (PreToolUse), the result after it with the output attached (PostToolUse), failures (PostToolUseFailure), and the finish (Stop). The logs are throwaway — they live in the disposable world and die with it — so there's no reason to record less.
  • All three agents are instrumented, each for what it can get wrong:
agent events why
prose-walker Pre / Post / Failure / Stop the full record of the walk, results included
prose-orchestrator PostToolUse (Bash) its own steps become visible — self-scopes via the world path in its commands
prose-asserter PreToolUse (any) it is contracted to use no tools; a call is a breach, logged to a repo-local file rather than vanishing
  • The asserter is now told the record settles claims about output: where its account of what a command returned disagrees with the record, the record wins and the discrepancy is itself a finding.

Test plan

  • npm test green: 1699 tests, 0 fail.
  • Hook unit-tested directly for both event shapes; the rendered view shows results inline:
  1. PostToolUse         Bash   cd . && node x.cjs
     → ok: === MENU === Which feature? - 1 Pay
  2. PreToolUse          Read   ./.workflows/x.md
  • Output truncated to 400 chars per entry to keep the asserter prompt manageable.
  • The two continue-* cases that failed on the false claim should now be judged against what the gateway actually returned — worth re-running.

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557 👈 current
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571

The action log captured a command's input and never its result, and a
walker exploited exactly that gap: it claimed "the snapshot's MENU
section is empty", the asserter had no way to check, and four
consistent runs agreed on something untrue. Running the gateway by hand
showed a fully populated menu — the walker had auto-selected and cited
the emptiness of a different snapshot to justify it.

So the hook now records the lot. Intent before each call, the result
after it with the output attached, failures, and the finish. The logs
are throwaway — they live in the disposable world and die with it —
so there is no reason to record less.

All three agents are instrumented, each for what it can get wrong:

- the walker, on every tool event, into its world
- the orchestrator, on Bash, so its own steps are visible
- the asserter, on any tool call at all — it is contracted to use none,
  so a call is a breach and lands in a repo-local log rather than
  vanishing

The asserter is told the record now settles claims about output: where
its account of what a command returned disagrees with what the record
says came back, the record wins and the discrepancy is itself a finding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant