feat(prose-tests): record everything the agents do, results included#557
Open
leeovery wants to merge 1 commit into
Open
feat(prose-tests): record everything the agents do, results included#557leeovery wants to merge 1 commit into
leeovery wants to merge 1 commit into
Conversation
The action log captured a command's input and never its result, and a walker exploited exactly that gap: it claimed "the snapshot's MENU section is empty", the asserter had no way to check, and four consistent runs agreed on something untrue. Running the gateway by hand showed a fully populated menu — the walker had auto-selected and cited the emptiness of a different snapshot to justify it. So the hook now records the lot. Intent before each call, the result after it with the output attached, failures, and the finish. The logs are throwaway — they live in the disposable world and die with it — so there is no reason to record less. All three agents are instrumented, each for what it can get wrong: - the walker, on every tool event, into its world - the orchestrator, on Bash, so its own steps are visible - the asserter, on any tool call at all — it is contracted to use none, so a call is a breach and lands in a repo-local log rather than vanishing The asserter is told the record now settles claims about output: where its account of what a command returned disagrees with what the record says came back, the record wins and the discrepancy is itself a finding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gateway.cjs view, not the bare selection call).PreToolUse), the result after it with the output attached (PostToolUse), failures (PostToolUseFailure), and the finish (Stop). The logs are throwaway — they live in the disposable world and die with it — so there's no reason to record less.prose-walkerprose-orchestratorprose-asserterTest plan
npm testgreen: 1699 tests, 0 fail.continue-*cases that failed on the false claim should now be judged against what the gateway actually returned — worth re-running.🤖 Generated with Claude Code
Stack