fix(prose-tests): keep the walk's closing turn, and name the one live difference#572
Open
leeovery wants to merge 1 commit into
Open
Conversation
… difference Both spec-entry cases failed on a claim about the handoff block their walks emitted correctly. The block was missing from the record, so the asserter had nothing to judge and rightly refused to pass an unevidenced step. Four walks, two cases, one cause. The cause is not the parser. Running the hook's own extraction over the settled transcript returns every turn including the one it missed, so the file simply did not hold that turn yet when the hook read it. This hook runs inside the stop sequence rather than after it: the agent has stopped thinking, and the runtime is still writing down the last thing it said. The turn is not lost, only not yet on disk — the stop payload carries it as `last_assistant_message`, which is the runtime handing it over at exactly the moment it is unreadable anywhere else. It is appended as the closing turn when it is not already there. Reading the transcript later instead, from the assert command, would also have worked, but it splits the record across two components and leaves the world holding a pointer rather than the walk; the world stays self-contained this way. Appending only when absent is not deduplication. Two identical turns that are genuinely in the transcript both survive, because a step that ran twice is something the asserter has to see. Truncation is reconsidered with it. The caps were never about disk — the logs are throwaway — but about the asserter's prompt, which every recorded action is read into. That justifies trimming a file body no claim ever rests on, and does not justify trimming evidence: the walk log now caps nothing at all, and a command's output gets a ceiling that bounds a runaway without ever reaching real output. Separately, the one genuine difference between a walk and a live session is written down rather than rediscovered each run. An inline directive is substituted when a skill loads live; a walk reads the prose as a file, so the walker takes the fallback the prose supplies for that case. Correct, expected, and not a marker. The asserter is also told what it costs — the primary arm is the one real sessions take and it is never exercised here, so a claim resting on the substituted value is unprovable rather than passed. It is named as the only such exemption: anything else resembling an environment quirk is a finding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
Both spec-entry cases failed on a claim about a handoff block their walks emitted correctly. The block was missing from the record, so the asserter had nothing to judge and rightly refused an unevidenced step. Four walks, two cases, one cause.
It isn't the parser. Running the hook's own extraction over the settled transcript returns every turn including the missing one — so the file didn't hold that turn yet when the hook read it.
The hook runs inside the stop sequence, not after it. The agent has stopped thinking; the runtime is still writing down the last thing it said.
The fix
The turn isn't lost, just not yet on disk — the stop payload carries it as
last_assistant_message, the runtime handing it over at exactly the moment it's unreadable anywhere else. It's appended as the closing turn when not already present.Reading the transcript later from
assertwould also have worked, but it splits the record across two components and leaves the world holding a pointer rather than the walk. This keeps the world self-contained.Appending-when-absent is not deduplication. Two identical turns genuinely in the transcript both survive — a step that ran twice is something the asserter must see.
Truncation, reconsidered alongside it
The caps were never about disk (the logs are throwaway) but about the asserter's prompt, which every recorded action is read into. That justifies trimming a file body no claim rests on; it doesn't justify trimming evidence.
The one known live difference, written down
An inline
!`command`directive is substituted when a skill loads live. A walk reads prose as a file, so the walker takes the fallback the prose supplies for exactly that case — correct, expected, and no longer reported as a marker.The asserter is also told what it costs: the primary arm is the one real sessions almost always take and it is never exercised here, so a claim resting on the substituted value is unprovable rather than passed. Named as the only such exemption — anything else resembling an environment quirk is a finding, not a licence.
Test plan
🤖 Generated with Claude Code
Stack