Skip to content

fix(prose-tests): keep the walk's closing turn, and name the one live difference#572

Open
leeovery wants to merge 1 commit into
test/prose-discovery-epic-armfrom
fix/prose-closing-turn-and-known-difference
Open

fix(prose-tests): keep the walk's closing turn, and name the one live difference#572
leeovery wants to merge 1 commit into
test/prose-discovery-epic-armfrom
fix/prose-closing-turn-and-known-difference

Conversation

@leeovery

@leeovery leeovery commented Jul 27, 2026

Copy link
Copy Markdown
Owner

The failure

Both spec-entry cases failed on a claim about a handoff block their walks emitted correctly. The block was missing from the record, so the asserter had nothing to judge and rightly refused an unevidenced step. Four walks, two cases, one cause.

It isn't the parser. Running the hook's own extraction over the settled transcript returns every turn including the missing one — so the file didn't hold that turn yet when the hook read it.

The hook runs inside the stop sequence, not after it. The agent has stopped thinking; the runtime is still writing down the last thing it said.

The fix

The turn isn't lost, just not yet on disk — the stop payload carries it as last_assistant_message, the runtime handing it over at exactly the moment it's unreadable anywhere else. It's appended as the closing turn when not already present.

Reading the transcript later from assert would also have worked, but it splits the record across two components and leaves the world holding a pointer rather than the walk. This keeps the world self-contained.

Appending-when-absent is not deduplication. Two identical turns genuinely in the transcript both survive — a step that ran twice is something the asserter must see.

Truncation, reconsidered alongside it

The caps were never about disk (the logs are throwaway) but about the asserter's prompt, which every recorded action is read into. That justifies trimming a file body no claim rests on; it doesn't justify trimming evidence.

  • walk log — caps nothing at all
  • command output — 2000 → 10000, a ceiling that bounds a runaway without ever reaching real output
  • file-read bodies — 400 unchanged, the one place content is genuinely incidental

The one known live difference, written down

An inline !`command` directive is substituted when a skill loads live. A walk reads prose as a file, so the walker takes the fallback the prose supplies for exactly that case — correct, expected, and no longer reported as a marker.

The asserter is also told what it costs: the primary arm is the one real sessions almost always take and it is never exercised here, so a claim resting on the substituted value is unprovable rather than passed. Named as the only such exemption — anything else resembling an environment quirk is a finding, not a licence.

Test plan

  • 81/81 across the prose suites, including four new cases: the closing turn arrives and in order, a turn the transcript already caught isn't double-recorded, a genuine repeat survives, and the walk log truncates nothing
  • Both spec-entry cases want a re-run to confirm they now pass on their own evidence

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571
  28. fix(prose-tests): keep the walk's closing turn, and name the one live difference #572 👈 current
  29. fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope #573
  30. test(prose): declare the prose each case actually walks #574
  31. fix(investigation): the symptom interview never ran for work shaped in discovery #575

… difference

Both spec-entry cases failed on a claim about the handoff block their
walks emitted correctly. The block was missing from the record, so the
asserter had nothing to judge and rightly refused to pass an unevidenced
step. Four walks, two cases, one cause.

The cause is not the parser. Running the hook's own extraction over the
settled transcript returns every turn including the one it missed, so the
file simply did not hold that turn yet when the hook read it. This hook
runs inside the stop sequence rather than after it: the agent has stopped
thinking, and the runtime is still writing down the last thing it said.

The turn is not lost, only not yet on disk — the stop payload carries it
as `last_assistant_message`, which is the runtime handing it over at
exactly the moment it is unreadable anywhere else. It is appended as the
closing turn when it is not already there. Reading the transcript later
instead, from the assert command, would also have worked, but it splits
the record across two components and leaves the world holding a pointer
rather than the walk; the world stays self-contained this way.

Appending only when absent is not deduplication. Two identical turns that
are genuinely in the transcript both survive, because a step that ran
twice is something the asserter has to see.

Truncation is reconsidered with it. The caps were never about disk — the
logs are throwaway — but about the asserter's prompt, which every
recorded action is read into. That justifies trimming a file body no
claim ever rests on, and does not justify trimming evidence: the walk log
now caps nothing at all, and a command's output gets a ceiling that
bounds a runaway without ever reaching real output.

Separately, the one genuine difference between a walk and a live session
is written down rather than rediscovered each run. An inline directive is
substituted when a skill loads live; a walk reads the prose as a file, so
the walker takes the fallback the prose supplies for that case. Correct,
expected, and not a marker. The asserter is also told what it costs — the
primary arm is the one real sessions take and it is never exercised here,
so a claim resting on the substituted value is unprovable rather than
passed. It is named as the only such exemption: anything else resembling
an environment quirk is a finding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant