Skip to content

fix(prose-tests): what a write put in a file is evidence, not incident#577

Merged
leeovery merged 1 commit into
mainfrom
fix/prose-writes-are-evidence
Jul 27, 2026
Merged

fix(prose-tests): what a write put in a file is evidence, not incident#577
leeovery merged 1 commit into
mainfrom
fix/prose-writes-are-evidence

Conversation

@leeovery

@leeovery leeovery commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Summary

The symptom-gathering case failed on a claim that could not be verified — not one that was wrong. The claim: the user's ignorance was recorded in the investigation file rather than left as template placeholders. The asserter's refusal was correct:

Nothing in the prompt shows the file's post-interview content. The Edit record truncates the replacement text… the world delta names the added path only, carrying no content. Not shown, and an unshown claim does not pass.

The cap was a read's rule applied to the wrong thing. Only a read is incidental — nothing is ever claimed about bytes a walker read back out of a file, so those stay trimmed at 400. But what a write put in a file settles every claim about the artifact a phase leaves behind, and in claims mode the tool response is the only copy of that content the asserter will ever see.

The discovery cases escaped this by luck alone: their content lands via workunit create, a Bash call, which already kept 10,000 characters. Anything written by Write/Edit was invisible.

Fix: Write, Edit, NotebookEdit responses now get the produced-output ceiling alongside Bash. Reads keep the hard trim.

Test plan

  • test-prose-record-action.cjs — 25/25, with two new cases: a long Edit newString survives untruncated and legible; a long Write the same
  • Full proof is the case re-run above this in the stack

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571
  28. fix(prose-tests): keep the walk's closing turn, and name the one live difference #572
  29. fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope #573
  30. test(prose): declare the prose each case actually walks #574
  31. fix(investigation): the symptom interview never ran for work shaped in discovery #575
  32. feat(prose-tests): conduct — playing a person where a script cannot reach #576
  33. fix(prose-tests): what a write put in a file is evidence, not incident #577 👈 current
  34. fix(investigation): the interview starts from the carrier, not over it #578
  35. test(prose): quick-fix scoping, the whole definition stage in one walk #579
  36. fix(prose-tests): the world log is the walker's record #580
  37. fix(prose-tests): a confirmed failure reports both runs' checks #581
  38. fix(prose-tests): each prescribed command runs as written #582
  39. fix(scoping): context gathering reads the whole carrier, asks only the gaps #583

This was referenced Jul 27, 2026
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from 2bcfad6 to 59e6967 Compare July 27, 2026 15:54
@leeovery
leeovery force-pushed the feat/prose-conduct branch from 78d87b0 to ac14bca Compare July 27, 2026 15:56
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from 59e6967 to 0ff3350 Compare July 27, 2026 15:56
@leeovery
leeovery force-pushed the feat/prose-conduct branch from ac14bca to 3a13c80 Compare July 27, 2026 15:57
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from 0ff3350 to 5f44049 Compare July 27, 2026 15:57
@leeovery
leeovery force-pushed the feat/prose-conduct branch from 3a13c80 to 99409d7 Compare July 27, 2026 15:58
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from 5f44049 to 7a33436 Compare July 27, 2026 15:58
@leeovery
leeovery force-pushed the feat/prose-conduct branch from 99409d7 to 9ec1b03 Compare July 27, 2026 15:59
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from 7a33436 to ef6682b Compare July 27, 2026 15:59
@leeovery
leeovery force-pushed the feat/prose-conduct branch from 9ec1b03 to 534c255 Compare July 27, 2026 16:00
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from ef6682b to e8e2840 Compare July 27, 2026 16:00
@leeovery
leeovery force-pushed the feat/prose-conduct branch from 534c255 to 0ae49af Compare July 27, 2026 16:01
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from e8e2840 to ff03d53 Compare July 27, 2026 16:01
@leeovery
leeovery force-pushed the feat/prose-conduct branch from 0ae49af to bbf15cb Compare July 27, 2026 16:02
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from ff03d53 to b006d1e Compare July 27, 2026 16:02
@leeovery
leeovery force-pushed the feat/prose-conduct branch from bbf15cb to 79f2273 Compare July 27, 2026 17:05
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from b006d1e to 84ade1a Compare July 27, 2026 17:05
@leeovery
leeovery force-pushed the feat/prose-conduct branch from 79f2273 to dc96a58 Compare July 27, 2026 17:06
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from 84ade1a to 555d8f6 Compare July 27, 2026 17:06
@leeovery
leeovery force-pushed the feat/prose-conduct branch from dc96a58 to bf3ebe0 Compare July 27, 2026 17:06
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from 555d8f6 to 5776dff Compare July 27, 2026 17:06
@leeovery
leeovery force-pushed the feat/prose-conduct branch from bf3ebe0 to 53ee32a Compare July 27, 2026 17:07
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from 5776dff to f9b57ff Compare July 27, 2026 17:07
@leeovery
leeovery force-pushed the feat/prose-conduct branch from 53ee32a to 9f4c13c Compare July 27, 2026 17:07
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from f9b57ff to 8477688 Compare July 27, 2026 17:07
@leeovery
leeovery changed the base branch from feat/prose-conduct to main July 27, 2026 17:08
The symptom-gathering case failed on a claim that could not be verified:
that the user's ignorance was recorded in the investigation file rather
than left as template placeholders. The asserter was right to refuse —
nothing in its prompt showed the file's post-interview content. The Edit
record truncated the written text at 400 characters, and the world delta
names added paths without carrying content.

That cap was a read's rule applied to the wrong thing. Only a read is
incidental: nothing is ever claimed about the bytes a walker read back
out of a file, so those are trimmed hard and stay trimmed. But what a
write put in a file settles every claim about the artifact a phase
leaves behind — and in claims mode the response is the only copy of that
content the asserter will ever see. The discovery cases escaped this by
luck alone: their content lands via workunit create, a Bash call, which
already kept 10,000 characters.

Write, Edit and NotebookEdit responses now get the produced-output
ceiling alongside Bash. Reads keep the hard trim.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the fix/prose-writes-are-evidence branch from 8477688 to 96ed05f Compare July 27, 2026 17:08
@leeovery
leeovery merged commit fea9bed into main Jul 27, 2026
@leeovery
leeovery deleted the fix/prose-writes-are-evidence branch July 27, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant