Skip to content

test(prose): claims assert consequences, not what was displayed#556

Open
leeovery wants to merge 1 commit into
prose-tests/discussion-conversation-stubfrom
prose-tests/consequence-claims
Open

test(prose): claims assert consequences, not what was displayed#556
leeovery wants to merge 1 commit into
prose-tests/discussion-conversation-stubfrom
prose-tests/consequence-claims

Conversation

@leeovery

Copy link
Copy Markdown
Owner

Summary

  • P4c was written and not applied. The runs then showed why it matters. root-cause-validation-clean-verdict failed only on "offers the choice and waits" and "announces that validation is running". implementation-picks-first-task came back FLAKY on "asks the question, gathering the answer" — evidenced in one run's narrative, absent from the other, with an identical world both times. Display claims don't merely fail, they fail intermittently, which is worse than failing.
  • Every display-only claim in the corpus is rewritten as something the record can prove:
was now
"offers the choice and waits" the scripted yes was consumed, so the run arm was taken — a skip would have returned with no dispatch
"announces running, then dispatches exactly one agent" one dispatch recorded, one report written, no second of either
"asks the question, gathering without acting on it" the answer was consumed there, and the entry wrote and committed no setup document
"renders the phase note and emits it as produced" renders it through the engine with the verb Starting — the render is an engine call; the emission isn't
  • Swept the whole corpus rather than only the two failing cases, so the rule is applied uniformly.

No prose touched — test authoring only.

Test plan

  • npm test green: 1699 tests, 0 fail. Corpus valid.
  • Both previously-failing cases should now be judged on evidence that exists — worth re-running to confirm, though a PASS then means the claims are provable rather than that anything in the prose changed.

🤖 Generated with Claude Code

@leeovery
leeovery force-pushed the prose-tests/discussion-conversation-stub branch from f83df8e to 9d3d1b3 Compare July 26, 2026 13:43
P4c was written and not applied. The runs then demonstrated why it
matters: root-cause-validation failed only on "offers the choice and
waits" and "announces that validation is running", and
implementation-picks-first-task came back FLAKY on "asks the question,
gathering the answer" — evidenced in one run's narrative, absent from
the other, with an identical world both times. Display claims don't
merely fail; they fail intermittently, which is worse than failing.

Each is rewritten as something the record can prove:

- "offers the choice and waits" → the scripted yes was consumed, so the
  run arm was taken; a skip would have returned with no dispatch
- "announces running, dispatches exactly one agent" → one dispatch
  recorded, one report written, no second of either
- "asks the question, gathering without acting" → the answer was
  consumed there, and no setup document was written or committed by the
  entry
- "renders the phase note and emits it as produced" → renders it through
  the engine with the verb Starting; the render is an engine call, the
  emission is not

No prose touched — this is test authoring only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from b55f4f4 to 4790742 Compare July 26, 2026 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant