Skip to content

fix(prose-tests): the asserter is told which substitutions were armed#554

Open
leeovery wants to merge 1 commit into
fix/environment-check-gathers-onlyfrom
prose-tests/asserter-knows-stubs
Open

fix(prose-tests): the asserter is told which substitutions were armed#554
leeovery wants to merge 1 commit into
fix/environment-check-gathers-onlyfrom
prose-tests/asserter-knows-stubs

Conversation

@leeovery

Copy link
Copy Markdown
Owner

Summary

  • The gap the recording created. The action log shows a stub as the walker writing a report an agent would have produced. An asserter that has never heard of the stub reads that as a missing agent dispatch — root-cause-validation-clean-verdict failed 2/5 on precisely that, having behaved correctly throughout. Armed stubs now reach the asserter with their triggers and descriptions, as a distinct prompt section (P4b).
  • An authoring rule from the same round (P4c). A claim about something displayed — "offers the choice and waits" — can never be evidenced by the action log, because emitting text is not a tool call. It rests on the walker's narrative, the one piece of evidence it authors itself. Most display claims have a consequence that can be asserted: the answer consumed, the arm taken, the state produced. Prefer the consequence.
  • Deletes smoke-boot-in-world. It absorbed three authoring fixes in one day — a contradictory stop point, then claims about renderings after that stop, then a scope anchor (#Boot) fighting its own claims — for a framework smoke test whose job start-lists-active-work already does properly and passes 5/5.

Test plan

  • npm test green: 1698 tests, 0 fail (one fewer suite entry: the deleted case's snapshot check).
  • Corpus revalidates: 12 cases, 1 stub.
  • root-cause-validation-clean-verdict should now be judged correctly rather than failing on its own substitution — worth a re-run once this lands.

🤖 Generated with Claude Code

@leeovery
leeovery force-pushed the fix/environment-check-gathers-only branch from f4c27d7 to 42108f0 Compare July 26, 2026 12:52
@leeovery
leeovery force-pushed the prose-tests/asserter-knows-stubs branch from 872d47c to db46617 Compare July 26, 2026 12:53
@leeovery
leeovery force-pushed the fix/environment-check-gathers-only branch from 42108f0 to 4c916fa Compare July 26, 2026 13:37
@leeovery
leeovery force-pushed the prose-tests/asserter-knows-stubs branch from db46617 to e81714b Compare July 26, 2026 13:37
@leeovery
leeovery force-pushed the fix/environment-check-gathers-only branch from 4c916fa to 1976e37 Compare July 26, 2026 13:43
First round with recording live exposed the gap: the action log shows a
stub as the walker writing a report an agent would have produced, and an
asserter that has never heard of the stub reads that as a missing
dispatch. root-cause-validation failed 2/5 on exactly that, having
behaved correctly throughout. Armed stubs now reach the asserter with
their triggers and descriptions.

Also from the same round, as an authoring rule (P4c): a claim about
something *displayed* can never be evidenced by the action log, because
emitting text is not a tool call — it rests on the walker's narrative,
the one piece of evidence it authors itself. Most display claims have a
consequence that can be asserted instead: the answer consumed, the arm
taken, the state produced.

And smoke-boot-in-world is deleted. It absorbed three authoring fixes —
a contradictory stop point, claims about post-stop renderings, then a
scope anchor fighting its own claims — for a framework smoke test whose
job start-lists-active-work already does properly, and passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the prose-tests/asserter-knows-stubs branch from e81714b to f4cfb5a Compare July 26, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant