test(prose): bugfix corpus — the investigation-centric surfaces#548
Open
leeovery wants to merge 11 commits into
Open
test(prose): bugfix corpus — the investigation-centric surfaces#548leeovery wants to merge 11 commits into
leeovery wants to merge 11 commits into
Conversation
Stage 3b of design/prose-tests.md. Three `crash-fix` worlds and four cases covering only what bugfix does differently from the linear pipeline: continue-bugfix routing a fresh bug to investigation, investigation entry seeding from the carrier without re-asking, the root-cause validation agent's dispatch → scan → incorporate lifecycle, and a spec sourced from the investigation rather than a discussion. Two framework extensions the corpus demanded: a free-text `stub:` section (agent-dispatching prose is walked with the walker playing the agent — the lifecycle is under test, not the agent's judgment) and a `json` state assertion (the agent row's closing status is checked in the cache store, not read out of a transcript). Both were proven by hand-walking the lifecycle in a live world first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lee's call, restored: cases were bundled several-per-file against what
he'd asked for. Each case now lives at tests/prose/{flow}/{case-id}.md
with the filename validated against the heading id, so the convention
can't drift; a flow's intro prose moves to its README.md, which the
loader ignores. 13 cases split out verbatim — no grammar, fixture, or
claim changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…wned stub triggers
Lee's review of the corpus, four corrections:
- Cases go flat — tests/prose/{case-id}.md, filename = id. Grouping by
work type organised the corpus around the build sequence, not the
prose under test, and no single-parent taxonomy fits cases spanning
skills. Selection always ran off `files:`, never the directory.
- The assertion DSL is deleted. The expectation is a whole committed
world; code diffs acted-vs-expected and the asserting agent classifies
each difference volatile or material. Nothing goes unchecked because
nobody thought to assert it, and there is no normalisation table or
engine-vs-model file split to maintain.
- Stub triggers move to the case. Binding a substitution to a moment is
what prevents reuse — the same content is wanted at different moments.
A stub is named content; the case says when. Trigger-less stubs fail
validation as arrange in disguise.
- The trace is granular. Coarse assertions let an agent silently
course-correct around broken prose; a step-by-step expected trace
turns that into a mismatch. The When stays coarse so the walker must
still derive the path from the prose. Adds a DEVIATION: marker.
Proven end to end: walked the validation lifecycle by hand in a live
world, diffed against the expected fixture — the sole difference was the
volatile agent-row timestamp.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lee reading the case file: a stub declared in `given` states a trigger for a moment the reader hasn't reached yet. The format contradicted itself — scripted `answers:` already sat in `when`, and a stub is the same kind of thing: harness input consumed during the act, one playing the user, the other the dispatched agent. `given` is now the world and nothing else; `when` carries the instruction plus everything fed in while it runs. Structure-only cases drop `given` entirely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… no CI claims Three of Lee's points from the TUI review: - `given` carries situational prose beside world_before — what the world represents, what has already happened, where the session stands. It reaches the walker as a SITUATION block. The root-cause case had this context smuggled into its `when`, mixing the situation with the act. - A stub trigger names an observable event, not a narrative moment: "when the engine records a dispatch of kind root-cause-validation", not "when the prose dispatches the validation agent". The loose form is only unambiguous while a walk contains exactly one dispatch, and it leaves nothing in the transcript for the asserter to check the substitution against. - The docs claimed CI. This project has no .github/workflows — the gate is `npm test`, run before every commit. Reworded throughout rather than implying anything runs on push. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lee's review, three faults in the flat-markdown shape: - Code regex-parsed values out of markdown, the exact pattern the analysis-state campaign eliminated. Now case.json holds what code branches on, prose files are read whole, recipes are required as modules. Nothing is parsed and prose never shares a file with code — which also deletes ~150 lines of parser. - world_before/world_after named foreign fixtures the case merely pointed at, scattering one test across six locations. A case now owns its worlds: fixture-state.cjs and assertion-state.cjs, split because they change for different reasons, snapshots beside them. Duplication accepted; reuse moves to mainlines/ as composable stage functions. - Naming follows the stage served — fixture, act, assert — so assert.md and assertion-state.cjs sit together by design. act.md and assert.md stay separate files because that IS the walker/asserter boundary, enforced by the filesystem rather than by convention. Adds a recipe-hash skip, without which per-case worlds would grow the gate linearly. Proven both ways: 0.09s when nothing moved, and a full rebuild reporting DRIFT when a mainline's content changed. Whole pipe re-proven live — world built, lifecycle walked by hand, delta against the assertion state was the volatile timestamp alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The case-directory restructure was meant to remove tests/prose/fixtures but the rm ran from the wrong working directory and silently deleted nothing, leaving 98 committed files that no code references. Spotted by Lee reading the PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…f code Lee's design, taken properly this time: /prose-test dispatches one prose-orchestrator per case, which owns the world, dispatches prose-walker and prose-asserter, escalates a FAIL to Opus, and returns a verdict alone. Transcripts never reach the main session — the reason this scales past a handful of cases. Standing instructions move to .claude/agents/prose-*.md; run.cjs emits only the per-case payload from tests/prose/prompts/. That removes the last 52 lines of prose living as JS string literals in the runner. Every agent is told in prose that its job is to run the test and report — never to fix, never to work out why. The first real runs showed an asserter wandering into engine source to explain a failure; tool restrictions can't be relied on to stop that, so the instruction is explicit in each definition. Also from those runs: the walker must quote every block the prose directs it to emit, since a skipped emission was otherwise invisible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n is worth nothing Three runs of evidence: a Sonnet walker performed the walk correctly every time but narrated it in summary, omitting the quoted evidence the asserter requires — even with a worked transcript example in front of it. The asserter was right to fail it; the tiering was wrong. Tiered models produced a DISAGREEMENT on every run, which is noise rather than signal, so the cheaper tier was a false economy. Walker and asserter both run on Opus, and the orchestrator never overrides a model — each definition names the model its result is trusted at. A FAIL now confirms by repetition from a fresh world rather than by promotion to a bigger model; a failure that doesn't reproduce reports as FLAKY. Also: renames test-prose-fixtures.cjs to test-prose-snapshots.cjs (it tests snapshots), and clears two stale fragments left in the contract by the Given-When-Then reshape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lee's call, recorded before it becomes tempting: deriving assert.md from a recorded walk would turn the corpus into approval testing — pinning what the prose currently does, defects included, so a mummified bug passes forever. The authoring effort is what finds pre-existing defects; it is the point, not the overhead. Snapshots stay generated because the engine authors them and drift is visible; expectations are written because only a person can say what correct means. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four real runs exposed the framework's worst failure mode: three walkers
skipped straight into the middle of a flow — one began at the gateway
call instead of initialisation, another at project-skills discovery,
past the entire entry skill. Both then "failed" at steps the walk never
reached. Lee's read is the decisive one: he has never seen the
project-skills prompt in real use, so a live session very likely
course-corrects past that guard exactly as our walker did — which means
every finding from those runs is unsafe until the walker is disciplined.
The walker now starts at the entry point the task names and performs
every step including ones whose effect the world already holds
("already booted", "the plan already exists" are named as the reasoning
that invalidates a walk). The asserter returns INVALID WALK when a
transcript opens mid-flow and judges nothing further; the orchestrator
retries once, then reports INVALID rather than FAIL.
A false finding is worse than no finding: it spends attention on prose
that was never exercised and teaches distrust of the runs that are real.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 26, 2026
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
crash-fixworlds (created / investigating / investigated) and four cases covering only what bugfix does differently from the linear pipeline — the delivery phases it shares with feature are already covered and deliberately not repeated: continue-bugfix routing a fresh bug to investigation, investigation entry seeding from the carrier without re-asking, the root-cause validation agent's full lifecycle (dispatch → scan → incorporate → verdict routing), and a specification whose source is the investigation rather than a discussion.### stubsection — prose that dispatches a background agent is walked with the walker playing the agent (writing its report at the path the engine's dispatch allocated, returning the stated result) rather than spending a real sub-agent analysing fixture content. What the case tests is the lifecycle around the agent; the agent's own judgment is not under test.json {file} {dot.path} {value}state assertion, so the agent row's closing status is verified deterministically in the cache store instead of being read out of a transcript by a grader.bugfix-investigatingworld exists precisely at the pre-validation moment, so the lifecycle case starts from a clean agent store.Test plan
npm testgreen: 1692 tests, 0 fail — corpus validation (now covering the stub section andjsongrammar) plus all nine fixtures rebuilding byte-identical (~25s).bugfix-investigatingworld, hand-walked the validation lifecycle exactly as the stub instructs (dispatch → stub report at the returned path → scan → incorporate), then ranrun.cjs grade— both state assertions passed, andrun.cjs promptwas checked to confirm stubs reach the walker whileexpect:claims never do.🤖 Generated with Claude Code
Stack