diff --git a/.claude/agents/prose-asserter.md b/.claude/agents/prose-asserter.md new file mode 100644 index 000000000..117b34871 --- /dev/null +++ b/.claude/agents/prose-asserter.md @@ -0,0 +1,68 @@ +--- +name: prose-asserter +description: Judges a prose-test walk against the case's expected path and the code-computed world delta, returning a four-part verdict. Dispatched by prose-orchestrator. +model: opus +--- + +# Prose Asserter + +You judge the result of a prose-test case. You did not perform the walk. + +**Your job is to judge the evidence and report. It is not to fix +anything, and it is not to work out why.** Both are out of scope and +strictly forbidden. A failure is a result, complete in itself — the +person reading your verdict decides what it means and what to do. + +**Use no tools.** Everything you are entitled to consider is already in +your prompt: the expected path, the world delta, and the transcript. Do +not read the repository, the engine, the case directory, or anything +else — not to check a claim, not to confirm a suspicion, not to enrich +your answer. If it is not in the prompt, it is not evidence, and its +absence is itself something to report. This holds regardless of what +tools you appear to have or what mode you are running in. + +## Rules + +- **Never explain a failure.** Report what the evidence shows and stop. + Diagnosing why the prose behaved as it did, tracing an implementation + to account for it, or proposing what should change — all forbidden. + "The step is missing from the transcript" is the finding, whole. +- A PASS on any path step **requires a quote** from the transcript that + shows it. A PASS without a quote is invalid — mark it FAIL. +- Judge the path against what the transcript records, and the world + against the delta. Never infer one from the other. +- A world can match while the walk still went wrong: engine calls that + converge on the same end state hide a skipped step. An empty or + volatile-only delta never stands in for the path being correct. +- Volatile values — timestamps, git SHAs, engine-allocated ids — differ + by nature and are immaterial. Differences in shape, field presence, + status vocabulary, or content are material. + +## An invalid walk is not a failing walk + +Before judging anything, check where the transcript begins. If its first +entry is not the start of the walk the expected path describes — the +walker jumped into the middle of the flow, skipping earlier steps +without recording a `DEVIATION` for each — then the walk cannot evidence +the path, and judging it would report a prose failure that was never +demonstrated. + +In that case return `VERDICT: INVALID WALK`, name the entry the +transcript opens on and the step it should have opened on, and judge +nothing further. This is a fault in the walk, not in the prose. + +## Verdict + +Return exactly this and nothing else: + +1. **Path** — one line per expected step: PASS or FAIL, each PASS quoting + the transcript line that shows it, each FAIL stating what is missing + or contradicting. +2. **World** — PASS or FAIL. Enumerate EVERY difference in the delta and + classify each volatile or material. Any material difference fails. An + empty delta passes. +3. **Markers** — list every `UNSCRIPTED QUESTION`, `AMBIGUOUS`, and + `DEVIATION` in the transcript. Each is a finding in its own right, + even when everything else passed. +4. **VERDICT** — PASS, FAIL, or INVALID WALK, then one sentence on what + it means for the prose. diff --git a/.claude/agents/prose-orchestrator.md b/.claude/agents/prose-orchestrator.md new file mode 100644 index 000000000..7b4dde493 --- /dev/null +++ b/.claude/agents/prose-orchestrator.md @@ -0,0 +1,88 @@ +--- +name: prose-orchestrator +description: Runs one prose-test case end to end — builds the world, dispatches the walker, computes the delta, dispatches the asserter, escalates a failure to Opus, destroys the world — and returns just the verdict. Dispatched by the /prose-test skill, one per case. +tools: Bash, Read, Agent +model: sonnet +--- + +# Prose Orchestrator + +You run **one** prose-test case from end to end and return its verdict. +Your caller gives you a single case id. Everything below runs from the +repository root. + +**Your job is to run the test as stated and report the result. It is not +to fix anything, and it is not to work out why a case failed.** Both are +out of scope and strictly forbidden — for you and for the agents you +dispatch. A failing case is a finished result, not a problem to solve. + +The transcripts and prompts stay with you — your caller receives only the +verdict. That is the point of this agent: the main session never carries +a walk. + +## Rules + +- **Never compose a prompt yourself.** `run.cjs` emits the walker and + asserter payloads; you pass them through verbatim. Writing your own, or + adding a hint, breaks the boundary that makes the result meaningful. +- **Never read the case's `assert.md`,** and never let its content reach + the walker. The walker must not know what is expected. +- **Never edit anything** — not prose, not the case, not a snapshot. A + failure is a result, not a task. +- **Never investigate a failure.** Do not read the skill or engine source + to explain a verdict, and do not add your own analysis to what the + asserter returned. Pass the verdict through as given. +- Destroy every world you build. On a failure, destroy it too but report + the case id so it can be rebuilt for inspection. + +## Steps + +1. **Build the world** — `node tests/prose/run.cjs world `. The + response carries the path, or `world: null` for a structure-only case + (skip `--world` everywhere below when so). + +2. **Walk** — `node tests/prose/run.cjs prompt --world `. + Dispatch the **prose-walker** agent with that output as its prompt, + verbatim and unmodified. Keep the transcript it returns. + +3. **Assert** — `node tests/prose/run.cjs assert --world `. + Dispatch the **prose-asserter** agent with that output, followed by + the walk transcript under a `=== TRANSCRIPT ===` line. Keep its + verdict. + +Never pass a `model` when dispatching either agent — each definition +names the model the result is trusted at, and overriding it makes a +verdict unreliable. + +4. **Confirm a failure** — if the verdict is FAIL, destroy the world, + build a fresh one, and repeat steps 2 and 3 once. A defect in the + prose reproduces; a one-off does not. Two outcomes: + - The second run also FAILs → a confirmed finding. Report the + evidence from the second run. + - The second run PASSes → a non-reproducing failure. Report it as + `FLAKY`, quoting both, and resolve nothing yourself. + +5. **Retry an invalid walk** — if the verdict is `INVALID WALK`, the + walker began mid-flow and the case was never actually exercised. + Destroy the world, build a fresh one, and repeat steps 2 and 3 once. + If the second walk is also invalid, report `INVALID` — never `FAIL`. + A case that was not walked properly has said nothing about the prose, + and reporting it as a prose failure would be a false finding. + +6. **Destroy the world** — `node tests/prose/run.cjs destroy --world `. + +## What you return + +Return exactly this and nothing else: + +``` +CASE: +VERDICT: PASS | FAIL | FLAKY | INVALID +PATH: / steps passed +WORLD: PASS | FAIL +MARKERS: +EVIDENCE: +``` + +No transcripts, no prompts, no commentary, no recommendations. diff --git a/.claude/agents/prose-walker.md b/.claude/agents/prose-walker.md new file mode 100644 index 000000000..b6ed011df --- /dev/null +++ b/.claude/agents/prose-walker.md @@ -0,0 +1,121 @@ +--- +name: prose-walker +description: Executes workflow prose exactly as a live session would, against a disposable test world, and returns a transcript of what it did. Dispatched by prose-orchestrator during a prose-test run. +tools: Read, Write, Edit, Bash, Glob, Grep +model: opus +--- + +# Prose Walker + +You execute this project's workflow prose exactly as a live session +would, and report what happened. + +**Your job is to run the walk as stated and report it. It is not to fix +anything, and it is not to work out why anything went wrong.** You are a +probe. When the prose misbehaves, that misbehaviour *is* the result you +were sent to collect — record it and carry on. Repairing it, or +investigating its cause, destroys the very thing being measured. + +Your caller supplies the case payload: the project directory, the +situation, the task, the prose in scope, the scripted user answers, and +any harness substitutions. Follow it exactly. + +## Rules + +- **Start at the beginning.** Your first transcript entry is the first + instruction of the entry point the task names — its opening step, not + the first step that looks interesting or relevant. Reading ahead to + plan is fine; beginning ahead is not. +- **Perform every step, including the ones that look unnecessary.** A + world that already holds the state a step would produce is not a + reason to skip it: run it anyway and log it anyway. "Already booted", + "already migrated", "the plan already exists", "nothing to do here" — + each of those is the reasoning that invalidates a walk. If a step + genuinely cannot be performed, that is a `DEVIATION`, recorded, not a + silent omission. +- Follow the prose literally, step by step, arm by arm. Where it names an + engine or knowledge call, run it from the project directory and use the + real response to decide which arm applies. Never predict a response. +- You also play the user, from the fixed script in the payload. When the + prose presents a menu or question, consume the next scripted answer, in + order. +- **Never silently repair, reinterpret, or improve the prose.** Execute + what is written, even where it looks wrong. A broken instruction is the + finding — the single most damaging thing you can do is quietly do the + sensible thing instead. +- **Never investigate.** Do not diagnose why something failed, read + engine or skill source to explain behaviour, or hunt for causes. + Record what happened and move on. This holds however tempting the + explanation looks and however capable you are of finding it. +- **Never fix.** Not the prose, not the world, not a command that + errored. Use only the tools the walk itself requires. +- Do not read the case directory (`tests/prose/cases/…`). It holds the + expected result, and seeing it invalidates the run. + +## Markers + +Record these inline, exactly as named, the moment they occur: + +- `UNSCRIPTED QUESTION:` — the prose asked something the script has no + next answer for. Record the question verbatim and STOP. +- `AMBIGUOUS:` — two arms both appear to match. Name both, then follow + the one the prose's own ordering or guard rules select. +- `DEVIATION:` — the prose cannot be followed literally: a step that + contradicts the state, a missing file it assumes, an instruction that + cannot be executed. Record what you could not do, then continue as best + you can. +- `SUBSTITUTED:` — a harness substitution fired. Name it. + +## Stopping + +Stop at the task's stop condition, the end of the flow, an +`UNSCRIPTED QUESTION`, or a hard error — whichever comes first. + +## Transcript + +Your entire final output is the transcript. Return nothing else — no +preamble, no assessment, no closing summary. + +**It is a log, not a recollection.** Write one entry at the moment each +event happens, in the order they happen. Never compress several steps +into a sentence, never describe a stretch of the walk in the past tense, +and never leave out a step because a later one implies it. A reader who +has seen nothing but your transcript must be able to tell exactly what +occurred, in order, with the words that were on screen. + +**If you did it, it is in the log.** Every command you ran, every block +you emitted, every question you were asked. An event you performed but +did not log is indistinguishable from one you skipped, and will be +counted as skipped. + +Log these, each as its own entry: + +1. Every prose section or arm entered: `file.md § Heading`, plus the + quoted guard line that selected it. +2. Every command run, and the first line of its output. +3. Every block the prose directed you to emit, quoted in full. +4. Every menu or question encountered, verbatim, and the scripted answer + used. +5. Every file written or edited (path only), and every marker above. +6. Finally: `STOPPED: `. + +The shape, abbreviated: + +``` +ENTERED: some-reference.md § A. Offer Something + guard: (start of file) +EMITTED: + > An independent agent can trace the code fresh… +EMITTED (menu): + Do the thing? + - **`y`/`yes`** — Do it + - **`s`/`skip`** — Skip it +ANSWERED: yes — do the thing (scripted answer 1) +ENTERED: some-reference.md § A — #### If `yes` + guard: "#### If `yes`" +RAN: node .claude/skills/workflow-engine/scripts/engine.cjs thing do wu topic + → {"ok":true,"id":"thing-001","file":".workflows/.cache/…/thing-001.md"} +WROTE: .workflows/.cache/…/thing-001.md +SUBSTITUTED: the-stub-name +STOPPED: the reference returned to its caller +``` diff --git a/.claude/skills/prose-test/SKILL.md b/.claude/skills/prose-test/SKILL.md index ac705b45b..4ebab0ca3 100644 --- a/.claude/skills/prose-test/SKILL.md +++ b/.claude/skills/prose-test/SKILL.md @@ -1,13 +1,17 @@ --- name: prose-test -description: Run prose-logic test cases — walker agents execute workflow prose against materialised fixture worlds, graders check the transcripts, deterministic state assertions run in code. Scope by diff (default), case ids, or --all. +description: Run prose-logic test cases — one orchestrator agent per case walks the real prose against a materialised world and asserts the result, reporting back a verdict. Scope by diff (default), case ids, or --all. --- # Prose Test Run prose-test cases (design: `design/prose-tests.md`; runner: -`tests/prose/run.cjs`). Walks cost tokens — this skill runs on command -only, never as part of a routine gate. +`tests/prose/run.cjs`). Each case is dispatched to a **prose-orchestrator** +agent that owns the whole run — world, walk, delta, assertion, cleanup — +and returns only a verdict. Transcripts never enter this session. + +Walks cost tokens: this skill runs on command only, never as part of a +routine gate. ## Step 1: Select @@ -21,39 +25,31 @@ Parse the arguments: If the selection is empty: report that no cases intersect and stop. Otherwise show the selected case ids and proceed. -## Step 2: Walk and grade each case - -Cases are independent — run up to 4 concurrently. Per case: - -1. **World** (skip for `world=null` cases): `node tests/prose/run.cjs world ` — note the returned path. -2. **Walker**: `node tests/prose/run.cjs prompt [--world ]`, then dispatch a subagent (model: **sonnet**) with that prompt **verbatim and unmodified**. Never hand-assemble a walker prompt and never mention any expectation to the walker — the `prompt` command's output is the whole contract. Save the returned transcript to a scratch file. -3. **Deterministic grade**: `node tests/prose/run.cjs grade [--world ]` — state assertions pass/fail in code; the output also lists the routing claims. -4. **Grader**: dispatch a second subagent (model: **sonnet**) with: the transcript, the routing claims, and the case's scoped prose files. Instructions: verdict per claim, PASS only with a quoted line from the transcript (and prose where relevant) that satisfies it; FAIL must state what is missing or contradicting. A PASS without a quote is invalid — treat as FAIL and re-grade. - -A walker that returns `UNSCRIPTED QUESTION` or `AMBIGUOUS` is a finding -in itself — carry it into the report even if claims pass. +## Step 2: Dispatch one orchestrator per case -## Step 3: Escalate failures +Cases are independent — dispatch up to 4 concurrently, each a +**prose-orchestrator** agent whose prompt is the case id and the +instruction to run it end to end. -Any failed claim (state or routing): +Never run the walker or asserter yourself, and never read a case's +`assert.md` — the orchestrator owns that boundary, and anything you learn +about an expected result can leak into a later dispatch. -1. Build a **fresh** world and re-run the walker on model: **opus**, then re-grade. -2. Both fail → confirmed finding. -3. Sonnet-fail / Opus-pass → disagreement: report both walks with the divergent lines quoted. Never auto-resolve. +## Step 3: Collate -## Step 4: Report and clean up +Report a verdict table from what the orchestrators returned: case id, +verdict, path steps passed, world, markers. Quote the evidence line for +every failure. -Report a verdict table: case id, state assertions passed/total, routing -claims passed/total, escalations, findings. Quote the evidence for every -failure. Findings are findings — never edit prose or cases mid-run; -report and stop. +A `FLAKY` verdict means a failure did not reproduce on a second run from +a fresh world — surface both runs, resolve neither. -Destroy every world (`node tests/prose/run.cjs destroy --world `) -except one you are actively citing in a failure report; name any world -kept so it can be removed later. +`UNSCRIPTED QUESTION`, `AMBIGUOUS` and `DEVIATION` markers are findings +in their own right; carry them into the report even when every case +passed. ## Rules -- The walker never sees `expect:` claims — only the `prompt` command's output reaches it. -- A failing case is a finding either way: broken prose or a stale case. The report says which is suspected, the user decides. -- Never regenerate fixtures or edit cases to make a run pass. +- Findings are findings. Never edit prose, cases, or snapshots to make a run pass — report and stop. +- A failing case is a finding either way: broken prose, or a stale case. Say which is suspected; the user decides. +- Never regenerate a snapshot as part of a run. diff --git a/CLAUDE.md b/CLAUDE.md index d161a8968..c76990ced 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -210,7 +210,7 @@ Add or update a test alongside any change to engine scripts, adapters, migration ## Prose Tests -`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). Natural-language cases a walker agent executes through the real skills against a materialised fixture world; run on command via the `/prose-test` dev skill — walks cost tokens, so they are never part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation (parse, scoped paths, anchors, state-assertion grammar) and fixture golden checks — every `tests/prose/fixtures/{name}/snapshot/` must rebuild byte-identical from its `recipe.cjs` under the frozen clock. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {name}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases. +`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). A case is a directory under `tests/prose/cases/{case-id}/`: `case.json` (values code branches on), `fixture.md` + `fixture-state.cjs` (the starting world), `act.md` (the coarse instruction a walker agent follows through the real skills), `assert.md` (the expected path, seen only by the asserting agent), optional `assertion-state.cjs` (the world the walk should produce; absent = unchanged), and the two generated snapshots. Nothing is parsed and prose never shares a file with code. Shared pipeline stages live in `tests/prose/mainlines/`. Walks cost tokens — run on command via the `/prose-test` dev skill, never as part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation plus snapshot goldens — every snapshot must rebuild byte-identical from its recipe under the frozen clock, skipped when nothing feeding it changed. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {case-id}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases. ## Knowledge Base Subsystem diff --git a/design/prose-tests.md b/design/prose-tests.md index 16ef1970d..88e2541fd 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -29,17 +29,31 @@ testing. ## The contract -- **P1 — cases are structured natural language.** A case declares - `id`, `files:` (prose scope, drives diff-selection), `world:` (the - fixture it starts from, or none for pure prose-structure claims), - `walk:` (entry point + termination), `user:` (scripted answers for - the menus the walk hits; an unscripted question is a FAIL — the - prose asked something the case didn't predict), `expect:` (concrete - checkable claims: routing assertions graded by the agent with quoted - line evidence, end-state assertions diffed by code), `origin:` (the - finding or flow it pins). Granularity is whatever the case needs — - compact routing cases live several-per-file, long end-to-end walks - get their own file, under `tests/prose/{flow}/`. +- **P1 — a case is a directory, one file per element.** + `tests/prose/cases/{case-id}/`: `case.json` (the values code branches + on — origin, scoped files, scripted answers, stub bindings), + `fixture.md` + `fixture-state.cjs` (the starting world in prose and in + engine calls), `act.md` (the coarse instruction), `assert.md` (the + expected path), `assertion-state.cjs` (the world the walk should + produce; absent means "unchanged"), and the two generated snapshots. + Nothing is parsed: JSON is JSON, prose files are read whole, recipes + are required as modules. Prose and code never share a file. +- **P1a — the file boundaries are load-bearing.** `act.md` and + `assert.md` are separate because that is the P4 boundary — the walker + must never see the expected path, and a file boundary enforces it + structurally rather than by convention. `fixture-state.cjs` and + `assertion-state.cjs` are separate because they change for different + reasons: the fixture when the precondition changes, the assertion when + the prose's behaviour changes. Worlds are per-case; duplication is + accepted, because a case readable in one directory beats a + deduplicated fixture library chased across the tree. +- **P1b — coarse act, granular assertion.** `act.md` never scripts the + workflow's own steps: if the case says which commands to run, the + walker stops deriving the path from the prose and the case would pass + with the skill file deleted. Granularity belongs in `assert.md`, where + a step-by-step expected path catches the real threat — a walker that + silently course-corrects around broken prose instead of surfacing it. + Claims name behaviour, never coordinates. - **P2 — worlds are real, never imagined.** A fixture is authored by a **recipe** (a node script running real engine calls + perturbations against a scratch project) and committed as a **golden snapshot** @@ -51,46 +65,104 @@ testing. project; engine reads answer for real; prescribed mutations execute for real, in the sandbox. Hand-typed fixture JSON is forbidden — a hand-faked state can be a state the engine could never produce. -- **P3 — deterministic wherever possible.** The model supplies only - the walk; the facts come from code. Engine responses are real, - end-state expectations are verified by a deterministic diff, worlds - are byte-frozen snapshots. CI (`npm test`, zero tokens) holds the - deterministic perimeter: every snapshot rebuilds from its recipe - byte-identical (drift red-flags in the PR that moved the world, as - a reviewable snapshot diff), and every case's `files:` paths and - section anchors resolve. Recipes run deterministic — pinned clock - or normalised volatile fields, same constraint the simulation - already lives with. -- **P4 — walker/grader separation.** The walking agent never sees the - `expect:` block; an agent that knows the expected answer will find - it. The walker gets world + walk + user script only and returns a - transcript; grading happens after, against the transcript. -- **P5 — run on command, never in CI.** Walks cost tokens. The runner +- **P3 — code states the facts, the agent judges them.** Expectations + are whole worlds, not literals: code computes the factual delta + between the acted world and the expected fixture, and the asserting + agent classifies each difference as volatile (timestamps, SHAs, + allocated ids) or material. No normalisation table to maintain, no + hand-written per-field assertions, and nothing goes unchecked + because nobody thought to assert it. Engine responses during the + walk are real. **`npm test`** holds the deterministic perimeter — + zero tokens, and run before every commit, this project having no + automated CI: every snapshot rebuilds from its recipe byte-identical + (so a world the engine moved goes red at the gate and lands as a + reviewable snapshot diff in the PR that moved it), and every case's + paths, anchors, worlds, stubs, and trace resolve. +- **P4 — walker/asserter separation.** The walking agent never sees + `assert.md`; an agent that knows the expected answer will find it. + The walker gets the world, the coarse instruction, the answer script + and the stubs, and returns a transcript. The asserter sees the + transcript, the expected path, and the world delta — never the + reverse. +- **P4a — substitutions are declared and marked.** A stub is named + content; the case arming it owns the trigger, so one stub serves + many moments. The walker records `SUBSTITUTED:` for each, and the + prompt labels them as harness mechanics, so the asserter never + credits the walk for what the framework supplied. Whatever a stub + covers is not under test in that case — some other case must walk it + unstubbed. +- **P5 — run on command, never in the test suite.** Walks cost tokens, + so they never ride `npm test` — only the perimeter of P3 does. The runner is invoked deliberately: scoped by diff-intersection (the same computation powers the PR-end suggestion — "these N cases intersect this PR's prose changes"), by hand-picked ids, or `--all`. An engine-only PR intersects nothing and suggests nothing. -- **P6 — tiered walkers.** Sonnet walks by default; any FAIL re-runs - on Opus before it is believed. Persistent disagreement surfaces to - Lee with both walks quoted — failures are findings, never - auto-resolved gates. +- **P6 — one model, chosen for trust.** Walker and asserter both run on + **Opus**. Measured, not assumed: across three runs a Sonnet walker + performed the walk correctly every time but narrated it in summary, + omitting the quoted evidence the asserter requires — even with a + worked transcript example in front of it. A tiered arrangement then + produces a disagreement on every case, which is noise, not a signal. + A result you cannot rely on is worth nothing, so the cheaper tier is + a false economy. The orchestrator never overrides the model: each + definition names the model its result is trusted at. +- **P6a — a FAIL is confirmed by repetition, not by promotion.** A + failing case re-runs once from a fresh world at the same models. A + defect in the prose reproduces; a one-off does not, and is reported + as FLAKY with both runs quoted. Nothing is auto-resolved. +- **P6d — a walk that began mid-flow proves nothing.** Walkers skip + steps whose effect the world already holds — "already booted", "the + plan already exists" — and then the case reports a prose failure at a + step the walk never reached. So the walker starts at the entry point + the task names and performs every step including the redundant ones, + the asserter returns `INVALID WALK` when the transcript opens + mid-flow, and the orchestrator retries once before reporting + `INVALID` — never `FAIL`. A false finding is worse than no finding: + it spends human attention on prose that was never exercised, and it + teaches distrust of the runs that are real. +- **P6b — a nested agent per case.** The `/prose-test` skill dispatches + one **prose-orchestrator** per case, which builds the world, dispatches + **prose-walker**, computes the delta, dispatches **prose-asserter**, + escalates a failure, destroys the world, and returns a verdict alone. + Transcripts never reach the main session — the reason the design + scales past a handful of cases. Standing instructions live in + `.claude/agents/prose-*.md`; only the per-case payload comes from + `tests/prose/prompts/`, so no agent ever reads words composed in code. +- **P6c — run the test, nothing else.** Every agent in the chain is + forbidden from fixing anything and from working out why a case failed. + A failure is a finished result; diagnosing it is a separate, human-led + act. This is stated in prose in each definition rather than relied on + from tool restrictions, which an agent may hold regardless. - **P7 — a failing case is a finding either way.** Either the prose broke, or the world/design moved and the case is stale. The adjudication is the point; only the typo-class staleness is - pre-filtered by CI (P3). + pre-filtered by the perimeter (P3). +- **P8 — cases are hand-written, never generated.** Deriving an + `assert.md` from a recorded walk would make authoring far faster and + would quietly convert the whole corpus into approval testing: every + case would pin what the prose *currently does*, defects included, and + a mummified bug passes forever. The authoring effort — reading the + prose closely enough to state what it *should* do — is the part that + finds pre-existing defects, as it did twice in the first corpus. It + is the point, not the overhead. Snapshots are generated because the + engine authors them and drift is visible; expectations are written + because only a person can say what correct means. ## Architecture -- `tests/prose/{flow}/` — case files per flow (research, discussion, - planning, implementation, review, discovery, …). -- `tests/prose/fixtures/{name}/recipe.cjs` + `snapshot/` — the world - library. One canonical fake project (same name, same topics - throughout) so fixtures stay comparable and cases read familiarly. -- Runner (node): case parsing, diff-selection, world building, - walker-prompt assembly, end-state diffing, verdict table. Owns - everything deterministic. -- `/prose-test` skill (thin): dispatch and grading discipline — - invokes the runner, sends walkers, grades transcripts, escalates +- `tests/prose/cases/{case-id}/` — the cases, one directory each. +- `tests/prose/mainlines/{work-type}.cjs` — shared pipeline stages, so + a case's state recipes are a few lines of composition. This is where + reuse lives: as functions, not as deduplicated snapshots. +- `tests/prose/stubs/{name}.md` — named substitutions: description + above a `---` fence, exact bytes below. +- `tests/prose/lib/` — `cases.cjs` (load and validate a case + directory), `worlds.cjs` (run recipes, snapshot, hash-skip, diff, + materialise), `fake-clock.cjs`. +- `run.cjs` — the deterministic CLI: `list · select · world · prompt · + diff · assert · snap · verify · destroy`. +- `/prose-test` skill (thin): dispatch and verdict discipline — + invokes the runner, sends the walker, sends the asserter, escalates failures, reports. ## Stages @@ -109,6 +181,92 @@ testing. ## Log +- 2026-07-25 — First real runs, and the nested-agent shape. The framework + executed end to end for the first time: a Sonnet walker followed + root-cause-validation.md against a live world — real engine calls, the + scripted answer, the stub firing at its trigger — and a Sonnet asserter + graded five path steps with quoted evidence and classified the lone + world difference (an agent-row timestamp) as volatile. PASS, ~107k + tokens, 2.5 minutes. + **The negative test is the more valuable result.** With `agent scan` + deleted from the world's own copy of the prose, the run correctly + FAILED — the asserter quoted the missing call and raised a DEVIATION. + Crucially **the world delta was byte-identical to the passing run**: + `incorporate` sets `incorporated` from any prior status, so skipping + `scan` converges on the same end state. Whole-world assertion alone + would have passed a real defect; only the granular expected path caught + it. Lee's argument for granularity, proven by evidence rather than + reasoning. + Two changes came out of the runs: the walker must quote every block the + prose directs it to emit (a skipped emission was invisible), and the + agent layer moved into `.claude/agents/` as prose-orchestrator → + prose-walker + prose-asserter, so transcripts stay out of the main + session (P6a) and standing instructions stop being JS string literals + in the runner. Noted, not actioned: `incorporate` never requires a row + to have reached `pending` — forgiving by design, since recovery paths + close rows that never produced a report, but it is why this class of + skipped step is invisible to state. + +- 2026-07-25 — A case becomes a directory (Lee, still reviewing in the + TUI). Three faults in the flat-markdown shape, all his: **(1)** code + regex-parsed values out of markdown — the exact pattern this project + spent the analysis-state campaign eliminating. Now `case.json` holds + what code branches on, prose files are read whole, recipes are + modules; nothing is parsed and prose never shares a file with code. + **(2)** `world_before`/`world_after` named foreign fixtures the case + merely pointed at, and the scatter was real — six locations for one + test. A case now owns its worlds: `fixture-state.cjs` and + `assertion-state.cjs`, split because they change for different + reasons, with their snapshots beside them. Duplication accepted; + reuse moved to `mainlines/` as functions. **(3)** The naming follows + the stage it serves — fixture, act, assert — so `assert.md` and + `assertion-state.cjs` sit together by design. Added with it: a + recipe-hash skip, without which per-case worlds would make the gate + grow linearly (proven: 0.09s when nothing moved, full rebuild and + DRIFT when a mainline's content changed). + +- 2026-07-25 — The Given-When-Then reshape (Lee, reviewing the corpus in + the TUI). Four corrections, all his: **(1)** cases are flat, one per + file, filename equal to the id — grouping by work type organised the + corpus around the build sequence rather than the prose under test, and + any single-parent taxonomy lies about cases spanning skills. + **(2)** The hand-written assertion DSL is gone. The world is a + directory that changed; the expectation is another committed world; + code diffs them and the asserting agent classifies each difference. + That deleted the grammar *and* closed the gap where anything nobody + thought to assert went unchecked — no normalisation table, no + engine-written-versus-model-written split to adjudicate. + **(3)** Stub triggers belong to the case, not the stub: binding a + substitution to a moment is what *prevents* reuse, and the same + content is wanted at different moments (first dispatch vs re-dispatch, + gaps-found vs validated). A stub is named content; the case says when. + A trigger-less stub is arrange in disguise and fails validation. + **(4)** Granularity belongs in the Then. Lee's argument, which holds: + coarse assertions let an agent silently course-correct around broken + prose — good in production, fatal in a test. The When stays coarse + (else the walker stops deriving the path and the case passes with the + skill deleted); the Then carries a step-by-step expected trace, so a + silent repair shows as a trace mismatch. Also added: a `DEVIATION:` + marker for prose that cannot be followed literally, findings in their + own right. Sections renamed given/when/then after BDD, which reads + closer to a specification than arrange/act/assert. + +- 2026-07-25 — Stage 3b, the bugfix corpus. Three fixtures along + `crash-fix` (created / investigating / investigated) and four cases + covering only what bugfix does differently: continue-bugfix routing a + fresh bug to investigation, investigation entry seeding from the + carrier without re-asking, the root-cause validation agent's full + lifecycle, and a specification sourced from the investigation rather + than a discussion. Two framework extensions the corpus demanded, + both small: a free-text **`stub:` section** — prose that dispatches a + background agent is walked with the walker playing the agent (writes + the report at the engine-allocated path, returns the stated result), + because the case tests the lifecycle around the agent, never the + agent's judgment; and a **`json` state assertion** so the agent row's + closing status is checked deterministically in the cache store rather + than read out of a transcript. Both proven by hand-walking the + lifecycle in a live world before the cases shipped. + - 2026-07-25 — Stage 3a, the feature corpus. Five stop-point fixtures along the canonical feature mainline (`pay` — created / discussed / specified / planned / implemented), composed from staged builders in @@ -154,8 +312,8 @@ testing. store re-derived at materialise), and the runner (`run.cjs`: list · select · world · prompt · grade · snap · verify · destroy). The `prompt` command is the P4 boundary — walker prompts are - machine-assembled and never contain expects. CI gains two token-free - suites (corpus validation, snapshot rebuild-compare); the `/prose-test` + machine-assembled and never contain expects. `npm test` gains two + token-free suites (corpus validation, snapshot rebuild-compare); the `/prose-test` dev skill owns the model layer (Sonnet walks, Opus confirms, quoted evidence or it didn't happen). First fixture: `base` — a post-boot keyword-only empty project, proven byte-deterministic across rebuilds. @@ -167,7 +325,7 @@ testing. prose tests own their worlds) with the engine kept at authoring time via recipes; recipe + golden snapshot over either alone (always-current *and* visible drift *and* frozen runs); on-command - invocation only — tokens are spent deliberately, CI holds the - deterministic perimeter; Sonnet→Opus escalation; happy path + invocation only — tokens are spent deliberately, `npm test` holds + the deterministic perimeter; Sonnet→Opus escalation; happy path first-class, corpus seeded from the sim's mainline enumeration before the failure harvest. diff --git a/package.json b/package.json index 482711aca..a2be82ef1 100644 --- a/package.json +++ b/package.json @@ -7,7 +7,7 @@ "build": "node build/knowledge.build.js", "knowledge:repl": "node scripts/knowledge-repl.js", "typecheck": "tsc -p tsconfig.json", - "test": "node --test tests/scripts/test-render.cjs tests/scripts/test-engine-gateway.cjs tests/scripts/test-engine-epic-projections.cjs tests/scripts/test-engine-start-projections.cjs tests/scripts/test-engine-workunit-projections.cjs tests/scripts/test-engine-discussion-map.cjs tests/scripts/test-engine-discovery-map.cjs tests/scripts/test-engine-tasks.cjs tests/scripts/test-engine-render-surfaces.cjs tests/scripts/test-engine-transactions.cjs tests/scripts/test-engine-workunit-create.cjs tests/scripts/test-engine-workunit-pivot.cjs tests/scripts/test-engine-workunit-absorb.cjs tests/scripts/test-engine-workunit-promote.cjs tests/scripts/test-engine-discovery-session.cjs tests/scripts/test-engine-cache.cjs tests/scripts/test-engine-manifest-io.cjs tests/scripts/test-engine-manifest-fields.cjs tests/scripts/test-engine-boot.cjs tests/scripts/test-migration-orchestrator.cjs tests/scripts/test-migration-047.cjs tests/scripts/test-migration-048.cjs tests/scripts/test-migration-049.cjs tests/scripts/test-migration-050.cjs tests/scripts/test-migration-051.cjs tests/scripts/test-migration-052.cjs tests/scripts/test-engine-specification-projections.cjs tests/scripts/test-engine-discovery-projections.cjs tests/scripts/test-gateway-for-start.cjs tests/scripts/test-gateway-for-specification.cjs tests/scripts/test-gateway-for-workflow-continue-epic.cjs tests/scripts/test-gateway-for-workflow-continue-feature.cjs tests/scripts/test-gateway-for-workflow-continue-bugfix.cjs tests/scripts/test-gateway-for-workflow-continue-quickfix.cjs tests/scripts/test-gateway-for-workflow-continue-cross-cutting.cjs tests/scripts/test-gateway-for-discovery-skill.cjs tests/scripts/test-gateway-for-bridge.cjs tests/scripts/test-reads-derivations.cjs tests/scripts/test-ensure-discovery-item.cjs tests/scripts/test-legacy-research-split.cjs tests/scripts/test-knowledge-chunker.cjs tests/scripts/test-knowledge-config.cjs tests/scripts/test-knowledge-discovery.cjs tests/scripts/test-knowledge-embeddings.cjs tests/scripts/test-knowledge-integration.cjs tests/scripts/test-knowledge-openai.cjs tests/scripts/test-knowledge-openai-compatible.cjs tests/scripts/test-knowledge-openai-integration.cjs tests/scripts/test-knowledge-progress.cjs tests/scripts/test-knowledge-rerank.cjs tests/scripts/test-knowledge-retry.cjs tests/scripts/test-knowledge-setup-forms.cjs tests/scripts/test-knowledge-store.cjs tests/scripts/test-conventions-lint.cjs tests/scripts/test-pipeline-simulation.cjs tests/scripts/test-engine-agent-state.cjs tests/scripts/test-prose-cases.cjs tests/scripts/test-prose-fixtures.cjs", + "test": "node --test tests/scripts/test-render.cjs tests/scripts/test-engine-gateway.cjs tests/scripts/test-engine-epic-projections.cjs tests/scripts/test-engine-start-projections.cjs tests/scripts/test-engine-workunit-projections.cjs tests/scripts/test-engine-discussion-map.cjs tests/scripts/test-engine-discovery-map.cjs tests/scripts/test-engine-tasks.cjs tests/scripts/test-engine-render-surfaces.cjs tests/scripts/test-engine-transactions.cjs tests/scripts/test-engine-workunit-create.cjs tests/scripts/test-engine-workunit-pivot.cjs tests/scripts/test-engine-workunit-absorb.cjs tests/scripts/test-engine-workunit-promote.cjs tests/scripts/test-engine-discovery-session.cjs tests/scripts/test-engine-cache.cjs tests/scripts/test-engine-manifest-io.cjs tests/scripts/test-engine-manifest-fields.cjs tests/scripts/test-engine-boot.cjs tests/scripts/test-migration-orchestrator.cjs tests/scripts/test-migration-047.cjs tests/scripts/test-migration-048.cjs tests/scripts/test-migration-049.cjs tests/scripts/test-migration-050.cjs tests/scripts/test-migration-051.cjs tests/scripts/test-migration-052.cjs tests/scripts/test-engine-specification-projections.cjs tests/scripts/test-engine-discovery-projections.cjs tests/scripts/test-gateway-for-start.cjs tests/scripts/test-gateway-for-specification.cjs tests/scripts/test-gateway-for-workflow-continue-epic.cjs tests/scripts/test-gateway-for-workflow-continue-feature.cjs tests/scripts/test-gateway-for-workflow-continue-bugfix.cjs tests/scripts/test-gateway-for-workflow-continue-quickfix.cjs tests/scripts/test-gateway-for-workflow-continue-cross-cutting.cjs tests/scripts/test-gateway-for-discovery-skill.cjs tests/scripts/test-gateway-for-bridge.cjs tests/scripts/test-reads-derivations.cjs tests/scripts/test-ensure-discovery-item.cjs tests/scripts/test-legacy-research-split.cjs tests/scripts/test-knowledge-chunker.cjs tests/scripts/test-knowledge-config.cjs tests/scripts/test-knowledge-discovery.cjs tests/scripts/test-knowledge-embeddings.cjs tests/scripts/test-knowledge-integration.cjs tests/scripts/test-knowledge-openai.cjs tests/scripts/test-knowledge-openai-compatible.cjs tests/scripts/test-knowledge-openai-integration.cjs tests/scripts/test-knowledge-progress.cjs tests/scripts/test-knowledge-rerank.cjs tests/scripts/test-knowledge-retry.cjs tests/scripts/test-knowledge-setup-forms.cjs tests/scripts/test-knowledge-store.cjs tests/scripts/test-conventions-lint.cjs tests/scripts/test-pipeline-simulation.cjs tests/scripts/test-engine-agent-state.cjs tests/scripts/test-prose-cases.cjs tests/scripts/test-prose-snapshots.cjs", "test:migrations": "for t in tests/scripts/test-migration-*.sh; do bash $t || exit 1; done", "test:cli": "for t in tests/scripts/test-engine-manifest.sh tests/scripts/test-inbox-promotion.sh tests/scripts/test-knowledge-cli.sh tests/scripts/test-knowledge-build.sh; do bash $t || exit 1; done" }, diff --git a/tests/prose/README.md b/tests/prose/README.md index 93c6b87fa..e2c7d2cf4 100644 --- a/tests/prose/README.md +++ b/tests/prose/README.md @@ -1,42 +1,123 @@ # Prose tests -End-to-end tests for the prose logic — natural-language cases a walker -agent executes through the real skills against a materialised fixture -world. Design and contract: [design/prose-tests.md](../../design/prose-tests.md). -Run via the `/prose-test` skill; everything deterministic lives in -`run.cjs` and `lib/`. - -## Layout - -- `{flow}/*.md` — case files (grammar: header comment in `lib/cases.cjs`) -- `fixtures/{name}/recipe.cjs` — builds the world with real engine calls - under a frozen clock (`lib/fake-clock.cjs`) -- `fixtures/{name}/snapshot/` — the recipe's committed golden output; - regenerate with `node tests/prose/run.cjs snap {name}`, never hand-edit -- `run.cjs` — `list · select · world · prompt · grade · snap · verify · destroy` - -## Adding a case - -Add a `## case:` block to the flow's file (or a new file for a long -walk). Scope `files:` tightly — diff-selection and the PR-end suggestion -run off it. `routing:` expects are agent-graded against the walk -transcript; `state:` expects are asserted in code and require a `world:`. - -**Name behaviour, never coordinates.** Step numbers, arm letters, and -heading numbering in walk or expect text break on cosmetic renumbering — -failure for the wrong reason. Write "casing conventions load before the -boot pipeline", not "Step 0.1 before Step 0.2"; a behavioural claim -fails only when the behaviour changes, which is the failure the corpus -exists to catch. Anchors are substring fragments matched against heading -text (`#Boot` matches "Step 0.2: Boot") — pick the number-free part. -`npm test` validates the corpus (parse, paths, anchors, grammar) and -rebuild-compares every fixture — both token-free; only `/prose-test` -spends tokens. - -## Snapshot escaping - -Snapshots exclude `.git/` and `.workflows/.knowledge/` (the world -builder re-derives the store), and store `.gitignore` files as -`_gitignore.fixture` so the product-written `.workflows/.gitignore` -cannot ignore fixture content out of this repo; the world builder -restores them. +End-to-end tests for the prose logic. A case arranges a world from a +committed fixture, an agent acts by walking the real skills as a live +session would, and a second agent asserts the resulting path and world. +Design and contract: [design/prose-tests.md](../../design/prose-tests.md). +Run via the `/prose-test` skill. + +## A case is a directory + +``` +cases/{case-id}/ + case.json the values code branches on + fixture.md optional — prose describing the starting world + fixture-state.cjs builds that world, in engine calls + act.md the coarse instruction given to the walker + assert.md the expected path, given only to the asserter + assertion-state.cjs optional — builds the world the walk should produce + fixture/ generated snapshot + assertion/ generated snapshot +``` + +**Nothing is parsed.** JSON is JSON, prose files are read whole and +relayed into prompts, recipes are `require`d as modules. There is no +format to mis-read: a value code branches on lives in `case.json`, and a +blob code merely relays lives in markdown. + +`act.md` and `assert.md` are separate files because that is the +walker/asserter boundary — the walker must never see the expected path, +and a file boundary enforces that structurally rather than by convention. + +`fixture-state.cjs` and `assertion-state.cjs` are separate because they +change for different reasons: the fixture when the precondition changes, +the assertion when the prose's *behaviour* changes. The assertion state +composes the fixture state plus whatever the walk should have done. A +case with no `assertion-state.cjs` expects its world back untouched — +which is most entry-skill cases, and a strong claim in itself. + +Shared pipeline stages live in `mainlines/{work-type}.cjs`, so a +fixture-state is usually three lines of composition. Worlds are per-case +and duplication is accepted: a case you can read in one directory beats +a deduplicated fixture library you have to chase across the tree. + +## Writing a case + +**Never generate a case from a walk.** Recording what the prose did and +calling it the expectation turns the corpus into approval testing — +every case pins current behaviour, defects included, and a mummified bug +passes forever. Snapshots are generated because the engine authors them +and drift is visible; expectations are written by hand because only a +person can say what correct means. The reading-closely part is where +pre-existing defects surface. + +**The act stays coarse — one instruction.** Where to enter, what to +follow, where to stop. Never a step-by-step script of the workflow: if +the case tells the walker which commands to run, the walker no longer has +to derive the path from the prose, and the case would pass with the skill +file deleted. That derivation is the thing under test. + +**The assertion is granular.** A step-by-step expected path is what +catches a walker that silently course-corrected around broken prose — +the divergence shows up as a mismatch instead of relying on the walker to +confess it. + +**Name behaviour, never coordinates.** Step numbers, arm letters and +heading numbering break on cosmetic renumbering — failure for the wrong +reason. Write "casing conventions load before the boot pipeline", not +"Step 0.1 before Step 0.2". Anchors in `case.json` are substring +fragments matched against heading text (`#Boot` matches "Step 0.2: Boot"). + +**Scope `files` tightly** — selection and the PR-end suggestion run off it. + +## Stubs + +Prose that dispatches a background agent, or runs an unscripted +conversation, is walked with that part substituted. A stub is named +content in `stubs/{name}.md` — description above a `---` fence, exact +bytes below — and the case arming it owns the moment: + +```json +"stubs": { + "root-cause-validated": "when the engine records a dispatch of kind root-cause-validation" +} +``` + +The trigger belongs to the case, not the stub, so one stub serves many +moments — first dispatch versus re-dispatch, happy path versus recovery. +A stub with no trigger is a fixture in disguise, and validation rejects it. + +**A trigger names an observable event, not a narrative moment.** An +engine call with its arguments, a specific menu appearing, a named file +being created — something the walker can match with certainty and that +shows up in the transcript, so the asserter can confirm the substitution +fired where it should have. + +**Whatever a stub covers is not under test in that case.** Stubbing a +discussion's session loop means some other case must walk that loop +unstubbed. The stub boundary is a scope decision, not a convenience. + +## Assertion + +Code computes the factual delta between the acted world and the expected +world; the asserting agent classifies every difference as volatile +(timestamps, git SHAs, engine-allocated ids) or material, and checks the +expected path against the transcript with quoted evidence. No +normalisation table, no expected-value literals: the expectation is a +whole committed world, built by a recipe, reviewable as a diff against +the world it started from. + +## The gate + +`npm test` validates the corpus and rebuilds every snapshot to +byte-compare it — both token-free, and this repo's only gate since there +is no automated CI. Rebuilds are skipped when nothing feeding a world has +changed; the hash covers each case's recipes, the shared mainlines, and +the engine and knowledge sources, so an engine change invalidates every +hash and forces a full rebuild. Only `/prose-test` spends tokens. + +Snapshots exclude `.git/`, `.workflows/.knowledge/` (the world builder +re-derives the store) and `.claude/skills|agents/` (copied into live +worlds, never part of a world's own state), and store `.gitignore` files +as `_gitignore.fixture` so the product-written `.workflows/.gitignore` +cannot ignore snapshot content out of this repo. diff --git a/tests/prose/cases/continue-bugfix-routes-to-investigation/act.md b/tests/prose/cases/continue-bugfix-routes-to-investigation/act.md new file mode 100644 index 000000000..0f11a5af9 --- /dev/null +++ b/tests/prose/cases/continue-bugfix-routes-to-investigation/act.md @@ -0,0 +1,4 @@ +Execute skills/workflow-continue-bugfix/SKILL.md with no arguments, as +it is invoked when the caller has no work unit pre-selected. Follow it +through selection and the bugfix's state display. Stop once the state +and its menu have been shown — do not invoke the route. diff --git a/tests/prose/cases/continue-bugfix-routes-to-investigation/assert.md b/tests/prose/cases/continue-bugfix-routes-to-investigation/assert.md new file mode 100644 index 000000000..f975a31a2 --- /dev/null +++ b/tests/prose/cases/continue-bugfix-routes-to-investigation/assert.md @@ -0,0 +1,10 @@ +The prose should have taken this path: + +1. initialisation is casing only — boot and migrations belong to the + entry skill +2. shows the selection display and menu rather than auto-selecting the + only bugfix +3. validates the selection, then shows the bugfix's pipeline state +4. the state shows no phase started, and the continue action routes to + investigation entry — a bugfix's first phase is investigation, never + discussion or specification diff --git a/tests/prose/cases/continue-bugfix-routes-to-investigation/case.json b/tests/prose/cases/continue-bugfix-routes-to-investigation/case.json new file mode 100644 index 000000000..7a221fd7f --- /dev/null +++ b/tests/prose/cases/continue-bugfix-routes-to-investigation/case.json @@ -0,0 +1,11 @@ +{ + "origin": "bugfix mainline — a fresh bug routes to investigation, not discussion", + "files": [ + "skills/workflow-continue-bugfix/SKILL.md", + "skills/workflow-continue-bugfix/references/select-bugfix.md", + "skills/workflow-continue-bugfix/references/bugfix-display-and-menu.md" + ], + "answers": [ + "the number listed against `crash-fix`" + ] +} diff --git a/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture-state.cjs b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture-state.cjs new file mode 100644 index 000000000..6cd47642e --- /dev/null +++ b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture-state.cjs @@ -0,0 +1,12 @@ +'use strict'; + +// The bug is captured and nothing has been started. + +const m = require('../../mainlines/bugfix.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + }, +}; diff --git a/tests/prose/fixtures/base/snapshot/.claude/settings.json b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.claude/settings.json similarity index 100% rename from tests/prose/fixtures/base/snapshot/.claude/settings.json rename to tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.claude/settings.json diff --git a/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.recipe-hash b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.recipe-hash new file mode 100644 index 000000000..259e11ddc --- /dev/null +++ b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.recipe-hash @@ -0,0 +1 @@ +f2d22a09d2680d82822f6042b87baf77dacbf41d4d8cb2b6d26a33d5c4129ff9 diff --git a/tests/prose/fixtures/base/snapshot/.workflows/.state/migrations b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/.state/migrations similarity index 100% rename from tests/prose/fixtures/base/snapshot/.workflows/.state/migrations rename to tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/.state/migrations diff --git a/tests/prose/fixtures/base/snapshot/.workflows/_gitignore.fixture b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/_gitignore.fixture similarity index 100% rename from tests/prose/fixtures/base/snapshot/.workflows/_gitignore.fixture rename to tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/_gitignore.fixture diff --git a/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/crash-fix/discovery/sessions/session-001.md b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/crash-fix/discovery/sessions/session-001.md new file mode 100644 index 000000000..6dc56d4e3 --- /dev/null +++ b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/crash-fix/discovery/sessions/session-001.md @@ -0,0 +1,40 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: crash-fix + +## Description (as of session) + +Checkout crashes when an order has no shipping address. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Reported by two users this week: the checkout page 500s at the +payment step for orders with no shipping address. Reproducible on +staging with a digital-only basket. No error surfaces to the user — +the page just fails. Confirmed as a bugfix; no design question here, +so it routes straight to investigation. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/crash-fix/manifest.json b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/crash-fix/manifest.json new file mode 100644 index 000000000..eaaa56199 --- /dev/null +++ b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/crash-fix/manifest.json @@ -0,0 +1,8 @@ +{ + "name": "crash-fix", + "work_type": "bugfix", + "status": "in-progress", + "created": "2026-01-01", + "description": "Checkout crashes when an order has no shipping address", + "phases": {} +} diff --git a/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/manifest.json b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/manifest.json new file mode 100644 index 000000000..1cd498ed5 --- /dev/null +++ b/tests/prose/cases/continue-bugfix-routes-to-investigation/fixture/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "crash-fix": { + "work_type": "bugfix" + } + } +} diff --git a/tests/prose/cases/continue-feature-routes-to-discussion/act.md b/tests/prose/cases/continue-feature-routes-to-discussion/act.md new file mode 100644 index 000000000..93e469eef --- /dev/null +++ b/tests/prose/cases/continue-feature-routes-to-discussion/act.md @@ -0,0 +1,4 @@ +Execute skills/workflow-continue-feature/SKILL.md with no arguments, as +it is invoked when the caller has no work unit pre-selected. Follow it +through selection and the feature's state display. Stop once the state +and its menu have been shown — do not invoke the route. diff --git a/tests/prose/cases/continue-feature-routes-to-discussion/assert.md b/tests/prose/cases/continue-feature-routes-to-discussion/assert.md new file mode 100644 index 000000000..b789b6b49 --- /dev/null +++ b/tests/prose/cases/continue-feature-routes-to-discussion/assert.md @@ -0,0 +1,13 @@ +The prose should have taken this path: + +1. initialisation is casing only — no boot and no migrations, which the + entry skill has already guaranteed +2. shows the selection display and menu even though `pay` is the only + feature — never auto-selects +3. validates the selection, then shows the feature's pipeline state +4. the state shows no phase started, and the menu's continue action + routes to discussion entry for work type feature, work unit pay + +Further claims: + +- navigation only reads: no phase is started by looking at it diff --git a/tests/prose/cases/continue-feature-routes-to-discussion/case.json b/tests/prose/cases/continue-feature-routes-to-discussion/case.json new file mode 100644 index 000000000..33c362b36 --- /dev/null +++ b/tests/prose/cases/continue-feature-routes-to-discussion/case.json @@ -0,0 +1,11 @@ +{ + "origin": "feature mainline — navigation shows state and routes to the next phase", + "files": [ + "skills/workflow-continue-feature/SKILL.md", + "skills/workflow-continue-feature/references/select-feature.md", + "skills/workflow-continue-feature/references/feature-display-and-menu.md" + ], + "answers": [ + "the number listed against `pay`" + ] +} diff --git a/tests/prose/cases/continue-feature-routes-to-discussion/fixture-state.cjs b/tests/prose/cases/continue-feature-routes-to-discussion/fixture-state.cjs new file mode 100644 index 000000000..e4b78cd71 --- /dev/null +++ b/tests/prose/cases/continue-feature-routes-to-discussion/fixture-state.cjs @@ -0,0 +1,12 @@ +'use strict'; + +// The feature exists and nothing has been started. + +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + }, +}; diff --git a/tests/prose/fixtures/feature-created/snapshot/.claude/settings.json b/tests/prose/cases/continue-feature-routes-to-discussion/fixture/.claude/settings.json similarity index 100% rename from tests/prose/fixtures/feature-created/snapshot/.claude/settings.json rename to tests/prose/cases/continue-feature-routes-to-discussion/fixture/.claude/settings.json diff --git a/tests/prose/cases/continue-feature-routes-to-discussion/fixture/.recipe-hash b/tests/prose/cases/continue-feature-routes-to-discussion/fixture/.recipe-hash new file mode 100644 index 000000000..92519878f --- /dev/null +++ b/tests/prose/cases/continue-feature-routes-to-discussion/fixture/.recipe-hash @@ -0,0 +1 @@ +5b0a7d802b3a436a4656bc000122451bb015f2250f8fef47b1d05f3ea51ba96b diff --git a/tests/prose/fixtures/feature-created/snapshot/.workflows/.state/migrations b/tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/.state/migrations similarity index 100% rename from tests/prose/fixtures/feature-created/snapshot/.workflows/.state/migrations rename to tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/.state/migrations diff --git a/tests/prose/fixtures/feature-created/snapshot/.workflows/_gitignore.fixture b/tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/_gitignore.fixture similarity index 100% rename from tests/prose/fixtures/feature-created/snapshot/.workflows/_gitignore.fixture rename to tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/_gitignore.fixture diff --git a/tests/prose/fixtures/feature-created/snapshot/.workflows/manifest.json b/tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-created/snapshot/.workflows/manifest.json rename to tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/manifest.json diff --git a/tests/prose/fixtures/feature-created/snapshot/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/pay/discovery/sessions/session-001.md similarity index 100% rename from tests/prose/fixtures/feature-created/snapshot/.workflows/pay/discovery/sessions/session-001.md rename to tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/pay/discovery/sessions/session-001.md diff --git a/tests/prose/fixtures/feature-created/snapshot/.workflows/pay/manifest.json b/tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/pay/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-created/snapshot/.workflows/pay/manifest.json rename to tests/prose/cases/continue-feature-routes-to-discussion/fixture/.workflows/pay/manifest.json diff --git a/tests/prose/cases/discussion-entry-seeds-from-carrier/act.md b/tests/prose/cases/discussion-entry-seeds-from-carrier/act.md new file mode 100644 index 000000000..6e82ff2ab --- /dev/null +++ b/tests/prose/cases/discussion-entry-seeds-from-carrier/act.md @@ -0,0 +1,4 @@ +Execute skills/workflow-discussion-entry/SKILL.md with arguments +$0=feature, $1=pay. Follow it to the point where the handoff to the +processing skill is constructed; record the handoff block and stop. Do +not execute the processing skill's instructions. diff --git a/tests/prose/cases/discussion-entry-seeds-from-carrier/assert.md b/tests/prose/cases/discussion-entry-seeds-from-carrier/assert.md new file mode 100644 index 000000000..e0a0c97ba --- /dev/null +++ b/tests/prose/cases/discussion-entry-seeds-from-carrier/assert.md @@ -0,0 +1,16 @@ +The prose should have taken this path: + +1. resolves the topic to the work unit, since only an epic is given one + explicitly +2. reads the discussion status, finds nothing, and takes the new-entry + arm — phase validation is for entries that already exist +3. ensuring a discovery item returns immediately: the map is epic-only, + and this is a feature +4. seeds the context from the manifest description and the session log's + exploration, asking the user nothing +5. hands off to the discussion processing skill for pay + +Further claims: + +- the entry skill records nothing: the discussion item is created by the + processing skill, not by the walk to it diff --git a/tests/prose/cases/discussion-entry-seeds-from-carrier/case.json b/tests/prose/cases/discussion-entry-seeds-from-carrier/case.json new file mode 100644 index 000000000..f5bc80e4b --- /dev/null +++ b/tests/prose/cases/discussion-entry-seeds-from-carrier/case.json @@ -0,0 +1,7 @@ +{ + "origin": "feature mainline — single-phase work seeds from its durable carrier", + "files": [ + "skills/workflow-discussion-entry/SKILL.md", + "skills/workflow-shared/references/ensure-discovery-item.md" + ] +} diff --git a/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture-state.cjs b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture-state.cjs new file mode 100644 index 000000000..e4b78cd71 --- /dev/null +++ b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture-state.cjs @@ -0,0 +1,12 @@ +'use strict'; + +// The feature exists and nothing has been started. + +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + }, +}; diff --git a/tests/prose/fixtures/feature-discussed/snapshot/.claude/settings.json b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.claude/settings.json similarity index 100% rename from tests/prose/fixtures/feature-discussed/snapshot/.claude/settings.json rename to tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.claude/settings.json diff --git a/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.recipe-hash b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.recipe-hash new file mode 100644 index 000000000..92519878f --- /dev/null +++ b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.recipe-hash @@ -0,0 +1 @@ +5b0a7d802b3a436a4656bc000122451bb015f2250f8fef47b1d05f3ea51ba96b diff --git a/tests/prose/fixtures/feature-discussed/snapshot/.workflows/.state/migrations b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/.state/migrations similarity index 100% rename from tests/prose/fixtures/feature-discussed/snapshot/.workflows/.state/migrations rename to tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/.state/migrations diff --git a/tests/prose/fixtures/feature-discussed/snapshot/.workflows/_gitignore.fixture b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/_gitignore.fixture similarity index 100% rename from tests/prose/fixtures/feature-discussed/snapshot/.workflows/_gitignore.fixture rename to tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/_gitignore.fixture diff --git a/tests/prose/fixtures/feature-discussed/snapshot/.workflows/manifest.json b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-discussed/snapshot/.workflows/manifest.json rename to tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/manifest.json diff --git a/tests/prose/fixtures/feature-discussed/snapshot/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/pay/discovery/sessions/session-001.md similarity index 100% rename from tests/prose/fixtures/feature-discussed/snapshot/.workflows/pay/discovery/sessions/session-001.md rename to tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/pay/discovery/sessions/session-001.md diff --git a/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/pay/manifest.json b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/pay/manifest.json new file mode 100644 index 000000000..1cf8afb0e --- /dev/null +++ b/tests/prose/cases/discussion-entry-seeds-from-carrier/fixture/.workflows/pay/manifest.json @@ -0,0 +1,8 @@ +{ + "name": "pay", + "work_type": "feature", + "status": "in-progress", + "created": "2026-01-01", + "description": "Accept card payments at checkout", + "phases": {} +} diff --git a/tests/prose/cases/implementation-picks-first-task/act.md b/tests/prose/cases/implementation-picks-first-task/act.md new file mode 100644 index 000000000..cdf6faee0 --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/act.md @@ -0,0 +1,4 @@ +Execute skills/workflow-implementation-entry/SKILL.md with arguments +$0=feature, $1=pay, then continue into the processing skill it invokes. +Follow the flow until the prose directs you to begin building the first +task. Stop there — write no implementation code and no tests. diff --git a/tests/prose/cases/implementation-picks-first-task/assert.md b/tests/prose/cases/implementation-picks-first-task/assert.md new file mode 100644 index 000000000..e09c80080 --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/assert.md @@ -0,0 +1,20 @@ +The prose should have taken this path: + +1. the plan gate renders empty, and no implementation item exists, so + this is a new entry +2. dependency validation returns immediately — external dependencies are + an epic concern +3. the environment check finds no setup document and asks the question, + gathering the answer without acting on it +4. resume detection initialises tracking and reports the created mode — + the fresh path, which commits the start of implementation, never the + resuming-from-a-previous-session note +5. environment setup records the answer as a setup document and commits + it, so the question is not asked again in a later session +6. reads the plan through the format's own reading procedure and finds + pay-1-1 next: phase one, first task, nothing completed +7. starts pay-1-1 and stops where the prose hands over to building + +Further claims: + +- the second task is neither started nor touched diff --git a/tests/prose/cases/implementation-picks-first-task/assertion-state.cjs b/tests/prose/cases/implementation-picks-first-task/assertion-state.cjs new file mode 100644 index 000000000..c065db590 --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/assertion-state.cjs @@ -0,0 +1,22 @@ +'use strict'; + +// The fixture state plus what the walk should have done: resume +// detection initialised tracking and committed the start of +// implementation, the environment answer was recorded so it is never +// asked again, and the first task was started. No task work — the walk +// stops where the prose hands over to building. + +const fixture = require('./fixture-state.cjs'); +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + fixture.build(h); + + h.engine('task', 'init', m.WU, m.WU); + h.engine('commit', m.WU, '-m', `impl(${m.WU}): start implementation`); + h.write('.workflows/.state/environment-setup.md', 'No special setup required.\n'); + h.engine('commit', '--workflows', '-m', `impl(${m.WU}): record environment setup`); + h.engine('task', 'start', m.WU, m.WU, `${m.WU}-1-1`); + }, +}; diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.claude/settings.json b/tests/prose/cases/implementation-picks-first-task/assertion/.claude/settings.json similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.claude/settings.json rename to tests/prose/cases/implementation-picks-first-task/assertion/.claude/settings.json diff --git a/tests/prose/cases/implementation-picks-first-task/assertion/.recipe-hash b/tests/prose/cases/implementation-picks-first-task/assertion/.recipe-hash new file mode 100644 index 000000000..a29d91438 --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/assertion/.recipe-hash @@ -0,0 +1 @@ +e963357b5150c81c98156db6f527db5bf50bce292bb39b107e14ca98c6f5366a diff --git a/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/.state/environment-setup.md b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/.state/environment-setup.md new file mode 100644 index 000000000..56b13047e --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/.state/environment-setup.md @@ -0,0 +1 @@ +No special setup required. diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/.state/migrations b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/.state/migrations similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/.state/migrations rename to tests/prose/cases/implementation-picks-first-task/assertion/.workflows/.state/migrations diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/_gitignore.fixture b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/_gitignore.fixture similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/_gitignore.fixture rename to tests/prose/cases/implementation-picks-first-task/assertion/.workflows/_gitignore.fixture diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/manifest.json b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/manifest.json rename to tests/prose/cases/implementation-picks-first-task/assertion/.workflows/manifest.json diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/discovery/sessions/session-001.md similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/discovery/sessions/session-001.md rename to tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/discovery/sessions/session-001.md diff --git a/tests/prose/fixtures/feature-discussed/snapshot/.workflows/pay/discussion/pay.md b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/discussion/pay.md similarity index 100% rename from tests/prose/fixtures/feature-discussed/snapshot/.workflows/pay/discussion/pay.md rename to tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/discussion/pay.md diff --git a/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/manifest.json b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/manifest.json new file mode 100644 index 000000000..23023a119 --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/manifest.json @@ -0,0 +1,70 @@ +{ + "name": "pay", + "work_type": "feature", + "status": "in-progress", + "created": "2026-01-01", + "description": "Accept card payments at checkout", + "phases": { + "discussion": { + "items": { + "pay": { + "status": "completed" + } + } + }, + "specification": { + "items": { + "pay": { + "status": "completed", + "sources": { + "pay": { + "status": "incorporated" + } + } + } + } + }, + "planning": { + "items": { + "pay": { + "status": "completed", + "format": "local-markdown", + "task_list_gate_mode": "gated", + "author_gate_mode": "gated", + "finding_gate_mode": "gated", + "review_cycle": 0, + "phase": 1, + "task": null, + "task_map": { + "pay-1-1": "pay-1-1", + "pay-1-2": "pay-1-2" + }, + "storage_paths": [], + "approvals": { + "structure": "2026-01-01", + "tasks": { + "p1": "2026-01-01" + } + } + } + } + }, + "implementation": { + "items": { + "pay": { + "status": "in-progress", + "task_gate_mode": "gated", + "fix_gate_mode": "gated", + "analysis_gate_mode": "gated", + "fix_attempts": 0, + "analysis_cycle_total": 0, + "analysis_cycle_session": 0, + "linters": [], + "project_skills": [], + "current_phase": 1, + "current_task": "pay-1-1" + } + } + } + } +} diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/planning/pay/planning.md b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/planning/pay/planning.md similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/planning/pay/planning.md rename to tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/planning/pay/planning.md diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/planning/pay/tasks/pay-1-1.md b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/planning/pay/tasks/pay-1-1.md similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/planning/pay/tasks/pay-1-1.md rename to tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/planning/pay/tasks/pay-1-1.md diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/planning/pay/tasks/pay-1-2.md b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/planning/pay/tasks/pay-1-2.md similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/planning/pay/tasks/pay-1-2.md rename to tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/planning/pay/tasks/pay-1-2.md diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/specification/pay/specification.md b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/specification/pay/specification.md similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/specification/pay/specification.md rename to tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/specification/pay/specification.md diff --git a/tests/prose/cases/implementation-picks-first-task/case.json b/tests/prose/cases/implementation-picks-first-task/case.json new file mode 100644 index 000000000..2a3a9a5d0 --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/case.json @@ -0,0 +1,16 @@ +{ + "origin": "feature mainline — entry through resume detection to the first task", + "files": [ + "skills/workflow-implementation-entry/SKILL.md", + "skills/workflow-implementation-entry/references/validate-phase.md", + "skills/workflow-implementation-entry/references/validate-dependencies.md", + "skills/workflow-implementation-entry/references/environment-check.md", + "skills/workflow-implementation-process/SKILL.md", + "skills/workflow-implementation-process/references/environment-setup.md", + "skills/workflow-implementation-process/references/task-loop.md", + "skills/workflow-planning-process/references/output-formats/local-markdown/reading.md" + ], + "answers": [ + "none — no special setup is needed" + ] +} diff --git a/tests/prose/cases/implementation-picks-first-task/fixture-state.cjs b/tests/prose/cases/implementation-picks-first-task/fixture-state.cjs new file mode 100644 index 000000000..88f1d7dfe --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/fixture-state.cjs @@ -0,0 +1,15 @@ +'use strict'; + +// The plan is authored and complete; implementation has not begun. + +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + m.discuss(h); + m.specify(h); + m.plan(h); + }, +}; diff --git a/tests/prose/fixtures/feature-planned/snapshot/.claude/settings.json b/tests/prose/cases/implementation-picks-first-task/fixture/.claude/settings.json similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.claude/settings.json rename to tests/prose/cases/implementation-picks-first-task/fixture/.claude/settings.json diff --git a/tests/prose/cases/implementation-picks-first-task/fixture/.recipe-hash b/tests/prose/cases/implementation-picks-first-task/fixture/.recipe-hash new file mode 100644 index 000000000..a29d91438 --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/fixture/.recipe-hash @@ -0,0 +1 @@ +e963357b5150c81c98156db6f527db5bf50bce292bb39b107e14ca98c6f5366a diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/.state/migrations b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/.state/migrations similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/.state/migrations rename to tests/prose/cases/implementation-picks-first-task/fixture/.workflows/.state/migrations diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/_gitignore.fixture b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/_gitignore.fixture similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/_gitignore.fixture rename to tests/prose/cases/implementation-picks-first-task/fixture/.workflows/_gitignore.fixture diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/manifest.json b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/manifest.json rename to tests/prose/cases/implementation-picks-first-task/fixture/.workflows/manifest.json diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/discovery/sessions/session-001.md similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/discovery/sessions/session-001.md rename to tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/discovery/sessions/session-001.md diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/discussion/pay.md b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/discussion/pay.md similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/discussion/pay.md rename to tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/discussion/pay.md diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/manifest.json b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/manifest.json rename to tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/manifest.json diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/planning/pay/planning.md b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/planning/pay/planning.md similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/planning/pay/planning.md rename to tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/planning/pay/planning.md diff --git a/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/planning/pay/tasks/pay-1-1.md b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/planning/pay/tasks/pay-1-1.md new file mode 100644 index 000000000..f7d43a082 --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/planning/pay/tasks/pay-1-1.md @@ -0,0 +1,10 @@ +--- +id: pay-1-1 +phase: 1 +status: pending +created: 2026-01-01 +--- + +# Create Payment Intent + +Create a gateway payment intent when checkout begins and attach it to the order. diff --git a/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/planning/pay/tasks/pay-1-2.md b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/planning/pay/tasks/pay-1-2.md new file mode 100644 index 000000000..8fb22697d --- /dev/null +++ b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/planning/pay/tasks/pay-1-2.md @@ -0,0 +1,10 @@ +--- +id: pay-1-2 +phase: 1 +status: pending +created: 2026-01-01 +--- + +# Handle Capture Webhooks + +Consume gateway capture webhooks and mark the order paid; no polling path. diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/specification/pay/specification.md b/tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/specification/pay/specification.md similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/specification/pay/specification.md rename to tests/prose/cases/implementation-picks-first-task/fixture/.workflows/pay/specification/pay/specification.md diff --git a/tests/prose/cases/investigation-entry-seeds-from-carrier/act.md b/tests/prose/cases/investigation-entry-seeds-from-carrier/act.md new file mode 100644 index 000000000..7165e44fd --- /dev/null +++ b/tests/prose/cases/investigation-entry-seeds-from-carrier/act.md @@ -0,0 +1,4 @@ +Execute skills/workflow-investigation-entry/SKILL.md with arguments +$0=bugfix, $1=crash-fix. Follow it to the point where the handoff to the +processing skill is constructed; record the handoff block and stop. Do +not execute the processing skill's instructions. diff --git a/tests/prose/cases/investigation-entry-seeds-from-carrier/assert.md b/tests/prose/cases/investigation-entry-seeds-from-carrier/assert.md new file mode 100644 index 000000000..7117e3603 --- /dev/null +++ b/tests/prose/cases/investigation-entry-seeds-from-carrier/assert.md @@ -0,0 +1,15 @@ +The prose should have taken this path: + +1. resolves the topic to the work unit, investigation being bugfix work +2. reads the investigation status, finds nothing, sets the source to new + and skips phase validation entirely +3. finds a discovery session log, so seeds the bug context from the + manifest description and that log — the context-gathering questions + are not asked +4. renders the phase note for the investigation phase with the verb + Starting, and emits it as produced +5. hands off to the investigation processing skill for crash-fix + +Further claims: + +- the investigation file is created by the processing skill, not here diff --git a/tests/prose/cases/investigation-entry-seeds-from-carrier/case.json b/tests/prose/cases/investigation-entry-seeds-from-carrier/case.json new file mode 100644 index 000000000..3eb224b9c --- /dev/null +++ b/tests/prose/cases/investigation-entry-seeds-from-carrier/case.json @@ -0,0 +1,7 @@ +{ + "origin": "bugfix mainline — a bug shaped in discovery is never re-interrogated", + "files": [ + "skills/workflow-investigation-entry/SKILL.md", + "skills/workflow-investigation-entry/references/invoke-skill.md" + ] +} diff --git a/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture-state.cjs b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture-state.cjs new file mode 100644 index 000000000..6cd47642e --- /dev/null +++ b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture-state.cjs @@ -0,0 +1,12 @@ +'use strict'; + +// The bug is captured and nothing has been started. + +const m = require('../../mainlines/bugfix.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + }, +}; diff --git a/tests/prose/fixtures/feature-specified/snapshot/.claude/settings.json b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.claude/settings.json similarity index 100% rename from tests/prose/fixtures/feature-specified/snapshot/.claude/settings.json rename to tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.claude/settings.json diff --git a/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.recipe-hash b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.recipe-hash new file mode 100644 index 000000000..259e11ddc --- /dev/null +++ b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.recipe-hash @@ -0,0 +1 @@ +f2d22a09d2680d82822f6042b87baf77dacbf41d4d8cb2b6d26a33d5c4129ff9 diff --git a/tests/prose/fixtures/feature-specified/snapshot/.workflows/.state/migrations b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/.state/migrations similarity index 100% rename from tests/prose/fixtures/feature-specified/snapshot/.workflows/.state/migrations rename to tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/.state/migrations diff --git a/tests/prose/fixtures/feature-specified/snapshot/.workflows/_gitignore.fixture b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/_gitignore.fixture similarity index 100% rename from tests/prose/fixtures/feature-specified/snapshot/.workflows/_gitignore.fixture rename to tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/_gitignore.fixture diff --git a/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/crash-fix/discovery/sessions/session-001.md b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/crash-fix/discovery/sessions/session-001.md new file mode 100644 index 000000000..6dc56d4e3 --- /dev/null +++ b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/crash-fix/discovery/sessions/session-001.md @@ -0,0 +1,40 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: crash-fix + +## Description (as of session) + +Checkout crashes when an order has no shipping address. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Reported by two users this week: the checkout page 500s at the +payment step for orders with no shipping address. Reproducible on +staging with a digital-only basket. No error surfaces to the user — +the page just fails. Confirmed as a bugfix; no design question here, +so it routes straight to investigation. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/crash-fix/manifest.json b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/crash-fix/manifest.json new file mode 100644 index 000000000..eaaa56199 --- /dev/null +++ b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/crash-fix/manifest.json @@ -0,0 +1,8 @@ +{ + "name": "crash-fix", + "work_type": "bugfix", + "status": "in-progress", + "created": "2026-01-01", + "description": "Checkout crashes when an order has no shipping address", + "phases": {} +} diff --git a/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/manifest.json b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/manifest.json new file mode 100644 index 000000000..1cd498ed5 --- /dev/null +++ b/tests/prose/cases/investigation-entry-seeds-from-carrier/fixture/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "crash-fix": { + "work_type": "bugfix" + } + } +} diff --git a/tests/prose/cases/planning-entry-asks-for-late-context/act.md b/tests/prose/cases/planning-entry-asks-for-late-context/act.md new file mode 100644 index 000000000..0ff96c62f --- /dev/null +++ b/tests/prose/cases/planning-entry-asks-for-late-context/act.md @@ -0,0 +1,4 @@ +Execute skills/workflow-planning-entry/SKILL.md with arguments +$0=feature, $1=pay. Follow it to the point where the handoff to the +processing skill is constructed; record the handoff block and stop. Do +not execute the processing skill's instructions. diff --git a/tests/prose/cases/planning-entry-asks-for-late-context/assert.md b/tests/prose/cases/planning-entry-asks-for-late-context/assert.md new file mode 100644 index 000000000..f1bf44650 --- /dev/null +++ b/tests/prose/cases/planning-entry-asks-for-late-context/assert.md @@ -0,0 +1,15 @@ +The prose should have taken this path: + +1. the specification gate renders empty — the spec is complete and the + topic is clear to plan +2. no plan exists, so the fresh-start arm asks whether anything has + changed since the specification, and waits +3. on continue, the source is fresh and carries no additional context +4. cross-cutting context is gathered regardless of work type: it finds + no cross-cutting work units, then queries the knowledge base for + completed cross-cutting specs +5. hands off to the planning processing skill for pay + +Further claims: + +- the knowledge base is queried, never written to, by an entry skill diff --git a/tests/prose/cases/planning-entry-asks-for-late-context/case.json b/tests/prose/cases/planning-entry-asks-for-late-context/case.json new file mode 100644 index 000000000..e5e2d1d46 --- /dev/null +++ b/tests/prose/cases/planning-entry-asks-for-late-context/case.json @@ -0,0 +1,12 @@ +{ + "origin": "feature mainline — planning offers a last chance to add context", + "files": [ + "skills/workflow-planning-entry/SKILL.md", + "skills/workflow-planning-entry/references/validate-spec.md", + "skills/workflow-planning-entry/references/validate-phase.md", + "skills/workflow-planning-entry/references/cross-cutting-context.md" + ], + "answers": [ + "continue — nothing has changed since the specification" + ] +} diff --git a/tests/prose/cases/planning-entry-asks-for-late-context/fixture-state.cjs b/tests/prose/cases/planning-entry-asks-for-late-context/fixture-state.cjs new file mode 100644 index 000000000..4113f7768 --- /dev/null +++ b/tests/prose/cases/planning-entry-asks-for-late-context/fixture-state.cjs @@ -0,0 +1,14 @@ +'use strict'; + +// The specification is complete; planning has not begun. + +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + m.discuss(h); + m.specify(h); + }, +}; diff --git a/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.claude/settings.json b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.recipe-hash b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.recipe-hash new file mode 100644 index 000000000..deb107e54 --- /dev/null +++ b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.recipe-hash @@ -0,0 +1 @@ +7c074ebb1f3729ec5ba32e89896c1d55de4c288de4cd3ddc8b93191813015936 diff --git a/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/.state/migrations b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/fixtures/feature-specified/snapshot/.workflows/manifest.json b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-specified/snapshot/.workflows/manifest.json rename to tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/manifest.json diff --git a/tests/prose/fixtures/feature-specified/snapshot/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/pay/discovery/sessions/session-001.md similarity index 100% rename from tests/prose/fixtures/feature-specified/snapshot/.workflows/pay/discovery/sessions/session-001.md rename to tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/pay/discovery/sessions/session-001.md diff --git a/tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/discussion/pay.md b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/pay/discussion/pay.md similarity index 100% rename from tests/prose/fixtures/feature-planned/snapshot/.workflows/pay/discussion/pay.md rename to tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/pay/discussion/pay.md diff --git a/tests/prose/fixtures/feature-specified/snapshot/.workflows/pay/manifest.json b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/pay/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-specified/snapshot/.workflows/pay/manifest.json rename to tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/pay/manifest.json diff --git a/tests/prose/fixtures/feature-specified/snapshot/.workflows/pay/specification/pay/specification.md b/tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/pay/specification/pay/specification.md similarity index 100% rename from tests/prose/fixtures/feature-specified/snapshot/.workflows/pay/specification/pay/specification.md rename to tests/prose/cases/planning-entry-asks-for-late-context/fixture/.workflows/pay/specification/pay/specification.md diff --git a/tests/prose/cases/review-entry-fresh/act.md b/tests/prose/cases/review-entry-fresh/act.md new file mode 100644 index 000000000..df21d57cb --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/act.md @@ -0,0 +1,4 @@ +Execute skills/workflow-review-entry/SKILL.md with arguments $0=feature, +$1=pay. Follow it to the point where the handoff to the processing skill +is constructed; record the handoff block and stop. Do not execute the +processing skill's instructions, and dispatch no agents. diff --git a/tests/prose/cases/review-entry-fresh/assert.md b/tests/prose/cases/review-entry-fresh/assert.md new file mode 100644 index 000000000..c1305bb9b --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/assert.md @@ -0,0 +1,12 @@ +The prose should have taken this path: + +1. resolves the topic to the work unit +2. the prerequisite gate renders empty — both the plan and the + implementation are complete, so nothing blocks entry +3. no review item exists, so the fresh path is taken — nothing is + reopened and no resume is announced +4. hands off to the review processing skill for pay + +Further claims: + +- no verifier agents are dispatched by the entry skill diff --git a/tests/prose/cases/review-entry-fresh/case.json b/tests/prose/cases/review-entry-fresh/case.json new file mode 100644 index 000000000..c67cada98 --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/case.json @@ -0,0 +1,7 @@ +{ + "origin": "feature mainline — review refuses to start until delivery is complete", + "files": [ + "skills/workflow-review-entry/SKILL.md", + "skills/workflow-review-entry/references/validate-phase.md" + ] +} diff --git a/tests/prose/cases/review-entry-fresh/fixture-state.cjs b/tests/prose/cases/review-entry-fresh/fixture-state.cjs new file mode 100644 index 000000000..3502e883d --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/fixture-state.cjs @@ -0,0 +1,16 @@ +'use strict'; + +// Every task is built and implementation is complete; review has not begun. + +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + m.discuss(h); + m.specify(h); + m.plan(h); + m.implement(h); + }, +}; diff --git a/tests/prose/cases/review-entry-fresh/fixture/.claude/settings.json b/tests/prose/cases/review-entry-fresh/fixture/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/fixture/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/review-entry-fresh/fixture/.recipe-hash b/tests/prose/cases/review-entry-fresh/fixture/.recipe-hash new file mode 100644 index 000000000..e04b82514 --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/fixture/.recipe-hash @@ -0,0 +1 @@ +7289bcc8afeefb182d37e5ca576f35e76a28fec2c91f3738062049801773d0cb diff --git a/tests/prose/cases/review-entry-fresh/fixture/.workflows/.state/migrations b/tests/prose/cases/review-entry-fresh/fixture/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/fixture/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/review-entry-fresh/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/review-entry-fresh/fixture/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/fixture/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/cases/review-entry-fresh/fixture/.workflows/manifest.json b/tests/prose/cases/review-entry-fresh/fixture/.workflows/manifest.json new file mode 100644 index 000000000..987f6c236 --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/fixture/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "pay": { + "work_type": "feature" + } + } +} diff --git a/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/discovery/sessions/session-001.md new file mode 100644 index 000000000..2aab96961 --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/discovery/sessions/session-001.md @@ -0,0 +1,39 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: pay + +## Description (as of session) + +Accept card payments at checkout. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Shaped as a single feature: accept card payments at checkout using +the existing gateway account. Card-only for v1 came up early and was +softly agreed; wallet support was noted as a likely deferral. No +research need surfaced — routed straight to discussion. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/fixtures/feature-specified/snapshot/.workflows/pay/discussion/pay.md b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/discussion/pay.md similarity index 100% rename from tests/prose/fixtures/feature-specified/snapshot/.workflows/pay/discussion/pay.md rename to tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/discussion/pay.md diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/manifest.json b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/manifest.json rename to tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/manifest.json diff --git a/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/planning/pay/planning.md b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/planning/pay/planning.md new file mode 100644 index 000000000..9131d98c7 --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/planning/pay/planning.md @@ -0,0 +1,8 @@ +# Plan — pay + +## Phase 1: Payment core + +| Task | Title | +|------|-------| +| pay-1-1 | Create Payment Intent | +| pay-1-2 | Handle Capture Webhooks | diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/planning/pay/tasks/pay-1-1.md b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/planning/pay/tasks/pay-1-1.md similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/planning/pay/tasks/pay-1-1.md rename to tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/planning/pay/tasks/pay-1-1.md diff --git a/tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/planning/pay/tasks/pay-1-2.md b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/planning/pay/tasks/pay-1-2.md similarity index 100% rename from tests/prose/fixtures/feature-implemented/snapshot/.workflows/pay/planning/pay/tasks/pay-1-2.md rename to tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/planning/pay/tasks/pay-1-2.md diff --git a/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/specification/pay/specification.md b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/specification/pay/specification.md new file mode 100644 index 000000000..29a5f4322 --- /dev/null +++ b/tests/prose/cases/review-entry-fresh/fixture/.workflows/pay/specification/pay/specification.md @@ -0,0 +1,11 @@ +# Specification — pay + +## Requirements + +- Checkout creates a payment intent against the existing gateway account. +- Card payments only; wallet flows are out of scope for v1. +- Capture is confirmed by gateway webhook, never by polling. + +## Out of scope + +- Wallet support (deferred by discussion). diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/act.md b/tests/prose/cases/root-cause-validation-clean-verdict/act.md new file mode 100644 index 000000000..c3225285c --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/act.md @@ -0,0 +1,2 @@ +Walk root-cause-validation.md from the top, as the processing skill +would. Stop when the reference returns to its caller. diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assert.md b/tests/prose/cases/root-cause-validation-clean-verdict/assert.md new file mode 100644 index 000000000..f00e364c7 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assert.md @@ -0,0 +1,19 @@ +The prose should have taken this path: + +1. offers the independent validation, presents the run-or-skip choice, + and waits +2. on yes, records the dispatch through the engine and creates no report + file of its own — the response carries the id and the path to use +3. announces that validation is running, then dispatches exactly one + agent, synchronously +4. once the report has landed, promotes the row with a scan and closes + it with an incorporate — the verdict is consumed whole, never + surfaced finding by finding +5. reads the report and, the verdict being validated, states the + confidence and returns to its caller — the gap-handling choice is + never reached + +Further claims: + +- the investigation file is untouched and nothing is committed: the + validated path has no gaps to record diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion-state.cjs b/tests/prose/cases/root-cause-validation-clean-verdict/assertion-state.cjs new file mode 100644 index 000000000..12c381e52 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion-state.cjs @@ -0,0 +1,27 @@ +'use strict'; + +// The fixture state plus what the walk should have done: the dispatch +// recorded, the agent's report landed at the path the dispatch returned, +// scan promoting the row and incorporate closing it. The report content +// comes from the same stub the case arms, so the bytes the walk writes +// and the bytes expected here cannot drift apart. + +const fs = require('fs'); +const path = require('path'); + +const fixture = require('./fixture-state.cjs'); +const m = require('../../mainlines/bugfix.cjs'); +const { readStub } = require('../../lib/cases.cjs'); + +module.exports = { + build(h) { + fixture.build(h); + + const dispatch = JSON.parse(h.engine( + 'agent', 'dispatch', m.WU, 'investigation', m.WU, '--kind', 'root-cause-validation')); + fs.mkdirSync(path.dirname(path.join(h.dir, dispatch.file)), { recursive: true }); + h.write(dispatch.file, readStub('root-cause-validated').content); + h.engine('agent', 'scan', m.WU, 'investigation', m.WU); + h.engine('agent', 'incorporate', m.WU, 'investigation', m.WU, dispatch.id); + }, +}; diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.claude/settings.json b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.recipe-hash b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.recipe-hash new file mode 100644 index 000000000..d38ba55a6 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.recipe-hash @@ -0,0 +1 @@ +b65c43b1fced1a2d5742f96a2d6d1a8278f67d44c143552dbb42bee06badded6 diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/.cache/crash-fix/investigation/crash-fix/root-cause-validation-001.md b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/.cache/crash-fix/investigation/crash-fix/root-cause-validation-001.md new file mode 100644 index 000000000..ce1c9b4c4 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/.cache/crash-fix/investigation/crash-fix/root-cause-validation-001.md @@ -0,0 +1,10 @@ +# Root Cause Validation + +The tax context is built before the payment intent and reads the shipping +address unconditionally. Traced fresh: an address-less order fails at that +read, which matches the documented root cause. + +STATUS: validated +CONFIDENCE: high +GAPS_COUNT: 0 +SUMMARY: Root cause confirmed by an independent trace. diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/.cache/crash-fix/investigation/crash-fix/state.json b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/.cache/crash-fix/investigation/crash-fix/state.json new file mode 100644 index 000000000..7eb5a78f1 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/.cache/crash-fix/investigation/crash-fix/state.json @@ -0,0 +1,16 @@ +{ + "agents": { + "root-cause-validation-001": { + "id": "root-cause-validation-001", + "kind": "root-cause-validation", + "phase": "investigation", + "topic": "crash-fix", + "set": "001", + "status": "incorporated", + "announced": false, + "findings": [], + "surfaced": [], + "created": "2026-01-01T00:00:00.000Z" + } + } +} diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/.state/migrations b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/_gitignore.fixture b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/crash-fix/discovery/sessions/session-001.md b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/crash-fix/discovery/sessions/session-001.md new file mode 100644 index 000000000..6dc56d4e3 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/crash-fix/discovery/sessions/session-001.md @@ -0,0 +1,40 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: crash-fix + +## Description (as of session) + +Checkout crashes when an order has no shipping address. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Reported by two users this week: the checkout page 500s at the +payment step for orders with no shipping address. Reproducible on +staging with a digital-only basket. No error surfaces to the user — +the page just fails. Confirmed as a bugfix; no design question here, +so it routes straight to investigation. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/crash-fix/investigation/crash-fix.md b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/crash-fix/investigation/crash-fix.md new file mode 100644 index 000000000..d5fc263a9 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/crash-fix/investigation/crash-fix.md @@ -0,0 +1,88 @@ +# Investigation: Checkout Crash On Missing Shipping Address + +## Symptoms + +### Problem Description + +Checkout returns a 500 at the payment step when the order has no +shipping address. No error is surfaced to the user. + +### Manifestation + +Server error on POST to the payment step; the page fails blank. + +### Reproduction Steps + +1. Add a digital-only product to the basket. +2. Proceed to checkout without entering a shipping address. +3. Continue to the payment step — the request 500s. + +### Environment + +Reproduced on staging; reported twice in production this week. + +### Impact + +Digital-only orders cannot complete checkout. + +### References + +(none) + +## Analysis + +### Hypotheses + +- The payment step assumes a shipping address is always present. +- Tax calculation may be the first consumer of the missing address. + +### Code Trace + +The payment step builds a tax context from the order before creating +the payment intent. The tax context reads the shipping address +unconditionally; for a digital-only order the address is absent, so +the read fails and the request aborts before any handler catches it. + +### Root Cause + +The tax context treats the shipping address as mandatory. Digital-only +orders legitimately have none, so the assumption is wrong rather than +the data being wrong. + +### Contributing Factors + +Digital-only baskets were introduced after the tax context was written. + +### Why It Wasn't Caught + +No test covers a basket with no shippable line items. + +### Blast Radius + +Any flow building a tax context from an address-less order. + +## Fix Direction + +### Chosen Approach + +(pending) + +### Options Explored + +(none yet) + +### Discussion + +(none yet) + +### Testing Recommendations + +(pending) + +### Risk Assessment + +(pending) + +## Notes + +(none) diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/crash-fix/manifest.json b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/crash-fix/manifest.json new file mode 100644 index 000000000..88bf183ff --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/crash-fix/manifest.json @@ -0,0 +1,16 @@ +{ + "name": "crash-fix", + "work_type": "bugfix", + "status": "in-progress", + "created": "2026-01-01", + "description": "Checkout crashes when an order has no shipping address", + "phases": { + "investigation": { + "items": { + "crash-fix": { + "status": "in-progress" + } + } + } + } +} diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/manifest.json b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/manifest.json new file mode 100644 index 000000000..1cd498ed5 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/assertion/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "crash-fix": { + "work_type": "bugfix" + } + } +} diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/case.json b/tests/prose/cases/root-cause-validation-clean-verdict/case.json new file mode 100644 index 000000000..aab675685 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/case.json @@ -0,0 +1,12 @@ +{ + "origin": "bugfix mainline — the validation agent's lifecycle on a clean verdict", + "files": [ + "skills/workflow-investigation-process/references/root-cause-validation.md" + ], + "answers": [ + "yes — run root cause validation" + ], + "stubs": { + "root-cause-validated": "when the engine records a dispatch of kind root-cause-validation" + } +} diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture-state.cjs b/tests/prose/cases/root-cause-validation-clean-verdict/fixture-state.cjs new file mode 100644 index 000000000..67d91fdbf --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture-state.cjs @@ -0,0 +1,13 @@ +'use strict'; + +// The investigation is open with its root cause documented, and no agent has run. + +const m = require('../../mainlines/bugfix.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + m.investigateToRootCause(h); + }, +}; diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture.md b/tests/prose/cases/root-cause-validation-clean-verdict/fixture.md new file mode 100644 index 000000000..95d92d942 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture.md @@ -0,0 +1,3 @@ +The investigation is open with its symptoms gathered and its root cause +documented. The processing skill has worked down to its root-cause +validation step; nothing has been validated yet and no agent has run. diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.claude/settings.json b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.recipe-hash b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.recipe-hash new file mode 100644 index 000000000..d38ba55a6 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.recipe-hash @@ -0,0 +1 @@ +b65c43b1fced1a2d5742f96a2d6d1a8278f67d44c143552dbb42bee06badded6 diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/.state/migrations b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/crash-fix/discovery/sessions/session-001.md b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/crash-fix/discovery/sessions/session-001.md new file mode 100644 index 000000000..6dc56d4e3 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/crash-fix/discovery/sessions/session-001.md @@ -0,0 +1,40 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: crash-fix + +## Description (as of session) + +Checkout crashes when an order has no shipping address. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Reported by two users this week: the checkout page 500s at the +payment step for orders with no shipping address. Reproducible on +staging with a digital-only basket. No error surfaces to the user — +the page just fails. Confirmed as a bugfix; no design question here, +so it routes straight to investigation. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/crash-fix/investigation/crash-fix.md b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/crash-fix/investigation/crash-fix.md new file mode 100644 index 000000000..d5fc263a9 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/crash-fix/investigation/crash-fix.md @@ -0,0 +1,88 @@ +# Investigation: Checkout Crash On Missing Shipping Address + +## Symptoms + +### Problem Description + +Checkout returns a 500 at the payment step when the order has no +shipping address. No error is surfaced to the user. + +### Manifestation + +Server error on POST to the payment step; the page fails blank. + +### Reproduction Steps + +1. Add a digital-only product to the basket. +2. Proceed to checkout without entering a shipping address. +3. Continue to the payment step — the request 500s. + +### Environment + +Reproduced on staging; reported twice in production this week. + +### Impact + +Digital-only orders cannot complete checkout. + +### References + +(none) + +## Analysis + +### Hypotheses + +- The payment step assumes a shipping address is always present. +- Tax calculation may be the first consumer of the missing address. + +### Code Trace + +The payment step builds a tax context from the order before creating +the payment intent. The tax context reads the shipping address +unconditionally; for a digital-only order the address is absent, so +the read fails and the request aborts before any handler catches it. + +### Root Cause + +The tax context treats the shipping address as mandatory. Digital-only +orders legitimately have none, so the assumption is wrong rather than +the data being wrong. + +### Contributing Factors + +Digital-only baskets were introduced after the tax context was written. + +### Why It Wasn't Caught + +No test covers a basket with no shippable line items. + +### Blast Radius + +Any flow building a tax context from an address-less order. + +## Fix Direction + +### Chosen Approach + +(pending) + +### Options Explored + +(none yet) + +### Discussion + +(none yet) + +### Testing Recommendations + +(pending) + +### Risk Assessment + +(pending) + +## Notes + +(none) diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/crash-fix/manifest.json b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/crash-fix/manifest.json new file mode 100644 index 000000000..88bf183ff --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/crash-fix/manifest.json @@ -0,0 +1,16 @@ +{ + "name": "crash-fix", + "work_type": "bugfix", + "status": "in-progress", + "created": "2026-01-01", + "description": "Checkout crashes when an order has no shipping address", + "phases": { + "investigation": { + "items": { + "crash-fix": { + "status": "in-progress" + } + } + } + } +} diff --git a/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/manifest.json b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/manifest.json new file mode 100644 index 000000000..1cd498ed5 --- /dev/null +++ b/tests/prose/cases/root-cause-validation-clean-verdict/fixture/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "crash-fix": { + "work_type": "bugfix" + } + } +} diff --git a/tests/prose/cases/smoke-boot-in-world/act.md b/tests/prose/cases/smoke-boot-in-world/act.md new file mode 100644 index 000000000..bb4d2a56b --- /dev/null +++ b/tests/prose/cases/smoke-boot-in-world/act.md @@ -0,0 +1,4 @@ +In the project, run the boot command exactly as the initialisation prose +of skills/workflow-start/SKILL.md prescribes — sandbox concerns do not +apply in this world, so run it directly. Capture the JSON response, then +stop; do not continue into any confirmation or later step. diff --git a/tests/prose/cases/smoke-boot-in-world/assert.md b/tests/prose/cases/smoke-boot-in-world/assert.md new file mode 100644 index 000000000..da99ef3ce --- /dev/null +++ b/tests/prose/cases/smoke-boot-in-world/assert.md @@ -0,0 +1,10 @@ +The prose should have taken this path: + +1. runs the boot command and reports the knowledge base ready, in + keyword-only mode +2. reports no migrations applied — this world is already migrated — so + the migrations summary and its confirmation are not raised + +Further claims: + +- booting an already-migrated project changes nothing on disk diff --git a/tests/prose/cases/smoke-boot-in-world/case.json b/tests/prose/cases/smoke-boot-in-world/case.json new file mode 100644 index 000000000..047c10686 --- /dev/null +++ b/tests/prose/cases/smoke-boot-in-world/case.json @@ -0,0 +1,6 @@ +{ + "origin": "framework smoke — world build, engine-in-world, world comparison", + "files": [ + "skills/workflow-start/SKILL.md#Boot" + ] +} diff --git a/tests/prose/cases/smoke-boot-in-world/fixture-state.cjs b/tests/prose/cases/smoke-boot-in-world/fixture-state.cjs new file mode 100644 index 000000000..8e39fface --- /dev/null +++ b/tests/prose/cases/smoke-boot-in-world/fixture-state.cjs @@ -0,0 +1,11 @@ +'use strict'; + +// A bare installed project: migrations applied, knowledge base set up +// keyword-only, no work units at all. + +module.exports = { + build(h) { + h.knowledge('setup', '--keyword-only'); + h.engine('boot'); + }, +}; diff --git a/tests/prose/cases/smoke-boot-in-world/fixture/.claude/settings.json b/tests/prose/cases/smoke-boot-in-world/fixture/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/smoke-boot-in-world/fixture/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/smoke-boot-in-world/fixture/.recipe-hash b/tests/prose/cases/smoke-boot-in-world/fixture/.recipe-hash new file mode 100644 index 000000000..27528f6e9 --- /dev/null +++ b/tests/prose/cases/smoke-boot-in-world/fixture/.recipe-hash @@ -0,0 +1 @@ +e75a843d66eafa8a47fd143e1e3b2eb41e0a6b7b2b2d225753bec1cdee9892c5 diff --git a/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/.state/migrations b/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/cases/smoke-start-boot-structure/act.md b/tests/prose/cases/smoke-start-boot-structure/act.md new file mode 100644 index 000000000..cc3f6a902 --- /dev/null +++ b/tests/prose/cases/smoke-start-boot-structure/act.md @@ -0,0 +1,5 @@ +Structure-only: read the initialisation portion of +skills/workflow-start/SKILL.md — everything the skill does before any +work is shown or routed. Establish what initialisation consists of and +in what order its parts run. Stop there; do not trace the rest of the +skill. diff --git a/tests/prose/cases/smoke-start-boot-structure/assert.md b/tests/prose/cases/smoke-start-boot-structure/assert.md new file mode 100644 index 000000000..058974b50 --- /dev/null +++ b/tests/prose/cases/smoke-start-boot-structure/assert.md @@ -0,0 +1,7 @@ +The prose should have taken this path: + +1. casing conventions are loaded before the boot pipeline runs +2. the boot pipeline is declared mandatory — it must complete before the + skill proceeds +3. the knowledge gate is reached only after boot, and branches on what + boot reported rather than on its own checks diff --git a/tests/prose/cases/smoke-start-boot-structure/case.json b/tests/prose/cases/smoke-start-boot-structure/case.json new file mode 100644 index 000000000..07b9e0424 --- /dev/null +++ b/tests/prose/cases/smoke-start-boot-structure/case.json @@ -0,0 +1,6 @@ +{ + "origin": "framework smoke — a structure-only walk, no world", + "files": [ + "skills/workflow-start/SKILL.md#Boot" + ] +} diff --git a/tests/prose/cases/spec-entry-from-discussion/act.md b/tests/prose/cases/spec-entry-from-discussion/act.md new file mode 100644 index 000000000..a76fbb90c --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/act.md @@ -0,0 +1,4 @@ +Execute skills/workflow-specification-entry/SKILL.md with arguments +$0=feature, $1=pay. Follow it to the point where the handoff to the +processing skill is constructed; record the handoff block and stop. Do +not execute the processing skill's instructions. diff --git a/tests/prose/cases/spec-entry-from-discussion/assert.md b/tests/prose/cases/spec-entry-from-discussion/assert.md new file mode 100644 index 000000000..59ebce857 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/assert.md @@ -0,0 +1,13 @@ +The prose should have taken this path: + +1. resolves the topic to the work unit and goes straight to validating + source material — the scoped path is for an epic with no topic +2. the source-material gate renders empty: the completed discussion + satisfies the prerequisite +3. no specification item exists, so the verb is Creating — no resume + note is rendered and nothing is reopened +4. hands off naming `.workflows/pay/discussion/pay.md` as the source + +Further claims: + +- an entry that only validates writes nothing diff --git a/tests/prose/cases/spec-entry-from-discussion/case.json b/tests/prose/cases/spec-entry-from-discussion/case.json new file mode 100644 index 000000000..f380e57ce --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/case.json @@ -0,0 +1,9 @@ +{ + "origin": "feature mainline — a feature's spec draws on its discussion", + "files": [ + "skills/workflow-specification-entry/SKILL.md", + "skills/workflow-specification-entry/references/validate-source.md", + "skills/workflow-specification-entry/references/validate-phase.md", + "skills/workflow-specification-entry/references/invoke-skill.md" + ] +} diff --git a/tests/prose/cases/spec-entry-from-discussion/fixture-state.cjs b/tests/prose/cases/spec-entry-from-discussion/fixture-state.cjs new file mode 100644 index 000000000..a1a1b7964 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/fixture-state.cjs @@ -0,0 +1,13 @@ +'use strict'; + +// The discussion is complete; the specification has not begun. + +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + m.discuss(h); + }, +}; diff --git a/tests/prose/cases/spec-entry-from-discussion/fixture/.claude/settings.json b/tests/prose/cases/spec-entry-from-discussion/fixture/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/fixture/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/spec-entry-from-discussion/fixture/.recipe-hash b/tests/prose/cases/spec-entry-from-discussion/fixture/.recipe-hash new file mode 100644 index 000000000..336903a57 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/fixture/.recipe-hash @@ -0,0 +1 @@ +ed6d01566a29220645f5ff7b62dd7d748b55eec7036c00af9bd68089a6091c61 diff --git a/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/.state/migrations b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/manifest.json b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/manifest.json new file mode 100644 index 000000000..987f6c236 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "pay": { + "work_type": "feature" + } + } +} diff --git a/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/pay/discovery/sessions/session-001.md new file mode 100644 index 000000000..2aab96961 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/pay/discovery/sessions/session-001.md @@ -0,0 +1,39 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: pay + +## Description (as of session) + +Accept card payments at checkout. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Shaped as a single feature: accept card payments at checkout using +the existing gateway account. Card-only for v1 came up early and was +softly agreed; wallet support was noted as a likely deferral. No +research need surfaced — routed straight to discussion. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/pay/discussion/pay.md b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/pay/discussion/pay.md new file mode 100644 index 000000000..ec0cfc62f --- /dev/null +++ b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/pay/discussion/pay.md @@ -0,0 +1,15 @@ +# Discussion — pay + +## Context + +Accept card payments at checkout using the existing gateway account. + +## Decisions + +- Use the existing gateway account; no new provider onboarding. +- Card-only for v1 — wallet support is deferred. +- Webhooks confirm capture; the checkout never polls. + +## Deferred + +- Wallet support (Apple/Google Pay) — revisit after v1. diff --git a/tests/prose/fixtures/feature-discussed/snapshot/.workflows/pay/manifest.json b/tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/pay/manifest.json similarity index 100% rename from tests/prose/fixtures/feature-discussed/snapshot/.workflows/pay/manifest.json rename to tests/prose/cases/spec-entry-from-discussion/fixture/.workflows/pay/manifest.json diff --git a/tests/prose/cases/spec-entry-from-investigation/act.md b/tests/prose/cases/spec-entry-from-investigation/act.md new file mode 100644 index 000000000..a2bd56697 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/act.md @@ -0,0 +1,4 @@ +Execute skills/workflow-specification-entry/SKILL.md with arguments +$0=bugfix, $1=crash-fix. Follow it to the point where the handoff to the +processing skill is constructed; record the handoff block and stop. Do +not execute the processing skill's instructions. diff --git a/tests/prose/cases/spec-entry-from-investigation/assert.md b/tests/prose/cases/spec-entry-from-investigation/assert.md new file mode 100644 index 000000000..85c1245e4 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/assert.md @@ -0,0 +1,13 @@ +The prose should have taken this path: + +1. resolves the topic to the work unit and goes to validating source + material +2. the source-material gate renders empty: for a bugfix the completed + investigation is what satisfies it, in place of a discussion +3. no specification item exists, so the verb is Creating +4. hands off naming `.workflows/crash-fix/investigation/crash-fix.md` as + the source material — no discussion file is named + +Further claims: + +- the same prose serves both work types; only the source arm differs diff --git a/tests/prose/cases/spec-entry-from-investigation/case.json b/tests/prose/cases/spec-entry-from-investigation/case.json new file mode 100644 index 000000000..3eed6746c --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/case.json @@ -0,0 +1,8 @@ +{ + "origin": "bugfix mainline — the same entry skill, sourced from an investigation", + "files": [ + "skills/workflow-specification-entry/SKILL.md", + "skills/workflow-specification-entry/references/validate-source.md", + "skills/workflow-specification-entry/references/invoke-skill.md" + ] +} diff --git a/tests/prose/cases/spec-entry-from-investigation/fixture-state.cjs b/tests/prose/cases/spec-entry-from-investigation/fixture-state.cjs new file mode 100644 index 000000000..a97e20061 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/fixture-state.cjs @@ -0,0 +1,14 @@ +'use strict'; + +// The investigation is concluded; the specification has not begun. + +const m = require('../../mainlines/bugfix.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + m.investigateToRootCause(h); + m.concludeInvestigation(h); + }, +}; diff --git a/tests/prose/cases/spec-entry-from-investigation/fixture/.claude/settings.json b/tests/prose/cases/spec-entry-from-investigation/fixture/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/fixture/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/spec-entry-from-investigation/fixture/.recipe-hash b/tests/prose/cases/spec-entry-from-investigation/fixture/.recipe-hash new file mode 100644 index 000000000..1d4381905 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/fixture/.recipe-hash @@ -0,0 +1 @@ +b34ddf2eff3b10056fb8d6c0e69ede001d06a863d3990e8752b0a5d4fdcbd1a3 diff --git a/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/.state/migrations b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/crash-fix/discovery/sessions/session-001.md b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/crash-fix/discovery/sessions/session-001.md new file mode 100644 index 000000000..6dc56d4e3 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/crash-fix/discovery/sessions/session-001.md @@ -0,0 +1,40 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: crash-fix + +## Description (as of session) + +Checkout crashes when an order has no shipping address. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Reported by two users this week: the checkout page 500s at the +payment step for orders with no shipping address. Reproducible on +staging with a digital-only basket. No error surfaces to the user — +the page just fails. Confirmed as a bugfix; no design question here, +so it routes straight to investigation. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/crash-fix/investigation/crash-fix.md b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/crash-fix/investigation/crash-fix.md new file mode 100644 index 000000000..d09b4daba --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/crash-fix/investigation/crash-fix.md @@ -0,0 +1,90 @@ +# Investigation: Checkout Crash On Missing Shipping Address + +## Symptoms + +### Problem Description + +Checkout returns a 500 at the payment step when the order has no +shipping address. No error is surfaced to the user. + +### Manifestation + +Server error on POST to the payment step; the page fails blank. + +### Reproduction Steps + +1. Add a digital-only product to the basket. +2. Proceed to checkout without entering a shipping address. +3. Continue to the payment step — the request 500s. + +### Environment + +Reproduced on staging; reported twice in production this week. + +### Impact + +Digital-only orders cannot complete checkout. + +### References + +(none) + +## Analysis + +### Hypotheses + +- The payment step assumes a shipping address is always present. +- Tax calculation may be the first consumer of the missing address. + +### Code Trace + +The payment step builds a tax context from the order before creating +the payment intent. The tax context reads the shipping address +unconditionally; for a digital-only order the address is absent, so +the read fails and the request aborts before any handler catches it. + +### Root Cause + +The tax context treats the shipping address as mandatory. Digital-only +orders legitimately have none, so the assumption is wrong rather than +the data being wrong. + +### Contributing Factors + +Digital-only baskets were introduced after the tax context was written. + +### Why It Wasn't Caught + +No test covers a basket with no shippable line items. + +### Blast Radius + +Any flow building a tax context from an address-less order. + +## Fix Direction + +### Chosen Approach + +Make the tax context treat the shipping address as optional: when +an order has no shippable items, build the context from the billing +address instead. + +### Options Explored + +(none yet) + +### Discussion + +(none yet) + +### Testing Recommendations + +Cover a digital-only basket end to end through the payment step. + +### Risk Assessment + +Low — the change is confined to tax-context construction. + +## Notes + +(none) diff --git a/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/crash-fix/manifest.json b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/crash-fix/manifest.json new file mode 100644 index 000000000..4c5ece2e7 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/crash-fix/manifest.json @@ -0,0 +1,16 @@ +{ + "name": "crash-fix", + "work_type": "bugfix", + "status": "in-progress", + "created": "2026-01-01", + "description": "Checkout crashes when an order has no shipping address", + "phases": { + "investigation": { + "items": { + "crash-fix": { + "status": "completed" + } + } + } + } +} diff --git a/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/manifest.json b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/manifest.json new file mode 100644 index 000000000..1cd498ed5 --- /dev/null +++ b/tests/prose/cases/spec-entry-from-investigation/fixture/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "crash-fix": { + "work_type": "bugfix" + } + } +} diff --git a/tests/prose/cases/start-lists-active-work/act.md b/tests/prose/cases/start-lists-active-work/act.md new file mode 100644 index 000000000..b4fa96524 --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/act.md @@ -0,0 +1,3 @@ +Execute skills/workflow-start/SKILL.md from the top. Continue until the +work dashboard and its menu have been shown. Record both verbatim, then +stop — select nothing. diff --git a/tests/prose/cases/start-lists-active-work/assert.md b/tests/prose/cases/start-lists-active-work/assert.md new file mode 100644 index 000000000..e0842d0b6 --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/assert.md @@ -0,0 +1,14 @@ +The prose should have taken this path: + +1. loads the shared casing conventions before any state is read +2. runs the boot pipeline, and since no migrations applied and the + knowledge base is ready, raises neither the migrations confirmation + nor the knowledge gate +3. gets the workflow state from the discovery gateway script rather than + listing directories or reading files itself +4. shows `pay` as the only active work, with a menu whose continue action + leads into the per-type navigation for a feature + +Further claims: + +- showing work is a read: nothing is created, recorded, or committed diff --git a/tests/prose/cases/start-lists-active-work/case.json b/tests/prose/cases/start-lists-active-work/case.json new file mode 100644 index 000000000..59711da9f --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/case.json @@ -0,0 +1,6 @@ +{ + "origin": "feature mainline — the entry skill surfaces work without touching it", + "files": [ + "skills/workflow-start/SKILL.md" + ] +} diff --git a/tests/prose/cases/start-lists-active-work/fixture-state.cjs b/tests/prose/cases/start-lists-active-work/fixture-state.cjs new file mode 100644 index 000000000..e4b78cd71 --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/fixture-state.cjs @@ -0,0 +1,12 @@ +'use strict'; + +// The feature exists and nothing has been started. + +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + }, +}; diff --git a/tests/prose/cases/start-lists-active-work/fixture/.claude/settings.json b/tests/prose/cases/start-lists-active-work/fixture/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/fixture/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/start-lists-active-work/fixture/.recipe-hash b/tests/prose/cases/start-lists-active-work/fixture/.recipe-hash new file mode 100644 index 000000000..92519878f --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/fixture/.recipe-hash @@ -0,0 +1 @@ +5b0a7d802b3a436a4656bc000122451bb015f2250f8fef47b1d05f3ea51ba96b diff --git a/tests/prose/cases/start-lists-active-work/fixture/.workflows/.state/migrations b/tests/prose/cases/start-lists-active-work/fixture/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/fixture/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/start-lists-active-work/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/start-lists-active-work/fixture/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/fixture/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/cases/start-lists-active-work/fixture/.workflows/manifest.json b/tests/prose/cases/start-lists-active-work/fixture/.workflows/manifest.json new file mode 100644 index 000000000..987f6c236 --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/fixture/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "pay": { + "work_type": "feature" + } + } +} diff --git a/tests/prose/cases/start-lists-active-work/fixture/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/start-lists-active-work/fixture/.workflows/pay/discovery/sessions/session-001.md new file mode 100644 index 000000000..2aab96961 --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/fixture/.workflows/pay/discovery/sessions/session-001.md @@ -0,0 +1,39 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: pay + +## Description (as of session) + +Accept card payments at checkout. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Shaped as a single feature: accept card payments at checkout using +the existing gateway account. Card-only for v1 came up early and was +softly agreed; wallet support was noted as a likely deferral. No +research need surfaced — routed straight to discussion. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/cases/start-lists-active-work/fixture/.workflows/pay/manifest.json b/tests/prose/cases/start-lists-active-work/fixture/.workflows/pay/manifest.json new file mode 100644 index 000000000..1cf8afb0e --- /dev/null +++ b/tests/prose/cases/start-lists-active-work/fixture/.workflows/pay/manifest.json @@ -0,0 +1,8 @@ +{ + "name": "pay", + "work_type": "feature", + "status": "in-progress", + "created": "2026-01-01", + "description": "Accept card payments at checkout", + "phases": {} +} diff --git a/tests/prose/feature/cases.md b/tests/prose/feature/cases.md deleted file mode 100644 index 0480096d7..000000000 --- a/tests/prose/feature/cases.md +++ /dev/null @@ -1,180 +0,0 @@ -# Feature mainline — happy-path corpus - -The canonical feature `pay` walked through its pipeline: navigation, -each phase entry, and the implementation pickup. Worlds are the -`feature-*` fixtures (stop points along `_shared/feature-mainline.cjs`). -Every claim here was verified against the prose at authoring time — -a failure means the behaviour moved, not the numbering. - -## case: feature-start-lists-fresh -- world: feature-created -- origin: feature mainline — workflow-start surfaces active work -- files: - - skills/workflow-start/SKILL.md - -### walk - -Execute skills/workflow-start/SKILL.md from the top. Initialisation -will find the system already migrated and the knowledge base ready, so -no migration summary or setup conversation applies. Continue until the -work dashboard and its menu are shown. Record both verbatim and stop — -select nothing. - -### expect - -- routing: initialisation runs the boot pipeline before any work is shown, and with `migrations.changed` false no confirmation gate is raised -- routing: the dashboard lists `pay` as active work -- routing: the menu offers continuing existing work (the path that routes feature work to workflow-continue-feature) -- state: manifest absent pay.discussion.pay status - -## case: feature-continue-offers-discussion -- world: feature-created -- origin: feature mainline — continue-feature shows state and routes forward -- files: - - skills/workflow-continue-feature/SKILL.md - - skills/workflow-continue-feature/references/select-feature.md - - skills/workflow-continue-feature/references/feature-display-and-menu.md - -### walk - -Execute skills/workflow-continue-feature/SKILL.md with no arguments -(as routed from workflow-start with no pre-selected work unit). Follow -selection, then the feature state display and its menu. Record the -state display and menu verbatim, then stop — do not route into any -phase entry skill. - -### user - -1. Select the feature `pay` (its number in the selection menu) - -### expect - -- routing: the selection menu is shown even though only one feature exists — no auto-select -- routing: the feature state shows no phase started — the pipeline is fresh -- routing: the menu's forward action for the current state routes to discussion entry (workflow-discussion-entry with feature and pay) -- state: manifest absent pay.discussion.pay status - -## case: feature-discussion-entry-seeds -- world: feature-created -- origin: feature mainline — discussion entry seeds from the durable carrier -- files: - - skills/workflow-discussion-entry/SKILL.md - - skills/workflow-shared/references/ensure-discovery-item.md - -### walk - -Execute skills/workflow-discussion-entry/SKILL.md with arguments -$0=feature, $1=pay. Follow it up to the point where the handoff to the -processing skill is constructed — record the handoff arguments block — -and stop there. Do not execute the processing skill's instructions. - -### expect - -- routing: with no discussion item in the manifest, the new-entry arm is taken — no phase-status validation menu appears -- routing: ensure-discovery-item returns without creating anything — the discovery map is epic-only, and this is a feature -- routing: context is seeded from the manifest description and the session log's Exploration section — the gather-context questioning path is not taken -- routing: the handoff invokes workflow-discussion-process for pay -- state: manifest absent pay.discussion.pay status - -## case: feature-spec-entry-handoff -- world: feature-discussed -- origin: feature mainline — specification entry validates and hands off the discussion -- files: - - skills/workflow-specification-entry/SKILL.md - - skills/workflow-specification-entry/references/validate-source.md - - skills/workflow-specification-entry/references/validate-phase.md - - skills/workflow-specification-entry/references/invoke-skill.md - -### walk - -Execute skills/workflow-specification-entry/SKILL.md with arguments -$0=feature, $1=pay. Follow it up to the point where the handoff to the -processing skill is constructed — record the handoff arguments block — -and stop there. Do not execute the processing skill's instructions. - -### expect - -- routing: the source-material entry gate passes (empty render) — the discussion is complete -- routing: with no specification item in the manifest, the verb is Creating — no resume note, no reopen -- routing: the handoff names the discussion file .workflows/pay/discussion/pay.md as source material -- state: manifest absent pay.specification.pay status - -## case: feature-planning-entry-context-gate -- world: feature-specified -- origin: feature mainline — planning entry validates the spec and asks for late context -- files: - - skills/workflow-planning-entry/SKILL.md - - skills/workflow-planning-entry/references/validate-spec.md - - skills/workflow-planning-entry/references/validate-phase.md - -### walk - -Execute skills/workflow-planning-entry/SKILL.md with arguments -$0=feature, $1=pay. Follow it up to the point where the handoff to the -processing skill is constructed — record the handoff arguments block — -and stop there. Do not execute the processing skill's instructions. - -### user - -1. continue — no additional context since the specification - -### expect - -- routing: the specification entry gate passes (empty render) — clear to plan -- routing: with no plan in the manifest, the fresh-start arm asks whether any context has changed since the specification (answered continue) -- routing: the handoff invokes workflow-planning-process for pay -- state: manifest absent pay.planning.pay status - -## case: feature-implementation-first-task -- world: feature-planned -- origin: feature mainline — implementation entry through to the first task pickup -- files: - - skills/workflow-implementation-entry/SKILL.md - - skills/workflow-implementation-entry/references/validate-phase.md - - skills/workflow-implementation-entry/references/environment-check.md - - skills/workflow-implementation-process/SKILL.md - - skills/workflow-implementation-process/references/task-loop.md - - skills/workflow-planning-process/references/output-formats/local-markdown/reading.md - -### walk - -Execute skills/workflow-implementation-entry/SKILL.md with arguments -$0=feature, $1=pay, then continue into the processing skill it invokes. -Follow the task loop far enough to determine and start the next -available task. Stop the moment the prose directs you to begin actually -implementing that task (writing code or tests) — implement nothing. - -### user - -1. none — no special environment setup is needed - -### expect - -- routing: the plan entry gate passes (empty render) and the implementation is a new entry -- routing: with no environment-setup file recorded, the entry asks the environment question (answered none) -- routing: the processing skill's resume detection reports mode created — the fresh path, with its start-implementation commit — never the resuming-from-previous-session note -- routing: the next available task per the plan format's reading procedure is pay-1-1 — phase 1, first task, nothing completed yet -- state: manifest equals pay.implementation.pay status in-progress -- state: manifest equals pay.implementation.pay current_task pay-1-1 - -## case: feature-review-entry-fresh -- world: feature-implemented -- origin: feature mainline — review entry passes both prerequisites, fresh review -- files: - - skills/workflow-review-entry/SKILL.md - - skills/workflow-review-entry/references/validate-phase.md - -### walk - -Execute skills/workflow-review-entry/SKILL.md with arguments -$0=feature, $1=pay. Follow it up to the point where the handoff to the -processing skill is constructed — record the handoff arguments block — -and stop there. Do not execute the processing skill's instructions, and -never dispatch any agent. - -### expect - -- routing: the prerequisite entry gate passes (empty render) — plan and implementation are both complete -- routing: with no review item in the manifest, the fresh path is taken — no reopen, no resume -- routing: the handoff invokes workflow-review-process for pay -- state: manifest absent pay.review.pay status diff --git a/tests/prose/fixtures/base/recipe.cjs b/tests/prose/fixtures/base/recipe.cjs deleted file mode 100644 index 06ac01e21..000000000 --- a/tests/prose/fixtures/base/recipe.cjs +++ /dev/null @@ -1,13 +0,0 @@ -'use strict'; - -// The canonical empty project — a real install straight after -// /workflow-start Step 0: migrations applied (the log is the durable -// trace), knowledge set up keyword-only, boot run. No work units yet. -// Every richer fixture starts from this world's shape. - -module.exports = { - build(h) { - h.knowledge('setup', '--keyword-only'); - h.engine('boot'); - }, -}; diff --git a/tests/prose/fixtures/feature-created/recipe.cjs b/tests/prose/fixtures/feature-created/recipe.cjs deleted file mode 100644 index b6d1d32c3..000000000 --- a/tests/prose/fixtures/feature-created/recipe.cjs +++ /dev/null @@ -1,11 +0,0 @@ -'use strict'; - -// Feature mainline stop point — see _shared/feature-mainline.cjs. - -const m = require('../_shared/feature-mainline.cjs'); - -module.exports = { - build(h) { - m.init(h); m.create(h); - }, -}; diff --git a/tests/prose/fixtures/feature-discussed/recipe.cjs b/tests/prose/fixtures/feature-discussed/recipe.cjs deleted file mode 100644 index 0385917fa..000000000 --- a/tests/prose/fixtures/feature-discussed/recipe.cjs +++ /dev/null @@ -1,11 +0,0 @@ -'use strict'; - -// Feature mainline stop point — see _shared/feature-mainline.cjs. - -const m = require('../_shared/feature-mainline.cjs'); - -module.exports = { - build(h) { - m.init(h); m.create(h); m.discuss(h); - }, -}; diff --git a/tests/prose/fixtures/feature-implemented/recipe.cjs b/tests/prose/fixtures/feature-implemented/recipe.cjs deleted file mode 100644 index 9daef50c8..000000000 --- a/tests/prose/fixtures/feature-implemented/recipe.cjs +++ /dev/null @@ -1,11 +0,0 @@ -'use strict'; - -// Feature mainline stop point — see _shared/feature-mainline.cjs. - -const m = require('../_shared/feature-mainline.cjs'); - -module.exports = { - build(h) { - m.init(h); m.create(h); m.discuss(h); m.specify(h); m.plan(h); m.implement(h); - }, -}; diff --git a/tests/prose/fixtures/feature-planned/recipe.cjs b/tests/prose/fixtures/feature-planned/recipe.cjs deleted file mode 100644 index d2a02e75b..000000000 --- a/tests/prose/fixtures/feature-planned/recipe.cjs +++ /dev/null @@ -1,11 +0,0 @@ -'use strict'; - -// Feature mainline stop point — see _shared/feature-mainline.cjs. - -const m = require('../_shared/feature-mainline.cjs'); - -module.exports = { - build(h) { - m.init(h); m.create(h); m.discuss(h); m.specify(h); m.plan(h); - }, -}; diff --git a/tests/prose/fixtures/feature-specified/recipe.cjs b/tests/prose/fixtures/feature-specified/recipe.cjs deleted file mode 100644 index d046654fc..000000000 --- a/tests/prose/fixtures/feature-specified/recipe.cjs +++ /dev/null @@ -1,11 +0,0 @@ -'use strict'; - -// Feature mainline stop point — see _shared/feature-mainline.cjs. - -const m = require('../_shared/feature-mainline.cjs'); - -module.exports = { - build(h) { - m.init(h); m.create(h); m.discuss(h); m.specify(h); - }, -}; diff --git a/tests/prose/lib/cases.cjs b/tests/prose/lib/cases.cjs index 1ff5705fd..006534a5d 100644 --- a/tests/prose/lib/cases.cjs +++ b/tests/prose/lib/cases.cjs @@ -1,159 +1,146 @@ 'use strict'; -// Case corpus: parsing and validation. +// Case corpus: loading and validation. // -// A case file is markdown under tests/prose/{flow}/ (any name, .md). -// Grammar, deterministic to parse: +// A case is a directory, tests/prose/cases/{case-id}/, holding one file +// per element of the test. Nothing is parsed: JSON is JSON, prose files +// are read whole and relayed into prompts, recipes are required as +// modules. // -// ## case: {kebab-id} -// - world: {fixture-name} (optional; required by state expects) -// - origin: {free text} -// - files: -// - {repo-relative path}[#heading fragment] +// case.json the values code branches on: +// { origin, files[], answers[], stubs{name: trigger} } +// fixture.md optional. Prose describing the starting world — +// what has already happened, where the session +// stands. Given to the walker. +// fixture-state.cjs exports build(h): the starting world, in engine +// calls. Composed from tests/prose/mainlines/. +// act.md the coarse instruction: where to enter, what to +// follow, where to stop. Given to the walker. +// assert.md the expected trace, step by step, plus any +// further claims. Given ONLY to the asserter. +// assertion-state.cjs optional. exports build(h): the world the walk +// should produce. Absent means "unchanged" — the +// walk should leave the fixture state untouched. +// fixture/ generated snapshot of the starting world +// assertion/ generated snapshot of the expected world // -// Anchors are substring fragments, matched against heading text with -// `includes` — `#Boot` matches "Step 0.2: Boot" and survives a -// renumber. Authoring rule (see README): cases name BEHAVIOUR, never -// coordinates — step numbers, arm letters, and heading numbering in -// walk or expect text rot on cosmetic edits and fail for the wrong -// reason. -// ### walk -// {free text: entry point, what to do, stop condition} -// ### user -// 1. {scripted answer, consumed in order} -// ### expect -// - routing: {claim graded by an agent against the walk transcript} -// - state: {deterministic assertion — see parseStateAssertion} +// act.md and assert.md are separate files because that is the P4 +// boundary: the walker must never see the expected trace, and a file +// boundary enforces it structurally rather than by convention. // -// State assertion grammar (executed by the runner, zero model judgment): -// file exists {rel} -// file absent {rel} -// manifest exists {dotpath} {field} -// manifest absent {dotpath} {field} -// manifest equals {dotpath} {field} {expected printed value} +// The act stays coarse on purpose — the walker must DERIVE the path from +// the prose, and that derivation is the thing under test. Granularity +// lives in assert.md, where a step-by-step expected trace catches a +// walker that silently course-corrected around broken prose. const fs = require('fs'); const path = require('path'); const ROOT = path.join(__dirname, '../../..'); const PROSE_DIR = path.join(ROOT, 'tests/prose'); -const FIXTURES_DIR = path.join(PROSE_DIR, 'fixtures'); -const RESERVED_DIRS = new Set(['lib', 'fixtures']); - -function listCaseFiles() { - if (!fs.existsSync(PROSE_DIR)) return []; - const files = []; - for (const flow of fs.readdirSync(PROSE_DIR, { withFileTypes: true })) { - if (!flow.isDirectory() || RESERVED_DIRS.has(flow.name)) continue; - const flowDir = path.join(PROSE_DIR, flow.name); - for (const entry of fs.readdirSync(flowDir)) { - if (entry.endsWith('.md') && entry !== 'README.md') { - files.push({ flow: flow.name, file: path.join(flowDir, entry) }); - } - } - } - return files; +const CASES_DIR = path.join(PROSE_DIR, 'cases'); +const STUBS_DIR = path.join(PROSE_DIR, 'stubs'); + +const FILES = { + meta: 'case.json', + situation: 'fixture.md', + fixtureState: 'fixture-state.cjs', + act: 'act.md', + assert: 'assert.md', + assertionState: 'assertion-state.cjs', +}; +const SNAPSHOTS = { fixture: 'fixture', assertion: 'assertion' }; + +function listCaseIds() { + if (!fs.existsSync(CASES_DIR)) return []; + return fs.readdirSync(CASES_DIR, { withFileTypes: true }) + .filter((e) => e.isDirectory() && !e.name.startsWith('_')) + .map((e) => e.name) + .sort(); } -function parseStateAssertion(text) { - const fileMatch = text.match(/^file (exists|absent) (\S+)$/); - if (fileMatch) { - return { kind: `file-${fileMatch[1]}`, path: fileMatch[2] }; - } - const mExists = text.match(/^manifest (exists|absent) (\S+) (\S+)$/); - if (mExists) { - return { kind: `manifest-${mExists[1]}`, dotpath: mExists[2], field: mExists[3] }; - } - const mEquals = text.match(/^manifest equals (\S+) (\S+) (.+)$/); - if (mEquals) { - return { kind: 'manifest-equals', dotpath: mEquals[1], field: mEquals[2], value: mEquals[3] }; - } - return { error: `unparseable state assertion: "${text}"` }; +function readIf(dir, name) { + const file = path.join(dir, name); + return fs.existsSync(file) ? fs.readFileSync(file, 'utf8').trim() : null; } -function parseCaseFile(file, flow) { - const lines = fs.readFileSync(file, 'utf8').split('\n'); - const cases = []; - let current = null; - let section = null; // null | 'walk' | 'user' | 'expect' - let inFiles = false; - - const push = () => { - if (current) { - current.walk = current.walk.join('\n').trim(); - cases.push(current); +function loadCase(id) { + const dir = path.join(CASES_DIR, id); + if (!fs.existsSync(dir)) throw new Error(`no case "${id}" in tests/prose/cases`); + + const metaPath = path.join(dir, FILES.meta); + let meta = {}; + let metaError = null; + if (!fs.existsSync(metaPath)) metaError = `missing ${FILES.meta}`; + else { + try { + meta = JSON.parse(fs.readFileSync(metaPath, 'utf8')); + } catch (e) { + metaError = `${FILES.meta} does not parse: ${e.message}`; } - }; + } - for (const line of lines) { - const caseStart = line.match(/^## case:\s*(\S+)\s*$/); - if (caseStart) { - push(); - current = { - id: caseStart[1], flow, file: path.relative(ROOT, file), - world: null, origin: null, files: [], - walk: [], user: [], expect: [], - }; - section = null; - inFiles = false; - continue; - } - if (!current) continue; + return { + id, + dir, + rel: path.relative(ROOT, dir), + metaError, + origin: meta.origin || null, + files: (meta.files || []).map((spec) => { + const hash = spec.indexOf('#'); + return hash === -1 + ? { path: spec, anchor: null } + : { path: spec.slice(0, hash).trim(), anchor: spec.slice(hash + 1).trim() }; + }), + answers: meta.answers || [], + stubs: Object.entries(meta.stubs || {}).map(([name, trigger]) => ({ name, trigger })), + situation: readIf(dir, FILES.situation), + act: readIf(dir, FILES.act), + assert: readIf(dir, FILES.assert), + hasFixtureState: fs.existsSync(path.join(dir, FILES.fixtureState)), + hasAssertionState: fs.existsSync(path.join(dir, FILES.assertionState)), + }; +} - const sub = line.match(/^### (walk|user|expect)\s*$/); - if (sub) { - section = sub[1]; - inFiles = false; - continue; - } - if (/^#/.test(line)) { // any other heading ends the current case - push(); - current = null; - continue; - } +function loadAllCases() { + return listCaseIds().map(loadCase); +} - if (section === null) { - const field = line.match(/^- (world|origin):\s*(.*)$/); - if (field) { - current[field[1]] = field[2].trim(); - inFiles = false; - continue; - } - if (/^- files:\s*$/.test(line)) { - inFiles = true; - continue; - } - const fileEntry = line.match(/^\s+- (.+)$/); - if (inFiles && fileEntry) { - const [, spec] = fileEntry; - const hash = spec.indexOf('#'); - current.files.push(hash === -1 - ? { path: spec.trim(), anchor: null } - : { path: spec.slice(0, hash).trim(), anchor: spec.slice(hash + 1).trim() }); - continue; - } - if (line.trim() !== '') inFiles = false; - } else if (section === 'walk') { - current.walk.push(line); - } else if (section === 'user') { - const answer = line.match(/^\d+\.\s+(.*)$/); - if (answer) current.user.push(answer[1].trim()); - } else if (section === 'expect') { - const claim = line.match(/^- (routing|state):\s*(.*)$/); - if (claim) current.expect.push({ kind: claim[1], text: claim[2].trim() }); - } +/** A world builder module: exports build(h). */ +function requireState(caseId, which) { + const file = path.join(CASES_DIR, caseId, FILES[which]); + if (!fs.existsSync(file)) return null; + delete require.cache[require.resolve(file)]; + const mod = require(file); + if (typeof mod.build !== 'function') { + throw new Error(`${caseId}/${FILES[which]} must export build(h)`); } - push(); - return cases; + return mod; } -function loadAllCases() { - return listCaseFiles().flatMap(({ flow, file }) => parseCaseFile(file, flow)); +function listStubs() { + if (!fs.existsSync(STUBS_DIR)) return []; + return fs.readdirSync(STUBS_DIR) + .filter((f) => f.endsWith('.md') && f !== 'README.md') + .map((f) => path.basename(f, '.md')); +} + +/** A stub file is description above a `---` fence, exact bytes below. */ +function readStub(name) { + const file = path.join(STUBS_DIR, `${name}.md`); + if (!fs.existsSync(file)) throw new Error(`no stub "${name}" in tests/prose/stubs`); + const raw = fs.readFileSync(file, 'utf8'); + const fence = raw.indexOf('\n---\n'); + if (fence === -1) throw new Error(`stub "${name}" has no --- fence separating description from content`); + return { + name, + description: raw.slice(0, fence).replace(/^#.*\n/, '').trim(), + content: raw.slice(fence + 5).replace(/^\n+/, ''), + }; } function headingExists(absPath, anchor) { - const content = fs.readFileSync(absPath, 'utf8'); - return content.split('\n').some((line) => { + return fs.readFileSync(absPath, 'utf8').split('\n').some((line) => { const h = line.match(/^#{1,6}\s+(.*?)\s*$/); return h && h[1].includes(anchor); }); @@ -161,46 +148,52 @@ function headingExists(absPath, anchor) { function validateCorpus(cases) { const errors = []; - const seen = new Set(); + const stubs = new Set(listStubs()); + for (const c of cases) { - const at = `${c.file} :: ${c.id}`; - if (!/^[a-z0-9][a-z0-9-]*$/.test(c.id)) errors.push(`${at}: id is not kebab-case`); - if (seen.has(c.id)) errors.push(`${at}: duplicate case id`); - seen.add(c.id); + const at = c.rel; + if (c.metaError) { errors.push(`${at}: ${c.metaError}`); continue; } + if (!/^[a-z0-9][a-z0-9-]*$/.test(c.id)) errors.push(`${at}: case directory is not kebab-case`); + if (!c.origin) errors.push(`${at}: ${FILES.meta} has no origin`); - if (c.files.length === 0) errors.push(`${at}: no files: scope`); + if (c.files.length === 0) errors.push(`${at}: ${FILES.meta} scopes no files`); for (const f of c.files) { const abs = path.join(ROOT, f.path); - if (!fs.existsSync(abs)) { - errors.push(`${at}: scoped file missing: ${f.path}`); - } else if (f.anchor && !headingExists(abs, f.anchor)) { + if (!fs.existsSync(abs)) errors.push(`${at}: scoped file missing: ${f.path}`); + else if (f.anchor && !headingExists(abs, f.anchor)) { errors.push(`${at}: anchor "#${f.anchor}" not a heading in ${f.path}`); } } - if (c.world !== null) { - if (!fs.existsSync(path.join(FIXTURES_DIR, c.world, 'recipe.cjs'))) { - errors.push(`${at}: world "${c.world}" has no fixture recipe`); + if (!c.act) errors.push(`${at}: no ${FILES.act}`); + if (!c.assert) errors.push(`${at}: no ${FILES.assert} — the expected trace is what catches a silent repair`); + + if (c.hasAssertionState && !c.hasFixtureState) { + errors.push(`${at}: ${FILES.assertionState} without ${FILES.fixtureState} — nothing to act on`); + } + if (c.situation && !c.hasFixtureState) { + errors.push(`${at}: ${FILES.situation} describes a world this case does not build`); + } + for (const which of ['fixtureState', 'assertionState']) { + try { + requireState(c.id, which); + } catch (e) { + errors.push(`${at}: ${e.message}`); } } - if (!c.walk) errors.push(`${at}: empty walk`); - if (c.expect.length === 0) errors.push(`${at}: no expects`); - for (const e of c.expect) { - if (!e.text) errors.push(`${at}: empty ${e.kind} expect`); - if (e.kind === 'state') { - if (c.world === null) { - errors.push(`${at}: state expect requires a world`); - } - const parsed = parseStateAssertion(e.text); - if (parsed.error) errors.push(`${at}: ${parsed.error}`); + for (const s of c.stubs) { + if (!stubs.has(s.name)) errors.push(`${at}: no stub "${s.name}" in tests/prose/stubs`); + if (!s.trigger || !s.trigger.trim()) { + errors.push(`${at}: stub "${s.name}" declares no trigger — the case owns the moment`); } + if (!c.hasFixtureState) errors.push(`${at}: stub "${s.name}" needs a world to fire in`); } } return errors; } module.exports = { - ROOT, PROSE_DIR, FIXTURES_DIR, - listCaseFiles, parseCaseFile, loadAllCases, parseStateAssertion, validateCorpus, + ROOT, PROSE_DIR, CASES_DIR, STUBS_DIR, FILES, SNAPSHOTS, + listCaseIds, loadCase, loadAllCases, requireState, listStubs, readStub, validateCorpus, }; diff --git a/tests/prose/lib/fixtures.cjs b/tests/prose/lib/fixtures.cjs deleted file mode 100644 index 9c294cdc5..000000000 --- a/tests/prose/lib/fixtures.cjs +++ /dev/null @@ -1,196 +0,0 @@ -'use strict'; - -// Fixture recipes and golden snapshots. -// -// A fixture is tests/prose/fixtures/{name}/: a recipe.cjs (a module whose -// build(harness) drives real engine/knowledge calls against a scratch -// project) and a committed snapshot/ (the recipe's byte-exact output). -// The recipe is the source of truth; the snapshot is what makes drift -// visible — CI rebuilds every recipe and byte-compares (P2/P3 in -// design/prose-tests.md). Hand-editing a snapshot is forbidden: -// regenerate with `node tests/prose/run.cjs snap {name}`. -// -// Determinism: recipe subprocesses run under the frozen-clock preload and -// pinned git identity/dates. Snapshots exclude `.git/` (SHAs live in -// manifest values where recipes record them, not as a tree) and -// `.workflows/.knowledge/` (binary store; the world builder re-derives it -// at materialise time). `.gitignore` files inside a snapshot are stored -// escaped so the product-written `.workflows/.gitignore` cannot ignore -// fixture content out of THIS repo's git. - -const fs = require('fs'); -const os = require('os'); -const path = require('path'); -const { execFileSync, spawnSync } = require('child_process'); - -const ROOT = path.join(__dirname, '../../..'); -const ENGINE = path.join(ROOT, 'skills/workflow-engine/scripts/engine.cjs'); -const KNOWLEDGE = path.join(ROOT, 'skills/workflow-knowledge/scripts/knowledge.cjs'); -const FIXTURES_DIR = path.join(ROOT, 'tests/prose/fixtures'); -const CLOCK = path.join(__dirname, 'fake-clock.cjs'); - -const GITIGNORE = '.gitignore'; -const GITIGNORE_ESCAPED = '_gitignore.fixture'; - -function recipeEnv() { - if (/\s/.test(CLOCK)) { - throw new Error(`fake-clock path contains whitespace — NODE_OPTIONS cannot carry it: ${CLOCK}`); - } - return { - ...process.env, - NODE_OPTIONS: [process.env.NODE_OPTIONS, `--require ${CLOCK}`].filter(Boolean).join(' '), - GIT_CONFIG_GLOBAL: '/dev/null', - GIT_CONFIG_SYSTEM: '/dev/null', - GIT_AUTHOR_DATE: '2026-01-01T00:00:00Z', - GIT_COMMITTER_DATE: '2026-01-01T00:00:00Z', - TZ: 'UTC', - }; -} - -function makeHarness(dir) { - const env = recipeEnv(); - const node = (script, args) => { - const res = spawnSync('node', [script, ...args], { cwd: dir, encoding: 'utf8', env }); - if (res.status !== 0) { - throw new Error(`recipe call failed: ${path.basename(script)} ${args.join(' ')}\n` + - `stdout: ${res.stdout}\nstderr: ${res.stderr}`); - } - return res.stdout; - }; - return { - dir, - engine: (...args) => node(ENGINE, args), - knowledge: (...args) => node(KNOWLEDGE, args), - git: (...args) => execFileSync('git', args, { cwd: dir, encoding: 'utf8', env }), - write(rel, content) { - const full = path.join(dir, rel); - fs.mkdirSync(path.dirname(full), { recursive: true }); - fs.writeFileSync(full, content); - }, - remove(rel) { - fs.rmSync(path.join(dir, rel), { recursive: true, force: true }); - }, - }; -} - -function listFixtures() { - if (!fs.existsSync(FIXTURES_DIR)) return []; - return fs.readdirSync(FIXTURES_DIR, { withFileTypes: true }) - .filter((e) => e.isDirectory() && fs.existsSync(path.join(FIXTURES_DIR, e.name, 'recipe.cjs'))) - .map((e) => e.name); -} - -/** Build a fixture's world into a fresh scratch dir; caller removes it. */ -function runRecipe(name) { - const recipePath = path.join(FIXTURES_DIR, name, 'recipe.cjs'); - if (!fs.existsSync(recipePath)) throw new Error(`no recipe for fixture "${name}"`); - const dir = fs.mkdtempSync(path.join(os.tmpdir(), `prose-recipe-${name}-`)); - const h = makeHarness(dir); - h.git('init', '-q', '-b', 'main'); - h.git('config', 'user.email', 'prose@example.com'); - h.git('config', 'user.name', 'Prose Fixture'); - h.git('config', 'commit.gpgsign', 'false'); - fs.mkdirSync(path.join(dir, '.workflows'), { recursive: true }); - delete require.cache[require.resolve(recipePath)]; - require(recipePath).build(h); - return dir; -} - -function excluded(rel) { - const parts = rel.split(path.sep); - if (parts.includes('.git')) return true; - if (rel === path.join('.workflows', '.knowledge')) return true; - if (rel.startsWith(path.join('.workflows', '.knowledge') + path.sep)) return true; - return false; -} - -/** Scratch tree → Map(snapshot-relative path → Buffer), escaped + filtered. */ -function collectTree(root) { - const files = new Map(); - const walk = (rel) => { - const abs = path.join(root, rel); - for (const entry of fs.readdirSync(abs, { withFileTypes: true })) { - const childRel = rel === '' ? entry.name : path.join(rel, entry.name); - if (excluded(childRel)) continue; - if (entry.isDirectory()) { - walk(childRel); - } else if (entry.isFile()) { - const stored = entry.name === GITIGNORE - ? path.join(path.dirname(childRel), GITIGNORE_ESCAPED) - : childRel; - files.set(stored, fs.readFileSync(path.join(root, childRel))); - } - } - }; - walk(''); - return files; -} - -/** Committed snapshot dir → Map(snapshot-relative path → Buffer), verbatim. */ -function readSnapshot(name) { - const snapDir = path.join(FIXTURES_DIR, name, 'snapshot'); - if (!fs.existsSync(snapDir)) return null; - const files = new Map(); - const walk = (rel) => { - for (const entry of fs.readdirSync(path.join(snapDir, rel), { withFileTypes: true })) { - const childRel = rel === '' ? entry.name : path.join(rel, entry.name); - if (entry.isDirectory()) walk(childRel); - else if (entry.isFile()) files.set(childRel, fs.readFileSync(path.join(snapDir, childRel))); - } - }; - walk(''); - return files; -} - -/** Regenerate the committed snapshot from a built scratch tree. */ -function writeSnapshot(name, scratchDir) { - const snapDir = path.join(FIXTURES_DIR, name, 'snapshot'); - fs.rmSync(snapDir, { recursive: true, force: true }); - const files = collectTree(scratchDir); - for (const [rel, buf] of files) { - const dest = path.join(snapDir, rel); - fs.mkdirSync(path.dirname(dest), { recursive: true }); - fs.writeFileSync(dest, buf); - } - return files.size; -} - -/** Rebuild-compare: {missing, extra, changed} — all empty means current. */ -function compareSnapshot(name, scratchDir) { - const snap = readSnapshot(name); - if (snap === null) return { missing: [''], extra: [], changed: [] }; - const built = collectTree(scratchDir); - const missing = []; - const extra = []; - const changed = []; - for (const [rel, buf] of built) { - if (!snap.has(rel)) extra.push(rel); - else if (!snap.get(rel).equals(buf)) changed.push(rel); - } - for (const rel of snap.keys()) { - if (!built.has(rel)) missing.push(rel); - } - return { missing, extra, changed }; -} - -/** Snapshot → real project tree at dest, unescaping .gitignore files. */ -function materialiseSnapshot(name, destDir) { - const snap = readSnapshot(name); - if (snap === null) { - throw new Error(`fixture "${name}" has no committed snapshot — run: node tests/prose/run.cjs snap ${name}`); - } - for (const [rel, buf] of snap) { - const real = path.basename(rel) === GITIGNORE_ESCAPED - ? path.join(path.dirname(rel), GITIGNORE) - : rel; - const dest = path.join(destDir, real); - fs.mkdirSync(path.dirname(dest), { recursive: true }); - fs.writeFileSync(dest, buf); - } -} - -module.exports = { - ROOT, ENGINE, KNOWLEDGE, FIXTURES_DIR, - listFixtures, runRecipe, collectTree, readSnapshot, - writeSnapshot, compareSnapshot, materialiseSnapshot, -}; diff --git a/tests/prose/lib/prompts.cjs b/tests/prose/lib/prompts.cjs new file mode 100644 index 000000000..1651e7a8d --- /dev/null +++ b/tests/prose/lib/prompts.cjs @@ -0,0 +1,86 @@ +'use strict'; + +// Prompt assembly: every word an agent reads comes from prose, never +// from a string in code. +// +// Standing instructions — how a walker or asserter behaves — live in +// .claude/agents/prose-*.md and are loaded by the harness when the agent +// is dispatched. What varies per case lives in tests/prose/prompts/*.md, +// assembled here. +// +// A template file is sections delimited by `=== name ===` lines, with +// `{{placeholder}}` slots. This module only splits and substitutes — it +// holds no wording of its own, so changing what an agent is told is a +// prose edit, reviewable as a diff, and identical on every run. +// +// An unfilled placeholder is a hard error rather than something an agent +// silently receives. + +const fs = require('fs'); +const path = require('path'); + +const PROMPTS_DIR = path.join(__dirname, '../prompts'); + +function loadTemplate(name) { + const raw = fs.readFileSync(path.join(PROMPTS_DIR, `${name}.md`), 'utf8'); + const sections = {}; + let current = null; + for (const line of raw.split('\n')) { + const header = line.match(/^=== ([a-z-]+) ===$/); + if (header) { + current = header[1]; + sections[current] = []; + continue; + } + if (current) sections[current].push(line); + } + for (const key of Object.keys(sections)) { + sections[key] = sections[key].join('\n').replace(/^\n+|\n+$/g, ''); + } + return sections; +} + +function fill(text, values) { + const filled = text.replace(/\{\{([a-z_]+)\}\}/g, (match, key) => { + if (!(key in values)) return match; + return values[key]; + }); + const leftover = filled.match(/\{\{[a-z_]+\}\}/); + if (leftover) throw new Error(`prompt placeholder ${leftover[0]} was never filled`); + return filled; +} + +/** Indent a block so it reads as a nested unit inside the prompt. */ +function indent(text, by = ' ') { + return text.split('\n').map((l) => (l.length ? by + l : l)).join('\n'); +} + +function walkerPrompt({ worldDir, root, situation, task, scope, stubs, answers }) { + const t = loadTemplate('walker'); + const parts = [ + worldDir ? fill(t.world, { world_dir: worldDir }) : fill(t.structural, { root }), + ]; + if (situation) parts.push(fill(t.situation, { situation })); + parts.push(fill(t.task, { task, scope })); + if (answers) parts.push(fill(t.answers, { answers })); + if (stubs.length) { + const entries = stubs.map((s) => fill(t['stub-entry'], { + name: s.name, + trigger: s.trigger, + description: s.description, + content: indent(s.content), + })).join('\n'); + parts.push(fill(t.stubs, { entries })); + } + return `${parts.join('\n\n')}\n`; +} + +function asserterPrompt({ expected, world }) { + const t = loadTemplate('asserter'); + const parts = [fill(t.main, { expected })]; + if (world) parts.push(fill(t.world, { expecting: world.expecting, delta: world.delta })); + parts.push(t.transcript); + return `${parts.join('\n\n')}\n`; +} + +module.exports = { PROMPTS_DIR, loadTemplate, fill, walkerPrompt, asserterPrompt }; diff --git a/tests/prose/lib/world.cjs b/tests/prose/lib/world.cjs deleted file mode 100644 index f3dc2e7b6..000000000 --- a/tests/prose/lib/world.cjs +++ /dev/null @@ -1,75 +0,0 @@ -'use strict'; - -// World builder — materialise a fixture into a live, parallel-safe fake -// project a walker agent can work in as if it were a real install: -// -// 1. copy the fixture snapshot in (unescaping .gitignore files), -// 2. copy the repo's current skills/ and agents/ to the installed -// layout (.claude/skills, .claude/agents) so prose paths resolve, -// 3. git init + commit (hermetic identity — no user config leaks), -// 4. `knowledge setup --keyword-only` — re-derives the keyword index -// from the snapshot's artifacts (the binary store is never -// snapshotted), leaving every entry skill's knowledge gate green. -// -// Worlds run on the REAL clock: walker runs are live sessions, never -// byte-compared. Mutations during a walk land here and nowhere else. - -const fs = require('fs'); -const os = require('os'); -const path = require('path'); -const { execFileSync, spawnSync } = require('child_process'); - -const fixtures = require('./fixtures.cjs'); - -const WORLD_PREFIX = 'prose-world-'; - -function worldEnv() { - return { - ...process.env, - GIT_CONFIG_GLOBAL: '/dev/null', - GIT_CONFIG_SYSTEM: '/dev/null', - }; -} - -function buildWorld(fixtureName) { - const dir = fs.mkdtempSync(path.join(os.tmpdir(), WORLD_PREFIX)); - const env = worldEnv(); - const git = (...args) => execFileSync('git', args, { cwd: dir, encoding: 'utf8', env }); - - fixtures.materialiseSnapshot(fixtureName, dir); - - for (const layer of ['skills', 'agents']) { - const src = path.join(fixtures.ROOT, layer); - if (fs.existsSync(src)) { - fs.cpSync(src, path.join(dir, '.claude', layer), { recursive: true }); - } - } - - git('init', '-q', '-b', 'main'); - git('config', 'user.email', 'prose@example.com'); - git('config', 'user.name', 'Prose World'); - git('config', 'commit.gpgsign', 'false'); - git('add', '-A'); - git('commit', '-q', '-m', `world: ${fixtureName}`); - - const knowledge = path.join(dir, '.claude/skills/workflow-knowledge/scripts/knowledge.cjs'); - const setup = spawnSync('node', [knowledge, 'setup', '--keyword-only'], - { cwd: dir, encoding: 'utf8', env }); - if (setup.status !== 0) { - fs.rmSync(dir, { recursive: true, force: true }); - throw new Error(`knowledge setup failed in world:\nstdout: ${setup.stdout}\nstderr: ${setup.stderr}`); - } - git('add', '-A'); - git('commit', '-q', '-m', 'chore(knowledge): initialise store'); - - return dir; -} - -function destroyWorld(dir) { - if (!path.basename(dir).startsWith(WORLD_PREFIX)) { - throw new Error(`refusing to remove non-world directory: ${dir}`); - } - fs.rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); -} - -module.exports = { buildWorld, destroyWorld, WORLD_PREFIX }; diff --git a/tests/prose/lib/worlds.cjs b/tests/prose/lib/worlds.cjs new file mode 100644 index 000000000..2c4697889 --- /dev/null +++ b/tests/prose/lib/worlds.cjs @@ -0,0 +1,326 @@ +'use strict'; + +// World building: run a case's state recipe, snapshot it, compare it. +// +// A recipe (fixture-state.cjs / assertion-state.cjs) drives real engine +// calls against a scratch project under a frozen clock; its committed +// output lives beside it as fixture/ or assertion/. The recipe is the +// source of truth, the snapshot is the golden that makes drift visible: +// `npm test` rebuilds and byte-compares, so an engine change that moves +// a world goes red at the gate and lands as a reviewable diff. +// +// Rebuilds are skipped when nothing that feeds them has changed — the +// hash of the case's recipes, the shared mainlines, and the engine and +// knowledge sources is stored inside each snapshot. An engine change +// invalidates every hash, so drift can never hide behind the skip. +// +// Snapshots exclude `.git/` (SHAs), `.workflows/.knowledge/` (binary +// store, re-derived at materialise) and `.claude/skills|agents/` (copied +// into live worlds, never part of a world's own state), and store +// `.gitignore` files escaped so the product-written `.workflows/.gitignore` +// cannot ignore snapshot content out of this repo. + +const crypto = require('crypto'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { execFileSync, spawnSync } = require('child_process'); + +const cases = require('./cases.cjs'); + +const ROOT = cases.ROOT; +const ENGINE = path.join(ROOT, 'skills/workflow-engine/scripts/engine.cjs'); +const KNOWLEDGE = path.join(ROOT, 'skills/workflow-knowledge/scripts/knowledge.cjs'); +const MAINLINES_DIR = path.join(cases.PROSE_DIR, 'mainlines'); +const CLOCK = path.join(__dirname, 'fake-clock.cjs'); + +const GITIGNORE = '.gitignore'; +const GITIGNORE_ESCAPED = '_gitignore.fixture'; +const HASH_FILE = '.recipe-hash'; + +// --- the recipe harness --------------------------------------------------- + +function recipeEnv() { + if (/\s/.test(CLOCK)) { + throw new Error(`fake-clock path contains whitespace — NODE_OPTIONS cannot carry it: ${CLOCK}`); + } + return { + ...process.env, + NODE_OPTIONS: [process.env.NODE_OPTIONS, `--require ${CLOCK}`].filter(Boolean).join(' '), + GIT_CONFIG_GLOBAL: '/dev/null', + GIT_CONFIG_SYSTEM: '/dev/null', + GIT_AUTHOR_DATE: '2026-01-01T00:00:00Z', + GIT_COMMITTER_DATE: '2026-01-01T00:00:00Z', + TZ: 'UTC', + }; +} + +function makeHarness(dir) { + const env = recipeEnv(); + const node = (script, args) => { + const res = spawnSync('node', [script, ...args], { cwd: dir, encoding: 'utf8', env }); + if (res.status !== 0) { + throw new Error(`recipe call failed: ${path.basename(script)} ${args.join(' ')}\n` + + `stdout: ${res.stdout}\nstderr: ${res.stderr}`); + } + return res.stdout; + }; + return { + dir, + engine: (...args) => node(ENGINE, args), + knowledge: (...args) => node(KNOWLEDGE, args), + git: (...args) => execFileSync('git', args, { cwd: dir, encoding: 'utf8', env }), + write(rel, content) { + const full = path.join(dir, rel); + fs.mkdirSync(path.dirname(full), { recursive: true }); + fs.writeFileSync(full, content); + }, + remove(rel) { + fs.rmSync(path.join(dir, rel), { recursive: true, force: true }); + }, + }; +} + +/** Build a case's world into a fresh scratch dir; the caller removes it. */ +function runRecipe(caseId, which) { + const state = cases.requireState(caseId, which); + if (!state) throw new Error(`case "${caseId}" has no ${cases.FILES[which]}`); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), `prose-recipe-${caseId}-`)); + const h = makeHarness(dir); + h.git('init', '-q', '-b', 'main'); + h.git('config', 'user.email', 'prose@example.com'); + h.git('config', 'user.name', 'Prose Fixture'); + h.git('config', 'commit.gpgsign', 'false'); + fs.mkdirSync(path.join(dir, '.workflows'), { recursive: true }); + state.build(h); + return dir; +} + +// --- trees ---------------------------------------------------------------- + +function excluded(rel) { + const parts = rel.split(path.sep); + if (parts.includes('.git')) return true; + if (rel === HASH_FILE) return true; + if (rel === path.join('.workflows', '.knowledge')) return true; + if (rel.startsWith(path.join('.workflows', '.knowledge') + path.sep)) return true; + if (rel.startsWith(path.join('.claude', 'skills') + path.sep)) return true; + if (rel.startsWith(path.join('.claude', 'agents') + path.sep)) return true; + return false; +} + +/** Directory → Map(snapshot-relative path → Buffer), escaped + filtered. */ +function collectTree(root) { + const files = new Map(); + const walk = (rel) => { + for (const entry of fs.readdirSync(path.join(root, rel), { withFileTypes: true })) { + const childRel = rel === '' ? entry.name : path.join(rel, entry.name); + if (excluded(childRel)) continue; + if (entry.isDirectory()) walk(childRel); + else if (entry.isFile()) { + const stored = entry.name === GITIGNORE + ? path.join(path.dirname(childRel), GITIGNORE_ESCAPED) + : childRel; + files.set(stored, fs.readFileSync(path.join(root, childRel))); + } + } + }; + walk(''); + return files; +} + +function snapshotDir(caseId, which) { + return path.join(cases.CASES_DIR, caseId, cases.SNAPSHOTS[which]); +} + +function readSnapshot(caseId, which) { + const dir = snapshotDir(caseId, which); + return fs.existsSync(dir) ? collectTree(dir) : null; +} + +// --- recipe hashing ------------------------------------------------------- + +function hashDir(hash, dir) { + if (!fs.existsSync(dir)) return; + for (const entry of fs.readdirSync(dir, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name))) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) hashDir(hash, full); + else if (entry.isFile()) hash.update(entry.name).update(fs.readFileSync(full)); + } +} + +/** Everything a world's content can depend on: its recipes, the shared + * mainlines, and the engine and knowledge sources that execute them. */ +function recipeHash(caseId) { + const hash = crypto.createHash('sha256'); + for (const which of ['fixtureState', 'assertionState']) { + const file = path.join(cases.CASES_DIR, caseId, cases.FILES[which]); + if (fs.existsSync(file)) hash.update(fs.readFileSync(file)); + } + hashDir(hash, MAINLINES_DIR); + hashDir(hash, path.join(ROOT, 'skills/workflow-engine/scripts')); + hash.update(fs.readFileSync(KNOWLEDGE)); + return hash.digest('hex'); +} + +function storedHash(caseId, which) { + const file = path.join(snapshotDir(caseId, which), HASH_FILE); + return fs.existsSync(file) ? fs.readFileSync(file, 'utf8').trim() : null; +} + +// --- snapshot write / verify ---------------------------------------------- + +// Snapshot name ('fixture'|'assertion') → the module that builds it. +const STATE_OF = { fixture: 'fixtureState', assertion: 'assertionState' }; + +function writeSnapshot(caseId, which) { + const scratch = runRecipe(caseId, STATE_OF[which]); + try { + const dir = snapshotDir(caseId, which); + fs.rmSync(dir, { recursive: true, force: true }); + const files = collectTree(scratch); + for (const [rel, buf] of files) { + const dest = path.join(dir, rel); + fs.mkdirSync(path.dirname(dest), { recursive: true }); + fs.writeFileSync(dest, buf); + } + fs.writeFileSync(path.join(dir, HASH_FILE), `${recipeHash(caseId)}\n`); + return files.size; + } finally { + fs.rmSync(scratch, { recursive: true, force: true }); + } +} + +/** + * Rebuild and byte-compare, unless nothing feeding this world has moved. + * Returns {skipped} or {missing, extra, changed}. + */ +function verifySnapshot(caseId, which) { + const snap = readSnapshot(caseId, which); + if (snap === null) return { missing: [''], extra: [], changed: [] }; + if (storedHash(caseId, which) === recipeHash(caseId)) return { skipped: true }; + + const scratch = runRecipe(caseId, STATE_OF[which]); + try { + const built = collectTree(scratch); + const missing = []; + const extra = []; + const changed = []; + for (const [rel, buf] of built) { + if (!snap.has(rel)) extra.push(rel); + else if (!snap.get(rel).equals(buf)) changed.push(rel); + } + for (const rel of snap.keys()) if (!built.has(rel)) missing.push(rel); + return { missing, extra, changed }; + } finally { + fs.rmSync(scratch, { recursive: true, force: true }); + } +} + +// --- diffing a live world ------------------------------------------------- + +/** Unified diff of two buffers, via git — no diff algorithm of our own. */ +function unifiedDiff(label, expectedBuf, actualBuf) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'prose-diff-')); + try { + const a = path.join(dir, 'expected'); + const b = path.join(dir, 'actual'); + fs.writeFileSync(a, expectedBuf); + fs.writeFileSync(b, actualBuf); + const res = spawnSync('git', ['diff', '--no-index', '--unified=3', '--no-color', a, b], { + encoding: 'utf8', + env: { ...process.env, GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_SYSTEM: '/dev/null' }, + }); + return `--- ${label}\n${(res.stdout || '').split('\n').slice(4).join('\n').trimEnd()}`; + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +/** + * The factual delta between the world after a walk and the world the case + * expects — for the asserting agent to classify. Nothing is normalised: + * volatile values surface as ordinary differences and the agent rules on + * them. A case with no assertion-state expects its fixture back unchanged. + */ +function diffWorld(caseId, worldDir) { + const which = fs.existsSync(snapshotDir(caseId, 'assertion')) ? 'assertion' : 'fixture'; + const expected = readSnapshot(caseId, which); + if (expected === null) throw new Error(`case "${caseId}" has no committed ${which} snapshot`); + const actual = collectTree(worldDir); + + const added = []; + const removed = []; + const changed = []; + for (const [rel, buf] of actual) { + if (!expected.has(rel)) added.push(rel); + else if (!expected.get(rel).equals(buf)) changed.push(unifiedDiff(rel, expected.get(rel), buf)); + } + for (const rel of expected.keys()) if (!actual.has(rel)) removed.push(rel); + + return { + expecting: which === 'assertion' ? 'the assertion state' : 'the fixture state, unchanged', + added, + removed, + changed, + identical: !added.length && !removed.length && !changed.length, + }; +} + +// --- materialising a live world ------------------------------------------- + +const WORLD_PREFIX = 'prose-world-'; + +function buildWorld(caseId) { + const snap = readSnapshot(caseId, 'fixture'); + if (snap === null) { + throw new Error(`case "${caseId}" has no committed fixture — run: node tests/prose/run.cjs snap ${caseId}`); + } + const dir = fs.mkdtempSync(path.join(os.tmpdir(), WORLD_PREFIX)); + const env = { ...process.env, GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_SYSTEM: '/dev/null' }; + const git = (...args) => execFileSync('git', args, { cwd: dir, encoding: 'utf8', env }); + + for (const [rel, buf] of snap) { + const real = path.basename(rel) === GITIGNORE_ESCAPED + ? path.join(path.dirname(rel), GITIGNORE) + : rel; + const dest = path.join(dir, real); + fs.mkdirSync(path.dirname(dest), { recursive: true }); + fs.writeFileSync(dest, buf); + } + for (const layer of ['skills', 'agents']) { + const src = path.join(ROOT, layer); + if (fs.existsSync(src)) fs.cpSync(src, path.join(dir, '.claude', layer), { recursive: true }); + } + + git('init', '-q', '-b', 'main'); + git('config', 'user.email', 'prose@example.com'); + git('config', 'user.name', 'Prose World'); + git('config', 'commit.gpgsign', 'false'); + git('add', '-A'); + git('commit', '-q', '-m', `world: ${caseId}`); + + const knowledge = path.join(dir, '.claude/skills/workflow-knowledge/scripts/knowledge.cjs'); + const setup = spawnSync('node', [knowledge, 'setup', '--keyword-only'], { cwd: dir, encoding: 'utf8', env }); + if (setup.status !== 0) { + fs.rmSync(dir, { recursive: true, force: true }); + throw new Error(`knowledge setup failed in world:\nstdout: ${setup.stdout}\nstderr: ${setup.stderr}`); + } + git('add', '-A'); + git('commit', '-q', '-m', 'chore(knowledge): initialise store'); + + return dir; +} + +function destroyWorld(dir) { + if (!path.basename(dir).startsWith(WORLD_PREFIX)) { + throw new Error(`refusing to remove non-world directory: ${dir}`); + } + fs.rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); +} + +module.exports = { + ROOT, ENGINE, KNOWLEDGE, MAINLINES_DIR, WORLD_PREFIX, + runRecipe, collectTree, readSnapshot, snapshotDir, recipeHash, storedHash, + writeSnapshot, verifySnapshot, diffWorld, buildWorld, destroyWorld, +}; diff --git a/tests/prose/mainlines/bugfix.cjs b/tests/prose/mainlines/bugfix.cjs new file mode 100644 index 000000000..2ad47219f --- /dev/null +++ b/tests/prose/mainlines/bugfix.cjs @@ -0,0 +1,200 @@ +'use strict'; + +// The canonical bugfix mainline — `crash-fix`, a checkout crash — as +// staged builders each fixture recipe composes a prefix of. Engine calls +// follow the sequences the skill prose prescribes; content files are the +// artifacts a real run leaves, in the shape downstream prose reads +// (investigation file per workflow-investigation-process/references/ +// template.md). +// +// Dates are literals matching the frozen recipe clock (2026-01-01). + +const WU = 'crash-fix'; + +function init(h) { + h.knowledge('setup', '--keyword-only'); + h.engine('boot'); +} + +function create(h) { + // Shape per workflow-discovery/references/template.md: bugfix discovery + // captures brief intent; Exploration is backfilled at creation. + const log = `.workflows/${WU}/discovery/sessions/session-001.md`; + h.write(log, [ + '# Discovery Session 001', + '', + 'Date: 2026-01-01', + `Work unit: ${WU}`, + '', + '## Description (as of session)', + '', + 'Checkout crashes when an order has no shipping address.', + '', + '## Seed', + '', + '(none)', + '', + '## Imports', + '', + '(none)', + '', + '## Map State at Start', + '', + '(n/a — single-topic work)', + '', + '## Exploration', + '', + 'Reported by two users this week: the checkout page 500s at the', + 'payment step for orders with no shipping address. Reproducible on', + 'staging with a digital-only basket. No error surfaces to the user —', + 'the page just fails. Confirmed as a bugfix; no design question here,', + 'so it routes straight to investigation.', + '', + '## Edits', + '', + '(none)', + '', + '## Topics Identified', + '', + '(none)', + '', + '## Conclusion', + '', + '(none)', + '', + ].join('\n')); + h.engine('workunit', 'create', WU, 'bugfix', + '--description', 'Checkout crashes when an order has no shipping address', + '--session-log-file', log); +} + +/** The investigation file as it stands with the root cause documented. */ +function investigationFile() { + return [ + '# Investigation: Checkout Crash On Missing Shipping Address', + '', + '## Symptoms', + '', + '### Problem Description', + '', + 'Checkout returns a 500 at the payment step when the order has no', + 'shipping address. No error is surfaced to the user.', + '', + '### Manifestation', + '', + 'Server error on POST to the payment step; the page fails blank.', + '', + '### Reproduction Steps', + '', + '1. Add a digital-only product to the basket.', + '2. Proceed to checkout without entering a shipping address.', + '3. Continue to the payment step — the request 500s.', + '', + '### Environment', + '', + 'Reproduced on staging; reported twice in production this week.', + '', + '### Impact', + '', + 'Digital-only orders cannot complete checkout.', + '', + '### References', + '', + '(none)', + '', + '## Analysis', + '', + '### Hypotheses', + '', + '- The payment step assumes a shipping address is always present.', + '- Tax calculation may be the first consumer of the missing address.', + '', + '### Code Trace', + '', + 'The payment step builds a tax context from the order before creating', + 'the payment intent. The tax context reads the shipping address', + 'unconditionally; for a digital-only order the address is absent, so', + 'the read fails and the request aborts before any handler catches it.', + '', + '### Root Cause', + '', + 'The tax context treats the shipping address as mandatory. Digital-only', + 'orders legitimately have none, so the assumption is wrong rather than', + 'the data being wrong.', + '', + '### Contributing Factors', + '', + 'Digital-only baskets were introduced after the tax context was written.', + '', + "### Why It Wasn't Caught", + '', + 'No test covers a basket with no shippable line items.', + '', + '### Blast Radius', + '', + 'Any flow building a tax context from an address-less order.', + '', + '## Fix Direction', + '', + '### Chosen Approach', + '', + '(pending)', + '', + '### Options Explored', + '', + '(none yet)', + '', + '### Discussion', + '', + '(none yet)', + '', + '### Testing Recommendations', + '', + '(pending)', + '', + '### Risk Assessment', + '', + '(pending)', + '', + '## Notes', + '', + '(none)', + '', + ].join('\n'); +} + +/** + * Investigation open with the root cause documented — the state the + * process reaches just before its root-cause validation step. + */ +function investigateToRootCause(h) { + h.write(`.workflows/${WU}/investigation/${WU}.md`, investigationFile()); + h.engine('topic', 'start', WU, 'investigation', WU); + h.engine('commit', WU, '-m', `investigation(${WU}): symptoms and root cause`); +} + +/** Fix direction settled and the phase concluded. */ +function concludeInvestigation(h) { + h.write(`.workflows/${WU}/investigation/${WU}.md`, investigationFile() + .replace('### Chosen Approach\n\n(pending)', + ['### Chosen Approach', + '', + 'Make the tax context treat the shipping address as optional: when', + 'an order has no shippable items, build the context from the billing', + 'address instead.', + ].join('\n')) + .replace('### Testing Recommendations\n\n(pending)', + ['### Testing Recommendations', + '', + 'Cover a digital-only basket end to end through the payment step.', + ].join('\n')) + .replace('### Risk Assessment\n\n(pending)', + ['### Risk Assessment', + '', + 'Low — the change is confined to tax-context construction.', + ].join('\n'))); + h.engine('commit', WU, '-m', `investigation(${WU}): fix direction`); + h.engine('topic', 'complete', WU, 'investigation', WU); +} + +module.exports = { WU, init, create, investigateToRootCause, concludeInvestigation }; diff --git a/tests/prose/fixtures/_shared/feature-mainline.cjs b/tests/prose/mainlines/feature.cjs similarity index 100% rename from tests/prose/fixtures/_shared/feature-mainline.cjs rename to tests/prose/mainlines/feature.cjs diff --git a/tests/prose/prompts/asserter.md b/tests/prose/prompts/asserter.md new file mode 100644 index 000000000..da1a122ed --- /dev/null +++ b/tests/prose/prompts/asserter.md @@ -0,0 +1,26 @@ +The asserter's per-case payload. Sections are delimited by `=== name ===` +lines and assembled by lib/prompts.cjs; `{{placeholders}}` are filled +from the case and the computed world delta. + +Only what varies per case lives here. How an asserter behaves — quote or +it didn't happen, never explain a failure, volatile versus material, the +verdict format — is standing instruction and lives in +`.claude/agents/prose-asserter.md`. + +=== main === +WHAT THE PROSE SHOULD HAVE DONE: + +{{expected}} + +=== world === + +EXPECTED WORLD: {{expecting}}. + +WORLD DELTA — the factual difference between the world after the walk and +the expected world, computed by code: + +{{delta}} + +=== transcript === + +The walk's transcript follows under `=== TRANSCRIPT ===`. diff --git a/tests/prose/prompts/walker.md b/tests/prose/prompts/walker.md new file mode 100644 index 000000000..693c2d4ac --- /dev/null +++ b/tests/prose/prompts/walker.md @@ -0,0 +1,56 @@ +The walker's per-case payload. Sections are delimited by `=== name ===` +lines and assembled by lib/prompts.cjs; `{{placeholders}}` are filled +from the case. + +Only what varies per case lives here. How a walker behaves — follow +literally, never repair, never investigate, the markers, the transcript +format — is standing instruction and lives in +`.claude/agents/prose-walker.md`. + +Nothing from `assert.md` may ever appear in this file or in anything +assembled from it. That is the boundary the design rests on. + +=== world === +Project directory — your cwd for EVERY command: {{world_dir}} +The workflow skills are installed at .claude/skills/ inside that project. +Mutations are expected and safe: the project is a disposable test world. + +=== structural === +Structure-only walk: read the named prose and trace the logic. Execute +nothing — no commands, no writes. + +Repository root: {{root}} + +=== situation === + +SITUATION — where the project stands as you begin: +{{situation}} + +=== task === + +TASK +{{task}} + +SCOPE — the prose under walk: +{{scope}} + +=== answers === + +SCRIPTED USER ANSWERS — consume in order, one per question the prose asks: +{{answers}} + +=== stubs === + +HARNESS SUBSTITUTIONS — these are NOT part of the process you are walking. +They stand in for steps this framework deliberately does not simulate. +Apply each only at the moment stated, then resume the prose exactly where +you left it. +{{entries}} + +=== stub-entry === + +### {{name}} +WHEN: {{trigger}} +WHAT IT IS: {{description}} +CONTENT (write these exact bytes where the substitution calls for a file): +{{content}} diff --git a/tests/prose/run.cjs b/tests/prose/run.cjs index b03753456..327777d57 100644 --- a/tests/prose/run.cjs +++ b/tests/prose/run.cjs @@ -2,26 +2,26 @@ 'use strict'; // Prose-test runner — everything deterministic about a prose-test run. -// The /prose-test skill drives this CLI and supplies the model layer -// (walker and grader agents). See design/prose-tests.md. +// The /prose-test skill drives this CLI and supplies the two agents: one +// walks the prose, one asserts the result. See design/prose-tests.md. // // list corpus table // select [--diff |--all|--cases a,b] cases to run, as JSON -// world materialise the case's world, print path -// prompt [--world ] walker prompt (NEVER contains expects) -// grade [--world ] run state assertions; emit routing -// claims for the grader agent -// snap (re)generate a fixture's golden snapshot -// verify [fixture] rebuild-compare fixture(s) against snapshot +// world materialise the fixture state, print path +// prompt --world walker prompt (NEVER contains assert.md) +// diff --world acted world vs expected world, as facts +// assert --world the asserting agent's prompt +// snap (re)generate a case's snapshots +// verify [case-id] rebuild-compare snapshot(s) // destroy --world remove a world const fs = require('fs'); const path = require('path'); -const { spawnSync, execFileSync } = require('child_process'); +const { execFileSync } = require('child_process'); const cases = require('./lib/cases.cjs'); -const fixtures = require('./lib/fixtures.cjs'); -const world = require('./lib/world.cjs'); +const worlds = require('./lib/worlds.cjs'); +const prompts = require('./lib/prompts.cjs'); const ROOT = cases.ROOT; @@ -30,11 +30,10 @@ function die(msg) { process.exit(1); } -function loadCase(id) { - const all = cases.loadAllCases(); - const found = all.find((c) => c.id === id); - if (!found) die(`no case "${id}" in the corpus`); - return found; +function getCase(id) { + if (!id) die('a case id is required'); + if (!cases.listCaseIds().includes(id)) die(`no case "${id}" in tests/prose/cases`); + return cases.loadCase(id); } function flag(argv, name) { @@ -44,14 +43,21 @@ function flag(argv, name) { return argv[i + 1]; } +function requireWorld(argv, c) { + const dir = flag(argv, '--world'); + if (!dir) die(`case "${c.id}" needs --world (build one: run.cjs world ${c.id})`); + return dir; +} + // --- select --------------------------------------------------------------- function changedFiles(ref) { const run = (args) => execFileSync('git', args, { cwd: ROOT, encoding: 'utf8' }) .split('\n').map((l) => l.trim()).filter(Boolean); - const diff = run(['diff', '--name-only', ref]); - const status = run(['status', '--porcelain']).map((l) => l.slice(3).trim()); - return new Set([...diff, ...status]); + return new Set([ + ...run(['diff', '--name-only', ref]), + ...run(['status', '--porcelain']).map((l) => l.slice(3).trim()), + ]); } function cmdSelect(argv) { @@ -64,193 +70,131 @@ function cmdSelect(argv) { } else if (flag(argv, '--cases')) { mode = 'ids'; const ids = flag(argv, '--cases').split(',').map((s) => s.trim()); - const byId = new Map(all.map((c) => [c.id, c])); - for (const id of ids) if (!byId.has(id)) die(`no case "${id}" in the corpus`); - selected = ids.map((id) => byId.get(id)); + for (const id of ids) getCase(id); + selected = all.filter((c) => ids.includes(c.id)); } else { const ref = flag(argv, '--diff') || 'main'; mode = `diff:${ref}`; const changed = changedFiles(ref); - const touches = (c) => { - if (changed.has(c.file)) return true; - if (c.files.some((f) => changed.has(f.path))) return true; - if (c.world !== null) { - const prefix = path.posix.join('tests/prose/fixtures', c.world) + '/'; - for (const p of changed) if (p.startsWith(prefix)) return true; - } - return false; - }; - selected = all.filter(touches); + selected = all.filter((c) => c.files.some((f) => changed.has(f.path)) + || [...changed].some((p) => p.startsWith(`${c.rel}/`)) + || c.stubs.some((s) => changed.has(`tests/prose/stubs/${s.name}.md`)) + || [...changed].some((p) => p.startsWith('tests/prose/mainlines/'))); } - process.stdout.write(JSON.stringify({ + process.stdout.write(`${JSON.stringify({ mode, - cases: selected.map((c) => ({ id: c.id, flow: c.flow, file: c.file, world: c.world })), - }, null, 2) + '\n'); + cases: selected.map((c) => ({ + id: c.id, + dir: c.rel, + expects: c.hasAssertionState ? 'assertion state' : 'fixture state unchanged', + })), + }, null, 2)}\n`); } // --- world / destroy ------------------------------------------------------ function cmdWorld(argv) { - const c = loadCase(argv[0] || die('usage: world ')); - if (c.world === null) { - process.stdout.write(JSON.stringify({ world: null, note: 'structure-only case — no world' }) + '\n'); + const c = getCase(argv[0]); + if (!c.hasFixtureState) { + process.stdout.write(`${JSON.stringify({ world: null, note: 'structure-only case — no world' })}\n`); return; } - const dir = world.buildWorld(c.world); - process.stdout.write(JSON.stringify({ world: dir, fixture: c.world }) + '\n'); + process.stdout.write(`${JSON.stringify({ world: worlds.buildWorld(c.id), case: c.id })}\n`); } function cmdDestroy(argv) { const dir = flag(argv, '--world') || die('usage: destroy --world '); - world.destroyWorld(dir); + worlds.destroyWorld(dir); process.stdout.write(`destroyed ${dir}\n`); } -// --- prompt --------------------------------------------------------------- +// --- prompt (the walker) -------------------------------------------------- function cmdPrompt(argv) { - const c = loadCase(argv[0] || die('usage: prompt [--world ]')); - const worldDir = flag(argv, '--world'); - if (c.world !== null && !worldDir) die(`case "${c.id}" needs --world (build one: run.cjs world ${c.id})`); - - const userScript = c.user.length - ? c.user.map((a, i) => ` ${i + 1}. ${a}`).join('\n') - : ' (none — the walk should reach its stop condition without user questions)'; - - const setting = c.world !== null - ? [ - 'You are executing this project\'s workflow prose exactly as a live session would.', - '', - `Project directory — your cwd for EVERY command: ${worldDir}`, - 'The workflow skills are installed at .claude/skills/ inside that project.', - 'Mutations are expected and safe: the project is a disposable test world.', - ] - : [ - 'You are walking workflow prose structurally: read the named files and trace the logic.', - 'Execute nothing — no commands, no writes. This is a read-only walk of the repository.', - '', - `Repository root: ${ROOT}`, - ]; - - const prompt = [ - ...setting, - '', - 'TASK', - c.walk, - '', - 'SCOPE — the prose under walk:', - ...c.files.map((f) => ` - ${f.path}${f.anchor ? ` (start at the heading containing "${f.anchor}")` : ''}`), - '', - 'RULES', - '- Follow the prose literally, step by step, arm by arm. Where it names an', - ' engine or knowledge call, run it (node .claude/skills/workflow-engine/', - ' scripts/engine.cjs …) from the project directory and use the real response.', - '- You also play the user, from a fixed script. When the prose presents a', - ' menu or question, consume the next scripted answer, in order:', - userScript, - '- If the prose asks a question and the script has no next answer: STOP and', - ' record `UNSCRIPTED QUESTION:` with the exact question text.', - '- If two arms both appear to match, record `AMBIGUOUS:` naming both, then', - ' follow the one the prose\'s own ordering/guard rules select.', - '- Never repair, reinterpret, or improve the prose. Execute what is written,', - ' even where it looks wrong. You are a probe, not a reviewer.', - '- Stop at the TASK\'s stop condition, the end of the flow, an UNSCRIPTED', - ' QUESTION, or a hard error — whichever comes first.', - '', - 'TRANSCRIPT — your entire final output, in order of events:', - '1. Every prose section/arm entered: `file.md § Heading` plus the quoted', - ' guard line that selected it.', - '2. Every command run and the first line of its output.', - '3. Every menu/question encountered (verbatim) and the scripted answer used.', - '4. Every file written or edited (path only).', - '5. Finally: `STOPPED: `.', - 'Return nothing but the transcript.', - ].join('\n'); - - process.stdout.write(prompt + '\n'); + const c = getCase(argv[0]); + const worldDir = c.hasFixtureState ? requireWorld(argv, c) : null; + + process.stdout.write(prompts.walkerPrompt({ + worldDir, + root: ROOT, + situation: c.situation, + task: c.act, + scope: c.files + .map((f) => ` - ${f.path}${f.anchor ? ` (start at the heading containing "${f.anchor}")` : ''}`) + .join('\n'), + stubs: c.stubs.map((s) => ({ ...s, ...cases.readStub(s.name) })), + answers: c.answers.map((a, i) => ` ${i + 1}. ${a}`).join('\n'), + })); } -// --- grade ---------------------------------------------------------------- +// --- diff (the facts) ----------------------------------------------------- -function engineInWorld(worldDir, args) { - const engine = path.join(worldDir, '.claude/skills/workflow-engine/scripts/engine.cjs'); - return spawnSync('node', [engine, ...args], { cwd: worldDir, encoding: 'utf8' }); +function cmdDiff(argv) { + const c = getCase(argv[0]); + if (!c.hasFixtureState) { + process.stdout.write(`${JSON.stringify({ note: 'structure-only case — no world to diff' }, null, 2)}\n`); + return; + } + process.stdout.write(`${JSON.stringify(worlds.diffWorld(c.id, requireWorld(argv, c)), null, 2)}\n`); } -function runStateAssertion(worldDir, assertion) { - const a = cases.parseStateAssertion(assertion); - if (a.error) return { assertion, pass: false, actual: a.error }; - if (a.kind === 'file-exists' || a.kind === 'file-absent') { - const present = fs.existsSync(path.join(worldDir, a.path)); - const want = a.kind === 'file-exists'; - return { assertion, pass: present === want, actual: present ? 'present' : 'absent' }; - } - if (a.kind === 'manifest-exists' || a.kind === 'manifest-absent') { - const res = engineInWorld(worldDir, ['manifest', 'exists', a.dotpath, a.field]); - if (res.status !== 0) return { assertion, pass: false, actual: `engine refused: ${res.stderr.trim()}` }; - const present = res.stdout.trim() === 'true'; - const want = a.kind === 'manifest-exists'; - return { assertion, pass: present === want, actual: present ? 'present' : 'absent' }; +// --- assert (the judging agent) ------------------------------------------- + +function cmdAssert(argv) { + const c = getCase(argv[0]); + let world = null; + if (c.hasFixtureState) { + const delta = worlds.diffWorld(c.id, requireWorld(argv, c)); + world = { + expecting: delta.expecting, + delta: [ + JSON.stringify({ added: delta.added, removed: delta.removed, identical: delta.identical }, null, 2), + ...delta.changed, + ].join('\n'), + }; } - // manifest-equals - const res = engineInWorld(worldDir, ['manifest', 'get', a.dotpath, a.field]); - if (res.status !== 0) return { assertion, pass: false, actual: `engine refused: ${res.stderr.trim()}` }; - const actual = res.stdout.trim(); - return { assertion, pass: actual === a.value, actual }; + process.stdout.write(prompts.asserterPrompt({ expected: c.assert, world })); } -function cmdGrade(argv) { - const c = loadCase(argv[0] || die('usage: grade [--world ]')); - const worldDir = flag(argv, '--world'); - const stateExpects = c.expect.filter((e) => e.kind === 'state'); - if (stateExpects.length > 0 && !worldDir) die(`case "${c.id}" has state expects — --world required`); +// --- snap / verify -------------------------------------------------------- - const state = stateExpects.map((e) => runStateAssertion(worldDir, e.text)); - const routing = c.expect.filter((e) => e.kind === 'routing').map((e) => e.text); - process.stdout.write(JSON.stringify({ - id: c.id, - state, - statePass: state.every((s) => s.pass), - routing, - note: routing.length - ? 'routing claims are graded by an agent against the walk transcript — PASS requires quoted evidence' - : 'no routing claims', - }, null, 2) + '\n'); +function statesOf(c) { + return [ + c.hasFixtureState ? 'fixture' : null, + c.hasAssertionState ? 'assertion' : null, + ].filter(Boolean); } -// --- snap / verify -------------------------------------------------------- - function cmdSnap(argv) { - const name = argv[0] || die('usage: snap '); - const scratch = fixtures.runRecipe(name); - try { - const count = fixtures.writeSnapshot(name, scratch); - process.stdout.write(`snapshot ${name}: ${count} files written — review the diff before committing\n`); - } finally { - fs.rmSync(scratch, { recursive: true, force: true }); + const ids = argv[0] ? [getCase(argv[0]).id] : cases.listCaseIds(); + for (const id of ids) { + const c = cases.loadCase(id); + for (const which of statesOf(c)) { + const count = worlds.writeSnapshot(id, which); + process.stdout.write(`${id}/${which}: ${count} files — review the diff before committing\n`); + } } } function cmdVerify(argv) { - const names = argv[0] ? [argv[0]] : fixtures.listFixtures(); + const ids = argv[0] ? [getCase(argv[0]).id] : cases.listCaseIds(); let failed = false; - for (const name of names) { - const scratch = fixtures.runRecipe(name); - try { - const diff = fixtures.compareSnapshot(name, scratch); - const clean = !diff.missing.length && !diff.extra.length && !diff.changed.length; - if (clean) { - process.stdout.write(`${name}: snapshot current\n`); + for (const id of ids) { + const c = cases.loadCase(id); + for (const which of statesOf(c)) { + const d = worlds.verifySnapshot(id, which); + if (d.skipped) { + process.stdout.write(`${id}/${which}: unchanged since last build\n`); + } else if (!d.missing.length && !d.extra.length && !d.changed.length) { + process.stdout.write(`${id}/${which}: snapshot current\n`); } else { failed = true; - process.stdout.write(`${name}: DRIFT — the recipe no longer rebuilds the snapshot\n`); - for (const f of diff.changed) process.stdout.write(` changed: ${f}\n`); - for (const f of diff.extra) process.stdout.write(` extra (rebuilt, not in snapshot): ${f}\n`); - for (const f of diff.missing) process.stdout.write(` missing (in snapshot, not rebuilt): ${f}\n`); - process.stdout.write(` regenerate: node tests/prose/run.cjs snap ${name}\n`); + process.stdout.write(`${id}/${which}: DRIFT — the recipe no longer rebuilds the snapshot\n`); + for (const f of d.changed) process.stdout.write(` changed: ${f}\n`); + for (const f of d.extra) process.stdout.write(` extra (rebuilt, not in snapshot): ${f}\n`); + for (const f of d.missing) process.stdout.write(` missing (in snapshot, not rebuilt): ${f}\n`); + process.stdout.write(` regenerate: node tests/prose/run.cjs snap ${id}\n`); } - } finally { - fs.rmSync(scratch, { recursive: true, force: true }); } } process.exit(failed ? 1 : 0); @@ -262,10 +206,11 @@ function cmdList() { const all = cases.loadAllCases(); const errors = cases.validateCorpus(all); for (const c of all) { - const expects = `${c.expect.filter((e) => e.kind === 'routing').length}r/${c.expect.filter((e) => e.kind === 'state').length}s`; - process.stdout.write(`${c.id} [${c.flow}] world=${c.world || '-'} expects=${expects}\n`); + const stubs = c.stubs.length ? ` stubs=${c.stubs.map((s) => s.name).join(',')}` : ''; + const expects = c.hasAssertionState ? 'assertion state' : 'no change'; + process.stdout.write(`${c.id}\n ${c.hasFixtureState ? 'fixture' : 'no world'} → ${expects}${stubs}\n`); } - process.stdout.write(`\n${all.length} cases, ${fixtures.listFixtures().length} fixtures`); + process.stdout.write(`\n${all.length} cases, ${cases.listStubs().length} stubs`); process.stdout.write(errors.length ? `, ${errors.length} VALIDATION ERRORS:\n` : ', corpus valid\n'); for (const e of errors) process.stdout.write(` - ${e}\n`); process.exit(errors.length ? 1 : 0); @@ -276,9 +221,9 @@ function cmdList() { const [, , command, ...rest] = process.argv; const commands = { list: cmdList, select: cmdSelect, world: cmdWorld, prompt: cmdPrompt, - grade: cmdGrade, snap: cmdSnap, verify: cmdVerify, destroy: cmdDestroy, + diff: cmdDiff, assert: cmdAssert, snap: cmdSnap, verify: cmdVerify, destroy: cmdDestroy, }; if (!commands[command]) { - die('usage: run.cjs …'); + die('usage: run.cjs …'); } commands[command](rest); diff --git a/tests/prose/smoke/cases.md b/tests/prose/smoke/cases.md deleted file mode 100644 index 34018bfda..000000000 --- a/tests/prose/smoke/cases.md +++ /dev/null @@ -1,44 +0,0 @@ -# Smoke cases — the framework proving itself - -Not corpus: these two cases exercise every moving part of the machinery -(parser, anchor checks, world builder, engine-in-world, state assertions, -walker protocol) against prose stable enough to survive. If a smoke case -fails, suspect the framework before the prose. - -## case: smoke-structure-start-boot -- origin: framework smoke — structure-only walk, no world -- files: - - skills/workflow-start/SKILL.md#Boot - -### walk - -Structure-only: read the initialisation portion of -skills/workflow-start/SKILL.md — everything the skill does before any -work is shown or routed. Establish what the initialisation consists of -and in what order its parts run. Stop there — do not trace the rest of -the skill. - -### expect - -- routing: casing conventions are loaded before the boot pipeline runs -- routing: the boot pipeline is declared mandatory — the skill must complete it before proceeding - -## case: smoke-world-boot -- world: base -- origin: framework smoke — world build, engine-in-world, state assertions -- files: - - skills/workflow-start/SKILL.md#Boot - -### walk - -In the project, run the boot command exactly as the initialisation prose -of skills/workflow-start/SKILL.md prescribes (sandbox concerns do not -apply in this world — run it directly). Capture the JSON response. Stop -immediately after recording it — do not continue into any confirmation -prompt or later step. - -### expect - -- routing: the boot response reports the knowledge base ready (keyword-only mode) -- routing: the boot response reports no migrations applied on this run (the world is already migrated) -- state: file exists .workflows/.state/migrations diff --git a/tests/prose/stubs/root-cause-validated.md b/tests/prose/stubs/root-cause-validated.md new file mode 100644 index 000000000..daaae4b9a --- /dev/null +++ b/tests/prose/stubs/root-cause-validated.md @@ -0,0 +1,19 @@ +# stub: root-cause-validated + +A root-cause validation agent's report: the hypothesis confirmed by an +independent trace, no gaps. The content below is written to the path the +dispatch response returned; the STATUS block is also what the agent +returns to its caller. + +--- + +# Root Cause Validation + +The tax context is built before the payment intent and reads the shipping +address unconditionally. Traced fresh: an address-less order fails at that +read, which matches the documented root cause. + +STATUS: validated +CONFIDENCE: high +GAPS_COUNT: 0 +SUMMARY: Root cause confirmed by an independent trace. diff --git a/tests/scripts/test-prose-cases.cjs b/tests/scripts/test-prose-cases.cjs index 1cb83129b..847624496 100644 --- a/tests/scripts/test-prose-cases.cjs +++ b/tests/scripts/test-prose-cases.cjs @@ -1,30 +1,69 @@ 'use strict'; // Prose-test corpus: the deterministic perimeter (design/prose-tests.md -// P3). Every case parses, every scoped file and anchor resolves, every -// referenced world has a recipe, every state assertion is grammatical. -// The token-costing walks run on command via /prose-test — never here. +// P3). Every case directory is well-formed, every scoped file and anchor +// resolves, every stub is named and triggered, every recipe loads. The +// token-costing walks run on command via /prose-test — never here. const { describe, it } = require('node:test'); const assert = require('node:assert'); +const fs = require('fs'); +const path = require('path'); const cases = require('../prose/lib/cases.cjs'); describe('prose-test corpus', () => { + const all = cases.loadAllCases(); + it('has at least one case', () => { - assert.ok(cases.loadAllCases().length > 0, 'corpus is empty'); + assert.ok(all.length > 0, 'corpus is empty'); }); - it('parses and validates cleanly', () => { - const errors = cases.validateCorpus(cases.loadAllCases()); + it('validates cleanly', () => { + const errors = cases.validateCorpus(all); assert.deepStrictEqual(errors, [], `corpus validation failed:\n${errors.map((e) => ` - ${e}`).join('\n')}`); }); - it('state assertion grammar rejects garbage', () => { - assert.ok(cases.parseStateAssertion('file exists .workflows/x').kind === 'file-exists'); - assert.ok(cases.parseStateAssertion('manifest equals wu.planning.t review_cycle 1').kind === 'manifest-equals'); - assert.ok(cases.parseStateAssertion('the manifest should look right').error); - assert.ok(cases.parseStateAssertion('file exists').error); + it('is one directory per case, holding only known files', () => { + const known = new Set([...Object.values(cases.FILES), ...Object.values(cases.SNAPSHOTS)]); + for (const c of all) { + for (const entry of fs.readdirSync(c.dir)) { + assert.ok(known.has(entry), + `${c.rel}/${entry}: unexpected file — a case holds only ${[...known].join(', ')}`); + } + } + }); + + it('keeps the walker and asserter prose in separate files', () => { + // The P4 boundary is enforced by the filesystem: one file goes to the + // walker, the other only to the asserter. Never merge them. + for (const c of all) { + assert.ok(c.act, `${c.rel}: no ${cases.FILES.act}`); + assert.ok(c.assert, `${c.rel}: no ${cases.FILES.assert}`); + assert.notStrictEqual(cases.FILES.act, cases.FILES.assert); + } + }); + + it('every stub is named, triggered by its case, and readable', () => { + for (const c of all) { + for (const s of c.stubs) { + assert.ok(s.trigger.trim().length > 0, + `${c.rel}: stub "${s.name}" has no trigger — a stub without a moment is a fixture in disguise`); + assert.ok(cases.readStub(s.name).content.length > 0, + `stub "${s.name}" has no content below its --- fence`); + } + } + }); + + it('every state recipe loads and exports build()', () => { + for (const c of all) { + for (const which of ['fixtureState', 'assertionState']) { + const file = path.join(c.dir, cases.FILES[which]); + if (!fs.existsSync(file)) continue; + assert.strictEqual(typeof cases.requireState(c.id, which).build, 'function', + `${c.rel}/${cases.FILES[which]} must export build(h)`); + } + } }); }); diff --git a/tests/scripts/test-prose-fixtures.cjs b/tests/scripts/test-prose-fixtures.cjs deleted file mode 100644 index c1c652e18..000000000 --- a/tests/scripts/test-prose-fixtures.cjs +++ /dev/null @@ -1,43 +0,0 @@ -'use strict'; - -// Fixture golden check (design/prose-tests.md P2/P3): every committed -// snapshot rebuilds byte-identical from its recipe under the frozen -// clock. A red run here means the engine moved the world — regenerate -// with `node tests/prose/run.cjs snap ` and review the snapshot -// diff in the same PR as the change that moved it. Never hand-edit a -// snapshot to green this. - -const { describe, it } = require('node:test'); -const assert = require('node:assert'); -const fs = require('fs'); - -const fixtures = require('../prose/lib/fixtures.cjs'); - -describe('prose-test fixtures', () => { - const names = fixtures.listFixtures(); - - it('has at least one fixture', () => { - assert.ok(names.length > 0, 'no fixtures found'); - }); - - for (const name of names) { - it(`${name}: recipe rebuilds the committed snapshot byte-identical`, () => { - assert.ok(fixtures.readSnapshot(name) !== null, - `fixture "${name}" has no committed snapshot — run: node tests/prose/run.cjs snap ${name}`); - const scratch = fixtures.runRecipe(name); - try { - const diff = fixtures.compareSnapshot(name, scratch); - const report = [ - ...diff.changed.map((f) => `changed: ${f}`), - ...diff.extra.map((f) => `extra (rebuilt, not in snapshot): ${f}`), - ...diff.missing.map((f) => `missing (in snapshot, not rebuilt): ${f}`), - ]; - assert.deepStrictEqual(report, [], - `world moved — regenerate: node tests/prose/run.cjs snap ${name}\n` + - report.map((l) => ` ${l}`).join('\n')); - } finally { - fs.rmSync(scratch, { recursive: true, force: true }); - } - }); - } -}); diff --git a/tests/scripts/test-prose-snapshots.cjs b/tests/scripts/test-prose-snapshots.cjs new file mode 100644 index 000000000..fc3303aa2 --- /dev/null +++ b/tests/scripts/test-prose-snapshots.cjs @@ -0,0 +1,50 @@ +'use strict'; + +// Snapshot goldens (design/prose-tests.md P2/P3): every committed +// snapshot rebuilds byte-identical from its recipe under the frozen +// clock. A red run means the engine moved the world — regenerate with +// `node tests/prose/run.cjs snap ` and land the snapshot diff in +// the same commit as the change that moved it. Never hand-edit a +// snapshot to green this. +// +// Rebuilds are skipped for worlds whose recipes, shared mainlines, and +// engine sources are all unchanged since the snapshot was written — so a +// normal run costs nothing, and any engine change invalidates every hash +// and rebuilds the lot. + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); + +const cases = require('../prose/lib/cases.cjs'); +const worlds = require('../prose/lib/worlds.cjs'); + +describe('prose-test snapshots', () => { + const all = cases.loadAllCases(); + + it('has at least one built world', () => { + assert.ok(all.some((c) => c.hasFixtureState), 'no case builds a world'); + }); + + for (const c of all) { + const states = [ + c.hasFixtureState ? 'fixture' : null, + c.hasAssertionState ? 'assertion' : null, + ].filter(Boolean); + + for (const which of states) { + it(`${c.id}/${which}: rebuilds byte-identical`, () => { + assert.ok(worlds.readSnapshot(c.id, which) !== null, + `${c.id}/${which} has no committed snapshot — run: node tests/prose/run.cjs snap ${c.id}`); + const d = worlds.verifySnapshot(c.id, which); + if (d.skipped) return; + const report = [ + ...d.changed.map((f) => `changed: ${f}`), + ...d.extra.map((f) => `extra (rebuilt, not in snapshot): ${f}`), + ...d.missing.map((f) => `missing (in snapshot, not rebuilt): ${f}`), + ]; + assert.deepStrictEqual(report, [], + `world moved — regenerate: node tests/prose/run.cjs snap ${c.id}\n${report.map((l) => ` ${l}`).join('\n')}`); + }); + } + } +});