feat(prose-tests): the framework — cases, worlds, runner, skill

.claude/skills/prose-test/SKILL.md

@@ -0,0 +1,59 @@
+---
+name: prose-test
+description: Run prose-logic test cases — walker agents execute workflow prose against materialised fixture worlds, graders check the transcripts, deterministic state assertions run in code. Scope by diff (default), case ids, or --all.
+---
+
+# Prose Test
+
+Run prose-test cases (design: `design/prose-tests.md`; runner:
+`tests/prose/run.cjs`). Walks cost tokens — this skill runs on command
+only, never as part of a routine gate.
+
+## Step 1: Select
+
+Parse the arguments:
+
+- Case ids (comma or space separated) → `node tests/prose/run.cjs select --cases <a,b>`
+- `--all` → `node tests/prose/run.cjs select --all`
+- `--diff <ref>` → `node tests/prose/run.cjs select --diff <ref>`
+- No arguments → `node tests/prose/run.cjs select --diff main`
+
+If the selection is empty: report that no cases intersect and stop.
+Otherwise show the selected case ids and proceed.
+
+## Step 2: Walk and grade each case
+
+Cases are independent — run up to 4 concurrently. Per case:
+
+1. **World** (skip for `world=null` cases): `node tests/prose/run.cjs world <id>` — note the returned path.
+2. **Walker**: `node tests/prose/run.cjs prompt <id> [--world <dir>]`, then dispatch a subagent (model: **sonnet**) with that prompt **verbatim and unmodified**. Never hand-assemble a walker prompt and never mention any expectation to the walker — the `prompt` command's output is the whole contract. Save the returned transcript to a scratch file.
+3. **Deterministic grade**: `node tests/prose/run.cjs grade <id> [--world <dir>]` — state assertions pass/fail in code; the output also lists the routing claims.
+4. **Grader**: dispatch a second subagent (model: **sonnet**) with: the transcript, the routing claims, and the case's scoped prose files. Instructions: verdict per claim, PASS only with a quoted line from the transcript (and prose where relevant) that satisfies it; FAIL must state what is missing or contradicting. A PASS without a quote is invalid — treat as FAIL and re-grade.
+
+A walker that returns `UNSCRIPTED QUESTION` or `AMBIGUOUS` is a finding
+in itself — carry it into the report even if claims pass.
+
+## Step 3: Escalate failures
+
+Any failed claim (state or routing):
+
+1. Build a **fresh** world and re-run the walker on model: **opus**, then re-grade.
+2. Both fail → confirmed finding.
+3. Sonnet-fail / Opus-pass → disagreement: report both walks with the divergent lines quoted. Never auto-resolve.
+
+## Step 4: Report and clean up
+
+Report a verdict table: case id, state assertions passed/total, routing
+claims passed/total, escalations, findings. Quote the evidence for every
+failure. Findings are findings — never edit prose or cases mid-run;
+report and stop.
+
+Destroy every world (`node tests/prose/run.cjs destroy --world <dir>`)
+except one you are actively citing in a failure report; name any world
+kept so it can be removed later.
+
+## Rules
+
+- The walker never sees `expect:` claims — only the `prompt` command's output reaches it.
+- A failing case is a finding either way: broken prose or a stale case. The report says which is suspected, the user decides.
+- Never regenerate fixtures or edit cases to make a run pass.

CLAUDE.md

@@ -208,6 +208,10 @@ Add or update a test alongside any change to engine scripts, adapters, migration
 
 **Any change to the workflows must update the simulation** — a new engine verb, a changed prose call sequence, a new phase ordering, a new manifest field: extend an existing scenario, add a new permutation, or re-pin changed expectations. A red simulation is the design speaking — decide deliberately whether the flow or the scenario is wrong, never paper over it. New permutations are cheap: a scenario is an ordered list of engine calls with assertions.
 
+## Prose Tests
+
+`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). Natural-language cases a walker agent executes through the real skills against a materialised fixture world; run on command via the `/prose-test` dev skill — walks cost tokens, so they are never part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation (parse, scoped paths, anchors, state-assertion grammar) and fixture golden checks — every `tests/prose/fixtures/{name}/snapshot/` must rebuild byte-identical from its `recipe.cjs` under the frozen clock. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {name}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases.
+
 ## Knowledge Base Subsystem
 
 Retrieval-augmented store of completed workflow artifacts (research, discussion, investigation, specification — never planning/implementation/review), plus epic discovery **session logs** (indexed under a `discovery` phase — the running exploration record) and seed material for early-phase context: user-shared `imports` and the inbox-promoted `seeds` (the work unit's origin). Every entry-point skill gates on knowledge base initialisation before any phase runs.

design/prose-tests.md

@@ -109,6 +109,39 @@ testing.
 
 ## Log
 
+- 2026-07-25 — Brittleness review (Lee, on the stage-2 smoke cases):
+  cases written in prose coordinates (step numbers, numbered headings)
+  break on cosmetic renumbering — failure for the wrong reason. Ruling:
+  **cases name behaviour, never coordinates** (the walker is an agent
+  reading prose — the binding is semantic and rename-robust; only
+  coordinate-phrased claim text rots), and anchors became substring
+  fragments matched with `includes` (`#Boot` survives a renumber, still
+  goes red token-free if the heading vanishes). Prose test hooks
+  (`data-test`-style markers in skill files) considered and **parked**:
+  the only machine consumer of anchors is the staleness check, so hooks
+  would put permanent plumbing into shipped prose to serve a consumer
+  that barely needs it — revisit on evidence if arm-level references in
+  the failure harvest (generic headings like "If valid") produce real
+  renumber noise; the grammar extends to `probe:` references without
+  breakage.
+
+- 2026-07-24 — Stage 2, the framework. `tests/prose/`: case parser and
+  corpus validation (`lib/cases.cjs`), fixture recipes under a frozen
+  clock with golden snapshots (`lib/fixtures.cjs`, `lib/fake-clock.cjs`
+  — snapshots exclude `.git` and the binary knowledge store, escape
+  product-written `.gitignore` files), the world builder
+  (`lib/world.cjs` — installed-layout skills, hermetic git, keyword-only
+  store re-derived at materialise), and the runner (`run.cjs`: list ·
+  select · world · prompt · grade · snap · verify · destroy). The
+  `prompt` command is the P4 boundary — walker prompts are
+  machine-assembled and never contain expects. CI gains two token-free
+  suites (corpus validation, snapshot rebuild-compare); the `/prose-test`
+  dev skill owns the model layer (Sonnet walks, Opus confirms, quoted
+  evidence or it didn't happen). First fixture: `base` — a post-boot
+  keyword-only empty project, proven byte-deterministic across rebuilds.
+  Two smoke cases exercise every moving part end-to-end; the full pipe
+  (world → boot-in-world → deterministic grade) executed clean.
+
 - 2026-07-24 — Design agreed with Lee. Key decisions: static fixtures
   over sim-replay (decoupling; the sim stays the engine's test bed,
   prose tests own their worlds) with the engine kept at authoring

package.json

@@ -7,7 +7,7 @@
     "build": "node build/knowledge.build.js",
     "knowledge:repl": "node scripts/knowledge-repl.js",
     "typecheck": "tsc -p tsconfig.json",
-    "test": "node --test tests/scripts/test-render.cjs tests/scripts/test-engine-gateway.cjs tests/scripts/test-engine-epic-projections.cjs tests/scripts/test-engine-start-projections.cjs tests/scripts/test-engine-workunit-projections.cjs tests/scripts/test-engine-discussion-map.cjs tests/scripts/test-engine-discovery-map.cjs tests/scripts/test-engine-tasks.cjs tests/scripts/test-engine-render-surfaces.cjs tests/scripts/test-engine-transactions.cjs tests/scripts/test-engine-workunit-create.cjs tests/scripts/test-engine-workunit-pivot.cjs tests/scripts/test-engine-workunit-absorb.cjs tests/scripts/test-engine-workunit-promote.cjs tests/scripts/test-engine-discovery-session.cjs tests/scripts/test-engine-cache.cjs tests/scripts/test-engine-manifest-io.cjs tests/scripts/test-engine-manifest-fields.cjs tests/scripts/test-engine-boot.cjs tests/scripts/test-migration-orchestrator.cjs tests/scripts/test-migration-047.cjs tests/scripts/test-migration-048.cjs tests/scripts/test-migration-049.cjs tests/scripts/test-migration-050.cjs tests/scripts/test-migration-051.cjs tests/scripts/test-migration-052.cjs tests/scripts/test-engine-specification-projections.cjs tests/scripts/test-engine-discovery-projections.cjs tests/scripts/test-gateway-for-start.cjs tests/scripts/test-gateway-for-specification.cjs tests/scripts/test-gateway-for-workflow-continue-epic.cjs tests/scripts/test-gateway-for-workflow-continue-feature.cjs tests/scripts/test-gateway-for-workflow-continue-bugfix.cjs tests/scripts/test-gateway-for-workflow-continue-quickfix.cjs tests/scripts/test-gateway-for-workflow-continue-cross-cutting.cjs tests/scripts/test-gateway-for-discovery-skill.cjs tests/scripts/test-gateway-for-bridge.cjs tests/scripts/test-reads-derivations.cjs tests/scripts/test-ensure-discovery-item.cjs tests/scripts/test-legacy-research-split.cjs tests/scripts/test-knowledge-chunker.cjs tests/scripts/test-knowledge-config.cjs tests/scripts/test-knowledge-discovery.cjs tests/scripts/test-knowledge-embeddings.cjs tests/scripts/test-knowledge-integration.cjs tests/scripts/test-knowledge-openai.cjs tests/scripts/test-knowledge-openai-compatible.cjs tests/scripts/test-knowledge-openai-integration.cjs tests/scripts/test-knowledge-progress.cjs tests/scripts/test-knowledge-rerank.cjs tests/scripts/test-knowledge-retry.cjs tests/scripts/test-knowledge-setup-forms.cjs tests/scripts/test-knowledge-store.cjs tests/scripts/test-conventions-lint.cjs tests/scripts/test-pipeline-simulation.cjs tests/scripts/test-engine-agent-state.cjs",
+    "test": "node --test tests/scripts/test-render.cjs tests/scripts/test-engine-gateway.cjs tests/scripts/test-engine-epic-projections.cjs tests/scripts/test-engine-start-projections.cjs tests/scripts/test-engine-workunit-projections.cjs tests/scripts/test-engine-discussion-map.cjs tests/scripts/test-engine-discovery-map.cjs tests/scripts/test-engine-tasks.cjs tests/scripts/test-engine-render-surfaces.cjs tests/scripts/test-engine-transactions.cjs tests/scripts/test-engine-workunit-create.cjs tests/scripts/test-engine-workunit-pivot.cjs tests/scripts/test-engine-workunit-absorb.cjs tests/scripts/test-engine-workunit-promote.cjs tests/scripts/test-engine-discovery-session.cjs tests/scripts/test-engine-cache.cjs tests/scripts/test-engine-manifest-io.cjs tests/scripts/test-engine-manifest-fields.cjs tests/scripts/test-engine-boot.cjs tests/scripts/test-migration-orchestrator.cjs tests/scripts/test-migration-047.cjs tests/scripts/test-migration-048.cjs tests/scripts/test-migration-049.cjs tests/scripts/test-migration-050.cjs tests/scripts/test-migration-051.cjs tests/scripts/test-migration-052.cjs tests/scripts/test-engine-specification-projections.cjs tests/scripts/test-engine-discovery-projections.cjs tests/scripts/test-gateway-for-start.cjs tests/scripts/test-gateway-for-specification.cjs tests/scripts/test-gateway-for-workflow-continue-epic.cjs tests/scripts/test-gateway-for-workflow-continue-feature.cjs tests/scripts/test-gateway-for-workflow-continue-bugfix.cjs tests/scripts/test-gateway-for-workflow-continue-quickfix.cjs tests/scripts/test-gateway-for-workflow-continue-cross-cutting.cjs tests/scripts/test-gateway-for-discovery-skill.cjs tests/scripts/test-gateway-for-bridge.cjs tests/scripts/test-reads-derivations.cjs tests/scripts/test-ensure-discovery-item.cjs tests/scripts/test-legacy-research-split.cjs tests/scripts/test-knowledge-chunker.cjs tests/scripts/test-knowledge-config.cjs tests/scripts/test-knowledge-discovery.cjs tests/scripts/test-knowledge-embeddings.cjs tests/scripts/test-knowledge-integration.cjs tests/scripts/test-knowledge-openai.cjs tests/scripts/test-knowledge-openai-compatible.cjs tests/scripts/test-knowledge-openai-integration.cjs tests/scripts/test-knowledge-progress.cjs tests/scripts/test-knowledge-rerank.cjs tests/scripts/test-knowledge-retry.cjs tests/scripts/test-knowledge-setup-forms.cjs tests/scripts/test-knowledge-store.cjs tests/scripts/test-conventions-lint.cjs tests/scripts/test-pipeline-simulation.cjs tests/scripts/test-engine-agent-state.cjs tests/scripts/test-prose-cases.cjs tests/scripts/test-prose-fixtures.cjs",
     "test:migrations": "for t in tests/scripts/test-migration-*.sh; do bash $t || exit 1; done",
     "test:cli": "for t in tests/scripts/test-engine-manifest.sh tests/scripts/test-inbox-promotion.sh tests/scripts/test-knowledge-cli.sh tests/scripts/test-knowledge-build.sh; do bash $t || exit 1; done"
   },

tests/prose/README.md

@@ -0,0 +1,42 @@
+# Prose tests
+
+End-to-end tests for the prose logic — natural-language cases a walker
+agent executes through the real skills against a materialised fixture
+world. Design and contract: [design/prose-tests.md](../../design/prose-tests.md).
+Run via the `/prose-test` skill; everything deterministic lives in
+`run.cjs` and `lib/`.
+
+## Layout
+
+- `{flow}/*.md` — case files (grammar: header comment in `lib/cases.cjs`)
+- `fixtures/{name}/recipe.cjs` — builds the world with real engine calls
+  under a frozen clock (`lib/fake-clock.cjs`)
+- `fixtures/{name}/snapshot/` — the recipe's committed golden output;
+  regenerate with `node tests/prose/run.cjs snap {name}`, never hand-edit
+- `run.cjs` — `list · select · world · prompt · grade · snap · verify · destroy`
+
+## Adding a case
+
+Add a `## case:` block to the flow's file (or a new file for a long
+walk). Scope `files:` tightly — diff-selection and the PR-end suggestion
+run off it. `routing:` expects are agent-graded against the walk
+transcript; `state:` expects are asserted in code and require a `world:`.
+
+**Name behaviour, never coordinates.** Step numbers, arm letters, and
+heading numbering in walk or expect text break on cosmetic renumbering —
+failure for the wrong reason. Write "casing conventions load before the
+boot pipeline", not "Step 0.1 before Step 0.2"; a behavioural claim
+fails only when the behaviour changes, which is the failure the corpus
+exists to catch. Anchors are substring fragments matched against heading
+text (`#Boot` matches "Step 0.2: Boot") — pick the number-free part.
+`npm test` validates the corpus (parse, paths, anchors, grammar) and
+rebuild-compares every fixture — both token-free; only `/prose-test`
+spends tokens.
+
+## Snapshot escaping
+
+Snapshots exclude `.git/` and `.workflows/.knowledge/` (the world
+builder re-derives the store), and store `.gitignore` files as
+`_gitignore.fixture` so the product-written `.workflows/.gitignore`
+cannot ignore fixture content out of this repo; the world builder
+restores them.

tests/prose/fixtures/base/recipe.cjs

@@ -0,0 +1,13 @@
+'use strict';
+
+// The canonical empty project — a real install straight after
+// /workflow-start Step 0: migrations applied (the log is the durable
+// trace), knowledge set up keyword-only, boot run. No work units yet.
+// Every richer fixture starts from this world's shape.
+
+module.exports = {
+  build(h) {
+    h.knowledge('setup', '--keyword-only');
+    h.engine('boot');
+  },
+};

tests/prose/fixtures/base/snapshot/.claude/settings.json

@@ -0,0 +1,9 @@
+{
+  "showClearContextOnPlanAccept": true,
+  "permissions": {
+    "allow": [
+      "Edit(.workflows/**)",
+      "Bash(mv .workflows/:*)"
+    ]
+  }
+}

tests/prose/fixtures/base/snapshot/.workflows/.state/migrations

@@ -0,0 +1,52 @@
+001
+002
+003
+004
+005
+006
+007
+008
+009
+010
+011
+012
+013
+014
+015
+016
+017
+018
+019
+020
+021
+022
+023
+024
+025
+026
+027
+028
+029
+030
+031
+032
+033
+034
+035
+036
+037
+038
+039
+040
+041
+042
+043
+044
+045
+046
+047
+048
+049
+050
+051
+052

tests/prose/fixtures/base/snapshot/.workflows/_gitignore.fixture

@@ -0,0 +1,2 @@
+.cache/
+.manifest.json.*.tmp