From cee751ba0c10a80c9602b777940a3771db1af9b1 Mon Sep 17 00:00:00 2001 From: Lee Overy Date: Fri, 24 Jul 2026 16:52:32 +0100 Subject: [PATCH 1/2] =?UTF-8?q?feat(prose-tests):=20the=20framework=20?= =?UTF-8?q?=E2=80=94=20cases,=20worlds,=20runner,=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 2 of design/prose-tests.md. The deterministic layer in code: case parser + corpus validation, fixture recipes under a frozen clock with golden snapshots (rebuild-compare in npm test), the world builder (installed-layout skills, hermetic git, keyword-only knowledge store), and the runner whose `prompt` command is the P4 boundary — walker prompts never carry expects. The /prose-test dev skill owns the model layer: Sonnet walks, Opus confirms failures, quoted evidence required. First fixture (base) proven byte-deterministic; two smoke cases exercised the full pipe end-to-end. Co-Authored-By: Claude Fable 5 --- .claude/skills/prose-test/SKILL.md | 59 ++++ CLAUDE.md | 4 + design/prose-tests.md | 17 ++ package.json | 2 +- tests/prose/README.md | 34 +++ tests/prose/fixtures/base/recipe.cjs | 13 + .../base/snapshot/.claude/settings.json | 9 + .../snapshot/.workflows/.state/migrations | 52 ++++ .../snapshot/.workflows/_gitignore.fixture | 2 + tests/prose/lib/cases.cjs | 199 ++++++++++++ tests/prose/lib/fake-clock.cjs | 30 ++ tests/prose/lib/fixtures.cjs | 196 ++++++++++++ tests/prose/lib/world.cjs | 75 +++++ tests/prose/run.cjs | 284 ++++++++++++++++++ tests/prose/smoke/cases.md | 43 +++ tests/scripts/test-prose-cases.cjs | 30 ++ tests/scripts/test-prose-fixtures.cjs | 43 +++ 17 files changed, 1091 insertions(+), 1 deletion(-) create mode 100644 .claude/skills/prose-test/SKILL.md create mode 100644 tests/prose/README.md create mode 100644 tests/prose/fixtures/base/recipe.cjs create mode 100644 tests/prose/fixtures/base/snapshot/.claude/settings.json create mode 100644 tests/prose/fixtures/base/snapshot/.workflows/.state/migrations create mode 100644 tests/prose/fixtures/base/snapshot/.workflows/_gitignore.fixture create mode 100644 tests/prose/lib/cases.cjs create mode 100644 tests/prose/lib/fake-clock.cjs create mode 100644 tests/prose/lib/fixtures.cjs create mode 100644 tests/prose/lib/world.cjs create mode 100644 tests/prose/run.cjs create mode 100644 tests/prose/smoke/cases.md create mode 100644 tests/scripts/test-prose-cases.cjs create mode 100644 tests/scripts/test-prose-fixtures.cjs diff --git a/.claude/skills/prose-test/SKILL.md b/.claude/skills/prose-test/SKILL.md new file mode 100644 index 000000000..ac705b45b --- /dev/null +++ b/.claude/skills/prose-test/SKILL.md @@ -0,0 +1,59 @@ +--- +name: prose-test +description: Run prose-logic test cases — walker agents execute workflow prose against materialised fixture worlds, graders check the transcripts, deterministic state assertions run in code. Scope by diff (default), case ids, or --all. +--- + +# Prose Test + +Run prose-test cases (design: `design/prose-tests.md`; runner: +`tests/prose/run.cjs`). Walks cost tokens — this skill runs on command +only, never as part of a routine gate. + +## Step 1: Select + +Parse the arguments: + +- Case ids (comma or space separated) → `node tests/prose/run.cjs select --cases ` +- `--all` → `node tests/prose/run.cjs select --all` +- `--diff ` → `node tests/prose/run.cjs select --diff ` +- No arguments → `node tests/prose/run.cjs select --diff main` + +If the selection is empty: report that no cases intersect and stop. +Otherwise show the selected case ids and proceed. + +## Step 2: Walk and grade each case + +Cases are independent — run up to 4 concurrently. Per case: + +1. **World** (skip for `world=null` cases): `node tests/prose/run.cjs world ` — note the returned path. +2. **Walker**: `node tests/prose/run.cjs prompt [--world ]`, then dispatch a subagent (model: **sonnet**) with that prompt **verbatim and unmodified**. Never hand-assemble a walker prompt and never mention any expectation to the walker — the `prompt` command's output is the whole contract. Save the returned transcript to a scratch file. +3. **Deterministic grade**: `node tests/prose/run.cjs grade [--world ]` — state assertions pass/fail in code; the output also lists the routing claims. +4. **Grader**: dispatch a second subagent (model: **sonnet**) with: the transcript, the routing claims, and the case's scoped prose files. Instructions: verdict per claim, PASS only with a quoted line from the transcript (and prose where relevant) that satisfies it; FAIL must state what is missing or contradicting. A PASS without a quote is invalid — treat as FAIL and re-grade. + +A walker that returns `UNSCRIPTED QUESTION` or `AMBIGUOUS` is a finding +in itself — carry it into the report even if claims pass. + +## Step 3: Escalate failures + +Any failed claim (state or routing): + +1. Build a **fresh** world and re-run the walker on model: **opus**, then re-grade. +2. Both fail → confirmed finding. +3. Sonnet-fail / Opus-pass → disagreement: report both walks with the divergent lines quoted. Never auto-resolve. + +## Step 4: Report and clean up + +Report a verdict table: case id, state assertions passed/total, routing +claims passed/total, escalations, findings. Quote the evidence for every +failure. Findings are findings — never edit prose or cases mid-run; +report and stop. + +Destroy every world (`node tests/prose/run.cjs destroy --world `) +except one you are actively citing in a failure report; name any world +kept so it can be removed later. + +## Rules + +- The walker never sees `expect:` claims — only the `prompt` command's output reaches it. +- A failing case is a finding either way: broken prose or a stale case. The report says which is suspected, the user decides. +- Never regenerate fixtures or edit cases to make a run pass. diff --git a/CLAUDE.md b/CLAUDE.md index 47e4ffa52..d161a8968 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -208,6 +208,10 @@ Add or update a test alongside any change to engine scripts, adapters, migration **Any change to the workflows must update the simulation** — a new engine verb, a changed prose call sequence, a new phase ordering, a new manifest field: extend an existing scenario, add a new permutation, or re-pin changed expectations. A red simulation is the design speaking — decide deliberately whether the flow or the scenario is wrong, never paper over it. New permutations are cheap: a scenario is an ordered list of engine calls with assertions. +## Prose Tests + +`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). Natural-language cases a walker agent executes through the real skills against a materialised fixture world; run on command via the `/prose-test` dev skill — walks cost tokens, so they are never part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation (parse, scoped paths, anchors, state-assertion grammar) and fixture golden checks — every `tests/prose/fixtures/{name}/snapshot/` must rebuild byte-identical from its `recipe.cjs` under the frozen clock. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {name}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases. + ## Knowledge Base Subsystem Retrieval-augmented store of completed workflow artifacts (research, discussion, investigation, specification — never planning/implementation/review), plus epic discovery **session logs** (indexed under a `discovery` phase — the running exploration record) and seed material for early-phase context: user-shared `imports` and the inbox-promoted `seeds` (the work unit's origin). Every entry-point skill gates on knowledge base initialisation before any phase runs. diff --git a/design/prose-tests.md b/design/prose-tests.md index 045cdd587..5a6ca14db 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -109,6 +109,23 @@ testing. ## Log +- 2026-07-24 — Stage 2, the framework. `tests/prose/`: case parser and + corpus validation (`lib/cases.cjs`), fixture recipes under a frozen + clock with golden snapshots (`lib/fixtures.cjs`, `lib/fake-clock.cjs` + — snapshots exclude `.git` and the binary knowledge store, escape + product-written `.gitignore` files), the world builder + (`lib/world.cjs` — installed-layout skills, hermetic git, keyword-only + store re-derived at materialise), and the runner (`run.cjs`: list · + select · world · prompt · grade · snap · verify · destroy). The + `prompt` command is the P4 boundary — walker prompts are + machine-assembled and never contain expects. CI gains two token-free + suites (corpus validation, snapshot rebuild-compare); the `/prose-test` + dev skill owns the model layer (Sonnet walks, Opus confirms, quoted + evidence or it didn't happen). First fixture: `base` — a post-boot + keyword-only empty project, proven byte-deterministic across rebuilds. + Two smoke cases exercise every moving part end-to-end; the full pipe + (world → boot-in-world → deterministic grade) executed clean. + - 2026-07-24 — Design agreed with Lee. Key decisions: static fixtures over sim-replay (decoupling; the sim stays the engine's test bed, prose tests own their worlds) with the engine kept at authoring diff --git a/package.json b/package.json index 0a3b9fc67..482711aca 100644 --- a/package.json +++ b/package.json @@ -7,7 +7,7 @@ "build": "node build/knowledge.build.js", "knowledge:repl": "node scripts/knowledge-repl.js", "typecheck": "tsc -p tsconfig.json", - "test": "node --test tests/scripts/test-render.cjs tests/scripts/test-engine-gateway.cjs tests/scripts/test-engine-epic-projections.cjs tests/scripts/test-engine-start-projections.cjs tests/scripts/test-engine-workunit-projections.cjs tests/scripts/test-engine-discussion-map.cjs tests/scripts/test-engine-discovery-map.cjs tests/scripts/test-engine-tasks.cjs tests/scripts/test-engine-render-surfaces.cjs tests/scripts/test-engine-transactions.cjs tests/scripts/test-engine-workunit-create.cjs tests/scripts/test-engine-workunit-pivot.cjs tests/scripts/test-engine-workunit-absorb.cjs tests/scripts/test-engine-workunit-promote.cjs tests/scripts/test-engine-discovery-session.cjs tests/scripts/test-engine-cache.cjs tests/scripts/test-engine-manifest-io.cjs tests/scripts/test-engine-manifest-fields.cjs tests/scripts/test-engine-boot.cjs tests/scripts/test-migration-orchestrator.cjs tests/scripts/test-migration-047.cjs tests/scripts/test-migration-048.cjs tests/scripts/test-migration-049.cjs tests/scripts/test-migration-050.cjs tests/scripts/test-migration-051.cjs tests/scripts/test-migration-052.cjs tests/scripts/test-engine-specification-projections.cjs tests/scripts/test-engine-discovery-projections.cjs tests/scripts/test-gateway-for-start.cjs tests/scripts/test-gateway-for-specification.cjs tests/scripts/test-gateway-for-workflow-continue-epic.cjs tests/scripts/test-gateway-for-workflow-continue-feature.cjs tests/scripts/test-gateway-for-workflow-continue-bugfix.cjs tests/scripts/test-gateway-for-workflow-continue-quickfix.cjs tests/scripts/test-gateway-for-workflow-continue-cross-cutting.cjs tests/scripts/test-gateway-for-discovery-skill.cjs tests/scripts/test-gateway-for-bridge.cjs tests/scripts/test-reads-derivations.cjs tests/scripts/test-ensure-discovery-item.cjs tests/scripts/test-legacy-research-split.cjs tests/scripts/test-knowledge-chunker.cjs tests/scripts/test-knowledge-config.cjs tests/scripts/test-knowledge-discovery.cjs tests/scripts/test-knowledge-embeddings.cjs tests/scripts/test-knowledge-integration.cjs tests/scripts/test-knowledge-openai.cjs tests/scripts/test-knowledge-openai-compatible.cjs tests/scripts/test-knowledge-openai-integration.cjs tests/scripts/test-knowledge-progress.cjs tests/scripts/test-knowledge-rerank.cjs tests/scripts/test-knowledge-retry.cjs tests/scripts/test-knowledge-setup-forms.cjs tests/scripts/test-knowledge-store.cjs tests/scripts/test-conventions-lint.cjs tests/scripts/test-pipeline-simulation.cjs tests/scripts/test-engine-agent-state.cjs", + "test": "node --test tests/scripts/test-render.cjs tests/scripts/test-engine-gateway.cjs tests/scripts/test-engine-epic-projections.cjs tests/scripts/test-engine-start-projections.cjs tests/scripts/test-engine-workunit-projections.cjs tests/scripts/test-engine-discussion-map.cjs tests/scripts/test-engine-discovery-map.cjs tests/scripts/test-engine-tasks.cjs tests/scripts/test-engine-render-surfaces.cjs tests/scripts/test-engine-transactions.cjs tests/scripts/test-engine-workunit-create.cjs tests/scripts/test-engine-workunit-pivot.cjs tests/scripts/test-engine-workunit-absorb.cjs tests/scripts/test-engine-workunit-promote.cjs tests/scripts/test-engine-discovery-session.cjs tests/scripts/test-engine-cache.cjs tests/scripts/test-engine-manifest-io.cjs tests/scripts/test-engine-manifest-fields.cjs tests/scripts/test-engine-boot.cjs tests/scripts/test-migration-orchestrator.cjs tests/scripts/test-migration-047.cjs tests/scripts/test-migration-048.cjs tests/scripts/test-migration-049.cjs tests/scripts/test-migration-050.cjs tests/scripts/test-migration-051.cjs tests/scripts/test-migration-052.cjs tests/scripts/test-engine-specification-projections.cjs tests/scripts/test-engine-discovery-projections.cjs tests/scripts/test-gateway-for-start.cjs tests/scripts/test-gateway-for-specification.cjs tests/scripts/test-gateway-for-workflow-continue-epic.cjs tests/scripts/test-gateway-for-workflow-continue-feature.cjs tests/scripts/test-gateway-for-workflow-continue-bugfix.cjs tests/scripts/test-gateway-for-workflow-continue-quickfix.cjs tests/scripts/test-gateway-for-workflow-continue-cross-cutting.cjs tests/scripts/test-gateway-for-discovery-skill.cjs tests/scripts/test-gateway-for-bridge.cjs tests/scripts/test-reads-derivations.cjs tests/scripts/test-ensure-discovery-item.cjs tests/scripts/test-legacy-research-split.cjs tests/scripts/test-knowledge-chunker.cjs tests/scripts/test-knowledge-config.cjs tests/scripts/test-knowledge-discovery.cjs tests/scripts/test-knowledge-embeddings.cjs tests/scripts/test-knowledge-integration.cjs tests/scripts/test-knowledge-openai.cjs tests/scripts/test-knowledge-openai-compatible.cjs tests/scripts/test-knowledge-openai-integration.cjs tests/scripts/test-knowledge-progress.cjs tests/scripts/test-knowledge-rerank.cjs tests/scripts/test-knowledge-retry.cjs tests/scripts/test-knowledge-setup-forms.cjs tests/scripts/test-knowledge-store.cjs tests/scripts/test-conventions-lint.cjs tests/scripts/test-pipeline-simulation.cjs tests/scripts/test-engine-agent-state.cjs tests/scripts/test-prose-cases.cjs tests/scripts/test-prose-fixtures.cjs", "test:migrations": "for t in tests/scripts/test-migration-*.sh; do bash $t || exit 1; done", "test:cli": "for t in tests/scripts/test-engine-manifest.sh tests/scripts/test-inbox-promotion.sh tests/scripts/test-knowledge-cli.sh tests/scripts/test-knowledge-build.sh; do bash $t || exit 1; done" }, diff --git a/tests/prose/README.md b/tests/prose/README.md new file mode 100644 index 000000000..263c32e94 --- /dev/null +++ b/tests/prose/README.md @@ -0,0 +1,34 @@ +# Prose tests + +End-to-end tests for the prose logic — natural-language cases a walker +agent executes through the real skills against a materialised fixture +world. Design and contract: [design/prose-tests.md](../../design/prose-tests.md). +Run via the `/prose-test` skill; everything deterministic lives in +`run.cjs` and `lib/`. + +## Layout + +- `{flow}/*.md` — case files (grammar: header comment in `lib/cases.cjs`) +- `fixtures/{name}/recipe.cjs` — builds the world with real engine calls + under a frozen clock (`lib/fake-clock.cjs`) +- `fixtures/{name}/snapshot/` — the recipe's committed golden output; + regenerate with `node tests/prose/run.cjs snap {name}`, never hand-edit +- `run.cjs` — `list · select · world · prompt · grade · snap · verify · destroy` + +## Adding a case + +Add a `## case:` block to the flow's file (or a new file for a long +walk). Scope `files:` tightly — diff-selection and the PR-end suggestion +run off it. `routing:` expects are agent-graded against the walk +transcript; `state:` expects are asserted in code and require a `world:`. +`npm test` validates the corpus (parse, paths, anchors, grammar) and +rebuild-compares every fixture — both token-free; only `/prose-test` +spends tokens. + +## Snapshot escaping + +Snapshots exclude `.git/` and `.workflows/.knowledge/` (the world +builder re-derives the store), and store `.gitignore` files as +`_gitignore.fixture` so the product-written `.workflows/.gitignore` +cannot ignore fixture content out of this repo; the world builder +restores them. diff --git a/tests/prose/fixtures/base/recipe.cjs b/tests/prose/fixtures/base/recipe.cjs new file mode 100644 index 000000000..06ac01e21 --- /dev/null +++ b/tests/prose/fixtures/base/recipe.cjs @@ -0,0 +1,13 @@ +'use strict'; + +// The canonical empty project — a real install straight after +// /workflow-start Step 0: migrations applied (the log is the durable +// trace), knowledge set up keyword-only, boot run. No work units yet. +// Every richer fixture starts from this world's shape. + +module.exports = { + build(h) { + h.knowledge('setup', '--keyword-only'); + h.engine('boot'); + }, +}; diff --git a/tests/prose/fixtures/base/snapshot/.claude/settings.json b/tests/prose/fixtures/base/snapshot/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/fixtures/base/snapshot/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/fixtures/base/snapshot/.workflows/.state/migrations b/tests/prose/fixtures/base/snapshot/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/fixtures/base/snapshot/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/fixtures/base/snapshot/.workflows/_gitignore.fixture b/tests/prose/fixtures/base/snapshot/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/fixtures/base/snapshot/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/lib/cases.cjs b/tests/prose/lib/cases.cjs new file mode 100644 index 000000000..5e065aba8 --- /dev/null +++ b/tests/prose/lib/cases.cjs @@ -0,0 +1,199 @@ +'use strict'; + +// Case corpus: parsing and validation. +// +// A case file is markdown under tests/prose/{flow}/ (any name, .md). +// Grammar, deterministic to parse: +// +// ## case: {kebab-id} +// - world: {fixture-name} (optional; required by state expects) +// - origin: {free text} +// - files: +// - {repo-relative path}[#Heading text] +// ### walk +// {free text: entry point, what to do, stop condition} +// ### user +// 1. {scripted answer, consumed in order} +// ### expect +// - routing: {claim graded by an agent against the walk transcript} +// - state: {deterministic assertion — see parseStateAssertion} +// +// State assertion grammar (executed by the runner, zero model judgment): +// file exists {rel} +// file absent {rel} +// manifest exists {dotpath} {field} +// manifest absent {dotpath} {field} +// manifest equals {dotpath} {field} {expected printed value} + +const fs = require('fs'); +const path = require('path'); + +const ROOT = path.join(__dirname, '../../..'); +const PROSE_DIR = path.join(ROOT, 'tests/prose'); +const FIXTURES_DIR = path.join(PROSE_DIR, 'fixtures'); +const RESERVED_DIRS = new Set(['lib', 'fixtures']); + +function listCaseFiles() { + if (!fs.existsSync(PROSE_DIR)) return []; + const files = []; + for (const flow of fs.readdirSync(PROSE_DIR, { withFileTypes: true })) { + if (!flow.isDirectory() || RESERVED_DIRS.has(flow.name)) continue; + const flowDir = path.join(PROSE_DIR, flow.name); + for (const entry of fs.readdirSync(flowDir)) { + if (entry.endsWith('.md') && entry !== 'README.md') { + files.push({ flow: flow.name, file: path.join(flowDir, entry) }); + } + } + } + return files; +} + +function parseStateAssertion(text) { + const fileMatch = text.match(/^file (exists|absent) (\S+)$/); + if (fileMatch) { + return { kind: `file-${fileMatch[1]}`, path: fileMatch[2] }; + } + const mExists = text.match(/^manifest (exists|absent) (\S+) (\S+)$/); + if (mExists) { + return { kind: `manifest-${mExists[1]}`, dotpath: mExists[2], field: mExists[3] }; + } + const mEquals = text.match(/^manifest equals (\S+) (\S+) (.+)$/); + if (mEquals) { + return { kind: 'manifest-equals', dotpath: mEquals[1], field: mEquals[2], value: mEquals[3] }; + } + return { error: `unparseable state assertion: "${text}"` }; +} + +function parseCaseFile(file, flow) { + const lines = fs.readFileSync(file, 'utf8').split('\n'); + const cases = []; + let current = null; + let section = null; // null | 'walk' | 'user' | 'expect' + let inFiles = false; + + const push = () => { + if (current) { + current.walk = current.walk.join('\n').trim(); + cases.push(current); + } + }; + + for (const line of lines) { + const caseStart = line.match(/^## case:\s*(\S+)\s*$/); + if (caseStart) { + push(); + current = { + id: caseStart[1], flow, file: path.relative(ROOT, file), + world: null, origin: null, files: [], + walk: [], user: [], expect: [], + }; + section = null; + inFiles = false; + continue; + } + if (!current) continue; + + const sub = line.match(/^### (walk|user|expect)\s*$/); + if (sub) { + section = sub[1]; + inFiles = false; + continue; + } + if (/^#/.test(line)) { // any other heading ends the current case + push(); + current = null; + continue; + } + + if (section === null) { + const field = line.match(/^- (world|origin):\s*(.*)$/); + if (field) { + current[field[1]] = field[2].trim(); + inFiles = false; + continue; + } + if (/^- files:\s*$/.test(line)) { + inFiles = true; + continue; + } + const fileEntry = line.match(/^\s+- (.+)$/); + if (inFiles && fileEntry) { + const [, spec] = fileEntry; + const hash = spec.indexOf('#'); + current.files.push(hash === -1 + ? { path: spec.trim(), anchor: null } + : { path: spec.slice(0, hash).trim(), anchor: spec.slice(hash + 1).trim() }); + continue; + } + if (line.trim() !== '') inFiles = false; + } else if (section === 'walk') { + current.walk.push(line); + } else if (section === 'user') { + const answer = line.match(/^\d+\.\s+(.*)$/); + if (answer) current.user.push(answer[1].trim()); + } else if (section === 'expect') { + const claim = line.match(/^- (routing|state):\s*(.*)$/); + if (claim) current.expect.push({ kind: claim[1], text: claim[2].trim() }); + } + } + push(); + return cases; +} + +function loadAllCases() { + return listCaseFiles().flatMap(({ flow, file }) => parseCaseFile(file, flow)); +} + +function headingExists(absPath, anchor) { + const content = fs.readFileSync(absPath, 'utf8'); + return content.split('\n').some((line) => { + const h = line.match(/^#{1,6}\s+(.*?)\s*$/); + return h && h[1] === anchor; + }); +} + +function validateCorpus(cases) { + const errors = []; + const seen = new Set(); + for (const c of cases) { + const at = `${c.file} :: ${c.id}`; + if (!/^[a-z0-9][a-z0-9-]*$/.test(c.id)) errors.push(`${at}: id is not kebab-case`); + if (seen.has(c.id)) errors.push(`${at}: duplicate case id`); + seen.add(c.id); + + if (c.files.length === 0) errors.push(`${at}: no files: scope`); + for (const f of c.files) { + const abs = path.join(ROOT, f.path); + if (!fs.existsSync(abs)) { + errors.push(`${at}: scoped file missing: ${f.path}`); + } else if (f.anchor && !headingExists(abs, f.anchor)) { + errors.push(`${at}: anchor "#${f.anchor}" not a heading in ${f.path}`); + } + } + + if (c.world !== null) { + if (!fs.existsSync(path.join(FIXTURES_DIR, c.world, 'recipe.cjs'))) { + errors.push(`${at}: world "${c.world}" has no fixture recipe`); + } + } + + if (!c.walk) errors.push(`${at}: empty walk`); + if (c.expect.length === 0) errors.push(`${at}: no expects`); + for (const e of c.expect) { + if (!e.text) errors.push(`${at}: empty ${e.kind} expect`); + if (e.kind === 'state') { + if (c.world === null) { + errors.push(`${at}: state expect requires a world`); + } + const parsed = parseStateAssertion(e.text); + if (parsed.error) errors.push(`${at}: ${parsed.error}`); + } + } + } + return errors; +} + +module.exports = { + ROOT, PROSE_DIR, FIXTURES_DIR, + listCaseFiles, parseCaseFile, loadAllCases, parseStateAssertion, validateCorpus, +}; diff --git a/tests/prose/lib/fake-clock.cjs b/tests/prose/lib/fake-clock.cjs new file mode 100644 index 000000000..ab1391536 --- /dev/null +++ b/tests/prose/lib/fake-clock.cjs @@ -0,0 +1,30 @@ +'use strict'; + +// Frozen clock preload for fixture recipes (injected via NODE_OPTIONS +// --require). Recipes must be byte-deterministic — the same recipe always +// builds the same snapshot — but the engine stamps real times (work-unit +// `created`, agent-state rows, cache payload headers). Pinning Date in +// every recipe subprocess removes the only nondeterministic input. +// +// Worlds materialised for live walks do NOT use this: walker runs are +// real sessions and never byte-compared. + +const FIXED_MS = 1767225600000; // 2026-01-01T00:00:00.000Z + +const RealDate = Date; + +class FrozenDate extends RealDate { + constructor(...args) { + if (args.length === 0) { + super(FIXED_MS); + } else { + super(...args); + } + } + + static now() { + return FIXED_MS; + } +} + +global.Date = FrozenDate; diff --git a/tests/prose/lib/fixtures.cjs b/tests/prose/lib/fixtures.cjs new file mode 100644 index 000000000..9c294cdc5 --- /dev/null +++ b/tests/prose/lib/fixtures.cjs @@ -0,0 +1,196 @@ +'use strict'; + +// Fixture recipes and golden snapshots. +// +// A fixture is tests/prose/fixtures/{name}/: a recipe.cjs (a module whose +// build(harness) drives real engine/knowledge calls against a scratch +// project) and a committed snapshot/ (the recipe's byte-exact output). +// The recipe is the source of truth; the snapshot is what makes drift +// visible — CI rebuilds every recipe and byte-compares (P2/P3 in +// design/prose-tests.md). Hand-editing a snapshot is forbidden: +// regenerate with `node tests/prose/run.cjs snap {name}`. +// +// Determinism: recipe subprocesses run under the frozen-clock preload and +// pinned git identity/dates. Snapshots exclude `.git/` (SHAs live in +// manifest values where recipes record them, not as a tree) and +// `.workflows/.knowledge/` (binary store; the world builder re-derives it +// at materialise time). `.gitignore` files inside a snapshot are stored +// escaped so the product-written `.workflows/.gitignore` cannot ignore +// fixture content out of THIS repo's git. + +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { execFileSync, spawnSync } = require('child_process'); + +const ROOT = path.join(__dirname, '../../..'); +const ENGINE = path.join(ROOT, 'skills/workflow-engine/scripts/engine.cjs'); +const KNOWLEDGE = path.join(ROOT, 'skills/workflow-knowledge/scripts/knowledge.cjs'); +const FIXTURES_DIR = path.join(ROOT, 'tests/prose/fixtures'); +const CLOCK = path.join(__dirname, 'fake-clock.cjs'); + +const GITIGNORE = '.gitignore'; +const GITIGNORE_ESCAPED = '_gitignore.fixture'; + +function recipeEnv() { + if (/\s/.test(CLOCK)) { + throw new Error(`fake-clock path contains whitespace — NODE_OPTIONS cannot carry it: ${CLOCK}`); + } + return { + ...process.env, + NODE_OPTIONS: [process.env.NODE_OPTIONS, `--require ${CLOCK}`].filter(Boolean).join(' '), + GIT_CONFIG_GLOBAL: '/dev/null', + GIT_CONFIG_SYSTEM: '/dev/null', + GIT_AUTHOR_DATE: '2026-01-01T00:00:00Z', + GIT_COMMITTER_DATE: '2026-01-01T00:00:00Z', + TZ: 'UTC', + }; +} + +function makeHarness(dir) { + const env = recipeEnv(); + const node = (script, args) => { + const res = spawnSync('node', [script, ...args], { cwd: dir, encoding: 'utf8', env }); + if (res.status !== 0) { + throw new Error(`recipe call failed: ${path.basename(script)} ${args.join(' ')}\n` + + `stdout: ${res.stdout}\nstderr: ${res.stderr}`); + } + return res.stdout; + }; + return { + dir, + engine: (...args) => node(ENGINE, args), + knowledge: (...args) => node(KNOWLEDGE, args), + git: (...args) => execFileSync('git', args, { cwd: dir, encoding: 'utf8', env }), + write(rel, content) { + const full = path.join(dir, rel); + fs.mkdirSync(path.dirname(full), { recursive: true }); + fs.writeFileSync(full, content); + }, + remove(rel) { + fs.rmSync(path.join(dir, rel), { recursive: true, force: true }); + }, + }; +} + +function listFixtures() { + if (!fs.existsSync(FIXTURES_DIR)) return []; + return fs.readdirSync(FIXTURES_DIR, { withFileTypes: true }) + .filter((e) => e.isDirectory() && fs.existsSync(path.join(FIXTURES_DIR, e.name, 'recipe.cjs'))) + .map((e) => e.name); +} + +/** Build a fixture's world into a fresh scratch dir; caller removes it. */ +function runRecipe(name) { + const recipePath = path.join(FIXTURES_DIR, name, 'recipe.cjs'); + if (!fs.existsSync(recipePath)) throw new Error(`no recipe for fixture "${name}"`); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), `prose-recipe-${name}-`)); + const h = makeHarness(dir); + h.git('init', '-q', '-b', 'main'); + h.git('config', 'user.email', 'prose@example.com'); + h.git('config', 'user.name', 'Prose Fixture'); + h.git('config', 'commit.gpgsign', 'false'); + fs.mkdirSync(path.join(dir, '.workflows'), { recursive: true }); + delete require.cache[require.resolve(recipePath)]; + require(recipePath).build(h); + return dir; +} + +function excluded(rel) { + const parts = rel.split(path.sep); + if (parts.includes('.git')) return true; + if (rel === path.join('.workflows', '.knowledge')) return true; + if (rel.startsWith(path.join('.workflows', '.knowledge') + path.sep)) return true; + return false; +} + +/** Scratch tree → Map(snapshot-relative path → Buffer), escaped + filtered. */ +function collectTree(root) { + const files = new Map(); + const walk = (rel) => { + const abs = path.join(root, rel); + for (const entry of fs.readdirSync(abs, { withFileTypes: true })) { + const childRel = rel === '' ? entry.name : path.join(rel, entry.name); + if (excluded(childRel)) continue; + if (entry.isDirectory()) { + walk(childRel); + } else if (entry.isFile()) { + const stored = entry.name === GITIGNORE + ? path.join(path.dirname(childRel), GITIGNORE_ESCAPED) + : childRel; + files.set(stored, fs.readFileSync(path.join(root, childRel))); + } + } + }; + walk(''); + return files; +} + +/** Committed snapshot dir → Map(snapshot-relative path → Buffer), verbatim. */ +function readSnapshot(name) { + const snapDir = path.join(FIXTURES_DIR, name, 'snapshot'); + if (!fs.existsSync(snapDir)) return null; + const files = new Map(); + const walk = (rel) => { + for (const entry of fs.readdirSync(path.join(snapDir, rel), { withFileTypes: true })) { + const childRel = rel === '' ? entry.name : path.join(rel, entry.name); + if (entry.isDirectory()) walk(childRel); + else if (entry.isFile()) files.set(childRel, fs.readFileSync(path.join(snapDir, childRel))); + } + }; + walk(''); + return files; +} + +/** Regenerate the committed snapshot from a built scratch tree. */ +function writeSnapshot(name, scratchDir) { + const snapDir = path.join(FIXTURES_DIR, name, 'snapshot'); + fs.rmSync(snapDir, { recursive: true, force: true }); + const files = collectTree(scratchDir); + for (const [rel, buf] of files) { + const dest = path.join(snapDir, rel); + fs.mkdirSync(path.dirname(dest), { recursive: true }); + fs.writeFileSync(dest, buf); + } + return files.size; +} + +/** Rebuild-compare: {missing, extra, changed} — all empty means current. */ +function compareSnapshot(name, scratchDir) { + const snap = readSnapshot(name); + if (snap === null) return { missing: [''], extra: [], changed: [] }; + const built = collectTree(scratchDir); + const missing = []; + const extra = []; + const changed = []; + for (const [rel, buf] of built) { + if (!snap.has(rel)) extra.push(rel); + else if (!snap.get(rel).equals(buf)) changed.push(rel); + } + for (const rel of snap.keys()) { + if (!built.has(rel)) missing.push(rel); + } + return { missing, extra, changed }; +} + +/** Snapshot → real project tree at dest, unescaping .gitignore files. */ +function materialiseSnapshot(name, destDir) { + const snap = readSnapshot(name); + if (snap === null) { + throw new Error(`fixture "${name}" has no committed snapshot — run: node tests/prose/run.cjs snap ${name}`); + } + for (const [rel, buf] of snap) { + const real = path.basename(rel) === GITIGNORE_ESCAPED + ? path.join(path.dirname(rel), GITIGNORE) + : rel; + const dest = path.join(destDir, real); + fs.mkdirSync(path.dirname(dest), { recursive: true }); + fs.writeFileSync(dest, buf); + } +} + +module.exports = { + ROOT, ENGINE, KNOWLEDGE, FIXTURES_DIR, + listFixtures, runRecipe, collectTree, readSnapshot, + writeSnapshot, compareSnapshot, materialiseSnapshot, +}; diff --git a/tests/prose/lib/world.cjs b/tests/prose/lib/world.cjs new file mode 100644 index 000000000..f3dc2e7b6 --- /dev/null +++ b/tests/prose/lib/world.cjs @@ -0,0 +1,75 @@ +'use strict'; + +// World builder — materialise a fixture into a live, parallel-safe fake +// project a walker agent can work in as if it were a real install: +// +// 1. copy the fixture snapshot in (unescaping .gitignore files), +// 2. copy the repo's current skills/ and agents/ to the installed +// layout (.claude/skills, .claude/agents) so prose paths resolve, +// 3. git init + commit (hermetic identity — no user config leaks), +// 4. `knowledge setup --keyword-only` — re-derives the keyword index +// from the snapshot's artifacts (the binary store is never +// snapshotted), leaving every entry skill's knowledge gate green. +// +// Worlds run on the REAL clock: walker runs are live sessions, never +// byte-compared. Mutations during a walk land here and nowhere else. + +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { execFileSync, spawnSync } = require('child_process'); + +const fixtures = require('./fixtures.cjs'); + +const WORLD_PREFIX = 'prose-world-'; + +function worldEnv() { + return { + ...process.env, + GIT_CONFIG_GLOBAL: '/dev/null', + GIT_CONFIG_SYSTEM: '/dev/null', + }; +} + +function buildWorld(fixtureName) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), WORLD_PREFIX)); + const env = worldEnv(); + const git = (...args) => execFileSync('git', args, { cwd: dir, encoding: 'utf8', env }); + + fixtures.materialiseSnapshot(fixtureName, dir); + + for (const layer of ['skills', 'agents']) { + const src = path.join(fixtures.ROOT, layer); + if (fs.existsSync(src)) { + fs.cpSync(src, path.join(dir, '.claude', layer), { recursive: true }); + } + } + + git('init', '-q', '-b', 'main'); + git('config', 'user.email', 'prose@example.com'); + git('config', 'user.name', 'Prose World'); + git('config', 'commit.gpgsign', 'false'); + git('add', '-A'); + git('commit', '-q', '-m', `world: ${fixtureName}`); + + const knowledge = path.join(dir, '.claude/skills/workflow-knowledge/scripts/knowledge.cjs'); + const setup = spawnSync('node', [knowledge, 'setup', '--keyword-only'], + { cwd: dir, encoding: 'utf8', env }); + if (setup.status !== 0) { + fs.rmSync(dir, { recursive: true, force: true }); + throw new Error(`knowledge setup failed in world:\nstdout: ${setup.stdout}\nstderr: ${setup.stderr}`); + } + git('add', '-A'); + git('commit', '-q', '-m', 'chore(knowledge): initialise store'); + + return dir; +} + +function destroyWorld(dir) { + if (!path.basename(dir).startsWith(WORLD_PREFIX)) { + throw new Error(`refusing to remove non-world directory: ${dir}`); + } + fs.rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); +} + +module.exports = { buildWorld, destroyWorld, WORLD_PREFIX }; diff --git a/tests/prose/run.cjs b/tests/prose/run.cjs new file mode 100644 index 000000000..32c8bba2e --- /dev/null +++ b/tests/prose/run.cjs @@ -0,0 +1,284 @@ +#!/usr/bin/env node +'use strict'; + +// Prose-test runner — everything deterministic about a prose-test run. +// The /prose-test skill drives this CLI and supplies the model layer +// (walker and grader agents). See design/prose-tests.md. +// +// list corpus table +// select [--diff |--all|--cases a,b] cases to run, as JSON +// world materialise the case's world, print path +// prompt [--world ] walker prompt (NEVER contains expects) +// grade [--world ] run state assertions; emit routing +// claims for the grader agent +// snap (re)generate a fixture's golden snapshot +// verify [fixture] rebuild-compare fixture(s) against snapshot +// destroy --world remove a world + +const fs = require('fs'); +const path = require('path'); +const { spawnSync, execFileSync } = require('child_process'); + +const cases = require('./lib/cases.cjs'); +const fixtures = require('./lib/fixtures.cjs'); +const world = require('./lib/world.cjs'); + +const ROOT = cases.ROOT; + +function die(msg) { + process.stderr.write(`${msg}\n`); + process.exit(1); +} + +function loadCase(id) { + const all = cases.loadAllCases(); + const found = all.find((c) => c.id === id); + if (!found) die(`no case "${id}" in the corpus`); + return found; +} + +function flag(argv, name) { + const i = argv.indexOf(name); + if (i === -1) return null; + if (i + 1 >= argv.length) die(`${name} requires a value`); + return argv[i + 1]; +} + +// --- select --------------------------------------------------------------- + +function changedFiles(ref) { + const run = (args) => execFileSync('git', args, { cwd: ROOT, encoding: 'utf8' }) + .split('\n').map((l) => l.trim()).filter(Boolean); + const diff = run(['diff', '--name-only', ref]); + const status = run(['status', '--porcelain']).map((l) => l.slice(3).trim()); + return new Set([...diff, ...status]); +} + +function cmdSelect(argv) { + const all = cases.loadAllCases(); + let selected; + let mode; + if (argv.includes('--all')) { + mode = 'all'; + selected = all; + } else if (flag(argv, '--cases')) { + mode = 'ids'; + const ids = flag(argv, '--cases').split(',').map((s) => s.trim()); + const byId = new Map(all.map((c) => [c.id, c])); + for (const id of ids) if (!byId.has(id)) die(`no case "${id}" in the corpus`); + selected = ids.map((id) => byId.get(id)); + } else { + const ref = flag(argv, '--diff') || 'main'; + mode = `diff:${ref}`; + const changed = changedFiles(ref); + const touches = (c) => { + if (changed.has(c.file)) return true; + if (c.files.some((f) => changed.has(f.path))) return true; + if (c.world !== null) { + const prefix = path.posix.join('tests/prose/fixtures', c.world) + '/'; + for (const p of changed) if (p.startsWith(prefix)) return true; + } + return false; + }; + selected = all.filter(touches); + } + process.stdout.write(JSON.stringify({ + mode, + cases: selected.map((c) => ({ id: c.id, flow: c.flow, file: c.file, world: c.world })), + }, null, 2) + '\n'); +} + +// --- world / destroy ------------------------------------------------------ + +function cmdWorld(argv) { + const c = loadCase(argv[0] || die('usage: world ')); + if (c.world === null) { + process.stdout.write(JSON.stringify({ world: null, note: 'structure-only case — no world' }) + '\n'); + return; + } + const dir = world.buildWorld(c.world); + process.stdout.write(JSON.stringify({ world: dir, fixture: c.world }) + '\n'); +} + +function cmdDestroy(argv) { + const dir = flag(argv, '--world') || die('usage: destroy --world '); + world.destroyWorld(dir); + process.stdout.write(`destroyed ${dir}\n`); +} + +// --- prompt --------------------------------------------------------------- + +function cmdPrompt(argv) { + const c = loadCase(argv[0] || die('usage: prompt [--world ]')); + const worldDir = flag(argv, '--world'); + if (c.world !== null && !worldDir) die(`case "${c.id}" needs --world (build one: run.cjs world ${c.id})`); + + const userScript = c.user.length + ? c.user.map((a, i) => ` ${i + 1}. ${a}`).join('\n') + : ' (none — the walk should reach its stop condition without user questions)'; + + const setting = c.world !== null + ? [ + 'You are executing this project\'s workflow prose exactly as a live session would.', + '', + `Project directory — your cwd for EVERY command: ${worldDir}`, + 'The workflow skills are installed at .claude/skills/ inside that project.', + 'Mutations are expected and safe: the project is a disposable test world.', + ] + : [ + 'You are walking workflow prose structurally: read the named files and trace the logic.', + 'Execute nothing — no commands, no writes. This is a read-only walk of the repository.', + '', + `Repository root: ${ROOT}`, + ]; + + const prompt = [ + ...setting, + '', + 'TASK', + c.walk, + '', + 'SCOPE — the prose under walk:', + ...c.files.map((f) => ` - ${f.path}${f.anchor ? ` (start at "${f.anchor}")` : ''}`), + '', + 'RULES', + '- Follow the prose literally, step by step, arm by arm. Where it names an', + ' engine or knowledge call, run it (node .claude/skills/workflow-engine/', + ' scripts/engine.cjs …) from the project directory and use the real response.', + '- You also play the user, from a fixed script. When the prose presents a', + ' menu or question, consume the next scripted answer, in order:', + userScript, + '- If the prose asks a question and the script has no next answer: STOP and', + ' record `UNSCRIPTED QUESTION:` with the exact question text.', + '- If two arms both appear to match, record `AMBIGUOUS:` naming both, then', + ' follow the one the prose\'s own ordering/guard rules select.', + '- Never repair, reinterpret, or improve the prose. Execute what is written,', + ' even where it looks wrong. You are a probe, not a reviewer.', + '- Stop at the TASK\'s stop condition, the end of the flow, an UNSCRIPTED', + ' QUESTION, or a hard error — whichever comes first.', + '', + 'TRANSCRIPT — your entire final output, in order of events:', + '1. Every prose section/arm entered: `file.md § Heading` plus the quoted', + ' guard line that selected it.', + '2. Every command run and the first line of its output.', + '3. Every menu/question encountered (verbatim) and the scripted answer used.', + '4. Every file written or edited (path only).', + '5. Finally: `STOPPED: `.', + 'Return nothing but the transcript.', + ].join('\n'); + + process.stdout.write(prompt + '\n'); +} + +// --- grade ---------------------------------------------------------------- + +function engineInWorld(worldDir, args) { + const engine = path.join(worldDir, '.claude/skills/workflow-engine/scripts/engine.cjs'); + return spawnSync('node', [engine, ...args], { cwd: worldDir, encoding: 'utf8' }); +} + +function runStateAssertion(worldDir, assertion) { + const a = cases.parseStateAssertion(assertion); + if (a.error) return { assertion, pass: false, actual: a.error }; + if (a.kind === 'file-exists' || a.kind === 'file-absent') { + const present = fs.existsSync(path.join(worldDir, a.path)); + const want = a.kind === 'file-exists'; + return { assertion, pass: present === want, actual: present ? 'present' : 'absent' }; + } + if (a.kind === 'manifest-exists' || a.kind === 'manifest-absent') { + const res = engineInWorld(worldDir, ['manifest', 'exists', a.dotpath, a.field]); + if (res.status !== 0) return { assertion, pass: false, actual: `engine refused: ${res.stderr.trim()}` }; + const present = res.stdout.trim() === 'true'; + const want = a.kind === 'manifest-exists'; + return { assertion, pass: present === want, actual: present ? 'present' : 'absent' }; + } + // manifest-equals + const res = engineInWorld(worldDir, ['manifest', 'get', a.dotpath, a.field]); + if (res.status !== 0) return { assertion, pass: false, actual: `engine refused: ${res.stderr.trim()}` }; + const actual = res.stdout.trim(); + return { assertion, pass: actual === a.value, actual }; +} + +function cmdGrade(argv) { + const c = loadCase(argv[0] || die('usage: grade [--world ]')); + const worldDir = flag(argv, '--world'); + const stateExpects = c.expect.filter((e) => e.kind === 'state'); + if (stateExpects.length > 0 && !worldDir) die(`case "${c.id}" has state expects — --world required`); + + const state = stateExpects.map((e) => runStateAssertion(worldDir, e.text)); + const routing = c.expect.filter((e) => e.kind === 'routing').map((e) => e.text); + process.stdout.write(JSON.stringify({ + id: c.id, + state, + statePass: state.every((s) => s.pass), + routing, + note: routing.length + ? 'routing claims are graded by an agent against the walk transcript — PASS requires quoted evidence' + : 'no routing claims', + }, null, 2) + '\n'); +} + +// --- snap / verify -------------------------------------------------------- + +function cmdSnap(argv) { + const name = argv[0] || die('usage: snap '); + const scratch = fixtures.runRecipe(name); + try { + const count = fixtures.writeSnapshot(name, scratch); + process.stdout.write(`snapshot ${name}: ${count} files written — review the diff before committing\n`); + } finally { + fs.rmSync(scratch, { recursive: true, force: true }); + } +} + +function cmdVerify(argv) { + const names = argv[0] ? [argv[0]] : fixtures.listFixtures(); + let failed = false; + for (const name of names) { + const scratch = fixtures.runRecipe(name); + try { + const diff = fixtures.compareSnapshot(name, scratch); + const clean = !diff.missing.length && !diff.extra.length && !diff.changed.length; + if (clean) { + process.stdout.write(`${name}: snapshot current\n`); + } else { + failed = true; + process.stdout.write(`${name}: DRIFT — the recipe no longer rebuilds the snapshot\n`); + for (const f of diff.changed) process.stdout.write(` changed: ${f}\n`); + for (const f of diff.extra) process.stdout.write(` extra (rebuilt, not in snapshot): ${f}\n`); + for (const f of diff.missing) process.stdout.write(` missing (in snapshot, not rebuilt): ${f}\n`); + process.stdout.write(` regenerate: node tests/prose/run.cjs snap ${name}\n`); + } + } finally { + fs.rmSync(scratch, { recursive: true, force: true }); + } + } + process.exit(failed ? 1 : 0); +} + +// --- list ----------------------------------------------------------------- + +function cmdList() { + const all = cases.loadAllCases(); + const errors = cases.validateCorpus(all); + for (const c of all) { + const expects = `${c.expect.filter((e) => e.kind === 'routing').length}r/${c.expect.filter((e) => e.kind === 'state').length}s`; + process.stdout.write(`${c.id} [${c.flow}] world=${c.world || '-'} expects=${expects}\n`); + } + process.stdout.write(`\n${all.length} cases, ${fixtures.listFixtures().length} fixtures`); + process.stdout.write(errors.length ? `, ${errors.length} VALIDATION ERRORS:\n` : ', corpus valid\n'); + for (const e of errors) process.stdout.write(` - ${e}\n`); + process.exit(errors.length ? 1 : 0); +} + +// --- dispatch ------------------------------------------------------------- + +const [, , command, ...rest] = process.argv; +const commands = { + list: cmdList, select: cmdSelect, world: cmdWorld, prompt: cmdPrompt, + grade: cmdGrade, snap: cmdSnap, verify: cmdVerify, destroy: cmdDestroy, +}; +if (!commands[command]) { + die('usage: run.cjs …'); +} +commands[command](rest); diff --git a/tests/prose/smoke/cases.md b/tests/prose/smoke/cases.md new file mode 100644 index 000000000..1c28097dc --- /dev/null +++ b/tests/prose/smoke/cases.md @@ -0,0 +1,43 @@ +# Smoke cases — the framework proving itself + +Not corpus: these two cases exercise every moving part of the machinery +(parser, anchor checks, world builder, engine-in-world, state assertions, +walker protocol) against prose stable enough to survive. If a smoke case +fails, suspect the framework before the prose. + +## case: smoke-structure-start-boot +- origin: framework smoke — structure-only walk, no world +- files: + - skills/workflow-start/SKILL.md#Step 0.2: Boot + +### walk + +Structure-only: read skills/workflow-start/SKILL.md's Step 0 and its +sub-steps. Establish the order of Step 0's parts and what Step 0.2 +declares about the boot pipeline. Stop after Step 0 — do not trace the +rest of the skill. + +### expect + +- routing: Step 0.1 (casing conventions) is prescribed before Step 0.2 (boot) +- routing: Step 0.2 declares the boot pipeline mandatory — it must complete before proceeding + +## case: smoke-world-boot +- world: base +- origin: framework smoke — world build, engine-in-world, state assertions +- files: + - skills/workflow-start/SKILL.md#Step 0.2: Boot + +### walk + +In the project, run the boot command exactly as Step 0.2 of +skills/workflow-start/SKILL.md prescribes (sandbox concerns do not apply +in this world — run it directly). Capture the JSON response. Stop +immediately after recording it — do not continue into Step 0's +confirmation prompt or any later step. + +### expect + +- routing: the boot response reports the knowledge base ready (keyword-only mode) +- routing: the boot response reports no migrations applied on this run (the world is already migrated) +- state: file exists .workflows/.state/migrations diff --git a/tests/scripts/test-prose-cases.cjs b/tests/scripts/test-prose-cases.cjs new file mode 100644 index 000000000..1cb83129b --- /dev/null +++ b/tests/scripts/test-prose-cases.cjs @@ -0,0 +1,30 @@ +'use strict'; + +// Prose-test corpus: the deterministic perimeter (design/prose-tests.md +// P3). Every case parses, every scoped file and anchor resolves, every +// referenced world has a recipe, every state assertion is grammatical. +// The token-costing walks run on command via /prose-test — never here. + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); + +const cases = require('../prose/lib/cases.cjs'); + +describe('prose-test corpus', () => { + it('has at least one case', () => { + assert.ok(cases.loadAllCases().length > 0, 'corpus is empty'); + }); + + it('parses and validates cleanly', () => { + const errors = cases.validateCorpus(cases.loadAllCases()); + assert.deepStrictEqual(errors, [], + `corpus validation failed:\n${errors.map((e) => ` - ${e}`).join('\n')}`); + }); + + it('state assertion grammar rejects garbage', () => { + assert.ok(cases.parseStateAssertion('file exists .workflows/x').kind === 'file-exists'); + assert.ok(cases.parseStateAssertion('manifest equals wu.planning.t review_cycle 1').kind === 'manifest-equals'); + assert.ok(cases.parseStateAssertion('the manifest should look right').error); + assert.ok(cases.parseStateAssertion('file exists').error); + }); +}); diff --git a/tests/scripts/test-prose-fixtures.cjs b/tests/scripts/test-prose-fixtures.cjs new file mode 100644 index 000000000..c1c652e18 --- /dev/null +++ b/tests/scripts/test-prose-fixtures.cjs @@ -0,0 +1,43 @@ +'use strict'; + +// Fixture golden check (design/prose-tests.md P2/P3): every committed +// snapshot rebuilds byte-identical from its recipe under the frozen +// clock. A red run here means the engine moved the world — regenerate +// with `node tests/prose/run.cjs snap ` and review the snapshot +// diff in the same PR as the change that moved it. Never hand-edit a +// snapshot to green this. + +const { describe, it } = require('node:test'); +const assert = require('node:assert'); +const fs = require('fs'); + +const fixtures = require('../prose/lib/fixtures.cjs'); + +describe('prose-test fixtures', () => { + const names = fixtures.listFixtures(); + + it('has at least one fixture', () => { + assert.ok(names.length > 0, 'no fixtures found'); + }); + + for (const name of names) { + it(`${name}: recipe rebuilds the committed snapshot byte-identical`, () => { + assert.ok(fixtures.readSnapshot(name) !== null, + `fixture "${name}" has no committed snapshot — run: node tests/prose/run.cjs snap ${name}`); + const scratch = fixtures.runRecipe(name); + try { + const diff = fixtures.compareSnapshot(name, scratch); + const report = [ + ...diff.changed.map((f) => `changed: ${f}`), + ...diff.extra.map((f) => `extra (rebuilt, not in snapshot): ${f}`), + ...diff.missing.map((f) => `missing (in snapshot, not rebuilt): ${f}`), + ]; + assert.deepStrictEqual(report, [], + `world moved — regenerate: node tests/prose/run.cjs snap ${name}\n` + + report.map((l) => ` ${l}`).join('\n')); + } finally { + fs.rmSync(scratch, { recursive: true, force: true }); + } + }); + } +}); From 7aabce082bf548ae08198f8e6726f9de8656c0ea Mon Sep 17 00:00:00 2001 From: Lee Overy Date: Sat, 25 Jul 2026 10:01:50 +0100 Subject: [PATCH 2/2] =?UTF-8?q?fix(prose-tests):=20behaviour=20over=20coor?= =?UTF-8?q?dinates=20=E2=80=94=20de-brittle=20case=20authoring?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review finding (Lee): coordinate-phrased cases (step numbers, numbered heading anchors) break on cosmetic renumbering — failure for the wrong reason. Anchors become substring fragments (includes-matched, still red token-free when the heading vanishes), the smoke cases are rewritten purely behaviourally, and the authoring rule lands in the README. Prose test hooks considered and parked in the design log: the walker binds semantically, so shipped-prose plumbing isn't earning its keep — revisit on evidence at the failure harvest. Co-Authored-By: Claude Fable 5 --- design/prose-tests.md | 16 ++++++++++++++++ tests/prose/README.md | 8 ++++++++ tests/prose/lib/cases.cjs | 11 +++++++++-- tests/prose/run.cjs | 2 +- tests/prose/smoke/cases.md | 27 ++++++++++++++------------- 5 files changed, 48 insertions(+), 16 deletions(-) diff --git a/design/prose-tests.md b/design/prose-tests.md index 5a6ca14db..b314436d0 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -109,6 +109,22 @@ testing. ## Log +- 2026-07-25 — Brittleness review (Lee, on the stage-2 smoke cases): + cases written in prose coordinates (step numbers, numbered headings) + break on cosmetic renumbering — failure for the wrong reason. Ruling: + **cases name behaviour, never coordinates** (the walker is an agent + reading prose — the binding is semantic and rename-robust; only + coordinate-phrased claim text rots), and anchors became substring + fragments matched with `includes` (`#Boot` survives a renumber, still + goes red token-free if the heading vanishes). Prose test hooks + (`data-test`-style markers in skill files) considered and **parked**: + the only machine consumer of anchors is the staleness check, so hooks + would put permanent plumbing into shipped prose to serve a consumer + that barely needs it — revisit on evidence if arm-level references in + the failure harvest (generic headings like "If valid") produce real + renumber noise; the grammar extends to `probe:` references without + breakage. + - 2026-07-24 — Stage 2, the framework. `tests/prose/`: case parser and corpus validation (`lib/cases.cjs`), fixture recipes under a frozen clock with golden snapshots (`lib/fixtures.cjs`, `lib/fake-clock.cjs` diff --git a/tests/prose/README.md b/tests/prose/README.md index 263c32e94..93c6b87fa 100644 --- a/tests/prose/README.md +++ b/tests/prose/README.md @@ -21,6 +21,14 @@ Add a `## case:` block to the flow's file (or a new file for a long walk). Scope `files:` tightly — diff-selection and the PR-end suggestion run off it. `routing:` expects are agent-graded against the walk transcript; `state:` expects are asserted in code and require a `world:`. + +**Name behaviour, never coordinates.** Step numbers, arm letters, and +heading numbering in walk or expect text break on cosmetic renumbering — +failure for the wrong reason. Write "casing conventions load before the +boot pipeline", not "Step 0.1 before Step 0.2"; a behavioural claim +fails only when the behaviour changes, which is the failure the corpus +exists to catch. Anchors are substring fragments matched against heading +text (`#Boot` matches "Step 0.2: Boot") — pick the number-free part. `npm test` validates the corpus (parse, paths, anchors, grammar) and rebuild-compares every fixture — both token-free; only `/prose-test` spends tokens. diff --git a/tests/prose/lib/cases.cjs b/tests/prose/lib/cases.cjs index 5e065aba8..1ff5705fd 100644 --- a/tests/prose/lib/cases.cjs +++ b/tests/prose/lib/cases.cjs @@ -9,7 +9,14 @@ // - world: {fixture-name} (optional; required by state expects) // - origin: {free text} // - files: -// - {repo-relative path}[#Heading text] +// - {repo-relative path}[#heading fragment] +// +// Anchors are substring fragments, matched against heading text with +// `includes` — `#Boot` matches "Step 0.2: Boot" and survives a +// renumber. Authoring rule (see README): cases name BEHAVIOUR, never +// coordinates — step numbers, arm letters, and heading numbering in +// walk or expect text rot on cosmetic edits and fail for the wrong +// reason. // ### walk // {free text: entry point, what to do, stop condition} // ### user @@ -148,7 +155,7 @@ function headingExists(absPath, anchor) { const content = fs.readFileSync(absPath, 'utf8'); return content.split('\n').some((line) => { const h = line.match(/^#{1,6}\s+(.*?)\s*$/); - return h && h[1] === anchor; + return h && h[1].includes(anchor); }); } diff --git a/tests/prose/run.cjs b/tests/prose/run.cjs index 32c8bba2e..b03753456 100644 --- a/tests/prose/run.cjs +++ b/tests/prose/run.cjs @@ -139,7 +139,7 @@ function cmdPrompt(argv) { c.walk, '', 'SCOPE — the prose under walk:', - ...c.files.map((f) => ` - ${f.path}${f.anchor ? ` (start at "${f.anchor}")` : ''}`), + ...c.files.map((f) => ` - ${f.path}${f.anchor ? ` (start at the heading containing "${f.anchor}")` : ''}`), '', 'RULES', '- Follow the prose literally, step by step, arm by arm. Where it names an', diff --git a/tests/prose/smoke/cases.md b/tests/prose/smoke/cases.md index 1c28097dc..34018bfda 100644 --- a/tests/prose/smoke/cases.md +++ b/tests/prose/smoke/cases.md @@ -8,33 +8,34 @@ fails, suspect the framework before the prose. ## case: smoke-structure-start-boot - origin: framework smoke — structure-only walk, no world - files: - - skills/workflow-start/SKILL.md#Step 0.2: Boot + - skills/workflow-start/SKILL.md#Boot ### walk -Structure-only: read skills/workflow-start/SKILL.md's Step 0 and its -sub-steps. Establish the order of Step 0's parts and what Step 0.2 -declares about the boot pipeline. Stop after Step 0 — do not trace the -rest of the skill. +Structure-only: read the initialisation portion of +skills/workflow-start/SKILL.md — everything the skill does before any +work is shown or routed. Establish what the initialisation consists of +and in what order its parts run. Stop there — do not trace the rest of +the skill. ### expect -- routing: Step 0.1 (casing conventions) is prescribed before Step 0.2 (boot) -- routing: Step 0.2 declares the boot pipeline mandatory — it must complete before proceeding +- routing: casing conventions are loaded before the boot pipeline runs +- routing: the boot pipeline is declared mandatory — the skill must complete it before proceeding ## case: smoke-world-boot - world: base - origin: framework smoke — world build, engine-in-world, state assertions - files: - - skills/workflow-start/SKILL.md#Step 0.2: Boot + - skills/workflow-start/SKILL.md#Boot ### walk -In the project, run the boot command exactly as Step 0.2 of -skills/workflow-start/SKILL.md prescribes (sandbox concerns do not apply -in this world — run it directly). Capture the JSON response. Stop -immediately after recording it — do not continue into Step 0's -confirmation prompt or any later step. +In the project, run the boot command exactly as the initialisation prose +of skills/workflow-start/SKILL.md prescribes (sandbox concerns do not +apply in this world — run it directly). Capture the JSON response. Stop +immediately after recording it — do not continue into any confirmation +prompt or later step. ### expect