From 9d3d1b37e021aa83b0da62bd20b21f1e2ee6ec15 Mon Sep 17 00:00:00 2001 From: Lee Overy Date: Sun, 26 Jul 2026 14:00:24 +0100 Subject: [PATCH] feat(prose-tests): the mid-flow substitution, and a world only prose can describe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The conversation stub was designed at length and never exercised. Writing the case proved the mechanism needed no change — a stub is named content, the case owns the trigger, and "once the file is created from the template, the map seeded, and the initialisation commit made" is as observable a moment as an engine dispatch. What the case did need was a third shape of end state. A discussion's subtopics are derived by the model from whatever context it has, so their names and number differ run to run and no recipe can pin the resulting world. Cases like that now set `world: "claims"`: the delta is taken against the fixture, expected to be non-empty, and judged against the case's stated claims. It is the weakest of the three shapes and reached for last — a world a recipe can build must be built, and validation refuses claims mode alongside an assertion state. The case walks discussion initialisation for real — resume detection, topic start before any map command, the file from the template, subtopics recorded pending, the commit — then substitutes the session it cannot simulate and resumes to the review boundary. Co-Authored-By: Claude Opus 5 --- design/prose-tests.md | 9 ++++ tests/prose/README.md | 8 +++ .../discussion-session-substituted/act.md | 5 ++ .../discussion-session-substituted/assert.md | 25 +++++++++ .../discussion-session-substituted/case.json | 12 +++++ .../fixture-state.cjs | 13 +++++ .../discussion-session-substituted/fixture.md | 3 ++ .../fixture/.claude/settings.json | 9 ++++ .../fixture/.recipe-hash | 1 + .../fixture/.workflows/.state/migrations | 52 +++++++++++++++++++ .../fixture/.workflows/_gitignore.fixture | 2 + .../fixture/.workflows/manifest.json | 7 +++ .../pay/discovery/sessions/session-001.md | 39 ++++++++++++++ .../fixture/.workflows/pay/manifest.json | 8 +++ tests/prose/lib/cases.cjs | 25 ++++++++- tests/prose/lib/worlds.cjs | 11 ++-- tests/prose/run.cjs | 2 +- tests/prose/stubs/discussion-pay-decided.md | 44 ++++++++++++++++ 18 files changed, 269 insertions(+), 6 deletions(-) create mode 100644 tests/prose/cases/discussion-session-substituted/act.md create mode 100644 tests/prose/cases/discussion-session-substituted/assert.md create mode 100644 tests/prose/cases/discussion-session-substituted/case.json create mode 100644 tests/prose/cases/discussion-session-substituted/fixture-state.cjs create mode 100644 tests/prose/cases/discussion-session-substituted/fixture.md create mode 100644 tests/prose/cases/discussion-session-substituted/fixture/.claude/settings.json create mode 100644 tests/prose/cases/discussion-session-substituted/fixture/.recipe-hash create mode 100644 tests/prose/cases/discussion-session-substituted/fixture/.workflows/.state/migrations create mode 100644 tests/prose/cases/discussion-session-substituted/fixture/.workflows/_gitignore.fixture create mode 100644 tests/prose/cases/discussion-session-substituted/fixture/.workflows/manifest.json create mode 100644 tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/discovery/sessions/session-001.md create mode 100644 tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/manifest.json create mode 100644 tests/prose/stubs/discussion-pay-decided.md diff --git a/design/prose-tests.md b/design/prose-tests.md index a2b324fde..7307f07ce 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -84,6 +84,15 @@ testing. and the stubs, and returns a transcript. The asserter sees the transcript, the expected path, and the world delta — never the reverse. +- **P3a — three shapes of end state.** A walk either produces a world a + recipe can build (`assertion-state.cjs`, byte-compared), leaves the + fixture untouched (no assertion state — most entry-skill cases), or + mutates into a shape only prose can describe (`world: "claims"`). The + third exists because parts of some worlds are authored by the model: + a discussion's subtopic names are derived from context and differ run + to run. There the delta is taken against the fixture and judged + against the case's claims. It is the weakest of the three and is + reached for last: a world a recipe can build must be built. - **P4b — the asserter is told which substitutions were armed.** The recorded actions show a stub as the walker writing a file an agent would have produced; without knowing the stub exists, an asserter diff --git a/tests/prose/README.md b/tests/prose/README.md index 575586ea6..7071ef7a6 100644 --- a/tests/prose/README.md +++ b/tests/prose/README.md @@ -29,6 +29,14 @@ blob code merely relays lives in markdown. walker/asserter boundary — the walker must never see the expected path, and a file boundary enforces that structurally rather than by convention. +A case whose end state only prose can describe sets `"world": "claims"` +in `case.json` and carries no `assertion-state.cjs`. The delta is then +computed against the **fixture** — expected to be non-empty — and judged +against the case's stated claims. Reach for it only where the +variability is genuinely the model's to decide: a discussion's subtopic +names are derived from context and differ run to run, so no recipe can +pin them. A world a recipe *can* build must be built. + `fixture-state.cjs` and `assertion-state.cjs` are separate because they change for different reasons: the fixture when the precondition changes, the assertion when the prose's *behaviour* changes. The assertion state diff --git a/tests/prose/cases/discussion-session-substituted/act.md b/tests/prose/cases/discussion-session-substituted/act.md new file mode 100644 index 000000000..09ae57f9a --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/act.md @@ -0,0 +1,5 @@ +Execute skills/workflow-discussion-process/SKILL.md from its first step, +as the entry skill has just invoked it for work unit `pay`, topic `pay`. +Follow it through initialisation. Stop once the session loop is behind +you and the prose moves on to reviewing the discussion — do not begin +the review, dispatch any agent, or conclude the phase. diff --git a/tests/prose/cases/discussion-session-substituted/assert.md b/tests/prose/cases/discussion-session-substituted/assert.md new file mode 100644 index 000000000..a45e704a3 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/assert.md @@ -0,0 +1,25 @@ +The prose should have taken this path: + +1. resume detection finds no discussion file and treats this as a fresh + discussion — no resume choice is offered +2. initialisation registers the discussion in the manifest through the + engine before any map command, since the map requires the item to + exist +3. it creates the discussion file from the template, including the + terminal Triage section seeded as `(none)` +4. it derives initial subtopics from the context available and records + each on the map through the engine, each landing `pending` +5. it commits the initialised discussion +6. the session loop is where the conversation would happen, and the walk + resumes after it to reach the review step, which it does not enter + +Further claims: + +- the subtopic names are derived by the walker from the seed and context, + so which names appear and how many is expected to vary between runs. + What matters is that at least one was recorded through the engine, that + each began `pending`, and that by the end none remain `pending`. +- the discussion file's working sections hold the substituted content, + and its Triage section reads `(none)` +- the discussion item exists in the manifest with its subtopics; no other + phase has been touched diff --git a/tests/prose/cases/discussion-session-substituted/case.json b/tests/prose/cases/discussion-session-substituted/case.json new file mode 100644 index 000000000..c024e03f7 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/case.json @@ -0,0 +1,12 @@ +{ + "origin": "framework — the mid-flow substitution: an unscripted conversation stood in for, so the steps around it can be walked", + "world": "claims", + "files": [ + "skills/workflow-discussion-process/SKILL.md", + "skills/workflow-discussion-process/references/initialize-discussion.md", + "skills/workflow-discussion-process/references/template.md" + ], + "stubs": { + "discussion-pay-decided": "once the discussion file has been created from the template, the map seeded, and the initialisation commit made — the moment the prose would enter the session loop" + } +} diff --git a/tests/prose/cases/discussion-session-substituted/fixture-state.cjs b/tests/prose/cases/discussion-session-substituted/fixture-state.cjs new file mode 100644 index 000000000..546391b87 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture-state.cjs @@ -0,0 +1,13 @@ +'use strict'; + +// A feature routed to discussion with nothing started — the discussion +// file does not exist, so initialisation runs for real. + +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + }, +}; diff --git a/tests/prose/cases/discussion-session-substituted/fixture.md b/tests/prose/cases/discussion-session-substituted/fixture.md new file mode 100644 index 000000000..9f3a2c546 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture.md @@ -0,0 +1,3 @@ +The feature exists and no phase has started. Discovery shaped it and +routed it to discussion; the discussion entry skill has already run and +handed over, so the processing skill is starting from its first step. diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.claude/settings.json b/tests/prose/cases/discussion-session-substituted/fixture/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.recipe-hash b/tests/prose/cases/discussion-session-substituted/fixture/.recipe-hash new file mode 100644 index 000000000..af934cf8a --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.recipe-hash @@ -0,0 +1 @@ +9b48d4c996d773219dc60a44fcc014b1f534adc4d22537741a96575da66d2e46 diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/.state/migrations b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/manifest.json b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/manifest.json new file mode 100644 index 000000000..987f6c236 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "pay": { + "work_type": "feature" + } + } +} diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/discovery/sessions/session-001.md new file mode 100644 index 000000000..2aab96961 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/discovery/sessions/session-001.md @@ -0,0 +1,39 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: pay + +## Description (as of session) + +Accept card payments at checkout. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Shaped as a single feature: accept card payments at checkout using +the existing gateway account. Card-only for v1 came up early and was +softly agreed; wallet support was noted as a likely deferral. No +research need surfaced — routed straight to discussion. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/manifest.json b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/manifest.json new file mode 100644 index 000000000..1cf8afb0e --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/manifest.json @@ -0,0 +1,8 @@ +{ + "name": "pay", + "work_type": "feature", + "status": "in-progress", + "created": "2026-01-01", + "description": "Accept card payments at checkout", + "phases": {} +} diff --git a/tests/prose/lib/cases.cjs b/tests/prose/lib/cases.cjs index 006534a5d..a4b3d79df 100644 --- a/tests/prose/lib/cases.cjs +++ b/tests/prose/lib/cases.cjs @@ -8,7 +8,8 @@ // modules. // // case.json the values code branches on: -// { origin, files[], answers[], stubs{name: trigger} } +// { origin, files[], answers[], stubs{name: trigger}, +// world: "claims" (optional) } // fixture.md optional. Prose describing the starting world — // what has already happened, where the session // stands. Given to the walker. @@ -20,10 +21,19 @@ // further claims. Given ONLY to the asserter. // assertion-state.cjs optional. exports build(h): the world the walk // should produce. Absent means "unchanged" — the -// walk should leave the fixture state untouched. +// walk should leave the fixture state untouched, +// unless case.json sets world: "claims". // fixture/ generated snapshot of the starting world // assertion/ generated snapshot of the expected world // +// `world: "claims"` is the third case of a walk's end state: it mutates, +// but into a shape no recipe can pin because prose the model authors is +// part of it — derived subtopic names, written artifact text. The delta +// is then computed against the fixture and judged against the case's +// stated claims rather than byte-compared. Reach for it only when the +// variability is genuinely the model's to decide; a world a recipe can +// build must be built. +// // act.md and assert.md are separate files because that is the P4 // boundary: the walker must never see the expected trace, and a file // boundary enforces it structurally rather than by convention. @@ -93,6 +103,7 @@ function loadCase(id) { : { path: spec.slice(0, hash).trim(), anchor: spec.slice(hash + 1).trim() }; }), answers: meta.answers || [], + worldMode: meta.world || null, stubs: Object.entries(meta.stubs || {}).map(([name, trigger]) => ({ name, trigger })), situation: readIf(dir, FILES.situation), act: readIf(dir, FILES.act), @@ -168,6 +179,16 @@ function validateCorpus(cases) { if (!c.act) errors.push(`${at}: no ${FILES.act}`); if (!c.assert) errors.push(`${at}: no ${FILES.assert} — the expected trace is what catches a silent repair`); + if (c.worldMode && c.worldMode !== 'claims') { + errors.push(`${at}: ${FILES.meta} world must be "claims" when set, not "${c.worldMode}"`); + } + if (c.worldMode === 'claims' && c.hasAssertionState) { + errors.push(`${at}: world "claims" and ${FILES.assertionState} are exclusive — ` + + 'a world a recipe can build must be built'); + } + if (c.worldMode === 'claims' && !c.hasFixtureState) { + errors.push(`${at}: world "claims" needs a world to mutate`); + } if (c.hasAssertionState && !c.hasFixtureState) { errors.push(`${at}: ${FILES.assertionState} without ${FILES.fixtureState} — nothing to act on`); } diff --git a/tests/prose/lib/worlds.cjs b/tests/prose/lib/worlds.cjs index 566bb03eb..000f5e990 100644 --- a/tests/prose/lib/worlds.cjs +++ b/tests/prose/lib/worlds.cjs @@ -246,8 +246,8 @@ function unifiedDiff(label, expectedBuf, actualBuf) { * volatile values surface as ordinary differences and the agent rules on * them. A case with no assertion-state expects its fixture back unchanged. */ -function diffWorld(caseId, worldDir) { - const which = fs.existsSync(snapshotDir(caseId, 'assertion')) ? 'assertion' : 'fixture'; +function diffWorld(caseId, worldDir, claimsMode = false) { + const which = !claimsMode && fs.existsSync(snapshotDir(caseId, 'assertion')) ? 'assertion' : 'fixture'; const expected = readSnapshot(caseId, which); if (expected === null) throw new Error(`case "${caseId}" has no committed ${which} snapshot`); const actual = collectTree(worldDir); @@ -262,7 +262,12 @@ function diffWorld(caseId, worldDir) { for (const rel of expected.keys()) if (!actual.has(rel)) removed.push(rel); return { - expecting: which === 'assertion' ? 'the assertion state' : 'the fixture state, unchanged', + expecting: which === 'assertion' + ? 'the assertion state' + : (claimsMode + ? 'no fixed world — the delta below is against the STARTING state, is expected to be ' + + "non-empty, and must be judged against the case's stated claims" + : 'the fixture state, unchanged'), added, removed, changed, diff --git a/tests/prose/run.cjs b/tests/prose/run.cjs index 8464a9d33..66b4322a3 100644 --- a/tests/prose/run.cjs +++ b/tests/prose/run.cjs @@ -146,7 +146,7 @@ function cmdAssert(argv) { let actions = null; if (c.hasFixtureState) { const dir = requireWorld(argv, c); - const delta = worlds.diffWorld(c.id, dir); + const delta = worlds.diffWorld(c.id, dir, c.worldMode === 'claims'); world = { expecting: delta.expecting, delta: [ diff --git a/tests/prose/stubs/discussion-pay-decided.md b/tests/prose/stubs/discussion-pay-decided.md new file mode 100644 index 000000000..5a5575d28 --- /dev/null +++ b/tests/prose/stubs/discussion-pay-decided.md @@ -0,0 +1,44 @@ +# stub: discussion-pay-decided + +The outcome of a discussion session that this framework does not +simulate — the organic conversation between user and assistant. Stands in +for the whole session loop: + +1. Write the content below over the discussion file the prose created at + `.workflows/pay/discussion/pay.md`, replacing the working sections + while leaving the file's structure as the template made it. +2. Move every subtopic the map holds to a settled state through the + engine, so the map reflects a finished conversation: + `discussion-map set pay pay {subtopic} decided` for each, except any + named for wallets or deferral, which take `deferred`. +3. Resume the prose at the step that follows the session loop. + +--- + +# Discussion — pay + +## Context + +Accept card payments at checkout using the existing gateway account. +Shaped in discovery as a single feature; card-only for v1. + +## Decisions + +- Use the existing gateway account — no new provider onboarding, and no + second set of credentials to hold. +- Card-only for v1. The checkout collects card details directly; wallet + flows are a separate surface. +- Capture is confirmed by gateway webhook. The checkout never polls: a + poll would either be slow or hammer the gateway, and the webhook is + already guaranteed. +- An order with no shippable items still needs a tax context. Billing + address stands in where there is no shipping address. + +## Deferred + +- Wallet support (Apple and Google Pay) — revisit after v1 ships and we + know the card flow holds. + +## Triage + +(none)