diff --git a/design/prose-tests.md b/design/prose-tests.md index a2b324fde..7307f07ce 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -84,6 +84,15 @@ testing. and the stubs, and returns a transcript. The asserter sees the transcript, the expected path, and the world delta — never the reverse. +- **P3a — three shapes of end state.** A walk either produces a world a + recipe can build (`assertion-state.cjs`, byte-compared), leaves the + fixture untouched (no assertion state — most entry-skill cases), or + mutates into a shape only prose can describe (`world: "claims"`). The + third exists because parts of some worlds are authored by the model: + a discussion's subtopic names are derived from context and differ run + to run. There the delta is taken against the fixture and judged + against the case's claims. It is the weakest of the three and is + reached for last: a world a recipe can build must be built. - **P4b — the asserter is told which substitutions were armed.** The recorded actions show a stub as the walker writing a file an agent would have produced; without knowing the stub exists, an asserter diff --git a/tests/prose/README.md b/tests/prose/README.md index 575586ea6..7071ef7a6 100644 --- a/tests/prose/README.md +++ b/tests/prose/README.md @@ -29,6 +29,14 @@ blob code merely relays lives in markdown. walker/asserter boundary — the walker must never see the expected path, and a file boundary enforces that structurally rather than by convention. +A case whose end state only prose can describe sets `"world": "claims"` +in `case.json` and carries no `assertion-state.cjs`. The delta is then +computed against the **fixture** — expected to be non-empty — and judged +against the case's stated claims. Reach for it only where the +variability is genuinely the model's to decide: a discussion's subtopic +names are derived from context and differ run to run, so no recipe can +pin them. A world a recipe *can* build must be built. + `fixture-state.cjs` and `assertion-state.cjs` are separate because they change for different reasons: the fixture when the precondition changes, the assertion when the prose's *behaviour* changes. The assertion state diff --git a/tests/prose/cases/discussion-session-substituted/act.md b/tests/prose/cases/discussion-session-substituted/act.md new file mode 100644 index 000000000..09ae57f9a --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/act.md @@ -0,0 +1,5 @@ +Execute skills/workflow-discussion-process/SKILL.md from its first step, +as the entry skill has just invoked it for work unit `pay`, topic `pay`. +Follow it through initialisation. Stop once the session loop is behind +you and the prose moves on to reviewing the discussion — do not begin +the review, dispatch any agent, or conclude the phase. diff --git a/tests/prose/cases/discussion-session-substituted/assert.md b/tests/prose/cases/discussion-session-substituted/assert.md new file mode 100644 index 000000000..a45e704a3 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/assert.md @@ -0,0 +1,25 @@ +The prose should have taken this path: + +1. resume detection finds no discussion file and treats this as a fresh + discussion — no resume choice is offered +2. initialisation registers the discussion in the manifest through the + engine before any map command, since the map requires the item to + exist +3. it creates the discussion file from the template, including the + terminal Triage section seeded as `(none)` +4. it derives initial subtopics from the context available and records + each on the map through the engine, each landing `pending` +5. it commits the initialised discussion +6. the session loop is where the conversation would happen, and the walk + resumes after it to reach the review step, which it does not enter + +Further claims: + +- the subtopic names are derived by the walker from the seed and context, + so which names appear and how many is expected to vary between runs. + What matters is that at least one was recorded through the engine, that + each began `pending`, and that by the end none remain `pending`. +- the discussion file's working sections hold the substituted content, + and its Triage section reads `(none)` +- the discussion item exists in the manifest with its subtopics; no other + phase has been touched diff --git a/tests/prose/cases/discussion-session-substituted/case.json b/tests/prose/cases/discussion-session-substituted/case.json new file mode 100644 index 000000000..c024e03f7 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/case.json @@ -0,0 +1,12 @@ +{ + "origin": "framework — the mid-flow substitution: an unscripted conversation stood in for, so the steps around it can be walked", + "world": "claims", + "files": [ + "skills/workflow-discussion-process/SKILL.md", + "skills/workflow-discussion-process/references/initialize-discussion.md", + "skills/workflow-discussion-process/references/template.md" + ], + "stubs": { + "discussion-pay-decided": "once the discussion file has been created from the template, the map seeded, and the initialisation commit made — the moment the prose would enter the session loop" + } +} diff --git a/tests/prose/cases/discussion-session-substituted/fixture-state.cjs b/tests/prose/cases/discussion-session-substituted/fixture-state.cjs new file mode 100644 index 000000000..546391b87 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture-state.cjs @@ -0,0 +1,13 @@ +'use strict'; + +// A feature routed to discussion with nothing started — the discussion +// file does not exist, so initialisation runs for real. + +const m = require('../../mainlines/feature.cjs'); + +module.exports = { + build(h) { + m.init(h); + m.create(h); + }, +}; diff --git a/tests/prose/cases/discussion-session-substituted/fixture.md b/tests/prose/cases/discussion-session-substituted/fixture.md new file mode 100644 index 000000000..9f3a2c546 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture.md @@ -0,0 +1,3 @@ +The feature exists and no phase has started. Discovery shaped it and +routed it to discussion; the discussion entry skill has already run and +handed over, so the processing skill is starting from its first step. diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.claude/settings.json b/tests/prose/cases/discussion-session-substituted/fixture/.claude/settings.json new file mode 100644 index 000000000..ae019ede5 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.claude/settings.json @@ -0,0 +1,9 @@ +{ + "showClearContextOnPlanAccept": true, + "permissions": { + "allow": [ + "Edit(.workflows/**)", + "Bash(mv .workflows/:*)" + ] + } +} diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.recipe-hash b/tests/prose/cases/discussion-session-substituted/fixture/.recipe-hash new file mode 100644 index 000000000..af934cf8a --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.recipe-hash @@ -0,0 +1 @@ +9b48d4c996d773219dc60a44fcc014b1f534adc4d22537741a96575da66d2e46 diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/.state/migrations b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/.state/migrations new file mode 100644 index 000000000..3c7162acc --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/.state/migrations @@ -0,0 +1,52 @@ +001 +002 +003 +004 +005 +006 +007 +008 +009 +010 +011 +012 +013 +014 +015 +016 +017 +018 +019 +020 +021 +022 +023 +024 +025 +026 +027 +028 +029 +030 +031 +032 +033 +034 +035 +036 +037 +038 +039 +040 +041 +042 +043 +044 +045 +046 +047 +048 +049 +050 +051 +052 diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/_gitignore.fixture new file mode 100644 index 000000000..779cbf28a --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/_gitignore.fixture @@ -0,0 +1,2 @@ +.cache/ +.manifest.json.*.tmp diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/manifest.json b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/manifest.json new file mode 100644 index 000000000..987f6c236 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/manifest.json @@ -0,0 +1,7 @@ +{ + "work_units": { + "pay": { + "work_type": "feature" + } + } +} diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/discovery/sessions/session-001.md b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/discovery/sessions/session-001.md new file mode 100644 index 000000000..2aab96961 --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/discovery/sessions/session-001.md @@ -0,0 +1,39 @@ +# Discovery Session 001 + +Date: 2026-01-01 +Work unit: pay + +## Description (as of session) + +Accept card payments at checkout. + +## Seed + +(none) + +## Imports + +(none) + +## Map State at Start + +(n/a — single-topic work) + +## Exploration + +Shaped as a single feature: accept card payments at checkout using +the existing gateway account. Card-only for v1 came up early and was +softly agreed; wallet support was noted as a likely deferral. No +research need surfaced — routed straight to discussion. + +## Edits + +(none) + +## Topics Identified + +(none) + +## Conclusion + +(none) diff --git a/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/manifest.json b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/manifest.json new file mode 100644 index 000000000..1cf8afb0e --- /dev/null +++ b/tests/prose/cases/discussion-session-substituted/fixture/.workflows/pay/manifest.json @@ -0,0 +1,8 @@ +{ + "name": "pay", + "work_type": "feature", + "status": "in-progress", + "created": "2026-01-01", + "description": "Accept card payments at checkout", + "phases": {} +} diff --git a/tests/prose/lib/cases.cjs b/tests/prose/lib/cases.cjs index 006534a5d..a4b3d79df 100644 --- a/tests/prose/lib/cases.cjs +++ b/tests/prose/lib/cases.cjs @@ -8,7 +8,8 @@ // modules. // // case.json the values code branches on: -// { origin, files[], answers[], stubs{name: trigger} } +// { origin, files[], answers[], stubs{name: trigger}, +// world: "claims" (optional) } // fixture.md optional. Prose describing the starting world — // what has already happened, where the session // stands. Given to the walker. @@ -20,10 +21,19 @@ // further claims. Given ONLY to the asserter. // assertion-state.cjs optional. exports build(h): the world the walk // should produce. Absent means "unchanged" — the -// walk should leave the fixture state untouched. +// walk should leave the fixture state untouched, +// unless case.json sets world: "claims". // fixture/ generated snapshot of the starting world // assertion/ generated snapshot of the expected world // +// `world: "claims"` is the third case of a walk's end state: it mutates, +// but into a shape no recipe can pin because prose the model authors is +// part of it — derived subtopic names, written artifact text. The delta +// is then computed against the fixture and judged against the case's +// stated claims rather than byte-compared. Reach for it only when the +// variability is genuinely the model's to decide; a world a recipe can +// build must be built. +// // act.md and assert.md are separate files because that is the P4 // boundary: the walker must never see the expected trace, and a file // boundary enforces it structurally rather than by convention. @@ -93,6 +103,7 @@ function loadCase(id) { : { path: spec.slice(0, hash).trim(), anchor: spec.slice(hash + 1).trim() }; }), answers: meta.answers || [], + worldMode: meta.world || null, stubs: Object.entries(meta.stubs || {}).map(([name, trigger]) => ({ name, trigger })), situation: readIf(dir, FILES.situation), act: readIf(dir, FILES.act), @@ -168,6 +179,16 @@ function validateCorpus(cases) { if (!c.act) errors.push(`${at}: no ${FILES.act}`); if (!c.assert) errors.push(`${at}: no ${FILES.assert} — the expected trace is what catches a silent repair`); + if (c.worldMode && c.worldMode !== 'claims') { + errors.push(`${at}: ${FILES.meta} world must be "claims" when set, not "${c.worldMode}"`); + } + if (c.worldMode === 'claims' && c.hasAssertionState) { + errors.push(`${at}: world "claims" and ${FILES.assertionState} are exclusive — ` + + 'a world a recipe can build must be built'); + } + if (c.worldMode === 'claims' && !c.hasFixtureState) { + errors.push(`${at}: world "claims" needs a world to mutate`); + } if (c.hasAssertionState && !c.hasFixtureState) { errors.push(`${at}: ${FILES.assertionState} without ${FILES.fixtureState} — nothing to act on`); } diff --git a/tests/prose/lib/worlds.cjs b/tests/prose/lib/worlds.cjs index 566bb03eb..000f5e990 100644 --- a/tests/prose/lib/worlds.cjs +++ b/tests/prose/lib/worlds.cjs @@ -246,8 +246,8 @@ function unifiedDiff(label, expectedBuf, actualBuf) { * volatile values surface as ordinary differences and the agent rules on * them. A case with no assertion-state expects its fixture back unchanged. */ -function diffWorld(caseId, worldDir) { - const which = fs.existsSync(snapshotDir(caseId, 'assertion')) ? 'assertion' : 'fixture'; +function diffWorld(caseId, worldDir, claimsMode = false) { + const which = !claimsMode && fs.existsSync(snapshotDir(caseId, 'assertion')) ? 'assertion' : 'fixture'; const expected = readSnapshot(caseId, which); if (expected === null) throw new Error(`case "${caseId}" has no committed ${which} snapshot`); const actual = collectTree(worldDir); @@ -262,7 +262,12 @@ function diffWorld(caseId, worldDir) { for (const rel of expected.keys()) if (!actual.has(rel)) removed.push(rel); return { - expecting: which === 'assertion' ? 'the assertion state' : 'the fixture state, unchanged', + expecting: which === 'assertion' + ? 'the assertion state' + : (claimsMode + ? 'no fixed world — the delta below is against the STARTING state, is expected to be ' + + "non-empty, and must be judged against the case's stated claims" + : 'the fixture state, unchanged'), added, removed, changed, diff --git a/tests/prose/run.cjs b/tests/prose/run.cjs index 8464a9d33..66b4322a3 100644 --- a/tests/prose/run.cjs +++ b/tests/prose/run.cjs @@ -146,7 +146,7 @@ function cmdAssert(argv) { let actions = null; if (c.hasFixtureState) { const dir = requireWorld(argv, c); - const delta = worlds.diffWorld(c.id, dir); + const delta = worlds.diffWorld(c.id, dir, c.worldMode === 'claims'); world = { expecting: delta.expecting, delta: [ diff --git a/tests/prose/stubs/discussion-pay-decided.md b/tests/prose/stubs/discussion-pay-decided.md new file mode 100644 index 000000000..5a5575d28 --- /dev/null +++ b/tests/prose/stubs/discussion-pay-decided.md @@ -0,0 +1,44 @@ +# stub: discussion-pay-decided + +The outcome of a discussion session that this framework does not +simulate — the organic conversation between user and assistant. Stands in +for the whole session loop: + +1. Write the content below over the discussion file the prose created at + `.workflows/pay/discussion/pay.md`, replacing the working sections + while leaving the file's structure as the template made it. +2. Move every subtopic the map holds to a settled state through the + engine, so the map reflects a finished conversation: + `discussion-map set pay pay {subtopic} decided` for each, except any + named for wallets or deferral, which take `deferred`. +3. Resume the prose at the step that follows the session loop. + +--- + +# Discussion — pay + +## Context + +Accept card payments at checkout using the existing gateway account. +Shaped in discovery as a single feature; card-only for v1. + +## Decisions + +- Use the existing gateway account — no new provider onboarding, and no + second set of credentials to hold. +- Card-only for v1. The checkout collects card details directly; wallet + flows are a separate surface. +- Capture is confirmed by gateway webhook. The checkout never polls: a + poll would either be slow or hammer the gateway, and the webhook is + already guaranteed. +- An order with no shippable items still needs a tax context. Billing + address stands in where there is no shipping address. + +## Deferred + +- Wallet support (Apple and Google Pay) — revisit after v1 ships and we + know the card flow holds. + +## Triage + +(none)