diff --git a/design/prose-tests.md b/design/prose-tests.md index 1d44caf0c..a2b324fde 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -84,6 +84,19 @@ testing. and the stubs, and returns a transcript. The asserter sees the transcript, the expected path, and the world delta — never the reverse. +- **P4b — the asserter is told which substitutions were armed.** The + recorded actions show a stub as the walker writing a file an agent + would have produced; without knowing the stub exists, an asserter + reads that as a missing dispatch and fails a walk that behaved + correctly. Measured: it did exactly that on the first round with + recording live. Armed stubs now reach the asserter with their + triggers. +- **P4c — assert what actions and state can prove.** A claim about + something *displayed* cannot be evidenced by the action log, because + emitting text is not a tool call — it rests on the walker's + narrative, the one piece of evidence it authors itself. Most display + claims have a consequence that can be asserted instead: the answer + consumed, the arm taken, the state produced. Prefer the consequence. - **P4a — substitutions are declared and marked.** A stub is named content; the case arming it owns the trigger, so one stub serves many moments. The walker records `SUBSTITUTED:` for each, and the @@ -196,6 +209,22 @@ testing. ## Log +- 2026-07-26 — The first round that found more prose than instrument. + Recording finally live (the blocker was `hasTrustDialogAccepted`, which + gates frontmatter hooks in project subagents and fails silently), five + cases run: one clean PASS, **two confirmed prose defects**, one case of + mine at fault, one framework gap. The defects: every handoff arm was + meant to say "invoke the skill" and four across three entry skills did + not — the walker found one, enumeration found the rest; and + `environment-check.md` opened with "information gathering only" then + told the entry to write and commit the setup document the processing + skill also writes, so whichever ran first won and the other silently + no-opped. Neither is visible to a unit test or the simulation, because + neither reads prose. The framework gap was mine and is closed (P4b): + the asserter saw a stub's Write as a missing agent dispatch. The + residual limit is P4c — display claims have no machine evidence, and + that is a property of the medium rather than a bug. + - 2026-07-25 — First real runs, and the nested-agent shape. The framework executed end to end for the first time: a Sonnet walker followed root-cause-validation.md against a live world — real engine calls, the diff --git a/tests/prose/README.md b/tests/prose/README.md index e2c7d2cf4..575586ea6 100644 --- a/tests/prose/README.md +++ b/tests/prose/README.md @@ -68,6 +68,16 @@ reason. Write "casing conventions load before the boot pipeline", not "Step 0.1 before Step 0.2". Anchors in `case.json` are substring fragments matched against heading text (`#Boot` matches "Step 0.2: Boot"). +**Assert what actions and state can prove.** A claim about something +*displayed* — "offers the choice and waits", "emits the summary block" — +can never be evidenced by the action log, because emitting text is not a +tool call. Those claims rest on the walker's narrative, which is the one +piece of evidence it authors itself. Most have a consequence that can be +asserted instead: the answer that was consumed, the arm that was taken, +the state that resulted. Prefer the consequence; keep display claims for +where the display genuinely is the behaviour under test, and expect them +to be the weakest line in the case. + **Scope `files` tightly** — selection and the PR-end suggestion run off it. ## Stubs diff --git a/tests/prose/cases/smoke-boot-in-world/act.md b/tests/prose/cases/smoke-boot-in-world/act.md deleted file mode 100644 index 115963792..000000000 --- a/tests/prose/cases/smoke-boot-in-world/act.md +++ /dev/null @@ -1,4 +0,0 @@ -Execute the initialisation portion of skills/workflow-start/SKILL.md -from its first step — sandbox concerns do not apply in this world, so -run any command it prescribes directly. Stop once initialisation is -complete and the skill would move on to showing the user's work. diff --git a/tests/prose/cases/smoke-boot-in-world/assert.md b/tests/prose/cases/smoke-boot-in-world/assert.md deleted file mode 100644 index b91fedd12..000000000 --- a/tests/prose/cases/smoke-boot-in-world/assert.md +++ /dev/null @@ -1,15 +0,0 @@ -The prose should have taken this path: - -1. loads the shared casing conventions before any state is read -2. runs the boot pipeline as a single engine call -3. the response reports no migrations applied — this world is already - migrated — so the migrations summary and its confirmation gate are - not raised, and the skill says so rather than stopping -4. the response reports the knowledge base ready, so the knowledge gate - is not entered and initialisation completes - -Further claims: - -- booting an already-migrated project changes nothing on disk -- the user is asked nothing: neither gate in initialisation can fire - against a world that is already migrated and already has a store diff --git a/tests/prose/cases/smoke-boot-in-world/case.json b/tests/prose/cases/smoke-boot-in-world/case.json deleted file mode 100644 index 047c10686..000000000 --- a/tests/prose/cases/smoke-boot-in-world/case.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "origin": "framework smoke — world build, engine-in-world, world comparison", - "files": [ - "skills/workflow-start/SKILL.md#Boot" - ] -} diff --git a/tests/prose/cases/smoke-boot-in-world/fixture-state.cjs b/tests/prose/cases/smoke-boot-in-world/fixture-state.cjs deleted file mode 100644 index 8e39fface..000000000 --- a/tests/prose/cases/smoke-boot-in-world/fixture-state.cjs +++ /dev/null @@ -1,11 +0,0 @@ -'use strict'; - -// A bare installed project: migrations applied, knowledge base set up -// keyword-only, no work units at all. - -module.exports = { - build(h) { - h.knowledge('setup', '--keyword-only'); - h.engine('boot'); - }, -}; diff --git a/tests/prose/cases/smoke-boot-in-world/fixture/.claude/settings.json b/tests/prose/cases/smoke-boot-in-world/fixture/.claude/settings.json deleted file mode 100644 index ae019ede5..000000000 --- a/tests/prose/cases/smoke-boot-in-world/fixture/.claude/settings.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "showClearContextOnPlanAccept": true, - "permissions": { - "allow": [ - "Edit(.workflows/**)", - "Bash(mv .workflows/:*)" - ] - } -} diff --git a/tests/prose/cases/smoke-boot-in-world/fixture/.recipe-hash b/tests/prose/cases/smoke-boot-in-world/fixture/.recipe-hash deleted file mode 100644 index 27528f6e9..000000000 --- a/tests/prose/cases/smoke-boot-in-world/fixture/.recipe-hash +++ /dev/null @@ -1 +0,0 @@ -e75a843d66eafa8a47fd143e1e3b2eb41e0a6b7b2b2d225753bec1cdee9892c5 diff --git a/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/.state/migrations b/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/.state/migrations deleted file mode 100644 index 3c7162acc..000000000 --- a/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/.state/migrations +++ /dev/null @@ -1,52 +0,0 @@ -001 -002 -003 -004 -005 -006 -007 -008 -009 -010 -011 -012 -013 -014 -015 -016 -017 -018 -019 -020 -021 -022 -023 -024 -025 -026 -027 -028 -029 -030 -031 -032 -033 -034 -035 -036 -037 -038 -039 -040 -041 -042 -043 -044 -045 -046 -047 -048 -049 -050 -051 -052 diff --git a/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/_gitignore.fixture b/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/_gitignore.fixture deleted file mode 100644 index 779cbf28a..000000000 --- a/tests/prose/cases/smoke-boot-in-world/fixture/.workflows/_gitignore.fixture +++ /dev/null @@ -1,2 +0,0 @@ -.cache/ -.manifest.json.*.tmp diff --git a/tests/prose/lib/prompts.cjs b/tests/prose/lib/prompts.cjs index 87e65cf5d..50a13107e 100644 --- a/tests/prose/lib/prompts.cjs +++ b/tests/prose/lib/prompts.cjs @@ -75,11 +75,12 @@ function walkerPrompt({ worldDir, root, situation, task, scope, stubs, answers } return `${parts.join('\n\n')}\n`; } -function asserterPrompt({ expected, world, actions }) { +function asserterPrompt({ expected, world, actions, substitutions }) { const t = loadTemplate('asserter'); const parts = [fill(t.main, { expected })]; if (world) parts.push(fill(t.world, { expecting: world.expecting, delta: world.delta })); if (actions) parts.push(fill(t.actions, { actions })); + if (substitutions) parts.push(fill(t.substitutions, { substitutions })); parts.push(t.transcript); return `${parts.join('\n\n')}\n`; } diff --git a/tests/prose/prompts/asserter.md b/tests/prose/prompts/asserter.md index 5e680e730..c4eaa036e 100644 --- a/tests/prose/prompts/asserter.md +++ b/tests/prose/prompts/asserter.md @@ -29,6 +29,16 @@ not have edited or omitted from it. This is what the walk *did*. {{actions}} +=== substitutions === + +HARNESS SUBSTITUTIONS the case armed for this walk. Where the recorded +actions show the walker doing one of these itself — writing a report an +agent would have produced, for instance — that IS the substitution, and +it is expected. The absence of a real agent dispatch alongside it is not +a missing step. + +{{substitutions}} + === transcript === The walk's own account follows under `=== TRANSCRIPT ===`. It is the diff --git a/tests/prose/run.cjs b/tests/prose/run.cjs index 03c655125..8464a9d33 100644 --- a/tests/prose/run.cjs +++ b/tests/prose/run.cjs @@ -167,7 +167,13 @@ function cmdAssert(argv) { + '(hasTrustDialogAccepted). Do not judge this run.'); } } - process.stdout.write(prompts.asserterPrompt({ expected: c.assert, world, actions })); + const substitutions = c.stubs.length + ? c.stubs.map((s) => { + const stub = cases.readStub(s.name); + return `- ${s.name} — fires ${s.trigger}\n ${stub.description.replace(/\n/g, '\n ')}`; + }).join('\n') + : null; + process.stdout.write(prompts.asserterPrompt({ expected: c.assert, world, actions, substitutions })); } // --- snap / verify --------------------------------------------------------