diff --git a/.claude/agents/prose-asserter.md b/.claude/agents/prose-asserter.md index 117b34871..cd89bba9c 100644 --- a/.claude/agents/prose-asserter.md +++ b/.claude/agents/prose-asserter.md @@ -38,18 +38,38 @@ tools you appear to have or what mode you are running in. by nature and are immaterial. Differences in shape, field presence, status vocabulary, or content are material. +## Two kinds of evidence, and which one wins + +**Recorded actions** are captured by a harness hook as each tool call +happens. The walker did not write them and could not have edited them. +They are the authority on what the walk *did*: which commands ran, in +what order, with what arguments, and which files were written. + +**The transcript** is the walker's own account. It is the authority on +*reasoning* — which arm it entered, which guard line selected it, what +it emitted to the user — because none of that appears in a tool call. + +Judge each expected step against the right one. A step about an action +("records the dispatch through the engine", "promotes with a scan then +closes with an incorporate") is evidenced from the recorded actions, +even when the transcript omits it — walkers routinely under-report, and +a thin narrative is not a missing step. A step about reasoning or +output is evidenced from the transcript. Where the two disagree about +what was done, the recorded actions win. + ## An invalid walk is not a failing walk -Before judging anything, check where the transcript begins. If its first -entry is not the start of the walk the expected path describes — the -walker jumped into the middle of the flow, skipping earlier steps -without recording a `DEVIATION` for each — then the walk cannot evidence -the path, and judging it would report a prose failure that was never -demonstrated. +Check the **recorded actions**, not the transcript, for where the walk +began. If the actions show it starting mid-flow — the earlier steps +never attempted, and no `DEVIATION` recorded for them — then the walk +cannot evidence the path, and judging it would report a prose failure +that was never demonstrated. -In that case return `VERDICT: INVALID WALK`, name the entry the -transcript opens on and the step it should have opened on, and judge -nothing further. This is a fault in the walk, not in the prose. +In that case return `VERDICT: INVALID WALK`, name the action it opens +on and the step it should have opened on, and judge nothing further. +This is a fault in the walk, not in the prose. A walk that *did* the +work and merely described it poorly is **not** invalid — judge it from +the actions. ## Verdict diff --git a/.claude/agents/prose-orchestrator.md b/.claude/agents/prose-orchestrator.md index 7b4dde493..bd14394d3 100644 --- a/.claude/agents/prose-orchestrator.md +++ b/.claude/agents/prose-orchestrator.md @@ -43,13 +43,20 @@ a walk. 2. **Walk** — `node tests/prose/run.cjs prompt --world `. Dispatch the **prose-walker** agent with that output as its prompt, - verbatim and unmodified. Keep the transcript it returns. + verbatim and unmodified. Keep the account it returns. Its tool calls + are recorded into the world by its own hook — you do nothing to + collect them. 3. **Assert** — `node tests/prose/run.cjs assert --world `. Dispatch the **prose-asserter** agent with that output, followed by - the walk transcript under a `=== TRANSCRIPT ===` line. Keep its + the walker's account under a `=== TRANSCRIPT ===` line. Keep its verdict. + If that command **fails** rather than printing a prompt, the harness + is broken, not the prose. Stop there, destroy the world, and report + `VERDICT: HARNESS ERROR` with the message it printed. Never fall back + to judging a walk with no record of what it did. + Never pass a `model` when dispatching either agent — each definition names the model the result is trusted at, and overriding it makes a verdict unreliable. @@ -77,7 +84,7 @@ Return exactly this and nothing else: ``` CASE: -VERDICT: PASS | FAIL | FLAKY | INVALID +VERDICT: PASS | FAIL | FLAKY | INVALID | HARNESS ERROR PATH: / steps passed WORLD: PASS | FAIL MARKERS: diff --git a/.claude/agents/prose-walker.md b/.claude/agents/prose-walker.md index b6ed011df..d84f8fcc2 100644 --- a/.claude/agents/prose-walker.md +++ b/.claude/agents/prose-walker.md @@ -3,6 +3,12 @@ name: prose-walker description: Executes workflow prose exactly as a live session would, against a disposable test world, and returns a transcript of what it did. Dispatched by prose-orchestrator during a prose-test run. tools: Read, Write, Edit, Bash, Glob, Grep model: opus +hooks: + PostToolUse: + - matcher: "Bash|Write|Edit|Read" + hooks: + - type: command + command: "node \"$CLAUDE_PROJECT_DIR/tests/prose/lib/record-action.cjs\"" --- # Prose Walker diff --git a/design/prose-tests.md b/design/prose-tests.md index 88e2541fd..1d44caf0c 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -110,6 +110,21 @@ testing. failing case re-runs once from a fresh world at the same models. A defect in the prose reproduces; a one-off does not, and is reported as FLAKY with both runs quoted. Nothing is auto-resolved. +- **P6e — what the walk did is recorded by the harness, not reported by + the walker.** A `PostToolUse` hook declared in the walker's own agent + frontmatter (so it fires for walkers and nothing else) appends every + tool call — command, path, error flag — to `.walk-actions.log` in the + world, excluded from the diff. Evidence then splits by kind: the + **recorded actions** are the authority on what was *done*, the + walker's transcript on what was *reasoned* (which arm, which guard, + what was emitted), and where they disagree the recording wins. + Measured, not assumed: a diagnostic walker's world held the agent row + at `incorporated` and the report file — it had run dispatch, stub, + scan and incorporate — while its transcript began at the fifth step. + Two attempts to fix that with wording (a stricter log format, then + Opus) both failed, because a narrative an agent composes at the end + is a recollection however it is instructed. Asking it to write a log + as it goes would have relocated the same trust, not removed it. - **P6d — a walk that began mid-flow proves nothing.** Walkers skip steps whose effect the world already holds — "already booted", "the plan already exists" — and then the case reports a prose failure at a diff --git a/skills/workflow-implementation-entry/references/invoke-skill.md b/skills/workflow-implementation-entry/references/invoke-skill.md index 3d947786e..05d92e2b4 100644 --- a/skills/workflow-implementation-entry/references/invoke-skill.md +++ b/skills/workflow-implementation-entry/references/invoke-skill.md @@ -36,3 +36,4 @@ Implementation: {exists:[true|false]} Dependencies: {All satisfied | List any notes} Environment: {Setup required | No special setup required} +``` diff --git a/skills/workflow-investigation-entry/references/invoke-skill.md b/skills/workflow-investigation-entry/references/invoke-skill.md index 5356ded2b..4c760534c 100644 --- a/skills/workflow-investigation-entry/references/invoke-skill.md +++ b/skills/workflow-investigation-entry/references/invoke-skill.md @@ -31,6 +31,7 @@ Bug context: - Expected behavior: {from the carrier / gather-context} - Actual behavior: {from the carrier / gather-context} - Initial context: {error messages, reproduction steps — from the carrier / gather-context, or "(none captured yet)"} +``` #### If source is `continue` @@ -39,3 +40,4 @@ Investigation session for: {work_unit} Source: existing investigation Output: .workflows/{work_unit}/investigation/{topic}.md +``` diff --git a/skills/workflow-planning-entry/references/invoke-skill.md b/skills/workflow-planning-entry/references/invoke-skill.md index c4d5736a0..ee767f89b 100644 --- a/skills/workflow-planning-entry/references/invoke-skill.md +++ b/skills/workflow-planning-entry/references/invoke-skill.md @@ -21,6 +21,7 @@ Work unit: {work_unit} Specification: .workflows/{work_unit}/specification/{topic}/specification.md Additional context: {summary of user's additional context, or "none"} Cross-cutting references: {list of applicable cross-cutting specs with brief summaries, or "none"} +``` #### If continuing or reviewing existing plan @@ -31,3 +32,4 @@ Work unit: {work_unit} Specification: .workflows/{work_unit}/specification/{topic}/specification.md Existing plan: .workflows/{work_unit}/planning/{topic}/planning.md Cross-cutting references: {list of applicable cross-cutting specs with brief summaries, or "none"} +``` diff --git a/skills/workflow-review-entry/references/invoke-skill.md b/skills/workflow-review-entry/references/invoke-skill.md index 3b8b609b7..e2a284f72 100644 --- a/skills/workflow-review-entry/references/invoke-skill.md +++ b/skills/workflow-review-entry/references/invoke-skill.md @@ -26,3 +26,4 @@ Plans to review: topic: {topic} format: {format} specification: .workflows/{work_unit}/specification/{topic}/specification.md (exists: {true|false}) +``` diff --git a/skills/workflow-scoping-entry/references/invoke-skill.md b/skills/workflow-scoping-entry/references/invoke-skill.md index 14f17b19a..fa5721048 100644 --- a/skills/workflow-scoping-entry/references/invoke-skill.md +++ b/skills/workflow-scoping-entry/references/invoke-skill.md @@ -28,3 +28,4 @@ Invoke the **workflow-scoping-process** skill (Skill tool) with the next fenced Scoping session for: {topic} Work unit: {work_unit} Description: {description} +``` diff --git a/skills/workflow-specification-entry/references/invoke-skill.md b/skills/workflow-specification-entry/references/invoke-skill.md index d4d4389df..1c40141bd 100644 --- a/skills/workflow-specification-entry/references/invoke-skill.md +++ b/skills/workflow-specification-entry/references/invoke-skill.md @@ -22,6 +22,7 @@ Source material: Work unit: {work_unit} Action: {verb} specification +``` #### If `work_type` is `bugfix` @@ -33,6 +34,7 @@ Source material: Work unit: {work_unit} Action: {verb} specification +``` #### If `work_type` is `epic` @@ -51,6 +53,7 @@ Source material: Work unit: {work_unit} Topic: {topic} Action: {verb} specification +``` #### If `work_type` is `cross-cutting` @@ -65,3 +68,4 @@ Source material: Work unit: {work_unit} Action: {verb} specification +``` diff --git a/tests/prose/cases/implementation-picks-first-task/act.md b/tests/prose/cases/implementation-picks-first-task/act.md index cdf6faee0..5f15274c7 100644 --- a/tests/prose/cases/implementation-picks-first-task/act.md +++ b/tests/prose/cases/implementation-picks-first-task/act.md @@ -1,4 +1,5 @@ Execute skills/workflow-implementation-entry/SKILL.md with arguments $0=feature, $1=pay, then continue into the processing skill it invokes. -Follow the flow until the prose directs you to begin building the first -task. Stop there — write no implementation code and no tests. +Stop once the environment has been settled and the skill moves on to +its next concern — do not enter project-skill or linter discovery, and +write no implementation code. diff --git a/tests/prose/cases/implementation-picks-first-task/assert.md b/tests/prose/cases/implementation-picks-first-task/assert.md index e09c80080..c5dd079d0 100644 --- a/tests/prose/cases/implementation-picks-first-task/assert.md +++ b/tests/prose/cases/implementation-picks-first-task/assert.md @@ -11,10 +11,7 @@ The prose should have taken this path: resuming-from-a-previous-session note 5. environment setup records the answer as a setup document and commits it, so the question is not asked again in a later session -6. reads the plan through the format's own reading procedure and finds - pay-1-1 next: phase one, first task, nothing completed -7. starts pay-1-1 and stops where the prose hands over to building Further claims: -- the second task is neither started nor touched +- no task is started: the walk stops before any task is picked up diff --git a/tests/prose/cases/implementation-picks-first-task/assertion-state.cjs b/tests/prose/cases/implementation-picks-first-task/assertion-state.cjs index c065db590..125426f84 100644 --- a/tests/prose/cases/implementation-picks-first-task/assertion-state.cjs +++ b/tests/prose/cases/implementation-picks-first-task/assertion-state.cjs @@ -2,9 +2,8 @@ // The fixture state plus what the walk should have done: resume // detection initialised tracking and committed the start of -// implementation, the environment answer was recorded so it is never -// asked again, and the first task was started. No task work — the walk -// stops where the prose hands over to building. +// implementation, and the environment answer recorded so it is never +// asked again. No task is started — the walk stops before pickup. const fixture = require('./fixture-state.cjs'); const m = require('../../mainlines/feature.cjs'); @@ -17,6 +16,5 @@ module.exports = { h.engine('commit', m.WU, '-m', `impl(${m.WU}): start implementation`); h.write('.workflows/.state/environment-setup.md', 'No special setup required.\n'); h.engine('commit', '--workflows', '-m', `impl(${m.WU}): record environment setup`); - h.engine('task', 'start', m.WU, m.WU, `${m.WU}-1-1`); }, }; diff --git a/tests/prose/cases/implementation-picks-first-task/assertion/.recipe-hash b/tests/prose/cases/implementation-picks-first-task/assertion/.recipe-hash index a29d91438..b96080a96 100644 --- a/tests/prose/cases/implementation-picks-first-task/assertion/.recipe-hash +++ b/tests/prose/cases/implementation-picks-first-task/assertion/.recipe-hash @@ -1 +1 @@ -e963357b5150c81c98156db6f527db5bf50bce292bb39b107e14ca98c6f5366a +e2c25569e167ac5ea503277237f107306a620ad8f81ac078ba429ca2fbf8dbaf diff --git a/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/manifest.json b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/manifest.json index 23023a119..1adb2fd34 100644 --- a/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/manifest.json +++ b/tests/prose/cases/implementation-picks-first-task/assertion/.workflows/pay/manifest.json @@ -62,7 +62,7 @@ "linters": [], "project_skills": [], "current_phase": 1, - "current_task": "pay-1-1" + "current_task": null } } } diff --git a/tests/prose/cases/implementation-picks-first-task/case.json b/tests/prose/cases/implementation-picks-first-task/case.json index 2a3a9a5d0..6a43f6cd2 100644 --- a/tests/prose/cases/implementation-picks-first-task/case.json +++ b/tests/prose/cases/implementation-picks-first-task/case.json @@ -6,9 +6,7 @@ "skills/workflow-implementation-entry/references/validate-dependencies.md", "skills/workflow-implementation-entry/references/environment-check.md", "skills/workflow-implementation-process/SKILL.md", - "skills/workflow-implementation-process/references/environment-setup.md", - "skills/workflow-implementation-process/references/task-loop.md", - "skills/workflow-planning-process/references/output-formats/local-markdown/reading.md" + "skills/workflow-implementation-process/references/environment-setup.md" ], "answers": [ "none — no special setup is needed" diff --git a/tests/prose/cases/implementation-picks-first-task/fixture/.recipe-hash b/tests/prose/cases/implementation-picks-first-task/fixture/.recipe-hash index a29d91438..b96080a96 100644 --- a/tests/prose/cases/implementation-picks-first-task/fixture/.recipe-hash +++ b/tests/prose/cases/implementation-picks-first-task/fixture/.recipe-hash @@ -1 +1 @@ -e963357b5150c81c98156db6f527db5bf50bce292bb39b107e14ca98c6f5366a +e2c25569e167ac5ea503277237f107306a620ad8f81ac078ba429ca2fbf8dbaf diff --git a/tests/prose/cases/smoke-boot-in-world/act.md b/tests/prose/cases/smoke-boot-in-world/act.md index bb4d2a56b..115963792 100644 --- a/tests/prose/cases/smoke-boot-in-world/act.md +++ b/tests/prose/cases/smoke-boot-in-world/act.md @@ -1,4 +1,4 @@ -In the project, run the boot command exactly as the initialisation prose -of skills/workflow-start/SKILL.md prescribes — sandbox concerns do not -apply in this world, so run it directly. Capture the JSON response, then -stop; do not continue into any confirmation or later step. +Execute the initialisation portion of skills/workflow-start/SKILL.md +from its first step — sandbox concerns do not apply in this world, so +run any command it prescribes directly. Stop once initialisation is +complete and the skill would move on to showing the user's work. diff --git a/tests/prose/cases/smoke-boot-in-world/assert.md b/tests/prose/cases/smoke-boot-in-world/assert.md index da99ef3ce..b91fedd12 100644 --- a/tests/prose/cases/smoke-boot-in-world/assert.md +++ b/tests/prose/cases/smoke-boot-in-world/assert.md @@ -1,10 +1,15 @@ The prose should have taken this path: -1. runs the boot command and reports the knowledge base ready, in - keyword-only mode -2. reports no migrations applied — this world is already migrated — so - the migrations summary and its confirmation are not raised +1. loads the shared casing conventions before any state is read +2. runs the boot pipeline as a single engine call +3. the response reports no migrations applied — this world is already + migrated — so the migrations summary and its confirmation gate are + not raised, and the skill says so rather than stopping +4. the response reports the knowledge base ready, so the knowledge gate + is not entered and initialisation completes Further claims: - booting an already-migrated project changes nothing on disk +- the user is asked nothing: neither gate in initialisation can fire + against a world that is already migrated and already has a store diff --git a/tests/prose/lib/prompts.cjs b/tests/prose/lib/prompts.cjs index 1651e7a8d..87e65cf5d 100644 --- a/tests/prose/lib/prompts.cjs +++ b/tests/prose/lib/prompts.cjs @@ -75,10 +75,11 @@ function walkerPrompt({ worldDir, root, situation, task, scope, stubs, answers } return `${parts.join('\n\n')}\n`; } -function asserterPrompt({ expected, world }) { +function asserterPrompt({ expected, world, actions }) { const t = loadTemplate('asserter'); const parts = [fill(t.main, { expected })]; if (world) parts.push(fill(t.world, { expecting: world.expecting, delta: world.delta })); + if (actions) parts.push(fill(t.actions, { actions })); parts.push(t.transcript); return `${parts.join('\n\n')}\n`; } diff --git a/tests/prose/lib/record-action.cjs b/tests/prose/lib/record-action.cjs new file mode 100644 index 000000000..e8a47c71a --- /dev/null +++ b/tests/prose/lib/record-action.cjs @@ -0,0 +1,67 @@ +#!/usr/bin/env node +'use strict'; + +// PostToolUse hook for the prose-walker agent: records what the walker +// ACTUALLY did, as the harness sees it. +// +// Declared in .claude/agents/prose-walker.md frontmatter, so it fires +// only while a walker is active. The walker plays no part in it — it +// cannot forget an entry, summarise one away, or write the log late, +// which is exactly why the log exists. Walkers were repeatedly found +// doing a full walk and then reporting only its tail; the narrative is +// now evidence of reasoning alone, and this file is the evidence of +// action. +// +// Self-scoping: every prose world lives at a temp path containing +// `prose-world-`. The payload is scanned for that path, and anything +// happening outside a world is ignored. Nothing is configured per run. +// +// The log lands at /.walk-actions.log, which collectTree() +// excludes, so recording never shows up as a world difference. + +const fs = require('fs'); +const path = require('path'); + +const LOG = '.walk-actions.log'; +const WORLD = /(^|[\s"'`])(\/[^\s"'`]*\/prose-world-[A-Za-z0-9]+)/; + +function read() { + try { + return JSON.parse(fs.readFileSync(0, 'utf8')); + } catch { + return null; + } +} + +/** The salient argument, per tool — what the claim will be about. */ +function summarise(tool, input) { + if (!input || typeof input !== 'object') return ''; + const raw = input.command || input.file_path || input.pattern || input.path || ''; + return String(raw).replace(/\s+/g, ' ').trim(); +} + +function main() { + const payload = read(); + if (!payload) return; + + const probe = JSON.stringify(payload); + const found = probe.match(WORLD); + // Not a prose world: this hook has nothing to say. + if (!found) return; + const world = found[2].replace(/\\+/g, ''); + + if (!fs.existsSync(world)) return; + + const tool = payload.tool_name || '?'; + const detail = summarise(tool, payload.tool_input); + const failed = payload.tool_output_is_error ? ' [ERROR]' : ''; + const line = `${tool}${failed}\t${detail.replace(world, '.')}\n`; + + try { + fs.appendFileSync(path.join(world, LOG), line); + } catch { + // A hook must never break the walk it is observing. + } +} + +main(); diff --git a/tests/prose/lib/worlds.cjs b/tests/prose/lib/worlds.cjs index 2c4697889..566bb03eb 100644 --- a/tests/prose/lib/worlds.cjs +++ b/tests/prose/lib/worlds.cjs @@ -37,6 +37,9 @@ const CLOCK = path.join(__dirname, 'fake-clock.cjs'); const GITIGNORE = '.gitignore'; const GITIGNORE_ESCAPED = '_gitignore.fixture'; const HASH_FILE = '.recipe-hash'; +// Written by the walker's PostToolUse hook (lib/record-action.cjs) — +// observation of the walk, not part of the world it acted on. +const ACTION_LOG = '.walk-actions.log'; // --- the recipe harness --------------------------------------------------- @@ -101,7 +104,7 @@ function runRecipe(caseId, which) { function excluded(rel) { const parts = rel.split(path.sep); if (parts.includes('.git')) return true; - if (rel === HASH_FILE) return true; + if (rel === HASH_FILE || rel === ACTION_LOG) return true; if (rel === path.join('.workflows', '.knowledge')) return true; if (rel.startsWith(path.join('.workflows', '.knowledge') + path.sep)) return true; if (rel.startsWith(path.join('.claude', 'skills') + path.sep)) return true; @@ -319,8 +322,22 @@ function destroyWorld(dir) { fs.rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); } +/** What the walk actually did, as recorded by the walker's hook. */ +function readActionLog(worldDir) { + const file = path.join(worldDir, ACTION_LOG); + if (!fs.existsSync(file)) return null; + return fs.readFileSync(file, 'utf8') + .split('\n') + .filter(Boolean) + .map((l, i) => { + const [tool, detail = ''] = l.split('\t'); + return `${String(i + 1).padStart(3)}. ${tool.padEnd(10)} ${detail}`; + }) + .join('\n'); +} + module.exports = { - ROOT, ENGINE, KNOWLEDGE, MAINLINES_DIR, WORLD_PREFIX, + ROOT, ENGINE, KNOWLEDGE, MAINLINES_DIR, WORLD_PREFIX, ACTION_LOG, readActionLog, runRecipe, collectTree, readSnapshot, snapshotDir, recipeHash, storedHash, writeSnapshot, verifySnapshot, diffWorld, buildWorld, destroyWorld, }; diff --git a/tests/prose/prompts/asserter.md b/tests/prose/prompts/asserter.md index da1a122ed..5e680e730 100644 --- a/tests/prose/prompts/asserter.md +++ b/tests/prose/prompts/asserter.md @@ -21,6 +21,17 @@ the expected world, computed by code: {{delta}} +=== actions === + +RECORDED ACTIONS — every tool call the walker made, in order, captured by a +harness hook as it happened. The walker did not write this and could +not have edited or omitted from it. This is what the walk *did*. + +{{actions}} + === transcript === -The walk's transcript follows under `=== TRANSCRIPT ===`. +The walk's own account follows under `=== TRANSCRIPT ===`. It is the +walker's narrative: evidence of reasoning — which arm it entered, which +guard it read, what it emitted — and nothing more. Where it disagrees +with the recorded actions about what was done, the recorded actions win. diff --git a/tests/prose/run.cjs b/tests/prose/run.cjs index 327777d57..03c655125 100644 --- a/tests/prose/run.cjs +++ b/tests/prose/run.cjs @@ -143,8 +143,10 @@ function cmdDiff(argv) { function cmdAssert(argv) { const c = getCase(argv[0]); let world = null; + let actions = null; if (c.hasFixtureState) { - const delta = worlds.diffWorld(c.id, requireWorld(argv, c)); + const dir = requireWorld(argv, c); + const delta = worlds.diffWorld(c.id, dir); world = { expecting: delta.expecting, delta: [ @@ -152,8 +154,20 @@ function cmdAssert(argv) { ...delta.changed, ].join('\n'), }; + actions = worlds.readActionLog(dir); + // No log is the harness failing, not the walk: a walk in a world + // always runs commands, and the walker's hook records every one. + // Refuse loudly rather than let an agent judge on the narrative + // alone — which is the thing the recording exists to replace. + if (!actions) { + die(`no action log at ${path.join(dir, worlds.ACTION_LOG)} — the prose-walker ` + + 'PostToolUse hook did not fire, so there is no record of what the walk did.\n' + + 'Check the hooks block in .claude/agents/prose-walker.md, that the agent ' + + 'registry has reloaded since it changed, and that this project is trusted ' + + '(hasTrustDialogAccepted). Do not judge this run.'); + } } - process.stdout.write(prompts.asserterPrompt({ expected: c.assert, world })); + process.stdout.write(prompts.asserterPrompt({ expected: c.assert, world, actions })); } // --- snap / verify --------------------------------------------------------