From a921fb838ebd69edb8fe4a39cf18049a1866706b Mon Sep 17 00:00:00 2001 From: Lee Overy Date: Sun, 26 Jul 2026 19:24:55 +0100 Subject: [PATCH] feat(prose-tests): judge the walk as told, not the summary returned MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An agent returns one message. A walk happens across dozens of turns. We were reading only the return value and judging it as though it were the whole walk — so every step the walker narrated at the time and then compressed out of its closing summary read as a step never taken. This is what produced the under-reporting we spent a long time blaming on the walker, and then on the model. It was neither. A Sonnet walk of continue-feature was marked down for skipping the selection menu; its transcript shows it emitted the display, emitted the menu verbatim, stopped, consumed the scripted answer and validated the selection. An Opus walk of the same case made byte-identical tool calls. The evidence was there the whole time and we were discarding it. The runtime already writes every turn to a transcript, and the stop payload names its path. The hook now lifts the turns into the world beside the action log, where the judging happens. Nothing is asked of the walker, which is what makes it dependable — the same reason the action log is trusted. That in turn lets the walker be asked for less. It no longer has to make its final message a complete log, which was the instruction driving the compression; commands, outputs and file writes come off its list entirely, since the harness records those with more fidelity than a narrative can. What remains is only what no tool call can show: the arm entered, the guard that selected it, and what went on screen. The asserter drops the caveat that a thin narrative is not a missing step. It existed to paper over this blindness and is now false — the walk it receives is complete, so an absent step is absent. Missing walk data is a harness failure, refused loudly like a missing action log, never quietly downgraded to judging the closing summary. Co-Authored-By: Claude Opus 5 --- .claude/agents/prose-asserter.md | 30 ++++++++-------- .claude/agents/prose-orchestrator.md | 12 +++---- .claude/agents/prose-walker.md | 37 ++++++++------------ tests/prose/lib/prompts.cjs | 4 +-- tests/prose/lib/record-action.cjs | 40 +++++++++++++++++++--- tests/prose/lib/worlds.cjs | 16 +++++++-- tests/prose/prompts/asserter.md | 17 ++++++--- tests/prose/run.cjs | 16 ++++++++- tests/scripts/test-prose-record-action.cjs | 29 ++++++++++++++++ 9 files changed, 143 insertions(+), 58 deletions(-) diff --git a/.claude/agents/prose-asserter.md b/.claude/agents/prose-asserter.md index c635f3189..1541c6592 100644 --- a/.claude/agents/prose-asserter.md +++ b/.claude/agents/prose-asserter.md @@ -19,7 +19,7 @@ strictly forbidden. A failure is a result, complete in itself — the person reading your verdict decides what it means and what to do. **Use no tools.** Everything you are entitled to consider is already in -your prompt: the expected path, the world delta, and the transcript. Do +your prompt: the expected path, the world delta, and the walk. Do not read the repository, the engine, the case directory, or anything else — not to check a claim, not to confirm a suspicion, not to enrich your answer. If it is not in the prompt, it is not evidence, and its @@ -31,10 +31,10 @@ tools you appear to have or what mode you are running in. - **Never explain a failure.** Report what the evidence shows and stop. Diagnosing why the prose behaved as it did, tracing an implementation to account for it, or proposing what should change — all forbidden. - "The step is missing from the transcript" is the finding, whole. -- A PASS on any path step **requires a quote** from the transcript that + "The step is missing from the walk" is the finding, whole. +- A PASS on any path step **requires a quote** from the walk that shows it. A PASS without a quote is invalid — mark it FAIL. -- Judge the path against what the transcript records, and the world +- Judge the path against what the walk records, and the world against the delta. Never infer one from the other. - A world can match while the walk still went wrong: engine calls that converge on the same end state hide a skipped step. An empty or @@ -53,21 +53,21 @@ were written. A walker has been known to describe a command's output inaccurately; where its account and the record disagree about what came back, the record is right and the disagreement is a finding. -**The transcript** is the walker's own account. It is the authority on -*reasoning* — which arm it entered, which guard line selected it, what -it emitted to the user — because none of that appears in a tool call. +**The walk** is the walker's own account, turn by turn, as it was told at +the time — lifted from the runtime's transcript by the harness, not +summarised and not editable after the fact. It is the authority on +*reasoning*: which arm was entered, which guard line selected it, what +was emitted to the user, because none of that appears in a tool call. Judge each expected step against the right one. A step about an action ("records the dispatch through the engine", "promotes with a scan then -closes with an incorporate") is evidenced from the recorded actions, -even when the transcript omits it — walkers routinely under-report, and -a thin narrative is not a missing step. A step about reasoning or -output is evidenced from the transcript. Where the two disagree about -what was done, the recorded actions win. +closes with an incorporate") is evidenced from the recorded actions. A +step about reasoning or output is evidenced from the walk. Where the two +disagree about what was done, the recorded actions win. ## An invalid walk is not a failing walk -Check the **recorded actions**, not the transcript, for where the walk +Check the **recorded actions**, not the walk, for where the walk began. If the actions show it starting mid-flow — the earlier steps never attempted, and no `DEVIATION` recorded for them — then the walk cannot evidence the path, and judging it would report a prose failure @@ -88,13 +88,13 @@ Return exactly this and nothing else: only as trustworthy as the model that produced the walk, so it is stated before anything is judged, never inferred, and never omitted. 1. **Path** — one line per expected step: PASS or FAIL, each PASS quoting - the transcript line that shows it, each FAIL stating what is missing + the line of the walk that shows it, each FAIL stating what is missing or contradicting. 2. **World** — PASS or FAIL. Enumerate EVERY difference in the delta and classify each volatile or material. Any material difference fails. An empty delta passes. 3. **Markers** — list every `UNSCRIPTED QUESTION`, `AMBIGUOUS`, and - `DEVIATION` in the transcript. Each is a finding in its own right, + `DEVIATION` in the walk. Each is a finding in its own right, even when everything else passed. 4. **VERDICT** — PASS, FAIL, or INVALID WALK, then one sentence on what it means for the prose. diff --git a/.claude/agents/prose-orchestrator.md b/.claude/agents/prose-orchestrator.md index 5db9c4c68..06564ca76 100644 --- a/.claude/agents/prose-orchestrator.md +++ b/.claude/agents/prose-orchestrator.md @@ -49,14 +49,14 @@ a walk. 2. **Walk** — `node tests/prose/run.cjs prompt --world `. Dispatch the **prose-walker** agent with that output as its prompt, - verbatim and unmodified. Keep the account it returns. Its tool calls - are recorded into the world by its own hook — you do nothing to - collect them. + verbatim and unmodified. Its tool calls and every turn it takes are + recorded into the world by its own hook — you do nothing to collect + them, and the message it returns to you is not evidence. 3. **Assert** — `node tests/prose/run.cjs assert --world `. - Dispatch the **prose-asserter** agent with that output, followed by - the walker's account under a `=== TRANSCRIPT ===` line. Keep its - verdict. + Dispatch the **prose-asserter** agent with that output verbatim. It + already carries the walk; never append the walker's returned message + to it, and never paste it anywhere else. If that command **fails** rather than printing a prompt, the harness is broken, not the prose. Stop there, destroy the world, and report diff --git a/.claude/agents/prose-walker.md b/.claude/agents/prose-walker.md index fcc802ae7..bfb5ddf8d 100644 --- a/.claude/agents/prose-walker.md +++ b/.claude/agents/prose-walker.md @@ -91,33 +91,27 @@ Record these inline, exactly as named, the moment they occur: Stop at the task's stop condition, the end of the flow, an `UNSCRIPTED QUESTION`, or a hard error — whichever comes first. -## Transcript +## Narrate as you go -Your entire final output is the transcript. Return nothing else — no -preamble, no assessment, no closing summary. +**Write each entry as it happens, not afterwards.** The harness captures +every turn you take, so the account it keeps is built from what you say +along the way. Your final message is not the record and does not need to +recap the walk — a summary written at the end is worth less than the +entries written at the time. -**It is a log, not a recollection.** Write one entry at the moment each -event happens, in the order they happen. Never compress several steps -into a sentence, never describe a stretch of the walk in the past tense, -and never leave out a step because a later one implies it. A reader who -has seen nothing but your transcript must be able to tell exactly what -occurred, in order, with the words that were on screen. +Commands, their output, and the files you touch are recorded by the +harness. Do not restate them. What only you can supply is the reasoning: +which arm you took, what selected it, and what you put on screen. -**If you did it, it is in the log.** Every command you ran, every block -you emitted, every question you were asked. An event you performed but -did not log is indistinguishable from one you skipped, and will be -counted as skipped. - -Log these, each as its own entry: +Narrate these, each as its own entry, the moment it happens: 1. Every prose section or arm entered: `file.md § Heading`, plus the quoted guard line that selected it. -2. Every command run, and the first line of its output. -3. Every block the prose directed you to emit, quoted in full. -4. Every menu or question encountered, verbatim, and the scripted answer +2. Every block the prose directed you to emit, quoted in full. +3. Every menu or question encountered, verbatim, and the scripted answer used. -5. Every file written or edited (path only), and every marker above. -6. Finally: `STOPPED: `. +4. Every marker above. +5. Finally: `STOPPED: `. The shape, abbreviated: @@ -133,9 +127,6 @@ EMITTED (menu): ANSWERED: yes — do the thing (scripted answer 1) ENTERED: some-reference.md § A — #### If `yes` guard: "#### If `yes`" -RAN: node .claude/skills/workflow-engine/scripts/engine.cjs thing do wu topic - → {"ok":true,"id":"thing-001","file":".workflows/.cache/…/thing-001.md"} -WROTE: .workflows/.cache/…/thing-001.md SUBSTITUTED: the-stub-name STOPPED: the reference returned to its caller ``` diff --git a/tests/prose/lib/prompts.cjs b/tests/prose/lib/prompts.cjs index 50a13107e..b6a7cb1a4 100644 --- a/tests/prose/lib/prompts.cjs +++ b/tests/prose/lib/prompts.cjs @@ -75,13 +75,13 @@ function walkerPrompt({ worldDir, root, situation, task, scope, stubs, answers } return `${parts.join('\n\n')}\n`; } -function asserterPrompt({ expected, world, actions, substitutions }) { +function asserterPrompt({ expected, world, actions, walk, substitutions }) { const t = loadTemplate('asserter'); const parts = [fill(t.main, { expected })]; if (world) parts.push(fill(t.world, { expecting: world.expecting, delta: world.delta })); if (actions) parts.push(fill(t.actions, { actions })); if (substitutions) parts.push(fill(t.substitutions, { substitutions })); - parts.push(t.transcript); + if (walk) parts.push(fill(t.walk, { walk })); return `${parts.join('\n\n')}\n`; } diff --git a/tests/prose/lib/record-action.cjs b/tests/prose/lib/record-action.cjs index 66e452d03..c58e78ac7 100644 --- a/tests/prose/lib/record-action.cjs +++ b/tests/prose/lib/record-action.cjs @@ -39,6 +39,7 @@ const fs = require('fs'); const path = require('path'); const LOG = '.walk-actions.log'; +const WALK = '.walk-transcript.log'; const VIOLATIONS = 'tests/prose/.agent-tool-use.log'; const WORLD = /(^|[\s"'`])(\/[^\s"'`]*\/prose-world-[A-Za-z0-9]+)/; const MAX_OUTPUT = 400; @@ -94,7 +95,7 @@ function responseText(response, limit) { * neither of which a stop payload states. */ function fromTranscript(transcriptPath) { - const blank = { world: null, model: '' }; + const blank = { world: null, model: '', turns: [] }; if (!transcriptPath) return blank; let raw; try { @@ -103,19 +104,49 @@ function fromTranscript(transcriptPath) { return blank; } const models = new Set(); + const turns = []; for (const line of raw.split('\n')) { + let entry; try { - const model = JSON.parse(line)?.message?.model; - if (model) models.add(model); - } catch { /* a partial line is not worth failing the record over */ } + entry = JSON.parse(line); + } catch { continue; /* a partial line is not worth failing the record over */ } + const message = entry && entry.message; + if (!message) continue; + if (message.model) models.add(message.model); + if (!Array.isArray(message.content)) continue; + for (const block of message.content) { + if (block && block.type === 'text' && block.text.trim()) turns.push(block.text.trim()); + } } const found = raw.match(WORLD); return { world: found ? found[2].replace(/\\+/g, '') : null, model: [...models].join(',') || '', + turns, }; } +/** + * The walk as it was actually told, turn by turn. + * + * An agent returns one final message, and a walk runs across dozens of + * turns — so a caller who reads only the return value sees a summary the + * walker wrote after the fact, and every step compressed out of it looks + * like a step never taken. The turns are all in the runtime's transcript + * already; this lifts them into the world beside the action log, where + * the judging is done. Nothing is asked of the walker, which is what + * makes it dependable. + */ +function writeWalk(world, turns) { + if (!turns || !turns.length) return; + try { + fs.writeFileSync( + path.join(world, WALK), + `${turns.join('\n\n---\n\n').split(world).join('.')}\n`, + ); + } catch { /* a hook must never break what it observes */ } +} + function main() { const payload = read(); if (!payload) return; @@ -172,6 +203,7 @@ function main() { ); } else if (stop) { parts.push(flatten(payload.last_assistant_message, MAX_OUTPUT).split(world).join('.')); + writeWalk(world, traced.turns); } try { diff --git a/tests/prose/lib/worlds.cjs b/tests/prose/lib/worlds.cjs index ec6b87d71..6cc652a77 100644 --- a/tests/prose/lib/worlds.cjs +++ b/tests/prose/lib/worlds.cjs @@ -40,6 +40,10 @@ const HASH_FILE = '.recipe-hash'; // Written by the walker's PostToolUse hook (lib/record-action.cjs) — // observation of the walk, not part of the world it acted on. const ACTION_LOG = '.walk-actions.log'; +// Written by the same hook at SubagentStop: the walk as it was told, +// turn by turn, lifted from the runtime's own transcript. An agent +// returns one final message; the walk happens across dozens of turns. +const WALK_LOG = '.walk-transcript.log'; // --- the recipe harness --------------------------------------------------- @@ -104,7 +108,7 @@ function runRecipe(caseId, which) { function excluded(rel) { const parts = rel.split(path.sep); if (parts.includes('.git')) return true; - if (rel === HASH_FILE || rel === ACTION_LOG) return true; + if (rel === HASH_FILE || rel === ACTION_LOG || rel === WALK_LOG) return true; if (rel === path.join('.workflows', '.knowledge')) return true; if (rel.startsWith(path.join('.workflows', '.knowledge') + path.sep)) return true; if (rel.startsWith(path.join('.claude', 'skills') + path.sep)) return true; @@ -343,8 +347,16 @@ function readActionLog(worldDir) { .join('\n'); } +/** The walk as it was told, turn by turn — not the summary it returned. */ +function readWalkLog(worldDir) { + const file = path.join(worldDir, WALK_LOG); + if (!fs.existsSync(file)) return null; + return fs.readFileSync(file, 'utf8').trim() || null; +} + module.exports = { - ROOT, ENGINE, KNOWLEDGE, MAINLINES_DIR, WORLD_PREFIX, ACTION_LOG, readActionLog, + ROOT, ENGINE, KNOWLEDGE, MAINLINES_DIR, WORLD_PREFIX, + ACTION_LOG, readActionLog, WALK_LOG, readWalkLog, runRecipe, collectTree, readSnapshot, snapshotDir, recipeHash, storedHash, writeSnapshot, verifySnapshot, diffWorld, buildWorld, destroyWorld, }; diff --git a/tests/prose/prompts/asserter.md b/tests/prose/prompts/asserter.md index f7a31f191..ff8f54bc6 100644 --- a/tests/prose/prompts/asserter.md +++ b/tests/prose/prompts/asserter.md @@ -46,9 +46,16 @@ a missing step. {{substitutions}} -=== transcript === +=== walk === + +THE WALK, TURN BY TURN — the walker's own account as it was told at the +time, lifted from the runtime's transcript by the harness. It was not +summarised, and the walker had no chance to edit it after the fact. + +Read it for what only it can show: which section or arm was entered and +the guard line that selected it, the blocks emitted to the user, the +questions asked and the scripted answers consumed. What a command ran and +returned is settled by the recorded actions above, not here. + +{{walk}} -The walk's own account follows under `=== TRANSCRIPT ===`. It is the -walker's narrative: evidence of reasoning — which arm it entered, which -guard it read, what it emitted — and nothing more. Where it disagrees -with the recorded actions about what was done, the recorded actions win. diff --git a/tests/prose/run.cjs b/tests/prose/run.cjs index 66b4322a3..bc6c4b251 100644 --- a/tests/prose/run.cjs +++ b/tests/prose/run.cjs @@ -144,6 +144,7 @@ function cmdAssert(argv) { const c = getCase(argv[0]); let world = null; let actions = null; + let walk = null; if (c.hasFixtureState) { const dir = requireWorld(argv, c); const delta = worlds.diffWorld(c.id, dir, c.worldMode === 'claims'); @@ -166,6 +167,17 @@ function cmdAssert(argv) { + 'registry has reloaded since it changed, and that this project is trusted ' + '(hasTrustDialogAccepted). Do not judge this run.'); } + walk = worlds.readWalkLog(dir); + // Same stance as the action log: the walk is harness-captured, so its + // absence is a broken hook, not a quiet walk. Judging without it would + // fall back to whatever the walker chose to say at the end — the very + // summary this record exists to replace. + if (!walk) { + die(`no walk log at ${path.join(dir, worlds.WALK_LOG)} — the prose-walker ` + + 'SubagentStop hook did not fire, so there is no turn-by-turn record of ' + + 'the walk.\nCheck the hooks block in .claude/agents/prose-walker.md and ' + + 'that the agent registry has reloaded since it changed. Do not judge this run.'); + } } const substitutions = c.stubs.length ? c.stubs.map((s) => { @@ -173,7 +185,9 @@ function cmdAssert(argv) { return `- ${s.name} — fires ${s.trigger}\n ${stub.description.replace(/\n/g, '\n ')}`; }).join('\n') : null; - process.stdout.write(prompts.asserterPrompt({ expected: c.assert, world, actions, substitutions })); + process.stdout.write( + prompts.asserterPrompt({ expected: c.assert, world, actions, walk, substitutions }), + ); } // --- snap / verify -------------------------------------------------------- diff --git a/tests/scripts/test-prose-record-action.cjs b/tests/scripts/test-prose-record-action.cjs index 06f11da9e..c0b3b1129 100644 --- a/tests/scripts/test-prose-record-action.cjs +++ b/tests/scripts/test-prose-record-action.cjs @@ -213,6 +213,35 @@ describe('prose recorder — the stop event', () => { assert.equal(logLines()[0].split('\t')[2], 'claude-sonnet-5'); }); + it('lifts the walk out turn by turn, not just the message returned', () => { + const transcript = writeTranscript([ + { message: { model: 'claude-sonnet-5', content: [{ type: 'text', text: 'ENTERED: a.md § Step 1' }] }, cwd: world }, + { message: { content: [{ type: 'tool_use', name: 'Bash' }] } }, + { message: { content: [{ type: 'text', text: 'EMITTED (menu):\n Which feature?' }] } }, + { message: { content: [{ type: 'text', text: 'STOPPED: end of flow' }] } }, + ]); + fire({ + hook_event_name: 'SubagentStop', + agent_type: 'prose-walker', + agent_transcript_path: transcript, + last_assistant_message: 'STOPPED: end of flow', + }); + const walk = fs.readFileSync(path.join(world, '.walk-transcript.log'), 'utf8'); + assert.ok(walk.includes('ENTERED: a.md § Step 1'), 'an early turn survives'); + assert.ok(walk.includes('Which feature?'), 'and what it emitted mid-walk'); + assert.ok(walk.includes('STOPPED: end of flow')); + assert.ok(walk.indexOf('ENTERED') < walk.indexOf('Which feature?'), 'in the order they happened'); + }); + + it('writes no walk file when the transcript holds no turns', () => { + fire({ + hook_event_name: 'SubagentStop', + agent_type: 'prose-walker', + agent_transcript_path: writeTranscript([{ message: { model: 'x' }, cwd: world }]), + }); + assert.equal(fs.existsSync(path.join(world, '.walk-transcript.log')), false); + }); + it('stays silent when a stop names no readable transcript', () => { // A real transcript lives outside the world, so an unreadable one // leaves nothing anywhere in the payload to scope the record to.