Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 29 additions & 9 deletions .claude/agents/prose-asserter.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,18 +38,38 @@ tools you appear to have or what mode you are running in.
by nature and are immaterial. Differences in shape, field presence,
status vocabulary, or content are material.

## Two kinds of evidence, and which one wins

**Recorded actions** are captured by a harness hook as each tool call
happens. The walker did not write them and could not have edited them.
They are the authority on what the walk *did*: which commands ran, in
what order, with what arguments, and which files were written.

**The transcript** is the walker's own account. It is the authority on
*reasoning* — which arm it entered, which guard line selected it, what
it emitted to the user — because none of that appears in a tool call.

Judge each expected step against the right one. A step about an action
("records the dispatch through the engine", "promotes with a scan then
closes with an incorporate") is evidenced from the recorded actions,
even when the transcript omits it — walkers routinely under-report, and
a thin narrative is not a missing step. A step about reasoning or
output is evidenced from the transcript. Where the two disagree about
what was done, the recorded actions win.

## An invalid walk is not a failing walk

Before judging anything, check where the transcript begins. If its first
entry is not the start of the walk the expected path describes — the
walker jumped into the middle of the flow, skipping earlier steps
without recording a `DEVIATION` for each — then the walk cannot evidence
the path, and judging it would report a prose failure that was never
demonstrated.
Check the **recorded actions**, not the transcript, for where the walk
began. If the actions show it starting mid-flow — the earlier steps
never attempted, and no `DEVIATION` recorded for them — then the walk
cannot evidence the path, and judging it would report a prose failure
that was never demonstrated.

In that case return `VERDICT: INVALID WALK`, name the entry the
transcript opens on and the step it should have opened on, and judge
nothing further. This is a fault in the walk, not in the prose.
In that case return `VERDICT: INVALID WALK`, name the action it opens
on and the step it should have opened on, and judge nothing further.
This is a fault in the walk, not in the prose. A walk that *did* the
work and merely described it poorly is **not** invalid — judge it from
the actions.

## Verdict

Expand Down
13 changes: 10 additions & 3 deletions .claude/agents/prose-orchestrator.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,20 @@ a walk.

2. **Walk** — `node tests/prose/run.cjs prompt <case-id> --world <dir>`.
Dispatch the **prose-walker** agent with that output as its prompt,
verbatim and unmodified. Keep the transcript it returns.
verbatim and unmodified. Keep the account it returns. Its tool calls
are recorded into the world by its own hook — you do nothing to
collect them.

3. **Assert** — `node tests/prose/run.cjs assert <case-id> --world <dir>`.
Dispatch the **prose-asserter** agent with that output, followed by
the walk transcript under a `=== TRANSCRIPT ===` line. Keep its
the walker's account under a `=== TRANSCRIPT ===` line. Keep its
verdict.

If that command **fails** rather than printing a prompt, the harness
is broken, not the prose. Stop there, destroy the world, and report
`VERDICT: HARNESS ERROR` with the message it printed. Never fall back
to judging a walk with no record of what it did.

Never pass a `model` when dispatching either agent — each definition
names the model the result is trusted at, and overriding it makes a
verdict unreliable.
Expand Down Expand Up @@ -77,7 +84,7 @@ Return exactly this and nothing else:

```
CASE: <case-id>
VERDICT: PASS | FAIL | FLAKY | INVALID
VERDICT: PASS | FAIL | FLAKY | INVALID | HARNESS ERROR
PATH: <n>/<total> steps passed
WORLD: PASS | FAIL
MARKERS: <none, or one line each>
Expand Down
6 changes: 6 additions & 0 deletions .claude/agents/prose-walker.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@ name: prose-walker
description: Executes workflow prose exactly as a live session would, against a disposable test world, and returns a transcript of what it did. Dispatched by prose-orchestrator during a prose-test run.
tools: Read, Write, Edit, Bash, Glob, Grep
model: opus
hooks:
PostToolUse:
- matcher: "Bash|Write|Edit|Read"
hooks:
- type: command
command: "node \"$CLAUDE_PROJECT_DIR/tests/prose/lib/record-action.cjs\""
---

# Prose Walker
Expand Down
15 changes: 15 additions & 0 deletions design/prose-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,21 @@ testing.
failing case re-runs once from a fresh world at the same models. A
defect in the prose reproduces; a one-off does not, and is reported
as FLAKY with both runs quoted. Nothing is auto-resolved.
- **P6e — what the walk did is recorded by the harness, not reported by
the walker.** A `PostToolUse` hook declared in the walker's own agent
frontmatter (so it fires for walkers and nothing else) appends every
tool call — command, path, error flag — to `.walk-actions.log` in the
world, excluded from the diff. Evidence then splits by kind: the
**recorded actions** are the authority on what was *done*, the
walker's transcript on what was *reasoned* (which arm, which guard,
what was emitted), and where they disagree the recording wins.
Measured, not assumed: a diagnostic walker's world held the agent row
at `incorporated` and the report file — it had run dispatch, stub,
scan and incorporate — while its transcript began at the fifth step.
Two attempts to fix that with wording (a stricter log format, then
Opus) both failed, because a narrative an agent composes at the end
is a recollection however it is instructed. Asking it to write a log
as it goes would have relocated the same trust, not removed it.
- **P6d — a walk that began mid-flow proves nothing.** Walkers skip
steps whose effect the world already holds — "already booted", "the
plan already exists" — and then the case reports a prose failure at a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,4 @@ Implementation: {exists:[true|false]}
Dependencies: {All satisfied | List any notes}
Environment: {Setup required | No special setup required}
```
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Bug context:
- Expected behavior: {from the carrier / gather-context}
- Actual behavior: {from the carrier / gather-context}
- Initial context: {error messages, reproduction steps — from the carrier / gather-context, or "(none captured yet)"}
```

#### If source is `continue`

Expand All @@ -39,3 +40,4 @@ Investigation session for: {work_unit}
Source: existing investigation
Output: .workflows/{work_unit}/investigation/{topic}.md
```
2 changes: 2 additions & 0 deletions skills/workflow-planning-entry/references/invoke-skill.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Work unit: {work_unit}
Specification: .workflows/{work_unit}/specification/{topic}/specification.md
Additional context: {summary of user's additional context, or "none"}
Cross-cutting references: {list of applicable cross-cutting specs with brief summaries, or "none"}
```

#### If continuing or reviewing existing plan

Expand All @@ -31,3 +32,4 @@ Work unit: {work_unit}
Specification: .workflows/{work_unit}/specification/{topic}/specification.md
Existing plan: .workflows/{work_unit}/planning/{topic}/planning.md
Cross-cutting references: {list of applicable cross-cutting specs with brief summaries, or "none"}
```
1 change: 1 addition & 0 deletions skills/workflow-review-entry/references/invoke-skill.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,4 @@ Plans to review:
topic: {topic}
format: {format}
specification: .workflows/{work_unit}/specification/{topic}/specification.md (exists: {true|false})
```
1 change: 1 addition & 0 deletions skills/workflow-scoping-entry/references/invoke-skill.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,4 @@ Invoke the **workflow-scoping-process** skill (Skill tool) with the next fenced
Scoping session for: {topic}
Work unit: {work_unit}
Description: {description}
```
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Source material:

Work unit: {work_unit}
Action: {verb} specification
```

#### If `work_type` is `bugfix`

Expand All @@ -33,6 +34,7 @@ Source material:

Work unit: {work_unit}
Action: {verb} specification
```

#### If `work_type` is `epic`

Expand All @@ -51,6 +53,7 @@ Source material:
Work unit: {work_unit}
Topic: {topic}
Action: {verb} specification
```

#### If `work_type` is `cross-cutting`

Expand All @@ -65,3 +68,4 @@ Source material:

Work unit: {work_unit}
Action: {verb} specification
```
5 changes: 3 additions & 2 deletions tests/prose/cases/implementation-picks-first-task/act.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
Execute skills/workflow-implementation-entry/SKILL.md with arguments
$0=feature, $1=pay, then continue into the processing skill it invokes.
Follow the flow until the prose directs you to begin building the first
task. Stop there — write no implementation code and no tests.
Stop once the environment has been settled and the skill moves on to
its next concern — do not enter project-skill or linter discovery, and
write no implementation code.
5 changes: 1 addition & 4 deletions tests/prose/cases/implementation-picks-first-task/assert.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,7 @@ The prose should have taken this path:
resuming-from-a-previous-session note
5. environment setup records the answer as a setup document and commits
it, so the question is not asked again in a later session
6. reads the plan through the format's own reading procedure and finds
pay-1-1 next: phase one, first task, nothing completed
7. starts pay-1-1 and stops where the prose hands over to building

Further claims:

- the second task is neither started nor touched
- no task is started: the walk stops before any task is picked up
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,8 @@

// The fixture state plus what the walk should have done: resume
// detection initialised tracking and committed the start of
// implementation, the environment answer was recorded so it is never
// asked again, and the first task was started. No task work — the walk
// stops where the prose hands over to building.
// implementation, and the environment answer recorded so it is never
// asked again. No task is started — the walk stops before pickup.

const fixture = require('./fixture-state.cjs');
const m = require('../../mainlines/feature.cjs');
Expand All @@ -17,6 +16,5 @@ module.exports = {
h.engine('commit', m.WU, '-m', `impl(${m.WU}): start implementation`);
h.write('.workflows/.state/environment-setup.md', 'No special setup required.\n');
h.engine('commit', '--workflows', '-m', `impl(${m.WU}): record environment setup`);
h.engine('task', 'start', m.WU, m.WU, `${m.WU}-1-1`);
},
};
Original file line number Diff line number Diff line change
@@ -1 +1 @@
e963357b5150c81c98156db6f527db5bf50bce292bb39b107e14ca98c6f5366a
e2c25569e167ac5ea503277237f107306a620ad8f81ac078ba429ca2fbf8dbaf
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
"linters": [],
"project_skills": [],
"current_phase": 1,
"current_task": "pay-1-1"
"current_task": null
}
}
}
Expand Down
4 changes: 1 addition & 3 deletions tests/prose/cases/implementation-picks-first-task/case.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,7 @@
"skills/workflow-implementation-entry/references/validate-dependencies.md",
"skills/workflow-implementation-entry/references/environment-check.md",
"skills/workflow-implementation-process/SKILL.md",
"skills/workflow-implementation-process/references/environment-setup.md",
"skills/workflow-implementation-process/references/task-loop.md",
"skills/workflow-planning-process/references/output-formats/local-markdown/reading.md"
"skills/workflow-implementation-process/references/environment-setup.md"
],
"answers": [
"none — no special setup is needed"
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
e963357b5150c81c98156db6f527db5bf50bce292bb39b107e14ca98c6f5366a
e2c25569e167ac5ea503277237f107306a620ad8f81ac078ba429ca2fbf8dbaf
8 changes: 4 additions & 4 deletions tests/prose/cases/smoke-boot-in-world/act.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
In the project, run the boot command exactly as the initialisation prose
of skills/workflow-start/SKILL.md prescribes — sandbox concerns do not
apply in this world, so run it directly. Capture the JSON response, then
stop; do not continue into any confirmation or later step.
Execute the initialisation portion of skills/workflow-start/SKILL.md
from its first step — sandbox concerns do not apply in this world, so
run any command it prescribes directly. Stop once initialisation is
complete and the skill would move on to showing the user's work.
13 changes: 9 additions & 4 deletions tests/prose/cases/smoke-boot-in-world/assert.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
The prose should have taken this path:

1. runs the boot command and reports the knowledge base ready, in
keyword-only mode
2. reports no migrations applied — this world is already migrated — so
the migrations summary and its confirmation are not raised
1. loads the shared casing conventions before any state is read
2. runs the boot pipeline as a single engine call
3. the response reports no migrations applied — this world is already
migrated — so the migrations summary and its confirmation gate are
not raised, and the skill says so rather than stopping
4. the response reports the knowledge base ready, so the knowledge gate
is not entered and initialisation completes

Further claims:

- booting an already-migrated project changes nothing on disk
- the user is asked nothing: neither gate in initialisation can fire
against a world that is already migrated and already has a store
3 changes: 2 additions & 1 deletion tests/prose/lib/prompts.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,11 @@ function walkerPrompt({ worldDir, root, situation, task, scope, stubs, answers }
return `${parts.join('\n\n')}\n`;
}

function asserterPrompt({ expected, world }) {
function asserterPrompt({ expected, world, actions }) {
const t = loadTemplate('asserter');
const parts = [fill(t.main, { expected })];
if (world) parts.push(fill(t.world, { expecting: world.expecting, delta: world.delta }));
if (actions) parts.push(fill(t.actions, { actions }));
parts.push(t.transcript);
return `${parts.join('\n\n')}\n`;
}
Expand Down
67 changes: 67 additions & 0 deletions tests/prose/lib/record-action.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
#!/usr/bin/env node
'use strict';

// PostToolUse hook for the prose-walker agent: records what the walker
// ACTUALLY did, as the harness sees it.
//
// Declared in .claude/agents/prose-walker.md frontmatter, so it fires
// only while a walker is active. The walker plays no part in it — it
// cannot forget an entry, summarise one away, or write the log late,
// which is exactly why the log exists. Walkers were repeatedly found
// doing a full walk and then reporting only its tail; the narrative is
// now evidence of reasoning alone, and this file is the evidence of
// action.
//
// Self-scoping: every prose world lives at a temp path containing
// `prose-world-`. The payload is scanned for that path, and anything
// happening outside a world is ignored. Nothing is configured per run.
//
// The log lands at <world>/.walk-actions.log, which collectTree()
// excludes, so recording never shows up as a world difference.

const fs = require('fs');
const path = require('path');

const LOG = '.walk-actions.log';
const WORLD = /(^|[\s"'`])(\/[^\s"'`]*\/prose-world-[A-Za-z0-9]+)/;

function read() {
try {
return JSON.parse(fs.readFileSync(0, 'utf8'));
} catch {
return null;
}
}

/** The salient argument, per tool — what the claim will be about. */
function summarise(tool, input) {
if (!input || typeof input !== 'object') return '';
const raw = input.command || input.file_path || input.pattern || input.path || '';
return String(raw).replace(/\s+/g, ' ').trim();
}

function main() {
const payload = read();
if (!payload) return;

const probe = JSON.stringify(payload);
const found = probe.match(WORLD);
// Not a prose world: this hook has nothing to say.
if (!found) return;
const world = found[2].replace(/\\+/g, '');

if (!fs.existsSync(world)) return;

const tool = payload.tool_name || '?';
const detail = summarise(tool, payload.tool_input);
const failed = payload.tool_output_is_error ? ' [ERROR]' : '';
const line = `${tool}${failed}\t${detail.replace(world, '.')}\n`;

try {
fs.appendFileSync(path.join(world, LOG), line);
} catch {
// A hook must never break the walk it is observing.
}
}

main();
Loading