Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ Add or update a test alongside any change to engine scripts, adapters, migration

## Prose Tests

`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). A case is a directory under `tests/prose/cases/{case-id}/`: `case.json` (values code branches on, including the required `entry` and any deterministic `invariants`), `fixture.md` + `fixture-state.cjs` (the starting world), `act.md` (the coarse instruction a walker agent follows through the real skills), `assert.md` (the expected path, seen only by the asserting agent), optional `assertion-state.cjs` (the world the walk should produce; absent = unchanged), and the two generated snapshots. Nothing is parsed and prose never shares a file with code. **A case starts where a session starts and runs until state lands**: `entry` must name `workflow-start`, a `workflow-*-entry` skill, or `workflow-discovery` — never a navigation or processing skill, never a reference — because a walk begun mid-session lacks the context the prose was written for, and one that stops at a gate leaves an empty delta and nothing but display claims to judge. Claims that code can settle belong in `invariants` (`engine_before_write`, `calls_include`, `calls_exclude`), which run before the asserter and whose verdicts it cannot overturn. Shared pipeline stages live in `tests/prose/mainlines/`. Walks cost tokens — run on command via the `/prose-test` dev skill, never as part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation plus snapshot goldens — every snapshot must rebuild byte-identical from its recipe under the frozen clock, skipped when nothing feeding it changed. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {case-id}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases.
`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). A case is a directory under `tests/prose/cases/{case-id}/`: `case.json` (values code branches on: the required `entry`, deterministic `invariants`, scripted `answers` and/or a `conduct` description of the user, stub arming, `world: "claims"` for model-authored end states judged against stated claims), `fixture.md` + `fixture-state.cjs` (the starting world), `act.md` (the coarse instruction a walker agent follows through the real skills), `assert.md` (the expected path, seen only by the asserting agent), optional `assertion-state.cjs` (the world the walk should produce; absent = unchanged), and the two generated snapshots. Nothing is parsed and prose never shares a file with code. **A case starts where a session starts and stops where its claims can be answered**: `entry` must name `workflow-start`, a `workflow-*-entry` skill, or `workflow-discovery` — never a navigation or processing skill, never a reference — because a walk begun mid-session lacks the context the prose was written for. A handoff is a fine ending (recorded actions are as deterministic as a world delta); what a case must never do is stop where only display claims remain, since text put on screen is evidenced by no hook. Claims that code can settle belong in `invariants` (`engine_before_write` — declared only where the walk writes, `calls_include`, `calls_exclude`, `calls_in_order`), which run before the asserter and whose verdicts it cannot overturn. Shared pipeline stages live in `tests/prose/mainlines/`. Walks cost tokens — run on command via the `/prose-test` dev skill, never as part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation plus snapshot goldens — every snapshot must rebuild byte-identical from its recipe under the frozen clock, skipped when nothing feeding it changed. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {case-id}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases.

## Knowledge Base Subsystem

Expand Down
44 changes: 44 additions & 0 deletions design/prose-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,21 @@ testing.
recording and replaying one would freeze whatever the walker happened
to do, certifying broken prose instead of catching it.

- **P11 — the user is played two ways, and neither steers.** Scripted
`answers` cover discrete gates, one right response each, consumed in
arrival order. Where the prose explores open-endedly — an interview, a
shaping conversation — no turn count can be scripted, so `conduct`
describes the person instead and the walker answers as they would for
as long as the prose keeps asking. Conduct describes the user, never
the walk: it names no arm, ends no step, leaks no expectation — the
P4 boundary extended to roleplay — and any outcome a case depends on
(a menu choice carried as a disposition) is pinned by an invariant,
never trusted to the performance. The two modes never mix where an
open stretch precedes a scripted gate, because the queue is consumed
in arrival order and the interview would eat the menu's answer. A stub
replaces prose and untests it; conduct keeps prose live — where the
prose can genuinely run against a played user, conduct wins.

- **P7 — a failing case is a finding either way.** Either the prose
broke, or the world/design moved and the case is stale. The
adjudication is the point; only the typo-class staleness is
Expand Down Expand Up @@ -254,6 +269,35 @@ testing.

## Log

- 2026-07-27 — Day two: the evidence pipeline completed, conduct built,
three more shipped defects fixed, and the whole programme landed. The
walk's closing turn was being lost to a race — the stop hook reads the
transcript while the runtime is still appending the final message — and
that turn is where closing emissions live; the payload's own copy now
closes the walk log. Write and Edit responses were being trimmed by a
read's rule, leaving claims about written artifacts unprovable; a write
is evidence and is kept whole. `conduct` (P11) shipped with its first
two cases: investigation's symptom interview, and quick-fix scoping
walked entry-to-conclusion — the first whole-phase case, every user
interaction played from a description, the format choice pinned by
invariant. Those two cases found defects five through seven: a guard
keyed on the Symptoms content that initialisation itself writes (the
interview never ran for discovery-shaped work), the interview re-asking
the carrier, and scoping's gather-context blind to the carrier's
Exploration half with a static question fence — the recurring pattern
named: a guard keyed on artifact content an earlier step writes is
suspect; key on session facts or engine state. The harness grew teeth
the same day: vacuous checks report N/A rather than PASS,
`calls_in_order` covers what presence cannot, the scope lint ratchets
`files[]` from real walks (a link-crawl over-reports; one walk
undersamples), the world log is gated to walker events, a confirmed
failure reports both runs' checks labelled (divergence is walk
variance, not a prose property), and the walker is told a prescribed
command runs as written after an escalated walk skipped two calls yet
converged on a passing world — caught only by `calls_include`. All
forty PRs squash-landed to main across three rate-limit windows, every
ref audited intact; the full gate holds at 135/135.

- 2026-07-26 — Two findings that dissolved a long-running mystery, and
the rules that follow from them. First: we were judging a walk by the
single message an agent returns, while the walk itself happens across
Expand Down
41 changes: 40 additions & 1 deletion tests/prose/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,46 @@ the state that resulted. Prefer the consequence; keep display claims for
where the display genuinely is the behaviour under test, and expect them
to be the weakest line in the case.

**Scope `files` tightly** — selection and the PR-end suggestion run off it.
**Scope `files` tightly** — selection and the PR-end suggestion run off
it. It names the prose the walk actually traverses, not everything
reachable: harvest it from a real run's scope-lint markers rather than
following Load links (a crawl includes branches the case deliberately
stops before), and treat it as a ratchet — walks vary between runs, so a
later run may surface a file the first did not. The lint reports files
*opened*; a file read only to decide a branch that was not taken is not
walked, and stays off the list.

## Playing the user

The walker plays the user, two ways, declared in `case.json`:

**`answers`** is a queue — one entry per question, consumed in arrival
order at discrete gates (menus, yes/no prompts, named questions). Use it
where each question has one right response.

**`conduct`** is a description of the *person* — used where the prose
explores open-endedly (interviews, shaping conversations) and there is no
fixed number of turns to script. The walker answers as that person would,
in their words, for as long as the prose keeps asking. Give them concrete
knowledge, honest gaps ("they do not know the file names"), and
dispositions ("takes the simplest option that adds no external
dependencies") — a disposition can carry a menu choice, but pin any
outcome the case depends on with an invariant rather than trusting the
roleplay.

**Never mix them when an open stretch precedes a scripted gate.** The
queue is consumed in arrival order, so an interview before a menu eats
the menu's answer. Prefer conduct-only in such cases.

**Conduct describes the user, never the walk.** It must not name an arm,
end a step, or hint at what the prose ought to do — that is the same
boundary that keeps `assert.md` away from the walker. A described user
who would happily stop talking is not permission for the walker to cut a
loop the prose has not ended.

A stub *replaces* prose, so what it covers goes untested in that case;
conduct keeps the prose live. Where the prose can genuinely run against a
played user, prefer conduct.

## Stubs

Expand Down