From a898f94224753e035349fd0bcaf5dc720c7e10b0 Mon Sep 17 00:00:00 2001 From: Lee Overy Date: Mon, 27 Jul 2026 19:53:01 +0100 Subject: [PATCH] docs(prose-tests): the record catches up with day two MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CLAUDE.md still told a case author to run until state lands — the rule P9 overturned when a handoff proved as deterministic an ending as a delta, and the exact instruction that would have pushed short entry walks into needing conversation stubs. It now states the current rule, lists calls_in_order among the invariants, and names conduct and claims mode in the case anatomy, so an agent working from CLAUDE.md alone authors to the contract as it is. The prose-test README gains the two pieces day two added and one it never had: a Playing the User section covering answers and conduct — the arrival-order hazard that forbids mixing them around an interview, the pin-choices-with-invariants rule, and the P4 boundary extended to roleplay — and the files[] guidance rewritten from what was learned measuring it: harvested from walks not link-crawls, a ratchet across runs, opened-is-not-walked. The design log gains P11, naming conduct as contract rather than convenience, and the day-two entry: the closing-turn race, writes as evidence, the whole-phase quick-fix case, defects five through seven with the recurring pattern named — a guard keyed on artifact content an earlier step writes is suspect — and the forty-PR landing. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 2 +- design/prose-tests.md | 44 +++++++++++++++++++++++++++++++++++++++++++ tests/prose/README.md | 41 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 85 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index bde6d1ad7..bb8508353 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -218,7 +218,7 @@ Add or update a test alongside any change to engine scripts, adapters, migration ## Prose Tests -`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). A case is a directory under `tests/prose/cases/{case-id}/`: `case.json` (values code branches on, including the required `entry` and any deterministic `invariants`), `fixture.md` + `fixture-state.cjs` (the starting world), `act.md` (the coarse instruction a walker agent follows through the real skills), `assert.md` (the expected path, seen only by the asserting agent), optional `assertion-state.cjs` (the world the walk should produce; absent = unchanged), and the two generated snapshots. Nothing is parsed and prose never shares a file with code. **A case starts where a session starts and runs until state lands**: `entry` must name `workflow-start`, a `workflow-*-entry` skill, or `workflow-discovery` — never a navigation or processing skill, never a reference — because a walk begun mid-session lacks the context the prose was written for, and one that stops at a gate leaves an empty delta and nothing but display claims to judge. Claims that code can settle belong in `invariants` (`engine_before_write`, `calls_include`, `calls_exclude`), which run before the asserter and whose verdicts it cannot overturn. Shared pipeline stages live in `tests/prose/mainlines/`. Walks cost tokens — run on command via the `/prose-test` dev skill, never as part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation plus snapshot goldens — every snapshot must rebuild byte-identical from its recipe under the frozen clock, skipped when nothing feeding it changed. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {case-id}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases. +`tests/prose/` — end-to-end tests for the prose logic (design: `design/prose-tests.md`). A case is a directory under `tests/prose/cases/{case-id}/`: `case.json` (values code branches on: the required `entry`, deterministic `invariants`, scripted `answers` and/or a `conduct` description of the user, stub arming, `world: "claims"` for model-authored end states judged against stated claims), `fixture.md` + `fixture-state.cjs` (the starting world), `act.md` (the coarse instruction a walker agent follows through the real skills), `assert.md` (the expected path, seen only by the asserting agent), optional `assertion-state.cjs` (the world the walk should produce; absent = unchanged), and the two generated snapshots. Nothing is parsed and prose never shares a file with code. **A case starts where a session starts and stops where its claims can be answered**: `entry` must name `workflow-start`, a `workflow-*-entry` skill, or `workflow-discovery` — never a navigation or processing skill, never a reference — because a walk begun mid-session lacks the context the prose was written for. A handoff is a fine ending (recorded actions are as deterministic as a world delta); what a case must never do is stop where only display claims remain, since text put on screen is evidenced by no hook. Claims that code can settle belong in `invariants` (`engine_before_write` — declared only where the walk writes, `calls_include`, `calls_exclude`, `calls_in_order`), which run before the asserter and whose verdicts it cannot overturn. Shared pipeline stages live in `tests/prose/mainlines/`. Walks cost tokens — run on command via the `/prose-test` dev skill, never as part of a routine gate. The deterministic perimeter runs under `npm test`: corpus validation plus snapshot goldens — every snapshot must rebuild byte-identical from its recipe under the frozen clock, skipped when nothing feeding it changed. When an engine change moves a world, regenerate with `node tests/prose/run.cjs snap {case-id}` and land the snapshot diff in the same PR; never hand-edit a snapshot. Every review finding lands twice: the fix, and the case that would have caught it. At the end of a PR that touches skill prose, run `node tests/prose/run.cjs select --diff main` and suggest running the intersecting cases. ## Knowledge Base Subsystem diff --git a/design/prose-tests.md b/design/prose-tests.md index ab7be189c..8699f0be4 100644 --- a/design/prose-tests.md +++ b/design/prose-tests.md @@ -206,6 +206,21 @@ testing. recording and replaying one would freeze whatever the walker happened to do, certifying broken prose instead of catching it. +- **P11 — the user is played two ways, and neither steers.** Scripted + `answers` cover discrete gates, one right response each, consumed in + arrival order. Where the prose explores open-endedly — an interview, a + shaping conversation — no turn count can be scripted, so `conduct` + describes the person instead and the walker answers as they would for + as long as the prose keeps asking. Conduct describes the user, never + the walk: it names no arm, ends no step, leaks no expectation — the + P4 boundary extended to roleplay — and any outcome a case depends on + (a menu choice carried as a disposition) is pinned by an invariant, + never trusted to the performance. The two modes never mix where an + open stretch precedes a scripted gate, because the queue is consumed + in arrival order and the interview would eat the menu's answer. A stub + replaces prose and untests it; conduct keeps prose live — where the + prose can genuinely run against a played user, conduct wins. + - **P7 — a failing case is a finding either way.** Either the prose broke, or the world/design moved and the case is stale. The adjudication is the point; only the typo-class staleness is @@ -254,6 +269,35 @@ testing. ## Log +- 2026-07-27 — Day two: the evidence pipeline completed, conduct built, + three more shipped defects fixed, and the whole programme landed. The + walk's closing turn was being lost to a race — the stop hook reads the + transcript while the runtime is still appending the final message — and + that turn is where closing emissions live; the payload's own copy now + closes the walk log. Write and Edit responses were being trimmed by a + read's rule, leaving claims about written artifacts unprovable; a write + is evidence and is kept whole. `conduct` (P11) shipped with its first + two cases: investigation's symptom interview, and quick-fix scoping + walked entry-to-conclusion — the first whole-phase case, every user + interaction played from a description, the format choice pinned by + invariant. Those two cases found defects five through seven: a guard + keyed on the Symptoms content that initialisation itself writes (the + interview never ran for discovery-shaped work), the interview re-asking + the carrier, and scoping's gather-context blind to the carrier's + Exploration half with a static question fence — the recurring pattern + named: a guard keyed on artifact content an earlier step writes is + suspect; key on session facts or engine state. The harness grew teeth + the same day: vacuous checks report N/A rather than PASS, + `calls_in_order` covers what presence cannot, the scope lint ratchets + `files[]` from real walks (a link-crawl over-reports; one walk + undersamples), the world log is gated to walker events, a confirmed + failure reports both runs' checks labelled (divergence is walk + variance, not a prose property), and the walker is told a prescribed + command runs as written after an escalated walk skipped two calls yet + converged on a passing world — caught only by `calls_include`. All + forty PRs squash-landed to main across three rate-limit windows, every + ref audited intact; the full gate holds at 135/135. + - 2026-07-26 — Two findings that dissolved a long-running mystery, and the rules that follow from them. First: we were judging a walk by the single message an agent returns, while the walk itself happens across diff --git a/tests/prose/README.md b/tests/prose/README.md index 7071ef7a6..715b8a767 100644 --- a/tests/prose/README.md +++ b/tests/prose/README.md @@ -86,7 +86,46 @@ the state that resulted. Prefer the consequence; keep display claims for where the display genuinely is the behaviour under test, and expect them to be the weakest line in the case. -**Scope `files` tightly** — selection and the PR-end suggestion run off it. +**Scope `files` tightly** — selection and the PR-end suggestion run off +it. It names the prose the walk actually traverses, not everything +reachable: harvest it from a real run's scope-lint markers rather than +following Load links (a crawl includes branches the case deliberately +stops before), and treat it as a ratchet — walks vary between runs, so a +later run may surface a file the first did not. The lint reports files +*opened*; a file read only to decide a branch that was not taken is not +walked, and stays off the list. + +## Playing the user + +The walker plays the user, two ways, declared in `case.json`: + +**`answers`** is a queue — one entry per question, consumed in arrival +order at discrete gates (menus, yes/no prompts, named questions). Use it +where each question has one right response. + +**`conduct`** is a description of the *person* — used where the prose +explores open-endedly (interviews, shaping conversations) and there is no +fixed number of turns to script. The walker answers as that person would, +in their words, for as long as the prose keeps asking. Give them concrete +knowledge, honest gaps ("they do not know the file names"), and +dispositions ("takes the simplest option that adds no external +dependencies") — a disposition can carry a menu choice, but pin any +outcome the case depends on with an invariant rather than trusting the +roleplay. + +**Never mix them when an open stretch precedes a scripted gate.** The +queue is consumed in arrival order, so an interview before a menu eats +the menu's answer. Prefer conduct-only in such cases. + +**Conduct describes the user, never the walk.** It must not name an arm, +end a step, or hint at what the prose ought to do — that is the same +boundary that keeps `assert.md` away from the walker. A described user +who would happily stop talking is not permission for the walker to cut a +loop the prose has not ended. + +A stub *replaces* prose, so what it covers goes untested in that case; +conduct keeps the prose live. Where the prose can genuinely run against a +played user, prefer conduct. ## Stubs