From bc082477831e863d2e3397a04a0a4174da9e84e8 Mon Sep 17 00:00:00 2001 From: Elliot Taylor Date: Tue, 14 Jul 2026 10:17:28 +0200 Subject: [PATCH] Document plan B for field-testing while the gate is environmentally red The 2026-07-14 session showed the hostile gate fails for reasons no prompt edit can fix (stale published docs, release freshness), so document how to keep testing: stub self-test, standard persona, hostile rounds attributed by refusal reason, and a possible local-registry mode. Co-Authored-By: Claude Fable 5 --- field-test/README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/field-test/README.md b/field-test/README.md index d2d9bbc..8cdfd94 100644 --- a/field-test/README.md +++ b/field-test/README.md @@ -80,6 +80,20 @@ Everything is saved under `field-test/results/-/` (gitignore Keep `prompt.txt` in sync with the README prompt — it is the tested artifact. +## When the gate is red for environmental reasons (plan B) + +The agent audits the *published* tarball, so the gate's pass rate is a function of two things that no prompt edit can fix: + +- **Stale shipped docs.** If doc fixes are merged but unpublished, agents refuse over contradictions that no longer exist at HEAD (this is how a full 2026-07-14 session went 2-green/8-refused across every prompt variant, including the incumbent). Publishing is the only fix. +- **Release freshness.** "Published 11 hours ago" and same-day release bursts are cited as targeted-attack signals. This decays on its own; gate results are most representative ≥24–48h after the last publish. (Corollary: publishing doc fixes resets this clock — expect a noisy day, then re-gate.) + +Until a publish lands and ages, use this ladder instead of burning hostile rounds on a known-red gate: + +1. **Stub self-test** — `node field-test/run.mjs --agent-cmd "node $PWD/field-test/stub-compliant.mjs"`. Validates the harness, mock, and scoring in ~1 min. No AI, no registry dependency. +2. **Standard persona** — exercises the mechanical checks (guide accuracy, hook wiring, widget token) with less policy pressure; catches CLI/UX regressions immediately. +3. **Hostile rounds scored by refusal *reason*, not exit code.** Read DECISION ANALYSIS and attribute each refusal: one that quotes the published docs or release age is environmental noise; one that quotes the prompt's own wording is a real prompt bug. A variant is not worse than the incumbent unless it draws prompt-directed refusals the incumbent doesn't. +4. **(Not built) local-registry mode** — run a local registry (e.g. verdaccio), publish the working tree to it, and pin the fixture via the `npm_config_registry` env var (env pinning reads as platform plumbing, same as `PATCHSTACK_ENDPOINT`). This is the only way to exercise unpublished doc/CLI changes end-to-end. Caveat: the local record has no provenance attestation or signatures, which strict agents check — expect some artificial refusals on that ground. + ## Known limitations - The simulated agents are Claude-based; real platform agents (Bolt, Lovable, Cursor) carry stricter, unknown system prompts. A green harness is necessary, not sufficient — the fourth refusal mode was found by a real Bolt user after the harness passed v1 of a prompt. Treat real-world refusal reports as new personas: encode the pressure they applied into `personas/` so the regression stays covered.