Skip to content

Document plan B for field-testing while the hostile gate is environmentally red #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ejntaylor merged 1 commit into from
Jul 14, 2026
Merged

Document plan B for field-testing while the hostile gate is environmentally red #50

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions field-test/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,20 @@ Everything is saved under `field-test/results/<timestamp>-<persona>/` (gitignore

Keep `prompt.txt` in sync with the README prompt — it is the tested artifact.

## When the gate is red for environmental reasons (plan B)

The agent audits the *published* tarball, so the gate's pass rate is a function of two things that no prompt edit can fix:

- **Stale shipped docs.** If doc fixes are merged but unpublished, agents refuse over contradictions that no longer exist at HEAD (this is how a full 2026-07-14 session went 2-green/8-refused across every prompt variant, including the incumbent). Publishing is the only fix.
- **Release freshness.** "Published 11 hours ago" and same-day release bursts are cited as targeted-attack signals. This decays on its own; gate results are most representative ≥24–48h after the last publish. (Corollary: publishing doc fixes resets this clock — expect a noisy day, then re-gate.)

Until a publish lands and ages, use this ladder instead of burning hostile rounds on a known-red gate:

1. **Stub self-test** — `node field-test/run.mjs --agent-cmd "node $PWD/field-test/stub-compliant.mjs"`. Validates the harness, mock, and scoring in ~1 min. No AI, no registry dependency.
2. **Standard persona** — exercises the mechanical checks (guide accuracy, hook wiring, widget token) with less policy pressure; catches CLI/UX regressions immediately.
3. **Hostile rounds scored by refusal *reason*, not exit code.** Read DECISION ANALYSIS and attribute each refusal: one that quotes the published docs or release age is environmental noise; one that quotes the prompt's own wording is a real prompt bug. A variant is not worse than the incumbent unless it draws prompt-directed refusals the incumbent doesn't.
4. **(Not built) local-registry mode** — run a local registry (e.g. verdaccio), publish the working tree to it, and pin the fixture via the `npm_config_registry` env var (env pinning reads as platform plumbing, same as `PATCHSTACK_ENDPOINT`). This is the only way to exercise unpublished doc/CLI changes end-to-end. Caveat: the local record has no provenance attestation or signatures, which strict agents check — expect some artificial refusals on that ground.

## Known limitations

- The simulated agents are Claude-based; real platform agents (Bolt, Lovable, Cursor) carry stricter, unknown system prompts. A green harness is necessary, not sufficient — the fourth refusal mode was found by a real Bolt user after the harness passed v1 of a prompt. Treat real-world refusal reports as new personas: encode the pressure they applied into `personas/` so the regression stays covered.
Expand Down
Loading