skill-authoring §7: bare arm is environment-relative; harvest the arms' reasoning, not just verdicts - #86
skill-authoring §7: bare arm is environment-relative; harvest the arms' reasoning, not just verdicts#86firaen22 wants to merge 3 commits into
Conversation
…s' reasoning, not just verdicts Two clauses folded into the existing bare-probe bullet, distilled from a downstream consumer's first four fold probes of this repo's merged rules: - The bare arm runs in the executor's REAL baseline (standing global rules, always-loaded caches), so a bare-pass verdict means redundant in that environment only — record the baseline next to the verdict and never export non-discrimination to a different baseline. - Read each arm's stated reasoning before writing the fold: a failing bare arm supplies the exact excuse the rule must refute (carry it in near-verbatim); a passing ruled arm that over-fires by refusing the surrounding task is a wording warning, not a win. Provenance paragraph appended; both clauses ship unprobed per the covenant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third clause on the same bare-probe bullet, and it must ship with the first one: clause 1 says the bare arm KEEPS its environment (standing global rules, always-loaded caches); this says strip episodic recall of the finding under test. Stated apart, each reads as contradicting the other — the boundary is the rule. The test is what the arm would have without you: what every invocation in that environment ships with is baseline; an auto-memory store, a session summary, or notes from the run being probed are leakage. That store survives a fresh invocation, so the existing no-shared-state requirement does not catch it — the arm loads the conclusion, restates it, and scores as an unaided reproduction. Countermeasures: recall-dodging framing, an explicit disregard instruction, and a citation check on the output before the run counts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Pushed a third clause to this branch ( Separate baseline from leakage in the bare arm. Clause 1 says the bare arm keeps its environment (standing global rules, always-loaded caches). This clause says strip episodic recall of the finding under test. Stated apart, each reads as contradicting the other — the boundary between them is the actual rule, so they need to land together. The test is what the arm would have without you: what every invocation in that environment ships with is baseline; an auto-memory store, a session summary, or notes from the run being probed are leakage. That store survives a "fresh invocation", so the bullet's existing no-shared-state requirement does not catch it — the arm loads the conclusion, restates it, and scores as an unaided reproduction. Countermeasures added: recall-dodging framing (generic scenario, none of the finding's names or phrasing), an explicit disregard-prior-findings instruction, and a citation check on the output before the run counts — an arm citing the finding or its vocabulary is re-run, not scored. Provenance updated to cover all three clauses; ships |
A fold probe run at the probing session's own strong tier returned bare 3/3 on a citation-re-resolve rule (redundant with model capability); a same-day replication of the same fixture at a weak tier returned bare 0/3. Extends the environment-relative-baseline clause: tier is another axis the verdict does not automatically generalize across, and only a bare-pass (non-discrimination) result is poisoned by running above the file's audience — a strong-arm ruled FAIL or bare FAIL still stands.
Summary
Two clauses folded into the existing §7 bare-probe bullet, distilled from a downstream consumer's first four fold probes of this repo's merged rules (pre-registered arms, weaker-tier executor):
Both clauses ship
unprobedper the covenant; a Provenance paragraph documents the source incidents and queues their own probes.Test plan
git diff --word-diffagainst the branch point — zero words removed from the existing bullet, only inserted🤖 Generated with Claude Code