From 9ac61e1fe60b5914febab6684ac966bf6a6616f3 Mon Sep 17 00:00:00 2001 From: firaen22 Date: Mon, 27 Jul 2026 18:17:48 +0800 Subject: [PATCH] skill-authoring: repair the derived-file rule's trigger, record first probe results MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four rules folded 2026-07-25 shipped `unprobed` per the covenant. This is that probe, run against the merged text (n=1/arm, one weaker-tier executor — screens for a large effect only). - §7 derived-file rule: failed BOTH arms. Given a wrong value in a file with no provenance cue, each arm edited it in place. The written trigger named the classification ("a derived file") that the rule exists to force, so it fired only when provenance was already known. Trigger restated as the observable state; body unchanged. Repaired form ships unprobed. - §3 bound-invalidation, §7 bare-executor-probe: discriminated. Markers updated. - §3 world-fact staleness: did NOT discriminate — a bare arm produced the scoping unaided. Flagged in its marker for a demotion judgment, not demoted at n=1. - §2: derive a cross-reference's section number from the file, never from recall. One such reference reached main and cost fix commit 8f8413f. Provenance amended with the results, including two discarded runs (round-1 scenarios leaked the taught distinction; one bare arm was contaminated by the author's own session memory reaching the subagent). Co-Authored-By: Claude Opus 5 --- skills/skill-authoring/SKILL.md | 83 ++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 12 deletions(-) diff --git a/skills/skill-authoring/SKILL.md b/skills/skill-authoring/SKILL.md index a7941d1..64c7dfd 100644 --- a/skills/skill-authoring/SKILL.md +++ b/skills/skill-authoring/SKILL.md @@ -55,7 +55,16 @@ artifact-producing step. - Verify every command, flag, path, and claim against the actual repo/system before writing it down. **A wrong runbook is worse than none, because it is - trusted.** + trusted.** That includes the pointer inside a rule you are writing: a + cross-reference's section number is a claim about the target file — derive + it by opening the file, or from the line a search already quoted, never + from recall of the file's structure. Two dangling §-references were written + by one author in one day, each from memory while the correct location sat + already-quoted in that author's working context. One reached main and cost + a maintainer fix commit, verifiable in this repo as `8f8413f`; the second + was caught in the author's own draft before submission + (contributor-reported). + ❌ "the sync contract is in §4" (recalled; the quoted line was in §3). - **Verifying the incident does not verify the prescription.** Distilling an incident into a rule is a lossy transform that can introduce a bug the incident never had: the rule cites a real failure yet prescribes a @@ -108,7 +117,9 @@ artifact-producing step. - Date-stamp volatile facts (versions, flags, model names, defaults). - **Staleness concentrates in the world-fact rules — audit there first** - (`unprobed` — private incident as shape; see Provenance). A rule encoding a + (first in-house probe 2026-07-27 did NOT discriminate — a bare arm produced + the scoping unaided; one run screens for a large effect only, so it is + flagged rather than demoted — see Provenance). A rule encoding a mutable fact about the outside world (which model or tool is default, how a CLI behaves, a numeric threshold someone measured) can go stale within days; a rule encoding method (reproduce before trusting, verify by @@ -183,7 +194,8 @@ artifact-producing step. command for anything that may drift. A skill without a re-verification path decays into exactly the stale-instruction problem it was meant to solve. **And that command has to hang off something the work already touches** - (`unprobed` — private incident as shape; see Provenance): an invalidation + (first in-house probe 2026-07-27 discriminated: bare arm documented the + condition with nothing reading it, ruled arm bound it — see Provenance): an invalidation condition needing a separate act of remembering is inert no matter how precisely it is written. Bind it to a surface the next pass crosses anyway — a line in the maintenance entry that pass must read, an assertion in a gate @@ -592,7 +604,10 @@ default; an AI rewrite does not launder a derivative). that gates destructive actions, spending, or publishing — or any rule the user set explicitly. - **Probe a candidate rule against the bare executor before folding it in** - (`unprobed` — measured in-house; see Provenance). §6's behavioral probe hunts + (first in-house probe of this rule itself, 2026-07-27, discriminated: both + arms were armed — each faced the add-or-not decision — and the bare arm + failed it, reaching the instinct but settling the question by judgment + instead of by running the two arms — see Provenance). §6's behavioral probe hunts gaps: what the file fails to make happen. This asks the opposite question, before the rule exists — run the scenario twice against the file's target executor, as independent fresh invocations (no shared state or history @@ -615,11 +630,19 @@ default; an AI rewrite does not launder a derivative). is not the same as wrong — the rule can be true and still not worth its line, and that is the judgment the probe is for. ❌ "the rule is correct and clearly written, so it earns a line." -- **A derived file contradicting reality has not necessarily drifted — read its - source before fixing it** (`unprobed` — private incident as shape; see - Provenance). Where a file is compiled from another (a cache over a playbook, a - rules file over a spec), the obvious reading of a wrong line is that the - derivative fell behind. Check whether the SOURCE carries the same wrong +- **A file's content contradicts reality and you are about to correct it — + first establish whether anything generates that file** (trigger repaired + 2026-07-27 after an in-house probe found the original form ineffective; the + repaired form is `unprobed` — see Provenance). The trigger is that + observable state — wrong content, correction imminent — never the + classification "this is a derived file": recognizing a file as derived is + the discovery this rule exists to force, so a trigger phrased "where a file + is compiled from another" fires only when provenance is already known + (exactly when the rule is least needed) and stays silent when it is not. + Where the provenance check finds the file IS compiled from another (a cache + over a playbook, a rules file over a spec), the obvious reading of a wrong + line is that the derivative fell behind — and not necessarily the right + one. Check whether the SOURCE carries the same wrong content first — the identical text, or the source-side value it is generated from. If it does, this is not drift: the derivative is faithfully mirroring a source that itself disagrees with the world, and correcting only @@ -646,6 +669,8 @@ default; an AI rewrite does not launder a derivative). rather than in a gate); the drift branches above already cover the disagreeing pair. ❌ "the cache says X, reality says Y — so the cache drifted; fix the cache." + ❌ "nothing marks `src/config/timeouts.ts` as generated, so this rule does + not apply — edit the value in place." - Compaction triggers — act when any of these is true: a skill outgrows what a reader can hold (~150 lines for discipline skills; domain-reference packs run longer, but every line must still earn its place); its description no @@ -979,7 +1004,9 @@ verified live first or it ships a false instruction. Ships `unprobed` per the covenant; its probe joins the private round-5 queue. The §3 world-fact-staleness and bound-invalidation rules and the §7 -bare-executor-probe and read-the-source-before-fixing rules (2026-07-25) are +bare-executor-probe and read-the-source rules (2026-07-25 — the last of these +shipped under the bold name "a derived file contradicting reality has not +necessarily drifted", retriggered 2026-07-27 per the entry below) are class-distilled from a maintenance session on the owner's own rules files (no code taken), and are one incident seen from four sides. A finding measuring rule-following at two file sizes carried the clause "re-test if either file @@ -994,5 +1021,37 @@ the one file carrying model names and CLI behavior, none in the three carrying method — the world-fact rule. And one of those stale lines turned out to be copied faithfully from a source file carrying the same error, so fixing the derived file alone would have been undone by the next write-back — the -read-the-source rule. All four ship `unprobed` per the covenant; their probes -join the private round-5 queue. +read-the-source rule. All four shipped `unprobed` per the covenant; the +2026-07-27 entry below records their first probe results. +The §7 read-the-source trigger repair and the §2 derive-cross-references +clause (2026-07-27) come from the first probe run of the four 2026-07-25 +rules above — the covenant's queued probe, run against the merged text +(n=1 per arm, one weaker-tier executor, screens for a large effect only; a +first round was discarded because its scenarios leaked the taught +distinction, and one bare arm was discarded as contaminated by the author's +own session memory reaching the subagent — a bare arm must be checked for +citations of the finding under test). Results: bound-invalidation and the +bare-executor-probe each discriminated (bare arm failed, ruled arm passed; +the bare-probe rule's own bare arm reached the instinct — "correct and +non-duplicate clears the bar for truth, not for inclusion" — but decided by +judgment, never running an arm). The world-fact-staleness rule did not +discriminate — a bare arm produced the scoping unaided; at one run per arm +this is flagged in its marker for a demotion judgment, not demoted here. The +read-the-source rule failed BOTH arms: handed a wrong value in a file with +no provenance cue, each arm edited it in place — the written trigger named +the very classification ("a derived file") the rule exists to force, so it +fired only when provenance was already known. The repair restates the +trigger as the observable state; the repaired form is unprobed and its +marker records that. Across the four, the two that discriminated hand the +executor a procedure and the one that did not hands it an observation a +careful bare executor states unaided. That pattern was itself drafted as a +candidate rule — triage the probe queue by asking whether a candidate hands +the executor a procedure or an observation, and skip probes on the +observations — and probed the same day: a clean bare arm produced its +substance unaided, so by this section's own verdict table it is +non-discriminating and stays out of the always-loaded file. It is recorded +here as an observed pattern, not shipped as a rule. The §2 clause's incident is verifiable +in this repo's history: two dangling section references shipped by one +author in one day, each written from recall while the correct location sat +already-quoted in the working context; one reached main and cost the +maintainer fix commit `8f8413f`.