diff --git a/skills/cross-model-review/SKILL.md b/skills/cross-model-review/SKILL.md index 40f13fe..f836ac6 100644 --- a/skills/cross-model-review/SKILL.md +++ b/skills/cross-model-review/SKILL.md @@ -105,6 +105,42 @@ objection is not re-litigated — but a *reproduced* correctness or safety defec stays must-fix (§3 triage): it may be deferred only with the owner's explicit acceptance, never closed by the reviewers on their own. +**Pre-existing is a disposition you establish, not assume — reproduce the +finding on the merge-base too** (`unprobed` — see Provenance). On a +behavior-preserving refactor branch a +*reproduced* defect can still be one the branch faithfully carried over from +its base, not one it introduced; recording it against the branch is a false +regression that blocks a clean refactor for a fault it did not create. Before +you count a reproduced finding as the branch's defect, re-run it against the +merge-base: the SAME failure on both — same trigger, same outcome, +comparable blast radius — → `pre-existing-tracked`, spun off as its own +fix, never bloating the refactor — pre-existing moves where a reproduced +correctness or safety defect gets fixed, never whether (deferral stays the +owner's call per this section's triage), and any widening the branch adds +(a new trigger path, a broader blast radius) is not carried over: the +widened part is the branch's own regression; gone on the merge-base and +live on the branch → a real regression this branch owns. Reviewers do not +see the base (§2 packet is your +inlined lines), so this classification is yours to make, not theirs — a +reviewer's severity label ("not merge-ready", "CRITICAL") is a claim about +the code in front of it, blind to whether the branch introduced the behavior. +And applicability is itself a **runtime/deployment-state** question, not a +source one: adjudicate a migration or identifier case-collision finding +against what actually shipped (query the collection, check the deploy), +never only against synthetic data the bug reproduces on but the running +system never holds. A finding whose premise no shipped state satisfies — +empty store, writer never deployed — is recorded `rejected-with-reason` +with that runtime counter-evidence attached, or held as an owner-accepted +deferral where the state could yet arrive; do not file it under §4's +irreproducible-false-positive bucket, which is for findings that do not +reproduce at all. Non-applicability is severity evidence, not ownership +evidence: a latent path the branch itself introduces stays its own +must-fix however empty today's store is — deployment catches up. +neg: filing a reviewer's "P1, not merge-ready" against a refactor for a quirk +that reproduces identically on the merge-base — the review still pays off (spin +the quirk off as its own fix), but the branch is clean and the label was +baseline-blind. + **A proposed fix is a suggestion, not a patch.** Reproducing a finding licenses the finding, not its remedy — these are separate judgments, and a reviewer writes the remedy against only the lines you inlined (§2), so a real @@ -232,6 +268,19 @@ the reason recorded. The landed side is verifiable in-repo (PR #32's commits); the held side was never pushed, so the comparison itself is contributor-reported — the rule ships with an in-body `unprobed` marker per the README covenant's second branch. +The §3 baseline-classification and runtime-state adjudication rules +(2026-07-24) are class-distilled from a mining pass over the owner's own +sessions (no code taken): a moira-web behavior-preserving refactor where +codex's "not merge-ready" and agy's "CRITICAL" both dissolved under +reproduction as pre-existing base-branch behaviors the branch faithfully +preserved +(0 regressions, 4 pre-existing quirks spun off), and a TG-bot review where +codex's identifier-migration and case-collision findings were non-applicable +because the store was empty and the writer had never deployed. The lesson +is the general one — pre-existing-vs-regression is a baseline diff, and +applicability is a runtime-state question — not the specific findings. +Both rules ship `unprobed` per the covenant; their probes join the private +round-5 queue. Re-verify line: model families, CLI availability, "flagship" identity, and effort tiers are volatile — re-discover at session time; never trust a model name or tier diff --git a/skills/delegation-and-review/SKILL.md b/skills/delegation-and-review/SKILL.md index e031032..9422c65 100644 --- a/skills/delegation-and-review/SKILL.md +++ b/skills/delegation-and-review/SKILL.md @@ -576,6 +576,22 @@ reviewers that they silently absorb as implementers. its inputs arrived. ❌ "the synthesis stage returned a thorough report, so the finders must have run." +- **A read-only survey reports leads, not facts — re-verify each in source + before you spec work on it** (`unprobed` — private evidence as shape; see + Provenance). Fan-out finders scanning for dedup targets, + dead code, or duplication over-claim in the direction that makes the work + look bigger: a substring grep calls "66 import sites" what is really 2 + files; "byte-identical" components turn out to diverge on a prop or a + fallback; a "duplicated" helper is two semantically incompatible families + that must not be merged; "dead" code has a live registry entry the finder + never traced. Each finding is a hypothesis — open the cited files, tracing + the references its verdict depends on (registries, routes, call sites), and + confirm the claim before it enters the plan, and re-check after any finding + that reverses scope (a dissolved dedup, a resurrected dead file), because + the tier ranking built on the survey is now stale. Speccing on an + unverified survey buys a plan that reverses mid-execution. + ❌ "the survey found 66 three.js sites, so the dedup is a big win" — the + finder counted substrings; there are 2. - **Miss-is-costly audits** ("find ALL of X": security holes, money paths, data leaving the machine) need a different loop than one review pass: - Scout the work-list once in-context first — fan out over a known list, @@ -975,6 +991,19 @@ Rules bullet was also extended from "do not merge" to "do not commit, push, or merge" (generalizing mindfold-ai/Trellis's check-agent forbidden-operations list, AGPLv3 — strictly ideas only) — a widening of an existing prohibition, not a new standalone rule, so it owes no separate probe. +The §3 survey-reports-leads bullet (2026-07-24) is class-distilled from a +private mining pass over the owner's own sessions (no code taken): a single +codebase-lightening session whose read-only fan-out survey was wrong four +separate ways, each over-claiming — a substring count of "66 three.js sites" +that was 2 files, "byte-identical" components that diverged on a prop and a +fallback, a "NumField duplication" that dissolved into two incompatible +families, and a "dead" component with a live registry entry — each reversal +re-ordering the tiered plan. Distinct from the synthesizer-fed-nothing bullet +above (empty input → confident synthesis) and the settled-tree bullet +(re-reading an already-fixed file): here the survey ran and returned +non-empty, confidently wrong leads. Private evidence, cited as shape per the +README covenant's second branch; no in-repo probe has run — in-body +`unprobed` marker. Stable behavioral rules; re-check worktree/agent mechanics and any recorded hosted-endpoint behavioral claims against the current environment. diff --git a/skills/ground-truth-gates/SKILL.md b/skills/ground-truth-gates/SKILL.md index f69a1b9..536e4ce 100644 --- a/skills/ground-truth-gates/SKILL.md +++ b/skills/ground-truth-gates/SKILL.md @@ -269,6 +269,36 @@ A generic green test is not proof. A gate is real only if: elsewhere — is legitimate as-is and needs no anchor, as does an intrinsic gate like a syntax or forbidden-character scan. The rule bites only when agreement is made to stand in for an external fact.) +8. **A gate over hardcoded facts asserts the facts, not just the shape — and + the cross-check that established them belongs IN the fixture, not in the + chat** (`unprobed` — private incident as shape; see Provenance). When code + embeds domain constants (holiday dates, a tax rate, a fee schedule, a + jurisdiction's valid state codes), a suite that checks structure — the + array is non-empty, + each entry parses, the shape is right — passes identically whether the + values are correct or a later edit corrupted one. Those values were usually + cross-checked once, against an authority or several independent sources or a + reviewer's recall — but that check happened in the conversation and + evaporates when the session ends, so the next bad edit sails through a + shape-only gate. Anchor the fact: assert every load-bearing value + (a fixed holiday falls on its known date, the standard rate equals the + published number), each assertion naming its authority (source, and its + version or URL where it has one) and consultation date beside the value — + an unattributed literal is indistinguishable from item 3's copied-back + expected value — so a future silent change to a constant fails. This *extends* item 2's carve-out — + from an output-interface string to an embedded input constant — and is NOT + item 3's "hardcoded expected value" fake pass: the anchor's value comes from + an external authority, not copied back from the code's own output. Item 3's + tautology asserts the code agrees with its own output; this asserts the + code matches the world. It shares rule 7's remedy — an external anchor — but not its + trigger: rule 7 bites where *agreement between artifacts* is made to stand + in for an external fact, this one where *structure* is. A fact that + legitimately changes gets its anchor updated as a contract edit (rule 4: + state which contract changed and who approved it); an always-fixed one is + cheap to anchor permanently. + ❌ "the holiday tests pass" — they assert the list has the right count and + types, never that any date is the right day; a fat-fingered edit to one + date stays green. **A red result is not automatically a real defect** — but ruling one "environmental" is a gate change, not the worker's call (rule 4): quarantine it @@ -446,5 +476,13 @@ unless the project opted in (MIT, ideas only), corroborated by NYCU-Chung/my-claude-devteam's bypassPermissions-hook framing (MIT) — it matches this pack's own no-auto-registered-hooks invariant. All three ship `unprobed` per the covenant; their probes join the private round-5 queue. +Rule 8 (fact-anchoring; 2026-07-24) is class-distilled from a mining pass +over the owner's own sessions (no code taken): hardcoded public-holiday dates +were cross-checked against four independent sources, but a cross-model +reviewer noted the suite validated structure only — "tests check shape, not +date-truth" — so the truth evidence lived in the conversation, not the gate; +the fix added anchor-date assertions as a regression guard against a future +bad edit. Ships `unprobed` per the covenant; its probe joins the private +round-5 queue. `template/` scripts are self-contained (Node + bash, zero deps) and were run green on 2026-07-06 with Node v23; re-verify with `bash template/run-all.sh`. diff --git a/skills/skill-authoring/SKILL.md b/skills/skill-authoring/SKILL.md index 4354d23..a7941d1 100644 --- a/skills/skill-authoring/SKILL.md +++ b/skills/skill-authoring/SKILL.md @@ -84,6 +84,19 @@ artifact-producing step. - What cannot be verified is labeled `unverified` or `user-must-provide` — never silently invented. Unproven ideas stay labeled open/candidate; no oversell. +- **Verify-before-you-write-it bites hardest on a capability you describe for a + weaker executor** (`unprobed` — private incident as shape; see Provenance). + "The engine auto-searches every category", "this flag does X" — a weak model + executes such a line verbatim, with none of your context to catch it when it + is wrong, so an unverified capability claim is a false instruction to the one + reader least able to notice. When the behavior is new or just-patched, order + it: prove it against a real gate (ground-truth-gates), THEN finalize the doc + that describes it. A scarce live session (an auth window about to expire, a + costly remote setup) argues for verifying FIRST, not for shipping the doc on + the theory it probably works. + ❌ drafting "the engine now self-heals across categories" into the skill + while the self-heal fix is still unproven, to spend the login session before + it expires. - Embed the knowledge itself; do not make private paths or one person's memory a load-bearing reference. A hard-coded machine-absolute path is worse than a broken link: a stale duplicate clone resolves *silently* to an @@ -94,6 +107,17 @@ artifact-producing step. ## 3. Provenance and decay - Date-stamp volatile facts (versions, flags, model names, defaults). +- **Staleness concentrates in the world-fact rules — audit there first** + (`unprobed` — private incident as shape; see Provenance). A rule encoding a + mutable fact about the outside world (which model or tool is default, how a + CLI behaves, a numeric threshold someone measured) can go stale within + days; a rule encoding method (reproduce before trusting, verify by + execution) goes stale rarely and slowly. A staleness audit therefore + concentrates the search on the world-fact files and treats the + method-encoding files as low-yield — not as exempt. One sweep of four rules + files found every stale instance — 8 of 8 — in the single file that carried + world-facts; the three method files were clean. This scopes the SEARCH only: + a method file still gets corrected when something does surface in it. - **Capability-negative claims rot the worst** (`unprobed` — private incident as shape; see Provenance). About to write "no such flag", "only works interactively", "the API can't do X" into an instruction file — or @@ -158,6 +182,16 @@ artifact-producing step. - End each skill with a short provenance note and a one-line re-verification command for anything that may drift. A skill without a re-verification path decays into exactly the stale-instruction problem it was meant to solve. + **And that command has to hang off something the work already touches** + (`unprobed` — private incident as shape; see Provenance): an invalidation + condition needing a separate act of remembering is inert no matter how + precisely it is written. Bind it to a surface the next pass crosses anyway — + a line in the maintenance entry that pass must read, an assertion in a gate + that already runs, a trigger on a file someone edits regardless. A measured + finding carrying the clause "re-test if either file grows past ~250 lines" + sat at 297 and 318 for days, still cited as current: the threshold was + right, the condition was true, and nothing was reading it. + ❌ "the invalidation condition is documented at the end of the finding." - **A merged upstream integration is not necessarily the end of the campaign** (`unprobed` — the upstream half of the incident is verifiable in this repo's PR history, the sync half @@ -557,6 +591,61 @@ default; an AI rewrite does not launder a derivative). the §6 review. Ask the user first before weakening or deleting any rule that gates destructive actions, spending, or publishing — or any rule the user set explicitly. +- **Probe a candidate rule against the bare executor before folding it in** + (`unprobed` — measured in-house; see Provenance). §6's behavioral probe hunts + gaps: what the file fails to make happen. This asks the opposite question, + before the rule exists — run the scenario twice against the file's target + executor, as independent fresh invocations (no shared state or history + between the arms), once with no rule and once with it. An arm counts only + when its run demonstrably met the rule's trigger (ground-truth-gates' + not-armed discipline: a run that never hit the guarded condition is + excluded and re-run), and one run per arm screens for a large effect + only — a close or surprising call re-runs before deciding. Score each arm + against the rule's INTENDED outcome — for a preventive rule the intended + outcome is the abstention or refusal, so a bare arm that commits the + prohibited act FAILS its arm — then read the pair, not the bare arm + alone: both arms produce the intended outcome → the rule is + non-discriminating — it costs a line and buys nothing, and belongs in a + reference file or nowhere, not in the always-loaded one; only the ruled + arm produces it → the rule earns its line; neither arm does → the rule as + written is ineffective — rewrite or drop it, never fold it in on truth + alone; only the bare arm produces it → the rule is harmful — dropped, not + filed as reference. Eight rules folded into two files over one week were + probed this way afterwards; three were reproduced unaided. Non-discriminating + is not the same as wrong — the rule can be true and still not worth its line, + and that is the judgment the probe is for. + ❌ "the rule is correct and clearly written, so it earns a line." +- **A derived file contradicting reality has not necessarily drifted — read its + source before fixing it** (`unprobed` — private incident as shape; see + Provenance). Where a file is compiled from another (a cache over a playbook, a + rules file over a spec), the obvious reading of a wrong line is that the + derivative fell behind. Check whether the SOURCE carries the same wrong + content first — the identical text, or the source-side value it is + generated from. If it does, this is not drift: the derivative is faithfully + mirroring a source that itself disagrees with the world, and correcting only + the derivative gets it silently re-broken by the next compile or write-back. + Fix the source, then bring the derivative along through its regeneration or + write-back path; a machine-compiled artifact is never hand-edited + (ground-truth-gates' regenerate-and-diff rule), while a hand-maintained + derivative with no mechanical path is corrected by hand against the fixed + source. Record which side was authoritative. A source that checks out + clean does not yet establish true drift — with a mechanical path, + trial-regenerate and diff: correct output means the derivative had merely + fallen behind, so land that regenerated output; wrong output again means + the generation path itself is defective and gets fixed before any + regeneration is trusted. For a pair that is hand-maintained by design, a + clean source and a wrong derivative is manual drift: hand-correct the + derivative against the clean source and record it as such. A + machine-compiled artifact whose generator is merely unavailable is NOT + that case — restoring the generation path is the fix, and the artifact + stays un-hand-edited (blocked, not manually drifted). + Distinct from §3's sync contract, which + keeps coupled files agreeing and names which wins when they disagree — this + rule's trap subcase is the pair that agrees while both are wrong + (ground-truth-gates rule 7's mutual-agreement trap, arising in maintenance + rather than in a gate); the drift branches above already cover the + disagreeing pair. + ❌ "the cache says X, reality says Y — so the cache drifted; fix the cache." - Compaction triggers — act when any of these is true: a skill outgrows what a reader can hold (~150 lines for discipline skills; domain-reference packs run longer, but every line must still earn its place); its description no @@ -880,3 +969,30 @@ its scope-install / invoke-by-name / prune mitigation ladder (MathWorks field-of-use license — ideas only, no text). Both ship `unprobed` per the covenant; their probes join the private round-5 queue — the catalog-collision rule's is directly runnable against this pack's own 13 skills. +The §2 verify-capability-before-shipping-the-doc rule (2026-07-24) is +class-distilled from a mining pass over the owner's own sessions (no code +taken): with a scarce authenticated remote session about to expire, doc edits +describing a just-patched self-heal behavior were held until a regression gate +proved the behavior, on the explicit reasoning that telling the weaker model +"the engine auto-searches all categories" is a capability claim that must be +verified live first or it ships a false instruction. Ships `unprobed` per the +covenant; +its probe joins the private round-5 queue. +The §3 world-fact-staleness and bound-invalidation rules and the §7 +bare-executor-probe and read-the-source-before-fixing rules (2026-07-25) are +class-distilled from a maintenance session on the owner's own rules files (no +code taken), and are one incident seen from four sides. A finding measuring +rule-following at two file sizes carried the clause "re-test if either file +grows past ~250 lines"; both files reached 297 and 318 and the clause never +fired, because nothing read it — the bound-invalidation rule. The re-test that +eventually ran added a bare-executor control arm the original lacked, and found +3 of 8 recently-folded rules reproduced with no rules file at all — the +bare-executor-probe rule, the one item here with a measurement rather than a +shape behind it (n=8 rules, 1 probe per cell; it detects a large effect, not a +small one). The staleness sweep that followed found all 8 stale instances in +the one file carrying model names and CLI behavior, none in the three carrying +method — the world-fact rule. And one of those stale lines turned out to be +copied faithfully from a source file carrying the same error, so fixing the +derived file alone would have been undone by the next write-back — the +read-the-source rule. All four ship `unprobed` per the covenant; their probes +join the private round-5 queue.