Skip to content

skill-authoring: repair the derived-file rule's trigger (it fails both arms), record first probe results for the 2026-07-25 four - #85

Open
firaen22 wants to merge 1 commit into
F-e-u-e-r:mainfrom
firaen22:repair-derived-file-trigger
Open

skill-authoring: repair the derived-file rule's trigger (it fails both arms), record first probe results for the 2026-07-25 four#85
firaen22 wants to merge 1 commit into
F-e-u-e-r:mainfrom
firaen22:repair-derived-file-trigger

Conversation

@firaen22

Copy link
Copy Markdown
Contributor

The four rules folded on 2026-07-25 (#82, merged via #84) shipped with in-body
unprobed markers. This is that probe, run against the merged text rather than
what I submitted — the fold gate changed those rules materially, so the
submitted version was not the right thing to test.

One rule fails. The repair is the substance of this PR; the rest is recording
what the probe found.

§7 derived-file rule — failed both arms, trigger repaired

Round 1 said "a generated config file says the timeout is 30s; the running
system says 5s. Fix it." Both arms passed. That scenario leaks: it hands over
the provenance the rule exists to make you ask about. Discarded and re-run.

Round 2 said src/config/timeouts.ts with no provenance cue. Both arms
hand-edited the value.
Neither asked what produces the file.

The written trigger was "a derived file contradicting reality" — but
recognizing a file as derived is the discovery the rule exists to force. So
it fires when someone has already told you the file is generated (exactly when
you no longer need it) and stays silent when they haven't (exactly when you
do). Generalizable defect: a trigger naming a property the executor can only
know by having already applied the rule.

The repair restates the trigger as the observable state — content contradicts
reality, correction imminent, therefore establish provenance first — and keeps
the body, which was never the problem. The repaired form ships unprobed; its
marker says so.

The other three

rule bare ruled verdict
§3 bound-invalidation discriminates
§7 bare-executor-probe (self-applied) discriminates
§3 world-fact staleness scoping did not discriminate

The bare-probe rule's own bare arm reached the instinct — "correct and
non-duplicate clears the bar for truth, not for inclusion" — and then settled
the question by judgment instead of running the two arms. Armed, and failed.

World-fact scoping did not discriminate: on a scenario that doesn't pre-split
the files, a bare arm produced the prioritization unaided. At n=1 that is a
flag for a demotion judgment, not grounds to demote, so its marker records the
result and the rule stays. Your call whether it earns §3's line.

§2 — derive cross-references from the file, never from recall

Folded onto §2's existing verify-everything bullet rather than shipped as its
own rule: a pointer inside a rule is a claim about the target file, and the
same verification applies. Incident is in this repo — I wrote "§4's sync
contract" when the clause is in §3, from memory, while the correct line sat
already-quoted in my own working context. Cost you fix commit 8f8413f. A
second instance the same day I caught in my own draft (contributor-reported;
only the first is verifiable here).

A candidate this probe killed

Across the four, the two that discriminate hand the executor a procedure;
the one that doesn't hands it an observation a careful bare executor states
unaided. That looked like a cheap pre-filter for the probe queue, so I drafted
it as a rule and probed it the same day — and a clean bare arm produced its
substance unaided. Non-discriminating by this section's own verdict table, so
it is recorded in Provenance as an observed pattern and not shipped as a
rule.

Method, including what went wrong

n=1 per arm, one weaker-tier executor, independent fresh invocations. Screens
for a large effect only. Two runs were discarded:

  • Round-1 scenario leak (above) — the scenario handed over the distinction
    under test.
  • A contaminated bare arm. The subagent cited a private finding about this
    very rule from its own session memory, verbatim. A subagent bare arm is not
    bare by default; it needs framing that dodges recall relevance, an explicit
    disregard instruction, and a citation check on the output before the run
    counts. Noted in the Provenance entry, since anyone re-running these probes
    will hit it.

I also had to override a clause in my own pre-registration mid-grading: I had
frozen "an answer that never establishes provenance is not armed, excluded",
which conflates not armed with failed and would have excluded every failing
bare arm while counting the passes — a change detector inside the gate, written
while building the gate for it. Armed tests whether the scenario presented
the guarded situation, never whether the answer handled it.

Reviewed fresh-context before submitting; three findings fixed, including an
overclaim in my own §2 addition (I had asserted both dangling references as
repo-verifiable when only one is).

… probe results

The four rules folded 2026-07-25 shipped `unprobed` per the covenant. This is
that probe, run against the merged text (n=1/arm, one weaker-tier executor —
screens for a large effect only).

- §7 derived-file rule: failed BOTH arms. Given a wrong value in a file with no
  provenance cue, each arm edited it in place. The written trigger named the
  classification ("a derived file") that the rule exists to force, so it fired
  only when provenance was already known. Trigger restated as the observable
  state; body unchanged. Repaired form ships unprobed.
- §3 bound-invalidation, §7 bare-executor-probe: discriminated. Markers updated.
- §3 world-fact staleness: did NOT discriminate — a bare arm produced the
  scoping unaided. Flagged in its marker for a demotion judgment, not demoted
  at n=1.
- §2: derive a cross-reference's section number from the file, never from
  recall. One such reference reached main and cost fix commit 8f8413f.

Provenance amended with the results, including two discarded runs (round-1
scenarios leaked the taught distinction; one bare arm was contaminated by the
author's own session memory reaching the subagent).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant