From bb4116dd72aa581e561cae82959beda9115e7540 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Sat, 12 Sep 2026 01:35:15 +0000 Subject: [PATCH 1/6] test: an orphan ledger row is named, and pruned only if no history goes (#983) The comparison already existed and already printed the answer. rename-scan pairs an appearance with a disappearance: for new, old in zip(app, van): rc = 1 With two vanished names and nothing appeared, zip yields nothing, rc stays 0, and the summary prints "appeared=0, vanished=2". Two rows in the committed ledger named checks that no longer existed, the census counted both, and every run said so in a line nobody acted on. A guard that compels one list and ignores the second manufactures the confidence that the thing is handled. orphan-scan adds the second direction: * a row no record in its own part matches is an orphan, and rc=1; * rows in parts the run does not contain are counted OUT LOUD as "not checked", never as present -- otherwise a one-suite log certifies the whole ledger; * --prune removes an orphan carrying no history, and REFUSES THE WHOLE PRUNE if any carries history, because no run can recreate the catalogue and removing the safe ones would leave a partial job for whoever reads the output. IT REPORTS AND IS NOT WIRED INTO THE GATE, for a measured reason rather than caution. Scanned against a real run, 340-the-binary-must-be-built-from.sh records ONE skip under a name neither of its two arms has ("the unreadable-source refusal") whenever the box has no non-root user to read as. On such a box two live rows have no matching record, so absence does not yet mean removal and a gate refusing on it would redden a correct run. Arming it needs those branches to record a SKIP under the names they stand in for -- #965's conversion for the eleven timeout paths -- and an arm pins that shape so the day it changes, the arm says so. Red-green on both harnesses, independently: ten arms in 410, two tests in test_mutation_ledger.py. Against main's tool they fail -- got 0 want 1, and rc 2 from an unknown subcommand -- and against this one they pass. harness_selftest 881 checks 881 records 0 FAIL, own PASSED verdict pytest corpus 349 passed / 868 checks | docs_style 9/9 shellcheck -S error -s bash over the harness: 0 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- CHANGELOG.md | 35 ++++++ test/pgc_ledger.py | 107 ++++++++++++++++++ test/pytest/TESTS.md | 20 ++++ test/pytest/test_mutation_ledger.py | 88 ++++++++++++++ .../410-a-check-must-have-been-red.sh | 96 ++++++++++++++++ 5 files changed, 346 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d57dfad3..577620a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2021,6 +2021,41 @@ true until the next version shipped. unanchored `grep -F` finds 3 occurrences because both renames EXTEND the name rather than replace it, so each renamed line still matches its own old form. Anchoring on the closing quote gives 1, which is the one that matters. +- `pgc_ledger.py orphan-scan` reports a ledger row that no record in its own part + matches, and `--prune` removes it only when no history would be lost (#983). + + The comparison already existed and already printed the answer. `rename-scan` pairs an + appearance with a disappearance, so an UNPAIRED disappearance -- a check deleted, or + renamed in a run where nothing appeared -- printed `vanished=2` and returned 0. Two rows + in the committed ledger named checks that no longer existed; the census counted both, and + every run for days said so in a line nobody acted on. **A guard that compels one list and + ignores the second manufactures the confidence that the thing is handled.** + + Driven on the real ledger, which is the only instance that matters: + + orphan: harness_selftest 330-... premise: all three runner functions were extracted + orphan: harness_selftest 330-... premise: and all three are callable + not checked: 44 row(s) in 1 part(s) this run does not contain + orphan scan: parts in the run=43, rows in those parts=861, orphans=2 + (0 carrying history), not checked=44 + orphan prune: removed 2 row(s), the ledger now holds 925 + + **A row carrying history is never pruned**, and one such row refuses the WHOLE prune. + No run can recreate the catalogue of what has been seen red, and removing the safe rows + while naming the unsafe ones would leave a partial job for whoever reads the output. + + **Rows in parts the run does not contain are counted out loud as `not checked`**, never + as present. Otherwise a single-suite log would certify the whole ledger, which is the + same defect one level up. + + **It reports and is NOT wired into the gate, for a measured reason.** Scanned against a + real run, `340-the-binary-must-be-built-from.sh` records ONE skip under a name neither + of its two arms has (`the unreadable-source refusal`) whenever the box has no non-root + user to read as. On such a box two live ledger rows have no matching record, so an absent + record does not yet mean a removed check and a gate refusing on absence would redden a + correct run. Arming it needs those branches to record a SKIP under the names they stand + in for -- the conversion #965 made for the eleven timeout paths -- and an arm now pins + that shape so the day it changes, the arm says so. - Nine git bundles are out of the tree, `*.bundle` is ignored, and `310-a-compiled-artifact-must-not-be.sh` now covers transfer artifacts as well as diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index a717c354..15a96909 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -28,6 +28,14 @@ refuses an increase, so widening the debt is an edit a reviewer sees AND a gate refuses, rather than either alone. +THE COMPARISON RUNS BOTH WAYS +----------------------------- +`gate` refuses a check the ledger has never seen. `orphan-scan` reports a ledger +row no record in its own part matches -- the other direction, which for a long +time printed `vanished=N` and refused nothing while two rows named checks that no +longer existed. It reports rather than gates, for a measured reason given in its +own docstring: an absent record does not yet mean a removed check. + WHAT THE GATE REFUSES --------------------- A check the committed ledger has never seen. That is the allowlist the issue asks @@ -357,6 +365,96 @@ def cmd_rename_scan(args): return rc +def cmd_orphan_scan(args): + """A ledger row that no record in its OWN PART matches: the unpaired half. + + `rename-scan` pairs an appearance with a disappearance. An unpaired + disappearance -- a check deleted, or renamed in a run where nothing appeared -- + was printed as `vanished=N` and refused nothing. Two such rows sat in the + committed ledger naming checks that no longer existed; the census counted both, + and every run returned 0 while the note scrolled past. A guard that compels one + list and ignores the second manufactures the confidence that the thing is + handled. + + SCOPED TO THE PARTS THE RUN CONTAINS, and the scope is REPORTED, not assumed. + A one-suite log has nothing to say about another suite's rows. Counting those + as present would make a single-suite run certify the whole ledger, so they are + counted OUT LOUD as `not checked` instead. + + WHY THIS REPORTS AND IS NOT WIRED INTO THE GATE. Measured, not assumed: part + 340 records ONE skip under a DIFFERENT name ("the unreadable-source refusal") + when the box has no non-root user to read as, rather than skipping its two + named arms. On such a box two committed rows have no matching record and are + not removed checks, so a gate refusing on absence would redden a correct run. + Arming this needs those branches to record a SKIP under the names they stand + in for -- the same conversion #965 made for the eleven timeout paths. + + A ROW CARRYING HISTORY IS NEVER PRUNED. The catalogue of what has been seen + red is the thing this ledger exists to be, and no run can recreate it. Dropping + an entry because a name moved is the precise loss `rename-scan` was written to + prevent, so `--prune` refuses the WHOLE prune when any orphan carries history + rather than removing the safe ones and leaving a partial job to be finished by + whoever reads the output. + """ + runs = _by_run(args.logs) + if len(runs) > 1: + raise LedgerError( + f"orphan-scan compares ONE run against the ledger, but got {len(runs)} logs: " + "the union of a before-log and an after-log hides the disappearance") + rows = read_ledger(args.ledger) + now = set(runs[0][1]) + + parts = {(s, p) for s, p, _ in now} + checkable = {k for k in rows if (k[0], k[1]) in parts} + unchecked = sorted(set(rows) - checkable) + orphans = sorted(checkable - now) + + with_history = [k for k in orphans + if rows[k][0] != NEVER or rows[k][1]] + historyless = [k for k in orphans if k not in with_history] + + for k in orphans: + last, muts = rows[k] + if k in with_history: + print(f" ORPHAN CARRYING HISTORY: {k[0]}\t{k[1]}\t{k[2]} " + f"(last red {last}, mutations: {';'.join(sorted(muts)) or NONE})") + else: + print(f" orphan: {k[0]}\t{k[1]}\t{k[2]} (no history)") + + # The parts the run never mentioned, named rather than counted alone: a number + # with no names is a number nobody can act on. + if unchecked: + silent = sorted({(k[0], k[1]) for k in unchecked}) + print(f" not checked: {len(unchecked)} row(s) in {len(silent)} part(s) this run " + f"does not contain, so it cannot speak about them: " + + ", ".join(f"{a}/{b}" for a, b in silent[:5]) + + (" ..." if len(silent) > 5 else "")) + + print(f" orphan scan: parts in the run={len(parts)}, rows in those parts={len(checkable)}, " + f"orphans={len(orphans)} ({len(with_history)} carrying history), " + f"not checked={len(unchecked)}") + + if not args.prune: + return 1 if orphans else 0 + + if with_history: + print(f" refusing to prune: {len(with_history)} orphan row(s) carry history, and " + "the catalogue is what this ledger is for -- no run can recreate it") + print(" reconcile them instead: rename the ledger row to the check's new name, " + "or say in the commit why the history may go") + return 2 + + if not historyless: + return 0 + + for k in historyless: + print(f" pruned: {k[0]}\t{k[1]}\t{k[2]}") + del rows[k] + write_ledger(args.ledger, rows) + print(f" orphan prune: removed {len(historyless)} row(s), the ledger now holds {len(rows)}") + return 0 + + def read_budget(path): out = {} text = pathlib.Path(path).read_text() @@ -669,6 +767,15 @@ def main(argv=None): r.add_argument("logs", nargs="+") r.set_defaults(fn=cmd_rename_scan) + o = sub.add_parser("orphan-scan", + help="refuse a ledger row no record in its own part matches") + o.add_argument("--ledger", required=True) + o.add_argument("--prune", action="store_true", + help="remove orphan rows that carry no history; refuse the whole " + "prune if any of them does") + o.add_argument("logs", nargs="+") + o.set_defaults(fn=cmd_orphan_scan) + g = sub.add_parser("gate", help="refuse a check the ledger has never seen") g.add_argument("--ledger", required=True) g.add_argument("--budget", required=True) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 5e21c01f..0f6b4e49 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -2237,6 +2237,26 @@ A global positional pairing misses a real rename whenever unrelated movement in part shifts the ordering. Given a before-log and an after-log together the vanished name is present in the union, so the scan **refuses** rather than silently finding nothing. +### `test_an_orphan_row_is_named_and_the_unscanned_rows_are_counted` + +`rename-scan` pairs an appearance with a disappearance, so an **unpaired** disappearance +— a check deleted, or renamed in a run where nothing appeared — printed `vanished=N` and +refused nothing. Two rows in the committed ledger named checks that no longer existed; +the census counted both and every run returned 0. + +The assertion that matters is the **scope**. A row in a part the run does not contain is +not an orphan, because the run cannot speak about it — counting those as present would +let a one-suite log certify the whole ledger. So the scan states how many rows it could +not speak about, and this test pins that number as well as the orphan it found. + +### `test_prune_drops_a_historyless_orphan_and_refuses_one_carrying_history` + +The catalogue of what has been seen red is what the ledger exists to be, and no run can +recreate it. `--prune` therefore refuses the **whole** prune when any orphan carries +history, rather than removing the safe ones and leaving a partial job for whoever reads +the output. A historyless orphan is removed and named as it goes; the row in the part the +run never mentioned survives, which is the control that the scope holds under a write. + ### `test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers` The suite restriction is the *meaning* of `suites_not_covered`, not a softening: without diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index 0248af4b..30e588a9 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -173,6 +173,94 @@ def test_renames_are_grouped_by_part_and_scanned_against_one_run(tmp_path, expec "a before-log and an after-log together are refused, not silently empty") +def test_an_orphan_row_is_named_and_the_unscanned_rows_are_counted(tmp_path, expect): + """The other direction of the set-compare: a ledger row no record matches. + + `rename-scan` pairs an appearance with a disappearance, so an UNPAIRED + disappearance printed `vanished=N` and refused nothing -- two rows naming checks + that no longer existed sat in the committed ledger while the census counted both. + + THE SCOPE IS THE ASSERTION THAT MATTERS. A row in a part the run does not contain + is not an orphan, because the run cannot speak about it; counting those as present + would let a one-suite log certify the whole ledger. So the scan says how many rows + it could not speak about, and this test pins that number rather than only the + orphan it found. + """ + ledger = _w(tmp_path, "l.tsv", "") + before = _w(tmp_path, "b.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\n" + "RESULT\tdemo\tpart1\tgone tomorrow\tPASS\t\n" + "RESULT\tdemo\tpartZ\telsewhere\tPASS\t\nchecks run: 3\n") + after = _w(tmp_path, "a.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\nchecks run: 1\n") + _run("merge", "--ledger", ledger, "--date", "2026-09-01", before) + expect.num(len(_rows(ledger)), 3, "premise: the ledger holds all three rows") + + out, rc = _run("orphan-scan", "--ledger", ledger, after) + expect.num(out.count("orphan: demo\tpart1\tgone tomorrow"), 1, + "a row no record in its own part matches is named an orphan") + expect.num(rc, 1, "and it is refused, not merely printed") + expect.num(out.count("still here"), 0, "the check the run still emits is not an orphan") + expect.num(out.count("elsewhere"), 0, + "nor is a row in a part the run does not contain") + expect.num(out.count("not checked=1"), 1, + "and the scan states how many rows it could not speak about") + + expect.num(_run("orphan-scan", "--ledger", ledger, before)[1], 0, + "a run that emits every row in its parts is clean") + expect.num(_run("orphan-scan", "--ledger", ledger, before, after)[1], 2, + "a before-log and an after-log together are refused, as rename-scan refuses them") + + +def test_prune_drops_a_historyless_orphan_and_refuses_one_carrying_history(tmp_path, expect): + """The catalogue of what has been seen red is what this ledger exists to be. + + No run can recreate it, so dropping an entry because a name moved is the precise + loss `rename-scan` was written to prevent. `--prune` refuses the WHOLE prune when + any orphan carries history, rather than removing the safe ones and leaving a + partial job for whoever reads the output. + """ + after = _w(tmp_path, "a.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\nchecks run: 1\n") + + plain = _w(tmp_path, "plain.tsv", "") + _run("merge", "--ledger", plain, "--date", "2026-09-01", + _w(tmp_path, "p.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\n" + "RESULT\tdemo\tpart1\tgone tomorrow\tPASS\t\n" + "RESULT\tdemo\tpartZ\telsewhere\tPASS\t\nchecks run: 3\n")) + out, rc = _run("orphan-scan", "--prune", "--ledger", plain, after) + expect.num(out.count("pruned: demo\tpart1\tgone tomorrow"), 1, + "a historyless orphan is pruned, and named as it goes") + expect.num(rc, 0, "and a prune that did its job is not an error") + names = {r[2] for r in _rows(plain)} + expect.num(len(names), 2, "the ledger is one row shorter") + expect.num(1 if "elsewhere" in names else 0, 1, + "control: the row in the part the run never mentioned survives the prune") + expect.num(_run("orphan-scan", "--prune", "--ledger", plain, after)[1], 0, + "and a prune with nothing left to remove is clean, not an error") + + hist = _w(tmp_path, "hist.tsv", "") + _run("merge", "--reds-are-real", "--mutation", "drop the guard", "--ledger", hist, + "--date", "2026-09-01", + _w(tmp_path, "h.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\n" + "RESULT\tdemo\tpart1\tgone tomorrow\tFAIL\t\nchecks run: 2\n")) + expect.text({r[2]: r[3] for r in _rows(hist)}["gone tomorrow"], "2026-09-01", + "premise: the orphan now carries a date") + + out, rc = _run("orphan-scan", "--prune", "--ledger", hist, after) + expect.num(out.count("ORPHAN CARRYING HISTORY"), 1, + "an orphan carrying history is reported as carrying it") + expect.num(out.count("last red 2026-09-01"), 1, + "and the history it would lose is printed with it") + expect.num(rc, 2, "the prune is refused") + expect.num(out.count("the catalogue is what this ledger is for"), 1, + "and it says why, rather than only that it refused") + expect.num(len([r for r in _rows(hist) if r[2] == "gone tomorrow"]), 1, + "premise: and the refusal removed NOTHING") + + def test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers(tmp_path, expect): """The suite restriction is the MEANING of `suites_not_covered`, not a softening. diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index f537f724..79302961 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -228,6 +228,102 @@ printf 'RESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 1\n' > "$_lw/re check "nor is one merely removed" \ "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/removed.log" | grep -c 'possible rename')" "0" +# ---- and what it IS, which nothing asked until #983 ------------------------- +# +# `rename-scan` pairs an appearance with a disappearance. An UNPAIRED +# disappearance -- a check deleted, or renamed in a run where nothing appeared -- +# printed as `vanished=N` and refused nothing. Measured on the committed ledger: +# two rows named checks that no longer existed, the census counted both, and the +# gate returned 0 on every run while the note scrolled past. +# +# SCOPED TO THE PARTS THE RUN CONTAINS, and the scope is REPORTED rather than +# assumed. A one-suite log has nothing to say about another suite's rows, and +# counting those as present is how a guard manufactures the confidence that the +# ledger was checked -- the shape of the whole issue family. + +: > "$_lw/orph.tsv" +printf 'RESULT\tdemo\tpart1\tstill here\tPASS\t\nRESULT\tdemo\tpart1\tgone tomorrow\tPASS\t\nRESULT\tdemo\tpartZ\telsewhere\tPASS\t\nchecks run: 3\n' > "$_lw/o_before.log" +printf 'RESULT\tdemo\tpart1\tstill here\tPASS\t\nchecks run: 1\n' > "$_lw/o_after.log" +_led_run merge --ledger "$_lw/orph.tsv" --date 2026-09-01 "$_lw/o_before.log" >/dev/null +check "premise: the ledger holds all three rows before the scan" \ + "$(wc -l < "$_lw/orph.tsv" | tr -d ' ')" "3" +check "premise: and the run the scan is given holds only one of them" \ + "$(grep -c '^RESULT' "$_lw/o_after.log")" "1" + +check "a ledger row no record in its own part matches is named an orphan" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" \ + | grep -c 'orphan: demo part1 gone tomorrow')" "1" +check "and it is REFUSED, not merely printed" \ + "$(_led_rc orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log")" "1" +check "the check the run still emits is not called an orphan" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" | grep -c 'still here')" "0" + +# The half that decides whether this guard is honest. A row in a part the run does +# not contain is NOT an orphan -- the run cannot speak about it -- and saying so +# out loud is the difference between a scope and a blind spot. +check "a row in a part the run does not contain is not called an orphan" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" | grep -c 'elsewhere')" "0" +check "and the scan states how many rows it could not speak about" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" | grep -c 'not checked=1')" "1" +check "a run that emits every row in its parts is clean" \ + "$(_led_rc orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_before.log")" "0" +check "a before-log and an after-log together are refused, as rename-scan refuses them" \ + "$(_led_rc orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_before.log" "$_lw/o_after.log")" "2" + +# ---- pruning: historyless rows may go, a catalogue entry may not ------------ +# +# The catalogue of what has been seen red is the thing this ledger exists to be. +# Dropping an entry because the name moved is the exact loss `rename-scan` was +# written to prevent, so prune REFUSES on history rather than asking nicely. + +cp "$_lw/orph.tsv" "$_lw/prune.tsv" +check "premise: the orphan about to be pruned carries no history" \ + "$(awk -F'\t' '$3=="gone tomorrow"{print $4}' "$_lw/prune.tsv")" "never" +check "--prune removes a historyless orphan and says which" \ + "$(_led_run orphan-scan --prune --ledger "$_lw/prune.tsv" "$_lw/o_after.log" \ + | grep -c 'pruned: demo part1 gone tomorrow')" "1" +check "and the ledger is one row shorter afterwards" \ + "$(wc -l < "$_lw/prune.tsv" | tr -d ' ')" "2" +check "control: the row in the part the run never mentioned SURVIVES the prune" \ + "$(awk -F'\t' '$3=="elsewhere"' "$_lw/prune.tsv" | wc -l | tr -d ' ')" "1" +check "and a prune with nothing left to remove is clean, not an error" \ + "$(_led_rc orphan-scan --prune --ledger "$_lw/prune.tsv" "$_lw/o_after.log")" "0" + +# An orphan that has been seen red is the one case where deleting the row loses +# something no run can recreate. +: > "$_lw/hist.tsv" +printf 'RESULT\tdemo\tpart1\tstill here\tPASS\t\nRESULT\tdemo\tpart1\tgone tomorrow\tFAIL\t\nchecks run: 2\n' > "$_lw/h_before.log" +_led_run merge --reds-are-real --mutation "drop the guard" --ledger "$_lw/hist.tsv" \ + --date 2026-09-01 "$_lw/h_before.log" >/dev/null +check "premise: the orphan now carries a date and a mutation" \ + "$(awk -F'\t' '$3=="gone tomorrow"{print $4"/"$5}' "$_lw/hist.tsv")" "2026-09-01/drop the guard" +check "an orphan carrying history is reported as carrying it" \ + "$(_led_run orphan-scan --ledger "$_lw/hist.tsv" "$_lw/o_after.log" \ + | grep -c 'ORPHAN CARRYING HISTORY')" "1" +check "and the history it would lose is printed with it" \ + "$(_led_run orphan-scan --ledger "$_lw/hist.tsv" "$_lw/o_after.log" \ + | grep -c 'last red 2026-09-01')" "1" +check "--prune REFUSES the whole prune when any orphan carries history" \ + "$(_led_rc orphan-scan --prune --ledger "$_lw/hist.tsv" "$_lw/o_after.log")" "2" +check "premise: and the refusal removed NOTHING -- the row is still there" \ + "$(awk -F'\t' '$3=="gone tomorrow"' "$_lw/hist.tsv" | wc -l | tr -d ' ')" "1" +check "the refusal says why, rather than only that it refused" \ + "$(_led_run orphan-scan --prune --ledger "$_lw/hist.tsv" "$_lw/o_after.log" \ + | grep -c 'the catalogue is what this ledger is for')" "1" + +# WHY THIS REPORTS AND DOES NOT GATE, measured rather than asserted. Part 340 +# records ONE skip under a DIFFERENT name ("the unreadable-source refusal") when +# the box has no non-root user, instead of skipping its two named arms. So on such +# a box two committed rows have no record and are not removed checks -- a gate +# refusing on absence would redden a correct run. This arm pins the precondition +# for arming it: the branch must record under the names it skips. +check "the conditional branch that makes absence ambiguous is still the OTHER shape" \ + "$(grep -c 'check_skip "the unreadable-source refusal"' \ + "$PGC_TESTDIR/selftest/340-the-binary-must-be-built-from.sh")" "1" +check "premise: and the two arms it stands in for are still named in the ledger" \ + "$(grep -cP '^harness_selftest\t340-the-binary-must-be-built-from\tpremise: the unprivileged' \ + "$_ledger")" "2" + # ---- the gate refuses a check the ledger has never seen --------------------- : > "$_lw/g.tsv" From 8095dcbeaf113bea20f66d0a5dd292149521f4a7 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Sat, 12 Sep 2026 03:19:56 +0000 Subject: [PATCH 2/6] test: --prune refuses a part that skipped, and 410 pins the precondition not an instance TWO REVIEW FINDINGS, both @pgcolumnar-9b's, both on this change rather than on the tree it guards. 1. --PRUNE DELETED A SUITE. `not checked` protects a part the run does not contain. A part CONTAINED BUT SKIPPED WHOLESALE fell in the gap between the two: one SKIP record put the part in `parts`, every other row of that suite became an orphan, and --prune removed them while reporting `not checked=0` and rc=0 -- the most confident output the tool can produce. Reproduced here on a three-row fixture for analyze_differential, whose PG17 run is a single SKIP: before orphans=2, not checked=0, rc=0, rows 3 -> 1 (the suite, deleted) after unprunable=2, rows 3 -> 3, and it says why control the same rows with a PASS record: pruned, rows 3 -> 1 The rule is broader than the reported case on purpose. A SKIP anywhere in the part means some arm did not run, so the run cannot tell "deleted" from "skipped under a name that does not match it" -- #994's defect at suite granularity instead of branch granularity. I wrote a docstring naming that shape at 340 and then built it one level up, which is not a subtlety I get to claim. One skipped timing check now blocks pruning that whole part; prune is rare and deliberate, a refusal costs a sentence, and a deletion costs history no run can recreate. The four categories are asserted to account for every ledger row, because a classification that loses one silently is the failure this tool reports. 2. 410:320 PINNED AN INSTANCE, NOT THE PRECONDITION. It grepped 340 for `check_skip "the unreadable-source refusal"`, which #998 deletes -- and the two PRs compose with a conflict only in the budget file, so main would have gone red with no marker to read. The replacement reads #998's own sweep: while `interpolated` or `armless` is nonzero, absence is not removal and the scan stays a report. It rests on the authority for that number rather than re-deriving it, and it RETIRES ITSELF when both counters reach zero. That condition is sufficient, NOT necessary, and the comment says so: a bare suite-level skip is not a loop, so it can never raise either counter. Both can reach zero with wholesale-skip suites still in place. @pgcolumnar-9b is adding a third counter for those as a follow-up; until then that half is unmeasured and the arm does not pretend otherwise. harness_selftest 897 checks 897 records 0 FAIL, own PASSED verdict pytest corpus 350 passed / 876 checks | docs_style 9/9 shellcheck -S error -s bash over the harness: 0 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- CHANGELOG.md | 21 ++++ test/pgc_ledger.py | 56 +++++++++- test/pytest/TESTS.md | 16 +++ test/pytest/test_mutation_ledger.py | 48 +++++++++ .../410-a-check-must-have-been-red.sh | 102 ++++++++++++++++-- 5 files changed, 230 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 577620a1..bcc71ecd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2048,6 +2048,27 @@ true until the next version shipped. as present. Otherwise a single-suite log would certify the whole ledger, which is the same defect one level up. + **`--prune` refuses a part that skipped, which was this change's own worst bug.** + `not checked` protects a part the run does not contain. A part *contained but skipped + wholesale* fell in the gap: one SKIP record put the part in the run's `parts`, every other + row of that suite became an orphan, and `--prune` deleted the suite while reporting + `not checked=0` and `rc=0` -- the most confident output the tool can produce. Measured on + a three-row fixture for `analyze_differential`, whose PG17 run is a single SKIP; found by + @pgcolumnar-9b in review. Nine suites skip wholesale on PG17 and `suites_not_covered` is + 250, so seeding any one of them would have armed it. + + The rule is broader than that case deliberately: a SKIP **anywhere** in the part means + some arm did not run, so the run cannot tell "this row's check was deleted" from "this + row's check was skipped under a name that does not match it" -- #994's defect at suite + granularity rather than branch granularity. One skipped timing check therefore blocks + pruning that whole part, and that is the direction a deleting command should err in. The + control holds the other half: the same rows are still pruned when the part's record is a + PASS, so this is not a tool that refuses to prune anything. + + The four categories -- matched, orphan, unprunable, not checked -- are asserted to account + for every ledger row, because a classification that silently loses one is the failure this + tool exists to report. + **It reports and is NOT wired into the gate, for a measured reason.** Scanned against a real run, `340-the-binary-must-be-built-from.sh` records ONE skip under a name neither of its two arms has (`the unreadable-source refusal`) whenever the box has no non-root diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 15a96909..a792e930 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -395,6 +395,27 @@ def cmd_orphan_scan(args): prevent, so `--prune` refuses the WHOLE prune when any orphan carries history rather than removing the safe ones and leaving a partial job to be finished by whoever reads the output. + + AND A PART HOLDING ANY SKIP IS UNPRUNABLE, which is the first version's worst + bug rather than a refinement of it. `not checked` protects a part the run does + not contain at all. A part the run CONTAINS BUT SKIPPED WHOLESALE fell between + the two: one SKIP record put the part in `parts`, every other row of that suite + became an orphan, and `--prune` deleted the suite while reporting + `not checked=0` and rc=0 -- the most confident output the tool can produce. + Measured on a three-row fixture for `analyze_differential`, whose run is one + SKIP on PG17; reported by @pgcolumnar-9b reviewing this change. + + THE RULE IS BROADER THAN THAT CASE ON PURPOSE. A SKIP anywhere in the part + means some arm did not run, so the run cannot distinguish "this row's check was + deleted" from "this row's check was skipped under a name that does not match + it" -- which is #994's defect, at suite granularity instead of branch + granularity. That is the same sentence this docstring already uses about part + 340, so writing the hole one level up was not an oversight I get to call + subtle. + + It is deliberately conservative: one skipped timing check blocks pruning that + whole part. Prune is a rare, deliberate act; a refusal costs a sentence and a + deletion costs history no run can recreate. """ runs = _by_run(args.logs) if len(runs) > 1: @@ -402,12 +423,17 @@ def cmd_orphan_scan(args): f"orphan-scan compares ONE run against the ledger, but got {len(runs)} logs: " "the union of a before-log and an after-log hides the disappearance") rows = read_ledger(args.ledger) - now = set(runs[0][1]) + verdicts = runs[0][1] + now = set(verdicts) parts = {(s, p) for s, p, _ in now} + # A part holding ANY SKIP cannot speak about absence: see the docstring. + skipped_parts = {(s, p) for (s, p, _), v in verdicts.items() if "SKIP" in v} checkable = {k for k in rows if (k[0], k[1]) in parts} unchecked = sorted(set(rows) - checkable) - orphans = sorted(checkable - now) + absent = sorted(checkable - now) + orphans = [k for k in absent if (k[0], k[1]) not in skipped_parts] + unprunable = [k for k in absent if (k[0], k[1]) in skipped_parts] with_history = [k for k in orphans if rows[k][0] != NEVER or rows[k][1]] @@ -430,12 +456,34 @@ def cmd_orphan_scan(args): + ", ".join(f"{a}/{b}" for a, b in silent[:5]) + (" ..." if len(silent) > 5 else "")) + # NAMED, not just counted: a number with no names is a number nobody can act on. + if unprunable: + parts_named = sorted({(k[0], k[1]) for k in unprunable}) + print(f" unprunable: {len(unprunable)} row(s) in {len(parts_named)} part(s) that " + f"SKIPPED at least one check, so absence there is not removal: " + + ", ".join(f"{a}/{b}" for a, b in parts_named[:5]) + + (" ..." if len(parts_named) > 5 else "")) + for k in unprunable[:5]: + print(f" {k[0]}\t{k[1]}\t{k[2]}") + print(f" orphan scan: parts in the run={len(parts)}, rows in those parts={len(checkable)}, " f"orphans={len(orphans)} ({len(with_history)} carrying history), " - f"not checked={len(unchecked)}") + f"unprunable={len(unprunable)}, not checked={len(unchecked)}") + # The four categories must account for every row, or a row went missing in the + # classification itself -- which is the failure this tool exists to report. + matched = len(set(rows) & now) + if matched + len(orphans) + len(unprunable) + len(unchecked) != len(rows): + raise LedgerError( + f"classification lost rows: matched {matched} + orphans {len(orphans)} + " + f"unprunable {len(unprunable)} + not checked {len(unchecked)} != {len(rows)} " + f"ledger rows -- every row must land in exactly one of the four") if not args.prune: - return 1 if orphans else 0 + return 1 if (orphans or unprunable) else 0 + + if unprunable: + print(f" not pruning {len(unprunable)} row(s) in a part that skipped: the run did " + "not exercise those checks, so their absence is not removal") if with_history: print(f" refusing to prune: {len(with_history)} orphan row(s) carry history, and " diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 0f6b4e49..d4682c14 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -2249,6 +2249,22 @@ not an orphan, because the run cannot speak about it — counting those as prese let a one-suite log certify the whole ledger. So the scan states how many rows it could not speak about, and this test pins that number as well as the orphan it found. +### `test_a_part_that_skipped_is_unprunable_because_absence_is_not_removal` + +The first version of `--prune` **deleted a suite**. One SKIP record put the part in the +run's `parts`, so every other row of that suite became an orphan, and the prune removed them +while reporting `not checked=0` and `rc=0` — the most confident output the tool can produce. +`not checked` protects a part the run does not contain; a part *contained but skipped +wholesale* fell in the gap between the two. + +The rule is deliberately broader than that case: a SKIP **anywhere** in the part means some +arm did not run, so the run cannot tell a deleted check from one skipped under a name that +does not match it. One skipped timing check blocks pruning that whole part, which is the +direction a deleting command should err in. + +The control is the half that matters — the same two rows must still be pruned when the +part's record is a `PASS`, or this is simply a tool that refuses to prune anything. + ### `test_prune_drops_a_historyless_orphan_and_refuses_one_carrying_history` The catalogue of what has been seen red is what the ledger exists to be, and no run can diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index 30e588a9..c689a9a1 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -261,6 +261,54 @@ def test_prune_drops_a_historyless_orphan_and_refuses_one_carrying_history(tmp_p "premise: and the refusal removed NOTHING") +def test_a_part_that_skipped_is_unprunable_because_absence_is_not_removal(tmp_path, expect): + """The first version of `--prune` deleted a suite. + + One SKIP record put the part in `parts`, so every other row of that suite became an + orphan, and `--prune` removed them while reporting `not checked=0` and rc=0 -- the + most confident output the tool can produce. `not checked` protects a part the run does + not contain; a part CONTAINED BUT SKIPPED WHOLESALE fell in the gap between the two. + + The rule is deliberately broader than that case: a SKIP anywhere in the part means + some arm did not run, so the run cannot tell a deleted check from one skipped under a + name that does not match it. One skipped timing check blocks pruning that whole part, + which is the direction a deleting command should err in. + + The CONTROL is the half that matters: the same two rows must still be pruned when the + part's record is a PASS, or this is a tool that refuses to prune anything. + """ + ledger = _w(tmp_path, "l.tsv", "") + _run("merge", "--ledger", ledger, "--date", "2026-09-01", + _w(tmp_path, "full.log", + "RESULT\tdemo\tpart1\tarm one\tPASS\t\n" + "RESULT\tdemo\tpart1\tarm two\tPASS\t\n" + "RESULT\tdemo\tpart1\tthe whole thing\tPASS\t\nchecks run: 3\n")) + expect.num(len(_rows(ledger)), 3, "premise: the ledger holds all three rows") + skipped = _w(tmp_path, "skipped.log", + "RESULT\tdemo\tpart1\tthe whole thing\tSKIP\tno fixture on this box\n" + "checks run: 1\n") + + out, rc = _run("orphan-scan", "--ledger", ledger, skipped) + expect.num(out.count("unprunable: 2 row(s)"), 1, + "a row in a part that skipped is unprunable, not an orphan") + expect.num(out.count("orphans=0 (0 carrying history), unprunable=2"), 1, + "and the summary keeps the two apart") + expect.num(rc, 1, "it is still a finding, so the scan does not return success") + + _run("orphan-scan", "--prune", "--ledger", ledger, skipped) + expect.num(len(_rows(ledger)), 3, "--prune removes nothing from a part that skipped") + expect.num(_run("orphan-scan", "--prune", "--ledger", ledger, skipped)[0] + .count("the run did not exercise those checks"), 1, + "and it says so rather than declining silently") + + passed = _w(tmp_path, "pass.log", + "RESULT\tdemo\tpart1\tthe whole thing\tPASS\t\nchecks run: 1\n") + out, _ = _run("orphan-scan", "--prune", "--ledger", ledger, passed) + expect.num(out.count("removed 2 row(s)"), 1, + "control: the same rows ARE pruned when that part's record is a PASS") + expect.num(len(_rows(ledger)), 1, "control: and the ledger really is shorter") + + def test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers(tmp_path, expect): """The suite restriction is the MEANING of `suites_not_covered`, not a softening. diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 79302961..45918cd4 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -311,15 +311,99 @@ check "the refusal says why, rather than only that it refused" \ "$(_led_run orphan-scan --prune --ledger "$_lw/hist.tsv" "$_lw/o_after.log" \ | grep -c 'the catalogue is what this ledger is for')" "1" -# WHY THIS REPORTS AND DOES NOT GATE, measured rather than asserted. Part 340 -# records ONE skip under a DIFFERENT name ("the unreadable-source refusal") when -# the box has no non-root user, instead of skipping its two named arms. So on such -# a box two committed rows have no record and are not removed checks -- a gate -# refusing on absence would redden a correct run. This arm pins the precondition -# for arming it: the branch must record under the names it skips. -check "the conditional branch that makes absence ambiguous is still the OTHER shape" \ - "$(grep -c 'check_skip "the unreadable-source refusal"' \ - "$PGC_TESTDIR/selftest/340-the-binary-must-be-built-from.sh")" "1" +# ---- a part the run SKIPPED is not a part the run can speak about ------------ +# +# The first version of this deleted a suite. One SKIP record put the part in +# `parts`, so every other row of that suite became an orphan, and `--prune` removed +# them while reporting `not checked=0` and rc=0 -- the most confident output the +# tool can produce. `not checked` protects a part the run does not contain at all; +# a part CONTAINED BUT SKIPPED WHOLESALE fell in the gap between the two. +# +# Reported by @pgcolumnar-9b reviewing this change, on a three-row fixture for +# `analyze_differential`, whose run on PG17 is a single SKIP. Nine suites skip +# wholesale on PG17 and `suites_not_covered` is 250, so seeding any one of them +# would have armed it. +# +# THE RULE IS BROADER THAN THAT CASE DELIBERATELY: a SKIP anywhere in the part +# means some arm did not run, so the run cannot tell "this row's check was deleted" +# from "this row's check was skipped under a name that does not match it" -- #994's +# defect at suite granularity. One skipped timing check therefore blocks pruning +# that whole part, and that is the direction a deleting command should err in. + +: > "$_lw/skp.tsv" +printf 'RESULT\tdemo\tpart1\tarm one\tPASS\t\nRESULT\tdemo\tpart1\tarm two\tPASS\t\nRESULT\tdemo\tpart1\tthe whole thing\tPASS\t\nchecks run: 3\n' > "$_lw/sk_full.log" +printf 'RESULT\tdemo\tpart1\tthe whole thing\tSKIP\tno fixture on this box\nchecks run: 1\n' > "$_lw/sk_skipped.log" +_led_run merge --ledger "$_lw/skp.tsv" --date 2026-09-01 "$_lw/sk_full.log" >/dev/null +check "premise: the ledger holds all three of that part's rows" \ + "$(wc -l < "$_lw/skp.tsv" | tr -d ' ')" "3" +check "premise: and the skipped run emits exactly one of them, as a SKIP" \ + "$(awk -F'\t' '$5=="SKIP"' "$_lw/sk_skipped.log" | wc -l | tr -d ' ')" "1" + +check "a row in a part that SKIPPED is reported as unprunable, not as an orphan" \ + "$(_led_run orphan-scan --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log" \ + | grep -c 'unprunable: 2 row(s)')" "1" +check "and the summary keeps the two apart, so a zero orphan count is not a clean bill" \ + "$(_led_run orphan-scan --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log" \ + | grep -c 'orphans=0 (0 carrying history), unprunable=2')" "1" +check "it is still a finding, so the scan does not return success" \ + "$(_led_rc orphan-scan --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log")" "1" + +_led_run orphan-scan --prune --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log" >/dev/null +check "--prune removes NOTHING from a part that skipped" \ + "$(wc -l < "$_lw/skp.tsv" | tr -d ' ')" "3" +check "and it says so rather than declining silently" \ + "$(_led_run orphan-scan --prune --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log" \ + | grep -c 'the run did not exercise those checks')" "1" + +# CONTROL. Without this the arms above are satisfied by a tool that refuses to prune +# anything at all, which is the failure mode of every over-broad guard. +printf 'RESULT\tdemo\tpart1\tthe whole thing\tPASS\t\nchecks run: 1\n' > "$_lw/sk_pass.log" +check "control: the same two rows ARE pruned when that part's record is a PASS" \ + "$(_led_run orphan-scan --prune --ledger "$_lw/skp.tsv" "$_lw/sk_pass.log" \ + | grep -c 'removed 2 row(s)')" "1" +check "control: and the ledger really is shorter afterwards" \ + "$(wc -l < "$_lw/skp.tsv" | tr -d ' ')" "1" + +# Every row must land in exactly one of the four categories, or the classification +# itself lost one -- which is the failure this whole tool exists to report. +check "the four categories account for every ledger row" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" \ + | grep -c 'classification lost rows')" "0" + +# WHY THIS REPORTS AND DOES NOT GATE -- pinned to the PRECONDITION, not to one +# instance of it. +# +# This arm used to grep `340` for `check_skip "the unreadable-source refusal"`: one +# skip standing in for three named arms, so on a box with no non-root user three +# committed rows had no record and were not removed checks. #998 removed that line, +# which SATISFIED the premise rather than breaking it -- and the arm as written would +# then have failed. Worse, it would have failed in `main`: the two PRs compose with a +# conflict only in the budget file, so nothing would have presented a marker to read. +# Found by @pgcolumnar-9b, by composing the merge rather than reasoning about it. +# +# THE PRECONDITION IS NOT "340 HAS THAT LINE". It is that somewhere in the corpus a +# skip still cannot be matched to the arms it stands in for -- while that holds, an +# absent record does not reliably mean a removed check, and a gate refusing on absence +# would redden a correct run. #998's sweep is the authority on that number, so this +# reads it instead of re-deriving it: a second implementation of one count is how two +# numbers come to disagree, which is the defect this file exists to catch. +# +# AND IT RETIRES ITSELF. The day `interpolated` and `armless` both reach zero this arm +# fails, and the fix for that failure is to arm the gate -- which is the direction the +# whole issue wants to go, stated as a check rather than as a comment somebody has to +# remember to re-read. +_orph_tool="$PGC_TESTDIR/../.github/scripts/skip-loop-arms.py" +check "premise: the skip-loop sweep is present, so its counts can be read" \ + "$([ -r "$_orph_tool" ] && echo yes || echo no)" "yes" +_orph_sweep="$(python3 "$_orph_tool" "$PGC_TESTDIR" 2>&1)" || _orph_sweep="TOOL FAILED" +check "premise: and it reported all four of its categories, so a zero is a measurement" \ + "$(printf '%s\n' "$_orph_sweep" | grep -cE '^(loops|compared|interpolated|armless) [0-9]+$')" "4" +_orph_i="$(printf '%s\n' "$_orph_sweep" | sed -n 's/^interpolated \([0-9]*\)$/\1/p')" +_orph_a="$(printf '%s\n' "$_orph_sweep" | sed -n 's/^armless \([0-9]*\)$/\1/p')" +check "the orphan scan stays a REPORT while any skip loop cannot be compared to its arms" \ + "$({ [ "${_orph_i:-0}" -gt 0 ] || [ "${_orph_a:-0}" -gt 0 ]; } && echo "not yet armable" || echo "armable: arm the gate")" \ + "not yet armable" +unset _orph_tool _orph_sweep _orph_i _orph_a check "premise: and the two arms it stands in for are still named in the ledger" \ "$(grep -cP '^harness_selftest\t340-the-binary-must-be-built-from\tpremise: the unprivileged' \ "$_ledger")" "2" From 6bfd9d6d5a53689cc4720ebe52447111e6020fea Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 21:40:42 -0600 Subject: [PATCH 3/6] test: --prune cannot report success when it pruned nothing (#983) Reported by @jdatcmd in review. The two refusals exited differently and the new one looked like success: --prune, orphan CARRYING HISTORY rc=2 --prune, orphan in a SKIPPED part rc=0 <- nothing pruned, nothing prunable plain scan, same skipped part rc=1 So adding --prune took that case from 1 to 0. A caller that scans, sees 1, re-runs with --prune and sees 0 reads "it pruned them" -- when nothing was pruned and nothing could be. Prose covers a human; a script sees only the code. The exit codes are now a contract, written in the docstring because a caller only ever sees the code: 0 nothing left to report: no orphan and nothing unprunable 1 something is still there -- an orphan, or a row this run cannot speak for 2 an integrity failure, or a prune refused because history would be lost Driven, all four states: scan, skipped part rc=1 prune, skipped part rc=1 (was 0) rows unchanged prune, nothing outstanding rc=0 rows 3 -> 1, so 0 is not a refusal prune, history at stake rc=2 Six arms, and the control is the one that matters: 0 must still mean 0, or the code says "work remains" forever and nobody can use it in a script either. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- test/pgc_ledger.py | 19 ++++++++++++-- .../410-a-check-must-have-been-red.sh | 25 +++++++++++++++++++ 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index a792e930..55dfd166 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -416,6 +416,18 @@ def cmd_orphan_scan(args): It is deliberately conservative: one skipped timing check blocks pruning that whole part. Prune is a rare, deliberate act; a refusal costs a sentence and a deletion costs history no run can recreate. + + THE EXIT CODES, stated because a caller only ever sees the code: + + 0 nothing left to report: no orphan and nothing unprunable + 1 something is still there -- an orphan, or a row this run cannot speak for + 2 an integrity failure, or a prune refused because history would be lost + + `--prune` returning 0 when it had pruned NOTHING was the first version's subtler + bug, reported by @jdatcmd in review. A caller that scans, sees 1, re-runs with + `--prune` and sees 0 reads "it pruned them" -- when nothing was pruned and nothing + could be. Prose covers a human; a script sees only the code. So 0 now means the + ledger and the run agree, and anything outstanding keeps the 1 the scan gave. """ runs = _by_run(args.logs) if len(runs) > 1: @@ -493,14 +505,17 @@ def cmd_orphan_scan(args): return 2 if not historyless: - return 0 + # Nothing WAS pruned. If anything is still outstanding the caller must not read + # that as success, so the scan's own verdict stands. + return 1 if unprunable else 0 for k in historyless: print(f" pruned: {k[0]}\t{k[1]}\t{k[2]}") del rows[k] write_ledger(args.ledger, rows) print(f" orphan prune: removed {len(historyless)} row(s), the ledger now holds {len(rows)}") - return 0 + # Pruning some of it is not finishing it. + return 1 if unprunable else 0 def read_budget(path): diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 45918cd4..39d965f8 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -370,6 +370,31 @@ check "the four categories account for every ledger row" \ "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" \ | grep -c 'classification lost rows')" "0" +# ---- and the exit code cannot say "done" when nothing was done -------------- +# +# The first version returned 0 from `--prune` whenever it pruned nothing, including +# when everything it found was unprunable. A caller that scans, sees 1, re-runs with +# `--prune` and sees 0 reads "it pruned them" -- when nothing was pruned and nothing +# could be. Prose covers a human; a script sees only the code. Reported by @jdatcmd +# in review. + +: > "$_lw/rc.tsv" +_led_run merge --ledger "$_lw/rc.tsv" --date 2026-09-01 "$_lw/sk_full.log" >/dev/null +check "premise: that part has rows the skipped run cannot speak for" \ + "$(wc -l < "$_lw/rc.tsv" | tr -d ' ')" "3" +check "the scan reports a finding on a skipped part" \ + "$(_led_rc orphan-scan --ledger "$_lw/rc.tsv" "$_lw/sk_skipped.log")" "1" +check "and --prune does NOT turn that 1 into a 0, because it pruned nothing" \ + "$(_led_rc orphan-scan --prune --ledger "$_lw/rc.tsv" "$_lw/sk_skipped.log")" "1" +check "premise: and it really pruned nothing -- the rows are all still there" \ + "$(wc -l < "$_lw/rc.tsv" | tr -d ' ')" "3" +# CONTROL: 0 still means 0. A prune with nothing outstanding must report success, or +# the code says "work remains" forever and nobody can use it in a script either. +check "control: a prune that leaves nothing outstanding returns 0" \ + "$(_led_rc orphan-scan --prune --ledger "$_lw/rc.tsv" "$_lw/sk_pass.log")" "0" +check "control: and that one did prune, so 0 is not a refusal in disguise" \ + "$(wc -l < "$_lw/rc.tsv" | tr -d ' ')" "1" + # WHY THIS REPORTS AND DOES NOT GATE -- pinned to the PRECONDITION, not to one # instance of it. # From dfc0892103c2e25b30c61b85d2f0c58f9060e926 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 22:22:00 -0600 Subject: [PATCH 4/6] test: regenerate the ledger for the review round (#983) From a guarded harness_selftest run on this tree: rc=0, checks run 907, RESULT records 907, 907 distinct (suite, part, name) keys, ZERO COLLIDING, zero FAILs, the suite's own PASSED verdict. reconcile rows 951 against a 947 budget -> 951, BEFORE running merge no new rows: the run's checks were already all ledgered prune 0 orphans, 0 unprunable census DERIVED: rows=951 never=951, partition closes gate census 951/951 agree | ceiling 250 -> 250, which does not rise | rc=0 THE RECONCILE STEP IS WHY THIS COMMIT IS SELF-CONSISTENT. Rebasing across #1001 left rows=951 against a budget still stating 947: git merges both sides' rows into the tsv and keeps one side's number in the budget. An earlier version of this commit carried that contradiction, which would have left a commit in the history whose own selftest fails -- four arms, all tracing to that one cause. The order is now written into pgc_ledger.py's docstring so the next person does not rediscover it. The colliding-key count is asserted beside the record count because it earned its place on #1001: a new arm of mine shared a name with one four arms above it, and the merge reported `distinct checks this merge=866` against `checks run=867` -- #982's defect, in the session that removed the last of its 24. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- test/check_ledger.tsv | 42 ++++++++++++++++++++++++++++++++++-- test/check_ledger_budget.txt | 2 +- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 8511e774..d2e29c8a 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -322,9 +322,7 @@ harness_selftest 330-the-incomplete-path-must-run-whole control: and the major r harness_selftest 330-the-incomplete-path-must-run-whole control: the same loop leaves a passing suite passing never - harness_selftest 330-the-incomplete-path-must-run-whole every function the collect loop calls is defined here never - harness_selftest 330-the-incomplete-path-must-run-whole premise: all five runner functions were extracted, not empty ranges never - -harness_selftest 330-the-incomplete-path-must-run-whole premise: all three runner functions were extracted, not empty ranges never - harness_selftest 330-the-incomplete-path-must-run-whole premise: and all five are callable never - -harness_selftest 330-the-incomplete-path-must-run-whole premise: and all three are callable never - harness_selftest 330-the-incomplete-path-must-run-whole premise: and each extraction ends at its own closing brace never - harness_selftest 330-the-incomplete-path-must-run-whole premise: the major-verdict branch was extracted, not an empty range never - harness_selftest 330-the-incomplete-path-must-run-whole premise: the runner's collect loop was extracted, not an empty range never - @@ -655,6 +653,10 @@ harness_selftest 400-a-check-result-must-be-machine the same ratio check, ENABLE harness_selftest 400-a-check-result-must-be-machine the same timing check, ENABLED, emits one record and passes never - harness_selftest 400-a-check-result-must-be-machine the sweep finds a check inside a PIPED loop never - harness_selftest 410-a-check-must-have-been-red --mutation across two failing checks in one run is refused never - +harness_selftest 410-a-check-must-have-been-red --prune REFUSES the whole prune when any orphan carries history never - +harness_selftest 410-a-check-must-have-been-red --prune removes NOTHING from a part that skipped never - +harness_selftest 410-a-check-must-have-been-red --prune removes a historyless orphan and says which never - +harness_selftest 410-a-check-must-have-been-red a before-log and an after-log together are refused, as rename-scan refuses them never - harness_selftest 410-a-check-must-have-been-red a before-log and an after-log together are refused, not silently empty never - harness_selftest 410-a-check-must-have-been-red a budget stating no census is not refused, because absence is not a contradiction never - harness_selftest 410-a-check-must-have-been-red a budget that does not exist at the prior is a note, not a refusal never - @@ -670,6 +672,7 @@ harness_selftest 410-a-check-must-have-been-red a date that is not a date is ref harness_selftest 410-a-check-must-have-been-red a deliberate break says so with --mutation never - harness_selftest 410-a-check-must-have-been-red a gate over a nonexistent log is an integrity failure, not a pass never - harness_selftest 410-a-check-must-have-been-red a later green run does not erase an observation never - +harness_selftest 410-a-check-must-have-been-red a ledger row no record in its own part matches is named an orphan never - harness_selftest 410-a-check-must-have-been-red a log carrying a FAIL is refused when no reason is given never - harness_selftest 410-a-check-must-have-been-red a name that appeared while another disappeared is reported as a rename never - harness_selftest 410-a-check-must-have-been-red a named mutation is recorded against the check that reddened never - @@ -679,6 +682,9 @@ harness_selftest 410-a-check-must-have-been-red a ref that exists with the budge harness_selftest 410-a-check-must-have-been-red a ref that exists without the budget is still the bootstrap case never - harness_selftest 410-a-check-must-have-been-red a refusal keeps the regenerate-the-ledger wording never - harness_selftest 410-a-check-must-have-been-red a rename in one part survives an addition in another never - +harness_selftest 410-a-check-must-have-been-red a row in a part that SKIPPED is reported as unprunable, not as an orphan never - +harness_selftest 410-a-check-must-have-been-red a row in a part the run does not contain is not called an orphan never - +harness_selftest 410-a-check-must-have-been-red a run that emits every row in its parts is clean never - harness_selftest 410-a-check-must-have-been-red a run whose checks are all ledgered passes the gate never - harness_selftest 410-a-check-must-have-been-red a second mutation ACCUMULATES rather than replacing the first never - harness_selftest 410-a-check-must-have-been-red a stale prior is named WITH its distance from HEAD never - @@ -688,23 +694,28 @@ harness_selftest 410-a-check-must-have-been-red an empty check name is one too, harness_selftest 410-a-check-must-have-been-red an empty log is one too, because there is nothing to reconcile never - harness_selftest 410-a-check-must-have-been-red an integrity failure says regenerating will not help never - harness_selftest 410-a-check-must-have-been-red an older observation does not overwrite a newer one never - +harness_selftest 410-a-check-must-have-been-red an orphan carrying history is reported as carrying it never - +harness_selftest 410-a-check-must-have-been-red and --prune does NOT turn that 1 into a 0, because it pruned nothing never - harness_selftest 410-a-check-must-have-been-red and CI collects from the retained path rather than the deleted one never - harness_selftest 410-a-check-must-have-been-red and a genuine observation says so with --reds-are-real never - harness_selftest 410-a-check-must-have-been-red and a level prior carries no distance, so zero is silent never - harness_selftest 410-a-check-must-have-been-red and a log that does not reconcile with its own checks run: never - harness_selftest 410-a-check-must-have-been-red and a newer one does never - +harness_selftest 410-a-check-must-have-been-red and a prune with nothing left to remove is clean, not an error never - harness_selftest 410-a-check-must-have-been-red and a raise against that base ref is refused never - harness_selftest 410-a-check-must-have-been-red and a record missing its verdict never - harness_selftest 410-a-check-must-have-been-red and an empty mutation is a placeholder, not an empty last field never - harness_selftest 410-a-check-must-have-been-red and an undated merge does not erase a known date never - harness_selftest 410-a-check-must-have-been-red and it copies them BEFORE removing the build directory, which is the only order that works never - harness_selftest 410-a-check-must-have-been-red and it entered as debt, not as an observation nothing made never - +harness_selftest 410-a-check-must-have-been-red and it is REFUSED, not merely printed never - harness_selftest 410-a-check-must-have-been-red and it is a different sentence from the refusal, not the same one twice never - harness_selftest 410-a-check-must-have-been-red and it is named, so the author knows which one never - harness_selftest 410-a-check-must-have-been-red and it is that case that says the change introduces the file never - harness_selftest 410-a-check-must-have-been-red and it is the new one that is named, not the one already ledgered never - harness_selftest 410-a-check-must-have-been-red and it names the verdict, so the author knows which record never - harness_selftest 410-a-check-must-have-been-red and it runs before the build directory is removed, which is the only place it can never - +harness_selftest 410-a-check-must-have-been-red and it says so rather than declining silently never - harness_selftest 410-a-check-must-have-been-red and it says the change introduces the file rather than raising anything never - harness_selftest 410-a-check-must-have-been-red and it says the ref does not resolve, rather than claiming the file is new never - harness_selftest 410-a-check-must-have-been-red and it says why, rather than falling back to something weaker never - @@ -720,6 +731,8 @@ harness_selftest 410-a-check-must-have-been-red and the census line it always pr harness_selftest 410-a-check-must-have-been-red and the comparison passes when the ceiling did not rise never - harness_selftest 410-a-check-must-have-been-red and the gate prints that it compared them never - harness_selftest 410-a-check-must-have-been-red and the history it is about to lose travels with it never - +harness_selftest 410-a-check-must-have-been-red and the history it would lose is printed with it never - +harness_selftest 410-a-check-must-have-been-red and the ledger is one row shorter afterwards never - harness_selftest 410-a-check-must-have-been-red and the message says how to fix it, because regenerating is the intended action never - harness_selftest 410-a-check-must-have-been-red and the refusal names both values never - harness_selftest 410-a-check-must-have-been-red and the refusal names how many failed, so the author can narrow the run never - @@ -727,26 +740,41 @@ harness_selftest 410-a-check-must-have-been-red and the refusal names the check harness_selftest 410-a-check-must-have-been-red and the refusal names the raise never - harness_selftest 410-a-check-must-have-been-red and the refusal quotes the claim beside the measurement never - harness_selftest 410-a-check-must-have-been-red and the runner names no remote at that call site never - +harness_selftest 410-a-check-must-have-been-red and the scan states how many rows it could not speak about never - +harness_selftest 410-a-check-must-have-been-red and the summary keeps the two apart, so a zero orphan count is not a clean bill never - harness_selftest 410-a-check-must-have-been-red auto refuses when GITHUB_BASE_REF names a ref that is not here never - harness_selftest 410-a-check-must-have-been-red auto uses the base ref when it resolves, and names it never - harness_selftest 410-a-check-must-have-been-red auto with no base ref and no upstream is an integrity failure never - harness_selftest 410-a-check-must-have-been-red both failure arms fail the major never - harness_selftest 410-a-check-must-have-been-red but that suite is counted as not covered, which is the debt never - harness_selftest 410-a-check-must-have-been-red but the gate says so, so the skip is visible rather than silent never - +harness_selftest 410-a-check-must-have-been-red control: a prune that leaves nothing outstanding returns 0 never - harness_selftest 410-a-check-must-have-been-red control: a well-formed log still merges never - harness_selftest 410-a-check-must-have-been-red control: an all-PASS log still merges with no flag at all never - +harness_selftest 410-a-check-must-have-been-red control: and that one did prune, so 0 is not a refusal in disguise never - +harness_selftest 410-a-check-must-have-been-red control: and the ledger really is shorter afterwards never - +harness_selftest 410-a-check-must-have-been-red control: the row in the part the run never mentioned SURVIVES the prune never - harness_selftest 410-a-check-must-have-been-red control: the same log merges with a different reason, so the refusal above is --mutation-across-two-checks and not the log never - +harness_selftest 410-a-check-must-have-been-red control: the same two rows ARE pruned when that part's record is a PASS never - harness_selftest 410-a-check-must-have-been-red each says what was wrong with the input never - harness_selftest 410-a-check-must-have-been-red every committed row has five fields never - harness_selftest 410-a-check-must-have-been-red every row has five fields and no trailing tab never - +harness_selftest 410-a-check-must-have-been-red it is still a finding, so the scan does not return success never - harness_selftest 410-a-check-must-have-been-red lowering it is allowed, which is the direction the burn-down goes never - harness_selftest 410-a-check-must-have-been-red merging a green run records both checks never - harness_selftest 410-a-check-must-have-been-red nor is one merely removed never - harness_selftest 410-a-check-must-have-been-red once the suite is covered, a new check in it IS refused never - harness_selftest 410-a-check-must-have-been-red one --mutation cannot be attributed across several runs at once never - harness_selftest 410-a-check-must-have-been-red premise: and it is a file git has never seen, which is the case that used to fail open never - +harness_selftest 410-a-check-must-have-been-red premise: and it really pruned nothing -- the rows are all still there never - +harness_selftest 410-a-check-must-have-been-red premise: and it reported all four of its categories, so a zero is a measurement never - harness_selftest 410-a-check-must-have-been-red premise: and the nobudget branch does not, which is the bootstrap shape never - +harness_selftest 410-a-check-must-have-been-red premise: and the refusal removed NOTHING -- the row is still there never - +harness_selftest 410-a-check-must-have-been-red premise: and the run the scan is given holds only one of them never - +harness_selftest 410-a-check-must-have-been-red premise: and the skipped run emits exactly one of them, as a SKIP never - +harness_selftest 410-a-check-must-have-been-red premise: and the two arms it stands in for are still named in the ledger never - harness_selftest 410-a-check-must-have-been-red premise: both are never, so this ledger's census is two never - +harness_selftest 410-a-check-must-have-been-red premise: that part has rows the skipped run cannot speak for never - harness_selftest 410-a-check-must-have-been-red premise: the budget is a tracked file too never - harness_selftest 410-a-check-must-have-been-red premise: the budget was restored byte-exact never - harness_selftest 410-a-check-must-have-been-red premise: the check has history before the rename never - @@ -754,24 +782,33 @@ harness_selftest 410-a-check-must-have-been-red premise: the fixture ledger hold harness_selftest 410-a-check-must-have-been-red premise: the fixture log names checks the real ledger already knows never - harness_selftest 410-a-check-must-have-been-red premise: the gate's status block was found in the runner never - harness_selftest 410-a-check-must-have-been-red premise: the hasbudget branch carries the budget never - +harness_selftest 410-a-check-must-have-been-red premise: the ledger holds all three of that part's rows never - +harness_selftest 410-a-check-must-have-been-red premise: the ledger holds all three rows before the scan never - harness_selftest 410-a-check-must-have-been-red premise: the ledger is not empty, so the partition means something never - harness_selftest 410-a-check-must-have-been-red premise: the ledger itself is a tracked file, not a variable never - harness_selftest 410-a-check-must-have-been-red premise: the ledger tool exists never - +harness_selftest 410-a-check-must-have-been-red premise: the orphan about to be pruned carries no history never - +harness_selftest 410-a-check-must-have-been-red premise: the orphan now carries a date and a mutation never - harness_selftest 410-a-check-must-have-been-red premise: the raised copy really does carry a higher ceiling never - harness_selftest 410-a-check-must-have-been-red premise: the real budget is inside a git repository never - harness_selftest 410-a-check-must-have-been-red premise: the scratch prior really is three commits behind never - harness_selftest 410-a-check-must-have-been-red premise: the scratch repo has a committed ceiling and no upstream never - harness_selftest 410-a-check-must-have-been-red premise: the scratch repo has a prior ceiling committed never - +harness_selftest 410-a-check-must-have-been-red premise: the skip-loop sweep is present, so its counts can be read never - harness_selftest 410-a-check-must-have-been-red premise: the workflow file is where this part thinks it is never - harness_selftest 410-a-check-must-have-been-red raising the ceiling above its committed value is refused never - harness_selftest 410-a-check-must-have-been-red raising the ceiling in the tracked file is refused never - harness_selftest 410-a-check-must-have-been-red regenerating the ledger lets the new check through never - harness_selftest 410-a-check-must-have-been-red the budget names a ceiling and a census, and says which is which never - harness_selftest 410-a-check-must-have-been-red the ceiling refuses being exceeded never - +harness_selftest 410-a-check-must-have-been-red the check the run still emits is not called an orphan never - harness_selftest 410-a-check-must-have-been-red the committed census matches the committed ledger never - harness_selftest 410-a-check-must-have-been-red the distance travels with a refusal too, not only with a pass never - +harness_selftest 410-a-check-must-have-been-red the four categories account for every ledger row never - harness_selftest 410-a-check-must-have-been-red the gate refuses to run without the registered suite list never - harness_selftest 410-a-check-must-have-been-red the ledger partitions into observed and never never - +harness_selftest 410-a-check-must-have-been-red the orphan scan stays a REPORT while any skip loop cannot be compared to its arms never - +harness_selftest 410-a-check-must-have-been-red the refusal says why, rather than only that it refused never - harness_selftest 410-a-check-must-have-been-red the runner asks the tool to resolve the prior rather than naming one never - harness_selftest 410-a-check-must-have-been-red the runner captures the gate's status rather than only its success never - harness_selftest 410-a-check-must-have-been-red the runner invokes the ledger gate never - @@ -779,6 +816,7 @@ harness_selftest 410-a-check-must-have-been-red the runner keeps the logs somewh harness_selftest 410-a-check-must-have-been-red the runner passes --against to the gate never - harness_selftest 410-a-check-must-have-been-red the same check in two logs is two runs, not a duplicate never - harness_selftest 410-a-check-must-have-been-red the same name twice in ONE log is a duplicate, and is named never - +harness_selftest 410-a-check-must-have-been-red the scan reports a finding on a skipped part never - harness_selftest 410-a-check-must-have-been-red the stable check is not reported never - harness_selftest 410-a-check-must-have-been-red the suites job fetches the PR base for the ceiling comparison never - harness_selftest 420-a-deleted-part-must-be-visible and every name in the manifest is a part on disk, so a deletion reddens never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 762e8b5d..fb09efcd 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 913 +checks_never_observed_red 951 From 267cd935f73a3f2aea6b1e2b193ed1ecbdfd89f0 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 22:22:00 -0600 Subject: [PATCH 5/6] docs: the regeneration order belongs in the tool, not in my head (#983) Suggested by @jdatcmd after watching this PR pay the tax four times, the fourth because #1001 merged while it was in flight. A rebase moves the LEDGER without moving the BUDGET -- git merges both sides' rows into the tsv and keeps one side's number in the budget -- so the committed pair contradicts itself before anything runs. Four arms then fail and all four trace to that one cause: one asserts the pair agrees, three run the real gate, which correctly refuses a contradiction. An hour to diagnose from those four failures; one paragraph to avoid. 1. rebase onto the new base FIRST 2. DERIVE to reconcile, before running 3. run on the REBASED tree, and guard the log 4. merge the guarded run 5. prune orphans, if any 6. DERIVE again Two derives: step 2 makes the tree self-consistent enough for the suite to pass at all, step 6 records the result of 4 and 5. Both read back from the file, because `old + n` is right once and wrong every time after. NO CHECK CHANGES, and that is verified rather than assumed. Nothing in either harness reads this file's TEXT: the five greps in 410 and the one in test_mutation_ledger.py all read run_all_versions.sh for `pgc_ledger.py" gate`. So the check set cannot move, and the previous commit's run stands for this tree. "A docstring cannot matter" is exactly how a guard that reads a docstring gets broken, so it was checked. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- test/pgc_ledger.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 55dfd166..a981ab8c 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -54,6 +54,30 @@ It is not only mutation runs. Every real CI red fills it, every flake, every bisect. A mutation run is the deliberate accelerator. +REGENERATING ACROSS A REBASE, in this order, and the order is the point +----------------------------------------------------------------------- +Paid four times on one PR before it was written down. A rebase moves the LEDGER +without moving the BUDGET: git merges both sides' rows into the tsv and keeps one +side's number in the budget, so the committed pair contradicts itself before +anything is run. Four arms then fail and all four trace to that one cause -- one +asserts the pair agrees, three run the real gate, which correctly refuses a +contradiction. Diagnosing it from those four failures costs an hour. + + 1. rebase onto the new base FIRST + 2. DERIVE to reconcile the budget from the merged tsv, before running + 3. run the suite on the REBASED tree, and guard the log + 4. merge the guarded run + 5. prune orphans, if any + 6. DERIVE again the final census, read back from the file + +Two derives, not one: step 2 makes the tree self-consistent so the suite can pass at +all, step 6 records the result of steps 4 and 5. Both are READ BACK from the tsv -- +`old + n` is right once and wrong every time after. + +And re-run whatever your evidence names whose FILES moved in the rebase. A gate +statement is a claim about a tree, and a rebase silently changes which tree; saying +which suites you re-ran and which you did not is part of the claim. + FAIL CLOSED ----------- An unreadable file, an empty one, or a record with too few fields is an ERROR. From 2aecdd751ad9e3dc01cb0dd37eb6c75915f23bc2 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Sat, 12 Sep 2026 04:06:40 -0600 Subject: [PATCH 6/6] docs: the uncovered-suite count is removed, not updated (#983) #1002 seeds `differential` and takes suites_not_covered from 250 to 249, which makes four comments stale. @jdatcmd flagged one and suggested removing the number rather than updating it, because it goes stale on every seed. Agreed, and it is four places rather than one: test/pgc_ledger.py:727 test/selftest/410-a-check-must-have-been-red.sh:592 test/pytest/test_mutation_ledger.py:315 test/pytest/TESTS.md:2279 "every check of all 250 uncovered suites" -> "every check of every uncovered suite". The sentence is about the gate's SHAPE -- that it refuses only in a covered suite -- and the count was never load-bearing to it. A number in a comment that nothing asserts is a number that goes wrong quietly; the live one is in check_ledger_budget.txt, where the gate reads it. Fixing it here rather than in #1002 so that PR does not buy a conflict over prose: this PR already touches all four files. NO CHECK CHANGES. No check call is added, removed or renamed -- `git diff` over the four files touches comment and docstring lines only -- so the check set cannot move and the ledger commit stands. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- test/pgc_ledger.py | 2 +- test/pytest/TESTS.md | 2 +- test/pytest/test_mutation_ledger.py | 2 +- test/selftest/410-a-check-must-have-been-red.sh | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index a981ab8c..791170f7 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -724,7 +724,7 @@ def cmd_gate(args): # The suite restriction is not a softening, it is the meaning of # suites_not_covered: the gate cannot refuse a new check in a suite it has # never seen, because it has no idea which of that suite's checks are new. - # Without it the gate refuses every check of all 250 uncovered suites and + # Without it the gate refuses every check of every uncovered suite and # reddens the whole matrix on the first run -- which is a gate somebody turns # off, the failure mode this issue family exists to prevent. # diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index d4682c14..8cea5118 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -2276,7 +2276,7 @@ run never mentioned survives, which is the control that the scope holds under a ### `test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers` The suite restriction is the *meaning* of `suites_not_covered`, not a softening: without -it the gate refuses every check of all 250 uncovered suites and reddens the whole matrix +it the gate refuses every check of every uncovered suite and reddens the whole matrix on its first run. It tightens on its own as suites are seeded, and the deadlock that shipped is pinned as its own arm — regenerating the ledger lets a new check through. diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index c689a9a1..b3eb023f 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -312,7 +312,7 @@ def test_a_part_that_skipped_is_unprunable_because_absence_is_not_removal(tmp_pa def test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers(tmp_path, expect): """The suite restriction is the MEANING of `suites_not_covered`, not a softening. - Without it the gate refuses every check of all 250 uncovered suites and reddens the + Without it the gate refuses every check of every uncovered suite and reddens the whole matrix on its first run -- a gate somebody turns off within the week, which is the failure this issue family exists to prevent. It tightens on its own as suites are seeded. diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 39d965f8..5a241310 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -589,7 +589,7 @@ check "but the gate says so, so the skip is visible rather than silent" \ # ---- the gate cannot refuse a check in a suite it has never seen ------------- # # The suite restriction is the MEANING of suites_not_covered, not a softening of -# the refusal. Without it the gate refuses every check of all 250 uncovered +# the refusal. Without it the gate refuses every check of every uncovered # suites and reddens the whole matrix on its first run -- a gate somebody turns # off within the week, which is the failure this issue family exists to prevent. #