From 723e90194e36434dcdf65ed417c0e86ba9beb1c7 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 20:22:34 -0600 Subject: [PATCH 1/4] test: a red needs a reason before it becomes a ledger row (#946) `merge` already refused a log that does not RECONCILE, and reconciliation is not the property that matters. Both logs that poisoned this ledger on the day it landed reconciled: - mine, on #925: 827 records against `checks run: 827`, fifteen checks red because I had copied the tree without `.git`. Flawless self-consistency, fifteen environment reds, and I caught it by noticing the failures rather than by the tool refusing them. - @OffgridwithJD's, on #944: one FAIL from an unfinished change, recording a red for part 350's doc-coverage check that the check never earned. Two independent routes on day one, from the two people who knew the tool best. A third costs no imagination: a run against a stale `.so`. THE RATE IS THE ARGUMENT, not either instance. AN ENVIRONMENT RED AND A REAL REGRESSION ARE IDENTICAL IN THE LOG. Nothing in a RESULT record says which, so the tool cannot infer it and now makes the caller assert it -- the same move `check_ledger_budget.txt` already makes when it names a census apart from a ceiling. `merge` refuses a log carrying any FAIL unless the caller says which kind of red it is: `--mutation NAME` (exists, a deliberate break) or `--reds-are-real` (new, a genuine observation of the code under test). Refused BEFORE any row is built, so a declined merge is never half-applied. NOT "refuse FAILs unless --mutation", which @OffgridwithJD proposed and we rejected together. A genuine CI red is the most valuable row this ledger can hold and has no mutation to name, so that rule would refuse precisely the entry the ledger exists for -- the deadlock the budget file already argues against for `checks_never_observed_red`. A DEFAULT REFUSAL IS AFFORDABLE BECAUSE NOTHING AUTOMATED CALLS `merge`. `grep -rn 'pgc_ledger.py merge' .github/ test/` finds one hit and it is the gate's own advice string. `gate` is the automated path. That was the cost I expected to have to weigh and it does not exist. Both harnesses, independently implemented. The shell twin builds its own fixtures and names no pytest file; the pytest twin drives the tool directly. EIGHT EXISTING CALL SITES UPDATED, AND I CHECKED EACH RATHER THAN CHASING GREEN. Four arms broke, all of them using a red log as a VEHICLE for a different subject -- duplicate detection, rename grouping, date monotonicity, mutation attribution. None asserts that a red merges without a reason, so passing the new flag keeps each testing what it tests. One needed more than that: the control in `test_a_mutation_names_one_check_not_every_casualty` was named "the same log merges without --mutation", which stops being what it demonstrates. It now says the refusal above is --mutation-across-two-checks and not the log. Proved by removal, and by a control that must not move: guard neutered (`if False`) the new arm FAILS, alone control an all-PASS log merges with no flag, rc=0 driver-free pytest 196 passed Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- test/pgc_ledger.py | 35 +++++++++++ test/pytest/TESTS.md | 15 +++++ test/pytest/test_mutation_ledger.py | 61 ++++++++++++++++--- .../410-a-check-must-have-been-red.sh | 43 +++++++++++++ 4 files changed, 145 insertions(+), 9 deletions(-) diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 949dedc1..1b6b8648 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -258,6 +258,38 @@ def cmd_merge(args): f"collateral damage as evidence. Merge without --mutation, or narrow the " f"run to the check the mutation targets.") + # A RED NEEDS A REASON (#946). `merge` already refuses a log that does not + # RECONCILE, and reconciliation is not the property that matters: both logs that + # poisoned this ledger on the day it landed reconciled. One was 827 records + # against `checks run: 827` with fifteen checks red because the tree was copied + # without `.git`; the other was one FAIL from an unfinished change. Two + # independent routes on day one, from the two people who knew the tool best, and + # a third -- a run against a stale .so -- costs no imagination at all. + # + # An environment red and a real regression are IDENTICAL in the log. Nothing in a + # RESULT record says which, so the tool makes the caller assert it rather than + # guess, the same way check_ledger_budget.txt names a census apart from a ceiling. + # + # NOT "refuse FAILs unless --mutation". A genuine CI red is the most valuable row + # this ledger can hold and it has no mutation to name, so that rule would refuse + # precisely the entry the ledger exists for -- the deadlock the budget file + # already argues against for checks_never_observed_red. + # + # Refused BEFORE any row is built, so a declined merge is never half-applied. + if not args.mutation and not args.reds_are_real: + reddened = sorted({key for _p, seen in runs + for key, vs in seen.items() if "FAIL" in vs}) + if reddened: + listed = "\n".join(f" {s}\t{p}\t{n}" for s, p, n in reddened[:6]) + more = "" if len(reddened) <= 6 else f"\n ... and {len(reddened) - 6} more" + raise LedgerError( + f"{len(reddened)} check(s) are red in these logs and nothing says " + f"why:\n{listed}{more}\n A log can reconcile perfectly and still " + f"be evidence about your environment rather than about the code -- a " + f"tree without .git, an unfinished change, a stale .so. Pass " + f"--mutation NAME if you broke it deliberately, or --reds-are-real if " + f"this is a genuine observation of the code under test.") + for path, seen in runs: for key, verdicts in sorted(seen.items()): if key not in rows: @@ -591,6 +623,9 @@ def main(argv=None): m.add_argument("--ledger", required=True) m.add_argument("--date", default="unknown") m.add_argument("--mutation", default="") + m.add_argument("--reds-are-real", action="store_true", + help="the FAIL records in these logs are a genuine observation " + "of the code under test, not an artifact of the environment") m.add_argument("logs", nargs="+") m.set_defaults(fn=cmd_merge) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index b044b049..bbd7a754 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1941,6 +1941,21 @@ that the mutation kills that check. A run with more than one failing check is refused with the count, and a single failure still carries the mutation on the check that reddened. +### `test_a_reconciling_log_with_a_red_is_not_evidence_on_its_own` + +`merge` already refuses a log that does not **reconcile**, and reconciliation is not +the property that matters: both logs that poisoned this ledger on the day it landed +reconciled. One was 827 records against `checks run: 827`, with fifteen checks red +because the tree had been copied without `.git`; the other was a single `FAIL` from +an unfinished change. + +An environment red and a real regression are identical in the log, so the tool cannot +tell them apart and makes the caller say which it is: `--mutation NAME` for a +deliberate break, `--reds-are-real` for a genuine observation. Refusing reds outright +was rejected — a real CI red is the most valuable row the ledger holds and has no +mutation to name. An all-`PASS` log still merges with no flag, which is the control. +See #946. + ### `test_two_runs_of_a_check_are_not_a_duplicate_of_it` Merging logs first cannot tell *the same check in two runs* from *the same name twice in diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index 3ba7ff5c..ae6a90b6 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -136,7 +136,7 @@ def test_two_runs_of_a_check_are_not_a_duplicate_of_it(tmp_path, expect): twice = _w(tmp_path, "twice.log", "RESULT\tdemo\tpart1\tsame\tPASS\t\n" "RESULT\tdemo\tpart1\tsame\tFAIL\t\nchecks run: 2\n") - out, _ = _run("merge", "--ledger", _w(tmp_path, "l2.tsv", ""), "--date", "2026-09-10", twice) + out, _ = _run("merge", "--reds-are-real", "--ledger", _w(tmp_path, "l2.tsv", ""), "--date", "2026-09-10", twice) expect.num(out.count("duplicate check name in one run, so one ledger row covers 2: " "demo\tpart1\tsame"), 1, "the same name twice in ONE log is a duplicate, and is named") @@ -158,7 +158,7 @@ def test_renames_are_grouped_by_part_and_scanned_against_one_run(tmp_path, expec "RESULT\tdemo\tpartA\tnew A\tPASS\t\n" "RESULT\tdemo\tpartB\tstable B\tPASS\t\n" "RESULT\tdemo\tpartB\tadded B\tPASS\t\nchecks run: 3\n") - _run("merge", "--ledger", ledger, "--date", "2026-09-01", before) + _run("merge", "--reds-are-real", "--ledger", ledger, "--date", "2026-09-01", before) out, rc = _run("rename-scan", "--ledger", ledger, after) expect.num(out.count("possible rename: old A -> new A"), 1, @@ -354,14 +354,14 @@ def stored(): return f[3] return None - _run("merge", "--ledger", led, "--date", "2026-09-10", red) - _run("merge", "--ledger", led, "--date", "2026-09-01", red) + _run("merge", "--reds-are-real", "--ledger", led, "--date", "2026-09-10", red) + _run("merge", "--reds-are-real", "--ledger", led, "--date", "2026-09-01", red) expect.text(stored(), "2026-09-10", "an older observation does not overwrite a newer one") - _run("merge", "--ledger", led, "--date", "2026-09-20", red) + _run("merge", "--reds-are-real", "--ledger", led, "--date", "2026-09-20", red) expect.text(stored(), "2026-09-20", "and a newer one does") - _run("merge", "--ledger", led, red) + _run("merge", "--reds-are-real", "--ledger", led, red) expect.text(stored(), "2026-09-20", "and an undated merge does not erase a known date") - expect.num(_run("merge", "--ledger", led, "--date", "not-a-date", red)[1], 2, + expect.num(_run("merge", "--reds-are-real", "--ledger", led, "--date", "not-a-date", red)[1], 2, "a date that is not a date is refused rather than stored") @@ -380,8 +380,8 @@ def test_a_mutation_names_one_check_not_every_casualty(tmp_path, expect): expect.num(rc, 2, "--mutation across two failing checks in one run is refused") expect.num(out.count("2 checks failed"), 1, "and the refusal counts them, so the author can narrow the run") - expect.num(_run("merge", "--ledger", led, "--date", "2026-09-10", two)[1], 0, - "control: the same log merges without --mutation") + expect.num(_run("merge", "--reds-are-real", "--ledger", led, "--date", "2026-09-10", two)[1], 0, + "control: the same log merges with a different reason, so the\n refusal above is --mutation-across-two-checks and not the log") one = _w(tmp_path, "one.log", RED) _run("merge", "--ledger", led, "--date", "2026-09-10", "--mutation", "M", one) @@ -389,3 +389,46 @@ def test_a_mutation_names_one_check_not_every_casualty(tmp_path, expect): tagged = [r[2] for r in rows if len(r) > 4 and "M" in r[4].split(";")] expect.rows([[n] for n in sorted(tagged)], [["first check"]], "and a single failure still carries it, on the check that reddened") + + +def test_a_reconciling_log_with_a_red_is_not_evidence_on_its_own(tmp_path, expect): + """#946: self-consistency is not the property that matters. + + `merge` already refuses a log that does not reconcile. Both of the logs that + poisoned this ledger on the day it landed RECONCILED: @jdatcmd's was 827 records + against `checks run: 827` with fifteen checks red because the tree was copied + without `.git`, and @OffgridwithJD's was one FAIL from an unfinished change. A + rule about reconciliation would have caught neither. + + An environment red and a real regression are IDENTICAL in the log, so the tool + cannot tell them apart and must make the caller say which it is -- the same move + `check_ledger_budget.txt` makes when it names a census apart from a ceiling. + """ + led = _w(tmp_path, "l.tsv", "") + red = _w(tmp_path, "red.log", RED) + + out, rc = _run("merge", "--ledger", led, "--date", "2026-09-10", red) + expect.num(rc, 2, "a log carrying a FAIL is refused when no reason is given") + expect.at_least(out.count("first check"), 1, + "and the refusal names the check that reddened") + expect.text("absent" if not pathlib.Path(led).read_text().strip() else "written", + "absent", "and nothing is written, so a refused merge is not half-applied") + + # THE TWO WAYS TO SAY WHY, both of which must still work. Refusing reds outright + # would refuse the most valuable row the ledger can hold -- a genuine CI red, which + # has no mutation to name -- and that is the deadlock the budget file already + # argues against for checks_never_observed_red. + led2 = _w(tmp_path, "l2.tsv", "") + expect.num(_run("merge", "--ledger", led2, "--date", "2026-09-10", + "--mutation", "M", red)[1], 0, + "a deliberate break says so with --mutation") + led3 = _w(tmp_path, "l3.tsv", "") + expect.num(_run("merge", "--ledger", led3, "--date", "2026-09-10", + "--reds-are-real", red)[1], 0, + "and a genuine observation says so with --reds-are-real") + + # THE CONTROL, without which this arm passes on a tool that refuses everything. + led4 = _w(tmp_path, "l4.tsv", "") + green = _w(tmp_path, "green.log", GREEN) + expect.num(_run("merge", "--ledger", led4, "--date", "2026-09-10", green)[1], 0, + "control: an all-PASS log still merges with no flag at all") diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index f5e3fd27..69843a3f 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -648,3 +648,46 @@ check "and it is that case that says the change introduces the file" \ check "a ref that exists with the budget still compares" \ "$(cd "$_rr" && python3 "$_led" gate --ledger led --budget b.txt --registered reg \ --against hasbudget log 2>&1 | grep -c 'ceiling against hasbudget')" "1" + +# ---- a red needs a reason (#946) -------------------------------------------- +# +# `merge` already refuses a log that does not RECONCILE, and that is not the +# property that matters: both logs that poisoned this ledger on the day it landed +# reconciled. One was 827 records against `checks run: 827` with fifteen checks red +# because the tree was copied without `.git`; the other was a single FAIL from an +# unfinished change. A rule about reconciliation would have caught neither. +# +# An environment red and a real regression are IDENTICAL in the log, so the caller +# has to say which it is rather than the tool guessing. + +_r946="$PGC_WORKDIR/r946"; mkdir -p "$_r946" +printf 'RESULT\tdemo\tp\tthe check\tFAIL\t\nRESULT\tdemo\tp\tother\tPASS\t\nchecks run: 2\n' > "$_r946/red.log" +printf 'RESULT\tdemo\tp\tthe check\tPASS\t\nRESULT\tdemo\tp\tother\tPASS\t\nchecks run: 2\n' > "$_r946/green.log" + +: > "$_r946/a.tsv" +check "a log carrying a FAIL is refused when no reason is given" \ + "$(_led_rc merge --ledger "$_r946/a.tsv" --date 2026-09-10 "$_r946/red.log")" "2" +check "and the refusal names the check that reddened" \ + "$(_led_run merge --ledger "$_r946/a.tsv" --date 2026-09-10 "$_r946/red.log" \ + | grep -c 'the check')" "1" +# NOTHING HALF-APPLIED. A refusal that had already written rows would leave the +# ledger holding the very observation it just declined to accept. +check "and nothing is written, so a refused merge is not half-applied" \ + "$(wc -c < "$_r946/a.tsv" | tr -d ' ')" "0" + +# BOTH WAYS OF SAYING WHY MUST STILL WORK. Refusing reds outright would refuse a +# genuine CI red, which is the most valuable row this ledger can hold and has no +# mutation to name. +: > "$_r946/b.tsv" +check "a deliberate break says so with --mutation" \ + "$(_led_rc merge --ledger "$_r946/b.tsv" --date 2026-09-10 --mutation M "$_r946/red.log")" "0" +: > "$_r946/c.tsv" +check "and a genuine observation says so with --reds-are-real" \ + "$(_led_rc merge --ledger "$_r946/c.tsv" --date 2026-09-10 --reds-are-real "$_r946/red.log")" "0" + +# THE CONTROL, without which every arm above passes on a tool that refuses +# everything it is handed. +: > "$_r946/d.tsv" +check "control: an all-PASS log still merges with no flag at all" \ + "$(_led_rc merge --ledger "$_r946/d.tsv" --date 2026-09-10 "$_r946/green.log")" "0" +unset _r946 From f9593c265cb35252332ddfc0d4456e46ee426d15 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 20:25:03 -0600 Subject: [PATCH 2/4] test: the shell part's own eight vehicle call sites (#946) The shell twin holds its own copies of the same call sites, and they broke independently of the pytest ones -- which is the independence rule working rather than a duplicate failure. Eight merge calls feed a red log as a VEHICLE for a different subject (date monotonicity, duplicate naming, rename grouping, mutation attribution); each now says why. The control named 'the same log merges without --mutation' is renamed for the same reason its pytest counterpart was: that is no longer what it demonstrates. CHECKED, NOT ASSUMED: `--date not-a-date` still fails on the DATE, because the date validation runs at the top of cmd_merge and this guard sits below it. A deny arm is evidence only if the call reached the code that denies it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- .../selftest/410-a-check-must-have-been-red.sh | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 69843a3f..e3427a5f 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -93,14 +93,14 @@ check "control: a well-formed log still merges" \ # `unknown`. A free-form --date was accepted verbatim, so a typo became an # observation date the ledger treated as authoritative. : > "$_lw/date.tsv" -_led_run merge --ledger "$_lw/date.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null -_led_run merge --ledger "$_lw/date.tsv" --date 2026-09-01 "$_lw/red.log" >/dev/null +_led_run merge --reds-are-real --ledger "$_lw/date.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null +_led_run merge --reds-are-real --ledger "$_lw/date.tsv" --date 2026-09-01 "$_lw/red.log" >/dev/null check "an older observation does not overwrite a newer one" \ "$(awk -F'\t' '$3=="first check"{print $4}' "$_lw/date.tsv")" "2026-09-10" -_led_run merge --ledger "$_lw/date.tsv" --date 2026-09-20 "$_lw/red.log" >/dev/null +_led_run merge --reds-are-real --ledger "$_lw/date.tsv" --date 2026-09-20 "$_lw/red.log" >/dev/null check "and a newer one does" \ "$(awk -F'\t' '$3=="first check"{print $4}' "$_lw/date.tsv")" "2026-09-20" -_led_run merge --ledger "$_lw/date.tsv" "$_lw/red.log" >/dev/null +_led_run merge --reds-are-real --ledger "$_lw/date.tsv" "$_lw/red.log" >/dev/null check "and an undated merge does not erase a known date" \ "$(awk -F'\t' '$3=="first check"{print $4}' "$_lw/date.tsv")" "2026-09-20" check "a date that is not a date is refused rather than stored" \ @@ -118,8 +118,8 @@ check "--mutation across two failing checks in one run is refused" \ check "and the refusal names how many failed, so the author can narrow the run" \ "$(_led_run merge --ledger "$_lw/m2.tsv" --date 2026-09-10 --mutation M "$_lw/twofail.log" \ | grep -c '2 checks failed')" "1" -check "control: the same log merges without --mutation" \ - "$(_led_rc merge --ledger "$_lw/m2.tsv" --date 2026-09-10 "$_lw/twofail.log")" "0" +check "control: the same log merges with a different reason, so the refusal above is --mutation-across-two-checks and not the log" \ + "$(_led_rc merge --reds-are-real --ledger "$_lw/m2.tsv" --date 2026-09-10 "$_lw/twofail.log")" "0" # The three must be distinguishable from a REAL refusal, or fail-closed just # renames every outcome. @@ -143,7 +143,7 @@ check "every row has five fields and no trailing tab" \ check "and an empty mutation is a placeholder, not an empty last field" \ "$(grep -cP '\t$' "$_lw/ledger.tsv" || true)" "0" -_led_run merge --ledger "$_lw/ledger.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null +_led_run merge --reds-are-real --ledger "$_lw/ledger.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null check "a check observed red gains the date it was seen" \ "$(awk -F'\t' '$3=="first check"{print $4}' "$_lw/ledger.tsv")" "2026-09-10" check "and one that stayed green keeps its debt" \ @@ -182,7 +182,7 @@ check "the same check in two logs is two runs, not a duplicate" \ printf 'RESULT\tdemo\tpart1\tsame\tPASS\t\nRESULT\tdemo\tpart1\tsame\tFAIL\t\nchecks run: 2\n' > "$_lw/twice.log" : > "$_lw/dup2.tsv" check "the same name twice in ONE log is a duplicate, and is named" \ - "$(_led_run merge --ledger "$_lw/dup2.tsv" --date 2026-09-10 "$_lw/twice.log" \ + "$(_led_run merge --reds-are-real --ledger "$_lw/dup2.tsv" --date 2026-09-10 "$_lw/twice.log" \ | grep -c 'duplicate check name in one run, so one ledger row covers 2: demo part1 same')" "1" # ---- renames, grouped by part and scanned against ONE run ------------------- @@ -195,7 +195,7 @@ check "the same name twice in ONE log is a duplicate, and is named" \ : > "$_lw/ren.tsv" printf 'RESULT\tdemo\tpart1\tthe old name\tFAIL\t\nRESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 2\n' > "$_lw/before.log" printf 'RESULT\tdemo\tpart1\tthe new name\tPASS\t\nRESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 2\n' > "$_lw/after.log" -_led_run merge --ledger "$_lw/ren.tsv" --date 2026-09-01 "$_lw/before.log" >/dev/null +_led_run merge --reds-are-real --ledger "$_lw/ren.tsv" --date 2026-09-01 "$_lw/before.log" >/dev/null check "premise: the check has history before the rename" \ "$(awk -F'\t' '$3=="the old name"{print $4}' "$_lw/ren.tsv")" "2026-09-01" check "a name that appeared while another disappeared is reported as a rename" \ From 34b4590729b234ac7ae71bb30ea1837ad67fff21 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 20:26:33 -0600 Subject: [PATCH 3/4] test: the ledger and census the new checks oblige (#946) Seven new checks in part 410, so two tree-wide artifacts move with them. I enumerated what a new check obliges rather than fixing whatever broke first, which is the lesson #925 cost three rounds to learn: TESTS.md entry done in the first commit NO_CLUSTER membership already true, test_mutation_ledger.py is driver-free shell twin done in the second commit parts.manifest part 410 already listed, unchanged check_ledger.tsv +7 rows, this commit check_ledger_budget.txt census 826 -> 833, this commit REGENERATED FROM A GREEN RUN, and this change is its own demonstration: the log is 833 records with zero FAILs, so the guard added here does not refuse it. A red run would now be refused by the very thing being added, which is what #925's `.git`-less regeneration needed and did not have. harness_selftest, PG17 rc=0, 833 passed + 0 failed + 0 unrunnable + 0 skipped = 833 ledger 826 -> 833 rows, all 7 in 410-a-check-must-have-been-red rows removed or altered 0 census 826 -> 833, DERIVED from the ledger, not typed ceiling 250, untouched -- adding checks to a covered suite cannot move it, as the budget file predicts driver-free pytest 196 passed Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- test/check_ledger.tsv | 7 +++++++ test/check_ledger_budget.txt | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 7f10fbaa..b1547222 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -683,8 +683,10 @@ harness_selftest 410-a-check-must-have-been-red a check observed red gains the d harness_selftest 410-a-check-must-have-been-red a check the ledger has never seen is refused never - harness_selftest 410-a-check-must-have-been-red a clean status says nothing and does not fail the major never - harness_selftest 410-a-check-must-have-been-red a date that is not a date is refused rather than stored never - +harness_selftest 410-a-check-must-have-been-red a deliberate break says so with --mutation never - harness_selftest 410-a-check-must-have-been-red a gate over a nonexistent log is an integrity failure, not a pass never - harness_selftest 410-a-check-must-have-been-red a later green run does not erase an observation never - +harness_selftest 410-a-check-must-have-been-red a log carrying a FAIL is refused when no reason is given never - harness_selftest 410-a-check-must-have-been-red a name that appeared while another disappeared is reported as a rename never - harness_selftest 410-a-check-must-have-been-red a named mutation is recorded against the check that reddened never - harness_selftest 410-a-check-must-have-been-red a real refusal is a different status from an integrity failure never - @@ -703,6 +705,7 @@ harness_selftest 410-a-check-must-have-been-red an empty log is one too, because harness_selftest 410-a-check-must-have-been-red an integrity failure says regenerating will not help never - harness_selftest 410-a-check-must-have-been-red an older observation does not overwrite a newer one never - harness_selftest 410-a-check-must-have-been-red and CI collects from the retained path rather than the deleted one never - +harness_selftest 410-a-check-must-have-been-red and a genuine observation says so with --reds-are-real never - harness_selftest 410-a-check-must-have-been-red and a level prior carries no distance, so zero is silent never - harness_selftest 410-a-check-must-have-been-red and a log that does not reconcile with its own checks run: never - harness_selftest 410-a-check-must-have-been-red and a newer one does never - @@ -724,6 +727,7 @@ harness_selftest 410-a-check-must-have-been-red and it says why, rather than fal harness_selftest 410-a-check-must-have-been-red and never says a change introduces a file at a ref that is not there never - harness_selftest 410-a-check-must-have-been-red and none of them ends in a tab never - harness_selftest 410-a-check-must-have-been-red and not against one that stayed green never - +harness_selftest 410-a-check-must-have-been-red and nothing is written, so a refused merge is not half-applied never - harness_selftest 410-a-check-must-have-been-red and one that stayed green keeps its debt never - harness_selftest 410-a-check-must-have-been-red and only when there is a base, so a push build does not fail on it never - harness_selftest 410-a-check-must-have-been-red and records neither as ever having been red never - @@ -733,6 +737,7 @@ harness_selftest 410-a-check-must-have-been-red and the history it is about to l harness_selftest 410-a-check-must-have-been-red and the message says how to fix it, because regenerating is the intended action never - harness_selftest 410-a-check-must-have-been-red and the refusal names both values never - harness_selftest 410-a-check-must-have-been-red and the refusal names how many failed, so the author can narrow the run never - +harness_selftest 410-a-check-must-have-been-red and the refusal names the check that reddened never - harness_selftest 410-a-check-must-have-been-red and the refusal names the raise never - harness_selftest 410-a-check-must-have-been-red and the runner names no remote at that call site never - harness_selftest 410-a-check-must-have-been-red auto refuses when GITHUB_BASE_REF names a ref that is not here never - @@ -741,6 +746,8 @@ harness_selftest 410-a-check-must-have-been-red auto with no base ref and no ups harness_selftest 410-a-check-must-have-been-red both failure arms fail the major never - harness_selftest 410-a-check-must-have-been-red but that suite is counted as not covered, which is the debt never - harness_selftest 410-a-check-must-have-been-red control: a well-formed log still merges never - +harness_selftest 410-a-check-must-have-been-red control: an all-PASS log still merges with no flag at all never - +harness_selftest 410-a-check-must-have-been-red control: the same log merges with a different reason, so the refusal above is --mutation-across-two-checks and not the log never - harness_selftest 410-a-check-must-have-been-red control: the same log merges without --mutation never - harness_selftest 410-a-check-must-have-been-red each says what was wrong with the input never - harness_selftest 410-a-check-must-have-been-red every committed row has five fields never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 01a8c04f..5ff6723a 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 826 +checks_never_observed_red 833 From f80ca7d053fb74bb874ad84514801fcd99742df3 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 21:25:28 -0600 Subject: [PATCH 4/4] test: the census from a run on the composed tree, and one stale row (#946) #944 landing took the census 826 -> 756 while this branch takes it to 833, so neither number survives the compose. Arithmetic says 763. The rows are the source and the number is derived, so it came from a run rather than from addition. THE LEDGER AUTO-MERGED, WHICH IS THE DANGEROUS CASE, so I checked it against the run rather than against its own count. It took main's deletions and this branch's additions correctly: part 350 main 5 mine 53 -> 5 main's deletion kept part 370 main 0 mine 12 -> 0 main's deletion kept part 380 main 12 mine 21 -> 12 main's deletion kept part 410 main 111 mine 118 -> 118 this branch's addition kept Verified by set comparison against a real run on the composed tree, not by count: checks in the run absent from the ledger, 0. AND ONE STALE ROW THAT WAS MINE TO CLEAN. Three ledger rows did not appear in the run. Two are pre-existing conditional checks in part 330 that also sit in main's ledger. The third is the check THIS BRANCH RENAMED: main's ledger carries it under the old name, this branch adds it under the new one, and the merge kept both. The rename-scan subcommand reports vanished=3 and cannot pair them, because the new name was already present. Dropped after checking it carried nothing: verdict "never", no mutation, so no history was lost. A row with a date or a mutation would have had to travel with the rename instead. selftest on the composed tree 763 checks, 762 passed + 1 failed the one failure was the census arm naming 763, which is the arm doing its job ledger 763 -> 762 rows after the stale row census derived from the ledger, not typed ceiling 250, untouched driver-free pytest 208 passed AND A DEFECT IN THIS COMMIT'S OWN MESSAGE, fixed by amend: I wrote the first version through an unquoted heredoc, so the backticked words rename-scan and never were run as commands and left holes in the text. The message is written from a file now. A commit message is the durable record and a hole in it is a hole in the record. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- test/check_ledger.tsv | 1 - test/check_ledger_budget.txt | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 0c3e4b4b..422ae9f0 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -678,7 +678,6 @@ harness_selftest 410-a-check-must-have-been-red but that suite is counted as not harness_selftest 410-a-check-must-have-been-red control: a well-formed log still merges never - harness_selftest 410-a-check-must-have-been-red control: an all-PASS log still merges with no flag at all never - harness_selftest 410-a-check-must-have-been-red control: the same log merges with a different reason, so the refusal above is --mutation-across-two-checks and not the log never - -harness_selftest 410-a-check-must-have-been-red control: the same log merges without --mutation never - harness_selftest 410-a-check-must-have-been-red each says what was wrong with the input never - harness_selftest 410-a-check-must-have-been-red every committed row has five fields never - harness_selftest 410-a-check-must-have-been-red every row has five fields and no trailing tab never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 67080c34..08560d1b 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 756 +checks_never_observed_red 762