From 0dc46ea947bbc644857bec3238216831348c2045 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 06:26:04 +0000 Subject: [PATCH] test: the ledger gate compares the census it prints (#952) `pgc_ledger.py gate` printed `ledger census: rows=N` and never compared that number to the `checks_never_observed_red` the budget states, so it returned 0 on a twenty-row ledger claiming five. Reporting is not enforcing. The comparison already existed one layer out, in a selftest arm. That arm runs on a pull request, so it reports the disagreement after the merge that creates it rather than before. And a merge is what creates it: the census is a measurement of the tree, so every merge invalidates it. Two PRs each re-derive it from the same base, the merged ledger takes both sets of rows, and the budget keeps whichever side won the conflict. This branch demonstrated its own premise twice. Cut from a main stating 762, it has since been re-derived across the merges of #945, #948 and #943; main now states 819 and this states 829. Each time the correct operation was to regenerate both derived files from a run, never to merge them as text or to add up the parts. The new refusal is decidable from the two inputs alone. It needs no prior and no `--against`, which is what lets it speak about a merge commit: the composed tree is precisely where the prior is the thing in question, so a refusal needing a trustworthy prior would be unavailable exactly when it is needed. It does not make the census a ceiling. A ceiling refuses a rise, and bounding this number deadlocks: every added check enters as `never`, so landing one would mean raising a number the design says may only fall. What is refused is a contradiction, in either direction. A budget stating no census at all is reported rather than refused, because absence is not a contradiction. That is measured rather than preferred: every other gate fixture in both harnesses writes a budget stating only `suites_not_covered`, so refusing there would redden about twenty arms testing something else. What holds the committed budget to naming both numbers is a separate arm in each harness. Red first, in both harnesses, independently implemented: ten checks in selftest 410 and one test in `test_mutation_ledger.py`. Both were run against the unfixed tool and failed, and the core measurement was reproduced on its own: rc=0 for a budget claiming 1 and for one claiming 3 against a ledger holding 2. Gated on the composed tree: docs_style 9/0, harness_selftest 786/0, shellcheck -S error -s bash clean. The ten new ledger rows entered as `never` with no observed red, and the census was re-derived from the run rather than computed: 819 -> 829. Arithmetic would also have said 829, which is the dangerous case rather than the reassuring one, and only the run established it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- CHANGELOG.md | 23 ++++++++ test/check_ledger.tsv | 10 ++++ test/check_ledger_budget.txt | 2 +- test/pgc_ledger.py | 35 ++++++++++++ test/pytest/TESTS.md | 23 ++++++++ test/pytest/test_mutation_ledger.py | 51 ++++++++++++++++++ .../410-a-check-must-have-been-red.sh | 53 +++++++++++++++++++ 7 files changed, 196 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 327e0219..146321a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1259,6 +1259,29 @@ true until the next version shipped. now re-records. It names the two cases that remain: a declaration that no longer resolves, and the implicit base projection, which is not readable by name at all. +- `pgc_ledger.py gate` no longer certifies a census that contradicts its own + ledger (#952). + + The gate printed `ledger census: rows=N` and never compared that number to the + `checks_never_observed_red` the budget states, so it returned 0 on a twenty-row + ledger claiming five. Reporting is not enforcing. The comparison existed one layer + out, in a selftest arm, which runs on a pull request and therefore reports the + disagreement after the merge that creates it rather than before. + + It creates it because the census is a measurement of the tree, so every merge + invalidates it: two pull requests each re-derive it from the same base, the merged + ledger takes both sets of rows, and the budget keeps whichever side won the + conflict. Three in flight at once set 769, 762 and 800 against a base of 756, and + no two composed. The new refusal is decidable from the two inputs alone, needing no + prior and no `--against`, which is what lets it speak about a merge commit. + + It refuses a contradiction in either direction and does not make the census a + ceiling. A ceiling refuses a rise, and bounding this number deadlocks: every added + check enters as `never`, so landing one would mean raising a number the design says + may only fall. A budget that states no census at all is reported rather than + refused, since absence is not a contradiction; what holds the committed budget to + naming both numbers is a separate arm in each harness. + ## [1.0-alpha3] - 2026-09-02 ### Added diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 26563e13..165fa5e0 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -619,7 +619,11 @@ harness_selftest 400-a-check-result-must-be-machine the same timing check, ENABL harness_selftest 400-a-check-result-must-be-machine the sweep finds a check inside a PIPED loop never - harness_selftest 410-a-check-must-have-been-red --mutation across two failing checks in one run is refused never - harness_selftest 410-a-check-must-have-been-red a before-log and an after-log together are refused, not silently empty never - +harness_selftest 410-a-check-must-have-been-red a budget stating no census is not refused, because absence is not a contradiction never - harness_selftest 410-a-check-must-have-been-red a budget that does not exist at the prior is a note, not a refusal never - +harness_selftest 410-a-check-must-have-been-red a census that matches the ledger passes never - +harness_selftest 410-a-check-must-have-been-red a census that overstates the ledger is refused too, because this is not a ceiling never - +harness_selftest 410-a-check-must-have-been-red a census that understates the ledger is refused never - harness_selftest 410-a-check-must-have-been-red a check in an UNCOVERED suite is not refused never - harness_selftest 410-a-check-must-have-been-red a check merely added is not reported as a rename never - harness_selftest 410-a-check-must-have-been-red a check observed red gains the date it was seen never - @@ -675,19 +679,23 @@ harness_selftest 410-a-check-must-have-been-red and one that stayed green keeps harness_selftest 410-a-check-must-have-been-red and only when there is a base, so a push build does not fail on it never - harness_selftest 410-a-check-must-have-been-red and records neither as ever having been red never - harness_selftest 410-a-check-must-have-been-red and the addition in the other part is not called a rename never - +harness_selftest 410-a-check-must-have-been-red and the census line it always printed is still there never - harness_selftest 410-a-check-must-have-been-red and the comparison passes when the ceiling did not rise never - +harness_selftest 410-a-check-must-have-been-red and the gate prints that it compared them never - harness_selftest 410-a-check-must-have-been-red and the history it is about to lose travels with it never - harness_selftest 410-a-check-must-have-been-red and the message says how to fix it, because regenerating is the intended action never - harness_selftest 410-a-check-must-have-been-red and the refusal names both values never - harness_selftest 410-a-check-must-have-been-red and the refusal names how many failed, so the author can narrow the run never - harness_selftest 410-a-check-must-have-been-red and the refusal names the check that reddened never - harness_selftest 410-a-check-must-have-been-red and the refusal names the raise never - +harness_selftest 410-a-check-must-have-been-red and the refusal quotes the claim beside the measurement never - harness_selftest 410-a-check-must-have-been-red and the runner names no remote at that call site never - harness_selftest 410-a-check-must-have-been-red auto refuses when GITHUB_BASE_REF names a ref that is not here never - harness_selftest 410-a-check-must-have-been-red auto uses the base ref when it resolves, and names it never - harness_selftest 410-a-check-must-have-been-red auto with no base ref and no upstream is an integrity failure never - harness_selftest 410-a-check-must-have-been-red both failure arms fail the major never - harness_selftest 410-a-check-must-have-been-red but that suite is counted as not covered, which is the debt never - +harness_selftest 410-a-check-must-have-been-red but the gate says so, so the skip is visible rather than silent never - harness_selftest 410-a-check-must-have-been-red control: a well-formed log still merges never - harness_selftest 410-a-check-must-have-been-red control: an all-PASS log still merges with no flag at all never - harness_selftest 410-a-check-must-have-been-red control: the same log merges with a different reason, so the refusal above is --mutation-across-two-checks and not the log never - @@ -701,9 +709,11 @@ harness_selftest 410-a-check-must-have-been-red once the suite is covered, a new harness_selftest 410-a-check-must-have-been-red one --mutation cannot be attributed across several runs at once never - harness_selftest 410-a-check-must-have-been-red premise: and it is a file git has never seen, which is the case that used to fail open never - harness_selftest 410-a-check-must-have-been-red premise: and the nobudget branch does not, which is the bootstrap shape never - +harness_selftest 410-a-check-must-have-been-red premise: both are never, so this ledger's census is two never - harness_selftest 410-a-check-must-have-been-red premise: the budget is a tracked file too never - harness_selftest 410-a-check-must-have-been-red premise: the budget was restored byte-exact never - harness_selftest 410-a-check-must-have-been-red premise: the check has history before the rename never - +harness_selftest 410-a-check-must-have-been-red premise: the fixture ledger holds two rows never - harness_selftest 410-a-check-must-have-been-red premise: the fixture log names checks the real ledger already knows never - harness_selftest 410-a-check-must-have-been-red premise: the gate's status block was found in the runner never - harness_selftest 410-a-check-must-have-been-red premise: the hasbudget branch carries the budget never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 913055e2..a2175691 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 819 +checks_never_observed_red 829 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 1b6b8648..a717c354 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -561,6 +561,41 @@ def cmd_gate(args): print(f" ledger census: rows={len(rows)} | never observed red={never}, " f"ever red={len(rows) - never}, new this run={len(unknown)}") + # THE CENSUS IS COMPARED, NOT ONLY PRINTED (#952). Reporting is not enforcing: + # the line above stated the true number while the budget claimed another, and + # rc was 0 on a fifteen-row lie. + # + # DECIDABLE FROM THE TWO INPUTS ALONE. It needs no prior and no `--against`, + # which is the whole point: the case it catches is a MERGE COMMIT, where two + # PRs each re-derived the census from the same base, the ledger then takes both + # sets of rows, and the budget keeps whichever side won the conflict. A check + # that needed the prior could not speak about the commit that creates the + # disagreement. + # + # STILL NOT A CEILING, and this does not make it one. A ceiling refuses a RISE, + # and bounding this number deadlocks -- every added check enters as `never`, so + # landing one would require raising a number the design says may only fall. + # That argument is in check_ledger_budget.txt and nothing here changes it. What + # is refused is a CONTRADICTION, in either direction, which is what "a + # measurement that must be true" means. + stated = budget.get("checks_never_observed_red") + if stated is None: + # Absence is not a contradiction, and a silent skip is not acceptable + # either, so it is said out loud. Measured reason for not refusing: every + # other gate fixture in both harnesses writes a budget stating only + # suites_not_covered. What holds the COMMITTED budget to naming both is a + # separate arm in each harness. + print(" the budget names no checks_never_observed_red, so nothing asserts " + "the census") + elif stated != never: + print(f" the budget states checks_never_observed_red {stated}, the ledger " + f"holds {never}: these describe the same file and disagree") + print(" re-derive it from a run on THIS tree. Arithmetic across merges " + "has been right by accident and is not evidence.") + rc = 1 + else: + print(f" census stated {stated}, ledger holds {never}: they agree") + if not args.registered: raise LedgerError( "--registered is required: without the registered suite list the coverage " diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index d555a60e..3a9117c6 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -2159,6 +2159,29 @@ references in `.github/`, zero in the runner. If they disagree, one was edited by hand. `suites_not_covered` is 250 of 251, so the gate cannot refuse a new check in 250 suites — a real limit, counted rather than hidden, which falls as suites are seeded. + +### `test_the_gate_refuses_a_census_that_contradicts_its_own_ledger` + +The arm above asserts the two committed files agree. This one asserts the **tool +refuses a pair that does not** — because the gate printed `ledger census: rows=N` +and never compared it to the budget's claim, returning 0 on a fifteen-row lie +(#952). Reporting is not enforcing. + +It is decidable from the two inputs alone, with no prior and no `--against`, and +that is the point rather than an economy. The disagreement is created by a **merge**: +two PRs each re-derive the census from the same base, the ledger then takes both sets +of rows, and the budget keeps whichever side won the conflict. A check that needed the +prior could not speak about the commit that creates the problem. Three PRs in flight +at once set 769, 762 and 800 from a base of 756, and no two of them composed. + +Refused in **both** directions, which is what separates it from a ceiling: a ceiling +refuses a rise, and bounding this number deadlocks, as `check_ledger_budget.txt` +argues. Absence of the field is reported rather than refused, because absence is not a +contradiction — and because every other gate fixture in both harnesses states only +`suites_not_covered`, so refusing there would redden about twenty arms testing +something else. What holds the committed budget to naming both numbers is the arm +above. + ## 24. test_loop_coverage_premise.py: a loop that never ran asserted nothing **Why this file exists.** `assert-inside-a-loop-over-zero-rows` in VACUITY_MODES.md 3.5 diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index ae6a90b6..0248af4b 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -308,6 +308,57 @@ def test_the_committed_ledger_and_budget_agree(expect): "and the ceiling matches the suites with no rows") +def test_the_gate_refuses_a_census_that_contradicts_its_own_ledger(tmp_path, expect): + """#952. The gate PRINTED the census and never compared it, so rc=0 on a lie. + + Reporting is not enforcing. The case this catches is a MERGE, not a PR: two PRs + each rewrote the census from the same base, so the composed tree keeps whichever + side won the conflict while the ledger takes both sets of rows. Decidable from the + two inputs alone -- no prior, no `--against` -- which is exactly why it still + refuses on a merge commit, where the prior is the thing in question. + + Refused in BOTH directions. The census is not a ceiling and this does not make it + one: a ceiling refuses a rise, and bounding this number deadlocks, as the budget + file argues. What is refused here is a contradiction. + """ + ledger = _w(tmp_path, "l.tsv", "") + reg = _w(tmp_path, "reg", "demo\n") + log = _w(tmp_path, "g.log", GREEN) + _run("merge", "--ledger", ledger, "--date", "2026-09-10", log) + + # THE PREMISE, asserted: the numbers below mean nothing unless the ledger really + # holds two rows and both are `never`. + rows = _rows(ledger) + expect.num(len(rows), 2, "premise: the merged ledger holds two rows") + expect.num(len([r for r in rows if r[3] == "never"]), 2, + "premise: both entered as never, so this ledger's census is 2") + + ok = _w(tmp_path, "ok.txt", "suites_not_covered 0\nchecks_never_observed_red 2\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", ok, "--registered", reg, log) + expect.num(rc, 0, "a census that matches the ledger passes") + expect.num(out.count("census stated 2, ledger holds 2"), 1, + "and the agreement is printed, so a reader sees it was compared") + + for claim, why in (("1", "understates"), ("3", "overstates")): + b = _w(tmp_path, "b%s.txt" % claim, + "suites_not_covered 0\nchecks_never_observed_red %s\n" % claim) + out, rc = _run("gate", "--ledger", ledger, "--budget", b, "--registered", reg, log) + expect.num(rc, 1, "a census that %s the ledger is refused" % why) + expect.num(out.count("checks_never_observed_red %s, the ledger holds 2" % claim), 1, + "and the refusal quotes the claim and the measurement (%s)" % why) + + # ABSENCE IS NOT A MISMATCH, and that is measured rather than preferred: every + # other gate fixture here and in selftest 410 writes a budget stating only + # `suites_not_covered`, so refusing on absence would redden about twenty arms + # testing something else. What stops the COMMITTED budget dropping the field is + # the separate arm asserting it names both numbers, in both harnesses. + none = _w(tmp_path, "none.txt", "suites_not_covered 0\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", none, "--registered", reg, log) + expect.num(rc, 0, "a budget stating no census is not refused") + expect.num(out.count("names no checks_never_observed_red"), 1, + "but the gate says so, so the skip is visible rather than silent") + + def test_a_log_that_does_not_parse_is_not_evidence(tmp_path, expect): """`len(f) >= 5` accepted four shapes the emitter cannot produce. diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index e3427a5f..f537f724 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -328,6 +328,59 @@ check "the committed census matches the committed ledger" \ check "the budget names a ceiling and a census, and says which is which" \ "$(grep -cE '^(suites_not_covered|checks_never_observed_red) [0-9]+$' "$_budget")" "2" + +# ---- #952: the census is printed AND compared ------------------------------- +# +# `gate` printed `ledger census: rows=N` and never compared it to the budget's +# `checks_never_observed_red`. rc=0 on a twenty-row ledger claiming 5 was measured. +# Reporting is not enforcing. +# +# The case it catches is a MERGE, not a PR: two PRs each rewrite the census from the +# same base, so the composed tree keeps whichever side won the conflict while the +# ledger takes both sets of rows. So the comparison must need NO prior, and nothing +# in this block passes --against. +_cw="$PGC_WORKDIR/census952"; mkdir -p "$_cw" +printf 'demo\n' > "$_cw/reg" +: > "$_cw/led" +_led_run merge --ledger "$_cw/led" --date 2026-09-10 "$_lw/green.log" >/dev/null +check "premise: the fixture ledger holds two rows" \ + "$(grep -c . "$_cw/led" || true)" "2" +check "premise: both are never, so this ledger's census is two" \ + "$(awk -F'\t' '$4=="never"' "$_cw/led" | grep -c . || true)" "2" + +printf 'suites_not_covered 0\nchecks_never_observed_red 2\n' > "$_cw/ok.txt" +check "a census that matches the ledger passes" \ + "$(_led_rc gate --ledger "$_cw/led" --budget "$_cw/ok.txt" --registered "$_cw/reg" "$_lw/green.log")" "0" +check "and the gate prints that it compared them" \ + "$(_led_run gate --ledger "$_cw/led" --budget "$_cw/ok.txt" --registered "$_cw/reg" "$_lw/green.log" \ + | grep -c 'census stated 2, ledger holds 2')" "1" +check "and the census line it always printed is still there" \ + "$(_led_run gate --ledger "$_cw/led" --budget "$_cw/ok.txt" --registered "$_cw/reg" "$_lw/green.log" \ + | grep -c 'ledger census: rows=2')" "1" + +printf 'suites_not_covered 0\nchecks_never_observed_red 1\n' > "$_cw/low.txt" +check "a census that understates the ledger is refused" \ + "$(_led_rc gate --ledger "$_cw/led" --budget "$_cw/low.txt" --registered "$_cw/reg" "$_lw/green.log")" "1" +check "and the refusal quotes the claim beside the measurement" \ + "$(_led_run gate --ledger "$_cw/led" --budget "$_cw/low.txt" --registered "$_cw/reg" "$_lw/green.log" \ + | grep -c 'checks_never_observed_red 1, the ledger holds 2')" "1" + +# NOT A CEILING: a ceiling refuses a rise, and bounding this number deadlocks. What +# is refused is a contradiction, so overstating is refused too. +printf 'suites_not_covered 0\nchecks_never_observed_red 3\n' > "$_cw/high.txt" +check "a census that overstates the ledger is refused too, because this is not a ceiling" \ + "$(_led_rc gate --ledger "$_cw/led" --budget "$_cw/high.txt" --registered "$_cw/reg" "$_lw/green.log")" "1" + +# Absence is not a contradiction. Every other gate fixture in this part states only +# suites_not_covered, so refusing here would redden arms testing other things; the +# committed budget is held to naming both by the arm above. +printf 'suites_not_covered 0\n' > "$_cw/absent.txt" +check "a budget stating no census is not refused, because absence is not a contradiction" \ + "$(_led_rc gate --ledger "$_cw/led" --budget "$_cw/absent.txt" --registered "$_cw/reg" "$_lw/green.log")" "0" +check "but the gate says so, so the skip is visible rather than silent" \ + "$(_led_run gate --ledger "$_cw/led" --budget "$_cw/absent.txt" --registered "$_cw/reg" "$_lw/green.log" \ + | grep -c 'names no checks_never_observed_red')" "1" + # ---- the gate cannot refuse a check in a suite it has never seen ------------- # # The suite restriction is the MEANING of suites_not_covered, not a softening of