diff --git a/CHANGELOG.md b/CHANGELOG.md index 327e0219..146321a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1259,6 +1259,29 @@ true until the next version shipped. now re-records. It names the two cases that remain: a declaration that no longer resolves, and the implicit base projection, which is not readable by name at all. +- `pgc_ledger.py gate` no longer certifies a census that contradicts its own + ledger (#952). + + The gate printed `ledger census: rows=N` and never compared that number to the + `checks_never_observed_red` the budget states, so it returned 0 on a twenty-row + ledger claiming five. Reporting is not enforcing. The comparison existed one layer + out, in a selftest arm, which runs on a pull request and therefore reports the + disagreement after the merge that creates it rather than before. + + It creates it because the census is a measurement of the tree, so every merge + invalidates it: two pull requests each re-derive it from the same base, the merged + ledger takes both sets of rows, and the budget keeps whichever side won the + conflict. Three in flight at once set 769, 762 and 800 against a base of 756, and + no two composed. The new refusal is decidable from the two inputs alone, needing no + prior and no `--against`, which is what lets it speak about a merge commit. + + It refuses a contradiction in either direction and does not make the census a + ceiling. A ceiling refuses a rise, and bounding this number deadlocks: every added + check enters as `never`, so landing one would mean raising a number the design says + may only fall. A budget that states no census at all is reported rather than + refused, since absence is not a contradiction; what holds the committed budget to + naming both numbers is a separate arm in each harness. + ## [1.0-alpha3] - 2026-09-02 ### Added diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 26563e13..165fa5e0 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -619,7 +619,11 @@ harness_selftest 400-a-check-result-must-be-machine the same timing check, ENABL harness_selftest 400-a-check-result-must-be-machine the sweep finds a check inside a PIPED loop never - harness_selftest 410-a-check-must-have-been-red --mutation across two failing checks in one run is refused never - harness_selftest 410-a-check-must-have-been-red a before-log and an after-log together are refused, not silently empty never - +harness_selftest 410-a-check-must-have-been-red a budget stating no census is not refused, because absence is not a contradiction never - harness_selftest 410-a-check-must-have-been-red a budget that does not exist at the prior is a note, not a refusal never - +harness_selftest 410-a-check-must-have-been-red a census that matches the ledger passes never - +harness_selftest 410-a-check-must-have-been-red a census that overstates the ledger is refused too, because this is not a ceiling never - +harness_selftest 410-a-check-must-have-been-red a census that understates the ledger is refused never - harness_selftest 410-a-check-must-have-been-red a check in an UNCOVERED suite is not refused never - harness_selftest 410-a-check-must-have-been-red a check merely added is not reported as a rename never - harness_selftest 410-a-check-must-have-been-red a check observed red gains the date it was seen never - @@ -675,19 +679,23 @@ harness_selftest 410-a-check-must-have-been-red and one that stayed green keeps harness_selftest 410-a-check-must-have-been-red and only when there is a base, so a push build does not fail on it never - harness_selftest 410-a-check-must-have-been-red and records neither as ever having been red never - harness_selftest 410-a-check-must-have-been-red and the addition in the other part is not called a rename never - +harness_selftest 410-a-check-must-have-been-red and the census line it always printed is still there never - harness_selftest 410-a-check-must-have-been-red and the comparison passes when the ceiling did not rise never - +harness_selftest 410-a-check-must-have-been-red and the gate prints that it compared them never - harness_selftest 410-a-check-must-have-been-red and the history it is about to lose travels with it never - harness_selftest 410-a-check-must-have-been-red and the message says how to fix it, because regenerating is the intended action never - harness_selftest 410-a-check-must-have-been-red and the refusal names both values never - harness_selftest 410-a-check-must-have-been-red and the refusal names how many failed, so the author can narrow the run never - harness_selftest 410-a-check-must-have-been-red and the refusal names the check that reddened never - harness_selftest 410-a-check-must-have-been-red and the refusal names the raise never - +harness_selftest 410-a-check-must-have-been-red and the refusal quotes the claim beside the measurement never - harness_selftest 410-a-check-must-have-been-red and the runner names no remote at that call site never - harness_selftest 410-a-check-must-have-been-red auto refuses when GITHUB_BASE_REF names a ref that is not here never - harness_selftest 410-a-check-must-have-been-red auto uses the base ref when it resolves, and names it never - harness_selftest 410-a-check-must-have-been-red auto with no base ref and no upstream is an integrity failure never - harness_selftest 410-a-check-must-have-been-red both failure arms fail the major never - harness_selftest 410-a-check-must-have-been-red but that suite is counted as not covered, which is the debt never - +harness_selftest 410-a-check-must-have-been-red but the gate says so, so the skip is visible rather than silent never - harness_selftest 410-a-check-must-have-been-red control: a well-formed log still merges never - harness_selftest 410-a-check-must-have-been-red control: an all-PASS log still merges with no flag at all never - harness_selftest 410-a-check-must-have-been-red control: the same log merges with a different reason, so the refusal above is --mutation-across-two-checks and not the log never - @@ -701,9 +709,11 @@ harness_selftest 410-a-check-must-have-been-red once the suite is covered, a new harness_selftest 410-a-check-must-have-been-red one --mutation cannot be attributed across several runs at once never - harness_selftest 410-a-check-must-have-been-red premise: and it is a file git has never seen, which is the case that used to fail open never - harness_selftest 410-a-check-must-have-been-red premise: and the nobudget branch does not, which is the bootstrap shape never - +harness_selftest 410-a-check-must-have-been-red premise: both are never, so this ledger's census is two never - harness_selftest 410-a-check-must-have-been-red premise: the budget is a tracked file too never - harness_selftest 410-a-check-must-have-been-red premise: the budget was restored byte-exact never - harness_selftest 410-a-check-must-have-been-red premise: the check has history before the rename never - +harness_selftest 410-a-check-must-have-been-red premise: the fixture ledger holds two rows never - harness_selftest 410-a-check-must-have-been-red premise: the fixture log names checks the real ledger already knows never - harness_selftest 410-a-check-must-have-been-red premise: the gate's status block was found in the runner never - harness_selftest 410-a-check-must-have-been-red premise: the hasbudget branch carries the budget never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 913055e2..a2175691 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 819 +checks_never_observed_red 829 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 1b6b8648..a717c354 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -561,6 +561,41 @@ def cmd_gate(args): print(f" ledger census: rows={len(rows)} | never observed red={never}, " f"ever red={len(rows) - never}, new this run={len(unknown)}") + # THE CENSUS IS COMPARED, NOT ONLY PRINTED (#952). Reporting is not enforcing: + # the line above stated the true number while the budget claimed another, and + # rc was 0 on a fifteen-row lie. + # + # DECIDABLE FROM THE TWO INPUTS ALONE. It needs no prior and no `--against`, + # which is the whole point: the case it catches is a MERGE COMMIT, where two + # PRs each re-derived the census from the same base, the ledger then takes both + # sets of rows, and the budget keeps whichever side won the conflict. A check + # that needed the prior could not speak about the commit that creates the + # disagreement. + # + # STILL NOT A CEILING, and this does not make it one. A ceiling refuses a RISE, + # and bounding this number deadlocks -- every added check enters as `never`, so + # landing one would require raising a number the design says may only fall. + # That argument is in check_ledger_budget.txt and nothing here changes it. What + # is refused is a CONTRADICTION, in either direction, which is what "a + # measurement that must be true" means. + stated = budget.get("checks_never_observed_red") + if stated is None: + # Absence is not a contradiction, and a silent skip is not acceptable + # either, so it is said out loud. Measured reason for not refusing: every + # other gate fixture in both harnesses writes a budget stating only + # suites_not_covered. What holds the COMMITTED budget to naming both is a + # separate arm in each harness. + print(" the budget names no checks_never_observed_red, so nothing asserts " + "the census") + elif stated != never: + print(f" the budget states checks_never_observed_red {stated}, the ledger " + f"holds {never}: these describe the same file and disagree") + print(" re-derive it from a run on THIS tree. Arithmetic across merges " + "has been right by accident and is not evidence.") + rc = 1 + else: + print(f" census stated {stated}, ledger holds {never}: they agree") + if not args.registered: raise LedgerError( "--registered is required: without the registered suite list the coverage " diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index d555a60e..3a9117c6 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -2159,6 +2159,29 @@ references in `.github/`, zero in the runner. If they disagree, one was edited by hand. `suites_not_covered` is 250 of 251, so the gate cannot refuse a new check in 250 suites — a real limit, counted rather than hidden, which falls as suites are seeded. + +### `test_the_gate_refuses_a_census_that_contradicts_its_own_ledger` + +The arm above asserts the two committed files agree. This one asserts the **tool +refuses a pair that does not** — because the gate printed `ledger census: rows=N` +and never compared it to the budget's claim, returning 0 on a fifteen-row lie +(#952). Reporting is not enforcing. + +It is decidable from the two inputs alone, with no prior and no `--against`, and +that is the point rather than an economy. The disagreement is created by a **merge**: +two PRs each re-derive the census from the same base, the ledger then takes both sets +of rows, and the budget keeps whichever side won the conflict. A check that needed the +prior could not speak about the commit that creates the problem. Three PRs in flight +at once set 769, 762 and 800 from a base of 756, and no two of them composed. + +Refused in **both** directions, which is what separates it from a ceiling: a ceiling +refuses a rise, and bounding this number deadlocks, as `check_ledger_budget.txt` +argues. Absence of the field is reported rather than refused, because absence is not a +contradiction — and because every other gate fixture in both harnesses states only +`suites_not_covered`, so refusing there would redden about twenty arms testing +something else. What holds the committed budget to naming both numbers is the arm +above. + ## 24. test_loop_coverage_premise.py: a loop that never ran asserted nothing **Why this file exists.** `assert-inside-a-loop-over-zero-rows` in VACUITY_MODES.md 3.5 diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index ae6a90b6..0248af4b 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -308,6 +308,57 @@ def test_the_committed_ledger_and_budget_agree(expect): "and the ceiling matches the suites with no rows") +def test_the_gate_refuses_a_census_that_contradicts_its_own_ledger(tmp_path, expect): + """#952. The gate PRINTED the census and never compared it, so rc=0 on a lie. + + Reporting is not enforcing. The case this catches is a MERGE, not a PR: two PRs + each rewrote the census from the same base, so the composed tree keeps whichever + side won the conflict while the ledger takes both sets of rows. Decidable from the + two inputs alone -- no prior, no `--against` -- which is exactly why it still + refuses on a merge commit, where the prior is the thing in question. + + Refused in BOTH directions. The census is not a ceiling and this does not make it + one: a ceiling refuses a rise, and bounding this number deadlocks, as the budget + file argues. What is refused here is a contradiction. + """ + ledger = _w(tmp_path, "l.tsv", "") + reg = _w(tmp_path, "reg", "demo\n") + log = _w(tmp_path, "g.log", GREEN) + _run("merge", "--ledger", ledger, "--date", "2026-09-10", log) + + # THE PREMISE, asserted: the numbers below mean nothing unless the ledger really + # holds two rows and both are `never`. + rows = _rows(ledger) + expect.num(len(rows), 2, "premise: the merged ledger holds two rows") + expect.num(len([r for r in rows if r[3] == "never"]), 2, + "premise: both entered as never, so this ledger's census is 2") + + ok = _w(tmp_path, "ok.txt", "suites_not_covered 0\nchecks_never_observed_red 2\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", ok, "--registered", reg, log) + expect.num(rc, 0, "a census that matches the ledger passes") + expect.num(out.count("census stated 2, ledger holds 2"), 1, + "and the agreement is printed, so a reader sees it was compared") + + for claim, why in (("1", "understates"), ("3", "overstates")): + b = _w(tmp_path, "b%s.txt" % claim, + "suites_not_covered 0\nchecks_never_observed_red %s\n" % claim) + out, rc = _run("gate", "--ledger", ledger, "--budget", b, "--registered", reg, log) + expect.num(rc, 1, "a census that %s the ledger is refused" % why) + expect.num(out.count("checks_never_observed_red %s, the ledger holds 2" % claim), 1, + "and the refusal quotes the claim and the measurement (%s)" % why) + + # ABSENCE IS NOT A MISMATCH, and that is measured rather than preferred: every + # other gate fixture here and in selftest 410 writes a budget stating only + # `suites_not_covered`, so refusing on absence would redden about twenty arms + # testing something else. What stops the COMMITTED budget dropping the field is + # the separate arm asserting it names both numbers, in both harnesses. + none = _w(tmp_path, "none.txt", "suites_not_covered 0\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", none, "--registered", reg, log) + expect.num(rc, 0, "a budget stating no census is not refused") + expect.num(out.count("names no checks_never_observed_red"), 1, + "but the gate says so, so the skip is visible rather than silent") + + def test_a_log_that_does_not_parse_is_not_evidence(tmp_path, expect): """`len(f) >= 5` accepted four shapes the emitter cannot produce. diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index e3427a5f..f537f724 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -328,6 +328,59 @@ check "the committed census matches the committed ledger" \ check "the budget names a ceiling and a census, and says which is which" \ "$(grep -cE '^(suites_not_covered|checks_never_observed_red) [0-9]+$' "$_budget")" "2" + +# ---- #952: the census is printed AND compared ------------------------------- +# +# `gate` printed `ledger census: rows=N` and never compared it to the budget's +# `checks_never_observed_red`. rc=0 on a twenty-row ledger claiming 5 was measured. +# Reporting is not enforcing. +# +# The case it catches is a MERGE, not a PR: two PRs each rewrite the census from the +# same base, so the composed tree keeps whichever side won the conflict while the +# ledger takes both sets of rows. So the comparison must need NO prior, and nothing +# in this block passes --against. +_cw="$PGC_WORKDIR/census952"; mkdir -p "$_cw" +printf 'demo\n' > "$_cw/reg" +: > "$_cw/led" +_led_run merge --ledger "$_cw/led" --date 2026-09-10 "$_lw/green.log" >/dev/null +check "premise: the fixture ledger holds two rows" \ + "$(grep -c . "$_cw/led" || true)" "2" +check "premise: both are never, so this ledger's census is two" \ + "$(awk -F'\t' '$4=="never"' "$_cw/led" | grep -c . || true)" "2" + +printf 'suites_not_covered 0\nchecks_never_observed_red 2\n' > "$_cw/ok.txt" +check "a census that matches the ledger passes" \ + "$(_led_rc gate --ledger "$_cw/led" --budget "$_cw/ok.txt" --registered "$_cw/reg" "$_lw/green.log")" "0" +check "and the gate prints that it compared them" \ + "$(_led_run gate --ledger "$_cw/led" --budget "$_cw/ok.txt" --registered "$_cw/reg" "$_lw/green.log" \ + | grep -c 'census stated 2, ledger holds 2')" "1" +check "and the census line it always printed is still there" \ + "$(_led_run gate --ledger "$_cw/led" --budget "$_cw/ok.txt" --registered "$_cw/reg" "$_lw/green.log" \ + | grep -c 'ledger census: rows=2')" "1" + +printf 'suites_not_covered 0\nchecks_never_observed_red 1\n' > "$_cw/low.txt" +check "a census that understates the ledger is refused" \ + "$(_led_rc gate --ledger "$_cw/led" --budget "$_cw/low.txt" --registered "$_cw/reg" "$_lw/green.log")" "1" +check "and the refusal quotes the claim beside the measurement" \ + "$(_led_run gate --ledger "$_cw/led" --budget "$_cw/low.txt" --registered "$_cw/reg" "$_lw/green.log" \ + | grep -c 'checks_never_observed_red 1, the ledger holds 2')" "1" + +# NOT A CEILING: a ceiling refuses a rise, and bounding this number deadlocks. What +# is refused is a contradiction, so overstating is refused too. +printf 'suites_not_covered 0\nchecks_never_observed_red 3\n' > "$_cw/high.txt" +check "a census that overstates the ledger is refused too, because this is not a ceiling" \ + "$(_led_rc gate --ledger "$_cw/led" --budget "$_cw/high.txt" --registered "$_cw/reg" "$_lw/green.log")" "1" + +# Absence is not a contradiction. Every other gate fixture in this part states only +# suites_not_covered, so refusing here would redden arms testing other things; the +# committed budget is held to naming both by the arm above. +printf 'suites_not_covered 0\n' > "$_cw/absent.txt" +check "a budget stating no census is not refused, because absence is not a contradiction" \ + "$(_led_rc gate --ledger "$_cw/led" --budget "$_cw/absent.txt" --registered "$_cw/reg" "$_lw/green.log")" "0" +check "but the gate says so, so the skip is visible rather than silent" \ + "$(_led_run gate --ledger "$_cw/led" --budget "$_cw/absent.txt" --registered "$_cw/reg" "$_lw/green.log" \ + | grep -c 'names no checks_never_observed_red')" "1" + # ---- the gate cannot refuse a check in a suite it has never seen ------------- # # The suite restriction is the MEANING of suites_not_covered, not a softening of