From 4c0cb50035f06ed4871f66716a9d355abae13a99 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Sat, 12 Sep 2026 18:50:18 -0600 Subject: [PATCH] test: refuse two checks sharing one ledger key, and two rows sharing one (#982) A ledger row is keyed on (suite, part, name). Two checks with the same name in one part share a row, and two ROWS with the same key collapse into one. Neither is mis-recorded while everything passes. Both lose a red observation the moment one appears. --- THE GATE COULD NOT SEE TWO CHECKS IN ONE RUN ------------------------------- `merge` has printed "duplicate check name in one run" since #982 was filed and returns 0, which is how three of them sat in one part of selftest/400 for a day. The instance was fixed by c3b13aed; this is the mechanism the issue called the more valuable half. `cmd_gate` builds its records as `sorted({(s, p, n, m) for ...})`. A set collapses the duplicate before any arm can count it, so the same canonicalisation that makes the rest of that function correct made this one class unreachable. The count now comes from the raw records through `_by_run`, which already existed for the two-runs-versus-one-duplicate distinction. EVERY SUITE, DELIBERATELY UNLIKE THE NEW-CHECK REFUSAL. That one is restricted to covered suites because it cannot know which of an uncovered suite's checks are new. This one needs no history: two records, one key, one log is decidable from the log alone. The ledger covers four suites of 253, so copying the restriction would close the class in four places only, and the next collision would sit in one of the other 249 until that suite was seeded. --- AND THE LEDGER COULD NOT SEE TWO ROWS IN ONE FILE -------------------------- `read_ledger` did `rows[key] = [...]`, so a duplicated key in the tracked file collapsed silently and the LAST line won. Measured on a two-line fixture, both orders: never first, then 2026-09-01 survivor last_red='2026-09-01' 2026-09-01 first, then never survivor last_red='never' the red is GONE Line order decided whether a recorded red observation survived. A merge that keeps both sides of a changed row turns `ever red` back into `never`. NOTHING ELSE COULD CATCH IT, and bounding the census cannot. check_ledger_budget.txt says `checks_never_observed_red` is a CENSUS and must not become a ceiling, because every new check enters as `never` and bounding it deadlocks. The gate compares the budget's number with the ledger's, and both come from the same dict, so they agree either way. Measured: with the budget regenerated alongside, an erased red passes the gate at rc=0. It is now refused as an integrity failure (rc=2), beside the other inputs that do not parse. So the two halves are the same shape at two levels. A SET hid two checks in one run; a DICT hid two rows in one file. The question that found both is what the input canonicalises before the guard sees it. --- MEASURED BEFORE WIDENING IT ----------------------------------------------- A gate that reddens 250 unmeasured suites is a gate somebody turns off, so the refusal was not widened on reasoning. A full PG 18 matrix ran with it armed for every suite: 247 suites ran, 6 skipped, 0 incomplete | ALL VERSIONS PASSED, RC=0 | 0 shared keys Check names are static, so one major's matrix measures this class completely rather than sampling it. THE LOAD-BEARING EVIDENCE IS THAT RUN, not a log snapshot. An earlier draft of this message cited "244 logs, 6651 records, 6651 distinct keys, 0 collisions" from per-suite logs copied out of the build directory while the matrix was still running. That figure is withdrawn: the copies were taken mid-write, and measuring the same way again showed 150 of 253 had no `checks run:` line yet. Zero collisions across truncated logs is not evidence of zero across complete ones. The full-matrix figure stands because the gate reads the logs after each suite has finished. It does still answer what #982 said nobody had measured -- the gap between "checks that ran" and "rows the ledger can hold" across the other 250 suites -- because the gate saw every suite's complete log and refused nothing. --- VERIFICATION -------------------------------------------------------------- The committed ledger still loads at 1197 rows. The gate against the COMMITTED ledger and budget over a real 934-record harness_selftest log: rc=0, no shared-key line, census agrees at 1189. A planted duplicate check: rc=1, naming the key. A planted duplicate row: rc=2, naming the line and what it would lose. The refusal arm also asserts no Traceback in the output, because an unhandled exception exits 1 too and would satisfy an rc check on its own. Measured, it did: an earlier draft referenced `covered_suites` before it was defined, which compiles and fails at runtime. guard_tests 277 -> 283, re-derived by collection. 283 passed / 698 checks with --pgc-expect-tests 283 armed. docs_style.sh PASSED (11 checks). Long-sentence counts unchanged: CHANGELOG 860, TESTS.md 319. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- AND A THIRD INSTANCE, IN THE ARM THAT POLICES THE OTHER TRACKED FILE ------- test_harness_deps.py read expected_tests.txt with `nums[f[0]] = int(f[1])`, so a duplicated key collapsed and the last line won -- exactly as read_ledger did, one file over. Measured on one fixture read both ways: the line form names guard_tests as duplicated the dict form sees two keys and keeps 280, the LAST line NOT HYPOTHETICAL. Three PRs were open at once, each moving guard_tests, and resolving all three keep-both produced three of those lines. ci.yml reads the value with `awk '$1=="guard_tests"{print $2}'`, which prints one line per match. So WANT becomes multi-line, `test -n "$WANT"` still passes, and the flag refuses it: pytest: error: argument --pgc-expect-tests: invalid int value: '284\n280\n283' exit 4 It FAILS CLOSED, so these two arms are about legibility rather than a hole. What exit 4 does not say is that a line is duplicated. The removal proof is on the real file: with a duplicate planted, the new arm reddens and the pre-existing one stays green. Keep-both is right for a changelog and wrong for a key-value file, and nothing in the tree said so. cluster_tests 205 -> 207, NOT guard_tests: test_harness_deps.py DEFINES NO_CLUSTER and is not in it. I bumped the wrong number first and --pgc-expect-tests caught it -- `collected 283 test(s) but expected 285` -- which is the argument for the mechanism. Both halves re-derived by collection: guard 283, cluster 207. REBASED onto 14c9dd49 after #1029 merged, and the conflict was the exact shape the third arm here is about. `expected_tests.txt` conflicted on `guard_tests`: guard_tests 280 <- main, after #1029 guard_tests 283 <- this branch, from the old base Keep-both would have produced two lines and, through ci.yml's `awk '$1=="guard_tests"{print $2}'`, a multi-line value the flag refuses at exit 4. One line, and both halves re-derived by collection rather than by arithmetic: guard_tests 286, cluster_tests 207 Re-verified: 286 passed / 707 checks with --pgc-expect-tests 286 armed; docs_style.sh PASSED (11 checks); the gate against the committed ledger and budget over the real 934-record log still returns 0. Long-sentence counts unchanged against the new base: CHANGELOG 860, TESTS.md 319. --- FIXTURE CORRECTED AFTER REVIEW (@jdatcmd) ----------------------------------- test_the_gate_refuses_two_checks_sharing_one_ledger_key asserted rc == 1 on a fixture where the gate returned 1 for TWO independent reasons, so the arm did not discriminate. Found by mutation, and I reproduced it before changing anything: with `rc = 1` deleted from the shared-key block, the OLD fixture's arm PASSED with the same mutation, the corrected fixture's arm FAILS: got 0 want 1 The second route was the pre-existing new-check refusal. The ledger named only `some other check` in that part, which made `demo` a covered suite whose log carried a check the ledger had never seen: not in the ledger: demo part1 shared name (on major 18) The ledger now NAMES `shared name`, so that refusal has nothing to say and only the shared-key refusal can move rc. This is the same defect class the PR is about, in my own arm: an assertion that passes for a reason other than the one it is named for. My docstring claimed I had measured this and closed it -- I had closed the COVERAGE-CEILING route and not the new-check route, and the note said "routes" as though it were all of them. The comment now enumerates all three, including the unhandled-exception route that also exits 1. It also described the refusal as "restricted to suites a row exists for", which was the design I abandoned two pushes earlier when the matrix measurement said refusing everywhere was safe. That sentence is gone. --- REBASED onto 2ea73661 ------------------------------------------------------ guard_tests 293 and cluster_tests 207, both re-derived by COLLECTION on this base. And a sharper point than the one I had, measured by @jdatcmd on #1022: two branches each derived `guard_tests 280` correctly, each against a tree holding only its own three arms, and the merged value was 283. Both numbers were right for the tree they were taken on and both were wrong for the merge. So "derive rather than add" is necessary and not sufficient -- it has to be re-derived after every rebase, which expected_tests.txt now says. Re-verified: 293 passed / 725 checks with --pgc-expect-tests 293 armed; docs_style.sh PASSED (11 checks); the gate against the committed ledger and budget over the real 934-record log returns 0; long-sentence counts unchanged at CHANGELOG 860 and TESTS.md 319. REBASED again onto 1d76ccf4 after #1022 merged. Two hunks in expected_tests.txt, and the second one is worth a note: my side carried the new `cluster_tests 207` comment PLUS two stale fragments left by earlier keep-both resolutions of the same file ("177 -> 180", "166 -> 182"), describing values the file no longer holds. Resolved to ONE coherent block rather than the union, which is the same lesson the arms here are about: keep-both is right for a changelog and wrong for a file where each key appears once. guard_tests 296 and cluster_tests 207, both re-derived by collection on this base. Re-verified: 296 passed / 733 checks armed; docs_style.sh PASSED; the gate over the real 934-record log returns 0. --- CHANGELOG.md | 92 ++++++++++++++ test/pgc_ledger.py | 91 +++++++++++-- test/pytest/TESTS.md | 101 +++++++++++++++ test/pytest/expected_tests.txt | 19 ++- test/pytest/test_harness_deps.py | 77 +++++++++++ test/pytest/test_mutation_ledger.py | 191 ++++++++++++++++++++++++++++ 6 files changed, 560 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d6f5579e..2779b8c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,98 @@ true until the next version shipped. ### Added +- The gate refuses two checks that share one ledger key, instead of printing a note + about them (#982). + + AND THE SAME CLASS ONE LEVEL DOWN, found while building the arm for the first. `read_ledger` + did `rows[(f[0], f[1], f[2])] = [...]`, so a duplicated key in the TRACKED file collapsed + silently and the last line won. Measured on a two-line fixture, both orders: + + never first, then 2026-09-01 survivor last_red='2026-09-01' + 2026-09-01 first, then never survivor last_red='never' the red is GONE + + Line order decided whether a recorded red observation survived. A merge that keeps both + sides of a changed row turns `ever red` back into `never`. That is what #918 and #925 exist + to prevent, arriving from the opposite direction. + + NOTHING ELSE COULD CATCH IT, and bounding the census cannot. `check_ledger_budget.txt` says + `checks_never_observed_red` is a CENSUS and must not become a ceiling, because every new + check enters as `never` and bounding it deadlocks. The gate compares the budget's number + with the ledger's, and both come from the same dict, so they agree either way. Measured: + with the budget regenerated alongside, an erased red passes the gate at rc=0. + + It is refused as an INTEGRITY FAILURE (rc=2) rather than a gate verdict, beside the other + inputs that do not parse. A ledger that cannot be trusted is not a gate result. + + So the two halves are the same shape at two levels. A SET hid two checks in one run; a DICT + hid two rows in one file. The question that found both is what the input canonicalises + before the guard sees it. + + AND A THIRD, IN THE ARM THAT POLICES THE OTHER TRACKED KEY-VALUE FILE. + `test_harness_deps.py` read `expected_tests.txt` with `nums[f[0]] = int(f[1])`, so a + duplicated key collapsed and the last line won, exactly as `read_ledger` did. Measured on + one fixture read both ways: + + the line form names guard_tests as duplicated + the dict form sees two keys and keeps 280, the LAST line + + THIS IS NOT HYPOTHETICAL. Three PRs were open at once, each moving `guard_tests`, and + resolving all three keep-both produced three of those lines. `ci.yml` reads the value with + `awk '$1=="guard_tests"{print $2}'`, which prints one line per match. So `WANT` becomes + multi-line, `test -n "$WANT"` still passes, and the flag refuses it at exit 4: + + pytest: error: argument --pgc-expect-tests: invalid int value: '284\n280\n283' + + It FAILS CLOSED, so this is legibility rather than a hole. What exit 4 does not say is that a + line is duplicated. Two arms now say it. The removal proof on the real file reddens the new + arm and leaves the pre-existing one green, which is the point. + + Keep-both is right for a changelog and wrong for a key-value file. Nothing in the tree said + so. + + `cluster_tests` 205 -> 207, not `guard_tests`: `test_harness_deps.py` DEFINES `NO_CLUSTER` + and is not in it. I got that wrong first and the mechanism caught it, which is the argument + for the mechanism. + + A ledger row is keyed on `(suite, part, name)`, so two checks with the same name in one + part share a row. Nothing is mis-recorded while both pass. The hazard is exact: when one + goes red the row records `ever red`, and its namesake inherits a red observation nothing + attacked. `checks_never_observed_red` then falls by one for a check nobody attacked, and + that census is what #918 and #925 exist to make trustworthy. + + `merge` has detected this since #982 was filed, and returns 0. That is how three of them + sat in one part of `selftest/400` for a day. The instance was fixed by `c3b13aed`; this is + the mechanism, which that issue called the more valuable half. + + THE GATE COULD NOT SEE IT AT ALL, and the reason is worth recording. `cmd_gate` builds its + records as `sorted({(s, p, n, m) for ...})`. A set collapses the duplicate before any arm + can count it. So the same canonicalisation that makes the rest of the gate correct made + this one class unreachable. The count now comes from the raw records through `_by_run`. + + PER LOG, because one check observed in two logs is two RUNS of it. That is the normal case + and the way the ledger accumulates evidence at all. Only a repeat inside one log is a + collision. An arm pins the distinction, because written over the logs together the refusal + would reject every multi-day merge. + + EVERY SUITE, DELIBERATELY UNLIKE THE NEW-CHECK REFUSAL. That one is restricted to suites + the ledger covers because it cannot know which of an uncovered suite's checks are new. + This one needs no history: two records, one key, one log is decidable from the log alone. + The ledger covers four suites of 253. Copying the restriction would close the class in four + places only. The next collision would then sit in one of the other 249 until that suite was + seeded. + + MEASURED BEFORE WIDENING IT, because a gate that reddens 250 unmeasured suites is a gate + somebody turns off. A full PG 18 matrix run with the refusal armed for every suite: + + suites that ran 247 (6 skipped, 0 incomplete) + matrix verdict ALL VERSIONS PASSED, RC=0 + shared ledger keys found 0 + + Check names are static, so one major's matrix measures this class completely rather than + sampling it. A real `harness_selftest` log says the same end to end. Run against the + COMMITTED ledger and budget, the gate returns 0 with no shared-key line, over 934 records + and 934 distinct keys. + - `projection_privilege.sh` has a pytest twin, and both halves now attribute a refusal by SQLSTATE instead of by error text (#432, #562, #563). diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 0322cf12..25304ee4 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -311,7 +311,34 @@ def read_ledger(path): f"{path}:{n}: major {m!r} is neither a number nor " f"{MAJOR_UNKNOWN!r}, so this row names no version it applies to") muts = set() if f[5] == NONE else {m for m in f[5].split(";") if m} - rows[(f[0], f[1], f[2])] = [majors, f[4] or NEVER, muts] + key = (f[0], f[1], f[2]) + # A REPEATED KEY, REFUSED RATHER THAN OVERWRITTEN. This was + # `rows[key] = [...]`, so a duplicated row collapsed silently and the LAST line + # won. Measured on a two-line fixture, both orders: + # + # never first, then 2026-09-01 survivor last_red='2026-09-01' + # 2026-09-01 first, then never survivor last_red='never' <- the red is GONE + # + # So line order decided whether a recorded red observation survived, and a merge + # that keeps both sides of a changed row turns `ever red` back into `never` -- + # the corruption #918 and #925 exist to prevent. + # + # NOTHING ELSE CAN CATCH IT, and bounding the census cannot. The budget file says + # `checks_never_observed_red` is a CENSUS and must not become a ceiling, because + # every new check enters as `never` and bounding it deadlocks. The gate compares + # the budget's number with the ledger's, and both come from this dict, so they + # agree either way. Measured: with the budget regenerated alongside, an erased red + # passes the gate at rc=0. + # + # The same shape as the shared-key refusal in `cmd_gate`, one level down: there a + # set hid two checks in one RUN, here a dict hid two rows in one FILE. + if key in rows: + raise LedgerError( + f"{path}:{n}: a ledger row repeats a key already in this file: " + f"{f[0]}\t{f[1]}\t{f[2]}. One key is one check, and the later row would " + f"silently replace the earlier -- which loses a recorded red if the later " + f"row says {NEVER!r}. Keep one row per check.") + rows[key] = [majors, f[4] or NEVER, muts] return rows @@ -866,6 +893,60 @@ def cmd_gate(args): rc = 0 + covered_suites = {k[0] for k in rows} + # AND THE SAME ARGUMENT FOR THE MAJOR (#1010). The gate cannot refuse a new check on a + # major it holds no rows for, for the identical reason it cannot in a suite it has + # never seen: it has no idea which of that major's checks are new. Adding PG20 to the + # matrix would otherwise redden every check at once, which is a gate somebody turns + # off -- the failure this issue family exists to prevent. It tightens on its own the + # moment one run on that major is merged. + covered_majors = set().union(*(v[0] for v in rows.values())) if rows else set() + + # TWO CHECKS SHARING ONE LEDGER KEY, refused rather than noted (#982). + # + # A row is keyed on (suite, part, name), so two checks with the same name in one + # part share a row. Nothing is mis-recorded while both pass, and the hazard is + # exact: when one goes red the row records `ever red` and its namesake inherits a + # red observation nothing attacked. `checks_never_observed_red` then falls by one + # for a check nobody attacked, and that census is what #918 and #925 exist to make + # trustworthy. + # + # `merge` has printed this since #982 was filed and returns 0, which is how three + # of them sat in one part of selftest/400 for a day. THE GATE COULD NOT SEE IT AT + # ALL: `records` above is a set, and a set collapses the duplicate before any arm + # can count it. The same canonicalisation that makes the rest of this function + # correct made this one class unreachable, so the count comes from the raw records. + # + # PER LOG, via _by_run, because one check observed in two logs is two RUNS of it -- + # the normal case, and how the ledger accumulates evidence at all. Only a repeat + # inside one log is a collision. + # + # EVERY SUITE, DELIBERATELY UNLIKE THE NEW-CHECK REFUSAL BELOW. That one is + # restricted to suites the ledger covers because it cannot know which of an + # uncovered suite's checks are new. This one needs no history: two records, one + # key, one log is decidable from the log alone. Refusing everywhere is what makes + # the class impossible rather than impossible in the four suites seeded so far, and + # the next collision is likelier to arrive in one of the other 249. + # + # MEASURED BEFORE WIDENING IT, because a gate that reddens 250 unmeasured suites is + # a gate somebody turns off. A full PG 18 matrix run with this refusal armed for + # every suite: 247 suites ran, RC=0, ALL VERSIONS PASSED, zero shared keys. Check + # names are static, so one major's matrix is a complete measurement of this class + # rather than a sample of it. + shared = [] + for path, seen in _by_run(args.logs): + for (suite, part, name), verdicts in sorted(seen.items()): + if len(verdicts) > 1: + shared.append((path, suite, part, name, [v for v, _m in verdicts])) + for path, suite, part, name, verdicts in shared: + print(f" one ledger key covers {len(verdicts)} checks in {path}: " + f"{suite}\t{part}\t{name}\t({', '.join(verdicts)})") + if shared: + print(f" {len(shared)} ledger key(s) cover more than one check. Give each " + f"check a name that says which it is; a shared key records one check's " + f"red against the other.") + rc = 1 + # THE REFUSAL: a check the committed ledger has never seen, IN A SUITE THE # LEDGER COVERS. # @@ -878,14 +959,6 @@ def cmd_gate(args): # # It tightens on its own as suites are seeded, and the ceiling is what forces # that direction. - covered_suites = {k[0] for k in rows} - # AND THE SAME ARGUMENT FOR THE MAJOR (#1010). The gate cannot refuse a new check on a - # major it holds no rows for, for the identical reason it cannot in a suite it has - # never seen: it has no idea which of that major's checks are new. Adding PG20 to the - # matrix would otherwise redden every check at once, which is a gate somebody turns - # off -- the failure this issue family exists to prevent. It tightens on its own the - # moment one run on that major is merged. - covered_majors = set().union(*(v[0] for v in rows.values())) if rows else set() unknown = [] for key in records: if key[0] not in covered_suites or key[3] not in covered_majors: diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 2c88f5c1..93bf7fa4 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1599,6 +1599,8 @@ fixtures are read off `conftest.py` rather than named in the classifier. | `test_the_job_installs_no_database_driver` | the job asserts psycopg is absent rather than assuming it | | `test_the_cluster_job_runs_the_other_half_and_derives_it` | the complement of `NO_CLUSTER` is RUN, derived not listed, and asserts the driver IS present | | `test_both_pytest_jobs_assert_how_many_tests_they_collected` | both jobs pass `--pgc-expect-tests` from the tracked file, and each guards the read | +| `test_each_expected_count_is_stated_exactly_once` | `expected_tests.txt` states each key on one line — a keep-both merge duplicates it, `awk` then hands the flag a multi-line value, and the arm above could not see it because it builds a dict | +| `test_a_duplicated_count_is_caught_and_the_dict_form_is_not` | **removal proof**: the same fixture read both ways, so the dict form is shown keeping the LAST line while the line form names the duplicate | | `test_the_shell_reference_detector_sees_code_and_not_prose` | the premise: a docstring is prose, a string passed to bash is a reference, an f-string counts once | | `test_the_harness_independence_inventory_is_exactly_what_the_corpus_does` | CONTEXT.md's inventory, asserted in both directions | @@ -1612,6 +1614,23 @@ passed, so the numbers reach a reader from the run. A written count is a hand-maintained derived value, and the one in the job's comment was wrong the day it was written (#908). +**A keep-both merge of this file duplicates a key.** CI then says something unrecognisable. +`ci.yml` reads the value with `awk '$1=="guard_tests"{print $2}'`, which prints one line per +match. Two matches make `WANT` multi-line, `test -n "$WANT"` still passes, and the flag +refuses it: + + pytest: error: argument --pgc-expect-tests: invalid int value: '284\n280\n283' + exit 4 + +So it **fails closed**, which is why these arms are about legibility rather than a hole. What +exit 4 does not say is that a line is duplicated. The reader has to work back from an int +parse error to a merge resolution. + +**Not hypothetical.** Three PRs were open at once, each moving `guard_tests`, and resolving all +three keep-both produced exactly that. Keep-both is right for a changelog and wrong for a +key-value file, and nothing in the tree said so. The arm that read this file built +`nums[f[0]] = int(f[1])`, which is the same shape `read_ledger` had before #982 — one file over. + BUT THE ARMS IN THIS FILE DO NOT RUN IN THE GATE, and that is why `test/selftest/350-the-pytest-corpus-must-be.sh` runs the membership decision through this module's command line. The corpus is not in `SUITES` (README.md), and @@ -2364,6 +2383,88 @@ it the gate refuses every check of every uncovered suite and reddens the whole m on its first run. It tightens on its own as suites are seeded, and the deadlock that shipped is pinned as its own arm — regenerating the ledger lets a new check through. +### `test_the_ledger_refuses_two_rows_sharing_one_key` + +The same class one level down, found while building the arm below. `read_ledger` did +`rows[(f[0], f[1], f[2])] = [...]`, so a duplicated key in the **tracked file** collapsed +silently and the **last line won**. Measured on a two-line fixture, both orders: + + never first, then 2026-09-01 survivor last_red='2026-09-01' + 2026-09-01 first, then never survivor last_red='never' <- the red is GONE + +So line order decided whether a recorded red observation survived. A merge that keeps both +sides of a changed row turns `ever red` back into `never`. That is the corruption #918 and +#925 exist to prevent, arriving from the opposite direction to #982's. + +**Nothing else could catch it, and bounding the census cannot.** +`check_ledger_budget.txt` says `checks_never_observed_red` is a CENSUS and must not become a +ceiling. Every new check enters as `never`, so bounding it deadlocks. The gate compares the +budget's number with the ledger's, and both come from the same dict, so they agree either +way. Measured: with the budget regenerated alongside, an erased red passes the gate at +**rc=0**. + +It is refused as an **integrity failure** (rc=2) rather than a gate verdict, beside the other +inputs that do not parse. A ledger that cannot be trusted is not a gate result. + +### `test_a_ledger_with_no_duplicate_key_still_loads` + +The false-positive budget, and the premise the refusal needs. Two rows differing only in the +NAME are two checks and must load, which is the ordinary case for every part in the tree. The +arm also asserts `rows=3`, so the refusal cannot pass by eating a row. + +### `test_the_gate_refuses_two_checks_sharing_one_ledger_key` + +A row is keyed on `(suite, part, name)`, so two checks with the same name in one part +share a row. Nothing is mis-recorded while both pass. The hazard is exact: when one goes +red the row records `ever red` and its namesake inherits a red observation nothing +attacked. `checks_never_observed_red` then falls by one for a check nobody attacked, +which is the census #918 and #925 exist to make trustworthy. + +**`merge` already printed this and returned 0**, which is how three of them sat in one +part of `selftest/400` for a day (#982). The instance was fixed by `c3b13aed`; this is +the mechanism. + +**The gate could not see it at all, by construction.** `cmd_gate` builds its records as +`sorted({(s, p, n, m) for ...})`. A set collapses the duplicate before any arm can count +it. So the same canonicalisation that makes the rest of the gate correct made this one +class unreachable. The count now comes from the raw records, through `_by_run`. + +Measured on main at `03c6c9c8`. A real `harness_selftest` run emits **934 RESULT records +over 934 distinct keys, 0 collisions**. The gate run against the COMMITTED ledger and +budget returns 0 with no shared-key line. A full PG 18 matrix says the same for the whole +corpus: 247 suites, RC=0, zero shared keys. + +### `test_a_shared_key_is_refused_in_a_suite_the_ledger_does_not_cover` + +**The refusal covers every suite, unlike the new-check refusal.** That one is restricted +for a reason. It cannot know which of an uncovered suite's checks are new. This one needs +no history. Two records, one key, one log is decidable from the log alone. + +Why that matters rather than being a detail. The ledger covers **four suites of 253**. A +refusal restricted the same way would close the class in four places only. The next +collision would then arrive in one of the other 249 and sit there until that suite is +seeded. This arm is what stops the restriction being copied in by habit. + +**Measured before widening it**, because a gate that reddens 250 unmeasured suites is a gate +somebody turns off. A full PG 18 matrix ran with the refusal armed for every suite: + + 247 suites ran, RC=0, ALL VERSIONS PASSED, zero shared keys + +Check names are static, so one major's matrix measures this class completely rather than +sampling it. + +### `test_the_same_check_in_two_runs_is_not_a_shared_key` + +The distinction the refusal must not lose. One check observed on two days is two records +for one key and is the normal case. It is how the ledger accumulates evidence at all. +Only a repeat inside ONE log is a collision, which is why the count goes through +`_by_run` rather than over the logs together. Written the other way it would reject +every multi-day merge. + +### `test_a_clean_run_is_not_refused_for_a_shared_key` + +The false-positive budget: two different names in one part pass, and the gate returns 0. + ### `test_the_ceiling_may_only_fall_and_that_is_enforced` The tracked file says the ceiling may only fall. Without a mechanism that is prose, and diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 39e5213b..92192835 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -51,7 +51,12 @@ # 290 -> 298 when test_compare_to_bash.py landed: eight arms over the parity tool's own # extractors. Re-derived by collection on the merged tree, per the recipe above: # `298 tests collected`. -guard_tests 298 +# 298 -> 304: six arms in test_mutation_ledger.py about one ledger key covering +# more than one check (#982) -- the refusal, its uncovered-suite control, the two-runs +# distinction, the false-positive budget, and two for the same class one level down, where a +# duplicated row in the tracked ledger file collapsed silently and let the last line win. +# Re-derived by collection on the merged tree, per the recipe above: `304 tests collected`. +guard_tests 304 # The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran # in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and @@ -74,4 +79,14 @@ guard_tests 298 # 205 -> 217 when test_projection_privilege.py landed: twelve arms over the two projection # read helpers, parametrised per function so each carries the bash suite's own check name. # Re-derived by collection. -cluster_tests 217 +# 217 -> 219: two arms in test_harness_deps.py asserting THIS FILE states +# each key exactly once (#982). A keep-both merge of three PRs each moving guard_tests produced +# three of those lines, and the arm that read this file built a dict, so it could not see the +# duplicate. +# +# IN THE CLUSTER HALF, NOT THE GUARD HALF, which I got wrong first and this mechanism caught +# with `collected 283 test(s) but expected 285`: test_harness_deps.py DEFINES NO_CLUSTER and +# is not IN it. +# +# Re-derived by collection on the merged tree: `219 tests collected`. +cluster_tests 219 diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index 30c166bf..4ef25dd3 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -983,6 +983,83 @@ def test_both_pytest_jobs_assert_how_many_tests_they_collected(expect): "and each guards the read, because an empty value would fail open") +# A TRACKED KEY-VALUE FILE MUST NOT REPEAT A KEY, and the arm above could not see one. +# It builds `nums[f[0]] = int(f[1])`, so a duplicated key collapses and the last line +# wins -- the same shape as `read_ledger` before #982, one file over. A seam so the rule +# has fixtures, because the real file has no duplicate to redden on. +def _count_lines(text): + """-> [(key, value)] for every `name N` line, REPEATS INCLUDED.""" + out = [] + for line in text.splitlines(): + if line.startswith("#"): + continue + f = line.split() + if len(f) == 2 and f[1].isdigit(): + out.append((f[0], int(f[1]))) + return out + + +def test_each_expected_count_is_stated_exactly_once(expect): + """A merge that keeps both sides of this file duplicates a key, and CI says something + unrecognisable instead of saying that. + + `ci.yml` reads the value with `awk '$1=="guard_tests"{print $2}'`, which prints one + line per match. Two matches make `WANT` multi-line, `test -n "$WANT"` still passes, + and the flag then refuses it. Measured, with two and with three duplicated lines: + + pytest: error: argument --pgc-expect-tests: invalid int value: '284\n280\n283' + exit 4 + + SO IT FAILS CLOSED, which is why this is an arm about legibility rather than a hole. + Exit 4 reddens the job. What it does not do is say that a line is duplicated, and the + person reading it has to work back from an int parse error to a merge resolution. + + THIS IS NOT HYPOTHETICAL. Three PRs of mine were open at once, each moving + `guard_tests`, and a keep-both resolution across all three produced exactly this: + + guard_tests 284 + guard_tests 280 + guard_tests 283 + cluster_tests 205 + + Keep-both is the right resolution for a changelog and the wrong one for a key-value + file, and nothing in the tree said so. + """ + counts = HERE / "expected_tests.txt" + pairs = _count_lines(counts.read_text()) + expect.at_least(len(pairs), 2, "premise: the file states counts to check") + keys = [k for k, _v in pairs] + dupes = sorted({k for k in keys if keys.count(k) > 1}) + expect.text(", ".join(dupes) or "none", "none", + "no key is stated twice in expected_tests.txt") + expect.num(len(keys), len(set(keys)), + "so the lines and the distinct keys are the same count") + + +def test_a_duplicated_count_is_caught_and_the_dict_form_is_not(expect): + """The removal proof, and it shows WHY the arm above is not the one that existed. + + The older reading collapses the duplicate into a dict and reports a healthy file, so + the two forms are run side by side on the same fixture. + """ + fixture = ("# a comment\n" + "guard_tests 284\n" + "guard_tests 280\n" + "cluster_tests 205\n") + pairs = _count_lines(fixture) + keys = [k for k, _v in pairs] + expect.num(len(pairs), 3, "premise: the line reading sees all three lines") + expect.text(", ".join(sorted({k for k in keys if keys.count(k) > 1})), "guard_tests", + "the duplicated key is named") + + collapsed = {} + for k, v in pairs: + collapsed[k] = v + expect.num(len(collapsed), 2, "while the dict form sees only two keys") + expect.num(collapsed["guard_tests"], 280, + "and keeps the LAST line, which is how the duplicate stayed invisible") + + def test_the_job_installs_no_database_driver(expect): """The job's value is that it runs where there is no database. diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index b6e3bfe4..79436161 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -444,6 +444,197 @@ def test_the_committed_ledger_and_budget_agree(expect): "and the ceiling matches the suites with no rows") +def test_the_ledger_refuses_two_rows_sharing_one_key(tmp_path, expect): + """The same class one level down, found while building the arm above. + + `read_ledger` does `rows[(f[0], f[1], f[2])] = [...]`, so a duplicated key in the + TRACKED file collapsed silently and the LAST line won. Measured on a two-line + fixture, both orders: + + never first, then 2026-09-01 survivor last_red='2026-09-01' + 2026-09-01 first, then never survivor last_red='never' <- the red is GONE + + So line order decides whether a recorded red observation survives, and a merge that + keeps both sides of a changed row can turn `ever red` back into `never`. That is the + exact corruption #918 and #925 exist to prevent, arriving from the opposite direction + to #982's. + + NOTHING CAUGHT IT, and bounding the census cannot. `checks_never_observed_red` is a + CENSUS and the budget file says it must not become a ceiling, because every new check + enters as `never` and bounding it deadlocks. The gate only checks that the budget's + number equals the ledger's, and both are derived from the collapsed dict, so they + agree. Measured: with the budget regenerated alongside, the erasure passes at rc=0. + """ + dup = _w(tmp_path, "l.tsv", + "demo\tpart1\ta check\t18\t2026-09-01\tmut-A\n" + "demo\tpart1\ta check\t18\tnever\t-\n") + budget = _w(tmp_path, "b.txt", "suites_not_covered 0\n") + reg = _w(tmp_path, "reg", "demo\n") + log = _w(tmp_path, "r.log", + "RESULT\tdemo\tpart1\ta check\tPASS\t18\t\nchecks run: 1\n") + out, rc = _run("gate", "--ledger", dup, "--budget", budget, "--registered", reg, log) + expect.num(int("a ledger row repeats a key" in out), 1, + "a duplicated ledger key is refused by name") + expect.num(int("a check" in out), 1, "and the row is named") + expect.num(rc, 2, "as an integrity failure, not a gate verdict") + + +def test_a_ledger_with_no_duplicate_key_still_loads(tmp_path, expect): + """The false-positive budget, and the premise the refusal above needs. + + Two rows that differ only in the NAME are two checks and must load, which is the + ordinary case for every part in the tree. + """ + ok = _w(tmp_path, "l.tsv", + "demo\tpart1\tfirst check\t18\tnever\t-\n" + "demo\tpart1\tsecond check\t18\t2026-09-01\tmut-A\n" + "demo\tpart2\tfirst check\t18\tnever\t-\n") + budget = _w(tmp_path, "b.txt", "suites_not_covered 0\n") + reg = _w(tmp_path, "reg", "demo\n") + log = _w(tmp_path, "r.log", + "RESULT\tdemo\tpart1\tfirst check\tPASS\t18\t\nchecks run: 1\n") + out, rc = _run("gate", "--ledger", ok, "--budget", budget, "--registered", reg, log) + expect.num(int("a ledger row repeats a key" in out), 0, + "three distinct keys are not a duplicate") + expect.num(int("ledger census: rows=3" in out), 1, + "and all three rows loaded, so the refusal did not eat one") + + +def test_the_gate_refuses_two_checks_sharing_one_ledger_key(tmp_path, expect): + """#982's remaining half. `merge` PRINTS this and nothing fails on it. + + A ledger row is keyed on (suite, part, name), so two checks with the same name in + one part share a row. Neither is mis-recorded while both pass. The hazard is + conditional and exact: if one goes red the row records `ever red`, and its + namesake inherits a red observation nothing attacked -- which is the census + `checks_never_observed_red` exists to make trustworthy. + + THE GATE WAS STRUCTURALLY BLIND TO IT, which is why the note was not enough. + `cmd_gate` builds its records as `sorted({(s, p, n, m) for ...})`, and a set + collapses the duplicate before any arm can count it. The same canonicalisation + that makes the rest of the gate correct made this one class unreachable. + + Measured on main at `03c6c9c8`: a real `harness_selftest` run emits 934 RESULT + records over 934 distinct keys, 0 collisions, so this refusal is green on the tree + it lands in. The instance #982 reported was fixed by `c3b13aed`; this is the + mechanism that stops the next one. + """ + # THE FIXTURE HAS TO DEFEAT THREE OTHER ROUTES TO rc=1, and an arm asserting rc == 1 + # is worth nothing until it has. Each was measured reaching 1 on its own. + # + # 1. THE COVERAGE CEILING. With `suites_not_covered 0` the gate returns 1 for + # `suites_not_covered: 1 exceeds the ceiling of 0`. The budget names 1 instead. + # + # 2. THE NEW-CHECK REFUSAL, which is the route I missed and @jdatcmd found by + # mutation. The ledger must NAME `shared name`, not merely some other check in + # the part. With only `some other check` listed, `demo` is a covered suite whose + # log carries a check the ledger has never seen, and the pre-existing refusal + # sets rc by itself: + # + # not in the ledger: demo part1 shared name (on major 18) + # + # Measured: with `rc = 1` deleted from the shared-key block, the old fixture's + # arm STILL PASSED and this one fails. That is the whole difference between an + # arm about this refusal and an arm about the gate returning 1. + # + # 3. AN UNHANDLED EXCEPTION, which also exits 1 -- an earlier draft referenced + # `covered_suites` before it was defined, which compiles and fails at runtime. + # Hence the `Traceback` arm below. + # + # The ledger covering the suite is NOT why it is shaped this way: the refusal applies + # to every suite, which the next arm asserts. + ledger = _w(tmp_path, "l.tsv", + "demo\tpart1\tsome other check\t18\tnever\t-\n" + "demo\tpart1\tshared name\t18\tnever\t-\n") + budget = _w(tmp_path, "b.txt", "suites_not_covered 1\n") + reg = _w(tmp_path, "reg", "demo\n") + dup = _w(tmp_path, "dup.log", + "RESULT\tdemo\tpart1\tshared name\tPASS\t18\t\n" + "RESULT\tdemo\tpart1\tshared name\tPASS\t18\t\n" + "checks run: 2\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", budget, + "--registered", reg, dup) + expect.at_least(len(out), 20, "premise: the gate produced output to read") + expect.num(out.count("one ledger key covers 2 checks"), 1, + "a key covering two checks in one run is refused, and named") + expect.num(int("part1" in out), 1, "with the part, since the part is half the key") + expect.num(rc, 1, "and the gate fails rather than noting it") + expect.num(int("Traceback" in out), 0, + "premise: rc came from the refusal, not from a crash") + + +def test_a_shared_key_is_refused_in_a_suite_the_ledger_does_not_cover(expect, tmp_path): + """The refusal covers EVERY suite, deliberately unlike the new-check refusal. + + `test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers` is restricted because + it cannot know which of an uncovered suite's checks are new. This one needs no + history at all: two records, one key, one log is decidable from the log alone. + + WHY THAT MATTERS rather than being a detail. The ledger covers four suites of 253, so + a refusal restricted the same way would close the class in four places and leave the + next collision to arrive in one of the other 249 and sit there until that suite is + seeded. This arm is what stops the restriction being copied in by habit. + + MEASURED BEFORE WIDENING IT, because a gate that reddens 250 unmeasured suites is a + gate somebody turns off. A full PG 18 matrix with this refusal armed for every suite: + 247 suites ran, RC=0, ALL VERSIONS PASSED, zero shared keys. Check names are static, + so one major's matrix measures this class completely rather than sampling it. + """ + ledger = _w(tmp_path, "l.tsv", "") # covers NOTHING + budget = _w(tmp_path, "b.txt", "suites_not_covered 1\n") + reg = _w(tmp_path, "reg", "demo\n") + dup = _w(tmp_path, "dup.log", + "RESULT\tdemo\tpart1\tshared name\tPASS\t18\t\n" + "RESULT\tdemo\tpart1\tshared name\tPASS\t18\t\n" + "checks run: 2\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", budget, + "--registered", reg, dup) + expect.num(out.count("one ledger key covers 2 checks"), 1, + "a shared key in an uncovered suite is named") + expect.num(rc, 1, "and refused, with no row in the ledger for that suite") + expect.num(int("Traceback" in out), 0, + "premise: rc came from the refusal, not from a crash") + expect.num(out.count("not in the ledger:"), 0, + "premise: and not from the new-check refusal, which this suite escapes") + + +def test_the_same_check_in_two_runs_is_not_a_shared_key(tmp_path, expect): + """The control the refusal needs, and the distinction it must not lose. + + One check observed on two days is two records for one key and is the NORMAL case -- + it is how the ledger accumulates evidence at all. Only a repeat WITHIN one log is a + collision. Without this arm the refusal could be written as a count over all logs + together and would reject every multi-day merge. + """ + ledger = _w(tmp_path, "l.tsv", "demo\tpart1\tshared name\t18;19\tnever\t-\n") + budget = _w(tmp_path, "b.txt", "suites_not_covered 0\n") + reg = _w(tmp_path, "reg", "demo\n") + one = _w(tmp_path, "a.log", + "RESULT\tdemo\tpart1\tshared name\tPASS\t18\t\nchecks run: 1\n") + two = _w(tmp_path, "b.log", + "RESULT\tdemo\tpart1\tshared name\tPASS\t19\t\nchecks run: 1\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", budget, + "--registered", reg, one, two) + expect.num(out.count("one ledger key covers"), 0, + "the same check in two logs is not a shared key") + expect.num(rc, 0, "so a two-run merge is not refused") + + +def test_a_clean_run_is_not_refused_for_a_shared_key(tmp_path, expect): + """The false-positive budget: two DIFFERENT names in one part must pass.""" + ledger = _w(tmp_path, "l.tsv", + "demo\tpart1\tfirst check\t18\tnever\t-\n" + "demo\tpart1\tsecond check\t18\tnever\t-\n") + budget = _w(tmp_path, "b.txt", "suites_not_covered 0\n") + reg = _w(tmp_path, "reg", "demo\n") + clean = _w(tmp_path, "c.log", GREEN) + out, rc = _run("gate", "--ledger", ledger, "--budget", budget, + "--registered", reg, clean) + expect.num(out.count("one ledger key covers"), 0, + "two distinct names in one part are not a shared key") + expect.num(rc, 0, "and a clean run passes the gate") + + def test_the_gate_refuses_a_census_that_contradicts_its_own_ledger(tmp_path, expect): """#952. The gate PRINTED the census and never compared it, so rc=0 on a lie.