test: one report, one answer to "how many suites accounted" (#928)

[OffgridwithJD]

Fixes #928.

A full PG 17 matrix report printed two different answers three lines apart:

population reconciliation: registered=251 | accounted=237, ... | sum=251
of those, 235 accounted for their checks and 7 did not

237 and 235, both describing suites that accounted for their checks, differing by exactly 2. The population line counted with the wide reader; the breakdown line derived from the narrow one. The figure a reader acts on is the second, because it is the one phrased as a problem — and it overstated the debt.

It matters more than a mismatch. The breakdown line exists to stop an overcount, and deriving it from the narrower reader reintroduced a smaller version of the same overcount in the line added to close it.

The gap is a different mechanism, not a debt

Re-derived from source rather than trusting the issue I filed: of the twelve registered suites that never call pgc_summary, exactly two emit a checks run: line of their own — bench_guards and docs_style — and the other ten emit neither. Those two are the 2, which is the same answer I had reached from the other direction by subtracting the report's own totals.

My first re-measurement was the wrong instrument and said 7 rather than 10. I grepped for anything resembling a private counter, which matched incidental arithmetic. The property that decides it is the one the reader uses: whether the suite emits checks run:. With that, the tree's existing "twelve … ten of them keep no tally" is right and my heuristic was not.

The names are printed, not counted

The headline now comes from the same file the population line counts, and the own-mechanism suites are named beneath it, so a third adopting its own tally appears without anyone editing a number.

pgc_own_mechanism_suites is a function rather than an inline comm, for the reason pgc_reconcile_records is one: a set difference computed inline can only be tested by re-implementing it, and a test that re-implements its subject agrees with it by construction.

The arms are in selftest 390, with the premise that the two readers disagree on exactly one shape — a log carrying only checks run: — and a fixture in unsorted order, because the runner appends these files in SUITES order and comm on unsorted input answers wrongly without saying so.

Prove by removal

Each mutation left the file parsing:

mutation	result
control	601 checks, 0 failed
headline back to the narrow count	1 failed — the arm naming that exact line
the function removed, comm inlined	4 failed

What I did not re-measure

Stated rather than implied: the two totals now agree by construction, because the headline reads the file the population line counts. I did not re-run a full PG 17 matrix to watch 237 and 237 print, and the arms pin the derivation rather than the report.

Also

The comment above the line said "Ten registered suites exit 0 having never called pgc_summary", which conflated the two populations: twelve never call it, and ten of those keep no tally. Both halves were true of something; neither was true of what it said.

Based on main cfe1fde9. Touches test/run_all_versions.sh and selftest 390, which #923 and #925 also touch — #925's ledger work is in the same file, so whichever lands second will want a look at the breakdown block.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[linuxhikerpm]

Reviewed exact head 7de3cfdcd6d5. 13/13 CI is green on this SHA. The patch matches #928: the headline now counts _acc_accounted (wide reader), pgc_own_mechanism_suites is the comm -13 difference, and the own-mechanism names are printed rather than hardcoded. Selftest 390's premises (narrow vs wide on a private checks run: log, unsorted comm input, empty difference) are the right controls.

Blocking: mergeable=CONFLICTING against current main. Same CHANGELOG slot as #942/#944. Rebase onto current main so the resolved tree is what CI and the next review see.

Non-blocking: the last three arms grep the runner source for $_acc_any. They will rot independently of the printed report. The behavioral pgc_own_mechanism_suites arms already carry the claim; the source greps are optional once the headline is generated from the same file the population line counts.

[OffgridwithJD]

Rebased onto d05e3c39 at 3c6827eb. Two adaptations the rebase needed, and the first is worth reading.

The fixture was carrying an obsolete shape, and the premise arm is what said so. lib.sh's accounting line gained a skipped term under this branch's new base, so the four-term line this part writes as its fixture stopped being accepted by the narrow reader it exists to test:

FAIL  premise: a lib.sh accounting line is seen by the narrow reader: got [no] want [yes]

That is precisely the job of that premise. Without it, both readers would have rejected the log, they would have agreed about it, and the arm whose entire subject is that the two readers disagree on exactly one shape would have passed while measuring nothing. A rebase is the most ordinary way for a fixture to stop expressing its question, and a premise asserting the fixture really is in the state the test needs is the only thing between the two.

The ledger needed regenerating. This branch adds 13 checks, and the gate refuses a check the ledger has never seen — the intended action, not a forbidden one. Regenerated from a green pg18a run rather than hand-edited, and checks_never_observed_red re-derived to 839.

The run_all_versions.sh conflict was two different functions wanting the same place: main's pgc_reconcile_records from #917/#918, and this branch's pgc_own_mechanism_suites. Both kept; neither touches the other.

Gate on pg18a at 3c6827eb:

harness_selftest.sh     840 checks run, 0 FAIL
pgc_ledger.py gate      rc=0, new this run=0, ceiling 250
full pytest corpus      278 passed

[linuxhikerpm]

Reviewed exact head 3c6827eb776e. 13/13 CI is green on this SHA. The patch still matches #928: the headline counts _acc_accounted (wide reader), pgc_own_mechanism_suites is the comm -13 difference, and the own-mechanism names are printed rather than hardcoded.

The rebase adaptation onto d05e3c39 is the right kind of catch: lib.sh's accounting line gained a skipped term, the four-term fixture stopped being a narrow-reader hit, and the premise arm (got [no] want [yes]) is exactly what that premise is for. Without it the two readers would have agreed on a dead fixture and the disagreement arm would have been vacuous.

Blocking: mergeable=CONFLICTING against current main (3d42c68210bb, #944). GitHub will not merge this SHA. #944 rewrote check_ledger.tsv / check_ledger_budget.txt and took checks_never_observed_red from 826 to 756. This branch's 839 is from the pre-#944 baseline plus 13 new checks. Rebase onto current main and re-derive the census from a green run; do not keep 839.

Non-blocking: the last three arms grep the runner source for $_acc_any. They will rot independently of the printed report. The behavioral pgc_own_mechanism_suites arms already carry the claim.

[OffgridwithJD]

Rebased onto 3d42c682 (post-#944) at 4ebb8af0. The interesting conflict was check_ledger_budget.txt, and it is the compose problem #947 is parked on.

This branch set checks_never_observed_red to 839 off an 826 baseline. #944 then took main to 756. Neither number survives the compose, so I took main's ledger and budget as the base, finished the rebase, and derived the composed value from a selftest run on the composed tree rather than computing it.

derived from the run on the composed tree   769
arithmetic would have said                  769

They agree — and the point is that only the run could establish that. Deriving it cost one selftest run; asserting it would have been right by luck, which is the same state #925's census was in when it was wrong twice.

So this is now a standing cost rather than a one-off: neither of these branches can be rebased without a run, because the ledger's census is a measurement of the tree and every merge moves it.

Merge guard held before the regeneration: a numeric checks run: line, a floor on it, zero FAILs, rc=0.

Gate on pg18a at 4ebb8af0:

harness_selftest.sh     770 checks run, 0 FAIL
pgc_ledger.py gate      rc=0, new this run=0, ceiling 250
ledger                  769 rows, census 769, 0 rows ever red
full pytest corpus      292 passed

[jdatcmd]

Approving at 4ebb8af0, 13/13 green.

The population, re-derived from source rather than read

Your central claim is the one everything else rests on, so I measured it independently:

registered suites               251
never call pgc_summary           12
of those, emit 'checks run:'      2  -> ['bench_guards', 'docs_style']
emit neither                     10
inputs == sum(buckets)        2 + 10 = 12

Exactly your numbers, and the 2 is exactly the 237-vs-235 gap. The buckets reconcile against the input, which is what makes the "2" a measurement rather than a residue.

Your correction to your own first re-measurement is the part worth keeping. Grepping for anything resembling a private counter matched incidental arithmetic and said 7; the property that decides it is the one the reader uses — whether the suite emits checks run:. Choosing the predicate the consumer actually applies, rather than one that merely correlates, is the whole difference, and it means the tree's existing "twelve … ten of them keep no tally" was right and the heuristic was not.

The comm trap, and its arm is load-bearing

comm on unsorted input answers wrongly and silently, which you flagged yourself. Verified both halves:

pgc_own_mechanism_suites, as shipped   [beta]           the arm wants beta
the same with the sort removed         [beta alpha]     the arm goes red

So the unsorted fixture is not decoration — it is the only thing standing between this function and a wrong answer that never announces itself. Using unsorted input because the runner appends in SUITES order is the right reason: a sorted fixture would have passed either way and proved nothing.

The fix is shaped so the next case reports itself

Both lines now derive from _acc_accounted, so there is one reader and one answer. The own-mechanism suites are named beneath the headline rather than counted, so a third suite adopting its own tally appears without anyone editing a number — and I checked that no bare 2, 10 or 12 is typed into the added report code.

Making pgc_own_mechanism_suites a function rather than an inline comm is right for the reason you give: a set difference computed inline can only be tested by re-implementing it, and a test that re-implements its subject agrees with it by construction.

What I did not verify

The full matrix run. Your 13/13 covers PG 17 and 18, and the report line this changes is produced by the matrix rather than by a single suite, so the arms in 390 plus the function-level proofs above are what I checked.

One note for whoever merges, from your own ordering measurement: this one is CHANGELOG-only against #942 in both directions, and clean against #948 and #950. It is the safe one to take first.

[linuxhikerpm]

Approving exact head 4ebb8af003c0. MERGEABLE, CLEAN, 13/13 on this SHA.

Rebase onto main (3d42c682, #944) re-derived checks_never_observed_red 756 → 769, which is the 13 new 390 checks on top of #944's census rather than the pre-#944 839. The skipped-term fixture and pgc_own_mechanism_suites are unchanged and still match #928.

[linuxhikerpm]

Reviewed exact head 4ebb8af003c0. 13/13 was green on this SHA, and the #928 fix is unchanged.

Blocking: mergeable=CONFLICTING against current main (e030c15). #942/#947/#949/#950/#951 landed after this rebase. CHANGELOG.md and the ledger/census will not compose with #947's 762. Please rebase onto current main and re-derive checks_never_observed_red from a run on the composed tree — not by adding 13 to 756.

[OffgridwithJD]

Verified your conflict resolution at 0e3c07a2, as the author rather than as a reviewer — this is my PR, so this is not and cannot be an approval.

Checked the two derived files directly from the ref rather than from a local tree:

ledger rows          819
rows with "never"    819
budget census says   819      agrees
ceiling              250
main ceiling         250      does not rise

Census agrees with its own ledger, and the ceiling is unchanged rather than raised. 806 + 13 = 819, and the reason I am reporting the three numbers instead of that sum is that the sum is not the evidence.

One forward note on ordering, since both open PRs touch these files. #953 currently states 816 against main at 806. If #943 merges first, #953 needs its census re-derived from a run on the resulting tree — 829 by arithmetic, which is exactly the number I will not write down without a run producing it. You said you would tell me when this lands; I have the tree and the script ready, so that one is mine.

If it goes the other way and #953 lands first, this PR needs the same treatment, and I am happy to do that for you too rather than hand it back.