Split out of #858. Half done: the pytest harness reconciles, the bash harness does not.
The gap
registered == passed + failed + skipped + incomplete is asserted nowhere in the bash runner. The project already demands inputs == sum(buckets) for any list-derived claim; the runner does not apply that rule to itself.
Measured when #858 was filed: 237 registered suites, and CI legs that ran 228 and 231. Nothing reconciled those numbers and no skip carried a machine-readable reason.
Done: the pytest half, in #906
_RunShape (test/pytest/pgc_vacuity.py) keeps collected and reported node-id sets and refuses a session that finishes fewer tests than it started.
Its motivating measurement is in the source, and it is the case a count cannot see: a crashed xdist worker under --max-worker-restart=0 gave 8 collected, summary "1 failed, 6 passed", one named test never reported, and pytest printed no warning. Counting collected tests cannot detect that — the collected count is still right.
It also survived a real defect on the way in, worth recording because the same trap is waiting for the bash half: the first version reconciled against the pre-deselection set, so pytest -k failed a healthy run with exit 1. A guard whose subject is false greens producing a false red is the fastest way to get itself switched off. Fixed via pytest_deselected, and verified after the fix that it kept its teeth:
pytest -k refus 1 passed, 15 deselected exit 0
--pgc-expect-tests 999 vs 16 tests ERROR: collected 16 but expected 999 exit 4
Remaining: the bash half
run_all_versions.sh prints suites that ran: N of M and names the skips. That is reporting, not reconciliation — nothing refuses a batch that silently ran fewer suites than the list contains.
The measured limit any fix has to handle
12 of 238 suites are outside the accounting. bench_guards.sh and docs_style.sh never source lib.sh. smoke, audit, concurrency, phase2 through phase6, unique_conc and update_conc source it but never call pgc_summary and print their own verdicts — smoke.sh simply says SMOKE TEST PASSED.
Any reconciliation is a lie for those twelve until they are brought in or exempted with a premise that fails when the list grows. An exemption list that can silently lengthen is the same defect one level up.
Verification
Per the house rule, in both harnesses in the same change — though the pytest half already exists, so in practice this is the bash arm plus a twin that observes it. The arm that matters is the one that reddens when a suite is dropped from the run without being counted, not one that reddens when the printed total changes.
Split out of #858. Half done: the pytest harness reconciles, the bash harness does not.
The gap
registered == passed + failed + skipped + incompleteis asserted nowhere in the bash runner. The project already demandsinputs == sum(buckets)for any list-derived claim; the runner does not apply that rule to itself.Measured when #858 was filed: 237 registered suites, and CI legs that ran 228 and 231. Nothing reconciled those numbers and no skip carried a machine-readable reason.
Done: the pytest half, in #906
_RunShape(test/pytest/pgc_vacuity.py) keepscollectedandreportednode-id sets and refuses a session that finishes fewer tests than it started.Its motivating measurement is in the source, and it is the case a count cannot see: a crashed
xdistworker under--max-worker-restart=0gave 8 collected, summary "1 failed, 6 passed", one named test never reported, and pytest printed no warning. Counting collected tests cannot detect that — the collected count is still right.It also survived a real defect on the way in, worth recording because the same trap is waiting for the bash half: the first version reconciled against the pre-deselection set, so
pytest -kfailed a healthy run with exit 1. A guard whose subject is false greens producing a false red is the fastest way to get itself switched off. Fixed viapytest_deselected, and verified after the fix that it kept its teeth:Remaining: the bash half
run_all_versions.shprintssuites that ran: N of Mand names the skips. That is reporting, not reconciliation — nothing refuses a batch that silently ran fewer suites than the list contains.The measured limit any fix has to handle
12 of 238 suites are outside the accounting.
bench_guards.shanddocs_style.shnever sourcelib.sh.smoke,audit,concurrency,phase2throughphase6,unique_concandupdate_concsource it but never callpgc_summaryand print their own verdicts —smoke.shsimply saysSMOKE TEST PASSED.Any reconciliation is a lie for those twelve until they are brought in or exempted with a premise that fails when the list grows. An exemption list that can silently lengthen is the same defect one level up.
Verification
Per the house rule, in both harnesses in the same change — though the pytest half already exists, so in practice this is the bash arm plus a twin that observes it. The arm that matters is the one that reddens when a suite is dropped from the run without being counted, not one that reddens when the printed total changes.