test: a ledger of which checks have ever been red (#918)

.github/workflows/ci.yml

@@ -387,6 +387,21 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
+      # THE LEDGER GATE NEEDS THE BASE BRANCH TO EXIST (#918). It compares the
+      # tracked ceiling in `test/check_ledger_budget.txt` against the one already
+      # on the PR's target, and it FAILS CLOSED when no trustworthy prior can be
+      # read rather than quietly enforcing less.
+      #
+      # actions/checkout fetches one ref at depth 1, so without this the base is
+      # absent and the gate stops the run -- correctly, but for a reason that is
+      # this workflow's to fix rather than the author's. Depth 1 is enough: only
+      # the file at that commit is read.
+      - name: fetch the PR base, for the ledger ceiling comparison
+        if: github.base_ref != ''
+        run: |
+          git fetch --depth=1 origin \
+            "+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}"
+
       # The repository and the package lists first, on their own, because the
       # cache key below is derived from the versions apt resolves and cannot be
       # computed before the lists exist.
@@ -539,7 +554,10 @@ jobs:
       - name: Collect logs on failure
         if: failure()
         run: |
-          for f in /tmp/pgcolumnar-matrix-*/*.log; do
+          # /tmp/pgcolumnar-logs, not the build directory: the runner removes
+          # each builddir at the end of its major, so this step used to search a
+          # path that no longer existed and print nothing.
+          for f in /tmp/pgcolumnar-logs/*.log /tmp/pgcolumnar-matrix-*/*.log; do
             [ -e "$f" ] || continue
             echo "===== $f ====="
             tail -60 "$f"

CHANGELOG.md

@@ -18,6 +18,78 @@ true until the next version shipped.
 
 ### Added
 
+- Every check result is machine-readable, and counting a check is the same
+  operation as recording it (#917).
+
+  `check`, `check_num` and `check_text` printed `PASS` or `FAIL` and nothing
+  else, so proving that a mutation reddened one NAMED check meant grepping prose
+  and retyping the result. That is also how a reverted guard once reported plain
+  green while the check count fell from 190 to 186: the suite passed, and the only
+  evidence anything had changed was a number nobody compared.
+
+  `lib.sh` had eleven places that bumped `PGC_CHECKS`, each with its own outcome
+  line beside it, which is eleven chances to add a twelfth and forget the line.
+  `projections.sh` did exactly that with an `expect_fail` at ten call sites, for
+  as long as it existed. There is now one, `pgc_record`, so a helper cannot report
+  an outcome without being counted and cannot be counted without reporting one.
+  Every human line is byte-identical; 3,762 call sites is past what a careful
+  refactor can be trusted on, so both harnesses pin the exact strings.
+
+  Each record names the suite, the part, the check, the verdict and the reason.
+  The part matters because `harness_selftest` sources 40-odd parts into one shell
+  and phrases its premises to be copied, so a key of suite and name is a key of
+  check NAMES rather than of checks: 583 records give 579 distinct pairs against
+  582 distinct triples. It is derived from `BASH_SOURCE`, not from a convention.
+
+  A skipped wall-clock check is a fourth counted outcome. Under
+  `PGC_SKIP_TIMING`, `check_timing` and `check_ratio_needs_quiet_machine` printed
+  a `SKIP` line a reader sees while leaving the count at zero and emitting no
+  record, in branches no arm reached. `checks run:` now reports the checks a suite
+  encountered rather than the ones it evaluated, and the summary reconciles four
+  counters against it. A suite that skipped every check reports `SKIPPED` rather
+  than `PASSED`, which the old zero-check condition caught only by accident.
+
+  The matrix reconciles each suite's records against the count its log states, and
+  names the cause rather than the arithmetic: more records than counted is a check
+  that ran in a subshell, fewer is a counter bumped outside `pgc_record`.
+- A ledger of which checks have ever been seen red, and under what (#918).
+
+  Nothing recorded it. That is the gap that let 39 checks across 35 suites ship
+  unable to fail, three of them inside the suite whose whole purpose is to stop
+  exactly that: the gate answered "did anything print FAIL" and had never
+  answered "could anything print FAIL".
+
+  It records that a named check WAS OBSERVED RED in a recorded run. It does not
+  claim the check is proven able to fail, which needs a named mutation applied
+  deliberately; conflating the two would put a claim in the ledger that nothing
+  measured. It is fed by every real failure, not only by deliberate mutation
+  runs.
+
+  `run_all_versions.sh` runs the gate before it removes the build directory,
+  which is the only place a matrix run can reach every suite's log. What the gate
+  refuses is a check the committed ledger has never seen, in a suite the ledger
+  covers. Regenerating the ledger is the intended fix and a reviewable diff.
+
+  The two tracked numbers are different kinds of thing, and the first design
+  treated both as ceilings and deadlocked. `checks_never_observed_red` is a
+  CENSUS: every new check enters as `never`, so bounding it means every added
+  check breaks the gate and the only way to land one is to raise a number the
+  design says may only fall. It shipped that way once, at 614 rows, 614 never,
+  ceiling 614. `suites_not_covered` IS a ceiling, because adding a check to a
+  covered suite does not move it, and the gate refuses to see it raised above its
+  previously committed value rather than leaving that to review.
+
+  The row is keyed on suite, part and check name. The part matters because
+  `harness_selftest` sources 40-odd parts into one shell and phrases its premises
+  to be copied, so a name-only key is a key of check NAMES: 583 records give 579
+  distinct pairs against 582 distinct triples. A rename is detected and named
+  rather than silently resetting a check's history to `never`, and the mutation
+  column accumulates a set rather than keeping only the most recent attack.
+
+  Bad input fails closed. An unreadable log, an empty one, and a record missing
+  its verdict each returned success before, which is worse than no gate because
+  it certifies.
+
 - A write that wrote no rows no longer passes as a fixture that built something
   (#432).
 

CONTEXT.md

@@ -209,8 +209,7 @@ this was first counted. The file is the stable unit, so each file below is named
 with the mechanism that makes it a reference -- which is also what has to change
 for it to stop being one.
 
-**The debt this starts with, on 2026-09-10: 3 python files and 7 shell files**,
-with a fourth python file arriving in PR #923.
+**The debt this starts with, on 2026-09-10: 3 python files and 7 shell files.**
 
 Python that reaches into shell:
 
@@ -222,8 +221,6 @@ Python that reaches into shell:
 - `test_suite_accounting.py` -- reads `run_all_versions.sh`'s text, sources the
   real `lib.sh` from a suite it writes, and executes the real runner.
 - `pgc_cluster.py` -- sources the real `test/lib.sh`.
-- `test_check_results_are_machine_readable.py` -- sources `./lib.sh`. Arrives
-  with PR #923; not on `main` yet.
 
 Shell whose subject is python: `lib.sh`, and `selftest/030`, `040`, `350`, `360`,
 `370`, `380`.

test/advisory_lock_class.sh

@@ -127,7 +127,7 @@ if [ -n "$LK_CLASSID" ] && [ "$LK_CLASSID" -le 2147483647 ] && [ "$LK_OBJID" -le
 		sleep 0.2
 	done
 else
-	echo "SKIP  classid $LK_CLASSID or objid $LK_OBJID exceeds int4, so the SQL form cannot address it"
+	check_skip "the SQL form of the advisory lock" "SKIP  classid $LK_CLASSID or objid $LK_OBJID exceeds int4, so the SQL form cannot address it" "classid or objid exceeds int4"
 fi
 
 # ---------------------------------------------------------------------------

test/analyze_differential.sh

@@ -54,13 +54,12 @@ ROWS=${PGC_ANALYZE_DIFF_ROWS:-50000}
 # The major is asserted first so an unreadable version is not mistaken for an old
 # one and reported as "supported, skipped".
 if ! pgc_is_number "${PGC_MAJOR:-}"; then
-	echo "FAIL  could not read the server major, so the gate below cannot be trusted: got [${PGC_MAJOR:-<none>}]"
-	PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1))
-	PGC_FAIL=1
+	pgc_fail "could not read the server major, so the gate below cannot be trusted" \
+		"got [${PGC_MAJOR:-<none>}]"
 	pgc_summary
 fi
 if [ "$PGC_MAJOR" -lt 18 ]; then
-	echo "SKIP  pgcolumnar.analyze() needs pg_restore_attribute_stats (PG18+); this server is $PGC_MAJOR"
+	check_skip "the differential analyze path" "SKIP  pgcolumnar.analyze() needs pg_restore_attribute_stats (PG18+); this server is $PGC_MAJOR" "needs pg_restore_attribute_stats, PG18+"
 	pgc_summary
 fi
 

test/analyze_function.sh

@@ -63,13 +63,12 @@ ROWS=${PGC_ANALYZE_ROWS:-500000}
 # The major is asserted first. An unreadable version must not be mistaken for an
 # old one, or a broken environment would report SKIP and look supported.
 if ! pgc_is_number "${PGC_MAJOR:-}"; then
-	echo "FAIL  could not read the server major, so the gate below cannot be trusted: got [${PGC_MAJOR:-<none>}]"
-	PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1))
-	PGC_FAIL=1
+	pgc_fail "could not read the server major, so the gate below cannot be trusted" \
+		"got [${PGC_MAJOR:-<none>}]"
 	pgc_summary
 fi
 if [ "$PGC_MAJOR" -lt 18 ]; then
-	echo "SKIP  pgcolumnar.analyze() needs pg_restore_attribute_stats (PG18+); this server is $PGC_MAJOR"
+	check_skip "pgcolumnar.analyze()" "SKIP  pgcolumnar.analyze() needs pg_restore_attribute_stats (PG18+); this server is $PGC_MAJOR" "needs pg_restore_attribute_stats, PG18+"
 	pgc_summary
 fi