Eleven timeout paths print a FAIL nothing records, and the accounting balances at the wrong number

[OffgridwithJD]

Ten registered suites emit no RESULT lines and no checks run: total. They print their own PASS <name>: <value> lines and their own verdict — AUDIT TEST PASSED, CONCURRENCY TEST PASSED — and exit 0. They pass, and nothing that reads machine-readable results can see them.

Found while measuring #432's coverage half, by running all 252 registered suites on PG18.

Measured

suite          PASS-style lines   RESULT lines   checks run:   own verdict line
audit                   31              0            absent          yes
concurrency              7              0            absent          yes
phase2                  42              0            absent          yes
phase3                  32              0            absent          yes
phase4                  38              0            absent          yes
phase5                  36              0            absent          yes
phase6                  43              0            absent          yes
smoke                    9              0            absent          yes
unique_conc             30              0            absent          yes
update_conc             25              0            absent          yes
                       ---
                       293

293 checks. For comparison the whole machine-readable corpus is 6,422 checks across 240 suites, so this is about 4.4% of the suite surface, concentrated in ten files.

Read first: 293 is a PG18 number, and one version gate is why

The count in these suites moves with the PostgreSQL major, so anyone chasing
the remainder must re-count on their own box rather than reconcile against 293.

Measured, not inferred — test/audit.sh run on both majors from the #969 tree at
66507a92, each in its own build tree:

	PG18 (pg18a)	PG16 (pg16a)
checks run: / RESULT records	26	23
human PASS lines	31	27
of those, from expect_error (unrecorded)	5	4
the server_version_num >= 170000 arm	taken	not taken
rc	0	0

The whole divergence is one gate, test/audit.sh:293. PG16 and earlier refuse
PARTITION BY ... USING pgcolumnar, so the partitioned-parent arm is gated out,
and it holds exactly 3 check calls and 1 expect_error — which is the 3-record
and 1-unrecorded difference, with nothing left over. linuxhikerpm measured all
ten suites on PG16 independently and audit is the only one of the ten that moves.

So the totals for both majors:

	PG18	PG16
census of human PASS lines	293	289
recorded after #969	236	233
remainder still unrecorded	58	57

Each recorded figure includes one SKIP record for the citext case that has no
PASS line in the base census, which is why recorded + remainder exceeds the
census by one.

Correction to something I said earlier: I quoted the remainder as 57 on PG18.
That was linuxhikerpm's PG16 figure applied to my PG18 census. On PG18 it is 58 —
the audit row is 5 there, not 4.

The gate records nothing, which is how the denominator moves silently

The two version gates in these ten suites handle their skipped arm differently:

unique_conc.sh:546   else  check_skip "the NULLS NOT DISTINCT case" ...   -> a SKIP record
audit.sh:313         else  echo "-- PG16 refuses PARTITION BY ..."        -> nothing

On PG16 the ledger therefore receives 23 rows for audit with nothing saying an
arm was gated out. The count simply differs, and a reader reconciling it against
a PG18 figure sees three checks that look lost.

audit.sh:313 should call check_skip, exactly as unique_conc.sh already does.
That is the same class #969 fixed for citext — an arm that is not run, not recorded,
and not reported — and it belongs in the remainder pass below rather than in #969.

This also argues against ever seeding these suites into the ledger from one major:
the ledger has no major dimension, so a census taken on PG18 states 26 rows for
audit that a PG16 run contradicts.

What cannot see them

Everything built on the RESULT vocabulary since #917:

• the ledger — pgc_ledger.py merge reads RESULT lines, so none of these 293 checks can ever become a row. suites_not_covered therefore has a floor of 10 that no amount of seeding can lower.
• checks_never_observed_red — a census of checks the ledger holds, so these are outside it.
• the red-observation record — nothing can record that one of these was ever seen red, which is the gap Phase 4 of #858: nothing records whether a check has ever been red #918 and test: a ledger of which checks have ever been red (#918) #925 exist to close.
• duplicate-check-name detection — pgc_ledger.py warns when one run emits the same check name twice; these suites are exempt by invisibility.
• checks run: reconciliation — the merge guard refuses a log whose total does not reconcile with its records. A log with neither is not refused, it is simply empty.

What still does see them

Stated because the failure mode matters: their exit status. audit.sh runs set -euo pipefail with an EXIT trap that prints AUDIT TEST ABORTED (exit N) before reaching its verdict, so a failing command aborts the suite and the runner sees a non-zero exit. So these are not unguarded — a failure is caught. What is missing is per-check granularity and any record that survives the run.

That distinction is the reason this is not urgent: a regression in one of these suites still reddens the matrix. It just reddens it without saying which of 43 checks failed, and without any of the ledger machinery being able to say whether that check had ever passed before.

Why it matters for #432 specifically

suites_not_covered cannot go below 10 while these exist in this form, so the coverage burn-down has a floor that is not about effort. And the 293 checks are exactly the population where "a check that cannot fail" would be undetectable, because the mechanism that detects that (#918's ledger, #925's census) cannot see them.

Proposed

Convert them to check / check_num / check_text from lib.sh. That is mechanical per call site but 293 of them, and each conversion has to preserve what the existing line asserts rather than approximate it — several print a value alongside the name (PASS B both first deletes survived (ids 1,2 gone): 3,4,5,6), which maps onto check_text with the value as the comparison rather than onto check with a message.

Worth doing in batches by file, with each file's own verdict line removed only once its checks are individually counted, so the suite never reports less than it did before.

Not proposed

Deleting the suites or excluding them from the registry. They test real behaviour — audit alone holds 31 checks on add-column and transaction interaction — and the problem is their vocabulary, not their content.

Remaining after #969: 58 checks on PG18, 57 on PG16

#969 converted each suite's own check helper and recorded 236 of the 293. The
rest go through additional helpers with their own display formats, so they need
a second pass rather than the same substitution:

suite	helper	PG18	PG16
phase6	eq_on_off	39	39
phase4	assert_plan, assert_plan_seq, expect_fail	9	9
audit	expect_error	5	4
phase5	assert_plan	5	5
		58	57

phase6's 39 come from 40 call sites, one of which is shared. Each helper prints a
different line — expect_error emits PASS <name> (rejected), so the conversion
must keep (rejected) in the display or the human log says less than it did.

Also in that pass: audit.sh:313, so the gated-out arm records a SKIP.

Eleven timeout paths remain after #970, and a green run cannot see them

Found by sweeping the ten for verdict-printing lines after the conversion, not by
counting — no count could have found them, because on a green run not one executes.

concurrency.sh   4   timeout waiting for $name/$label, ...sentinel, ...to block, ...idle-in-transaction
unique_conc.sh   4   the same four
update_conc.sh   3   three of the four
                --
                11

Each prints FAIL timeout waiting for ..., sets the suite-local fail, and touches
neither PGC_CHECKS nor the record stream.

Measured by inducing one, rather than argued. send_wait's iteration bound was
lowered from 1200 (60 s) to 1 in a scratch tree, so the first wait reports a timeout
though its sentinel would have arrived. That changes when the path fires, not what it
does — the three lines under test are untouched, and a real 60-second hang runs the same
three. All four bounds in the file were rewritten and the count asserted before the run.

rc                              1
suite verdict                   CONCURRENCY TEST FAILED
FAIL lines printed              1   (FAIL  timeout waiting for s1/a_del)
RESULT records with FAIL        0
RESULT records total            7
checks run:                     7
human verdict lines             8
human - records                 1   <- the timeout, counted nowhere

So, precisely:

• it is not a false green today. All three suites derive their verdict from the local
  fail (concurrency.sh:394, unique_conc.sh:588, update_conc.sh:478), and the run
  above exits 1 and says CONCURRENCY TEST FAILED;
• but the reconciliation is self-consistent and still wrong. Records 7 and
  checks run: 7 agree with each other, so nothing refuses the log — on a run that
  printed a FAIL and exited 1. Once these suites emit an accounting: line it will read
  7 passed + 0 failed + 0 unrunnable + 0 skipped = 7, which is a reconciling statement
  asserting zero failures on a failing run. That is worse than invisibility: an
  aggregate that is merely missing data can be noticed, and one that balances cannot;
• and it becomes an outright false green the day any of the three derives its verdict
  from PGC_FAILED instead, which is the natural next step after this conversion.

Two things have to be decided before converting them, and neither is mechanical.

The name is dynamic. "timeout waiting for $name/$label" interpolates a session name
and a sentinel label, so a direct conversion puts unbounded names into a ledger keyed on
(suite, part, name) — rows nobody can enumerate and therefore nobody can seed. The
likely right shape is a fixed name per helper with the dynamic part in pgc_record's
REASON field, which exists for that, but that is a decision about the key space and
belongs with the seeding design.

It cannot be proven by running the suite. These lines do not execute on a green run,
so every count stays identical whether the conversion is right, wrong, or absent. It needs
an induced timeout — a fixture that makes a wait actually exhaust its 60 seconds — and
that arm is the work, not the substitution.

Provenance

Measured on PG18 at main 9cfa32d8, one run of every registered suite. Eight unrelated suites in that run were refused by the #960 stamp check because I had overwritten the prefix mid-run; those were re-run after restoring it and are green, and are not part of the ten above.

[OffgridwithJD]

Taking this, claimed here rather than in a message so it is on the record. @jdatcmd is on #937 phase 3; this is mine.

Converting one file first as the pattern-setter, so the shape is reviewed before it is repeated ten times. 293 check conversions across ten files is not a change to land in one diff, and the first file decides what the other nine look like.

Two things I intend to hold to, both from what the existing PASS lines actually print:

Several carry a value beside the name — PASS B both first deletes survived (ids 1,2 gone): 3,4,5,6. That is a comparison, not a message, so it maps onto check_text/check_num with the value as the comparison rather than onto check with a label. Converting it to check "..." "yes" would preserve the line and lose the assertion.

And each file ends with its own verdict — AUDIT TEST PASSED. That line is load-bearing today, because it is the only thing that says the suite reached its end; audit.sh has an EXIT trap printing AUDIT TEST ABORTED (exit N) before reaching its verdict precisely because a set -e abort is otherwise invisible. So the verdict line comes out only once the checks it summarises are individually counted and checks run: reconciles — never in the same step as the conversion, or the suite briefly reports less than it did before.

I will post the first file with a before/after of what each line asserted, and the count it now reports, before touching the other nine.

[OffgridwithJD]

The conversion is complete. All 293 checks in the ten suites now record, and this issue is retitled to what is left: the eleven timeout paths.

#969  merged  each suite's own check helper          236 records
#970  merged  the four remaining helpers              +58 records
                                                     ----
                                                      294 on PG18

294 rather than 293 because unique_conc.sh gained a SKIP record for the citext case, which had no PASS line in the original census — that was a check_skip call failing silently as command not found, fixed incidentally by #969 sourcing lib.sh.

Per suite, measured on PG18, with each suite's records now equal to both its own human check lines and its checks run: total:

suite          records   checks run:   human lines
audit               31            31            31
phase4              38            38            38
phase5              36            36            36
phase6              43            43            43
smoke                9             9             9
concurrency          7             7             7
phase2              42            42            42
phase3              32            32            32
unique_conc         31            31            31
update_conc         25            25            25

And audit.sh:293's version-gated arm now records a SKIP with its reason instead of printing a bare note, so PG16's 28 records carry the fact that an arm was gated out rather than leaving it to be inferred from three missing rows.

What this issue now tracks: the eleven

The section above has the measurement. In short: eleven timeout paths in the three concurrency suites print FAIL and record nothing, no count can find them because none executes on a green run, and the reconciliation balances at the wrong number — records 7, checks run: 7, agreeing with each other on a run that printed a FAIL and exited 1.

Two decisions block a mechanical conversion, and both belong here rather than in a substitution pass:

1. The name is dynamic. "timeout waiting for $name/$label" would put unbounded names into a ledger keyed on (suite, part, name). The likely shape is a fixed name per helper with the dynamic part in pgc_record's REASON field.
2. It cannot be proven by a green run, so it needs an induced-timeout arm. I have induced one by hand to measure the gap; a permanent arm is a different thing.

What this issue does NOT track

Seeding these suites into the ledger. Making a suite coverable and covering it are separate decisions, and the second is blocked on the ledger having no major dimension — now with two measured instances rather than one, since audit differs by major in exactly the way fk_referencing does. That lives in #432.

suites_not_covered no longer has a floor of 10 on account of this issue.

[jdatcmd]

The PG16 numbers in the table above were single-sourced from my runs. Here they are again from merged main, so anyone can re-derive them without trusting either of us.

@OffgridwithJD flagged that nine of the ten PG16 rows rest on my measurements alone, which is the right thing to say out loud — especially the day I found three instances of a stale ref failing in the flattering direction.

Measured on 972820e5, PG16, all ten suites

SUITE            records    checks run:    human lines
smoke                  9              9              9
audit                 28             28             28
concurrency            7              7              7
phase2                42             42             42
phase3                32             32             32
phase4                38             38             38
phase5                36             36             36
phase6                43             43             43
unique_conc           31             31             31
update_conc           25             25             25
                     ---
TOTAL                291

All three columns agree in every suite. That is the property worth stating rather than the total: two of the three could match while both were wrong about the third.

Why these are safe from the ref fault

The earlier numbers came from running both trees, not from a merge-base range, so the stale ref could never reach them — it only touched the base column of the four-suite table on #970, which was caught and re-run. But the copies those runs used were made with --exclude=.git, so their identity now rests on my command history rather than on the artifact. The table above has no such gap: it is merged main at a named sha, and the command is PGC_SKIP_BUILD=1 bash test/<suite>.sh /usr/local/pg16/bin/pg_config against a checkout of it.

The per-major delta, restated against these

PG16 totals 291. The difference from a PG18 run is audit, and only audit: below PG17 the partitioned-parent arm is refused by core, so PG16 emits one SKIP where PG17-and-above run the real checks. Every other suite in the ten is major-invariant here.

Which is the argument for @OffgridwithJD's skip-loop shape being the rule rather than an accident: the five names are emitted either way and only the verdict moves, so a count that differs by major does so for a reason the ledger can see, instead of by rows appearing and disappearing.