test: gate the hand-rebuild arms and say why it failed (#1248)

CHANGELOG.md

@@ -458,6 +458,52 @@ true until the next version shipped.
 
 ### Fixed
 
+- A failed hand rebuild now says why, and stops the arms beneath it reporting on
+  a rebuild that did not happen (#1248).
+
+  The 2026-09-24 nightly failed on aarch64 with the whole diagnosis being
+
+  ```
+    FAIL  premise: the hand rebuild itself succeeded: got [1] want [0]
+  ```
+
+  and nothing else. `rebuild.sh`'s output goes to a file the part deletes on the
+  way out, so the cause left with the workdir. **The three arms below that
+  premise then ran anyway and reported PASS.**
+
+  They were not vacuous that night, and only by luck: `rebuild.sh` writes both
+  stamps at step 3b and the unresolved-symbol check that rejected the build comes
+  after, so the stamps the three arms read had in fact been written. Had the
+  *build* failed, the same three arms would have read an absent stamp. That is
+  what "the premise does not gate" costs.
+
+  Both decisions are now functions the part drives directly, the way part 190
+  drives `pgc_build_needs_clean`, rather than inline branches that only a broken
+  machine exercises:
+
+  ```
+    _hr_dependents RC   ->  run | skip
+    _hr_diagnose  LOG   ->  the last 20 lines, indented, or nothing
+  ```
+
+  Driven with `rebuild.sh` forced to exit 1 *after* its stamp, reproducing the
+  aarch64 shape exactly:
+
+  ```
+    FAIL   premise: the hand rebuild itself succeeded
+    SKIP   and the freshness gate then reads the tree as built from this source
+    SKIP   and it reads a different source as stale rather than fresh
+    SKIP   and the installed library of the running major was never touched
+    the diagnosis is printed, and carries the failing line rather than a status
+  ```
+
+  **This does not fix the aarch64 failure and does not guess at it.** The
+  hypothesis is GCC's outline-atomic helpers (`__aarch64_*`) resolving from
+  libgcc and falling outside the `ldd` reference set the symbol check builds, and
+  there is no aarch64 box here to test it on. What this change does is make the
+  next nightly report the symbol names instead of a bare exit status, so the fix
+  after it is measured rather than reasoned about.
+
 - `pgc_reconcile_records` no longer reports an impossible mismatch on a log it
   cannot measure (#1242). Handed a log that states `checks run: N` and carries
   no `RESULT` records at all, it ran the arithmetic anyway and concluded that

test/check_ledger.tsv

@@ -1409,13 +1409,24 @@ harness_selftest	570-a-checkout-test-must-ask-git	premise: the suite defines the
 harness_selftest	570-a-checkout-test-must-ask-git	premise: the sweep sees .git mentioned in the suites at all	15;16;17;18;19	never	-
 harness_selftest	570-a-checkout-test-must-ask-git	the sweep matches every spelling that stats .git as a directory	15;16;17;18;19	2026-09-23	the same planted line, read by the pattern shipped in #1224;the sweep pattern shipped in #1224 is restored verbatim
 harness_selftest	570-a-checkout-test-must-ask-git	without the trailing boundary the same pattern flags three of them	15;16;17;18;19	2026-09-23	the boundary in the sweep pattern drifts, so the strip becomes a no-op;the derived control stops stripping the boundary
+harness_selftest	580-a-hand-rebuild-must-record	a log with content produces a diagnosis to print	15;16;17;18;19	2026-09-24	rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason
+harness_selftest	580-a-hand-rebuild-must-record	a rebuild that failed skips them rather than reading a stamp it did not write	15;16;17;18;19	2026-09-24	the gate reverted so a failed rebuild still runs the arms beneath it; driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them
+harness_selftest	580-a-hand-rebuild-must-record	a rebuild that succeeded runs the arms that read its stamp	15;16;17;18;19	2026-09-24	the gate always skips, so a successful rebuild runs none of the arms that read its stamp
+harness_selftest	580-a-hand-rebuild-must-record	an empty log produces nothing rather than an empty banner	15;16;17;18;19	2026-09-24	_hr_diagnose banners unconditionally, so an empty or absent log still prints
+harness_selftest	580-a-hand-rebuild-must-record	and a log that is not there produces nothing rather than an error	15;16;17;18;19	2026-09-24	_hr_diagnose banners unconditionally, so an empty or absent log still prints
+harness_selftest	580-a-hand-rebuild-must-record	and any other status skips too, rather than being read as success	15;16;17;18;19	2026-09-24	the gate reverted so a failed rebuild still runs the arms beneath it; driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them
 harness_selftest	580-a-hand-rebuild-must-record	and it reads a different source as stale rather than fresh	15;16;17;18;19	2026-09-23	the record call removed from rebuild.sh, so no stamp is written at all
+harness_selftest	580-a-hand-rebuild-must-record	and it writes a reason, so there is something for the gate to print	15;16;17;18;19	2026-09-24	rebuild.sh exits without saying why, so the gate has nothing to print
+harness_selftest	580-a-hand-rebuild-must-record	and the diagnosis carries the failing line, not just a status	15;16;17;18;19	2026-09-24	rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason
 harness_selftest	580-a-hand-rebuild-must-record	and the freshness gate then reads the tree as built from this source	15;16;17;18;19	2026-09-23	the record call removed from rebuild.sh, so no stamp is written at all
 harness_selftest	580-a-hand-rebuild-must-record	and the installed library of the running major was never touched	15;16;17;18;19	2026-09-23	the shim answers --pkglibdir with the real prefix, so the install is not redirected
 harness_selftest	580-a-hand-rebuild-must-record	and writes no stamp into the directory it was called from	15;16;17;18;19	2026-09-23	the argument guard removed from pgc_record_source_stamp
+harness_selftest	580-a-hand-rebuild-must-record	premise: a rebuild with no pg_config fails	15;16;17;18;19	never	-
 harness_selftest	580-a-hand-rebuild-must-record	premise: lib.sh has one recorder for what was installed	15;16;17;18;19	2026-09-23	pgc_record_source_stamp removed from lib.sh
 harness_selftest	580-a-hand-rebuild-must-record	premise: no stamp for this prefix before the rebuild	15;16;17;18;19	2026-09-23	the shim answers --pkglibdir with the real prefix, so the install is not redirected
-harness_selftest	580-a-hand-rebuild-must-record	premise: the hand rebuild itself succeeded	15;16;17;18;19	never	-
+harness_selftest	580-a-hand-rebuild-must-record	premise: the diagnosis is exposed too	15;16;17;18;19	never	-
+harness_selftest	580-a-hand-rebuild-must-record	premise: the gating decision is exposed to be judged	15;16;17;18;19	never	-
+harness_selftest	580-a-hand-rebuild-must-record	premise: the hand rebuild itself succeeded	15;16;17;18;19	2026-09-24	rebuild.sh forced to exit 1 after writing its stamp, reproducing the aarch64 shape;rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason;rebuild.sh forced to fail after its stamp AND the gate reverted, so a failed rebuild still runs the arms beneath it
 harness_selftest	580-a-hand-rebuild-must-record	premise: the hand rebuild tool installs, so a stale record is possible	15;16;17;18;19	never	-
 harness_selftest	580-a-hand-rebuild-must-record	premise: the shim redirects the install away from the real prefix	15;16;17;18;19	2026-09-23	the shim answers --pkglibdir with the real prefix, so the install is not redirected
 harness_selftest	580-a-hand-rebuild-must-record	the hand rebuild tool records what it installed rather than only naming it	15;16;17;18;19	2026-09-23	the record call removed from rebuild.sh, or the comment stripping removed

test/check_ledger_budget.txt

@@ -799,4 +799,40 @@ suites_not_covered 249
 # each. Counted on both trees rather than reasoned: registered == covered +
 # not_covered holds on each, and the only name in the difference either way is
 # catalog_delete_index.
-checks_never_observed_red 1530
+#
+# 1530 -> 1532 for #1248, composed onto the main that carries #1213.
+# RE-DERIVED BY COUNTING, decomposed by population:
+#
+#     awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l   ->  1532
+#     1673 rows total, 141 not `never`,  1532 + 141 == 1673
+#
+#     11 new rows: 3 arrive `never`, 8 arrive dated
+#      1 pre-existing row flips never -> dated
+#     1530 - 1 + 3 == 1532
+#
+# THIS BRANCH HAS CARRIED 1508 AND NOW 1532, and only the second describes a tree
+# that exists. BOTH figures were published before this merge existed -- mine and
+# @jdatcmd's, derived from different sides -- and both said 1532. A recount
+# compared only against itself is the derived-term problem: it always looks
+# consistent. Two predictions made independently and one count is not.
+#
+# THE ROWS AUTO-MERGED AND ONLY THIS LINE CONFLICTED, which is the right shape:
+# the rows are all true facts about checks and compose silently; the total is
+# derived and must not. Checked per part, because a total cannot see one row
+# lost and another duplicated: catalog_delete_index 30 rows from #1213, part 580
+# 22 rows from here, zero duplicate keys.
+#
+# SIX MUTATIONS, and the part has zero non-premise arms never observed red.
+# A, B and C force rebuild.sh to exit 1 AFTER it writes its stamp, which is the
+# aarch64 shape exactly; D, E and F leave rebuild.sh alone and attack the
+# healthy path. B reproduces the ORIGINAL defect beside the fix: with the gate
+# reverted the dependents PASS despite the failed premise.
+#
+# The two gate arms carry the SINGULAR gate mutation rather than the compound.
+# @jdatcmd drove the revert alone and got exactly those two: they call the gate
+# with the literals 1 and "", so the rebuild's status never reaches them and the
+# forced failure is inert for them.
+#
+# The three rows arriving `never` are premises: two `type -t` exposure checks
+# and one asserting that a rebuild with no pg_config fails.
+checks_never_observed_red 1532

test/selftest/580-a-hand-rebuild-must-record.sh

@@ -112,22 +112,107 @@ _hr_stamp="$(pgc_source_stamp_path "$_hr_tmp/tree" "$_hr_tmp/pg_config")"
 check_text "premise: no stamp for this prefix before the rebuild" \
 	"$(if [ -e "$_hr_stamp" ]; then echo present; else echo absent; fi)" "absent"
 
+# BOTH DECISIONS ARE FUNCTIONS so they can be driven from here rather than only
+# by a box that happens to break (#1248). The branch that matters is the one a
+# healthy machine never takes, which is exactly the branch that had never run
+# until aarch64 took it.
+_hr_dependents() {	# _hr_dependents RC -> run|skip
+	[ "${1:-}" = 0 ] && echo run || echo skip
+}
+_hr_diagnose() {	# _hr_diagnose LOGFILE -> the tail, indented, or nothing
+	[ -s "${1:-}" ] || return 0
+	echo "      ---- rebuild.log, last 20 lines ----"
+	tail -20 "$1" | sed 's/^/      /'
+}
+
 "$PGC_TESTDIR/rebuild.sh" "$_hr_tmp/pg_config" "$_hr_tmp/tree" >"$_hr_tmp/rebuild.log" 2>&1
 _hr_rc=$?
 check_num "premise: the hand rebuild itself succeeded" "$_hr_rc" "0"
 
-check_text "and the freshness gate then reads the tree as built from this source" \
-	"$(pgc_freshness_verdict \
-		"$(head -1 "$_hr_stamp" 2>/dev/null)" \
-		"$(pgc_source_fingerprint "$_hr_tmp/tree")")" "fresh"
+# ---- the premise must GATE, and the failure must SAY SOMETHING (#1248) ------
+#
+# The 2026-09-24 nightly failed here on aarch64 with `got [1] want [0]`, and that
+# was the entire diagnosis: the rebuild's output is captured to a file this part
+# deletes on the way out, so the cause died with the workdir. The three arms
+# below the premise then ran anyway and reported PASS.
+#
+# They were not vacuous that night, because rebuild.sh had in fact written its
+# stamps before failing -- it writes them at step 3b and the symbol check that
+# rejected the build comes after. That is luck rather than design. Had the BUILD
+# failed, the same three arms would have read an absent stamp and reported on a
+# rebuild that never happened.
+#
+# So both decisions are exposed as functions and judged, the way part 190 judges
+# pgc_build_needs_clean, rather than being written inline where nothing can drive
+# the branch that only a broken box takes.
+check_text "premise: the gating decision is exposed to be judged" \
+	"$(type -t _hr_dependents)" "function"
+check_text "premise: the diagnosis is exposed too" \
+	"$(type -t _hr_diagnose)" "function"
+
+check_text "a rebuild that succeeded runs the arms that read its stamp" \
+	"$(_hr_dependents 0)" "run"
+check_text "a rebuild that failed skips them rather than reading a stamp it did not write" \
+	"$(_hr_dependents 1)" "skip"
+check_text "and any other status skips too, rather than being read as success" \
+	"$(_hr_dependents "")" "skip"
+
+_hr_dtmp="$(mktemp -d)"
+printf 'rebuild: UNRESOLVED SYMBOLS against PostgreSQL 18\n__aarch64_ldadd4_acq_rel\n' \
+	> "$_hr_dtmp/full.log"
+: > "$_hr_dtmp/empty.log"
+check_num "a log with content produces a diagnosis to print" \
+	"$([ -n "$(_hr_diagnose "$_hr_dtmp/full.log")" ] && echo 1 || echo 0)" "1"
+check_num "and the diagnosis carries the failing line, not just a status" \
+	"$(_hr_diagnose "$_hr_dtmp/full.log" | grep -c 'UNRESOLVED SYMBOLS')" "1"
+check_num "an empty log produces nothing rather than an empty banner" \
+	"$([ -n "$(_hr_diagnose "$_hr_dtmp/empty.log")" ] && echo 1 || echo 0)" "0"
+check_num "and a log that is not there produces nothing rather than an error" \
+	"$([ -n "$(_hr_diagnose "$_hr_dtmp/nope.log" 2>/dev/null)" ] && echo 1 || echo 0)" "0"
+rm -rf "$_hr_dtmp"
+
+# AND THE REAL SCRIPT MUST LEAVE SOMETHING TO DIAGNOSE. A gate that prints a log
+# is worth nothing if the log is empty on the path that matters, so this drives
+# rebuild.sh's cheapest failure -- a pg_config that is not there -- and asserts it
+# wrote a reason rather than only exiting non-zero.
+_hr_fail_log="$(mktemp)"
+"$PGC_TESTDIR/rebuild.sh" /nonexistent/pg_config "$_hr_tmp/tree" >"$_hr_fail_log" 2>&1
+_hr_fail_rc=$?
+check_num "premise: a rebuild with no pg_config fails" \
+	"$([ "$_hr_fail_rc" != 0 ] && echo 1 || echo 0)" "1"
+check_num "and it writes a reason, so there is something for the gate to print" \
+	"$(grep -c '^rebuild: no such pg_config' "$_hr_fail_log")" "1"
+rm -f "$_hr_fail_log"
+
+if [ "$(_hr_dependents "$_hr_rc")" = run ]; then
+	check_text "and the freshness gate then reads the tree as built from this source" \
+		"$(pgc_freshness_verdict \
+			"$(head -1 "$_hr_stamp" 2>/dev/null)" \
+			"$(pgc_source_fingerprint "$_hr_tmp/tree")")" "fresh"
 
-# THE SAME READ AGAINST A FINGERPRINT THAT IS NOT THIS TREE'S, so the arm above
-# is known to distinguish rather than to answer "fresh" whatever it is given.
-check_text "and it reads a different source as stale rather than fresh" \
-	"$(pgc_freshness_verdict "$(head -1 "$_hr_stamp" 2>/dev/null)" deadbeefdead)" "stale"
+	# THE SAME READ AGAINST A FINGERPRINT THAT IS NOT THIS TREE'S, so the arm
+	# above is known to distinguish rather than to answer "fresh" whatever it is
+	# given.
+	check_text "and it reads a different source as stale rather than fresh" \
+		"$(pgc_freshness_verdict "$(head -1 "$_hr_stamp" 2>/dev/null)" deadbeefdead)" "stale"
 
-check_text "and the installed library of the running major was never touched" \
-	"$(if [ "$(stat -c %Y "$_hr_live_so" 2>/dev/null)" = "$_hr_live_before" ];
-	then echo untouched; else echo overwritten; fi)" "untouched"
+	check_text "and the installed library of the running major was never touched" \
+		"$(if [ "$(stat -c %Y "$_hr_live_so" 2>/dev/null)" = "$_hr_live_before" ];
+		then echo untouched; else echo overwritten; fi)" "untouched"
+else
+	# THE ONLY DIAGNOSIS THERE IS. rebuild.sh's output goes to a file this part
+	# deletes on the way out, so without this the run reports `got [1] want [0]`
+	# and the cause leaves with the workdir -- which is how the aarch64 failure
+	# arrived undiagnosable (#1248).
+	_hr_diagnose "$_hr_tmp/rebuild.log"
+	for _hr_arm in \
+		"and the freshness gate then reads the tree as built from this source" \
+		"and it reads a different source as stale rather than fresh" \
+		"and the installed library of the running major was never touched"; do
+		check_skip "$_hr_arm" \
+			"SKIP  $_hr_arm (the hand rebuild failed, so there is no stamp to read)" \
+			"the hand rebuild failed"
+	done
+fi
 
 rm -rf "$_hr_tmp"