From a348e2f26192ceff1f34f9f9145a7b342ff33564 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Thu, 24 Sep 2026 08:21:30 -0600 Subject: [PATCH 1/2] test: gate the hand-rebuild arms and say why it failed (#1248) The 2026-09-24 nightly failed on aarch64 with the whole diagnosis being FAIL premise: the hand rebuild itself succeeded: got [1] want [0] and nothing else. rebuild.sh's output goes to a file this part deletes on the way out, so the cause left with the workdir. The three arms below that premise then ran anyway and reported PASS. They were not vacuous that night, and only by luck: rebuild.sh writes both stamps at step 3b and the unresolved-symbol check that rejected the build comes after, so the stamps those arms read had in fact been written. Had the BUILD failed, the same three arms would have read an absent stamp and reported on a rebuild that never happened. Both decisions are now functions the part drives directly, the way part 190 drives pgc_build_needs_clean, rather than inline branches that only a broken machine exercises: _hr_dependents RC -> run | skip _hr_diagnose LOG -> the last 20 lines, indented, or nothing Six mutations, split so attribution stays separable. A, B and C force rebuild.sh to exit 1 AFTER its stamp, reproducing the aarch64 shape; D, E and F leave rebuild.sh alone and attack the healthy path. B reproduces the original defect beside the fix: with the gate reverted the dependents PASS despite the failed premise. The part now has zero non-premise arms never observed red. This does NOT fix the aarch64 failure and does not guess at it. The hypothesis is GCC's outline-atomic helpers resolving from libgcc and falling outside the ldd reference set the symbol check builds, and there is no aarch64 box here to test it on. What this does is make the next nightly report the symbol names instead of a bare exit status, so the fix after it is measured. Census re-derived by counting: 1643 rows, 135 not never, 1508 + 135 == 1643. The ceiling RISES, 1506 -> 1508, which is the honest direction for eleven arms that did not exist before. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP --- CHANGELOG.md | 46 ++++++++ test/check_ledger.tsv | 13 ++- test/check_ledger_budget.txt | 38 ++++++- .../580-a-hand-rebuild-must-record.sh | 107 ++++++++++++++++-- 4 files changed, 191 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fce0c0f..e22a8099 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -458,6 +458,52 @@ true until the next version shipped. ### Fixed +- A failed hand rebuild now says why, and stops the arms beneath it reporting on + a rebuild that did not happen (#1248). + + The 2026-09-24 nightly failed on aarch64 with the whole diagnosis being + + ``` + FAIL premise: the hand rebuild itself succeeded: got [1] want [0] + ``` + + and nothing else. `rebuild.sh`'s output goes to a file the part deletes on the + way out, so the cause left with the workdir. **The three arms below that + premise then ran anyway and reported PASS.** + + They were not vacuous that night, and only by luck: `rebuild.sh` writes both + stamps at step 3b and the unresolved-symbol check that rejected the build comes + after, so the stamps the three arms read had in fact been written. Had the + *build* failed, the same three arms would have read an absent stamp. That is + what "the premise does not gate" costs. + + Both decisions are now functions the part drives directly, the way part 190 + drives `pgc_build_needs_clean`, rather than inline branches that only a broken + machine exercises: + + ``` + _hr_dependents RC -> run | skip + _hr_diagnose LOG -> the last 20 lines, indented, or nothing + ``` + + Driven with `rebuild.sh` forced to exit 1 *after* its stamp, reproducing the + aarch64 shape exactly: + + ``` + FAIL premise: the hand rebuild itself succeeded + SKIP and the freshness gate then reads the tree as built from this source + SKIP and it reads a different source as stale rather than fresh + SKIP and the installed library of the running major was never touched + the diagnosis is printed, and carries the failing line rather than a status + ``` + + **This does not fix the aarch64 failure and does not guess at it.** The + hypothesis is GCC's outline-atomic helpers (`__aarch64_*`) resolving from + libgcc and falling outside the `ldd` reference set the symbol check builds, and + there is no aarch64 box here to test it on. What this change does is make the + next nightly report the symbol names instead of a bare exit status, so the fix + after it is measured rather than reasoned about. + - `pgc_reconcile_records` no longer reports an impossible mismatch on a log it cannot measure (#1242). Handed a log that states `checks run: N` and carries no `RESULT` records at all, it ran the arithmetic anyway and concluded that diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index d47661d1..0ead0f99 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1409,13 +1409,24 @@ harness_selftest 570-a-checkout-test-must-ask-git premise: the suite defines the harness_selftest 570-a-checkout-test-must-ask-git premise: the sweep sees .git mentioned in the suites at all 15;16;17;18;19 never - harness_selftest 570-a-checkout-test-must-ask-git the sweep matches every spelling that stats .git as a directory 15;16;17;18;19 2026-09-23 the same planted line, read by the pattern shipped in #1224;the sweep pattern shipped in #1224 is restored verbatim harness_selftest 570-a-checkout-test-must-ask-git without the trailing boundary the same pattern flags three of them 15;16;17;18;19 2026-09-23 the boundary in the sweep pattern drifts, so the strip becomes a no-op;the derived control stops stripping the boundary +harness_selftest 580-a-hand-rebuild-must-record a log with content produces a diagnosis to print 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason +harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND the gate reverted, so a failed rebuild still runs the arms beneath it +harness_selftest 580-a-hand-rebuild-must-record a rebuild that succeeded runs the arms that read its stamp 15;16;17;18;19 2026-09-24 the gate always skips, so a successful rebuild runs none of the arms that read its stamp +harness_selftest 580-a-hand-rebuild-must-record an empty log produces nothing rather than an empty banner 15;16;17;18;19 2026-09-24 _hr_diagnose banners unconditionally, so an empty or absent log still prints +harness_selftest 580-a-hand-rebuild-must-record and a log that is not there produces nothing rather than an error 15;16;17;18;19 2026-09-24 _hr_diagnose banners unconditionally, so an empty or absent log still prints +harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND the gate reverted, so a failed rebuild still runs the arms beneath it harness_selftest 580-a-hand-rebuild-must-record and it reads a different source as stale rather than fresh 15;16;17;18;19 2026-09-23 the record call removed from rebuild.sh, so no stamp is written at all +harness_selftest 580-a-hand-rebuild-must-record and it writes a reason, so there is something for the gate to print 15;16;17;18;19 2026-09-24 rebuild.sh exits without saying why, so the gate has nothing to print +harness_selftest 580-a-hand-rebuild-must-record and the diagnosis carries the failing line, not just a status 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason harness_selftest 580-a-hand-rebuild-must-record and the freshness gate then reads the tree as built from this source 15;16;17;18;19 2026-09-23 the record call removed from rebuild.sh, so no stamp is written at all harness_selftest 580-a-hand-rebuild-must-record and the installed library of the running major was never touched 15;16;17;18;19 2026-09-23 the shim answers --pkglibdir with the real prefix, so the install is not redirected harness_selftest 580-a-hand-rebuild-must-record and writes no stamp into the directory it was called from 15;16;17;18;19 2026-09-23 the argument guard removed from pgc_record_source_stamp +harness_selftest 580-a-hand-rebuild-must-record premise: a rebuild with no pg_config fails 15;16;17;18;19 never - harness_selftest 580-a-hand-rebuild-must-record premise: lib.sh has one recorder for what was installed 15;16;17;18;19 2026-09-23 pgc_record_source_stamp removed from lib.sh harness_selftest 580-a-hand-rebuild-must-record premise: no stamp for this prefix before the rebuild 15;16;17;18;19 2026-09-23 the shim answers --pkglibdir with the real prefix, so the install is not redirected -harness_selftest 580-a-hand-rebuild-must-record premise: the hand rebuild itself succeeded 15;16;17;18;19 never - +harness_selftest 580-a-hand-rebuild-must-record premise: the diagnosis is exposed too 15;16;17;18;19 never - +harness_selftest 580-a-hand-rebuild-must-record premise: the gating decision is exposed to be judged 15;16;17;18;19 never - +harness_selftest 580-a-hand-rebuild-must-record premise: the hand rebuild itself succeeded 15;16;17;18;19 2026-09-24 rebuild.sh forced to exit 1 after writing its stamp, reproducing the aarch64 shape;rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason;rebuild.sh forced to fail after its stamp AND the gate reverted, so a failed rebuild still runs the arms beneath it harness_selftest 580-a-hand-rebuild-must-record premise: the hand rebuild tool installs, so a stale record is possible 15;16;17;18;19 never - harness_selftest 580-a-hand-rebuild-must-record premise: the shim redirects the install away from the real prefix 15;16;17;18;19 2026-09-23 the shim answers --pkglibdir with the real prefix, so the install is not redirected harness_selftest 580-a-hand-rebuild-must-record the hand rebuild tool records what it installed rather than only naming it 15;16;17;18;19 2026-09-23 the record call removed from rebuild.sh, or the comment stripping removed diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 4d564a26..c5d24d38 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -799,4 +799,40 @@ suites_not_covered 249 # each. Counted on both trees rather than reasoned: registered == covered + # not_covered holds on each, and the only name in the difference either way is # catalog_delete_index. -checks_never_observed_red 1530 +# +# 1530 -> 1532 for #1248, composed onto the main that carries #1213. +# RE-DERIVED BY COUNTING, decomposed by population: +# +# awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l -> 1532 +# 1673 rows total, 141 not `never`, 1532 + 141 == 1673 +# +# 11 new rows: 3 arrive `never`, 8 arrive dated +# 1 pre-existing row flips never -> dated +# 1530 - 1 + 3 == 1532 +# +# THIS BRANCH HAS CARRIED 1508 AND NOW 1532, and only the second describes a tree +# that exists. BOTH figures were published before this merge existed -- mine and +# @jdatcmd's, derived from different sides -- and both said 1532. A recount +# compared only against itself is the derived-term problem: it always looks +# consistent. Two predictions made independently and one count is not. +# +# THE ROWS AUTO-MERGED AND ONLY THIS LINE CONFLICTED, which is the right shape: +# the rows are all true facts about checks and compose silently; the total is +# derived and must not. Checked per part, because a total cannot see one row +# lost and another duplicated: catalog_delete_index 30 rows from #1213, part 580 +# 22 rows from here, zero duplicate keys. +# +# SIX MUTATIONS, and the part has zero non-premise arms never observed red. +# A, B and C force rebuild.sh to exit 1 AFTER it writes its stamp, which is the +# aarch64 shape exactly; D, E and F leave rebuild.sh alone and attack the +# healthy path. B reproduces the ORIGINAL defect beside the fix: with the gate +# reverted the dependents PASS despite the failed premise. +# +# The two gate arms carry the SINGULAR gate mutation rather than the compound. +# @jdatcmd drove the revert alone and got exactly those two: they call the gate +# with the literals 1 and "", so the rebuild's status never reaches them and the +# forced failure is inert for them. +# +# The three rows arriving `never` are premises: two `type -t` exposure checks +# and one asserting that a rebuild with no pg_config fails. +checks_never_observed_red 1532 diff --git a/test/selftest/580-a-hand-rebuild-must-record.sh b/test/selftest/580-a-hand-rebuild-must-record.sh index b05b2e64..1d28fb8e 100644 --- a/test/selftest/580-a-hand-rebuild-must-record.sh +++ b/test/selftest/580-a-hand-rebuild-must-record.sh @@ -112,22 +112,107 @@ _hr_stamp="$(pgc_source_stamp_path "$_hr_tmp/tree" "$_hr_tmp/pg_config")" check_text "premise: no stamp for this prefix before the rebuild" \ "$(if [ -e "$_hr_stamp" ]; then echo present; else echo absent; fi)" "absent" +# BOTH DECISIONS ARE FUNCTIONS so they can be driven from here rather than only +# by a box that happens to break (#1248). The branch that matters is the one a +# healthy machine never takes, which is exactly the branch that had never run +# until aarch64 took it. +_hr_dependents() { # _hr_dependents RC -> run|skip + [ "${1:-}" = 0 ] && echo run || echo skip +} +_hr_diagnose() { # _hr_diagnose LOGFILE -> the tail, indented, or nothing + [ -s "${1:-}" ] || return 0 + echo " ---- rebuild.log, last 20 lines ----" + tail -20 "$1" | sed 's/^/ /' +} + "$PGC_TESTDIR/rebuild.sh" "$_hr_tmp/pg_config" "$_hr_tmp/tree" >"$_hr_tmp/rebuild.log" 2>&1 _hr_rc=$? check_num "premise: the hand rebuild itself succeeded" "$_hr_rc" "0" -check_text "and the freshness gate then reads the tree as built from this source" \ - "$(pgc_freshness_verdict \ - "$(head -1 "$_hr_stamp" 2>/dev/null)" \ - "$(pgc_source_fingerprint "$_hr_tmp/tree")")" "fresh" +# ---- the premise must GATE, and the failure must SAY SOMETHING (#1248) ------ +# +# The 2026-09-24 nightly failed here on aarch64 with `got [1] want [0]`, and that +# was the entire diagnosis: the rebuild's output is captured to a file this part +# deletes on the way out, so the cause died with the workdir. The three arms +# below the premise then ran anyway and reported PASS. +# +# They were not vacuous that night, because rebuild.sh had in fact written its +# stamps before failing -- it writes them at step 3b and the symbol check that +# rejected the build comes after. That is luck rather than design. Had the BUILD +# failed, the same three arms would have read an absent stamp and reported on a +# rebuild that never happened. +# +# So both decisions are exposed as functions and judged, the way part 190 judges +# pgc_build_needs_clean, rather than being written inline where nothing can drive +# the branch that only a broken box takes. +check_text "premise: the gating decision is exposed to be judged" \ + "$(type -t _hr_dependents)" "function" +check_text "premise: the diagnosis is exposed too" \ + "$(type -t _hr_diagnose)" "function" + +check_text "a rebuild that succeeded runs the arms that read its stamp" \ + "$(_hr_dependents 0)" "run" +check_text "a rebuild that failed skips them rather than reading a stamp it did not write" \ + "$(_hr_dependents 1)" "skip" +check_text "and any other status skips too, rather than being read as success" \ + "$(_hr_dependents "")" "skip" + +_hr_dtmp="$(mktemp -d)" +printf 'rebuild: UNRESOLVED SYMBOLS against PostgreSQL 18\n__aarch64_ldadd4_acq_rel\n' \ + > "$_hr_dtmp/full.log" +: > "$_hr_dtmp/empty.log" +check_num "a log with content produces a diagnosis to print" \ + "$([ -n "$(_hr_diagnose "$_hr_dtmp/full.log")" ] && echo 1 || echo 0)" "1" +check_num "and the diagnosis carries the failing line, not just a status" \ + "$(_hr_diagnose "$_hr_dtmp/full.log" | grep -c 'UNRESOLVED SYMBOLS')" "1" +check_num "an empty log produces nothing rather than an empty banner" \ + "$([ -n "$(_hr_diagnose "$_hr_dtmp/empty.log")" ] && echo 1 || echo 0)" "0" +check_num "and a log that is not there produces nothing rather than an error" \ + "$([ -n "$(_hr_diagnose "$_hr_dtmp/nope.log" 2>/dev/null)" ] && echo 1 || echo 0)" "0" +rm -rf "$_hr_dtmp" + +# AND THE REAL SCRIPT MUST LEAVE SOMETHING TO DIAGNOSE. A gate that prints a log +# is worth nothing if the log is empty on the path that matters, so this drives +# rebuild.sh's cheapest failure -- a pg_config that is not there -- and asserts it +# wrote a reason rather than only exiting non-zero. +_hr_fail_log="$(mktemp)" +"$PGC_TESTDIR/rebuild.sh" /nonexistent/pg_config "$_hr_tmp/tree" >"$_hr_fail_log" 2>&1 +_hr_fail_rc=$? +check_num "premise: a rebuild with no pg_config fails" \ + "$([ "$_hr_fail_rc" != 0 ] && echo 1 || echo 0)" "1" +check_num "and it writes a reason, so there is something for the gate to print" \ + "$(grep -c '^rebuild: no such pg_config' "$_hr_fail_log")" "1" +rm -f "$_hr_fail_log" + +if [ "$(_hr_dependents "$_hr_rc")" = run ]; then + check_text "and the freshness gate then reads the tree as built from this source" \ + "$(pgc_freshness_verdict \ + "$(head -1 "$_hr_stamp" 2>/dev/null)" \ + "$(pgc_source_fingerprint "$_hr_tmp/tree")")" "fresh" -# THE SAME READ AGAINST A FINGERPRINT THAT IS NOT THIS TREE'S, so the arm above -# is known to distinguish rather than to answer "fresh" whatever it is given. -check_text "and it reads a different source as stale rather than fresh" \ - "$(pgc_freshness_verdict "$(head -1 "$_hr_stamp" 2>/dev/null)" deadbeefdead)" "stale" + # THE SAME READ AGAINST A FINGERPRINT THAT IS NOT THIS TREE'S, so the arm + # above is known to distinguish rather than to answer "fresh" whatever it is + # given. + check_text "and it reads a different source as stale rather than fresh" \ + "$(pgc_freshness_verdict "$(head -1 "$_hr_stamp" 2>/dev/null)" deadbeefdead)" "stale" -check_text "and the installed library of the running major was never touched" \ - "$(if [ "$(stat -c %Y "$_hr_live_so" 2>/dev/null)" = "$_hr_live_before" ]; - then echo untouched; else echo overwritten; fi)" "untouched" + check_text "and the installed library of the running major was never touched" \ + "$(if [ "$(stat -c %Y "$_hr_live_so" 2>/dev/null)" = "$_hr_live_before" ]; + then echo untouched; else echo overwritten; fi)" "untouched" +else + # THE ONLY DIAGNOSIS THERE IS. rebuild.sh's output goes to a file this part + # deletes on the way out, so without this the run reports `got [1] want [0]` + # and the cause leaves with the workdir -- which is how the aarch64 failure + # arrived undiagnosable (#1248). + _hr_diagnose "$_hr_tmp/rebuild.log" + for _hr_arm in \ + "and the freshness gate then reads the tree as built from this source" \ + "and it reads a different source as stale rather than fresh" \ + "and the installed library of the running major was never touched"; do + check_skip "$_hr_arm" \ + "SKIP $_hr_arm (the hand rebuild failed, so there is no stamp to read)" \ + "the hand rebuild failed" + done +fi rm -rf "$_hr_tmp" From 8359ff028ba947877dbbdb512a0219b57a22138f Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Thu, 24 Sep 2026 09:24:38 -0600 Subject: [PATCH 2/2] test: decompose the ceiling by population, and narrow mutation B (#1248) Two corrections from @jdatcmd's review, neither changing a number. THE CEILING SENTENCE conflated two populations. It said "1506 + 11 new - 9 newly dated == 1508", and nine is eight new arrivals plus one pre-existing flip. Read the only way it can be read -- as nine pre-existing flips -- it gives 1516, and the reviewer had to decompose it to find the agreement. It now decomposes: 11 new rows of which 3 arrive `never` and 8 arrive dated, plus 1 pre-existing row flipping, so 1506 - 1 + 3 == 1508. Verified by counting the added and modified rows separately. MUTATION B'S ATTRIBUTION overstated itself for two rows. B is compound -- forced rebuild failure AND the gate reverted -- but @jdatcmd drove the gate revert ALONE and got exactly the two gate arms and nothing else. Those two call the gate with the literals 1 and "", so the rebuild's status never reaches them: for them the forced failure is inert. Their rows now carry the singular mutation. B keeps the original-defect demonstration, which is the one thing only a compound can show. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP --- test/check_ledger.tsv | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 0ead0f99..3e491229 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1410,11 +1410,11 @@ harness_selftest 570-a-checkout-test-must-ask-git premise: the sweep sees .git m harness_selftest 570-a-checkout-test-must-ask-git the sweep matches every spelling that stats .git as a directory 15;16;17;18;19 2026-09-23 the same planted line, read by the pattern shipped in #1224;the sweep pattern shipped in #1224 is restored verbatim harness_selftest 570-a-checkout-test-must-ask-git without the trailing boundary the same pattern flags three of them 15;16;17;18;19 2026-09-23 the boundary in the sweep pattern drifts, so the strip becomes a no-op;the derived control stops stripping the boundary harness_selftest 580-a-hand-rebuild-must-record a log with content produces a diagnosis to print 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason -harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND the gate reverted, so a failed rebuild still runs the arms beneath it +harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it; driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them harness_selftest 580-a-hand-rebuild-must-record a rebuild that succeeded runs the arms that read its stamp 15;16;17;18;19 2026-09-24 the gate always skips, so a successful rebuild runs none of the arms that read its stamp harness_selftest 580-a-hand-rebuild-must-record an empty log produces nothing rather than an empty banner 15;16;17;18;19 2026-09-24 _hr_diagnose banners unconditionally, so an empty or absent log still prints harness_selftest 580-a-hand-rebuild-must-record and a log that is not there produces nothing rather than an error 15;16;17;18;19 2026-09-24 _hr_diagnose banners unconditionally, so an empty or absent log still prints -harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND the gate reverted, so a failed rebuild still runs the arms beneath it +harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it; driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them harness_selftest 580-a-hand-rebuild-must-record and it reads a different source as stale rather than fresh 15;16;17;18;19 2026-09-23 the record call removed from rebuild.sh, so no stamp is written at all harness_selftest 580-a-hand-rebuild-must-record and it writes a reason, so there is something for the gate to print 15;16;17;18;19 2026-09-24 rebuild.sh exits without saying why, so the gate has nothing to print harness_selftest 580-a-hand-rebuild-must-record and the diagnosis carries the failing line, not just a status 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason