diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fce0c0f..e22a8099 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -458,6 +458,52 @@ true until the next version shipped. ### Fixed +- A failed hand rebuild now says why, and stops the arms beneath it reporting on + a rebuild that did not happen (#1248). + + The 2026-09-24 nightly failed on aarch64 with the whole diagnosis being + + ``` + FAIL premise: the hand rebuild itself succeeded: got [1] want [0] + ``` + + and nothing else. `rebuild.sh`'s output goes to a file the part deletes on the + way out, so the cause left with the workdir. **The three arms below that + premise then ran anyway and reported PASS.** + + They were not vacuous that night, and only by luck: `rebuild.sh` writes both + stamps at step 3b and the unresolved-symbol check that rejected the build comes + after, so the stamps the three arms read had in fact been written. Had the + *build* failed, the same three arms would have read an absent stamp. That is + what "the premise does not gate" costs. + + Both decisions are now functions the part drives directly, the way part 190 + drives `pgc_build_needs_clean`, rather than inline branches that only a broken + machine exercises: + + ``` + _hr_dependents RC -> run | skip + _hr_diagnose LOG -> the last 20 lines, indented, or nothing + ``` + + Driven with `rebuild.sh` forced to exit 1 *after* its stamp, reproducing the + aarch64 shape exactly: + + ``` + FAIL premise: the hand rebuild itself succeeded + SKIP and the freshness gate then reads the tree as built from this source + SKIP and it reads a different source as stale rather than fresh + SKIP and the installed library of the running major was never touched + the diagnosis is printed, and carries the failing line rather than a status + ``` + + **This does not fix the aarch64 failure and does not guess at it.** The + hypothesis is GCC's outline-atomic helpers (`__aarch64_*`) resolving from + libgcc and falling outside the `ldd` reference set the symbol check builds, and + there is no aarch64 box here to test it on. What this change does is make the + next nightly report the symbol names instead of a bare exit status, so the fix + after it is measured rather than reasoned about. + - `pgc_reconcile_records` no longer reports an impossible mismatch on a log it cannot measure (#1242). Handed a log that states `checks run: N` and carries no `RESULT` records at all, it ran the arithmetic anyway and concluded that diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index d47661d1..3e491229 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1409,13 +1409,24 @@ harness_selftest 570-a-checkout-test-must-ask-git premise: the suite defines the harness_selftest 570-a-checkout-test-must-ask-git premise: the sweep sees .git mentioned in the suites at all 15;16;17;18;19 never - harness_selftest 570-a-checkout-test-must-ask-git the sweep matches every spelling that stats .git as a directory 15;16;17;18;19 2026-09-23 the same planted line, read by the pattern shipped in #1224;the sweep pattern shipped in #1224 is restored verbatim harness_selftest 570-a-checkout-test-must-ask-git without the trailing boundary the same pattern flags three of them 15;16;17;18;19 2026-09-23 the boundary in the sweep pattern drifts, so the strip becomes a no-op;the derived control stops stripping the boundary +harness_selftest 580-a-hand-rebuild-must-record a log with content produces a diagnosis to print 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason +harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it; driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them +harness_selftest 580-a-hand-rebuild-must-record a rebuild that succeeded runs the arms that read its stamp 15;16;17;18;19 2026-09-24 the gate always skips, so a successful rebuild runs none of the arms that read its stamp +harness_selftest 580-a-hand-rebuild-must-record an empty log produces nothing rather than an empty banner 15;16;17;18;19 2026-09-24 _hr_diagnose banners unconditionally, so an empty or absent log still prints +harness_selftest 580-a-hand-rebuild-must-record and a log that is not there produces nothing rather than an error 15;16;17;18;19 2026-09-24 _hr_diagnose banners unconditionally, so an empty or absent log still prints +harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 the gate reverted so a failed rebuild still runs the arms beneath it; driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them harness_selftest 580-a-hand-rebuild-must-record and it reads a different source as stale rather than fresh 15;16;17;18;19 2026-09-23 the record call removed from rebuild.sh, so no stamp is written at all +harness_selftest 580-a-hand-rebuild-must-record and it writes a reason, so there is something for the gate to print 15;16;17;18;19 2026-09-24 rebuild.sh exits without saying why, so the gate has nothing to print +harness_selftest 580-a-hand-rebuild-must-record and the diagnosis carries the failing line, not just a status 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason harness_selftest 580-a-hand-rebuild-must-record and the freshness gate then reads the tree as built from this source 15;16;17;18;19 2026-09-23 the record call removed from rebuild.sh, so no stamp is written at all harness_selftest 580-a-hand-rebuild-must-record and the installed library of the running major was never touched 15;16;17;18;19 2026-09-23 the shim answers --pkglibdir with the real prefix, so the install is not redirected harness_selftest 580-a-hand-rebuild-must-record and writes no stamp into the directory it was called from 15;16;17;18;19 2026-09-23 the argument guard removed from pgc_record_source_stamp +harness_selftest 580-a-hand-rebuild-must-record premise: a rebuild with no pg_config fails 15;16;17;18;19 never - harness_selftest 580-a-hand-rebuild-must-record premise: lib.sh has one recorder for what was installed 15;16;17;18;19 2026-09-23 pgc_record_source_stamp removed from lib.sh harness_selftest 580-a-hand-rebuild-must-record premise: no stamp for this prefix before the rebuild 15;16;17;18;19 2026-09-23 the shim answers --pkglibdir with the real prefix, so the install is not redirected -harness_selftest 580-a-hand-rebuild-must-record premise: the hand rebuild itself succeeded 15;16;17;18;19 never - +harness_selftest 580-a-hand-rebuild-must-record premise: the diagnosis is exposed too 15;16;17;18;19 never - +harness_selftest 580-a-hand-rebuild-must-record premise: the gating decision is exposed to be judged 15;16;17;18;19 never - +harness_selftest 580-a-hand-rebuild-must-record premise: the hand rebuild itself succeeded 15;16;17;18;19 2026-09-24 rebuild.sh forced to exit 1 after writing its stamp, reproducing the aarch64 shape;rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason;rebuild.sh forced to fail after its stamp AND the gate reverted, so a failed rebuild still runs the arms beneath it harness_selftest 580-a-hand-rebuild-must-record premise: the hand rebuild tool installs, so a stale record is possible 15;16;17;18;19 never - harness_selftest 580-a-hand-rebuild-must-record premise: the shim redirects the install away from the real prefix 15;16;17;18;19 2026-09-23 the shim answers --pkglibdir with the real prefix, so the install is not redirected harness_selftest 580-a-hand-rebuild-must-record the hand rebuild tool records what it installed rather than only naming it 15;16;17;18;19 2026-09-23 the record call removed from rebuild.sh, or the comment stripping removed diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 4d564a26..c5d24d38 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -799,4 +799,40 @@ suites_not_covered 249 # each. Counted on both trees rather than reasoned: registered == covered + # not_covered holds on each, and the only name in the difference either way is # catalog_delete_index. -checks_never_observed_red 1530 +# +# 1530 -> 1532 for #1248, composed onto the main that carries #1213. +# RE-DERIVED BY COUNTING, decomposed by population: +# +# awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l -> 1532 +# 1673 rows total, 141 not `never`, 1532 + 141 == 1673 +# +# 11 new rows: 3 arrive `never`, 8 arrive dated +# 1 pre-existing row flips never -> dated +# 1530 - 1 + 3 == 1532 +# +# THIS BRANCH HAS CARRIED 1508 AND NOW 1532, and only the second describes a tree +# that exists. BOTH figures were published before this merge existed -- mine and +# @jdatcmd's, derived from different sides -- and both said 1532. A recount +# compared only against itself is the derived-term problem: it always looks +# consistent. Two predictions made independently and one count is not. +# +# THE ROWS AUTO-MERGED AND ONLY THIS LINE CONFLICTED, which is the right shape: +# the rows are all true facts about checks and compose silently; the total is +# derived and must not. Checked per part, because a total cannot see one row +# lost and another duplicated: catalog_delete_index 30 rows from #1213, part 580 +# 22 rows from here, zero duplicate keys. +# +# SIX MUTATIONS, and the part has zero non-premise arms never observed red. +# A, B and C force rebuild.sh to exit 1 AFTER it writes its stamp, which is the +# aarch64 shape exactly; D, E and F leave rebuild.sh alone and attack the +# healthy path. B reproduces the ORIGINAL defect beside the fix: with the gate +# reverted the dependents PASS despite the failed premise. +# +# The two gate arms carry the SINGULAR gate mutation rather than the compound. +# @jdatcmd drove the revert alone and got exactly those two: they call the gate +# with the literals 1 and "", so the rebuild's status never reaches them and the +# forced failure is inert for them. +# +# The three rows arriving `never` are premises: two `type -t` exposure checks +# and one asserting that a rebuild with no pg_config fails. +checks_never_observed_red 1532 diff --git a/test/selftest/580-a-hand-rebuild-must-record.sh b/test/selftest/580-a-hand-rebuild-must-record.sh index b05b2e64..1d28fb8e 100644 --- a/test/selftest/580-a-hand-rebuild-must-record.sh +++ b/test/selftest/580-a-hand-rebuild-must-record.sh @@ -112,22 +112,107 @@ _hr_stamp="$(pgc_source_stamp_path "$_hr_tmp/tree" "$_hr_tmp/pg_config")" check_text "premise: no stamp for this prefix before the rebuild" \ "$(if [ -e "$_hr_stamp" ]; then echo present; else echo absent; fi)" "absent" +# BOTH DECISIONS ARE FUNCTIONS so they can be driven from here rather than only +# by a box that happens to break (#1248). The branch that matters is the one a +# healthy machine never takes, which is exactly the branch that had never run +# until aarch64 took it. +_hr_dependents() { # _hr_dependents RC -> run|skip + [ "${1:-}" = 0 ] && echo run || echo skip +} +_hr_diagnose() { # _hr_diagnose LOGFILE -> the tail, indented, or nothing + [ -s "${1:-}" ] || return 0 + echo " ---- rebuild.log, last 20 lines ----" + tail -20 "$1" | sed 's/^/ /' +} + "$PGC_TESTDIR/rebuild.sh" "$_hr_tmp/pg_config" "$_hr_tmp/tree" >"$_hr_tmp/rebuild.log" 2>&1 _hr_rc=$? check_num "premise: the hand rebuild itself succeeded" "$_hr_rc" "0" -check_text "and the freshness gate then reads the tree as built from this source" \ - "$(pgc_freshness_verdict \ - "$(head -1 "$_hr_stamp" 2>/dev/null)" \ - "$(pgc_source_fingerprint "$_hr_tmp/tree")")" "fresh" +# ---- the premise must GATE, and the failure must SAY SOMETHING (#1248) ------ +# +# The 2026-09-24 nightly failed here on aarch64 with `got [1] want [0]`, and that +# was the entire diagnosis: the rebuild's output is captured to a file this part +# deletes on the way out, so the cause died with the workdir. The three arms +# below the premise then ran anyway and reported PASS. +# +# They were not vacuous that night, because rebuild.sh had in fact written its +# stamps before failing -- it writes them at step 3b and the symbol check that +# rejected the build comes after. That is luck rather than design. Had the BUILD +# failed, the same three arms would have read an absent stamp and reported on a +# rebuild that never happened. +# +# So both decisions are exposed as functions and judged, the way part 190 judges +# pgc_build_needs_clean, rather than being written inline where nothing can drive +# the branch that only a broken box takes. +check_text "premise: the gating decision is exposed to be judged" \ + "$(type -t _hr_dependents)" "function" +check_text "premise: the diagnosis is exposed too" \ + "$(type -t _hr_diagnose)" "function" + +check_text "a rebuild that succeeded runs the arms that read its stamp" \ + "$(_hr_dependents 0)" "run" +check_text "a rebuild that failed skips them rather than reading a stamp it did not write" \ + "$(_hr_dependents 1)" "skip" +check_text "and any other status skips too, rather than being read as success" \ + "$(_hr_dependents "")" "skip" + +_hr_dtmp="$(mktemp -d)" +printf 'rebuild: UNRESOLVED SYMBOLS against PostgreSQL 18\n__aarch64_ldadd4_acq_rel\n' \ + > "$_hr_dtmp/full.log" +: > "$_hr_dtmp/empty.log" +check_num "a log with content produces a diagnosis to print" \ + "$([ -n "$(_hr_diagnose "$_hr_dtmp/full.log")" ] && echo 1 || echo 0)" "1" +check_num "and the diagnosis carries the failing line, not just a status" \ + "$(_hr_diagnose "$_hr_dtmp/full.log" | grep -c 'UNRESOLVED SYMBOLS')" "1" +check_num "an empty log produces nothing rather than an empty banner" \ + "$([ -n "$(_hr_diagnose "$_hr_dtmp/empty.log")" ] && echo 1 || echo 0)" "0" +check_num "and a log that is not there produces nothing rather than an error" \ + "$([ -n "$(_hr_diagnose "$_hr_dtmp/nope.log" 2>/dev/null)" ] && echo 1 || echo 0)" "0" +rm -rf "$_hr_dtmp" + +# AND THE REAL SCRIPT MUST LEAVE SOMETHING TO DIAGNOSE. A gate that prints a log +# is worth nothing if the log is empty on the path that matters, so this drives +# rebuild.sh's cheapest failure -- a pg_config that is not there -- and asserts it +# wrote a reason rather than only exiting non-zero. +_hr_fail_log="$(mktemp)" +"$PGC_TESTDIR/rebuild.sh" /nonexistent/pg_config "$_hr_tmp/tree" >"$_hr_fail_log" 2>&1 +_hr_fail_rc=$? +check_num "premise: a rebuild with no pg_config fails" \ + "$([ "$_hr_fail_rc" != 0 ] && echo 1 || echo 0)" "1" +check_num "and it writes a reason, so there is something for the gate to print" \ + "$(grep -c '^rebuild: no such pg_config' "$_hr_fail_log")" "1" +rm -f "$_hr_fail_log" + +if [ "$(_hr_dependents "$_hr_rc")" = run ]; then + check_text "and the freshness gate then reads the tree as built from this source" \ + "$(pgc_freshness_verdict \ + "$(head -1 "$_hr_stamp" 2>/dev/null)" \ + "$(pgc_source_fingerprint "$_hr_tmp/tree")")" "fresh" -# THE SAME READ AGAINST A FINGERPRINT THAT IS NOT THIS TREE'S, so the arm above -# is known to distinguish rather than to answer "fresh" whatever it is given. -check_text "and it reads a different source as stale rather than fresh" \ - "$(pgc_freshness_verdict "$(head -1 "$_hr_stamp" 2>/dev/null)" deadbeefdead)" "stale" + # THE SAME READ AGAINST A FINGERPRINT THAT IS NOT THIS TREE'S, so the arm + # above is known to distinguish rather than to answer "fresh" whatever it is + # given. + check_text "and it reads a different source as stale rather than fresh" \ + "$(pgc_freshness_verdict "$(head -1 "$_hr_stamp" 2>/dev/null)" deadbeefdead)" "stale" -check_text "and the installed library of the running major was never touched" \ - "$(if [ "$(stat -c %Y "$_hr_live_so" 2>/dev/null)" = "$_hr_live_before" ]; - then echo untouched; else echo overwritten; fi)" "untouched" + check_text "and the installed library of the running major was never touched" \ + "$(if [ "$(stat -c %Y "$_hr_live_so" 2>/dev/null)" = "$_hr_live_before" ]; + then echo untouched; else echo overwritten; fi)" "untouched" +else + # THE ONLY DIAGNOSIS THERE IS. rebuild.sh's output goes to a file this part + # deletes on the way out, so without this the run reports `got [1] want [0]` + # and the cause leaves with the workdir -- which is how the aarch64 failure + # arrived undiagnosable (#1248). + _hr_diagnose "$_hr_tmp/rebuild.log" + for _hr_arm in \ + "and the freshness gate then reads the tree as built from this source" \ + "and it reads a different source as stale rather than fresh" \ + "and the installed library of the running major was never touched"; do + check_skip "$_hr_arm" \ + "SKIP $_hr_arm (the hand rebuild failed, so there is no stamp to read)" \ + "the hand rebuild failed" + done +fi rm -rf "$_hr_tmp"