Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,61 @@ true until the next version shipped.

### Fixed

- The hand-rebuild diagnosis can no longer be silent, which is how the first
version of it still told the 2026-09-24 nightly nothing (#1248).

#1249 added `_hr_diagnose` to print `rebuild.log`'s tail when the rebuild
fails. The nightly of the same day failed on aarch64 again, with the gate
working -- the three dependent arms skipped with a reason instead of cascading
-- and **the diagnosis printing nothing at all**, because `rebuild.log` was
empty and the function opened with

```sh
[ -s "${1:-}" ] || return 0
```

So the one path the diagnosis exists for was the one path it stayed quiet on.

`_hr_diagnose` now always says something, and distinguishes the two causes,
because they are different failures and merging them costs the next reader the
night:

```
---- rebuild.log, last 20 lines ----
---- rebuild.log is EMPTY (0 bytes): rebuild.sh exited
before its first echo ----
---- rebuild.log is MISSING: the redirect never created it ----
```

Two arms of selftest part 580 asserted the old silence and are replaced rather
than adjusted. They were provably wrong: `an empty log produces nothing rather
than an empty banner` and `and a log that is not there produces nothing rather
than an error` required exactly the behaviour that defeated the diagnosis on
aarch64. Four arms now require it to speak, and to name which of the two cases
it found.

**What the old diagnosis could not tell anyone, and this one can.** `[ -s ]`
is false for an empty file AND for one that is not there, so a silent
`_hr_diagnose` never distinguished them -- and neither did the first draft of
this entry, which asserted the log was empty when the only evidence was that
`-s` failed. The two have different causes and only one of them fits:

```
EMPTY rebuild.sh ran and wrote nothing before dying
MISSING the redirect could not open its target, so bash returned 1 and
never ran the command at all
```

Every exit in `rebuild.sh` before its first unconditional `echo` writes to
stderr first, and stderr is inside the same redirect, so EMPTY is not
something the script's control flow produces. MISSING is: measured, a failing
redirect returns 1, creates no file, and reports on the caller's stderr
rather than into the log. That matches `got [1]` and a silent diagnosis
exactly. Caught by @jdatcmd.

This does not explain why the aarch64 rebuild fails. It makes the next nightly
say which of the two it is.

- A failed hand rebuild now says why, and stops the arms beneath it reporting on
a rebuild that did not happen (#1248).

Expand Down
6 changes: 4 additions & 2 deletions test/check_ledger.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -1430,12 +1430,14 @@ harness_selftest 570-a-checkout-test-must-ask-git premise: the suite defines the
harness_selftest 570-a-checkout-test-must-ask-git premise: the sweep sees .git mentioned in the suites at all 15;16;17;18;19 never -
harness_selftest 570-a-checkout-test-must-ask-git the sweep matches every spelling that stats .git as a directory 15;16;17;18;19 2026-09-23 the same planted line, read by the pattern shipped in #1224;the sweep pattern shipped in #1224 is restored verbatim
harness_selftest 570-a-checkout-test-must-ask-git without the trailing boundary the same pattern flags three of them 15;16;17;18;19 2026-09-23 the boundary in the sweep pattern drifts, so the strip becomes a no-op;the derived control stops stripping the boundary
harness_selftest 580-a-hand-rebuild-must-record a log that is not there also produces a diagnosis 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all
harness_selftest 580-a-hand-rebuild-must-record a log with content produces a diagnosis to print 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason
harness_selftest 580-a-hand-rebuild-must-record a rebuild that failed skips them rather than reading a stamp it did not write 15;16;17;18;19 2026-09-24 driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them;the gate reverted so a failed rebuild still runs the arms beneath it
harness_selftest 580-a-hand-rebuild-must-record a rebuild that succeeded runs the arms that read its stamp 15;16;17;18;19 2026-09-24 the gate always skips, so a successful rebuild runs none of the arms that read its stamp
harness_selftest 580-a-hand-rebuild-must-record an empty log produces nothing rather than an empty banner 15;16;17;18;19 2026-09-24 _hr_diagnose banners unconditionally, so an empty or absent log still prints
harness_selftest 580-a-hand-rebuild-must-record and a log that is not there produces nothing rather than an error 15;16;17;18;19 2026-09-24 _hr_diagnose banners unconditionally, so an empty or absent log still prints
harness_selftest 580-a-hand-rebuild-must-record and any other status skips too, rather than being read as success 15;16;17;18;19 2026-09-24 driven alone, with no forced rebuild failure, because these two arms call the gate with literals and the rebuild's status never reaches them;the gate reverted so a failed rebuild still runs the arms beneath it
harness_selftest 580-a-hand-rebuild-must-record an empty log still produces a diagnosis, because silence is the bug 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all
harness_selftest 580-a-hand-rebuild-must-record and it distinguishes missing from empty, so the two causes do not merge 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all
harness_selftest 580-a-hand-rebuild-must-record and it names the log as empty rather than printing a bare banner 15;16;17;18;19 2026-09-24 _hr_diagnose restored to its silent `[ -s ] || return 0` form, so an empty or missing rebuild.log prints nothing at all
harness_selftest 580-a-hand-rebuild-must-record and it reads a different source as stale rather than fresh 15;16;17;18;19 2026-09-23 the record call removed from rebuild.sh, so no stamp is written at all
harness_selftest 580-a-hand-rebuild-must-record and it writes a reason, so there is something for the gate to print 15;16;17;18;19 2026-09-24 rebuild.sh exits without saying why, so the gate has nothing to print
harness_selftest 580-a-hand-rebuild-must-record and the diagnosis carries the failing line, not just a status 15;16;17;18;19 2026-09-24 rebuild.sh forced to fail after its stamp AND _hr_diagnose gutted, so the failure prints no reason
Expand Down
43 changes: 43 additions & 0 deletions test/check_ledger_budget.txt
Original file line number Diff line number Diff line change
Expand Up @@ -943,4 +943,47 @@ suites_not_covered 249
# projection ones reddens exactly one arm, at exactly the residual cost.
#
# suites_not_covered does NOT move: catalog_plan_index is already covered.

# #1248 FOLLOW-UP: THIS NUMBER DOES NOT MOVE. Four arms replace two in part 580
# and all six are dated, so `never` is untouched. RE-ANCHORED A SECOND TIME, on
# the main that carries #1254 (8bd5014b):
#
# ledger rows 1694 - 2 + 4 = 1696
# part 580 rows 22 - 2 + 4 = 24
# dated 155 - 2 + 4 = 157
# never 1539 unchanged
# 1539 + 157 == 1696
#
# The delta is constant at -2 + 4; the base has now moved twice under this
# branch -- 1673, then 1690, then 1694. Each earlier decomposition was counted
# correctly against a tree that no longer exists, which is why they are
# re-anchored rather than adjusted by a difference.
#
# THE ROWS CONFLICT, NOT JUST THIS LINE, AND UNION IS NOT THE WHOLE ANSWER.
# I wrote on #1251 that the rows auto-merge and only the total needs resolving.
# False, and @jdatcmd measured it: composing this branch conflicts in
# check_ledger.tsv itself, and taking a side drops rows silently while reporting
# a plausible total.
#
# BUT "UNION THE ROWS" IS ALSO NOT ENOUGH, which this rebase demonstrated.
# A union re-added the two rows this branch deliberately DELETES, giving
# 1698 = 1539 + 159 against a published prediction of 1696 = 1539 + 157 -- a
# number that looks like an ordinary miscount. Union the ADDITIONS, then honour
# the DELETIONS, then recount. A conflict hunk cannot tell a removed row from a
# row the other side never had.
#
# THE TWO REMOVED ROWS CARRY HISTORY AND IT IS DELIBERATELY NOT TRANSFERRED.
# `orphan-scan --prune` refused them and said to reconcile or to say in the
# commit why the history may go. It may go because these arms are not renamed,
# they are REVERSED. Their recorded mutation is
#
# _hr_diagnose banners unconditionally, so an empty or absent log still prints
#
# which is exactly what the four new arms REQUIRE. Carrying it forward would
# assert each new arm is reddened by the behaviour it demands.
#
# The four new rows are dated on all five majors from five separate runs, one
# merge each, under the single mutation `_hr_diagnose` restored to its silent
# `[ -s ] || return 0` form. 4 arms x 5 majors = 20 reds, and no other check went
# red in any of the five logs.
checks_never_observed_red 1539
40 changes: 33 additions & 7 deletions test/selftest/580-a-hand-rebuild-must-record.sh
Original file line number Diff line number Diff line change
Expand Up @@ -119,10 +119,24 @@ check_text "premise: no stamp for this prefix before the rebuild" \
_hr_dependents() { # _hr_dependents RC -> run|skip
[ "${1:-}" = 0 ] && echo run || echo skip
}
_hr_diagnose() { # _hr_diagnose LOGFILE -> the tail, indented, or nothing
[ -s "${1:-}" ] || return 0
_hr_diagnose() { # _hr_diagnose LOGFILE -> the tail, or WHY there is no tail
# NEVER SILENT. The first version returned early on `[ -s ]`, so the
# 2026-09-24 nightly -- which failed with an EMPTY rebuild.log on aarch64 --
# printed no diagnosis at all. An absent log and an empty one are different
# failures: one means the redirect never opened, the other means rebuild.sh
# exited before its first echo. Naming which one costs a line and saves a
# night.
local _f="${1:-}"
if [ ! -f "$_f" ]; then
echo " ---- rebuild.log is MISSING: the redirect never created it ----"
return 0
fi
if [ ! -s "$_f" ]; then
echo " ---- rebuild.log is EMPTY (0 bytes): rebuild.sh exited before its first echo ----"
return 0
fi
echo " ---- rebuild.log, last 20 lines ----"
tail -20 "$1" | sed 's/^/ /'
tail -20 "$_f" | sed 's/^/ /'
}

"$PGC_TESTDIR/rebuild.sh" "$_hr_tmp/pg_config" "$_hr_tmp/tree" >"$_hr_tmp/rebuild.log" 2>&1
Expand Down Expand Up @@ -165,10 +179,22 @@ check_num "a log with content produces a diagnosis to print" \
"$([ -n "$(_hr_diagnose "$_hr_dtmp/full.log")" ] && echo 1 || echo 0)" "1"
check_num "and the diagnosis carries the failing line, not just a status" \
"$(_hr_diagnose "$_hr_dtmp/full.log" | grep -c 'UNRESOLVED SYMBOLS')" "1"
check_num "an empty log produces nothing rather than an empty banner" \
"$([ -n "$(_hr_diagnose "$_hr_dtmp/empty.log")" ] && echo 1 || echo 0)" "0"
check_num "and a log that is not there produces nothing rather than an error" \
"$([ -n "$(_hr_diagnose "$_hr_dtmp/nope.log" 2>/dev/null)" ] && echo 1 || echo 0)" "0"
# THESE ARMS USED TO DEMAND SILENCE, AND THE SILENCE WAS THE DEFECT (#1248).
# The 2026-09-24 nightly failed the premise on aarch64 with an EMPTY rebuild.log.
# `[ -s ]` returned early, so the whole diagnosis was nothing at all -- the exact
# outcome this part exists to prevent, reached through the arm that required it.
# A diagnosis that can be silent is not a diagnosis. When there is no tail to
# print it must say so, and say which of the two causes it found, because
# "rebuild.sh wrote no reason" and "the log was never created" are different
# failures and merging them costs the next reader the night.
check_num "an empty log still produces a diagnosis, because silence is the bug" \
"$([ -n "$(_hr_diagnose "$_hr_dtmp/empty.log")" ] && echo 1 || echo 0)" "1"
check_num "and it names the log as empty rather than printing a bare banner" \
"$(_hr_diagnose "$_hr_dtmp/empty.log" | grep -c 'rebuild.log is EMPTY')" "1"
check_num "a log that is not there also produces a diagnosis" \
"$([ -n "$(_hr_diagnose "$_hr_dtmp/nope.log" 2>/dev/null)" ] && echo 1 || echo 0)" "1"
check_num "and it distinguishes missing from empty, so the two causes do not merge" \
"$(_hr_diagnose "$_hr_dtmp/nope.log" 2>/dev/null | grep -c 'rebuild.log is MISSING')" "1"
rm -rf "$_hr_dtmp"

# AND THE REAL SCRIPT MUST LEAVE SOMETHING TO DIAGNOSE. A gate that prints a log
Expand Down
Loading