From efeae75b0ae92fe1e28c848cc1a3b0939c1afb0d Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 9 Sep 2026 20:38:38 -0600 Subject: [PATCH 01/27] test: count a check and record it in one operation (#917) Check results were prose. `check`, `check_num` and `check_text` printed PASS or FAIL and nothing else, so proving that a mutation reddened one NAMED check meant grepping text -- which is how every mutation proof in this repository is currently made, a person reading `FAIL ` out of a log and retyping it. That is also how a reverted guard once reported plain green while the check count fell from 190 to 186: the suite passed, and the only evidence anything had changed was a number nobody was comparing. THE FIX IS NOT A SECOND EMITTER. A second source of truth for how many checks ran is the defect this family of issues exists to close. lib.sh had ELEVEN places that bumped PGC_CHECKS, each with its own outcome line beside it -- eleven chances to add a twelfth and forget the line, which is exactly what projections.sh's expect_fail did with ten call sites for as long as it existed. So counting a check and recording it are ONE operation, pgc_record. A helper cannot report an outcome without being counted, and cannot be counted without reporting one, because no code path does either alone. `checks run: N` and the N record lines are the same increment seen twice. Eleven sites became one, and the arm that holds it is structural: lib.sh may bump PGC_CHECKS in exactly one place, and that place must be pgc_record. The record is tab separated -- suite, name, verdict, reason -- so a check name containing spaces survives. The reason carries #915's REASON_CODE, which is what makes this more than a reformat: an unrunnable check is distinguishable from a passing one without parsing prose. A verdict pgc_record does not recognise is recorded as a FAIL rather than dropped, because dropping it would leave PGC_CHECKS bumped with no outcome recorded -- the reconciliation pgc_summary already refuses. THE HUMAN LINES DID NOT MOVE. DISPLAY is passed to pgc_record whole rather than composed inside it, and both harnesses pin the exact strings for check, check_text, check_num and check_unrunnable. 3,762 call sites, with suites, selftests and CI all grepping `^PASS` and `^FAIL`, is far past what a careful refactor can be trusted on. PGC_SUITE is resolved once at load rather than per check: pgc_record runs at every one of those call sites, and a basename fork at each is 3,762 forks a suite does not need. The runner reconciles the two artifacts per suite: a log states `checks run: N` and carries N records. That cannot fail by drifting, since one function does both, but it can fail -- a suite killed mid-way, a truncated log, a helper that prints an outcome without recording it. A log with no count at all never reached its summary, which is a different fault from a miscount and is reported as one rather than reading as a clean reconciliation. Stacked on #916, which supplies pgc_log_shows_accounting: only a suite that reached its summary has a count to reconcile against. Evidence: selftest exit 0, 535 checks, 535 records, 0 failures; 14 pytest tests; shellcheck -S error rc=0; docs_style PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/lib.sh | 127 +++++++----- test/pytest/TESTS.md | 73 ++++++- ...test_check_results_are_machine_readable.py | 183 ++++++++++++++++++ test/run_all_versions.sh | 42 ++++ .../400-a-check-result-must-be-machine.sh | 167 ++++++++++++++++ 5 files changed, 533 insertions(+), 59 deletions(-) create mode 100644 test/pytest/test_check_results_are_machine_readable.py create mode 100644 test/selftest/400-a-check-result-must-be-machine.sh diff --git a/test/lib.sh b/test/lib.sh index 8af23700..472bcb34 100755 --- a/test/lib.sh +++ b/test/lib.sh @@ -28,6 +28,10 @@ # failure and continue. PGC_FAIL=0 +# The suite's own name, resolved ONCE at load rather than per check: pgc_record +# runs at every one of 3,762 call sites, and a basename fork at each of them is +# 3,762 forks a suite does not need. +PGC_SUITE="$(basename "$0" .sh)" PGC_CHECKS=0 # The status pgc_summary uses for "ran no checks". @@ -969,17 +973,62 @@ psql_file() { # only supported way to add a check from outside this file, and selftest part 320 # sweeps for direct PGC_CHECKS writes so the next expect_fail is caught when it # is written rather than when it reddens something. -pgc_pass() { # pgc_pass NAME +# ---- one place that counts a check, and it is the same place that records it - +# +# lib.sh had ELEVEN sites bumping PGC_CHECKS, each with its own outcome line +# beside it. That is eleven chances to add a twelfth and forget the line, which +# is exactly what projections.sh's expect_fail did with ten call sites for as +# long as it existed. +# +# Counting and recording are therefore ONE operation. A helper cannot report an +# outcome without being counted, and cannot be counted without reporting one, +# because there is no code path that does either alone. `checks run: N` and the +# N record lines are the same increment seen twice. +# +# DISPLAY is passed whole rather than composed here, so every existing human line +# stays byte-identical: suites, selftests and CI all grep `^PASS` and `^FAIL`, +# and 3,762 call sites is far past what a careful refactor can be trusted on. +# +# The record is tab separated -- suite, name, verdict, reason -- so a check name +# containing spaces survives, and the reason carries the REASON_CODE #915 +# introduced. That is what makes this more than a reformat: an unrunnable check +# is distinguishable from a passing one without parsing prose. +pgc_record() { # pgc_record VERDICT NAME DISPLAY [REASON] + local _v="$1" _name="$2" _display="$3" _reason="${4:-}" PGC_CHECKS=$((PGC_CHECKS + 1)) - PGC_PASSED=$((PGC_PASSED + 1)) - echo "PASS $1" + case "$_v" in + PASS) PGC_PASSED=$((PGC_PASSED + 1)) ;; + FAIL) PGC_FAILED=$((PGC_FAILED + 1)); PGC_FAIL=1 ;; + UNRUN) PGC_UNRUN=$((PGC_UNRUN + 1)) ;; + *) + # An unknown verdict is a failure of the harness, not a check to + # drop. Dropping it would leave PGC_CHECKS bumped with no outcome + # recorded, which is the reconciliation failure pgc_summary refuses. + PGC_FAILED=$((PGC_FAILED + 1)); PGC_FAIL=1 + _display="FAIL $_name: pgc_record was given the verdict [$_v], which is not PASS, FAIL or UNRUN" + _v=FAIL + ;; + esac + printf '%s\n' "$_display" + # Tabs in a field would split it. Nothing in the tree puts one in a check + # name, and this makes that true rather than assumed. + printf 'RESULT\t%s\t%s\t%s\t%s\n' \ + "${PGC_SUITE:-unknown}" \ + "$(printf '%s' "$_name" | tr '\t' ' ')" \ + "$_v" \ + "$(printf '%s' "$_reason" | tr '\t' ' ')" +} + +pgc_pass() { # pgc_pass NAME + pgc_record PASS "$1" "PASS $1" } pgc_fail() { # pgc_fail NAME DETAIL - PGC_CHECKS=$((PGC_CHECKS + 1)) - PGC_FAILED=$((PGC_FAILED + 1)) - PGC_FAIL=1 - if [ -n "${2:-}" ]; then echo "FAIL $1: $2"; else echo "FAIL $1"; fi + if [ -n "${2:-}" ]; then + pgc_record FAIL "$1" "FAIL $1: $2" + else + pgc_record FAIL "$1" "FAIL $1" + fi } # A check that could not be evaluated is a third state, not a pass. @@ -998,30 +1047,23 @@ pgc_fail() { # pgc_fail NAME DETAIL # FAILED, because the failure is the more urgent fact. check_unrunnable() { # check_unrunnable NAME REASON_CODE DETAIL local name="$1" reason="${2:-}" detail="${3:-}" - PGC_CHECKS=$((PGC_CHECKS + 1)) case " $PGC_UNRUN_REASONS " in *" $reason "*) ;; *) - echo "FAIL $name: unrunnable reason [$reason] is not one of: $PGC_UNRUN_REASONS" - PGC_FAIL=1 - PGC_FAILED=$((PGC_FAILED + 1)) + pgc_record FAIL "$name" \ + "FAIL $name: unrunnable reason [$reason] is not one of: $PGC_UNRUN_REASONS" return ;; esac - PGC_UNRUN=$((PGC_UNRUN + 1)) - echo "UNRUN $name: $reason: $detail" + pgc_record UNRUN "$name" "UNRUN $name: $reason: $detail" "$reason" } check() { local name="$1" got="$2" want="$3" - PGC_CHECKS=$((PGC_CHECKS + 1)) if [ "$got" = "$want" ]; then - PGC_PASSED=$((PGC_PASSED + 1)) - echo "PASS $name" + pgc_record PASS "$name" "PASS $name" else - echo "FAIL $name: got [$got] want [$want]" - PGC_FAIL=1 - PGC_FAILED=$((PGC_FAILED + 1)) + pgc_record FAIL "$name" "FAIL $name: got [$got] want [$want]" fi } @@ -1071,11 +1113,8 @@ pgc_is_number() { # $1 -> 0 when $1 is a number check_text() { local name="$1" got="$2" want="$3" if [ -z "$got" ] || [ -z "$want" ]; then - PGC_CHECKS=$((PGC_CHECKS + 1)) - PGC_FAIL=1 - PGC_FAILED=$((PGC_FAILED + 1)) - echo "FAIL $name: a side is empty, so nothing was compared:" \ - "got [$got] want [$want]" + pgc_record FAIL "$name" \ + "FAIL $name: a side is empty, so nothing was compared: got [$got] want [$want]" return 1 fi check "$name" "$got" "$want" @@ -1085,11 +1124,8 @@ check_text() { check_num() { local name="$1" got="$2" want="$3" if ! pgc_is_number "$got" || ! pgc_is_number "$want"; then - PGC_CHECKS=$((PGC_CHECKS + 1)) - PGC_FAIL=1 - PGC_FAILED=$((PGC_FAILED + 1)) - echo "FAIL $name: not a measurement, so nothing was compared:" \ - "got [$got] want [$want]" + pgc_record FAIL "$name" \ + "FAIL $name: not a measurement, so nothing was compared: got [$got] want [$want]" return 1 fi check "$name" "$got" "$want" @@ -1121,30 +1157,20 @@ check_ratio() { # $1 label, $2 a, $3 b, $4 max local name="$1" a="$2" b="$3" max="$4" ratio if ! pgc_is_number "$a" || ! pgc_is_number "$b" || ! pgc_is_number "$max"; then - PGC_CHECKS=$((PGC_CHECKS + 1)) - PGC_FAIL=1 - PGC_FAILED=$((PGC_FAILED + 1)) - echo "FAIL $name: not a measurement, so no ratio was formed:" \ - "a=[$a] b=[$b] max=[$max]" + pgc_record FAIL "$name" \ + "FAIL $name: not a measurement, so no ratio was formed: a=[$a] b=[$b] max=[$max]" return 1 fi if [ "$(awk -v x="$a" -v y="$b" 'BEGIN { print (x + 0 == 0 || y + 0 == 0) ? "yes" : "no" }')" = yes ]; then - PGC_CHECKS=$((PGC_CHECKS + 1)) - PGC_FAIL=1 - PGC_FAILED=$((PGC_FAILED + 1)) - echo "FAIL $name: a side of the ratio is zero, so nothing was measured:" \ - "a=[$a] b=[$b]" + pgc_record FAIL "$name" \ + "FAIL $name: a side of the ratio is zero, so nothing was measured: a=[$a] b=[$b]" return 1 fi ratio="$(awk -v a="$a" -v b="$b" 'BEGIN { printf "%.2f", a / b }')" - PGC_CHECKS=$((PGC_CHECKS + 1)) if [ "$(awk -v r="$ratio" -v m="$max" 'BEGIN { print (r <= m) ? "yes" : "no" }')" = yes ]; then - PGC_PASSED=$((PGC_PASSED + 1)) - echo "PASS $name (${ratio}x, bound ${max}x, from a=$a b=$b)" + pgc_record PASS "$name" "PASS $name (${ratio}x, bound ${max}x, from a=$a b=$b)" else - echo "FAIL $name: ${ratio}x exceeds the ${max}x bound (a=$a b=$b)" - PGC_FAIL=1 - PGC_FAILED=$((PGC_FAILED + 1)) + pgc_record FAIL "$name" "FAIL $name: ${ratio}x exceeds the ${max}x bound (a=$a b=$b)" fi } @@ -1157,10 +1183,8 @@ pgc_require_tools() { command -v "$t" >/dev/null 2>&1 || missing="$missing $t" done if [ -n "$missing" ]; then - echo "FAIL the tools this suite measures with are missing:$missing" - PGC_CHECKS=$((PGC_CHECKS + 1)) - PGC_FAIL=1 - PGC_FAILED=$((PGC_FAILED + 1)) + pgc_record FAIL "the tools this suite measures with are missing" \ + "FAIL the tools this suite measures with are missing:$missing" return 1 fi return 0 @@ -1479,10 +1503,7 @@ pgc_skip() { # pgc_skip echo "SKIP $2 (waived by $allow_one or PGC_ALLOW_MISSING)" pgc_summary fi - PGC_CHECKS=$((PGC_CHECKS + 1)) - PGC_FAIL=1 - PGC_FAILED=$((PGC_FAILED + 1)) - echo "FAIL $2" + pgc_record FAIL "$2" "FAIL $2" echo " A missing dependency is an environment defect, not a pass. Install" echo " it, or set $allow_one=1 to run knowingly without this coverage." pgc_summary diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index c6b62532..8fa18435 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -60,9 +60,10 @@ behaviour, the source of that number is named. - [12. test_saop_element_pushdown.py: scattered set pruning](#12-test_saop_element_pushdownpy-scattered-set-pruning) - [13. test_hilbert_locality.py: what the Hilbert curve buys](#13-test_hilbert_localitypy-what-the-hilbert-curve-buys) - [14. test_suite_accounting.py: the matrix accounting for its own suites](#14-test_suite_accountingpy-the-matrix-accounting-for-its-own-suites) -- [15. Adding a test](#15-adding-a-test) -- [16. What this corpus does NOT yet refuse](#16-what-this-corpus-does-not-yet-refuse) -- [17. Traps this corpus records](#17-traps-this-corpus-records) +- [15. test_check_results_are_machine_readable.py: one counter, one record](#15-test_check_results_are_machine_readablepy-one-counter-one-record) +- [16. Adding a test](#16-adding-a-test) +- [17. What this corpus does NOT yet refuse](#17-what-this-corpus-does-not-yet-refuse) +- [18. Traps this corpus records](#18-traps-this-corpus-records) ## 1. How to read a test in here @@ -1105,7 +1106,67 @@ reproduces on long files and not short ones -- it passed every fixture and faile on the real population, naming two of the longest suites. Selftest 040 carries the same story from #473 and #476. -## 15. Adding a test +## 15. test_check_results_are_machine_readable.py: one counter, one record + +Check results were prose. `check`, `check_num` and `check_text` printed `PASS` or +`FAIL` and nothing else, so proving that a mutation reddened one **named** check meant +grepping text. That is how a reverted guard once reported plain green while the check +count fell from 190 to 186 -- the suite passed, and the only evidence anything had +changed was a number nobody was comparing. + +The fix is not a second emitter beside the counters. A second source of truth for how +many checks ran is the defect this family of issues exists to close, and `lib.sh` had +**eleven** places that bumped `PGC_CHECKS` -- eleven chances to add a twelfth and +forget the line beside it, which is exactly what `projections.sh`'s `expect_fail` did +with ten call sites for as long as it existed. + +So counting a check and recording it are **one operation**, `pgc_record`. A helper +cannot report an outcome without being counted, and cannot be counted without +reporting one, because no code path does either alone. `checks run: N` and the N +record lines are the same increment seen twice. + +The record is tab separated -- suite, name, verdict, reason -- so a check name with +spaces survives, and the reason carries the `REASON_CODE` from #915. That is what +makes it more than a reformat: an unrunnable check is distinguishable from a passing +one without parsing prose. + +### `test_lib_sh_counts_a_check_in_exactly_one_place` + +The structural arm, and the one that matters most. It stops the next `expect_fail` +from being written rather than catching it after a year of silent miscounting. + +### `test_each_verdict_emits_one_record_carrying_its_fields` + +PASS, FAIL and UNRUN each emit one record with the right verdict, and the name field +keeps its spaces. A reason code the enum does not hold is already a failure, and must +record the verdict it produced rather than the one it was asked for. + +### `test_every_helper_records_exactly_once` + +`check_text`, `check_num`, `check_ratio`, `pgc_pass` and `pgc_fail`, not a sample of +them. Each had its own counter bump and its own outcome line, and each was one place +the pair could come apart. + +### `test_the_human_lines_are_byte_identical` + +3,762 call sites, and suites, selftests and CI all grep `^PASS` and `^FAIL`. Adding a +record beside them is only safe if the prose did not move, so the exact strings are +pinned rather than the refactor trusted. + +### `test_the_record_count_equals_the_counter_the_summary_reports` + +One operation, so it cannot fail by drifting. It can fail if a helper is added that +prints an outcome without recording it, which is the `expect_fail` shape. + +### `test_the_runner_reconciles_records_against_the_stated_count` + +A suite's log states `checks run: N` and carries N records. Those are two artifacts of +the same run and they can genuinely disagree: a suite killed mid-way, a truncated log, +a helper that prints an outcome without recording it. A log with no count at all never +reached its summary -- a different fault from a miscount, and not a clean +reconciliation. + +## 16. Adding a test 0. **Write it twice.** Every test in this tree ships as a `.sh` suite and a pytest test **in the same change** (jd, 2026-09-09). Not ported later, not one or the @@ -1132,7 +1193,7 @@ story from #473 and #476. failed the selftest on both majors of the matrix, which is how it was found. A new directory under `test/` inherits every rule the old ones follow. -## 16. What this corpus does NOT yet refuse +## 17. What this corpus does NOT yet refuse `VACUITY_MODES.md` is the inventory: 79 ways a pytest harness can report a pass while asserting nothing, 73 of them demonstrated by an actual run. **This layer refuses 25 @@ -1144,7 +1205,7 @@ Read it before adding a test. The gaps most likely to affect a new test are that same family satisfies it, and that a write is not required to have written anything. Both are named there with the refusal each needs. -## 17. Traps this corpus records +## 18. Traps this corpus records Recorded because each one produced a confident wrong result before it was caught, and all are the same family as the defect the layer exists to prevent. diff --git a/test/pytest/test_check_results_are_machine_readable.py b/test/pytest/test_check_results_are_machine_readable.py new file mode 100644 index 00000000..29af7689 --- /dev/null +++ b/test/pytest/test_check_results_are_machine_readable.py @@ -0,0 +1,183 @@ +"""A check result must be machine-readable, and counted in ONE place. + +Check results were prose. `check`, `check_num` and `check_text` printed PASS or FAIL +and nothing else, so proving a mutation reddened one NAMED check meant grepping text. +That is how a reverted guard once reported plain green while the check count fell from +190 to 186: the suite passed, and the only evidence anything had changed was a number +nobody was comparing. + +The fix is not a second emitter beside the counters. A second source of truth for how +many checks ran is the defect this issue family exists to close, and `lib.sh` had +ELEVEN places that bumped `PGC_CHECKS` -- eleven chances to add a twelfth and forget +the line beside it, which is exactly what `projections.sh`'s `expect_fail` did with ten +call sites for as long as it existed. + +So counting a check and recording it are ONE operation, `pgc_record`. `checks run: N` +and the N record lines are the same increment seen twice. + +These tests drive the shell out of `lib.sh` rather than reimplementing it, for the same +reason `test_suite_accounting.py` does: a Python twin would agree with itself. +""" + +import pathlib +import subprocess + +REPO = pathlib.Path(__file__).resolve().parents[2] +LIB = REPO / "test" / "lib.sh" +RUNNER = REPO / "test" / "run_all_versions.sh" + + +def _sh(body): + """Run a snippet with lib.sh sourced, under the shell options the suites use.""" + script = f'set -uo pipefail\ncd "{LIB.parent}"\n. ./lib.sh >/dev/null 2>&1\n{body}\n' + return subprocess.run(["bash", "-c", script], capture_output=True, text=True).stdout + + +def _records(call): + return [l for l in _sh(call).splitlines() if l.startswith("RESULT\t")] + + +def _human(call): + return [l for l in _sh(call).splitlines() if not l.startswith("RESULT\t")] + + +def _extract(path, name): + out, keep = [], False + for line in path.read_text().splitlines(): + if line.startswith(f"{name}() "): + keep = True + if keep: + out.append(line) + if line == "}": + break + return "\n".join(out) + + +# ---- the structural arm ----------------------------------------------------- + + +def test_lib_sh_counts_a_check_in_exactly_one_place(expect): + """Eleven bump sites were eleven chances to add a twelfth and forget the outcome. + + This is the arm that stops the next `expect_fail` from being written, rather than + catching it after it has been silently miscounting for a year. + """ + text = LIB.read_text() + expect.num(text.count("PGC_CHECKS=$((PGC_CHECKS"), 1, + "lib.sh bumps PGC_CHECKS in exactly one place") + expect.num(_extract(LIB, "pgc_record").count("PGC_CHECKS=$((PGC_CHECKS"), 1, + "and that place is pgc_record") + + +# ---- the record line -------------------------------------------------------- + + +def test_each_verdict_emits_one_record_carrying_its_fields(expect): + """Tab separated -- suite, name, verdict, reason -- so a name with spaces survives. + + The reason carries the REASON_CODE, which is what makes this more than a reformat: + an unrunnable check is distinguishable from a passing one without parsing prose. + """ + for call, verdict in ((' check "a name" x x', "PASS"), + (' check "a name" x y', "FAIL")): + recs = _records(call) + expect.num(len(recs), 1, f"a {verdict} check emits exactly one record") + expect.text(recs[0].split("\t")[3], verdict, f"and its verdict field says {verdict}") + expect.text(_records(' check "a name" x x')[0].split("\t")[2], "a name", + "and the name field keeps its spaces") + + recs = _records(' check_unrunnable "a name" MISSING_DEPENDENCY "no jq"') + expect.num(len(recs), 1, "an unrunnable check emits exactly one record") + expect.text(recs[0].split("\t")[3], "UNRUN", + "and its verdict is UNRUN, which is neither of the other two") + expect.text(recs[0].split("\t")[4], "MISSING_DEPENDENCY", + "and the REASON_CODE travels in the reason field, not in prose") + + # A reason the enum does not hold is already a FAIL. It must record the verdict it + # produced, not the one it was asked for. + expect.text(_records(' check_unrunnable "n" NOT_A_REASON "x"')[0].split("\t")[3], + "FAIL", "a bogus reason code records FAIL, not UNRUN") + + +def test_every_helper_records_exactly_once(expect): + """Not a sample. Each of these had its own counter bump and its own outcome line, + and each was one place the pair could come apart.""" + cases = { + 'check_text "n" "" "x"': "FAIL", + 'check_num "n" abc 1': "FAIL", + 'check_ratio "n" abc 1 2': "FAIL", + 'check_ratio "n" 0 1 2': "FAIL", + 'check_ratio "n" 1 1 2': "PASS", + 'pgc_pass "n"': "PASS", + 'pgc_fail "n" "d"': "FAIL", + } + for call, verdict in cases.items(): + recs = _records(" " + call) + expect.num(len(recs), 1, f"{call.split()[0]} emits exactly one record") + expect.text(recs[0].split("\t")[3], verdict, f"and records {verdict}") + + +def test_the_human_lines_are_byte_identical(expect): + """3,762 call sites, and suites, selftests and CI all grep `^PASS` and `^FAIL`. + + Adding a record beside them is only safe if the prose did not move, so the exact + strings are pinned rather than the refactor trusted. + """ + cases = { + ' check "a name" x x': "PASS a name", + ' check "a name" x y': "FAIL a name: got [x] want [y]", + ' check_unrunnable "a name" MISSING_DEPENDENCY "no jq"': + "UNRUN a name: MISSING_DEPENDENCY: no jq", + ' check_text "n" "" "x"': + "FAIL n: a side is empty, so nothing was compared: got [] want [x]", + ' check_num "n" abc 1': + "FAIL n: not a measurement, so nothing was compared: got [abc] want [1]", + } + for call, want in cases.items(): + expect.text("\n".join(_human(call)), want, f"{call.strip().split()[0]} prints its old line") + + +def test_the_record_count_equals_the_counter_the_summary_reports(expect): + """One operation, so it cannot fail by drifting -- but it CAN fail if a helper is + added that prints an outcome without recording it, which is the expect_fail shape.""" + out = _sh(' check a x x; check b x y; check_text c "" x; check_num d abc 1\n' + ' check_ratio e 1 1 2; pgc_pass f; check_unrunnable g MISSING_DEPENDENCY h\n' + ' echo "COUNTED $PGC_CHECKS"') + records = len([l for l in out.splitlines() if l.startswith("RESULT\t")]) + counted = int(next(l.split()[1] for l in out.splitlines() if l.startswith("COUNTED "))) + expect.num(records, 7, "premise: the probe ran every helper shape once") + expect.num(records, counted, "the record count equals the counter the summary reports") + + +# ---- the runner reconciles the two -------------------------------------------- + + +def test_the_runner_reconciles_records_against_the_stated_count(tmp_path, expect): + """A log states `checks run: N` and carries N records. Those are two artifacts of + the same run, and they can genuinely disagree: a suite killed mid-way, a truncated + log, a helper that prints an outcome without recording it.""" + body = _extract(RUNNER, "pgc_reconcile_records") + expect.at_least(len(body), 1, "premise: the runner defines the reconciliation") + + def run(text): + log = tmp_path / "s.log" + log.write_text(text) + script = f'set -uo pipefail\n{body}\npgc_reconcile_records "{log}"\n' + r = subprocess.run(["bash", "-c", script], capture_output=True, text=True) + return r.stdout, r.returncode + + ok = "RESULT\ts\ta\tPASS\t\nRESULT\ts\tb\tPASS\t\nchecks run: 2\n" + expect.num(run(ok)[1], 0, "a log whose records match its stated count reconciles") + + out, rc = run("RESULT\ts\ta\tPASS\t\nchecks run: 2\n") + expect.num(rc, 1, "a log with fewer records than it claims is caught") + expect.num(out.count("records=1"), 1, "and both numbers are named, not just the verdict") + + expect.num(run("RESULT\ts\ta\tPASS\t\nRESULT\ts\tb\tPASS\t\n" + "RESULT\ts\tc\tPASS\t\nchecks run: 2\n")[1], 1, + "a log with more records than it claims is caught too") + + # A log with no count at all never reached its summary. That is a different fault + # from a miscount and must not read as a clean reconciliation. + expect.num(run("RESULT\ts\ta\tPASS\t\n")[1], 1, + "a log that never stated a count is not silently accepted") diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index 5898ae7c..506380da 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -940,6 +940,34 @@ pgc_log_shows_accounting() { # pgc_log_shows_accounting LOGFILE -> yes|no fi } +pgc_reconcile_records() { # pgc_reconcile_records LOGFILE -> 0 ok, 1 mismatch + # A suite's log states `checks run: N` and carries N record lines. They are + # the same increment seen twice -- pgc_record does both -- so this cannot + # fail by drifting. It CAN fail, which is why it is asserted: a suite killed + # mid-way, a truncated log, or a helper that prints an outcome without + # recording it all separate the two. + # + # A log with no `checks run:` line at all never reached its summary. That is a + # different fault from a miscount, and it must not read as a clean + # reconciliation just because there is nothing to compare against. + local _log="$1" _records _stated + if [ ! -f "$_log" ]; then + echo " no log to reconcile records against: $_log" + return 1 + fi + _records="$(grep -c '^RESULT ' "$_log" || true)" + _stated="$(sed -n 's/^checks run: \([0-9][0-9]*\)$/\1/p' "$_log" | tail -1)" + if [ -z "$_stated" ]; then + echo " records=$_records but the log never stated a count, so it did not reach its summary" + return 1 + fi + if [ "$_records" != "$_stated" ]; then + echo " records=$_records but the log states checks run: $_stated" + return 1 + fi + return 0 +} + pgc_reconcile_accounting() { # pgc_reconcile_accounting DECLARED OBSERVED [NOTDISPATCHED] -> 0 ok, 1 asymmetric # Set equality in both directions. The two directions catch opposite # mistakes and neither can stand in for the other: @@ -1088,11 +1116,25 @@ pgc_tally_suite() { # pgc_tally_suite NAME VERDICT LOGFILE # PG16 would report PG15's incomplete suites in its own summary line and # still print PASS, because verfail is per major and this count was not. suites_incomplete=0 + _rec_bad=0 for s in "${SUITES[@]}"; do _rc="$(cat "$builddir/${s}.rc" 2>/dev/null)" _verdict="$(pgc_classify_suite_rc "$_rc" "$builddir/${s}.log")" pgc_tally_suite "$s" "$_verdict" "$builddir/${s}.log" + # Only a suite that reached its summary has a count to reconcile against + # (#917). One that was never dispatched, or that does not use lib.sh's + # accounting at all, has nothing to compare and is not a mismatch. + if [ "$(pgc_log_shows_accounting "$builddir/${s}.log")" = yes ]; then + if ! pgc_reconcile_records "$builddir/${s}.log"; then + echo " in $s" + _rec_bad=$((_rec_bad + 1)) + fi + fi done + if [ "$_rec_bad" != 0 ]; then + echo " $_rec_bad suite(s) on PG$major state a check count their records do not match" + verfail=1 + fi # How many suites actually asserted something, said out loud (#447). # diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh new file mode 100644 index 00000000..ab66e7c3 --- /dev/null +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -0,0 +1,167 @@ +# ---- a check result must be machine-readable, from ONE counter -------------- +# +# Check results are prose. `check`, `check_num` and `check_text` print PASS or +# FAIL and nothing else, so proving that a mutation reddened one NAMED check +# means grepping text. Every mutation proof in this repository is currently a +# person reading `FAIL ` out of a log and retyping it into a comment. +# +# That is how a reverted guard once reported plain green while the check count +# fell from 190 to 186: the suite passed, and the only evidence anything had +# changed was a number nobody was comparing. +# +# The fix is NOT a second emitter beside the counters. A second source of truth +# for how many checks ran is the defect this issue family exists to close, and +# lib.sh had ELEVEN places that bumped PGC_CHECKS -- eleven chances to add the +# twelfth and forget the line beside it. +# +# So counting a check and recording it are ONE operation, pgc_record, and every +# helper routes through it. The arms below hold that shape rather than the +# behaviour of any one helper, because the shape is what stops the next +# expect_fail from being written. +# --------------------------------------------------------------------------- + +_libsh="$PGC_TESTDIR/lib.sh" + +check "premise: lib.sh is where the check helpers live" \ + "$(grep -c '^check() {' "$_libsh")" "1" + +# THE STRUCTURAL ARM. One counter site, not eleven. +check "lib.sh bumps PGC_CHECKS in exactly one place" \ + "$(grep -c 'PGC_CHECKS=\$((PGC_CHECKS' "$_libsh")" "1" +check "and that place is pgc_record" \ + "$(sed -n '/^pgc_record()/,/^}/p' "$_libsh" | grep -c 'PGC_CHECKS=\$((PGC_CHECKS')" "1" + +# ---- the record line itself ------------------------------------------------- +# +# Tab separated, so a name containing spaces survives. Fields: suite, name, +# verdict, reason. The reason carries phase 1's REASON_CODE, which is what makes +# this more than a reformat: an unrunnable check is distinguishable from a +# passing one without parsing prose. + +_rec() { # _rec HELPER ARGS... -> the RESULT lines that helper emitted + ( PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0 + "$@" 2>/dev/null | grep '^RESULT' ) +} +_human() { # _human HELPER ARGS... -> the human lines that helper emitted + ( PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0 + "$@" 2>/dev/null | grep -v '^RESULT' ) +} + +check "a passing check emits exactly one record" \ + "$(_rec check "a name" x x | wc -l)" "1" +check "and its verdict field says PASS" \ + "$(_rec check "a name" x x | cut -f4)" "PASS" +check "and its name field is the check's name, spaces intact" \ + "$(_rec check "a name" x x | cut -f3)" "a name" + +check "a failing check emits exactly one record" \ + "$(_rec check "a name" x y | wc -l)" "1" +check "and its verdict field says FAIL" \ + "$(_rec check "a name" x y | cut -f4)" "FAIL" + +check "an unrunnable check emits exactly one record" \ + "$(_rec check_unrunnable "a name" MISSING_DEPENDENCY "no jq" | wc -l)" "1" +check "and its verdict field says UNRUN, which is neither of the other two" \ + "$(_rec check_unrunnable "a name" MISSING_DEPENDENCY "no jq" | cut -f4)" "UNRUN" +check "and the REASON_CODE travels in the reason field, not in prose" \ + "$(_rec check_unrunnable "a name" MISSING_DEPENDENCY "no jq" | cut -f5)" "MISSING_DEPENDENCY" + +# A reason code the enum does not contain is already a FAIL. It must record that +# verdict, not the one it was asked for. +check "a bogus reason code records FAIL, not UNRUN" \ + "$(_rec check_unrunnable "a name" NOT_A_REASON "x" | cut -f4)" "FAIL" + +# ---- every helper, not just the two that were easy -------------------------- +# +# check_text, check_num, check_ratio and pgc_require_tools each had their own +# counter bump and their own outcome line. Each is one place the pair could come +# apart, which is why the arm is over ALL of them rather than a sample. + +check "check_text on an empty side emits one record" \ + "$(_rec check_text "n" "" "x" | wc -l)" "1" +check "and records FAIL, because nothing was compared" \ + "$(_rec check_text "n" "" "x" | cut -f4)" "FAIL" +check "check_num on a non-number emits one record" \ + "$(_rec check_num "n" "abc" "1" | wc -l)" "1" +check "and records FAIL" "$(_rec check_num "n" "abc" "1" | cut -f4)" "FAIL" +check "check_ratio on a non-number emits one record" \ + "$(_rec check_ratio "n" "abc" "1" "2" | wc -l)" "1" +check "check_ratio with a zero side emits one record" \ + "$(_rec check_ratio "n" "0" "1" "2" | wc -l)" "1" +check "check_ratio that forms a ratio emits one record" \ + "$(_rec check_ratio "n" "1" "1" "2" | wc -l)" "1" +check "and records PASS when the ratio is inside the bound" \ + "$(_rec check_ratio "n" "1" "1" "2" | cut -f4)" "PASS" +check "pgc_pass emits one record" "$(_rec pgc_pass "n" | wc -l)" "1" +check "pgc_fail emits one record" "$(_rec pgc_fail "n" "d" | wc -l)" "1" + +# ---- the human lines must not have changed ---------------------------------- +# +# 3,762 check sites, and suites, selftests and CI all grep `^FAIL` and `^PASS`. +# Adding a record beside them is only safe if the prose is byte-identical, so the +# arms pin the exact strings rather than trusting that a refactor was careful. + +check "a passing check still prints its old line" \ + "$(_human check "a name" x x)" "PASS a name" +check "a failing check still prints its old line" \ + "$(_human check "a name" x y)" "FAIL a name: got [x] want [y]" +check "an unrunnable check still prints its old line" \ + "$(_human check_unrunnable "a name" MISSING_DEPENDENCY "no jq")" "UNRUN a name: MISSING_DEPENDENCY: no jq" +check "check_text's empty-side line is unchanged" \ + "$(_human check_text "n" "" "x")" "FAIL n: a side is empty, so nothing was compared: got [] want [x]" +check "check_num's non-measurement line is unchanged" \ + "$(_human check_num "n" "abc" "1")" "FAIL n: not a measurement, so nothing was compared: got [abc] want [1]" + +# ---- the count and the records cannot come apart ---------------------------- +# +# They are one operation, so this cannot fail by drifting. It CAN fail if a +# helper is added that prints an outcome without recording it, which is exactly +# the expect_fail shape, so it is asserted rather than argued. + +_n_calls=7 +_recorded="$( ( PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0 + check a x x; check b x y; check_text c "" x; check_num d abc 1 + check_ratio e 1 1 2; pgc_pass f; check_unrunnable g MISSING_DEPENDENCY h + echo "COUNTED $PGC_CHECKS" ) )" +check "premise: the probe ran every helper shape once" \ + "$(printf '%s\n' "$_recorded" | grep -c '^RESULT')" "$_n_calls" +check "the record count equals the counter the summary reports" \ + "$(printf '%s\n' "$_recorded" | grep -c '^RESULT')" \ + "$(printf '%s\n' "$_recorded" | sed -n 's/^COUNTED //p')" + +# ---- and the RUNNER must reconcile them ------------------------------------- +# +# A suite's log states `checks run: N` and carries N record lines. Those are two +# artifacts of the same run and they can genuinely disagree: a suite killed +# mid-way, a truncated log, a helper that prints an outcome without recording it. + +_rv="$PGC_TESTDIR/run_all_versions.sh" +check "the runner defines the record reconciliation" \ + "$(grep -c '^pgc_reconcile_records()' "$_rv")" "1" + +eval "$(sed -n '/^pgc_reconcile_records()/,/^}/p' "$_rv")" +check "premise: it is callable" "$(type -t pgc_reconcile_records)" "function" + +_rl="$PGC_WORKDIR/rec.log" +printf 'RESULT\ts\ta\tPASS\t\nRESULT\ts\tb\tPASS\t\nchecks run: 2\n' > "$_rl" +check "a log whose records match its stated count reconciles" \ + "$(pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch)" "ok" + +printf 'RESULT\ts\ta\tPASS\t\nchecks run: 2\n' > "$_rl" +check "a log with fewer records than it claims is caught" \ + "$(pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch)" "mismatch" +check "and the two numbers are named, not just the verdict" \ + "$(pgc_reconcile_records "$_rl" 2>&1 | grep -c 'records=1 .*checks run: 2')" "1" + +printf 'RESULT\ts\ta\tPASS\t\nRESULT\ts\tb\tPASS\t\nRESULT\ts\tc\tPASS\t\nchecks run: 2\n' > "$_rl" +check "a log with more records than it claims is caught too" \ + "$(pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch)" "mismatch" + +# A log with no `checks run:` line at all did not reach its summary. That is a +# different fault from a miscount and must not read as a clean reconciliation. +printf 'RESULT\ts\ta\tPASS\t\n' > "$_rl" +check "a log that never stated a count is not silently accepted" \ + "$(pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch)" "mismatch" + +check "the runner calls the record reconciliation, not merely defines it" \ + "$(grep -c '[^_[:alnum:]]pgc_reconcile_records "' "$_rv")" "1" From dd6a4d3f2005381e33c6e82265c4736526a6deb0 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 07:11:28 -0600 Subject: [PATCH 02/27] test: pgc_record paid four forks per check, and direct counter writes made the reconciler wrong (#917) Both from OffgridwithJD's review, and the first is embarrassing in the specific way that makes it worth writing down. FOUR FORKS PER RECORD, IN THE FUNCTION THAT HOISTS PGC_SUITE OUT TO AVOID THEM. pgc_record ran `$(printf ... | tr)` twice to blank tabs -- two subshells and two tr processes -- at every one of 3,762 check sites, three lines below a comment explaining that a basename fork per check is 3,762 forks a suite does not need. Parameter expansion does it free. Measured on an idle box, 2,000 calls, output identical on every input including a real tab, a leading tab and a trailing one: printf | tr in $( ) 3.1577 ms per call ${var//tab/ } 0.0096 ms per call ratio 331x across 3,762 checks 11.9 s of pure fork overhead against 36 ms DIRECT COUNTER WRITES MADE THE RECONCILER WRONG, and the two rules disagreed. Selftest 320 blessed a direct PGC_CHECKS bump that records an outcome nearby; pgc_reconcile_records requires a RESULT line. Thirteen sites across ten suites took the first path, so on a failure the reconciler added `records=N but the log states checks run: N+1` on top of the real failure. The fix is not to soften the reconciler. Counting a check and recording it are one operation, which is this change's whole argument, and a direct write is a check counted with nothing recorded -- the hole that argument cannot have. All thirteen now call pgc_fail, which lib.sh's own header already calls "the only supported way to add a check from outside this file". Selftest 320 gains the stronger rule that was not satisfiable until now: no suite using lib.sh's accounting writes PGC_CHECKS directly. bench_guards keeps its own counter under the same name and never sources lib.sh, and is exempt by measurement rather than by name. TWO OF MY OWN ARMS WERE WRONG ON THE WAY THROUGH, both caught by running them. The old premise required FIVE direct writes to EXIST, which is a premise about the corpus rather than about the sweep -- so converting the thirteen turned it red for exactly the reason the change is for. A premise that fails when the thing it guards is fixed is the wrong premise. It now asserts the sweep read something, and both rules are proven on fixtures: the original flags a bump with no outcome and allows one with an outcome, the stronger one flags that same allowed bump, and both exempt a private counter. Two arms reading "[]" over a clean corpus are satisfied by a sweep that classifies nothing. And those fixtures, written out literally, made this file flag its own three generator lines -- the same mistake selftest 080's control avoids by living in a quoted heredoc. The bump is now assembled from the variable name. Evidence: selftest exit 0, 581 checks, 581 records, 0 failures; 133 pytest passed (the 35 errors are /usr/local/pg18a absent on this host, identical on main); shellcheck rc=0 across test/, selftest/ and bench/. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/analyze_differential.sh | 5 +- test/analyze_function.sh | 5 +- test/lib.sh | 12 ++- test/native_groupagg_batch.sh | 3 +- test/native_repack.sh | 4 +- test/objstore_module.sh | 6 +- test/objstore_stash_recovery.sh | 8 +- test/parquet_export_stats.sh | 11 +-- test/pg19_vacuum_options.sh | 4 +- .../320-a-check-that-could-not-run.sh | 88 ++++++++++++++++++- 10 files changed, 112 insertions(+), 34 deletions(-) diff --git a/test/analyze_differential.sh b/test/analyze_differential.sh index 16eaa8e7..443e09a8 100755 --- a/test/analyze_differential.sh +++ b/test/analyze_differential.sh @@ -54,9 +54,8 @@ ROWS=${PGC_ANALYZE_DIFF_ROWS:-50000} # The major is asserted first so an unreadable version is not mistaken for an old # one and reported as "supported, skipped". if ! pgc_is_number "${PGC_MAJOR:-}"; then - echo "FAIL could not read the server major, so the gate below cannot be trusted: got [${PGC_MAJOR:-}]" - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)) - PGC_FAIL=1 + pgc_fail "could not read the server major, so the gate below cannot be trusted" \ + "got [${PGC_MAJOR:-}]" pgc_summary fi if [ "$PGC_MAJOR" -lt 18 ]; then diff --git a/test/analyze_function.sh b/test/analyze_function.sh index d491e1ef..10a133f4 100755 --- a/test/analyze_function.sh +++ b/test/analyze_function.sh @@ -63,9 +63,8 @@ ROWS=${PGC_ANALYZE_ROWS:-500000} # The major is asserted first. An unreadable version must not be mistaken for an # old one, or a broken environment would report SKIP and look supported. if ! pgc_is_number "${PGC_MAJOR:-}"; then - echo "FAIL could not read the server major, so the gate below cannot be trusted: got [${PGC_MAJOR:-}]" - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)) - PGC_FAIL=1 + pgc_fail "could not read the server major, so the gate below cannot be trusted" \ + "got [${PGC_MAJOR:-}]" pgc_summary fi if [ "$PGC_MAJOR" -lt 18 ]; then diff --git a/test/lib.sh b/test/lib.sh index 472bcb34..17159186 100755 --- a/test/lib.sh +++ b/test/lib.sh @@ -1012,11 +1012,19 @@ pgc_record() { # pgc_record VERDICT NAME DISPLAY [REASON] printf '%s\n' "$_display" # Tabs in a field would split it. Nothing in the tree puts one in a check # name, and this makes that true rather than assumed. + # + # PARAMETER EXPANSION, not `printf | tr` in a command substitution. The first + # version paid four forks per record -- two subshells and two tr processes -- + # in the function that runs at every one of 3,762 check sites, and whose own + # comment hoists PGC_SUITE out of the body on exactly that ground. Measured on + # an idle box, 2,000 calls, identical output on every input including a real + # tab: 3.1577 ms per call against 0.0096 ms, 331x, or 11.9 seconds of pure + # fork overhead across a full suite against 36 ms. Reported by OffgridwithJD. printf 'RESULT\t%s\t%s\t%s\t%s\n' \ "${PGC_SUITE:-unknown}" \ - "$(printf '%s' "$_name" | tr '\t' ' ')" \ + "${_name//$'\t'/ }" \ "$_v" \ - "$(printf '%s' "$_reason" | tr '\t' ' ')" + "${_reason//$'\t'/ }" } pgc_pass() { # pgc_pass NAME diff --git a/test/native_groupagg_batch.sh b/test/native_groupagg_batch.sh index cdf3cfb8..72aa57d7 100755 --- a/test/native_groupagg_batch.sh +++ b/test/native_groupagg_batch.sh @@ -93,8 +93,7 @@ agree_in() { # agree_in TABLE LABEL "SELECT ... FROM %T ..." # md5 of empty input is a fixed string, so require the columnar arm produced # rows at all before trusting the comparison. if [ -z "$(q "${tmpl//%T/$tbl}" | head -1)" ]; then - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)); PGC_FAIL=1 - echo "FAIL $label: the columnar arm returned no rows, so nothing was compared" + pgc_fail "$label" "the columnar arm returned no rows, so nothing was compared" return 1 fi check_text "$label" "$col" "$heap" diff --git a/test/native_repack.sh b/test/native_repack.sh index b8ef2bd0..ef671131 100755 --- a/test/native_repack.sh +++ b/test/native_repack.sh @@ -54,9 +54,7 @@ srv="$(q 'SHOW server_version_num')" # on an older major depends on it being 0. Asserting the premise must not destroy # the skip it guards. if ! pgc_is_number "$srv"; then - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)) - PGC_FAIL=1 - echo "FAIL the server did not answer 'SHOW server_version_num': got [$srv]" + pgc_fail "the server did not answer 'SHOW server_version_num'" "got [$srv]" pgc_summary fi if [ "$srv" -lt 190000 ]; then diff --git a/test/objstore_module.sh b/test/objstore_module.sh index 7fb139ac..3aea7f2b 100755 --- a/test/objstore_module.sh +++ b/test/objstore_module.sh @@ -70,10 +70,8 @@ for stash in "$MOD.away" "$MOD.probe"; do # every check below would run against a broken installation and report the # confusing half of the truth, so stop here and say which file to look at. if ! stash_is_debris; then - echo "FAIL restored $stash to $MOD, but that is not a module either." - echo " This installation needs 'make install' before the suite can run." - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)) - PGC_FAIL=1 + pgc_fail "restored $stash to $MOD, but that is not a module either" \ + "this installation needs 'make install' before the suite can run" pgc_summary exit 1 fi diff --git a/test/objstore_stash_recovery.sh b/test/objstore_stash_recovery.sh index e0ef1906..81842b4e 100755 --- a/test/objstore_stash_recovery.sh +++ b/test/objstore_stash_recovery.sh @@ -52,13 +52,13 @@ echo "PG_CONFIG=$PG_CONFIG" if [ -z "${PGC_SKIP_BUILD:-}" ]; then echo "-- building" make -C "$SRCDIR" PG_CONFIG="$PG_CONFIG" >/dev/null || { - echo "FAIL build failed, so nothing below measures the guard" - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)); PGC_FAIL=1; pgc_summary + pgc_fail "build failed, so nothing below measures the guard" + pgc_summary } echo "-- installing" make -C "$SRCDIR" install PG_CONFIG="$PG_CONFIG" >/dev/null || { - echo "FAIL install failed, so nothing below measures the guard" - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)); PGC_FAIL=1; pgc_summary + pgc_fail "install failed, so nothing below measures the guard" + pgc_summary } fi diff --git a/test/parquet_export_stats.sh b/test/parquet_export_stats.sh index db83952b..f650d188 100755 --- a/test/parquet_export_stats.sh +++ b/test/parquet_export_stats.sh @@ -88,8 +88,7 @@ psql_run "SELECT pgcolumnar.export_parquet('es_c', '$PARQ');" if ! python3 "$STATS_PY" "$PARQ" > "$S" 2>"$PGC_WORKDIR/stats.err"; then echo "FAIL the footer parser could not read the exported file:" sed 's/^/ /' "$PGC_WORKDIR/stats.err" - PGC_FAIL=1 - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)) + pgc_fail "the statistics export failed" "see the indented output above" pgc_summary fi @@ -229,10 +228,7 @@ check_num "every bound is its physical width" \ check_float() { # check_float NAME GOT WANT local name="$1" got="$2" want="$3" if ! pgc_is_number "$got" || ! pgc_is_number "$want"; then - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)) - PGC_FAIL=1 - echo "FAIL $name: not a measurement, so nothing was compared:" \ - "got [$got] want [$want]" + pgc_fail "$name" "not a measurement, so nothing was compared: got [$got] want [$want]" return 1 fi check "$name" \ @@ -370,9 +366,8 @@ if python3 "$STATS_PY" "$NANQ" > "$NS" 2>&1; then check "a zero minimum is written as -0.0" \ "$(nf "$N_Z9" minhex)" "0000000000000080" else - echo "FAIL the footer parser could not read the NaN fixture:" + pgc_fail "the footer parser could not read the NaN fixture" "see the indented output below" sed 's/^/ /' "$NS" - PGC_FAIL=1; PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)) fi # ---- (f) column_orders, without which the bounds have no defined meaning ---- diff --git a/test/pg19_vacuum_options.sh b/test/pg19_vacuum_options.sh index 6cafd0aa..3ee7f6d0 100755 --- a/test/pg19_vacuum_options.sh +++ b/test/pg19_vacuum_options.sh @@ -40,9 +40,7 @@ srv="$(q 'SHOW server_version_num')" # on an older major depends on it being 0. Asserting the premise must not destroy # the skip it guards. if ! pgc_is_number "$srv"; then - PGC_CHECKS=$((PGC_CHECKS + 1)); PGC_FAILED=$((PGC_FAILED + 1)) - PGC_FAIL=1 - echo "FAIL the server did not answer 'SHOW server_version_num': got [$srv]" + pgc_fail "the server did not answer 'SHOW server_version_num'" "got [$srv]" pgc_summary fi if [ "$srv" -lt 190000 ]; then diff --git a/test/selftest/320-a-check-that-could-not-run.sh b/test/selftest/320-a-check-that-could-not-run.sh index e6b56e12..3a6a325c 100644 --- a/test/selftest/320-a-check-that-could-not-run.sh +++ b/test/selftest/320-a-check-that-could-not-run.sh @@ -201,8 +201,15 @@ while IFS= read -r _cnt_l; do done < <(grep -rn 'PGC_CHECKS=\$((PGC_CHECKS' "$_cnt_dir"/*.sh "$_cnt_dir"/selftest/*.sh 2>/dev/null \ | grep -v '/lib\.sh:' | sort) -check "premise: the sweep finds the direct writes it is meant to police" \ - "$([ "${#_cnt_sites[@]}" -ge 5 ] && echo enough || echo "${#_cnt_sites[@]}")" "enough" +# The premise used to require FIVE direct writes to exist, which was a premise +# about the corpus rather than about the sweep -- and #917 converted the thirteen +# that used lib.sh's accounting, so it went red for the reason the change is FOR. +# A premise that fails when the thing it guards is fixed is the wrong premise. +# +# It now asserts the sweep read something, and the classifier is proven on a +# FIXTURE below rather than on whatever the corpus happens to contain. +check "premise: the sweep read the corpus and found sites to classify" \ + "$([ "${#_cnt_sites[@]}" -ge 1 ] && echo yes || echo "no (${#_cnt_sites[@]})")" "yes" # A file that keeps its OWN counters and never calls pgc_summary is not bound by # this invariant, because nothing reconciles it. Asserted rather than assumed: @@ -223,6 +230,83 @@ done check "every direct write to PGC_CHECKS records an outcome too" \ "$(_tsm_fmt_cnt "$_cnt_n" "$_cnt_bad")" "[]" +# AND, SINCE #917, THERE MUST BE NONE AT ALL among files that use lib.sh's +# accounting. That is a stronger rule than the one above and it was not +# satisfiable before: counting a check and RECORDING it are now one operation, +# pgc_record, so a direct write to PGC_CHECKS counts a check that emits no +# RESULT line -- and pgc_reconcile_records then reports a records/checks mismatch +# on top of whatever the suite was actually failing for. +# +# The two rules disagreed until the thirteen sites were converted, which +# OffgridwithJD found: selftest 320 blessed a direct write with a nearby outcome, +# while the reconciler required a record. pgc_pass and pgc_fail exist precisely so +# a suite-local helper need not touch the counters, and lib.sh's own header +# already calls them "the only supported way to add a check from outside this +# file". This arm makes that a rule rather than a description. +# +# A file that keeps its OWN counters and never calls pgc_summary is exempt for the +# same reason as above, measured from the file rather than named: bench_guards +# reuses the variable name privately and never sources lib.sh. +_cnt_lib=0; _cnt_lib_bad="" +for _cnt_l in "${_cnt_sites[@]}"; do + _cnt_f="${_cnt_l%%:*}" + _cnt_ln="$(printf '%s' "$_cnt_l" | cut -d: -f2)" + [ "$(grep -c 'pgc_summary' "$_cnt_f" || true)" != 0 ] || continue + _cnt_lib=$((_cnt_lib + 1)) + [ "$_cnt_lib" -le 5 ] && _cnt_lib_bad="$_cnt_lib_bad ${_cnt_f##*/}:$_cnt_ln" +done +check "no suite that uses lib.sh's accounting writes PGC_CHECKS directly" \ + "$(_tsm_fmt_cnt "$_cnt_lib" "$_cnt_lib_bad")" "[]" + +# BOTH RULES, PROVEN ON A FIXTURE. With the corpus clean, two arms reading "[]" +# are satisfied by a sweep that classifies nothing, so the classifier is driven +# against files built to trip it. +_cnt_fx="$PGC_WORKDIR/cntfx"; mkdir -p "$_cnt_fx" + +# Uses lib.sh's accounting, bumps the counter, records no outcome nearby: both +# rules must see it. +# The bump is ASSEMBLED from the variable name rather than written out, so these +# generator lines do not themselves carry the shape the sweep looks for. Writing +# it literally made this file flag its own three fixtures -- the same mistake +# selftest 080's control avoids by living in a quoted heredoc. +_cnt_bump="$(printf '%s=$((%s + 1))' PGC_CHECKS PGC_CHECKS)" +_cnt_out="$(printf '%s=$((%s + 1))' PGC_FAILED PGC_FAILED)" +printf 'pgc_summary\n%s\necho hi\n' "$_cnt_bump" > "$_cnt_fx/bare.sh" +# Uses lib.sh's accounting, bumps the counter, DOES record an outcome: the old +# rule lets it through, the new one does not. That difference is the change. +printf 'pgc_summary\n%s\n%s\n' "$_cnt_bump" "$_cnt_out" > "$_cnt_fx/outcome.sh" +# Keeps its own counter and never calls pgc_summary: exempt from both. +printf '%s\necho "checks run: 1"\n' "$_cnt_bump" > "$_cnt_fx/private.sh" + +_cnt_fx_old() { # the ORIGINAL rule, applied to one file + local _f="$1" _l + [ "$(grep -c 'pgc_summary' "$_f" || true)" != 0 ] || { echo exempt; return; } + _l="$(grep -n 'PGC_CHECKS=\$((PGC_CHECKS' "$_f" | head -1 | cut -d: -f1)" + [ -n "$_l" ] || { echo none; return; } + if [ "$(sed -n "$((_l > 3 ? _l - 3 : 1)),$((_l + 6))p" "$_f" \ + | grep -cE 'PGC_PASSED=|PGC_FAILED=|PGC_UNRUN=' || true)" = 0 ]; then + echo flagged + else + echo allowed + fi +} +_cnt_fx_new() { # the STRONGER rule, applied to one file + local _f="$1" + [ "$(grep -c 'pgc_summary' "$_f" || true)" != 0 ] || { echo exempt; return; } + [ "$(grep -c 'PGC_CHECKS=\$((PGC_CHECKS' "$_f" || true)" != 0 ] && echo flagged || echo none +} + +check "premise: the fixtures carry the shapes these rules are about" \ + "$(grep -lc 'PGC_CHECKS=\$((PGC_CHECKS' "$_cnt_fx"/*.sh 2>/dev/null | grep -c . || true)" "3" +check "the original rule flags a bump that records no outcome" \ + "$(_cnt_fx_old "$_cnt_fx/bare.sh")" "flagged" +check "and allows one that does, which is what it was written to allow" \ + "$(_cnt_fx_old "$_cnt_fx/outcome.sh")" "allowed" +check "the stronger rule flags that same allowed bump, which is the change" \ + "$(_cnt_fx_new "$_cnt_fx/outcome.sh")" "flagged" +check "and both exempt a file that keeps its own counter without lib.sh" \ + "$(_cnt_fx_old "$_cnt_fx/private.sh")/$(_cnt_fx_new "$_cnt_fx/private.sh")" "exempt/exempt" + # ---- and the RUNNER must not report an INCOMPLETE suite as a pass ------------ # # lib.sh exiting 67 is only half the state. The runner decides what a status From 9260cea814a7222ff6aa0e92482ade5c57c66176 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 07:33:40 -0600 Subject: [PATCH 03/27] test: a ledger of which checks have ever been red (#918) Closes #918, phase 4 of #858, and the honest version of it. Nothing recorded whether a check had ever been red. That is the gap that let 39 checks across 35 suites ship unable to fail, three of them inside the suite whose whole purpose is to stop exactly that. The gate answered "did anything print FAIL" and had never answered "could anything print FAIL". WHAT THIS LEDGER CLAIMS, AND WHAT IT DOES NOT. It records that a named check WAS OBSERVED RED in a recorded run. It does NOT claim the check is proven able to fail: that is a stronger statement, it needs a named mutation applied deliberately, and conflating the two would put a claim in the ledger that nothing measured -- the `defeated: 0` shape from VACUITY_MODES section 1, a number that reads as evidence and is not. SO EVERY ENTRY CURRENTLY READS `never`, and that is the finding rather than an embarrassment. #918 asks "nothing records whether a check has ever been red"; the answer this ships is "and now something records that almost nothing has". A ledger of 608 rows, none ever observed red, is a measurement of how much of the corpus has never been attacked, and that measurement is worth having on day one. WHAT FILLS IT. Not only deliberate mutation runs. Every real CI red fills it, every flake, every bisect, and those arrive whether anyone remembers or not. A mutation run is the deliberate accelerator, not the only source. Raised by OffgridwithJD, and it matters because "only a mutation run can retire debt" invites someone to build a mutation gate before it is needed. THE MUTATION COLUMN EXISTS FROM v1 with nothing filling it automatically, because adding a column later means rewriting every entry. If an entry can record WHICH mutation reddened a check, the catalogue a mutation gate would need builds itself out of work people already do by hand -- the vacuity branches are writing nine to eleven per change, each chosen to revert one property. A RENAME IS REPORTED, NOT SILENTLY ABSORBED. The ledger is keyed by check name, and names here are prose that gets rewritten freely -- which is most of why #917 exists. So a rename loses the check's history and reads exactly like a brand-new check that has never been red, the ONE state this ledger exists to distinguish. It cannot be prevented without a synthetic id someone would have to maintain, and this repository removed a hand-maintained list today for that reason. So a name that appeared while another disappeared is NAMED. Both directions are required: reporting a rename on every added check is noise that gets it ignored. A DUPLICATED NAME SHARES ONE ROW, so one of the two going red would mark BOTH as observed red -- a claim about a check nothing attacked. Also reported rather than prevented. The real corpus carries four today, which is how it was noticed at all: 612 records reduce to 608 rows. TWO TRACKED FILES CARRY THE DEBT, per #858's own constraint. check_ledger.tsv and check_ledger_budget.txt are in the tree, so a change to either is a diff a reviewer sees. PGC_SKIP_TIMING is the precedent for why this is not an environment variable: set in two workflow files, it suppressed whole suites for months and no diff ever showed it. Both numbers may only go down. The second number is the one that is easy to forget: suites_not_covered is 250 of 251, because the ledger can only be seeded from suites whose logs exist, and the matrix does not preserve them. Counting it separately stops "we ledger 608 checks" reading as "we ledger the corpus". It is a real limit, stated rather than hidden, and it burns down as suites are seeded. The gate refuses a check the ledger has never seen, so a new check cannot enter as silent debt -- while the existing 608 are grandfathered, because a gate that fails on 3,762 unledgered sites is one somebody disables under deadline. Evidence: selftest exit 0, 612 checks, 0 failures; the gate rc=0 against the committed files; 27 pytest; shellcheck rc=0; docs_style PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/check_ledger.tsv | 608 ++++++++++++++++++ test/check_ledger_budget.txt | 28 + test/pgc_ledger.py | 265 ++++++++ test/pytest/TESTS.md | 99 ++- test/pytest/test_mutation_ledger.py | 229 +++++++ .../410-a-check-must-have-been-red.sh | 249 +++++++ 6 files changed, 1472 insertions(+), 6 deletions(-) create mode 100644 test/check_ledger.tsv create mode 100644 test/check_ledger_budget.txt create mode 100755 test/pgc_ledger.py create mode 100644 test/pytest/test_mutation_ledger.py create mode 100644 test/selftest/410-a-check-must-have-been-red.sh diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv new file mode 100644 index 00000000..d01a60dc --- /dev/null +++ b/test/check_ledger.tsv @@ -0,0 +1,608 @@ +harness_selftest 67 without its line is a failure, not an INCOMPLETE taken on trust never +harness_selftest CI runs the extension-upgrade guard somewhere (#741) never +harness_selftest GCOV_PREFIX is exported before the suites run (#740) never +harness_selftest PREMISE and the target really holds sources find would otherwise hash never +harness_selftest PREMISE the Makefile's recursion was actually parsed never +harness_selftest PREMISE the copy discovers the same build directories as the real tree never +harness_selftest PREMISE the fingerprint covers at least src never +harness_selftest PREMISE the fixture's src really is a symlink never +harness_selftest README.md quotes the number of modes the inventory names as refused never +harness_selftest TESTS.md states no totals line for a merge to get wrong never +harness_selftest TESTS.md states the counted number as well never +harness_selftest a /./ segment hashes the same tree the same way never +harness_selftest a /src/.. segment hashes the same tree the same way never +harness_selftest a NEW unaccounted suite fails even while the known debt is excused never +harness_selftest a bogus reason code records FAIL, not UNRUN never +harness_selftest a caller passing a major is caught never +harness_selftest a caller that reimplements the digest is caught never +harness_selftest a check merely added is not reported as a rename never +harness_selftest a check merely removed is not reported as a rename either never +harness_selftest a check observed red gains the date it was seen never +harness_selftest a check the ledger has never seen is refused, not absorbed never +harness_selftest a comment mentioning pgc_summary is not a declaration never +harness_selftest a comparison on the exit status is not counted as an assignment never +harness_selftest a counter that drifts is caught rather than absorbed never +harness_selftest a declared suite that produced no accounting is caught never +harness_selftest a declared suite the driver never dispatched reconciles never +harness_selftest a documented file that does not exist is named never +harness_selftest a documented test that does not exist is named, not passed over never +harness_selftest a drifted exit code is visible rather than absorbed never +harness_selftest a duplicated check name is reported by name never +harness_selftest a failing check emits exactly one record never +harness_selftest a failing check still prints its old line never +harness_selftest a failing suite names the first fatal event in its log never +harness_selftest a failure outranks an unrunnable check, and both are still counted never +harness_selftest a file that does not exist is reported absent, not exempt never +harness_selftest a file that is not a build input does not move it never +harness_selftest a file that uses comm pins the collation of every sort feeding it never +harness_selftest a fingerprint different from the record is stale never +harness_selftest a fingerprint equal to the record is fresh never +harness_selftest a fingerprint is 12 hex characters never +harness_selftest a fingerprint that reads src only is caught never +harness_selftest a free port beyond the old 300-probe bound is still found never +harness_selftest a hash inside a word does not hide the call after it never +harness_selftest a later green run does not erase an observation never +harness_selftest a library newer than the running server is REFUSED never +harness_selftest a library older than the running server is accepted never +harness_selftest a log carrying lib.sh's accounting line is accounted never +harness_selftest a log carrying neither is not accounted never +harness_selftest a log claiming PASSED without the accounting line shows none never +harness_selftest a log that never stated a count is not silently accepted never +harness_selftest a log whose records match its stated count reconciles never +harness_selftest a log with fewer records than it claims is caught never +harness_selftest a log with more records than it claims is caught too never +harness_selftest a long suite that calls pgc_summary still declares accounting never +harness_selftest a longer name containing pgc_summary is not a declaration never +harness_selftest a make_cluster with no cleanup is caught never +harness_selftest a merge that names its mutation records it against the check that reddened never +harness_selftest a missing binary timestamp is unknown, not predates never +harness_selftest a missing postmaster timestamp is unknown, not predates never +harness_selftest a mixed run reports both causes and neither as the whole story never +harness_selftest a name after the array's closing paren is not read as a registered suite never +harness_selftest a name defined in two files is named, not passed over never +harness_selftest a name that appeared while another disappeared is reported as a rename never +harness_selftest a neutered absent arm is caught never +harness_selftest a neutered empty-plan refusal is caught never +harness_selftest a neutered present arm is caught never +harness_selftest a new source file under objstore moves the fingerprint never +harness_selftest a passing check emits exactly one record never +harness_selftest a passing check still prints its old line never +harness_selftest a passing log shows accounting never +harness_selftest a passing ratio check is counted as a pass, not a failure never +harness_selftest a preflight that built nothing does not report PASSED never +harness_selftest a preflight that built nothing exits non-zero never +harness_selftest a preflight that built nothing says how many it built never +harness_selftest a prose total that disagrees with the ids is visible never +harness_selftest a registered suite that is accounted by nothing FAILS never +harness_selftest a relative path hashes the same tree the same way never +harness_selftest a run whose debt is within budget passes the gate never +harness_selftest a seed at the ceiling wraps past a busy top and still finds a port never +harness_selftest a server older than the binary predates it never +harness_selftest a server started after the binary is fresh never +harness_selftest a server started at the same second is fresh never +harness_selftest a stated total that disagrees with disk is visible never +harness_selftest a stated total that disagrees with the ids is visible never +harness_selftest a suite of nothing but unrunnable checks is INCOMPLETE, not SKIPPED never +harness_selftest a suite recorded as known debt passes never +harness_selftest a suite recorded as never dispatched that DID account is caught never +harness_selftest a suite that accounted passes never +harness_selftest a suite that calls pgc_summary declares accounting never +harness_selftest a suite that never calls it does not never +harness_selftest a suite that now accounts but is still listed as debt is reported never +harness_selftest a suite the driver never dispatched passes never +harness_selftest a suite whose checks all passed still exits 0 PASSED never +harness_selftest a suite with an unrunnable check exits 67 never +harness_selftest a suite with none says so as zero rather than staying silent never +harness_selftest a symlink to the tree hashes it the same way never +harness_selftest a symlinked src contributes nothing, as find -P contributes nothing never +harness_selftest a trailing comment after the call does not hide it never +harness_selftest a trailing slash hashes the same tree the same way never +harness_selftest a tree with no hashable file yields no fingerprint never +harness_selftest a write-only unrunnable field is caught never +harness_selftest adding a source file moves it never +harness_selftest an INCOMPLETE suite fails its major never +harness_selftest an INCOMPLETE suite sets the flag the major verdict actually reads never +harness_selftest an absent log shows no accounting rather than erroring never +harness_selftest an absent prose total is empty rather than a stray number never +harness_selftest an absent total is empty rather than a number that happens to match never +harness_selftest an accounting line that does not start its line is refused never +harness_selftest an added file appears in the manifest by name never +harness_selftest an added file shows up in the report never +harness_selftest an empty manifest is reported as empty, not as silence never +harness_selftest an entirely busy band reports itself full and terminates never +harness_selftest an id named twice counts once never +harness_selftest an id of fewer than three words is not counted as a mode never +harness_selftest an import from the pytest tree is caught never +harness_selftest an indented comment is still a comment never +harness_selftest an unbackticked name in prose is not treated as a claim never +harness_selftest an unconditional exit override is caught by the dominance arm never +harness_selftest an undeclared suite that DID account is caught too never +harness_selftest an undocumented file is caught along with the tests inside it never +harness_selftest an undocumented test is named rather than passed over never +harness_selftest an unhashable tree has an empty manifest never +harness_selftest an unknown provenance is not reported as a major never +harness_selftest an unparseable stamp cleans rather than guessing never +harness_selftest an unreadable b.c yields no fingerprint, not a wrong one never +harness_selftest an unreadable c.c yields no fingerprint, not a wrong one never +harness_selftest an unreadable library is not a failure never +harness_selftest an unrunnable check counts toward checks run never +harness_selftest an unrunnable check emits exactly one record never +harness_selftest an unrunnable check still prints its old line never +harness_selftest an unrunnable reason outside the enum fails rather than being accepted never +harness_selftest and 66 with its line a skip never +harness_selftest and 67 with its line INCOMPLETE, which is not a pass never +harness_selftest and a check that stayed green keeps its debt never +harness_selftest and a compiled artifact written beside its source never +harness_selftest and a failed population reconciliation fails the major never +harness_selftest and a failed reconciliation sets the per-major failure flag never +harness_selftest and a failing suite still does never +harness_selftest and a hard-coded module list is caught by the name arm never +harness_selftest and a log carrying only its OWN checks-run line is accounted too never +harness_selftest and a log with no fatal line still reports rather than staying silent never +harness_selftest and a non-numeric timestamp is unknown rather than compared as text never +harness_selftest and a prefix of a registered name is not treated as registered never +harness_selftest and a reworded producer line is refused, so the arm can fail never +harness_selftest and a skip does not, which is the one that must stay true never +harness_selftest and a skip, which reached the summary and counted zero never +harness_selftest and a suite with no unrunnable checks reconciles too never +harness_selftest and a unique one is not never +harness_selftest and absent is distinguishable from a present file that does not declare never +harness_selftest and allows one that does, which is what it was written to allow never +harness_selftest and an empty WANT is refused rather than compared never +harness_selftest and an incomplete never +harness_selftest and an ordinary failure is still a failure never +harness_selftest and an uncomputable current fingerprint is unknown, not stale never +harness_selftest and appears in the results string as INCOMPLETE never +harness_selftest and asks pgc_start_failure_message for the verdict never +harness_selftest and bench/ was in the scan, which is the hole this rule had never +harness_selftest and both exempt a file that keeps its own counter without lib.sh never +harness_selftest and comparing two manifests names it rather than saying 'changed' never +harness_selftest and counted as incomplete, so the tally can say so never +harness_selftest and counts both suites as having run never +harness_selftest and debt naming a suite that is not registered is reported too never +harness_selftest and does not cover the live one above it never +harness_selftest and every executable script declares one never +harness_selftest and every script a document names exists never +harness_selftest and every script a document names is executable never +harness_selftest and exactly one of them as incomplete never +harness_selftest and in the other direction too never +harness_selftest and is NOT made when our own postmaster died, which is the #537 case never +harness_selftest and is counted as having run never +harness_selftest and is not counted as skipped, nor is the skip count disturbed never +harness_selftest and it agrees with the real reader on a SHORT file, which is why it survived review never +harness_selftest and it catches BaseException, so an interrupt cleans up too never +harness_selftest and it is 3 bytes, not an escaped literal never +harness_selftest and it is NAMED, so the reader does not have to diff two lists never +harness_selftest and it is empty when nothing named a mutation never +harness_selftest and it is named as that fault, not as one of the other two never +harness_selftest and it is named as the opposite fault, not the same one never +harness_selftest and it is named, so the author knows which one never +harness_selftest and it is named, which the symmetry check could never do never +harness_selftest and it is not reported as having run no checks never +harness_selftest and it is the same suites, not merely the same count never +harness_selftest and it names no module directory, so it is a derivation and not a list never +harness_selftest and it says plainly that no major was recorded never +harness_selftest and it says so rather than staying silent never +harness_selftest and it stops a partially started cluster before removing the tree never +harness_selftest and its log carries the INCOMPLETE line the classifier needs never +harness_selftest and its name field is the check's name, spaces intact never +harness_selftest and its verdict field says FAIL never +harness_selftest and its verdict field says PASS never +harness_selftest and its verdict field says UNRUN, which is neither of the other two never +harness_selftest and names the suite and the check, not just a count never +harness_selftest and neither as skipped never +harness_selftest and no longer counts incompletes inline beside it never +harness_selftest and no longer mixes in the bare filename never +harness_selftest and no write-only failure flag survives in the runner never +harness_selftest and not against one that stayed green never +harness_selftest and one over budget does not never +harness_selftest and only ever moves a run off zero, so a failure still dominates never +harness_selftest and prose containing the word does not count as the line never +harness_selftest and records FAIL never +harness_selftest and records FAIL, because nothing was compared never +harness_selftest and records PASS when the ratio is inside the bound never +harness_selftest and records each suite's own verdict in the results string never +harness_selftest and records neither as ever having been red never +harness_selftest and removing it restores the fingerprint never +harness_selftest and reprints the suite's own UNRUN line beneath it never +harness_selftest and restoring it restores the fingerprint never +harness_selftest and restoring the partition restores the fingerprint never +harness_selftest and selftest/ was in the scan, which is the hole that reddened #923 never +harness_selftest and so does a failing one, which is the point never +harness_selftest and so is the attempt count never +harness_selftest and something READS it, rather than only writing it never +harness_selftest and still matches a PANIC never +harness_selftest and still matches a signal death never +harness_selftest and still matches an AddressSanitizer report never +harness_selftest and that place is pgc_record never +harness_selftest and that refusal is a VacuityError, not an ordinary assertion never +harness_selftest and the REASON_CODE travels in the reason field, not in prose never +harness_selftest and the UNRUN line the runner prints into the matrix output never +harness_selftest and the excused one is not named as a failure never +harness_selftest and the gate says which number was exceeded, by how much never +harness_selftest and the heredoc exemption covers the one inside the heredoc, not the other never +harness_selftest and the major is still readable in the name never +harness_selftest and the mistake empties the whole array rather than appending to it never +harness_selftest and the no-squatter verdict points at the server log never +harness_selftest and the old start-failure verdict is not echoed inline anywhere never +harness_selftest and the original error is re-raised rather than swallowed never +harness_selftest and the port it found is below the busy region, which is where wrapping lands never +harness_selftest and the reader answers no on it, which is the wrong answer the arm catches never +harness_selftest and the reader reads back the fingerprint the writer recorded never +harness_selftest and the real function reconciles the same input, so the arm is not noise never +harness_selftest and the reconciliation is given that record never +harness_selftest and the refusal says the server must be restarted never +harness_selftest and the registered file is written from the SUITES array itself never +harness_selftest and the run's overall status is failure never +harness_selftest and the same comparison agrees on the fixture that is right never +harness_selftest and the scan examined the suites rather than finding nothing to read never +harness_selftest and the shell keeps none either never +harness_selftest and the stable check is not reported never +harness_selftest and the start path asks pgc_start_fatal_pattern, its deliberately wider one never +harness_selftest and the suite holding it fails rather than reporting PASSED never +harness_selftest and the suite that holds it still passes never +harness_selftest and the summary line carries the incomplete count a reader needs never +harness_selftest and the tally announces it, with the reason lifted from the log never +harness_selftest and the tracked list names none of them never +harness_selftest and the tree ignores the directory Python writes them to never +harness_selftest and the two numbers are named, not just the verdict never +harness_selftest and the unrunnable ones are reported as their own count never +harness_selftest and two pg_configs for one prefix share a stamp, keyed on pkglibdir never +harness_selftest and without that record the same run is still caught never +harness_selftest building a DIFFERENT major needs a clean, which is the #536 case never +harness_selftest building the same major again needs no clean never +harness_selftest but a tree with no objects at all needs nothing, stamp or not never +harness_selftest but it says which question went unanswered never +harness_selftest but not a routine statement error never +harness_selftest check compares two empty strings and passes, which is why the rest exist never +harness_selftest check_num accepts a decimal and a sign never +harness_selftest check_num on a non-number emits one record never +harness_selftest check_num refuses a psql error message never +harness_selftest check_num refuses an md5, which is why check_text exists never +harness_selftest check_num refuses the word a yes/no check would produce never +harness_selftest check_num refuses two empty strings never +harness_selftest check_num still compares two real numbers never +harness_selftest check_num still fails two unequal numbers never +harness_selftest check_num's non-measurement line is unchanged never +harness_selftest check_ratio fails a ratio outside its bound never +harness_selftest check_ratio on a non-number emits one record never +harness_selftest check_ratio passes a ratio inside its bound never +harness_selftest check_ratio refuses a zero denominator rather than dividing by it never +harness_selftest check_ratio refuses a zero numerator, which is inside every bound never +harness_selftest check_ratio refuses an empty measurement never +harness_selftest check_ratio that forms a ratio emits one record never +harness_selftest check_ratio with a zero side emits one record never +harness_selftest check_text compares two md5 hashes, which check_num cannot never +harness_selftest check_text on an empty side emits one record never +harness_selftest check_text refuses one empty side never +harness_selftest check_text refuses two empty strings, where plain check passes never +harness_selftest check_text still fails two different strings never +harness_selftest check_text's empty-side line is unchanged never +harness_selftest control fixture: a suite whose checks all ran exits 0 never +harness_selftest control: a caller passing a pg_config is not flagged never +harness_selftest control: a document naming only what exists is clean never +harness_selftest control: a fully documented corpus reports nothing missing never +harness_selftest control: a readable run still reads fresh never +harness_selftest control: a real content change still moves the fingerprint never +harness_selftest control: a real src directory is still hashed never +harness_selftest control: an interpreter declared without the bit is caught never +harness_selftest control: and a sourced fragment, with neither, is correct never +harness_selftest control: and it still succeeds on a writable one never +harness_selftest control: and leaves the run's overall status alone never +harness_selftest control: and restoring the content restores the fingerprint never +harness_selftest control: and still announces it never +harness_selftest control: and still records that it ran, and how never +harness_selftest control: and the major reports PASS never +harness_selftest control: and the same file with the bit is not never +harness_selftest control: and the tree fingerprints again once it is readable never +harness_selftest control: cp -a preserves the execute bit, so a staged tree reads the same never +harness_selftest control: distinct names in the same corpus report no duplicate never +harness_selftest control: piping a large string into grep -q reports a match as absent never +harness_selftest control: reads, longer names, and the deliberate RANDOM/SECONDS seeds are not flagged never +harness_selftest control: the bit without an interpreter is caught too never +harness_selftest control: the same loop leaves a passing suite passing never +harness_selftest control: the same pg_config twice gives the same path never +harness_selftest control: the sweep catches an assignment to a bash special never +harness_selftest control: writing the value it was given never +harness_selftest detection distinguishes it from ours never +harness_selftest detection reports a foreign cluster's directory never +harness_selftest each manifest line is a tree-relative path and a digest never +harness_selftest editing a source file moves the fingerprint never +harness_selftest equal sets reconcile never +harness_selftest every PGC_RUN_UPGRADE-gated suite is excluded from the coverage runner (#741) never +harness_selftest every diff_query_ordered site actually names an ORDER BY never +harness_selftest every direct write to PGC_CHECKS records an outcome too never +harness_selftest every directory the Makefile builds from is in the fingerprint never +harness_selftest every ledger row carries four fields, the fourth being the mutation never +harness_selftest every nightly job is named in docs/testing.md (#741) never +harness_selftest every registered suite has a file never +harness_selftest every script that declares an interpreter is executable never +harness_selftest every suite is registered in run_all_versions.sh never +harness_selftest every suite that connects by socket path sets unix_socket_directories never +harness_selftest every suite using the ordered oracle asserts its premise never +harness_selftest every test file and every test in the corpus is named in TESTS.md never +harness_selftest every test the document names exists in the corpus never +harness_selftest guard accepts our own cluster never +harness_selftest guard rejects a foreign cluster never +harness_selftest lib.sh bumps PGC_CHECKS in exactly one place never +harness_selftest lib.sh defines check_unrunnable never +harness_selftest lib.sh defines the INCOMPLETE exit status never +harness_selftest make_cluster removes its tree when setup raises never +harness_selftest merging a green run records both checks never +harness_selftest moving bytes between files moves the fingerprint never +harness_selftest negative control: and does not find one that is not never +harness_selftest no caller passes a major where a pg_config belongs never +harness_selftest no compiled Python artifact is tracked never +harness_selftest no diff_query site names an ORDER BY it cannot test (use diff_query_ordered) never +harness_selftest no non-zero status is classified as a pass never +harness_selftest no record at all is unknown, not fresh never +harness_selftest no suite assigns to a bash special variable never +harness_selftest no suite calls set_options with a value it will reject never +harness_selftest no suite hands every run the same default port never +harness_selftest no suite pipes a captured string into an early-exit reader never +harness_selftest no suite that uses lib.sh's accounting writes PGC_CHECKS directly never +harness_selftest no test name is defined twice in the corpus never +harness_selftest no test picks a port from inside the ephemeral range never +harness_selftest nor an ordinary log line never +harness_selftest nor no for every one of them never +harness_selftest nothing leaked into the squatter never +harness_selftest objects with NO stamp are unknown provenance and must be cleaned never +harness_selftest one tree, one fingerprint, whatever the locale never +harness_selftest one unrunnable check makes the suite INCOMPLETE, not passed never +harness_selftest opposite errors do not cancel: both directions are reported never +harness_selftest pgc_fail emits one record never +harness_selftest pgc_pass emits one record never +harness_selftest pgc_port_free says the squatter's port is busy never +harness_selftest pgc_require_tools fails on one that does not never +harness_selftest pgc_require_tools passes on tools that exist never +harness_selftest pgc_setup reports the installed .so never +harness_selftest plan_marker keeps the arm that fails when the key is absent never +harness_selftest plan_marker keeps the arm that fails when the key is present never +harness_selftest plan_marker refuses a plan with no nodes at all never +harness_selftest positive control: and it is a whole list, not one lucky line never +harness_selftest positive control: the membership test finds a name that is registered never +harness_selftest positive control: the real runner's list is read, and contains isolation never +harness_selftest premise: C collation puts sort_status before sorted_projection never +harness_selftest premise: a runnable script one level down is inside the population never +harness_selftest premise: all three runner functions were extracted, not empty ranges never +harness_selftest premise: and all three are callable never +harness_selftest premise: and bench/ is in the population never +harness_selftest premise: and does NOT fire when the error is the point, across a continuation never +harness_selftest premise: and each extraction ends at its own closing brace never +harness_selftest premise: and git ls-files sees the harness it is being asked about never +harness_selftest premise: and it built none of them never +harness_selftest premise: and it is the right block (it sets the port and the preload) never +harness_selftest premise: and produced exactly one accounting line to be read never +harness_selftest premise: and really does allow the bottom never +harness_selftest premise: and so are the fixture host tools never +harness_selftest premise: and that a tracked source file is not never +harness_selftest premise: and that count excludes the definition line, which mentions it never +harness_selftest premise: and that same fixture does show the write, so the arm is not blind never +harness_selftest premise: and the accounted reader that feeds it never +harness_selftest premise: and the exemption covers a minority of them, not the corpus never +harness_selftest premise: and the line really does hold the reader it must not flag never +harness_selftest premise: and the old echo/printf pattern did NOT catch it never +harness_selftest premise: and the real function still does never +harness_selftest premise: and the real helper still carries its cleanup never +harness_selftest premise: and the stamp really was not written, so the arm is not vacuous never +harness_selftest premise: and they name at least one command in every swept directory never +harness_selftest premise: at least one suite connects by a socket path, so this is not vacuous never +harness_selftest premise: at least one suite drives the C-level encoding selftest never +harness_selftest premise: at least three nightly jobs were parsed, so the list is real never +harness_selftest premise: at least two locales are installed to compare never +harness_selftest premise: both comparison helpers are present never +harness_selftest premise: both fake configs report the same major, which is the whole point never +harness_selftest premise: both line numbers were found, so the ordering arm can mean something never +harness_selftest premise: both not_a_suite definitions were found never +harness_selftest premise: both oracles are present never +harness_selftest premise: both stray-counter probes were located never +harness_selftest premise: both the counter refusal and the lcov capture were located never +harness_selftest premise: check-ignore agrees a build object is already ignored never +harness_selftest premise: every file containing a set_options call is in the sweep never +harness_selftest premise: it is callable never +harness_selftest premise: lib.sh is readable, or every grep below approves nothing never +harness_selftest premise: lib.sh is where the check helpers live never +harness_selftest premise: lib.sh states an INCOMPLETE exit code this part could read never +harness_selftest premise: make_cluster's body was actually cut out of the file never +harness_selftest premise: no TCP suite is counted as a socket user never +harness_selftest premise: not_a_suite says no to an ordinary suite, so its yes means something never +harness_selftest premise: pipefail is on, which is the condition the bug needs never +harness_selftest premise: plan_marker's body was actually cut out of the file never +harness_selftest premise: run_coverage.sh defines not_a_suite and calls it never +harness_selftest premise: run_san.sh's default subset was found and is non-empty never +harness_selftest premise: some suite still uses comm, or the check below is vacuous never +harness_selftest premise: some suite uses the ordered oracle, or the next check is vacuous never +harness_selftest premise: the 40-line tail is filler, not the marker never +harness_selftest premise: the PGC_RUN_UPGRADE block was found and names at least one suite never +harness_selftest premise: the argument parser reads the second argument at all never +harness_selftest premise: the auxiliary band has a width to wrap within never +harness_selftest premise: the budget is a tracked file too never +harness_selftest premise: the build path ran to completion, so a stamp was due never +harness_selftest premise: the build-stamp decision is exposed to be judged never +harness_selftest premise: the check has history before the rename never +harness_selftest premise: the classifier evalled out of the runner is callable never +harness_selftest premise: the cluster-config block was located in lib.sh never +harness_selftest premise: the containment and the copy were both located never +harness_selftest premise: the corpus carries the documentation this part polices never +harness_selftest premise: the counting rule finds modes at all never +harness_selftest premise: the coverage runner is present and parses never +harness_selftest premise: the declaration reader evalled out of the runner is callable never +harness_selftest premise: the detector fires on an out-of-range value that is NOT expect_error never +harness_selftest premise: the detector fires on the line that caused #799 never +harness_selftest premise: the documents name a population of commands, not none never +harness_selftest premise: the drift changed the line the reader looks for never +harness_selftest premise: the fixture carries a well-formed accounting line, just indented never +harness_selftest premise: the fixture fingerprints at all never +harness_selftest premise: the fixture is long enough to lose the race never +harness_selftest premise: the fixture really does carry the stray name never +harness_selftest premise: the fixture really does hide its call from the stripper never +harness_selftest premise: the fixtures carry the shapes these rules are about never +harness_selftest premise: the guard's count directory and the capture's were both located never +harness_selftest premise: the harness exposes its fatal pattern to be judged never +harness_selftest premise: the harness library is where this part thinks it is never +harness_selftest premise: the heredoc exemption found heredoc lines to exempt never +harness_selftest premise: the ledger is not empty, so the partition means something never +harness_selftest premise: the ledger itself is a tracked file, not a variable never +harness_selftest premise: the ledger tool exists never +harness_selftest premise: the log report is a function that can be fed a fixture never +harness_selftest premise: the major-verdict branch was extracted, not an empty range never +harness_selftest premise: the major-verdict mapping evalled out of the runner is callable never +harness_selftest premise: the mode inventory is where this part thinks it is never +harness_selftest premise: the mutation applied -- the twin no longer sorts its inputs never +harness_selftest premise: the nightly paragraph was located and is not empty never +harness_selftest premise: the nightly workflow and the testing doc are both present never +harness_selftest premise: the observation reader evalled out of the runner is callable never +harness_selftest premise: the one fingerprint implementation is where this part thinks it is never +harness_selftest premise: the parts directory exists and was sourced never +harness_selftest premise: the population is the real test directory, not an empty glob never +harness_selftest premise: the population reconciliation is callable never +harness_selftest premise: the probe ran every helper shape once never +harness_selftest premise: the probe run skipped every major never +harness_selftest premise: the pytest cluster helper is where this part thinks it is never +harness_selftest premise: the pytest corpus is where this part thinks it is never +harness_selftest premise: the pytest layer is where this part thinks it is never +harness_selftest premise: the pytest layer states one too never +harness_selftest premise: the reader still finds a totals line when one is there never +harness_selftest premise: the real prober was restored, or every check after this lies never +harness_selftest premise: the real suite ran and reached its summary never +harness_selftest premise: the reconciliation evalled out of the runner is callable never +harness_selftest premise: the redirect, the suite invocation and the copy-back were located never +harness_selftest premise: the registered list is not empty, so the partition means something never +harness_selftest premise: the reverse sweep reads backticked names at all never +harness_selftest premise: the runner answered --list-suites, so the two checks below mean something never +harness_selftest premise: the runner defines the classifier this part is about to eval never +harness_selftest premise: the runner defines the declaration reader this part evals never +harness_selftest premise: the runner defines the observation reader this part evals never +harness_selftest premise: the runner defines the population reconciliation never +harness_selftest premise: the runner defines the reconciliation this part evals never +harness_selftest premise: the runner's collect loop was extracted, not an empty range never +harness_selftest premise: the same function returns a fingerprint for a real tree never +harness_selftest premise: the selftest has a workdir to build fixtures in never +harness_selftest premise: the set_options sweep read a substantial number of calls never +harness_selftest premise: the source tree is a git checkout never +harness_selftest premise: the sourced parts are inside the population, not pruned never +harness_selftest premise: the spelling fixture fingerprints at all never +harness_selftest premise: the stamp writer is a function that can be exercised never +harness_selftest premise: the start-failure path still exists to be judged never +harness_selftest premise: the stub frees exactly one port, 500 past the floor never +harness_selftest premise: the stub really does refuse the top of the band never +harness_selftest premise: the sub-suite failed, so its summary ran never +harness_selftest premise: the sweep finds the call sites it is meant to police never +harness_selftest premise: the sweep found the corpus rather than an empty glob never +harness_selftest premise: the sweep read lines to classify never +harness_selftest premise: the sweep read the corpus and found sites to classify never +harness_selftest premise: the sweep reads a population of scripts, not an empty find never +harness_selftest premise: the tree fingerprints to something when it is readable never +harness_selftest premise: the tree really contains continued diff_query calls to join never +harness_selftest premise: the twin script was written and is runnable never +harness_selftest premise: the unprivileged read agrees while everything is readable never +harness_selftest premise: the unsorted twin is callable never +harness_selftest premise: the verdict is composed somewhere it can be judged never +harness_selftest premise: the writer wrote a stamp at all never +harness_selftest premise: there are workflow files to search never +harness_selftest premise: while a real assignment on the same line shape IS counted never +harness_selftest premise: while the real body satisfies all three, so the greps work never +harness_selftest premise: while the real layer satisfies that same arm never +harness_selftest premise: while the real module satisfies the derivation arm never +harness_selftest renaming a source file moves the fingerprint too never +harness_selftest running the real loop over both fixtures fails the major never +harness_selftest section 1a's document total is the sum of its two sections never +harness_selftest section 1a's not-refused total is the count of ids in section 3 never +harness_selftest section 1a's refused total is the count of ids in section 2 never +harness_selftest section 2's opening states the counted number of refused modes never +harness_selftest so the tree still fingerprints from its root files alone never +harness_selftest so the verdict is fresh, not unknown never +harness_selftest so the verdict is unknown -- UNVERIFIED -- and never stale never +harness_selftest sorted_projection's two comparisons are ordered, its subject being order never +harness_selftest squatter survived untouched never +harness_selftest suite did not settle on the squatter's port never +harness_selftest suite's cluster is its own never +harness_selftest suite's own objects are visible to it never +harness_selftest the admitted gap is the run total minus what is written down never +harness_selftest the build path asks pgc_build_needs_clean rather than merely naming it never +harness_selftest the census reads a run's records never +harness_selftest the closing paragraph states the counted number too never +harness_selftest the committed budget matches the committed ledger's debt never +harness_selftest the committed budget names both debts never +harness_selftest the copy-back refuses a destination outside the tree (#740) never +harness_selftest the counter counts a fixture's section 2 never +harness_selftest the counter counts a fixture's section 3 never +harness_selftest the counter stops at the next heading never +harness_selftest the counters are returned beside their objects before the refusal (#740) never +harness_selftest the coverage runner refuses zero counters before it calls lcov (#740) never +harness_selftest the coverage runner's not_a_suite agrees with the selftest's, both ways (#741) never +harness_selftest the debt file is in the tree never +harness_selftest the driver holds no checks; they all live in parts never +harness_selftest the driver sources the parts by glob, not by a list never +harness_selftest the empty-plan refusal precedes the arm it protects never +harness_selftest the failure path asks pgc_start_log_report for the reason never +harness_selftest the fatal pattern matches a library that will not load never +harness_selftest the fingerprint derives its build directories from a Makefile on disk never +harness_selftest the fingerprint is the hash of the manifest never +harness_selftest the fixed fingerprint equals what the previous implementation produced never +harness_selftest the grep -q shape is the one that gets this wrong under pipefail never +harness_selftest the hash mixes in each file's path relative to the tree, not its name never +harness_selftest the identity catches comm reading unsorted input never +harness_selftest the installed .so is the one this run built never +harness_selftest the layer ends a session by setting its exit status never +harness_selftest the layer prints the unrunnable reason in lib.sh's shape never +harness_selftest the layer still writes the unrunnable state never +harness_selftest the ledger partitions into observed and never never +harness_selftest the loop delegates each verdict to pgc_tally_suite never +harness_selftest the manifest is tree-relative, never absolute never +harness_selftest the manifest names every file the fingerprint hashes never +harness_selftest the module imports nothing from the pytest tree never +harness_selftest the ordered oracle keeps the empty-result sentinel never +harness_selftest the ordered oracle keeps the unique query-error sentinel never +harness_selftest the ordered oracle numbers the rows as they arrive never +harness_selftest the ordered oracle orders by row_number, so it keeps the query's order never +harness_selftest the original rule flags a bump that records no outcome never +harness_selftest the ownership claim is made when a squatter held the port every time never +harness_selftest the partition over the real suite list adds up never +harness_selftest the per-suite escape hatch PGC_EXTRA_CONF is still applied to the config never +harness_selftest the population partitions, and prints inputs == sum(buckets) never +harness_selftest the port is named either way never +harness_selftest the probe is written outside the live source tree never +harness_selftest the pytest helper keeps no private fingerprint implementation never +harness_selftest the reader accepts the line the producer actually emits never +harness_selftest the reader does not answer yes for every registered suite never +harness_selftest the reconciliation prints inputs == sum(buckets) never +harness_selftest the record cannot introduce a suite the source never declared never +harness_selftest the record count equals the counter the summary reports never +harness_selftest the refusal looks in GCOV_PREFIX before the tree-wide walk (#740) never +harness_selftest the report names each hashed file never +harness_selftest the report names the symbol that was actually missing never +harness_selftest the report states how many files it hashed never +harness_selftest the row's value is read, not a digit inside its label never +harness_selftest the runner calls a clean exit a pass never +harness_selftest the runner calls the population reconciliation never +harness_selftest the runner calls the reconciliation, not merely defines it never +harness_selftest the runner calls the record reconciliation, not merely defines it never +harness_selftest the runner classifies the file that suite actually produced never +harness_selftest the runner defines the record reconciliation never +harness_selftest the runner's INCOMPLETE branch calls the mapping rather than a local flag never +harness_selftest the same tree fingerprints the same twice never +harness_selftest the sanitizer subset runs every suite that drives the encoding selftest never +harness_selftest the set oracle orders by the rendered row, so it is order-blind never +harness_selftest the shared cluster config sets no pgcolumnar.* GUC never +harness_selftest the skip branch records the suite it did not dispatch never +harness_selftest the stamp lib.sh writes is exactly the major never +harness_selftest the stamp writer reports failure on an unwritable target never +harness_selftest the stripper hides no pgc_summary call in any registered suite never +harness_selftest the stronger rule flags that same allowed bump, which is the change never +harness_selftest the suite list is sorted in C order, so two new suites land in different places never +harness_selftest the summary path asks pgc_fatal_pattern rather than hardcoding it never +harness_selftest the summary reconciles the three states against the total never +harness_selftest the sweep catches a producer that is neither echo nor printf never +harness_selftest the sweep counts the fixture's tests and files never +harness_selftest the sweep does not mistake the || operator for a pipe never +harness_selftest the sweep's pattern sees both lines of the probe never +harness_selftest the two collapse to one row, which is the loss being reported never +harness_selftest the two harnesses agree on the INCOMPLETE exit code never +harness_selftest the unrunnable check names itself, its reason code and its detail never +harness_selftest the writer writes the file the reader looks for never +harness_selftest the zero-counter guard counts the directory lcov captures (#740) never +harness_selftest two installations of one major get different stamp paths never +harness_selftest two unreadable pg_configs do not alias onto one stamp never +harness_selftest while a pass does not never +harness_selftest with an incomplete suite in the tally the major reports FAIL never diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt new file mode 100644 index 00000000..c7e49a16 --- /dev/null +++ b/test/check_ledger_budget.txt @@ -0,0 +1,28 @@ +# How much of the check corpus has never been seen red -- DEBT, in a tracked +# file so a change to it is a diff a reviewer sees. +# +# An environment variable would not be. PGC_SKIP_TIMING is the precedent: set in +# two workflow files, it suppressed whole suites for months and no diff ever +# showed it. +# +# BOTH NUMBERS MAY ONLY GO DOWN. A change that raises either is a change that +# adds debt, and it must read as exactly that in review rather than as a passing +# gate. +# +# checks_never_observed_red +# Checks in test/check_ledger.tsv that no recorded run has ever seen go red. +# It does NOT mean they cannot fail -- that is a stronger claim needing a named +# mutation, and this ledger does not make it. It means nothing has attacked +# them yet, which is worth knowing on its own. +# +# It drains from the project's actual failures as well as from deliberate +# mutation runs: every real CI red, every flake, every bisect fills the ledger, +# and those arrive whether anyone remembers to run something or not. +# +# suites_not_covered +# Registered suites with no rows in the ledger at all. Their checks are +# invisible to everything above: the gate cannot refuse a new check in a suite +# it has never seen. Counted separately so that "we ledger 605 checks" cannot +# read as "we ledger the corpus". +checks_never_observed_red 608 +suites_not_covered 250 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py new file mode 100755 index 00000000..cdab1ac0 --- /dev/null +++ b/test/pgc_ledger.py @@ -0,0 +1,265 @@ +#!/usr/bin/env python3 +"""The mutation ledger: which checks have ever been seen red, and under what. + +Nothing recorded whether a check had ever been red. That is the gap that let 39 +checks across 35 suites ship unable to fail, three of them inside the suite whose +whole purpose is to stop exactly that. The gate answered "did anything print FAIL" +and had never answered "could anything print FAIL". + +WHAT THIS LEDGER CLAIMS, AND WHAT IT DOES NOT +--------------------------------------------- +It records that a named check WAS OBSERVED RED in a recorded run. It does NOT +claim the check is proven able to fail: that is a stronger statement, it needs a +named mutation applied deliberately, and conflating the two would put a claim in +the ledger that nothing measured -- the `defeated: 0` shape from VACUITY_MODES +section 1, a number that reads as evidence and is not. + +So v1 fills the observed column honestly and leaves the rest as debt, counted. A +ledger whose entries all read `never` is a measurement of how much of the suite +has never been attacked, and that measurement is worth having on day one. + +WHAT FILLS IT +------------- +Not only deliberate mutation runs. Every real CI red fills it, every flake, every +bisect -- and those arrive whether anyone remembers or not. A mutation run is the +deliberate accelerator, not the only source. + +THE MUTATION COLUMN +------------------- +Present from v1 with nothing filling it automatically, because adding a column +later means rewriting every entry. If an entry can record WHICH mutation reddened +a check, the catalogue a mutation gate would need builds itself out of work people +already do by hand. + +FORMAT +------ +Tab separated, one row per check, sorted: + + suite check name last observed red mutation + +`last observed red` is a date, or the literal `never`. `mutation` is free text or +empty. Both are written by this tool, never by hand. +""" + +import argparse +import pathlib +import sys + +NEVER = "never" + + +def read_records(paths): + """(suite, name, verdict) for every RESULT line in the given logs.""" + out = [] + for p in paths: + try: + text = pathlib.Path(p).read_text(errors="replace") + except OSError: + continue + for line in text.splitlines(): + if not line.startswith("RESULT\t"): + continue + f = line.split("\t") + if len(f) < 4: + continue + out.append((f[1], f[2], f[3])) + return out + + +def read_ledger(path): + """{(suite, name): [last_red, mutation]} from a ledger file.""" + rows = {} + p = pathlib.Path(path) + if not p.exists(): + return rows + for line in p.read_text(errors="replace").splitlines(): + if not line.strip() or line.startswith("#"): + continue + f = line.split("\t") + while len(f) < 4: + f.append("") + rows[(f[0], f[1])] = [f[2] or NEVER, f[3]] + return rows + + +def write_ledger(path, rows): + lines = [ + "\t".join((suite, name, v[0], v[1])) + for (suite, name), v in sorted(rows.items()) + ] + pathlib.Path(path).write_text("\n".join(lines) + ("\n" if lines else "")) + + +def cmd_census(args): + for suite, name, verdict in read_records(args.logs): + print(f"{suite}\t{name}\t{verdict}") + return 0 + + +def cmd_merge(args): + rows = read_ledger(args.ledger) + for suite, name, verdict in read_records(args.logs): + key = (suite, name) + if key not in rows: + # A check this ledger has never seen enters as DEBT. A green run has + # observed nothing go red, so merging one must never record a red + # observation -- otherwise an ordinary CI run retires the debt it + # exists to count. + rows[key] = [NEVER, ""] + if verdict == "FAIL": + rows[key][0] = args.date + if args.mutation: + rows[key][1] = args.mutation + write_ledger(args.ledger, rows) + + # A DUPLICATED NAME SHARES ONE LEDGER ROW, so one of the two going red marks + # BOTH as observed red -- a claim about a check nothing attacked, which is + # precisely what this ledger must not make. It cannot be fixed by keying + # harder without a synthetic id someone would maintain, so it is reported. + records = read_records(args.logs) + counts = {} + for suite, name, _ in records: + counts[(suite, name)] = counts.get((suite, name), 0) + 1 + dupes = sorted(k for k, c in counts.items() if c > 1) + for suite, name in dupes: + print(f" duplicate check name, so one ledger row covers " + f"{counts[(suite, name)]}: {suite}\t{name}") + + seen = len({(s, n) for s, n, _ in read_records(args.logs)}) + red = sum(1 for v in rows.values() if v[0] != NEVER) + print(f" ledger: rows={len(rows)} | seen this run={seen}, " + f"observed red ever={red}, never={len(rows) - red}") + return 0 + + +def cmd_rename_scan(args): + """A name that appeared while another disappeared is probably a rename. + + Keyed by the display string, a rename loses the check's history and reads + exactly like a brand-new check that has never been red -- the one state this + ledger exists to distinguish. It cannot be prevented without a synthetic id + that someone would have to maintain, so it is DETECTED and named instead of + silently resetting a count to `never`. + + Both directions are required. A check merely added, or merely removed, is not + a rename, and reporting one on every new check is noise that gets the whole + thing ignored. + """ + rows = read_ledger(args.ledger) + now = {(s, n) for s, n, _ in read_records(args.logs)} + suites = {s for s, _ in now} + known = {(s, n) for (s, n) in rows if s in suites} + + appeared = sorted(now - known) + vanished = sorted(known - now) + rc = 0 + if appeared and vanished: + # Only pair within a suite, and only report while both sides remain. + for (s_a, n_a), (s_v, n_v) in zip(appeared, vanished): + if s_a != s_v: + continue + was = rows.get((s_v, n_v), [NEVER, ""])[0] + print(f" possible rename: {n_v} -> {n_a} " + f"(in {s_a}, history: last red {was})") + rc = 1 + print(f" rename scan: appeared={len(appeared)}, vanished={len(vanished)}") + return rc + + +def _read_budget(path): + out = {} + p = pathlib.Path(path) + if not p.exists(): + return out + for line in p.read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + parts = line.split() + if len(parts) == 2 and parts[1].isdigit(): + out[parts[0]] = int(parts[1]) + return out + + +def cmd_gate(args): + rows = read_ledger(args.ledger) + budget = _read_budget(args.budget) + seen = {(s, n) for s, n, _ in read_records(args.logs)} + + # A check the ledger has never heard of is NEW. The allowlist exists so a + # gate that fails on 3,762 unledgered sites is not what lands -- but a new + # one must not enter as silent debt either. + unknown = sorted(seen - set(rows)) + rc = 0 + for suite, name in unknown: + print(f" not in the ledger: {suite}\t{name}") + rc = 1 + + never = sum(1 for v in rows.values() if v[0] == NEVER) + want = budget.get("checks_never_observed_red") + print(f" ledger gate: rows={len(rows)} | never observed red={never}, " + f"budget={want if want is not None else 'unset'}, new={len(unknown)}") + if want is None: + print(" the budget file names no checks_never_observed_red, so nothing bounds the debt") + return 1 + if never > want: + print(f" checks_never_observed_red: {never} exceeds the budget of {want}") + rc = 1 + + # The SECOND debt, and the one that is easy to forget: a suite with no rows + # in the ledger is not covered at all, and its checks are invisible to + # everything above -- the gate cannot refuse a new check in a suite it has + # never seen. Counting it separately keeps "we ledger 605 checks" from + # reading as "we ledger the corpus". + if args.registered: + registered = {l.strip() for l in pathlib.Path(args.registered).read_text().split() + if l.strip()} + covered = {s for s, _ in rows} + uncovered = sorted(registered - covered) + want_s = budget.get("suites_not_covered") + print(f" ledger coverage: registered={len(registered)} | " + f"covered={len(registered) - len(uncovered)}, not covered={len(uncovered)}, " + f"budget={want_s if want_s is not None else 'unset'}") + if want_s is None: + print(" the budget file names no suites_not_covered, so nothing bounds the coverage") + return 1 + if len(uncovered) > want_s: + print(f" suites_not_covered: {len(uncovered)} exceeds the budget of {want_s}") + rc = 1 + return rc + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + sub = ap.add_subparsers(dest="cmd", required=True) + + c = sub.add_parser("census", help="print suite/name/verdict for each RESULT line") + c.add_argument("logs", nargs="+") + c.set_defaults(fn=cmd_census) + + m = sub.add_parser("merge", help="merge a run's records into the ledger") + m.add_argument("--ledger", required=True) + m.add_argument("--date", default="unknown") + m.add_argument("--mutation", default="") + m.add_argument("logs", nargs="+") + m.set_defaults(fn=cmd_merge) + + r = sub.add_parser("rename-scan", help="report names that look renamed") + r.add_argument("--ledger", required=True) + r.add_argument("logs", nargs="+") + r.set_defaults(fn=cmd_rename_scan) + + g = sub.add_parser("gate", help="refuse new checks and debt over budget") + g.add_argument("--ledger", required=True) + g.add_argument("--budget", required=True) + g.add_argument("--registered", default="", + help="file listing every registered suite, for the coverage debt") + g.add_argument("logs", nargs="+") + g.set_defaults(fn=cmd_gate) + + args = ap.parse_args(argv) + return args.fn(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index a6648765..226698de 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -61,9 +61,10 @@ behaviour, the source of that number is named. - [13. test_hilbert_locality.py: what the Hilbert curve buys](#13-test_hilbert_localitypy-what-the-hilbert-curve-buys) - [14. test_suite_accounting.py: the matrix accounting for its own suites](#14-test_suite_accountingpy-the-matrix-accounting-for-its-own-suites) - [15. test_check_results_are_machine_readable.py: one counter, one record](#15-test_check_results_are_machine_readablepy-one-counter-one-record) -- [16. Adding a test](#16-adding-a-test) -- [17. What this corpus does NOT yet refuse](#17-what-this-corpus-does-not-yet-refuse) -- [18. Traps this corpus records](#18-traps-this-corpus-records) +- [16. test_mutation_ledger.py: which checks have ever been red](#16-test_mutation_ledgerpy-which-checks-have-ever-been-red) +- [17. Adding a test](#17-adding-a-test) +- [18. What this corpus does NOT yet refuse](#18-what-this-corpus-does-not-yet-refuse) +- [19. Traps this corpus records](#19-traps-this-corpus-records) ## 1. How to read a test in here @@ -1209,7 +1210,93 @@ a helper that prints an outcome without recording it. A log with no count at all reached its summary -- a different fault from a miscount, and not a clean reconciliation. -## 16. Adding a test +## 16. test_mutation_ledger.py: which checks have ever been red + +Nothing recorded whether a check had ever been red. That is the gap that let **39 +checks across 35 suites** ship unable to fail, three of them inside the suite whose +whole purpose is to stop exactly that. The gate answered *did anything print FAIL* and +had never answered *could anything print FAIL*. + +### What this ledger claims, and what it does not + +It records that a named check **was observed red in a recorded run**. It does **not** +claim the check is proven able to fail — that is a stronger statement, it needs a named +mutation applied deliberately, and conflating the two would put a claim in the ledger +that nothing measured. That is the `defeated: 0` shape from `VACUITY_MODES` section 1: a +number that reads as evidence and is not. + +So v1 fills the observed column honestly and leaves the rest as debt, counted. **Every +entry currently reads `never`**, and that is the finding rather than an embarrassment: a +ledger of all-`never` is a measurement of how much of the corpus has never been +attacked. + +### What fills it + +Not only deliberate mutation runs. Every real CI red fills it, every flake, every +bisect — and those arrive whether anyone remembers to run something or not. A mutation +run is the deliberate accelerator, not the only source. + +### The format + +``` +suite check name last observed red mutation +``` + +`last observed red` is a date or the literal `never`. The **mutation column exists from +v1 with nothing filling it automatically**, because adding a column later means +rewriting every entry — and if an entry can record *which* mutation reddened a check, +the catalogue a mutation gate would need builds itself out of work people already do by +hand. + +Two tracked files carry the debt, `test/check_ledger.tsv` and +`test/check_ledger_budget.txt`, so a change to either is a diff a reviewer sees. +`PGC_SKIP_TIMING` is the precedent for why it is not an environment variable: set in two +workflow files, it suppressed whole suites for months and no diff ever showed it. + +### `test_a_green_run_records_debt_and_never_a_red_observation` + +The arm that matters most. A green run has seen nothing go red, so merging one must +never record a red observation — otherwise an ordinary CI run retires the debt the +ledger exists to count. + +### `test_a_red_observation_is_dated_and_survives_a_later_green_run` + +The ledger records that a check **was** seen red, which stays true. + +### `test_the_mutation_column_exists_from_v1` + +Empty when nothing named a mutation; recorded against the check that reddened and not +against one that stayed green. + +### `test_a_rename_is_reported_rather_than_silently_resetting_history` + +Keyed by the display string, a rename loses the check's history and reads exactly like a +brand-new check that has never been red — the one state this ledger exists to +distinguish. It cannot be prevented without a synthetic id someone would have to +maintain, so it is **detected**: a name that appeared while another disappeared is +named. Both directions are required, or every new check is reported as a rename and the +whole thing gets ignored. + +### `test_a_duplicated_check_name_shares_one_row_and_is_reported` + +Two checks with the same name in one suite share a ledger row, so one going red marks +**both** as observed red — a claim about a check nothing attacked. Reported rather than +prevented, for the same reason as the rename. The real corpus carries four today, which +is how this was noticed: 609 records reduced to 605 rows. + +### `test_the_gate_refuses_a_check_the_ledger_has_never_seen` + +A gate that fails on 3,762 unledgered sites is one somebody disables under deadline, and +then we are back at `PGC_SKIP_TIMING` with extra steps. The budget grandfathers what +exists; a new check must not enter as silent debt. + +### `test_the_committed_ledger_and_budget_agree` + +If they disagree, one was edited by hand. `suites_not_covered` is counted separately so +that "we ledger 605 checks" cannot read as "we ledger the corpus" — 250 of 251 suites +have no rows at all. + +## 17. Adding a test 0. **Write it twice.** Every test in this tree ships as a `.sh` suite and a pytest test **in the same change** (jd, 2026-09-09). Not ported later, not one or the @@ -1236,7 +1323,7 @@ reconciliation. failed the selftest on both majors of the matrix, which is how it was found. A new directory under `test/` inherits every rule the old ones follow. -## 17. What this corpus does NOT yet refuse +## 18. What this corpus does NOT yet refuse `VACUITY_MODES.md` is the inventory: 79 ways a pytest harness can report a pass while asserting nothing, 73 of them demonstrated by an actual run. **This layer refuses 25 @@ -1248,7 +1335,7 @@ Read it before adding a test. The gaps most likely to affect a new test are that same family satisfies it, and that a write is not required to have written anything. Both are named there with the refusal each needs. -## 18. Traps this corpus records +## 19. Traps this corpus records Recorded because each one produced a confident wrong result before it was caught, and all are the same family as the defect the layer exists to prevent. diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py new file mode 100644 index 00000000..4760012f --- /dev/null +++ b/test/pytest/test_mutation_ledger.py @@ -0,0 +1,229 @@ +"""The mutation ledger: which checks have ever been seen red, and under what. + +Nothing recorded whether a check had ever been red. That is the gap that let 39 +checks across 35 suites ship unable to fail, three of them inside the suite whose +whole purpose is to stop exactly that. The gate answered *did anything print FAIL* +and had never answered *could anything print FAIL*. + +**What this ledger claims, and what it does not.** It records that a named check was +OBSERVED RED in a recorded run. It does not claim the check is proven able to fail: +that is a stronger statement, it needs a named mutation applied deliberately, and +conflating the two would put a claim in the ledger that nothing measured -- the +`defeated: 0` shape from `VACUITY_MODES` section 1, a number that reads as evidence +and is not. + +So v1 fills the observed column honestly and leaves the rest as debt, counted. A +ledger whose entries all read `never` is a measurement of how much of the corpus has +never been attacked, and that measurement is worth having on day one. + +**What fills it.** Not only deliberate mutation runs. Every real CI red fills it, +every flake, every bisect -- and those arrive whether anyone remembers or not. A +mutation run is the deliberate accelerator, not the only source. + +These tests drive the real tool, for the same reason the other two files drive the +real shell: a Python twin of a Python tool would agree with itself. +""" + +import pathlib +import subprocess + +REPO = pathlib.Path(__file__).resolve().parents[2] +LEDGER_TOOL = REPO / "test" / "pgc_ledger.py" +RUNNER = REPO / "test" / "run_all_versions.sh" + + +def _run(*args): + r = subprocess.run(["python3", str(LEDGER_TOOL), *args], + capture_output=True, text=True) + return r.stdout + r.stderr, r.returncode + + +def _write(tmp_path, name, text): + p = tmp_path / name + p.write_text(text) + return str(p) + + +def _rows(path): + return [l.split("\t") for l in pathlib.Path(path).read_text().splitlines() if l] + + +GREEN = ("RESULT\tdemo\tfirst check\tPASS\t\n" + "RESULT\tdemo\tsecond check\tPASS\t\n" + "checks run: 2\n") +RED = ("RESULT\tdemo\tfirst check\tFAIL\t\n" + "RESULT\tdemo\tsecond check\tPASS\t\n" + "checks run: 2\n") + + +def test_a_green_run_records_debt_and_never_a_red_observation(tmp_path, expect): + """The arm that matters most. + + A green run has seen nothing go red, so merging one must never record a red + observation -- otherwise an ordinary CI run retires the debt the ledger exists + to count. + """ + ledger = _write(tmp_path, "l.tsv", "") + log = _write(tmp_path, "green.log", GREEN) + _run("merge", "--ledger", ledger, log) + rows = _rows(ledger) + expect.num(len(rows), 2, "merging a green run records both checks") + expect.text(",".join(sorted({r[2] for r in rows})), "never", + "and records neither as ever having been red") + + +def test_a_red_observation_is_dated_and_survives_a_later_green_run(tmp_path, expect): + """The ledger records that a check WAS seen red, which stays true.""" + ledger = _write(tmp_path, "l.tsv", "") + _run("merge", "--ledger", ledger, _write(tmp_path, "g.log", GREEN)) + _run("merge", "--ledger", ledger, "--date", "2026-09-10", + _write(tmp_path, "r.log", RED)) + by = {r[1]: r[2] for r in _rows(ledger)} + expect.text(by["first check"], "2026-09-10", + "a check observed red gains the date it was seen") + expect.text(by["second check"], "never", + "and one that stayed green keeps its debt") + + _run("merge", "--ledger", ledger, "--date", "2026-09-11", + _write(tmp_path, "g2.log", GREEN)) + expect.text({r[1]: r[2] for r in _rows(ledger)}["first check"], "2026-09-10", + "a later green run does not erase an observation") + + +def test_the_mutation_column_exists_from_v1(tmp_path, expect): + """Present with nothing filling it automatically, because adding a column later + means rewriting every entry. + + If an entry can record WHICH mutation reddened a check, the catalogue a mutation + gate would need builds itself out of work people already do by hand. + """ + ledger = _write(tmp_path, "l.tsv", "") + _run("merge", "--ledger", ledger, "--date", "2026-09-10", + _write(tmp_path, "r.log", RED)) + expect.num(len([r for r in _rows(ledger) if len(r) != 4]), 0, + "every row carries four fields, the fourth being the mutation") + expect.text("[" + {r[1]: r[3] for r in _rows(ledger)}["first check"] + "]", "[]", + "and it is empty when nothing named a mutation") + + _run("merge", "--ledger", ledger, "--date", "2026-09-10", + "--mutation", "SAOP limit 128 -> 0", _write(tmp_path, "r2.log", RED)) + by = {r[1]: r[3] for r in _rows(ledger)} + expect.text(by["first check"], "SAOP limit 128 -> 0", + "a named mutation is recorded against the check that reddened") + expect.text("[" + by["second check"] + "]", "[]", + "and not against one that stayed green") + + +def test_a_rename_is_reported_rather_than_silently_resetting_history(tmp_path, expect): + """Keyed by the display string, a rename loses history and reads exactly like a + brand-new check that has never been red -- the one state this ledger exists to + distinguish. It is detected and named instead. + + Both directions are required: a check merely added, or merely removed, is not a + rename, and reporting one on every new check is noise that gets it ignored. + """ + ledger = _write(tmp_path, "l.tsv", "") + before = ("RESULT\tdemo\tthe old name\tFAIL\t\n" + "RESULT\tdemo\ta stable check\tPASS\t\nchecks run: 2\n") + _run("merge", "--ledger", ledger, "--date", "2026-09-01", + _write(tmp_path, "b.log", before)) + + after = ("RESULT\tdemo\tthe new name\tPASS\t\n" + "RESULT\tdemo\ta stable check\tPASS\t\nchecks run: 2\n") + out, _ = _run("rename-scan", "--ledger", ledger, _write(tmp_path, "a.log", after)) + expect.num(out.count("possible rename: the old name -> the new name"), 1, + "a name that appeared while another disappeared is reported") + expect.num(out.count("a stable check"), 0, "and the stable check is not") + + added = after.replace("the new name", "the old name") + "" + added = ("RESULT\tdemo\tthe old name\tPASS\t\n" + "RESULT\tdemo\ta stable check\tPASS\t\n" + "RESULT\tdemo\ta genuinely new check\tPASS\t\nchecks run: 3\n") + out, _ = _run("rename-scan", "--ledger", ledger, _write(tmp_path, "add.log", added)) + expect.num(out.count("possible rename"), 0, + "a check merely added is not reported as a rename") + + removed = "RESULT\tdemo\ta stable check\tPASS\t\nchecks run: 1\n" + out, _ = _run("rename-scan", "--ledger", ledger, _write(tmp_path, "rm.log", removed)) + expect.num(out.count("possible rename"), 0, + "nor is one merely removed") + + +def test_a_duplicated_check_name_shares_one_row_and_is_reported(tmp_path, expect): + """Two checks with the same name in one suite share a ledger row, so one going + red marks BOTH as observed red -- a claim about a check nothing attacked, which + is exactly what this ledger must not make. + + It cannot be fixed by keying harder without a synthetic id someone would + maintain, so it is reported. The real corpus carries four today, which is how + this was noticed: 609 records reduced to 605 rows. + """ + ledger = _write(tmp_path, "l.tsv", "") + dupe = ("RESULT\tdemo\tthe same name\tPASS\t\n" + "RESULT\tdemo\tthe same name\tFAIL\t\n" + "RESULT\tdemo\ta unique name\tPASS\t\nchecks run: 3\n") + out, _ = _run("merge", "--ledger", ledger, "--date", "2026-09-10", + _write(tmp_path, "d.log", dupe)) + expect.num(out.count("duplicate check name, so one ledger row covers 2: " + "demo\tthe same name"), 1, + "a duplicated check name is reported by name") + expect.num(out.count("a unique name"), 0, "and a unique one is not") + expect.num(len(_rows(ledger)), 2, + "the two collapse to one row, which is the loss being reported") + + +def test_the_gate_refuses_a_check_the_ledger_has_never_seen(tmp_path, expect): + """A gate that fails on 3,762 unledgered sites is one somebody disables under + deadline, and then we are back at PGC_SKIP_TIMING with extra steps. So the + budget grandfathers what exists -- but a NEW check must not enter as silent + debt either.""" + ledger = _write(tmp_path, "l.tsv", "") + _run("merge", "--ledger", ledger, _write(tmp_path, "g.log", GREEN)) + budget = _write(tmp_path, "b.txt", "checks_never_observed_red 2\n") + log = _write(tmp_path, "g2.log", GREEN) + expect.num(_run("gate", "--ledger", ledger, "--budget", budget, log)[1], 0, + "a run whose debt is within budget passes") + + over = _write(tmp_path, "b2.txt", "checks_never_observed_red 1\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", over, log) + expect.num(rc, 1, "and one over budget does not") + expect.num(out.count("checks_never_observed_red: 2 exceeds the budget of 1"), 1, + "and the gate says which number was exceeded, by how much") + + newer = _write(tmp_path, "n.log", GREEN + "RESULT\tdemo\tbrand new\tPASS\t\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", budget, newer) + expect.num(rc, 1, "a check the ledger has never seen is refused") + expect.num(out.count("not in the ledger: demo\tbrand new"), 1, + "and it is named, so the author knows which one") + + +def test_the_committed_ledger_and_budget_agree(expect): + """Both are tracked files, so a change to either is a diff a reviewer sees. If + they disagree, one of them was edited by hand -- the failure this whole design + refuses.""" + ledger = REPO / "test" / "check_ledger.tsv" + budget = REPO / "test" / "check_ledger_budget.txt" + expect.text("yes" if ledger.exists() else "no", "yes", "the ledger is in the tree") + expect.text("yes" if budget.exists() else "no", "yes", "the budget is in the tree") + + rows = [l.split("\t") for l in ledger.read_text().splitlines() if l] + never = [r for r in rows if r[2] == "never"] + red = [r for r in rows if r[2] != "never"] + print(f" ledger: inputs={len(rows)} | observed red={len(red)}, " + f"never={len(never)} | sum={len(red) + len(never)}") + expect.num(len(red) + len(never), len(rows), "the ledger partitions") + expect.at_least(len(rows), 1, "premise: it is not empty") + + nums = {} + for line in budget.read_text().splitlines(): + parts = line.split() + if len(parts) == 2 and parts[1].isdigit() and not line.startswith("#"): + nums[parts[0]] = int(parts[1]) + expect.num(nums.get("checks_never_observed_red", -1), len(never), + "the committed budget matches the committed ledger's debt") + + listed = subprocess.run(["bash", str(RUNNER), "--list-suites"], + capture_output=True, text=True).stdout.split() + covered = {r[0] for r in rows} + expect.num(nums.get("suites_not_covered", -1), len(set(listed) - covered), + "and the coverage debt matches the suites with no rows") diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh new file mode 100644 index 00000000..5ad3051a --- /dev/null +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -0,0 +1,249 @@ +# ---- a check must have been seen red, or be counted as debt ----------------- +# +# Nothing records whether a check has ever been red. That is the gap that let 39 +# checks across 35 suites ship unable to fail, three of them inside this very +# suite. The gate answers "did anything print FAIL" and has never answered "could +# anything print FAIL". +# +# An audit fixes today; only a ledger keeps it fixed. And a ledger somebody +# maintains is a hand-maintained count, which this repository has spent a day +# proving the cost of: nine collisions on one written number, both sides wrong +# every time. So the ledger is DERIVED FROM RUNS. The only hand-written numbers +# are the two budgets, and they may only go down. +# +# WHAT THIS LEDGER CLAIMS, AND WHAT IT DOES NOT. It records that a named check +# WAS OBSERVED RED in a recorded run. It does NOT claim the check is proven able +# to fail: that is a stronger statement, it needs a named mutation applied +# deliberately, and conflating the two would put a claim in the ledger that +# nothing measured. v1 fills the observed column honestly and leaves the rest as +# debt, counted. +# --------------------------------------------------------------------------- + +_led="$PGC_TESTDIR/pgc_ledger.py" +_ledger="$PGC_TESTDIR/check_ledger.tsv" +_budget="$PGC_TESTDIR/check_ledger_budget.txt" + +check "premise: the ledger tool exists" "$([ -f "$_led" ] && echo yes || echo no)" "yes" +check "premise: the ledger itself is a tracked file, not a variable" \ + "$([ -f "$_ledger" ] && echo yes || echo no)" "yes" +check "premise: the budget is a tracked file too" \ + "$([ -f "$_budget" ] && echo yes || echo no)" "yes" + +_lw="$PGC_WORKDIR/ledger"; mkdir -p "$_lw" +_led_run() { python3 "$_led" "$@" 2>&1; } + +# ---- the census comes out of a run, not out of a list ----------------------- + +cat > "$_lw/green.log" <<'LOG' +RESULT demo first check PASS +RESULT demo second check PASS +checks run: 2 +LOG + +check "the census reads a run's records" \ + "$(_led_run census "$_lw/green.log" | wc -l)" "2" +check "and names the suite and the check, not just a count" \ + "$(_led_run census "$_lw/green.log" | head -1)" "demo first check PASS" + +# ---- merging a green run adds the checks as DEBT, not as proven ------------- +# +# The arm that matters. A green run has seen nothing go red, so merging one must +# never record a red observation. Anything else would let an ordinary CI run +# retire the debt it exists to count. + +: > "$_lw/ledger.tsv" +_led_run merge --ledger "$_lw/ledger.tsv" "$_lw/green.log" >/dev/null +check "merging a green run records both checks" \ + "$(grep -c . "$_lw/ledger.tsv")" "2" +check "and records neither as ever having been red" \ + "$(cut -f3 "$_lw/ledger.tsv" | sort -u | tr '\n' ' ')" "never " + +# ---- merging a run that DID go red records the observation ------------------ + +cat > "$_lw/red.log" <<'LOG' +RESULT demo first check FAIL +RESULT demo second check PASS +checks run: 2 +LOG + +_led_run merge --ledger "$_lw/ledger.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null +check "a check observed red gains the date it was seen" \ + "$(awk -F'\t' '$2=="first check"{print $3}' "$_lw/ledger.tsv")" "2026-09-10" +check "and a check that stayed green keeps its debt" \ + "$(awk -F'\t' '$2=="second check"{print $3}' "$_lw/ledger.tsv")" "never" + +# An observation is not undone by a later green run. The ledger records that the +# check WAS seen red, which stays true. +_led_run merge --ledger "$_lw/ledger.tsv" --date 2026-09-11 "$_lw/green.log" >/dev/null +check "a later green run does not erase an observation" \ + "$(awk -F'\t' '$2=="first check"{print $3}' "$_lw/ledger.tsv")" "2026-09-10" + +# ---- the gate: new checks must not be added to the debt silently ------------ +# +# A gate that fails on 3,762 unledgered checks is a gate somebody disables under +# deadline, and then we are back at PGC_SKIP_TIMING with extra steps. So the +# budget grandfathers what exists and refuses to grow. + +printf 'suites_not_covered 0\nchecks_never_observed_red 1\n' > "$_lw/budget.txt" +check "a run whose debt is within budget passes the gate" \ + "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/green.log" >/dev/null 2>&1 \ + && echo ok || echo over)" "ok" + +printf 'suites_not_covered 0\nchecks_never_observed_red 0\n' > "$_lw/budget.txt" +check "and one over budget does not" \ + "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/green.log" >/dev/null 2>&1 \ + && echo ok || echo over)" "over" +check "and the gate says which number was exceeded, by how much" \ + "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/green.log" 2>&1 \ + | grep -c 'checks_never_observed_red: 1 exceeds the budget of 0')" "1" + +# A check the run produced that the ledger has never heard of is the case the +# allowlist exists for: it is NEW, and it must not enter as silent debt. +printf 'suites_not_covered 0\nchecks_never_observed_red 1\n' > "$_lw/budget.txt" +cat > "$_lw/newcheck.log" <<'LOG' +RESULT demo first check PASS +RESULT demo second check PASS +RESULT demo a brand new check PASS +checks run: 3 +LOG +check "a check the ledger has never seen is refused, not absorbed" \ + "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/newcheck.log" >/dev/null 2>&1 \ + && echo ok || echo refused)" "refused" +check "and it is named, so the author knows which one" \ + "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/newcheck.log" 2>&1 \ + | grep -c 'not in the ledger: demo a brand new check')" "1" + +# ---- the budget may only go DOWN -------------------------------------------- +# +# Both numbers are debt. A change that raises either is a change that adds debt, +# and it must be visible in a diff as exactly that rather than as a passing gate. + +check "the committed budget names both debts" \ + "$(grep -cE '^(suites_not_covered|checks_never_observed_red) [0-9]+$' "$_budget")" "2" + +# ---- inputs == sum(buckets), over the real ledger --------------------------- + +_l_total="$(grep -c . "$_ledger" || true)" +_l_red="$(awk -F'\t' '$3!="never"' "$_ledger" | grep -c . || true)" +_l_never="$(awk -F'\t' '$3=="never"' "$_ledger" | grep -c . || true)" +echo " ledger: inputs=$_l_total | observed red=$_l_red, never=$_l_never | sum=$((_l_red + _l_never))" +check "the ledger partitions into observed and never" \ + "$((_l_red + _l_never))" "$_l_total" +check "premise: the ledger is not empty, so the partition means something" \ + "$([ "$_l_total" -gt 0 ] && echo yes || echo no)" "yes" + +# The committed budget must match the committed ledger. If it does not, one of +# the two was edited by hand -- which is the failure this whole design refuses. +check "the committed budget matches the committed ledger's debt" \ + "$(sed -n 's/^checks_never_observed_red //p' "$_budget")" "$_l_never" + +# ---- a rename is not a new check, and must not look like one ---------------- +# +# The ledger is keyed by check NAME, and check names in this harness are prose -- +# they are renamed freely, which is most of why #917 exists. So a rename loses +# the check's history and reads exactly like a brand-new check that has never +# been red, which is the ONE state the ledger exists to distinguish. +# +# Raised by OffgridwithJD, who also named the detector: a name appearing with no +# history in the same run another disappears is a rename, and the ledger should +# SAY so rather than quietly resetting a count to `never`. It is the same +# both-directions set comparison as the suite reconciliation, over check names. +# +# The alternative -- a synthetic stable id -- would have to be maintained, and +# this repository removed a hand-maintained list today for that exact reason. + +: > "$_lw/ren.tsv" +cat > "$_lw/before.log" <<'LOG' +RESULT demo the old name FAIL +RESULT demo a stable check PASS +checks run: 2 +LOG +_led_run merge --ledger "$_lw/ren.tsv" --date 2026-09-01 "$_lw/before.log" >/dev/null +check "premise: the check has history before the rename" \ + "$(awk -F'\t' '$2=="the old name"{print $3}' "$_lw/ren.tsv")" "2026-09-01" + +cat > "$_lw/after.log" <<'LOG' +RESULT demo the new name PASS +RESULT demo a stable check PASS +checks run: 2 +LOG +check "a name that appeared while another disappeared is reported as a rename" \ + "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/after.log" 2>&1 \ + | grep -c 'possible rename: the old name -> the new name')" "1" +check "and the stable check is not reported" \ + "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/after.log" 2>&1 \ + | grep -c 'a stable check')" "0" + +# The detector must not fire when a check is simply ADDED. Without this it names +# a rename on every new check, which is noise that gets it ignored. +cat > "$_lw/added.log" <<'LOG' +RESULT demo the old name PASS +RESULT demo a stable check PASS +RESULT demo a genuinely new check PASS +checks run: 3 +LOG +check "a check merely added is not reported as a rename" \ + "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/added.log" 2>&1 \ + | grep -c 'possible rename')" "0" + +# Nor when one is simply REMOVED. +cat > "$_lw/removed.log" <<'LOG' +RESULT demo a stable check PASS +checks run: 1 +LOG +check "a check merely removed is not reported as a rename either" \ + "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/removed.log" 2>&1 \ + | grep -c 'possible rename')" "0" + +# ---- the mutation field, present from v1 even though nothing fills it ------- +# +# If an entry can record WHICH mutation reddened a check, the mutation catalogue +# builds itself out of work people already do by hand -- the vacuity branches are +# writing nine to eleven per change tonight, each chosen to revert one property. +# OffgridwithJD's point, and the reason the column exists now: adding it later +# means rewriting every entry. +# +# NOTHING FILLS IT AUTOMATICALLY YET, and the arms say so rather than implying a +# capability that does not exist. + +: > "$_lw/mut.tsv" +_led_run merge --ledger "$_lw/mut.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null +check "every ledger row carries four fields, the fourth being the mutation" \ + "$(awk -F'\t' 'NF!=4' "$_lw/mut.tsv" | grep -c . || true)" "0" +check "and it is empty when nothing named a mutation" \ + "$(awk -F'\t' '$2=="first check"{print "[" $4 "]"}' "$_lw/mut.tsv")" "[]" + +_led_run merge --ledger "$_lw/mut.tsv" --date 2026-09-10 \ + --mutation 'PGCOLUMNAR_SAOP_ELEMENT_LIMIT 128 -> 0' "$_lw/red.log" >/dev/null +check "a merge that names its mutation records it against the check that reddened" \ + "$(awk -F'\t' '$2=="first check"{print $4}' "$_lw/mut.tsv")" "PGCOLUMNAR_SAOP_ELEMENT_LIMIT 128 -> 0" +check "and not against one that stayed green" \ + "$(awk -F'\t' '$2=="second check"{print "[" $4 "]"}' "$_lw/mut.tsv")" "[]" + +# ---- a duplicated check name shares one ledger row -------------------------- +# +# The ledger is keyed by (suite, name). Two checks with the same name in one +# suite therefore share a row, so ONE of them going red marks BOTH as observed +# red -- a claim about a check nothing attacked, which is exactly what this +# ledger must not make. +# +# It cannot be fixed by keying harder without a synthetic id someone would have +# to maintain. So it is REPORTED, and the number is printed rather than assumed: +# the selftest corpus carries some today, which is how this was noticed at all -- +# 609 records reduced to 605 rows. + +cat > "$_lw/dupe.log" <<'LOG' +RESULT demo the same name PASS +RESULT demo the same name FAIL +RESULT demo a unique name PASS +checks run: 3 +LOG +: > "$_lw/dupe.tsv" +check "a duplicated check name is reported by name" \ + "$(_led_run merge --ledger "$_lw/dupe.tsv" --date 2026-09-10 "$_lw/dupe.log" \ + | grep -c 'duplicate check name, so one ledger row covers 2: demo the same name')" "1" +check "and a unique one is not" \ + "$(_led_run merge --ledger "$_lw/dupe.tsv" --date 2026-09-10 "$_lw/dupe.log" \ + | grep -c 'a unique name')" "0" +check "the two collapse to one row, which is the loss being reported" \ + "$(grep -c . "$_lw/dupe.tsv")" "2" From 5bf0a20ab28f066ebd4d737a8c5620df795684f8 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 07:42:28 -0600 Subject: [PATCH 04/27] test: a record names WHICH PART asked the check, not just which suite (#917) Reported by OffgridwithJD, and the finding is that the four duplicate check names I reported from #918 are not four instances -- they are a CONVENTION, and the count grows with every part anyone writes. The phrasing is the cause. `premise: the pytest layer is where THIS PART thinks it is` says "this part" precisely so the sentence can be copied into any part, and main already carries two copies of it and two of `premise: the harness library is where this part thinks it is`. All six of the reviewer's own in-flight branches were adding more. SO (suite, name) IS NOT A KEY OF CHECKS. It is a key of check NAMES, and the two differ by however many parts share a boilerplate premise. One of them going red would mark every sharer as observed red -- a claim about a check nothing attacked, which is exactly what #918's ledger must not make. pgc_record now derives the part from BASH_SOURCE: the first frame that is not lib.sh. Not a convention change, so the next part written the same way is keyed correctly without anyone remembering, and a premise moving between parts stops being indistinguishable from a rename. Parameter expansion only -- no basename fork -- because this runs at every one of 3,762 call sites, which is the mistake this same function already made once. RESULT suite part name verdict reason MEASURED OVER A REAL RUN, 583 records: distinct (suite, name) 579 -> 4 collisions distinct (suite, part, name) 582 -> 1 38 distinct parts are named. The one survivor is a GENUINE duplicate -- 340-the-binary-must-be-built-from asks `premise: the fixture fingerprints at all` twice within the same part -- which is a real defect the ledger can now name precisely instead of losing it among convention artifacts. Evidence: selftest exit 0, 583 checks, 0 failures; pytest; shellcheck rc=0. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/lib.sh | 27 +++++++++++++- ...test_check_results_are_machine_readable.py | 24 +++++++++--- .../400-a-check-result-must-be-machine.sh | 37 ++++++++++++++----- 3 files changed, 72 insertions(+), 16 deletions(-) diff --git a/test/lib.sh b/test/lib.sh index 17159186..b57965e1 100755 --- a/test/lib.sh +++ b/test/lib.sh @@ -1009,6 +1009,30 @@ pgc_record() { # pgc_record VERDICT NAME DISPLAY [REASON] _v=FAIL ;; esac + # WHICH PART asked this question, derived from the call stack. + # + # The suite is not enough. harness_selftest sources 40-odd parts into one + # shell, and its premises are phrased to be COPIED -- "premise: the pytest + # layer is where THIS PART thinks it is" says "this part" precisely so the + # same sentence works in any of them. main carries two copies of that one and + # two of another, and the number grows with every part anyone adds. + # + # So a key of (suite, name) is not a key of checks, it is a key of check + # NAMES, and they differ by however many parts share a boilerplate premise. + # One of them going red would then mark every sharer as observed red -- a + # claim about a check nothing attacked. Found by OffgridwithJD, who noticed + # that all six of their own branches added more. + # + # BASH_SOURCE, not a convention change, so the next part written the same way + # is keyed correctly without anyone remembering. Parameter expansion only: no + # basename fork, at 3,762 call sites. + local _part="" _bs + for _bs in "${BASH_SOURCE[@]}"; do + case "$_bs" in */lib.sh|lib.sh) continue ;; esac + _part="${_bs##*/}"; _part="${_part%.sh}" + break + done + printf '%s\n' "$_display" # Tabs in a field would split it. Nothing in the tree puts one in a check # name, and this makes that true rather than assumed. @@ -1020,8 +1044,9 @@ pgc_record() { # pgc_record VERDICT NAME DISPLAY [REASON] # an idle box, 2,000 calls, identical output on every input including a real # tab: 3.1577 ms per call against 0.0096 ms, 331x, or 11.9 seconds of pure # fork overhead across a full suite against 36 ms. Reported by OffgridwithJD. - printf 'RESULT\t%s\t%s\t%s\t%s\n' \ + printf 'RESULT\t%s\t%s\t%s\t%s\t%s\n' \ "${PGC_SUITE:-unknown}" \ + "${_part:-${PGC_SUITE:-unknown}}" \ "${_name//$'\t'/ }" \ "$_v" \ "${_reason//$'\t'/ }" diff --git a/test/pytest/test_check_results_are_machine_readable.py b/test/pytest/test_check_results_are_machine_readable.py index 29af7689..bded933d 100644 --- a/test/pytest/test_check_results_are_machine_readable.py +++ b/test/pytest/test_check_results_are_machine_readable.py @@ -82,20 +82,32 @@ def test_each_verdict_emits_one_record_carrying_its_fields(expect): (' check "a name" x y', "FAIL")): recs = _records(call) expect.num(len(recs), 1, f"a {verdict} check emits exactly one record") - expect.text(recs[0].split("\t")[3], verdict, f"and its verdict field says {verdict}") - expect.text(_records(' check "a name" x x')[0].split("\t")[2], "a name", + expect.text(recs[0].split("\t")[4], verdict, f"and its verdict field says {verdict}") + expect.text(_records(' check "a name" x x')[0].split("\t")[3], "a name", "and the name field keeps its spaces") + # WHICH PART asked it. The suite is not enough: harness_selftest sources + # 40-odd parts into one shell and its premises are phrased to be COPIED -- + # "premise: the pytest layer is where THIS PART thinks it is" says "this part" + # so the same sentence works in any of them. So (suite, name) is a key of + # check NAMES, not of checks, and one sharer going red would mark them all. + # Derived from BASH_SOURCE rather than from a convention. Found by + # OffgridwithJD, whose own six branches were each adding more. + parts = {r.split("\t")[2] for r in _records(' check "a name" x x')} + expect.num(len(parts), 1, "the record names exactly one part") + expect.text("nonempty" if parts and next(iter(parts)) else "empty", "nonempty", + "and the part field is not blank") + recs = _records(' check_unrunnable "a name" MISSING_DEPENDENCY "no jq"') expect.num(len(recs), 1, "an unrunnable check emits exactly one record") - expect.text(recs[0].split("\t")[3], "UNRUN", + expect.text(recs[0].split("\t")[4], "UNRUN", "and its verdict is UNRUN, which is neither of the other two") - expect.text(recs[0].split("\t")[4], "MISSING_DEPENDENCY", + expect.text(recs[0].split("\t")[5], "MISSING_DEPENDENCY", "and the REASON_CODE travels in the reason field, not in prose") # A reason the enum does not hold is already a FAIL. It must record the verdict it # produced, not the one it was asked for. - expect.text(_records(' check_unrunnable "n" NOT_A_REASON "x"')[0].split("\t")[3], + expect.text(_records(' check_unrunnable "n" NOT_A_REASON "x"')[0].split("\t")[4], "FAIL", "a bogus reason code records FAIL, not UNRUN") @@ -114,7 +126,7 @@ def test_every_helper_records_exactly_once(expect): for call, verdict in cases.items(): recs = _records(" " + call) expect.num(len(recs), 1, f"{call.split()[0]} emits exactly one record") - expect.text(recs[0].split("\t")[3], verdict, f"and records {verdict}") + expect.text(recs[0].split("\t")[4], verdict, f"and records {verdict}") def test_the_human_lines_are_byte_identical(expect): diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index ab66e7c3..ac351999 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -50,26 +50,45 @@ _human() { # _human HELPER ARGS... -> the human lines that helper emitted check "a passing check emits exactly one record" \ "$(_rec check "a name" x x | wc -l)" "1" check "and its verdict field says PASS" \ - "$(_rec check "a name" x x | cut -f4)" "PASS" + "$(_rec check "a name" x x | cut -f5)" "PASS" check "and its name field is the check's name, spaces intact" \ - "$(_rec check "a name" x x | cut -f3)" "a name" + "$(_rec check "a name" x x | cut -f4)" "a name" + +# ---- and WHICH PART asked it ------------------------------------------------ +# +# The suite is not enough. harness_selftest sources 40-odd parts into one shell, +# and its premises are phrased to be COPIED: "premise: the pytest layer is where +# THIS PART thinks it is" says "this part" so the same sentence works in any of +# them. main carries two copies of that one and two of another, and the count +# grows with every part anyone adds -- OffgridwithJD found all six of their own +# branches adding more. +# +# So (suite, name) is not a key of checks, it is a key of check NAMES. The part +# makes it a key of the thing it identifies, and it is derived from BASH_SOURCE +# rather than from a convention, so the next part written the same way is keyed +# correctly without anyone remembering. +check "the record names the part the check was asked from" \ + "$(_rec check "a name" x x | cut -f3)" "$(basename "${BASH_SOURCE[0]}" .sh)" +check "premise: and that is this fragment, not the suite" \ + "$([ "$(_rec check "n" x x | cut -f3)" != "$(_rec check "n" x x | cut -f2)" ] \ + && echo different || echo same)" "different" check "a failing check emits exactly one record" \ "$(_rec check "a name" x y | wc -l)" "1" check "and its verdict field says FAIL" \ - "$(_rec check "a name" x y | cut -f4)" "FAIL" + "$(_rec check "a name" x y | cut -f5)" "FAIL" check "an unrunnable check emits exactly one record" \ "$(_rec check_unrunnable "a name" MISSING_DEPENDENCY "no jq" | wc -l)" "1" check "and its verdict field says UNRUN, which is neither of the other two" \ - "$(_rec check_unrunnable "a name" MISSING_DEPENDENCY "no jq" | cut -f4)" "UNRUN" + "$(_rec check_unrunnable "a name" MISSING_DEPENDENCY "no jq" | cut -f5)" "UNRUN" check "and the REASON_CODE travels in the reason field, not in prose" \ - "$(_rec check_unrunnable "a name" MISSING_DEPENDENCY "no jq" | cut -f5)" "MISSING_DEPENDENCY" + "$(_rec check_unrunnable "a name" MISSING_DEPENDENCY "no jq" | cut -f6)" "MISSING_DEPENDENCY" # A reason code the enum does not contain is already a FAIL. It must record that # verdict, not the one it was asked for. check "a bogus reason code records FAIL, not UNRUN" \ - "$(_rec check_unrunnable "a name" NOT_A_REASON "x" | cut -f4)" "FAIL" + "$(_rec check_unrunnable "a name" NOT_A_REASON "x" | cut -f5)" "FAIL" # ---- every helper, not just the two that were easy -------------------------- # @@ -80,10 +99,10 @@ check "a bogus reason code records FAIL, not UNRUN" \ check "check_text on an empty side emits one record" \ "$(_rec check_text "n" "" "x" | wc -l)" "1" check "and records FAIL, because nothing was compared" \ - "$(_rec check_text "n" "" "x" | cut -f4)" "FAIL" + "$(_rec check_text "n" "" "x" | cut -f5)" "FAIL" check "check_num on a non-number emits one record" \ "$(_rec check_num "n" "abc" "1" | wc -l)" "1" -check "and records FAIL" "$(_rec check_num "n" "abc" "1" | cut -f4)" "FAIL" +check "and records FAIL" "$(_rec check_num "n" "abc" "1" | cut -f5)" "FAIL" check "check_ratio on a non-number emits one record" \ "$(_rec check_ratio "n" "abc" "1" "2" | wc -l)" "1" check "check_ratio with a zero side emits one record" \ @@ -91,7 +110,7 @@ check "check_ratio with a zero side emits one record" \ check "check_ratio that forms a ratio emits one record" \ "$(_rec check_ratio "n" "1" "1" "2" | wc -l)" "1" check "and records PASS when the ratio is inside the bound" \ - "$(_rec check_ratio "n" "1" "1" "2" | cut -f4)" "PASS" + "$(_rec check_ratio "n" "1" "1" "2" | cut -f5)" "PASS" check "pgc_pass emits one record" "$(_rec pgc_pass "n" | wc -l)" "1" check "pgc_fail emits one record" "$(_rec pgc_fail "n" "d" | wc -l)" "1" From 5e1228cb3b4e2494f19a58b6f510f8452c74af64 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 07:49:18 -0600 Subject: [PATCH 05/27] test: the ledger keys on the part, because check names are a shared convention (#918) Reported by OffgridwithJD, who found it by asking whether their own six branches added duplicate check names. All six did. THE FOUR DUPLICATES I REPORTED ARE NOT FOUR INSTANCES, THEY ARE A CONVENTION. `premise: the pytest layer is where THIS PART thinks it is` says "this part" precisely so the sentence can be copied into any part, and main already carries two copies of it and two of `premise: the harness library is where this part thinks it is`. The count is the count TODAY and grows with every part anyone adds. So a ledger keyed on (suite, name) is not a ledger of checks. It is a ledger of check NAMES, and the two differ by however many parts share a boilerplate premise -- with the consequence this ledger cannot have: one sharer going red marks every other as observed red, a claim about a check nothing attacked. THE KEY IS NOW (suite, part, name). The part comes from pgc_record, derived from BASH_SOURCE, which #917 adds in the commit below this one. Not a convention change, so the next part written the same way is keyed correctly without anyone remembering, and it closes a blind spot in the rename detector: a premise moving between parts was indistinguishable from a rename and is now an appearance and a disappearance in two different parts, which the detector does not pair. Measured over a real run, 583 records: distinct (suite, name) 579 distinct (suite, part, name) 582 One duplicate survives, and it is a GENUINE one rather than a convention artifact: 340-the-binary-must-be-built-from asks `premise: the fixture fingerprints at all` twice within the same part. Naming that precisely, instead of losing it among three copied premises, is the point. The reviewer also renamed the premises in all six of their own in-flight branches so each names its own subject -- thirteen renames, each part now contributing zero duplicated names -- which is why this number does not grow by six the moment those land. They left main's copies alone as this PR's scope, which is right: this change reports rather than prevents. Re-seeded from a green run, keyed the new way: 613 rows, none ever observed red. Evidence: selftest exit 0, 614 checks, 0 failures; gate rc=0 against the committed files; 140 pytest passed; shellcheck rc=0. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/check_ledger.tsv | 1221 +++++++++-------- test/check_ledger_budget.txt | 2 +- test/pgc_ledger.py | 78 +- test/pytest/TESTS.md | 23 +- test/pytest/test_mutation_ledger.py | 52 +- .../410-a-check-must-have-been-red.sh | 64 +- 6 files changed, 738 insertions(+), 702 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index d01a60dc..1fe07ffc 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1,608 +1,613 @@ -harness_selftest 67 without its line is a failure, not an INCOMPLETE taken on trust never -harness_selftest CI runs the extension-upgrade guard somewhere (#741) never -harness_selftest GCOV_PREFIX is exported before the suites run (#740) never -harness_selftest PREMISE and the target really holds sources find would otherwise hash never -harness_selftest PREMISE the Makefile's recursion was actually parsed never -harness_selftest PREMISE the copy discovers the same build directories as the real tree never -harness_selftest PREMISE the fingerprint covers at least src never -harness_selftest PREMISE the fixture's src really is a symlink never -harness_selftest README.md quotes the number of modes the inventory names as refused never -harness_selftest TESTS.md states no totals line for a merge to get wrong never -harness_selftest TESTS.md states the counted number as well never -harness_selftest a /./ segment hashes the same tree the same way never -harness_selftest a /src/.. segment hashes the same tree the same way never -harness_selftest a NEW unaccounted suite fails even while the known debt is excused never -harness_selftest a bogus reason code records FAIL, not UNRUN never -harness_selftest a caller passing a major is caught never -harness_selftest a caller that reimplements the digest is caught never -harness_selftest a check merely added is not reported as a rename never -harness_selftest a check merely removed is not reported as a rename either never -harness_selftest a check observed red gains the date it was seen never -harness_selftest a check the ledger has never seen is refused, not absorbed never -harness_selftest a comment mentioning pgc_summary is not a declaration never -harness_selftest a comparison on the exit status is not counted as an assignment never -harness_selftest a counter that drifts is caught rather than absorbed never -harness_selftest a declared suite that produced no accounting is caught never -harness_selftest a declared suite the driver never dispatched reconciles never -harness_selftest a documented file that does not exist is named never -harness_selftest a documented test that does not exist is named, not passed over never -harness_selftest a drifted exit code is visible rather than absorbed never -harness_selftest a duplicated check name is reported by name never -harness_selftest a failing check emits exactly one record never -harness_selftest a failing check still prints its old line never -harness_selftest a failing suite names the first fatal event in its log never -harness_selftest a failure outranks an unrunnable check, and both are still counted never -harness_selftest a file that does not exist is reported absent, not exempt never -harness_selftest a file that is not a build input does not move it never -harness_selftest a file that uses comm pins the collation of every sort feeding it never -harness_selftest a fingerprint different from the record is stale never -harness_selftest a fingerprint equal to the record is fresh never -harness_selftest a fingerprint is 12 hex characters never -harness_selftest a fingerprint that reads src only is caught never -harness_selftest a free port beyond the old 300-probe bound is still found never -harness_selftest a hash inside a word does not hide the call after it never -harness_selftest a later green run does not erase an observation never -harness_selftest a library newer than the running server is REFUSED never -harness_selftest a library older than the running server is accepted never -harness_selftest a log carrying lib.sh's accounting line is accounted never -harness_selftest a log carrying neither is not accounted never -harness_selftest a log claiming PASSED without the accounting line shows none never -harness_selftest a log that never stated a count is not silently accepted never -harness_selftest a log whose records match its stated count reconciles never -harness_selftest a log with fewer records than it claims is caught never -harness_selftest a log with more records than it claims is caught too never -harness_selftest a long suite that calls pgc_summary still declares accounting never -harness_selftest a longer name containing pgc_summary is not a declaration never -harness_selftest a make_cluster with no cleanup is caught never -harness_selftest a merge that names its mutation records it against the check that reddened never -harness_selftest a missing binary timestamp is unknown, not predates never -harness_selftest a missing postmaster timestamp is unknown, not predates never -harness_selftest a mixed run reports both causes and neither as the whole story never -harness_selftest a name after the array's closing paren is not read as a registered suite never -harness_selftest a name defined in two files is named, not passed over never -harness_selftest a name that appeared while another disappeared is reported as a rename never -harness_selftest a neutered absent arm is caught never -harness_selftest a neutered empty-plan refusal is caught never -harness_selftest a neutered present arm is caught never -harness_selftest a new source file under objstore moves the fingerprint never -harness_selftest a passing check emits exactly one record never -harness_selftest a passing check still prints its old line never -harness_selftest a passing log shows accounting never -harness_selftest a passing ratio check is counted as a pass, not a failure never -harness_selftest a preflight that built nothing does not report PASSED never -harness_selftest a preflight that built nothing exits non-zero never -harness_selftest a preflight that built nothing says how many it built never -harness_selftest a prose total that disagrees with the ids is visible never -harness_selftest a registered suite that is accounted by nothing FAILS never -harness_selftest a relative path hashes the same tree the same way never -harness_selftest a run whose debt is within budget passes the gate never -harness_selftest a seed at the ceiling wraps past a busy top and still finds a port never -harness_selftest a server older than the binary predates it never -harness_selftest a server started after the binary is fresh never -harness_selftest a server started at the same second is fresh never -harness_selftest a stated total that disagrees with disk is visible never -harness_selftest a stated total that disagrees with the ids is visible never -harness_selftest a suite of nothing but unrunnable checks is INCOMPLETE, not SKIPPED never -harness_selftest a suite recorded as known debt passes never -harness_selftest a suite recorded as never dispatched that DID account is caught never -harness_selftest a suite that accounted passes never -harness_selftest a suite that calls pgc_summary declares accounting never -harness_selftest a suite that never calls it does not never -harness_selftest a suite that now accounts but is still listed as debt is reported never -harness_selftest a suite the driver never dispatched passes never -harness_selftest a suite whose checks all passed still exits 0 PASSED never -harness_selftest a suite with an unrunnable check exits 67 never -harness_selftest a suite with none says so as zero rather than staying silent never -harness_selftest a symlink to the tree hashes it the same way never -harness_selftest a symlinked src contributes nothing, as find -P contributes nothing never -harness_selftest a trailing comment after the call does not hide it never -harness_selftest a trailing slash hashes the same tree the same way never -harness_selftest a tree with no hashable file yields no fingerprint never -harness_selftest a write-only unrunnable field is caught never -harness_selftest adding a source file moves it never -harness_selftest an INCOMPLETE suite fails its major never -harness_selftest an INCOMPLETE suite sets the flag the major verdict actually reads never -harness_selftest an absent log shows no accounting rather than erroring never -harness_selftest an absent prose total is empty rather than a stray number never -harness_selftest an absent total is empty rather than a number that happens to match never -harness_selftest an accounting line that does not start its line is refused never -harness_selftest an added file appears in the manifest by name never -harness_selftest an added file shows up in the report never -harness_selftest an empty manifest is reported as empty, not as silence never -harness_selftest an entirely busy band reports itself full and terminates never -harness_selftest an id named twice counts once never -harness_selftest an id of fewer than three words is not counted as a mode never -harness_selftest an import from the pytest tree is caught never -harness_selftest an indented comment is still a comment never -harness_selftest an unbackticked name in prose is not treated as a claim never -harness_selftest an unconditional exit override is caught by the dominance arm never -harness_selftest an undeclared suite that DID account is caught too never -harness_selftest an undocumented file is caught along with the tests inside it never -harness_selftest an undocumented test is named rather than passed over never -harness_selftest an unhashable tree has an empty manifest never -harness_selftest an unknown provenance is not reported as a major never -harness_selftest an unparseable stamp cleans rather than guessing never -harness_selftest an unreadable b.c yields no fingerprint, not a wrong one never -harness_selftest an unreadable c.c yields no fingerprint, not a wrong one never -harness_selftest an unreadable library is not a failure never -harness_selftest an unrunnable check counts toward checks run never -harness_selftest an unrunnable check emits exactly one record never -harness_selftest an unrunnable check still prints its old line never -harness_selftest an unrunnable reason outside the enum fails rather than being accepted never -harness_selftest and 66 with its line a skip never -harness_selftest and 67 with its line INCOMPLETE, which is not a pass never -harness_selftest and a check that stayed green keeps its debt never -harness_selftest and a compiled artifact written beside its source never -harness_selftest and a failed population reconciliation fails the major never -harness_selftest and a failed reconciliation sets the per-major failure flag never -harness_selftest and a failing suite still does never -harness_selftest and a hard-coded module list is caught by the name arm never -harness_selftest and a log carrying only its OWN checks-run line is accounted too never -harness_selftest and a log with no fatal line still reports rather than staying silent never -harness_selftest and a non-numeric timestamp is unknown rather than compared as text never -harness_selftest and a prefix of a registered name is not treated as registered never -harness_selftest and a reworded producer line is refused, so the arm can fail never -harness_selftest and a skip does not, which is the one that must stay true never -harness_selftest and a skip, which reached the summary and counted zero never -harness_selftest and a suite with no unrunnable checks reconciles too never -harness_selftest and a unique one is not never -harness_selftest and absent is distinguishable from a present file that does not declare never -harness_selftest and allows one that does, which is what it was written to allow never -harness_selftest and an empty WANT is refused rather than compared never -harness_selftest and an incomplete never -harness_selftest and an ordinary failure is still a failure never -harness_selftest and an uncomputable current fingerprint is unknown, not stale never -harness_selftest and appears in the results string as INCOMPLETE never -harness_selftest and asks pgc_start_failure_message for the verdict never -harness_selftest and bench/ was in the scan, which is the hole this rule had never -harness_selftest and both exempt a file that keeps its own counter without lib.sh never -harness_selftest and comparing two manifests names it rather than saying 'changed' never -harness_selftest and counted as incomplete, so the tally can say so never -harness_selftest and counts both suites as having run never -harness_selftest and debt naming a suite that is not registered is reported too never -harness_selftest and does not cover the live one above it never -harness_selftest and every executable script declares one never -harness_selftest and every script a document names exists never -harness_selftest and every script a document names is executable never -harness_selftest and exactly one of them as incomplete never -harness_selftest and in the other direction too never -harness_selftest and is NOT made when our own postmaster died, which is the #537 case never -harness_selftest and is counted as having run never -harness_selftest and is not counted as skipped, nor is the skip count disturbed never -harness_selftest and it agrees with the real reader on a SHORT file, which is why it survived review never -harness_selftest and it catches BaseException, so an interrupt cleans up too never -harness_selftest and it is 3 bytes, not an escaped literal never -harness_selftest and it is NAMED, so the reader does not have to diff two lists never -harness_selftest and it is empty when nothing named a mutation never -harness_selftest and it is named as that fault, not as one of the other two never -harness_selftest and it is named as the opposite fault, not the same one never -harness_selftest and it is named, so the author knows which one never -harness_selftest and it is named, which the symmetry check could never do never -harness_selftest and it is not reported as having run no checks never -harness_selftest and it is the same suites, not merely the same count never -harness_selftest and it names no module directory, so it is a derivation and not a list never -harness_selftest and it says plainly that no major was recorded never -harness_selftest and it says so rather than staying silent never -harness_selftest and it stops a partially started cluster before removing the tree never -harness_selftest and its log carries the INCOMPLETE line the classifier needs never -harness_selftest and its name field is the check's name, spaces intact never -harness_selftest and its verdict field says FAIL never -harness_selftest and its verdict field says PASS never -harness_selftest and its verdict field says UNRUN, which is neither of the other two never -harness_selftest and names the suite and the check, not just a count never -harness_selftest and neither as skipped never -harness_selftest and no longer counts incompletes inline beside it never -harness_selftest and no longer mixes in the bare filename never -harness_selftest and no write-only failure flag survives in the runner never -harness_selftest and not against one that stayed green never -harness_selftest and one over budget does not never -harness_selftest and only ever moves a run off zero, so a failure still dominates never -harness_selftest and prose containing the word does not count as the line never -harness_selftest and records FAIL never -harness_selftest and records FAIL, because nothing was compared never -harness_selftest and records PASS when the ratio is inside the bound never -harness_selftest and records each suite's own verdict in the results string never -harness_selftest and records neither as ever having been red never -harness_selftest and removing it restores the fingerprint never -harness_selftest and reprints the suite's own UNRUN line beneath it never -harness_selftest and restoring it restores the fingerprint never -harness_selftest and restoring the partition restores the fingerprint never -harness_selftest and selftest/ was in the scan, which is the hole that reddened #923 never -harness_selftest and so does a failing one, which is the point never -harness_selftest and so is the attempt count never -harness_selftest and something READS it, rather than only writing it never -harness_selftest and still matches a PANIC never -harness_selftest and still matches a signal death never -harness_selftest and still matches an AddressSanitizer report never -harness_selftest and that place is pgc_record never -harness_selftest and that refusal is a VacuityError, not an ordinary assertion never -harness_selftest and the REASON_CODE travels in the reason field, not in prose never -harness_selftest and the UNRUN line the runner prints into the matrix output never -harness_selftest and the excused one is not named as a failure never -harness_selftest and the gate says which number was exceeded, by how much never -harness_selftest and the heredoc exemption covers the one inside the heredoc, not the other never -harness_selftest and the major is still readable in the name never -harness_selftest and the mistake empties the whole array rather than appending to it never -harness_selftest and the no-squatter verdict points at the server log never -harness_selftest and the old start-failure verdict is not echoed inline anywhere never -harness_selftest and the original error is re-raised rather than swallowed never -harness_selftest and the port it found is below the busy region, which is where wrapping lands never -harness_selftest and the reader answers no on it, which is the wrong answer the arm catches never -harness_selftest and the reader reads back the fingerprint the writer recorded never -harness_selftest and the real function reconciles the same input, so the arm is not noise never -harness_selftest and the reconciliation is given that record never -harness_selftest and the refusal says the server must be restarted never -harness_selftest and the registered file is written from the SUITES array itself never -harness_selftest and the run's overall status is failure never -harness_selftest and the same comparison agrees on the fixture that is right never -harness_selftest and the scan examined the suites rather than finding nothing to read never -harness_selftest and the shell keeps none either never -harness_selftest and the stable check is not reported never -harness_selftest and the start path asks pgc_start_fatal_pattern, its deliberately wider one never -harness_selftest and the suite holding it fails rather than reporting PASSED never -harness_selftest and the suite that holds it still passes never -harness_selftest and the summary line carries the incomplete count a reader needs never -harness_selftest and the tally announces it, with the reason lifted from the log never -harness_selftest and the tracked list names none of them never -harness_selftest and the tree ignores the directory Python writes them to never -harness_selftest and the two numbers are named, not just the verdict never -harness_selftest and the unrunnable ones are reported as their own count never -harness_selftest and two pg_configs for one prefix share a stamp, keyed on pkglibdir never -harness_selftest and without that record the same run is still caught never -harness_selftest building a DIFFERENT major needs a clean, which is the #536 case never -harness_selftest building the same major again needs no clean never -harness_selftest but a tree with no objects at all needs nothing, stamp or not never -harness_selftest but it says which question went unanswered never -harness_selftest but not a routine statement error never -harness_selftest check compares two empty strings and passes, which is why the rest exist never -harness_selftest check_num accepts a decimal and a sign never -harness_selftest check_num on a non-number emits one record never -harness_selftest check_num refuses a psql error message never -harness_selftest check_num refuses an md5, which is why check_text exists never -harness_selftest check_num refuses the word a yes/no check would produce never -harness_selftest check_num refuses two empty strings never -harness_selftest check_num still compares two real numbers never -harness_selftest check_num still fails two unequal numbers never -harness_selftest check_num's non-measurement line is unchanged never -harness_selftest check_ratio fails a ratio outside its bound never -harness_selftest check_ratio on a non-number emits one record never -harness_selftest check_ratio passes a ratio inside its bound never -harness_selftest check_ratio refuses a zero denominator rather than dividing by it never -harness_selftest check_ratio refuses a zero numerator, which is inside every bound never -harness_selftest check_ratio refuses an empty measurement never -harness_selftest check_ratio that forms a ratio emits one record never -harness_selftest check_ratio with a zero side emits one record never -harness_selftest check_text compares two md5 hashes, which check_num cannot never -harness_selftest check_text on an empty side emits one record never -harness_selftest check_text refuses one empty side never -harness_selftest check_text refuses two empty strings, where plain check passes never -harness_selftest check_text still fails two different strings never -harness_selftest check_text's empty-side line is unchanged never -harness_selftest control fixture: a suite whose checks all ran exits 0 never -harness_selftest control: a caller passing a pg_config is not flagged never -harness_selftest control: a document naming only what exists is clean never -harness_selftest control: a fully documented corpus reports nothing missing never -harness_selftest control: a readable run still reads fresh never -harness_selftest control: a real content change still moves the fingerprint never -harness_selftest control: a real src directory is still hashed never -harness_selftest control: an interpreter declared without the bit is caught never -harness_selftest control: and a sourced fragment, with neither, is correct never -harness_selftest control: and it still succeeds on a writable one never -harness_selftest control: and leaves the run's overall status alone never -harness_selftest control: and restoring the content restores the fingerprint never -harness_selftest control: and still announces it never -harness_selftest control: and still records that it ran, and how never -harness_selftest control: and the major reports PASS never -harness_selftest control: and the same file with the bit is not never -harness_selftest control: and the tree fingerprints again once it is readable never -harness_selftest control: cp -a preserves the execute bit, so a staged tree reads the same never -harness_selftest control: distinct names in the same corpus report no duplicate never -harness_selftest control: piping a large string into grep -q reports a match as absent never -harness_selftest control: reads, longer names, and the deliberate RANDOM/SECONDS seeds are not flagged never -harness_selftest control: the bit without an interpreter is caught too never -harness_selftest control: the same loop leaves a passing suite passing never -harness_selftest control: the same pg_config twice gives the same path never -harness_selftest control: the sweep catches an assignment to a bash special never -harness_selftest control: writing the value it was given never -harness_selftest detection distinguishes it from ours never -harness_selftest detection reports a foreign cluster's directory never -harness_selftest each manifest line is a tree-relative path and a digest never -harness_selftest editing a source file moves the fingerprint never -harness_selftest equal sets reconcile never -harness_selftest every PGC_RUN_UPGRADE-gated suite is excluded from the coverage runner (#741) never -harness_selftest every diff_query_ordered site actually names an ORDER BY never -harness_selftest every direct write to PGC_CHECKS records an outcome too never -harness_selftest every directory the Makefile builds from is in the fingerprint never -harness_selftest every ledger row carries four fields, the fourth being the mutation never -harness_selftest every nightly job is named in docs/testing.md (#741) never -harness_selftest every registered suite has a file never -harness_selftest every script that declares an interpreter is executable never -harness_selftest every suite is registered in run_all_versions.sh never -harness_selftest every suite that connects by socket path sets unix_socket_directories never -harness_selftest every suite using the ordered oracle asserts its premise never -harness_selftest every test file and every test in the corpus is named in TESTS.md never -harness_selftest every test the document names exists in the corpus never -harness_selftest guard accepts our own cluster never -harness_selftest guard rejects a foreign cluster never -harness_selftest lib.sh bumps PGC_CHECKS in exactly one place never -harness_selftest lib.sh defines check_unrunnable never -harness_selftest lib.sh defines the INCOMPLETE exit status never -harness_selftest make_cluster removes its tree when setup raises never -harness_selftest merging a green run records both checks never -harness_selftest moving bytes between files moves the fingerprint never -harness_selftest negative control: and does not find one that is not never -harness_selftest no caller passes a major where a pg_config belongs never -harness_selftest no compiled Python artifact is tracked never -harness_selftest no diff_query site names an ORDER BY it cannot test (use diff_query_ordered) never -harness_selftest no non-zero status is classified as a pass never -harness_selftest no record at all is unknown, not fresh never -harness_selftest no suite assigns to a bash special variable never -harness_selftest no suite calls set_options with a value it will reject never -harness_selftest no suite hands every run the same default port never -harness_selftest no suite pipes a captured string into an early-exit reader never -harness_selftest no suite that uses lib.sh's accounting writes PGC_CHECKS directly never -harness_selftest no test name is defined twice in the corpus never -harness_selftest no test picks a port from inside the ephemeral range never -harness_selftest nor an ordinary log line never -harness_selftest nor no for every one of them never -harness_selftest nothing leaked into the squatter never -harness_selftest objects with NO stamp are unknown provenance and must be cleaned never -harness_selftest one tree, one fingerprint, whatever the locale never -harness_selftest one unrunnable check makes the suite INCOMPLETE, not passed never -harness_selftest opposite errors do not cancel: both directions are reported never -harness_selftest pgc_fail emits one record never -harness_selftest pgc_pass emits one record never -harness_selftest pgc_port_free says the squatter's port is busy never -harness_selftest pgc_require_tools fails on one that does not never -harness_selftest pgc_require_tools passes on tools that exist never -harness_selftest pgc_setup reports the installed .so never -harness_selftest plan_marker keeps the arm that fails when the key is absent never -harness_selftest plan_marker keeps the arm that fails when the key is present never -harness_selftest plan_marker refuses a plan with no nodes at all never -harness_selftest positive control: and it is a whole list, not one lucky line never -harness_selftest positive control: the membership test finds a name that is registered never -harness_selftest positive control: the real runner's list is read, and contains isolation never -harness_selftest premise: C collation puts sort_status before sorted_projection never -harness_selftest premise: a runnable script one level down is inside the population never -harness_selftest premise: all three runner functions were extracted, not empty ranges never -harness_selftest premise: and all three are callable never -harness_selftest premise: and bench/ is in the population never -harness_selftest premise: and does NOT fire when the error is the point, across a continuation never -harness_selftest premise: and each extraction ends at its own closing brace never -harness_selftest premise: and git ls-files sees the harness it is being asked about never -harness_selftest premise: and it built none of them never -harness_selftest premise: and it is the right block (it sets the port and the preload) never -harness_selftest premise: and produced exactly one accounting line to be read never -harness_selftest premise: and really does allow the bottom never -harness_selftest premise: and so are the fixture host tools never -harness_selftest premise: and that a tracked source file is not never -harness_selftest premise: and that count excludes the definition line, which mentions it never -harness_selftest premise: and that same fixture does show the write, so the arm is not blind never -harness_selftest premise: and the accounted reader that feeds it never -harness_selftest premise: and the exemption covers a minority of them, not the corpus never -harness_selftest premise: and the line really does hold the reader it must not flag never -harness_selftest premise: and the old echo/printf pattern did NOT catch it never -harness_selftest premise: and the real function still does never -harness_selftest premise: and the real helper still carries its cleanup never -harness_selftest premise: and the stamp really was not written, so the arm is not vacuous never -harness_selftest premise: and they name at least one command in every swept directory never -harness_selftest premise: at least one suite connects by a socket path, so this is not vacuous never -harness_selftest premise: at least one suite drives the C-level encoding selftest never -harness_selftest premise: at least three nightly jobs were parsed, so the list is real never -harness_selftest premise: at least two locales are installed to compare never -harness_selftest premise: both comparison helpers are present never -harness_selftest premise: both fake configs report the same major, which is the whole point never -harness_selftest premise: both line numbers were found, so the ordering arm can mean something never -harness_selftest premise: both not_a_suite definitions were found never -harness_selftest premise: both oracles are present never -harness_selftest premise: both stray-counter probes were located never -harness_selftest premise: both the counter refusal and the lcov capture were located never -harness_selftest premise: check-ignore agrees a build object is already ignored never -harness_selftest premise: every file containing a set_options call is in the sweep never -harness_selftest premise: it is callable never -harness_selftest premise: lib.sh is readable, or every grep below approves nothing never -harness_selftest premise: lib.sh is where the check helpers live never -harness_selftest premise: lib.sh states an INCOMPLETE exit code this part could read never -harness_selftest premise: make_cluster's body was actually cut out of the file never -harness_selftest premise: no TCP suite is counted as a socket user never -harness_selftest premise: not_a_suite says no to an ordinary suite, so its yes means something never -harness_selftest premise: pipefail is on, which is the condition the bug needs never -harness_selftest premise: plan_marker's body was actually cut out of the file never -harness_selftest premise: run_coverage.sh defines not_a_suite and calls it never -harness_selftest premise: run_san.sh's default subset was found and is non-empty never -harness_selftest premise: some suite still uses comm, or the check below is vacuous never -harness_selftest premise: some suite uses the ordered oracle, or the next check is vacuous never -harness_selftest premise: the 40-line tail is filler, not the marker never -harness_selftest premise: the PGC_RUN_UPGRADE block was found and names at least one suite never -harness_selftest premise: the argument parser reads the second argument at all never -harness_selftest premise: the auxiliary band has a width to wrap within never -harness_selftest premise: the budget is a tracked file too never -harness_selftest premise: the build path ran to completion, so a stamp was due never -harness_selftest premise: the build-stamp decision is exposed to be judged never -harness_selftest premise: the check has history before the rename never -harness_selftest premise: the classifier evalled out of the runner is callable never -harness_selftest premise: the cluster-config block was located in lib.sh never -harness_selftest premise: the containment and the copy were both located never -harness_selftest premise: the corpus carries the documentation this part polices never -harness_selftest premise: the counting rule finds modes at all never -harness_selftest premise: the coverage runner is present and parses never -harness_selftest premise: the declaration reader evalled out of the runner is callable never -harness_selftest premise: the detector fires on an out-of-range value that is NOT expect_error never -harness_selftest premise: the detector fires on the line that caused #799 never -harness_selftest premise: the documents name a population of commands, not none never -harness_selftest premise: the drift changed the line the reader looks for never -harness_selftest premise: the fixture carries a well-formed accounting line, just indented never -harness_selftest premise: the fixture fingerprints at all never -harness_selftest premise: the fixture is long enough to lose the race never -harness_selftest premise: the fixture really does carry the stray name never -harness_selftest premise: the fixture really does hide its call from the stripper never -harness_selftest premise: the fixtures carry the shapes these rules are about never -harness_selftest premise: the guard's count directory and the capture's were both located never -harness_selftest premise: the harness exposes its fatal pattern to be judged never -harness_selftest premise: the harness library is where this part thinks it is never -harness_selftest premise: the heredoc exemption found heredoc lines to exempt never -harness_selftest premise: the ledger is not empty, so the partition means something never -harness_selftest premise: the ledger itself is a tracked file, not a variable never -harness_selftest premise: the ledger tool exists never -harness_selftest premise: the log report is a function that can be fed a fixture never -harness_selftest premise: the major-verdict branch was extracted, not an empty range never -harness_selftest premise: the major-verdict mapping evalled out of the runner is callable never -harness_selftest premise: the mode inventory is where this part thinks it is never -harness_selftest premise: the mutation applied -- the twin no longer sorts its inputs never -harness_selftest premise: the nightly paragraph was located and is not empty never -harness_selftest premise: the nightly workflow and the testing doc are both present never -harness_selftest premise: the observation reader evalled out of the runner is callable never -harness_selftest premise: the one fingerprint implementation is where this part thinks it is never -harness_selftest premise: the parts directory exists and was sourced never -harness_selftest premise: the population is the real test directory, not an empty glob never -harness_selftest premise: the population reconciliation is callable never -harness_selftest premise: the probe ran every helper shape once never -harness_selftest premise: the probe run skipped every major never -harness_selftest premise: the pytest cluster helper is where this part thinks it is never -harness_selftest premise: the pytest corpus is where this part thinks it is never -harness_selftest premise: the pytest layer is where this part thinks it is never -harness_selftest premise: the pytest layer states one too never -harness_selftest premise: the reader still finds a totals line when one is there never -harness_selftest premise: the real prober was restored, or every check after this lies never -harness_selftest premise: the real suite ran and reached its summary never -harness_selftest premise: the reconciliation evalled out of the runner is callable never -harness_selftest premise: the redirect, the suite invocation and the copy-back were located never -harness_selftest premise: the registered list is not empty, so the partition means something never -harness_selftest premise: the reverse sweep reads backticked names at all never -harness_selftest premise: the runner answered --list-suites, so the two checks below mean something never -harness_selftest premise: the runner defines the classifier this part is about to eval never -harness_selftest premise: the runner defines the declaration reader this part evals never -harness_selftest premise: the runner defines the observation reader this part evals never -harness_selftest premise: the runner defines the population reconciliation never -harness_selftest premise: the runner defines the reconciliation this part evals never -harness_selftest premise: the runner's collect loop was extracted, not an empty range never -harness_selftest premise: the same function returns a fingerprint for a real tree never -harness_selftest premise: the selftest has a workdir to build fixtures in never -harness_selftest premise: the set_options sweep read a substantial number of calls never -harness_selftest premise: the source tree is a git checkout never -harness_selftest premise: the sourced parts are inside the population, not pruned never -harness_selftest premise: the spelling fixture fingerprints at all never -harness_selftest premise: the stamp writer is a function that can be exercised never -harness_selftest premise: the start-failure path still exists to be judged never -harness_selftest premise: the stub frees exactly one port, 500 past the floor never -harness_selftest premise: the stub really does refuse the top of the band never -harness_selftest premise: the sub-suite failed, so its summary ran never -harness_selftest premise: the sweep finds the call sites it is meant to police never -harness_selftest premise: the sweep found the corpus rather than an empty glob never -harness_selftest premise: the sweep read lines to classify never -harness_selftest premise: the sweep read the corpus and found sites to classify never -harness_selftest premise: the sweep reads a population of scripts, not an empty find never -harness_selftest premise: the tree fingerprints to something when it is readable never -harness_selftest premise: the tree really contains continued diff_query calls to join never -harness_selftest premise: the twin script was written and is runnable never -harness_selftest premise: the unprivileged read agrees while everything is readable never -harness_selftest premise: the unsorted twin is callable never -harness_selftest premise: the verdict is composed somewhere it can be judged never -harness_selftest premise: the writer wrote a stamp at all never -harness_selftest premise: there are workflow files to search never -harness_selftest premise: while a real assignment on the same line shape IS counted never -harness_selftest premise: while the real body satisfies all three, so the greps work never -harness_selftest premise: while the real layer satisfies that same arm never -harness_selftest premise: while the real module satisfies the derivation arm never -harness_selftest renaming a source file moves the fingerprint too never -harness_selftest running the real loop over both fixtures fails the major never -harness_selftest section 1a's document total is the sum of its two sections never -harness_selftest section 1a's not-refused total is the count of ids in section 3 never -harness_selftest section 1a's refused total is the count of ids in section 2 never -harness_selftest section 2's opening states the counted number of refused modes never -harness_selftest so the tree still fingerprints from its root files alone never -harness_selftest so the verdict is fresh, not unknown never -harness_selftest so the verdict is unknown -- UNVERIFIED -- and never stale never -harness_selftest sorted_projection's two comparisons are ordered, its subject being order never -harness_selftest squatter survived untouched never -harness_selftest suite did not settle on the squatter's port never -harness_selftest suite's cluster is its own never -harness_selftest suite's own objects are visible to it never -harness_selftest the admitted gap is the run total minus what is written down never -harness_selftest the build path asks pgc_build_needs_clean rather than merely naming it never -harness_selftest the census reads a run's records never -harness_selftest the closing paragraph states the counted number too never -harness_selftest the committed budget matches the committed ledger's debt never -harness_selftest the committed budget names both debts never -harness_selftest the copy-back refuses a destination outside the tree (#740) never -harness_selftest the counter counts a fixture's section 2 never -harness_selftest the counter counts a fixture's section 3 never -harness_selftest the counter stops at the next heading never -harness_selftest the counters are returned beside their objects before the refusal (#740) never -harness_selftest the coverage runner refuses zero counters before it calls lcov (#740) never -harness_selftest the coverage runner's not_a_suite agrees with the selftest's, both ways (#741) never -harness_selftest the debt file is in the tree never -harness_selftest the driver holds no checks; they all live in parts never -harness_selftest the driver sources the parts by glob, not by a list never -harness_selftest the empty-plan refusal precedes the arm it protects never -harness_selftest the failure path asks pgc_start_log_report for the reason never -harness_selftest the fatal pattern matches a library that will not load never -harness_selftest the fingerprint derives its build directories from a Makefile on disk never -harness_selftest the fingerprint is the hash of the manifest never -harness_selftest the fixed fingerprint equals what the previous implementation produced never -harness_selftest the grep -q shape is the one that gets this wrong under pipefail never -harness_selftest the hash mixes in each file's path relative to the tree, not its name never -harness_selftest the identity catches comm reading unsorted input never -harness_selftest the installed .so is the one this run built never -harness_selftest the layer ends a session by setting its exit status never -harness_selftest the layer prints the unrunnable reason in lib.sh's shape never -harness_selftest the layer still writes the unrunnable state never -harness_selftest the ledger partitions into observed and never never -harness_selftest the loop delegates each verdict to pgc_tally_suite never -harness_selftest the manifest is tree-relative, never absolute never -harness_selftest the manifest names every file the fingerprint hashes never -harness_selftest the module imports nothing from the pytest tree never -harness_selftest the ordered oracle keeps the empty-result sentinel never -harness_selftest the ordered oracle keeps the unique query-error sentinel never -harness_selftest the ordered oracle numbers the rows as they arrive never -harness_selftest the ordered oracle orders by row_number, so it keeps the query's order never -harness_selftest the original rule flags a bump that records no outcome never -harness_selftest the ownership claim is made when a squatter held the port every time never -harness_selftest the partition over the real suite list adds up never -harness_selftest the per-suite escape hatch PGC_EXTRA_CONF is still applied to the config never -harness_selftest the population partitions, and prints inputs == sum(buckets) never -harness_selftest the port is named either way never -harness_selftest the probe is written outside the live source tree never -harness_selftest the pytest helper keeps no private fingerprint implementation never -harness_selftest the reader accepts the line the producer actually emits never -harness_selftest the reader does not answer yes for every registered suite never -harness_selftest the reconciliation prints inputs == sum(buckets) never -harness_selftest the record cannot introduce a suite the source never declared never -harness_selftest the record count equals the counter the summary reports never -harness_selftest the refusal looks in GCOV_PREFIX before the tree-wide walk (#740) never -harness_selftest the report names each hashed file never -harness_selftest the report names the symbol that was actually missing never -harness_selftest the report states how many files it hashed never -harness_selftest the row's value is read, not a digit inside its label never -harness_selftest the runner calls a clean exit a pass never -harness_selftest the runner calls the population reconciliation never -harness_selftest the runner calls the reconciliation, not merely defines it never -harness_selftest the runner calls the record reconciliation, not merely defines it never -harness_selftest the runner classifies the file that suite actually produced never -harness_selftest the runner defines the record reconciliation never -harness_selftest the runner's INCOMPLETE branch calls the mapping rather than a local flag never -harness_selftest the same tree fingerprints the same twice never -harness_selftest the sanitizer subset runs every suite that drives the encoding selftest never -harness_selftest the set oracle orders by the rendered row, so it is order-blind never -harness_selftest the shared cluster config sets no pgcolumnar.* GUC never -harness_selftest the skip branch records the suite it did not dispatch never -harness_selftest the stamp lib.sh writes is exactly the major never -harness_selftest the stamp writer reports failure on an unwritable target never -harness_selftest the stripper hides no pgc_summary call in any registered suite never -harness_selftest the stronger rule flags that same allowed bump, which is the change never -harness_selftest the suite list is sorted in C order, so two new suites land in different places never -harness_selftest the summary path asks pgc_fatal_pattern rather than hardcoding it never -harness_selftest the summary reconciles the three states against the total never -harness_selftest the sweep catches a producer that is neither echo nor printf never -harness_selftest the sweep counts the fixture's tests and files never -harness_selftest the sweep does not mistake the || operator for a pipe never -harness_selftest the sweep's pattern sees both lines of the probe never -harness_selftest the two collapse to one row, which is the loss being reported never -harness_selftest the two harnesses agree on the INCOMPLETE exit code never -harness_selftest the unrunnable check names itself, its reason code and its detail never -harness_selftest the writer writes the file the reader looks for never -harness_selftest the zero-counter guard counts the directory lcov captures (#740) never -harness_selftest two installations of one major get different stamp paths never -harness_selftest two unreadable pg_configs do not alias onto one stamp never -harness_selftest while a pass does not never -harness_selftest with an incomplete suite in the tally the major reports FAIL never +harness_selftest 030-assertions nothing leaked into the squatter never +harness_selftest 030-assertions pgc_port_free says the squatter's port is busy never +harness_selftest 030-assertions squatter survived untouched never +harness_selftest 030-assertions suite did not settle on the squatter's port never +harness_selftest 030-assertions suite's cluster is its own never +harness_selftest 030-assertions suite's own objects are visible to it never +harness_selftest 040-the-detection-primitive-itself detection distinguishes it from ours never +harness_selftest 040-the-detection-primitive-itself detection reports a foreign cluster's directory never +harness_selftest 040-the-detection-primitive-itself guard accepts our own cluster never +harness_selftest 040-the-detection-primitive-itself guard rejects a foreign cluster never +harness_selftest 040-the-detection-primitive-itself premise: the runner answered --list-suites, so the two checks below mean something never +harness_selftest 050-the-list-must-be-read-the a name after the array's closing paren is not read as a registered suite never +harness_selftest 050-the-list-must-be-read-the and the mistake empties the whole array rather than appending to it never +harness_selftest 050-the-list-must-be-read-the positive control: and it is a whole list, not one lucky line never +harness_selftest 050-the-list-must-be-read-the positive control: the real runner's list is read, and contains isolation never +harness_selftest 050-the-list-must-be-read-the premise: the fixture really does carry the stray name never +harness_selftest 060-the-list-stays-sorted-which-is premise: C collation puts sort_status before sorted_projection never +harness_selftest 060-the-list-stays-sorted-which-is the suite list is sorted in C order, so two new suites land in different places never +harness_selftest 070-and-comm-s-two-inputs-must a file that uses comm pins the collation of every sort feeding it never +harness_selftest 070-and-comm-s-two-inputs-must and a prefix of a registered name is not treated as registered never +harness_selftest 070-and-comm-s-two-inputs-must every registered suite has a file never +harness_selftest 070-and-comm-s-two-inputs-must every suite is registered in run_all_versions.sh never +harness_selftest 070-and-comm-s-two-inputs-must negative control: and does not find one that is not never +harness_selftest 070-and-comm-s-two-inputs-must positive control: the membership test finds a name that is registered never +harness_selftest 070-and-comm-s-two-inputs-must premise: some suite still uses comm, or the check below is vacuous never +harness_selftest 080-no-suite-pipes-a-captured-string and bench/ was in the scan, which is the hole this rule had never +harness_selftest 080-no-suite-pipes-a-captured-string and does not cover the live one above it never +harness_selftest 080-no-suite-pipes-a-captured-string and selftest/ was in the scan, which is the hole that reddened #923 never +harness_selftest 080-no-suite-pipes-a-captured-string and the heredoc exemption covers the one inside the heredoc, not the other never +harness_selftest 080-no-suite-pipes-a-captured-string and the scan examined the suites rather than finding nothing to read never +harness_selftest 080-no-suite-pipes-a-captured-string control: piping a large string into grep -q reports a match as absent never +harness_selftest 080-no-suite-pipes-a-captured-string no suite pipes a captured string into an early-exit reader never +harness_selftest 080-no-suite-pipes-a-captured-string premise: and the exemption covers a minority of them, not the corpus never +harness_selftest 080-no-suite-pipes-a-captured-string premise: and the line really does hold the reader it must not flag never +harness_selftest 080-no-suite-pipes-a-captured-string premise: and the old echo/printf pattern did NOT catch it never +harness_selftest 080-no-suite-pipes-a-captured-string premise: the heredoc exemption found heredoc lines to exempt never +harness_selftest 080-no-suite-pipes-a-captured-string premise: the sweep read lines to classify never +harness_selftest 080-no-suite-pipes-a-captured-string the sweep catches a producer that is neither echo nor printf never +harness_selftest 080-no-suite-pipes-a-captured-string the sweep does not mistake the || operator for a pipe never +harness_selftest 080-no-suite-pipes-a-captured-string the sweep's pattern sees both lines of the probe never +harness_selftest 090-no-suite-hands-every-run-the no suite hands every run the same default port never +harness_selftest 090-no-suite-hands-every-run-the no test picks a port from inside the ephemeral range never +harness_selftest 100-the-assertions-that-refuse-an-empty check compares two empty strings and passes, which is why the rest exist never +harness_selftest 100-the-assertions-that-refuse-an-empty check_num accepts a decimal and a sign never +harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses a psql error message never +harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses an md5, which is why check_text exists never +harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses the word a yes/no check would produce never +harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses two empty strings never +harness_selftest 100-the-assertions-that-refuse-an-empty check_num still compares two real numbers never +harness_selftest 100-the-assertions-that-refuse-an-empty check_num still fails two unequal numbers never +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio fails a ratio outside its bound never +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio passes a ratio inside its bound never +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio refuses a zero denominator rather than dividing by it never +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio refuses a zero numerator, which is inside every bound never +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio refuses an empty measurement never +harness_selftest 100-the-assertions-that-refuse-an-empty check_text compares two md5 hashes, which check_num cannot never +harness_selftest 100-the-assertions-that-refuse-an-empty check_text refuses one empty side never +harness_selftest 100-the-assertions-that-refuse-an-empty check_text refuses two empty strings, where plain check passes never +harness_selftest 100-the-assertions-that-refuse-an-empty check_text still fails two different strings never +harness_selftest 100-the-assertions-that-refuse-an-empty pgc_require_tools fails on one that does not never +harness_selftest 100-the-assertions-that-refuse-an-empty pgc_require_tools passes on tools that exist never +harness_selftest 110-the-harness-must-say-which-binary pgc_setup reports the installed .so never +harness_selftest 110-the-harness-must-say-which-binary the installed .so is the one this run built never +harness_selftest 120-a-failing-suite-must-surface-the a failing suite names the first fatal event in its log never +harness_selftest 120-a-failing-suite-must-surface-the premise: the 40-line tail is filler, not the marker never +harness_selftest 120-a-failing-suite-must-surface-the premise: the sub-suite failed, so its summary ran never +harness_selftest 130-the-sanitizer-subset-must-cover-the premise: at least one suite drives the C-level encoding selftest never +harness_selftest 130-the-sanitizer-subset-must-cover-the premise: run_san.sh's default subset was found and is non-empty never +harness_selftest 130-the-sanitizer-subset-must-cover-the the sanitizer subset runs every suite that drives the encoding selftest never +harness_selftest 140-a-cluster-that-will-not-start and still matches a PANIC never +harness_selftest 140-a-cluster-that-will-not-start and still matches a signal death never +harness_selftest 140-a-cluster-that-will-not-start and still matches an AddressSanitizer report never +harness_selftest 140-a-cluster-that-will-not-start but not a routine statement error never +harness_selftest 140-a-cluster-that-will-not-start nor an ordinary log line never +harness_selftest 140-a-cluster-that-will-not-start premise: the harness exposes its fatal pattern to be judged never +harness_selftest 140-a-cluster-that-will-not-start the fatal pattern matches a library that will not load never +harness_selftest 150-the-verdict-must-not-assert-a a mixed run reports both causes and neither as the whole story never +harness_selftest 150-the-verdict-must-not-assert-a and is NOT made when our own postmaster died, which is the #537 case never +harness_selftest 150-the-verdict-must-not-assert-a and so is the attempt count never +harness_selftest 150-the-verdict-must-not-assert-a and the no-squatter verdict points at the server log never +harness_selftest 150-the-verdict-must-not-assert-a premise: the verdict is composed somewhere it can be judged never +harness_selftest 150-the-verdict-must-not-assert-a the ownership claim is made when a squatter held the port every time never +harness_selftest 150-the-verdict-must-not-assert-a the port is named either way never +harness_selftest 160-and-the-log-report-must-show and a log with no fatal line still reports rather than staying silent never +harness_selftest 160-and-the-log-report-must-show premise: the log report is a function that can be fed a fixture never +harness_selftest 160-and-the-log-report-must-show the report names the symbol that was actually missing never +harness_selftest 170-and-lib-sh-must-ask-these and asks pgc_start_failure_message for the verdict never +harness_selftest 170-and-lib-sh-must-ask-these and the old start-failure verdict is not echoed inline anywhere never +harness_selftest 170-and-lib-sh-must-ask-these and the start path asks pgc_start_fatal_pattern, its deliberately wider one never +harness_selftest 170-and-lib-sh-must-ask-these premise: lib.sh is readable, or every grep below approves nothing never +harness_selftest 170-and-lib-sh-must-ask-these premise: the start-failure path still exists to be judged never +harness_selftest 170-and-lib-sh-must-ask-these the failure path asks pgc_start_log_report for the reason never +harness_selftest 170-and-lib-sh-must-ask-these the summary path asks pgc_fatal_pattern rather than hardcoding it never +harness_selftest 180-the-port-walk-must-wrap-not a free port beyond the old 300-probe bound is still found never +harness_selftest 180-the-port-walk-must-wrap-not a seed at the ceiling wraps past a busy top and still finds a port never +harness_selftest 180-the-port-walk-must-wrap-not an entirely busy band reports itself full and terminates never +harness_selftest 180-the-port-walk-must-wrap-not and the port it found is below the busy region, which is where wrapping lands never +harness_selftest 180-the-port-walk-must-wrap-not premise: and really does allow the bottom never +harness_selftest 180-the-port-walk-must-wrap-not premise: the auxiliary band has a width to wrap within never +harness_selftest 180-the-port-walk-must-wrap-not premise: the real prober was restored, or every check after this lies never +harness_selftest 180-the-port-walk-must-wrap-not premise: the stub frees exactly one port, 500 past the floor never +harness_selftest 180-the-port-walk-must-wrap-not premise: the stub really does refuse the top of the band never +harness_selftest 190-an-in-tree-build-must-not an unknown provenance is not reported as a major never +harness_selftest 190-an-in-tree-build-must-not an unparseable stamp cleans rather than guessing never +harness_selftest 190-an-in-tree-build-must-not and an empty WANT is refused rather than compared never +harness_selftest 190-an-in-tree-build-must-not and in the other direction too never +harness_selftest 190-an-in-tree-build-must-not and it is 3 bytes, not an escaped literal never +harness_selftest 190-an-in-tree-build-must-not and it says plainly that no major was recorded never +harness_selftest 190-an-in-tree-build-must-not building a DIFFERENT major needs a clean, which is the #536 case never +harness_selftest 190-an-in-tree-build-must-not building the same major again needs no clean never +harness_selftest 190-an-in-tree-build-must-not but a tree with no objects at all needs nothing, stamp or not never +harness_selftest 190-an-in-tree-build-must-not objects with NO stamp are unknown provenance and must be cleaned never +harness_selftest 190-an-in-tree-build-must-not premise: the build-stamp decision is exposed to be judged never +harness_selftest 190-an-in-tree-build-must-not premise: the stamp writer is a function that can be exercised never +harness_selftest 190-an-in-tree-build-must-not the build path asks pgc_build_needs_clean rather than merely naming it never +harness_selftest 190-an-in-tree-build-must-not the stamp lib.sh writes is exactly the major never +harness_selftest 200-additions-go-in-their-own-file premise: the parts directory exists and was sourced never +harness_selftest 200-additions-go-in-their-own-file the driver holds no checks; they all live in parts never +harness_selftest 200-additions-go-in-their-own-file the driver sources the parts by glob, not by a list never +harness_selftest 210-no-suite-assigns-a-bash-special control: reads, longer names, and the deliberate RANDOM/SECONDS seeds are not flagged never +harness_selftest 210-no-suite-assigns-a-bash-special control: the sweep catches an assignment to a bash special never +harness_selftest 210-no-suite-assigns-a-bash-special no suite assigns to a bash special variable never +harness_selftest 220-an-opt-in-upgrade-guard-must CI runs the extension-upgrade guard somewhere (#741) never +harness_selftest 220-an-opt-in-upgrade-guard-must every PGC_RUN_UPGRADE-gated suite is excluded from the coverage runner (#741) never +harness_selftest 220-an-opt-in-upgrade-guard-must premise: both not_a_suite definitions were found never +harness_selftest 220-an-opt-in-upgrade-guard-must premise: not_a_suite says no to an ordinary suite, so its yes means something never +harness_selftest 220-an-opt-in-upgrade-guard-must premise: run_coverage.sh defines not_a_suite and calls it never +harness_selftest 220-an-opt-in-upgrade-guard-must premise: the PGC_RUN_UPGRADE block was found and names at least one suite never +harness_selftest 220-an-opt-in-upgrade-guard-must premise: the population is the real test directory, not an empty glob never +harness_selftest 220-an-opt-in-upgrade-guard-must premise: there are workflow files to search never +harness_selftest 220-an-opt-in-upgrade-guard-must the coverage runner's not_a_suite agrees with the selftest's, both ways (#741) never +harness_selftest 230-a-suite-connecting-by-socket-must every suite that connects by socket path sets unix_socket_directories never +harness_selftest 230-a-suite-connecting-by-socket-must premise: at least one suite connects by a socket path, so this is not vacuous never +harness_selftest 230-a-suite-connecting-by-socket-must premise: no TCP suite is counted as a socket user never +harness_selftest 240-the-nightly-enumeration-must-not every nightly job is named in docs/testing.md (#741) never +harness_selftest 240-the-nightly-enumeration-must-not premise: at least three nightly jobs were parsed, so the list is real never +harness_selftest 240-the-nightly-enumeration-must-not premise: the nightly paragraph was located and is not empty never +harness_selftest 240-the-nightly-enumeration-must-not premise: the nightly workflow and the testing doc are both present never +harness_selftest 250-the-coverage-runner-must-refuse GCOV_PREFIX is exported before the suites run (#740) never +harness_selftest 250-the-coverage-runner-must-refuse premise: both stray-counter probes were located never +harness_selftest 250-the-coverage-runner-must-refuse premise: both the counter refusal and the lcov capture were located never +harness_selftest 250-the-coverage-runner-must-refuse premise: the containment and the copy were both located never +harness_selftest 250-the-coverage-runner-must-refuse premise: the coverage runner is present and parses never +harness_selftest 250-the-coverage-runner-must-refuse premise: the guard's count directory and the capture's were both located never +harness_selftest 250-the-coverage-runner-must-refuse premise: the redirect, the suite invocation and the copy-back were located never +harness_selftest 250-the-coverage-runner-must-refuse the copy-back refuses a destination outside the tree (#740) never +harness_selftest 250-the-coverage-runner-must-refuse the counters are returned beside their objects before the refusal (#740) never +harness_selftest 250-the-coverage-runner-must-refuse the coverage runner refuses zero counters before it calls lcov (#740) never +harness_selftest 250-the-coverage-runner-must-refuse the refusal looks in GCOV_PREFIX before the tree-wide walk (#740) never +harness_selftest 250-the-coverage-runner-must-refuse the zero-counter guard counts the directory lcov captures (#740) never +harness_selftest 260-an-ordered-comparison-must-use-the and it is the same suites, not merely the same count never +harness_selftest 260-an-ordered-comparison-must-use-the every diff_query_ordered site actually names an ORDER BY never +harness_selftest 260-an-ordered-comparison-must-use-the every suite using the ordered oracle asserts its premise never +harness_selftest 260-an-ordered-comparison-must-use-the no diff_query site names an ORDER BY it cannot test (use diff_query_ordered) never +harness_selftest 260-an-ordered-comparison-must-use-the premise: both comparison helpers are present never +harness_selftest 260-an-ordered-comparison-must-use-the premise: both oracles are present never +harness_selftest 260-an-ordered-comparison-must-use-the premise: some suite uses the ordered oracle, or the next check is vacuous never +harness_selftest 260-an-ordered-comparison-must-use-the premise: the tree really contains continued diff_query calls to join never +harness_selftest 260-an-ordered-comparison-must-use-the sorted_projection's two comparisons are ordered, its subject being order never +harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle keeps the empty-result sentinel never +harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle keeps the unique query-error sentinel never +harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle numbers the rows as they arrive never +harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle orders by row_number, so it keeps the query's order never +harness_selftest 260-an-ordered-comparison-must-use-the the set oracle orders by the rendered row, so it is order-blind never +harness_selftest 270-a-set-options-call-must-use-values no suite calls set_options with a value it will reject never +harness_selftest 270-a-set-options-call-must-use-values premise: and does NOT fire when the error is the point, across a continuation never +harness_selftest 270-a-set-options-call-must-use-values premise: every file containing a set_options call is in the sweep never +harness_selftest 270-a-set-options-call-must-use-values premise: the detector fires on an out-of-range value that is NOT expect_error never +harness_selftest 270-a-set-options-call-must-use-values premise: the set_options sweep read a substantial number of calls never +harness_selftest 280-the-shared-cluster-config-must-not premise: and it is the right block (it sets the port and the preload) never +harness_selftest 280-the-shared-cluster-config-must-not premise: the cluster-config block was located in lib.sh never +harness_selftest 280-the-shared-cluster-config-must-not premise: the detector fires on the line that caused #799 never +harness_selftest 280-the-shared-cluster-config-must-not the per-suite escape hatch PGC_EXTRA_CONF is still applied to the config never +harness_selftest 280-the-shared-cluster-config-must-not the shared cluster config sets no pgcolumnar.* GUC never +harness_selftest 290-a-preflight-that-built-nothing a preflight that built nothing does not report PASSED never +harness_selftest 290-a-preflight-that-built-nothing a preflight that built nothing exits non-zero never +harness_selftest 290-a-preflight-that-built-nothing a preflight that built nothing says how many it built never +harness_selftest 290-a-preflight-that-built-nothing premise: and it built none of them never +harness_selftest 290-a-preflight-that-built-nothing premise: the probe run skipped every major never +harness_selftest 300-a-test-script-must-be-runnable and every executable script declares one never +harness_selftest 300-a-test-script-must-be-runnable and every script a document names exists never +harness_selftest 300-a-test-script-must-be-runnable and every script a document names is executable never +harness_selftest 300-a-test-script-must-be-runnable control: an interpreter declared without the bit is caught never +harness_selftest 300-a-test-script-must-be-runnable control: and a sourced fragment, with neither, is correct never +harness_selftest 300-a-test-script-must-be-runnable control: and the same file with the bit is not never +harness_selftest 300-a-test-script-must-be-runnable control: cp -a preserves the execute bit, so a staged tree reads the same never +harness_selftest 300-a-test-script-must-be-runnable control: the bit without an interpreter is caught too never +harness_selftest 300-a-test-script-must-be-runnable every script that declares an interpreter is executable never +harness_selftest 300-a-test-script-must-be-runnable premise: a runnable script one level down is inside the population never +harness_selftest 300-a-test-script-must-be-runnable premise: and bench/ is in the population never +harness_selftest 300-a-test-script-must-be-runnable premise: and so are the fixture host tools never +harness_selftest 300-a-test-script-must-be-runnable premise: and they name at least one command in every swept directory never +harness_selftest 300-a-test-script-must-be-runnable premise: the documents name a population of commands, not none never +harness_selftest 300-a-test-script-must-be-runnable premise: the sourced parts are inside the population, not pruned never +harness_selftest 300-a-test-script-must-be-runnable premise: the sweep reads a population of scripts, not an empty find never +harness_selftest 310-a-compiled-artifact-must-not-be and a compiled artifact written beside its source never +harness_selftest 310-a-compiled-artifact-must-not-be and the tracked list names none of them never +harness_selftest 310-a-compiled-artifact-must-not-be and the tree ignores the directory Python writes them to never +harness_selftest 310-a-compiled-artifact-must-not-be no compiled Python artifact is tracked never +harness_selftest 310-a-compiled-artifact-must-not-be premise: and git ls-files sees the harness it is being asked about never +harness_selftest 310-a-compiled-artifact-must-not-be premise: and that a tracked source file is not never +harness_selftest 310-a-compiled-artifact-must-not-be premise: check-ignore agrees a build object is already ignored never +harness_selftest 310-a-compiled-artifact-must-not-be premise: the source tree is a git checkout never +harness_selftest 320-a-check-that-could-not-run 67 without its line is a failure, not an INCOMPLETE taken on trust never +harness_selftest 320-a-check-that-could-not-run a counter that drifts is caught rather than absorbed never +harness_selftest 320-a-check-that-could-not-run a failure outranks an unrunnable check, and both are still counted never +harness_selftest 320-a-check-that-could-not-run a passing ratio check is counted as a pass, not a failure never +harness_selftest 320-a-check-that-could-not-run a suite of nothing but unrunnable checks is INCOMPLETE, not SKIPPED never +harness_selftest 320-a-check-that-could-not-run a suite whose checks all passed still exits 0 PASSED never +harness_selftest 320-a-check-that-could-not-run a suite with none says so as zero rather than staying silent never +harness_selftest 320-a-check-that-could-not-run an INCOMPLETE suite fails its major never +harness_selftest 320-a-check-that-could-not-run an unrunnable check counts toward checks run never +harness_selftest 320-a-check-that-could-not-run an unrunnable reason outside the enum fails rather than being accepted never +harness_selftest 320-a-check-that-could-not-run and 66 with its line a skip never +harness_selftest 320-a-check-that-could-not-run and 67 with its line INCOMPLETE, which is not a pass never +harness_selftest 320-a-check-that-could-not-run and a failing suite still does never +harness_selftest 320-a-check-that-could-not-run and a skip does not, which is the one that must stay true never +harness_selftest 320-a-check-that-could-not-run and a suite with no unrunnable checks reconciles too never +harness_selftest 320-a-check-that-could-not-run and allows one that does, which is what it was written to allow never +harness_selftest 320-a-check-that-could-not-run and an ordinary failure is still a failure never +harness_selftest 320-a-check-that-could-not-run and both exempt a file that keeps its own counter without lib.sh never +harness_selftest 320-a-check-that-could-not-run and it is not reported as having run no checks never +harness_selftest 320-a-check-that-could-not-run and no write-only failure flag survives in the runner never +harness_selftest 320-a-check-that-could-not-run and the suite holding it fails rather than reporting PASSED never +harness_selftest 320-a-check-that-could-not-run and the suite that holds it still passes never +harness_selftest 320-a-check-that-could-not-run and the unrunnable ones are reported as their own count never +harness_selftest 320-a-check-that-could-not-run every direct write to PGC_CHECKS records an outcome too never +harness_selftest 320-a-check-that-could-not-run lib.sh defines check_unrunnable never +harness_selftest 320-a-check-that-could-not-run lib.sh defines the INCOMPLETE exit status never +harness_selftest 320-a-check-that-could-not-run no non-zero status is classified as a pass never +harness_selftest 320-a-check-that-could-not-run no suite that uses lib.sh's accounting writes PGC_CHECKS directly never +harness_selftest 320-a-check-that-could-not-run one unrunnable check makes the suite INCOMPLETE, not passed never +harness_selftest 320-a-check-that-could-not-run premise: the classifier evalled out of the runner is callable never +harness_selftest 320-a-check-that-could-not-run premise: the fixtures carry the shapes these rules are about never +harness_selftest 320-a-check-that-could-not-run premise: the harness library is where this part thinks it is never +harness_selftest 320-a-check-that-could-not-run premise: the major-verdict mapping evalled out of the runner is callable never +harness_selftest 320-a-check-that-could-not-run premise: the runner defines the classifier this part is about to eval never +harness_selftest 320-a-check-that-could-not-run premise: the sweep read the corpus and found sites to classify never +harness_selftest 320-a-check-that-could-not-run the original rule flags a bump that records no outcome never +harness_selftest 320-a-check-that-could-not-run the runner calls a clean exit a pass never +harness_selftest 320-a-check-that-could-not-run the runner's INCOMPLETE branch calls the mapping rather than a local flag never +harness_selftest 320-a-check-that-could-not-run the stronger rule flags that same allowed bump, which is the change never +harness_selftest 320-a-check-that-could-not-run the summary reconciles the three states against the total never +harness_selftest 320-a-check-that-could-not-run the unrunnable check names itself, its reason code and its detail never +harness_selftest 320-a-check-that-could-not-run while a pass does not never +harness_selftest 330-the-incomplete-path-must-run-whole a suite with an unrunnable check exits 67 never +harness_selftest 330-the-incomplete-path-must-run-whole an INCOMPLETE suite sets the flag the major verdict actually reads never +harness_selftest 330-the-incomplete-path-must-run-whole and appears in the results string as INCOMPLETE never +harness_selftest 330-the-incomplete-path-must-run-whole and counted as incomplete, so the tally can say so never +harness_selftest 330-the-incomplete-path-must-run-whole and counts both suites as having run never +harness_selftest 330-the-incomplete-path-must-run-whole and exactly one of them as incomplete never +harness_selftest 330-the-incomplete-path-must-run-whole and is counted as having run never +harness_selftest 330-the-incomplete-path-must-run-whole and is not counted as skipped, nor is the skip count disturbed never +harness_selftest 330-the-incomplete-path-must-run-whole and its log carries the INCOMPLETE line the classifier needs never +harness_selftest 330-the-incomplete-path-must-run-whole and neither as skipped never +harness_selftest 330-the-incomplete-path-must-run-whole and no longer counts incompletes inline beside it never +harness_selftest 330-the-incomplete-path-must-run-whole and records each suite's own verdict in the results string never +harness_selftest 330-the-incomplete-path-must-run-whole and reprints the suite's own UNRUN line beneath it never +harness_selftest 330-the-incomplete-path-must-run-whole and the UNRUN line the runner prints into the matrix output never +harness_selftest 330-the-incomplete-path-must-run-whole and the run's overall status is failure never +harness_selftest 330-the-incomplete-path-must-run-whole and the summary line carries the incomplete count a reader needs never +harness_selftest 330-the-incomplete-path-must-run-whole and the tally announces it, with the reason lifted from the log never +harness_selftest 330-the-incomplete-path-must-run-whole control fixture: a suite whose checks all ran exits 0 never +harness_selftest 330-the-incomplete-path-must-run-whole control: and leaves the run's overall status alone never +harness_selftest 330-the-incomplete-path-must-run-whole control: and still announces it never +harness_selftest 330-the-incomplete-path-must-run-whole control: and still records that it ran, and how never +harness_selftest 330-the-incomplete-path-must-run-whole control: and the major reports PASS never +harness_selftest 330-the-incomplete-path-must-run-whole control: the same loop leaves a passing suite passing never +harness_selftest 330-the-incomplete-path-must-run-whole premise: all three runner functions were extracted, not empty ranges never +harness_selftest 330-the-incomplete-path-must-run-whole premise: and all three are callable never +harness_selftest 330-the-incomplete-path-must-run-whole premise: and each extraction ends at its own closing brace never +harness_selftest 330-the-incomplete-path-must-run-whole premise: the major-verdict branch was extracted, not an empty range never +harness_selftest 330-the-incomplete-path-must-run-whole premise: the runner's collect loop was extracted, not an empty range never +harness_selftest 330-the-incomplete-path-must-run-whole premise: the selftest has a workdir to build fixtures in never +harness_selftest 330-the-incomplete-path-must-run-whole running the real loop over both fixtures fails the major never +harness_selftest 330-the-incomplete-path-must-run-whole the loop delegates each verdict to pgc_tally_suite never +harness_selftest 330-the-incomplete-path-must-run-whole the runner classifies the file that suite actually produced never +harness_selftest 330-the-incomplete-path-must-run-whole with an incomplete suite in the tally the major reports FAIL never +harness_selftest 340-the-binary-must-be-built-from PREMISE and the target really holds sources find would otherwise hash never +harness_selftest 340-the-binary-must-be-built-from PREMISE the Makefile's recursion was actually parsed never +harness_selftest 340-the-binary-must-be-built-from PREMISE the copy discovers the same build directories as the real tree never +harness_selftest 340-the-binary-must-be-built-from PREMISE the fingerprint covers at least src never +harness_selftest 340-the-binary-must-be-built-from PREMISE the fixture's src really is a symlink never +harness_selftest 340-the-binary-must-be-built-from a /./ segment hashes the same tree the same way never +harness_selftest 340-the-binary-must-be-built-from a /src/.. segment hashes the same tree the same way never +harness_selftest 340-the-binary-must-be-built-from a caller passing a major is caught never +harness_selftest 340-the-binary-must-be-built-from a file that is not a build input does not move it never +harness_selftest 340-the-binary-must-be-built-from a fingerprint different from the record is stale never +harness_selftest 340-the-binary-must-be-built-from a fingerprint equal to the record is fresh never +harness_selftest 340-the-binary-must-be-built-from a fingerprint is 12 hex characters never +harness_selftest 340-the-binary-must-be-built-from a library newer than the running server is REFUSED never +harness_selftest 340-the-binary-must-be-built-from a library older than the running server is accepted never +harness_selftest 340-the-binary-must-be-built-from a missing binary timestamp is unknown, not predates never +harness_selftest 340-the-binary-must-be-built-from a missing postmaster timestamp is unknown, not predates never +harness_selftest 340-the-binary-must-be-built-from a new source file under objstore moves the fingerprint never +harness_selftest 340-the-binary-must-be-built-from a relative path hashes the same tree the same way never +harness_selftest 340-the-binary-must-be-built-from a server older than the binary predates it never +harness_selftest 340-the-binary-must-be-built-from a server started after the binary is fresh never +harness_selftest 340-the-binary-must-be-built-from a server started at the same second is fresh never +harness_selftest 340-the-binary-must-be-built-from a symlink to the tree hashes it the same way never +harness_selftest 340-the-binary-must-be-built-from a symlinked src contributes nothing, as find -P contributes nothing never +harness_selftest 340-the-binary-must-be-built-from a trailing slash hashes the same tree the same way never +harness_selftest 340-the-binary-must-be-built-from a tree with no hashable file yields no fingerprint never +harness_selftest 340-the-binary-must-be-built-from adding a source file moves it never +harness_selftest 340-the-binary-must-be-built-from an added file appears in the manifest by name never +harness_selftest 340-the-binary-must-be-built-from an added file shows up in the report never +harness_selftest 340-the-binary-must-be-built-from an empty manifest is reported as empty, not as silence never +harness_selftest 340-the-binary-must-be-built-from an unhashable tree has an empty manifest never +harness_selftest 340-the-binary-must-be-built-from an unreadable b.c yields no fingerprint, not a wrong one never +harness_selftest 340-the-binary-must-be-built-from an unreadable c.c yields no fingerprint, not a wrong one never +harness_selftest 340-the-binary-must-be-built-from an unreadable library is not a failure never +harness_selftest 340-the-binary-must-be-built-from and a non-numeric timestamp is unknown rather than compared as text never +harness_selftest 340-the-binary-must-be-built-from and an uncomputable current fingerprint is unknown, not stale never +harness_selftest 340-the-binary-must-be-built-from and comparing two manifests names it rather than saying 'changed' never +harness_selftest 340-the-binary-must-be-built-from and it says so rather than staying silent never +harness_selftest 340-the-binary-must-be-built-from and removing it restores the fingerprint never +harness_selftest 340-the-binary-must-be-built-from and restoring it restores the fingerprint never +harness_selftest 340-the-binary-must-be-built-from and restoring the partition restores the fingerprint never +harness_selftest 340-the-binary-must-be-built-from and the major is still readable in the name never +harness_selftest 340-the-binary-must-be-built-from and the reader reads back the fingerprint the writer recorded never +harness_selftest 340-the-binary-must-be-built-from and the refusal says the server must be restarted never +harness_selftest 340-the-binary-must-be-built-from and two pg_configs for one prefix share a stamp, keyed on pkglibdir never +harness_selftest 340-the-binary-must-be-built-from but it says which question went unanswered never +harness_selftest 340-the-binary-must-be-built-from control: a caller passing a pg_config is not flagged never +harness_selftest 340-the-binary-must-be-built-from control: a readable run still reads fresh never +harness_selftest 340-the-binary-must-be-built-from control: a real content change still moves the fingerprint never +harness_selftest 340-the-binary-must-be-built-from control: a real src directory is still hashed never +harness_selftest 340-the-binary-must-be-built-from control: and it still succeeds on a writable one never +harness_selftest 340-the-binary-must-be-built-from control: and restoring the content restores the fingerprint never +harness_selftest 340-the-binary-must-be-built-from control: and the tree fingerprints again once it is readable never +harness_selftest 340-the-binary-must-be-built-from control: the same pg_config twice gives the same path never +harness_selftest 340-the-binary-must-be-built-from control: writing the value it was given never +harness_selftest 340-the-binary-must-be-built-from each manifest line is a tree-relative path and a digest never +harness_selftest 340-the-binary-must-be-built-from editing a source file moves the fingerprint never +harness_selftest 340-the-binary-must-be-built-from every directory the Makefile builds from is in the fingerprint never +harness_selftest 340-the-binary-must-be-built-from moving bytes between files moves the fingerprint never +harness_selftest 340-the-binary-must-be-built-from no caller passes a major where a pg_config belongs never +harness_selftest 340-the-binary-must-be-built-from no record at all is unknown, not fresh never +harness_selftest 340-the-binary-must-be-built-from one tree, one fingerprint, whatever the locale never +harness_selftest 340-the-binary-must-be-built-from premise: and the stamp really was not written, so the arm is not vacuous never +harness_selftest 340-the-binary-must-be-built-from premise: at least two locales are installed to compare never +harness_selftest 340-the-binary-must-be-built-from premise: both fake configs report the same major, which is the whole point never +harness_selftest 340-the-binary-must-be-built-from premise: the argument parser reads the second argument at all never +harness_selftest 340-the-binary-must-be-built-from premise: the build path ran to completion, so a stamp was due never +harness_selftest 340-the-binary-must-be-built-from premise: the fixture fingerprints at all never +harness_selftest 340-the-binary-must-be-built-from premise: the same function returns a fingerprint for a real tree never +harness_selftest 340-the-binary-must-be-built-from premise: the spelling fixture fingerprints at all never +harness_selftest 340-the-binary-must-be-built-from premise: the sweep finds the call sites it is meant to police never +harness_selftest 340-the-binary-must-be-built-from premise: the tree fingerprints to something when it is readable never +harness_selftest 340-the-binary-must-be-built-from premise: the unprivileged read agrees while everything is readable never +harness_selftest 340-the-binary-must-be-built-from premise: the writer wrote a stamp at all never +harness_selftest 340-the-binary-must-be-built-from renaming a source file moves the fingerprint too never +harness_selftest 340-the-binary-must-be-built-from so the tree still fingerprints from its root files alone never +harness_selftest 340-the-binary-must-be-built-from so the verdict is fresh, not unknown never +harness_selftest 340-the-binary-must-be-built-from so the verdict is unknown -- UNVERIFIED -- and never stale never +harness_selftest 340-the-binary-must-be-built-from the fingerprint is the hash of the manifest never +harness_selftest 340-the-binary-must-be-built-from the fixed fingerprint equals what the previous implementation produced never +harness_selftest 340-the-binary-must-be-built-from the manifest is tree-relative, never absolute never +harness_selftest 340-the-binary-must-be-built-from the manifest names every file the fingerprint hashes never +harness_selftest 340-the-binary-must-be-built-from the probe is written outside the live source tree never +harness_selftest 340-the-binary-must-be-built-from the report names each hashed file never +harness_selftest 340-the-binary-must-be-built-from the report states how many files it hashed never +harness_selftest 340-the-binary-must-be-built-from the same tree fingerprints the same twice never +harness_selftest 340-the-binary-must-be-built-from the stamp writer reports failure on an unwritable target never +harness_selftest 340-the-binary-must-be-built-from the writer writes the file the reader looks for never +harness_selftest 340-the-binary-must-be-built-from two installations of one major get different stamp paths never +harness_selftest 340-the-binary-must-be-built-from two unreadable pg_configs do not alias onto one stamp never +harness_selftest 350-the-pytest-corpus-must-be README.md quotes the number of modes the inventory names as refused never +harness_selftest 350-the-pytest-corpus-must-be TESTS.md states no totals line for a merge to get wrong never +harness_selftest 350-the-pytest-corpus-must-be TESTS.md states the counted number as well never +harness_selftest 350-the-pytest-corpus-must-be a documented file that does not exist is named never +harness_selftest 350-the-pytest-corpus-must-be a documented test that does not exist is named, not passed over never +harness_selftest 350-the-pytest-corpus-must-be a name defined in two files is named, not passed over never +harness_selftest 350-the-pytest-corpus-must-be a prose total that disagrees with the ids is visible never +harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with disk is visible never +harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with the ids is visible never +harness_selftest 350-the-pytest-corpus-must-be an absent prose total is empty rather than a stray number never +harness_selftest 350-the-pytest-corpus-must-be an absent total is empty rather than a number that happens to match never +harness_selftest 350-the-pytest-corpus-must-be an id named twice counts once never +harness_selftest 350-the-pytest-corpus-must-be an id of fewer than three words is not counted as a mode never +harness_selftest 350-the-pytest-corpus-must-be an unbackticked name in prose is not treated as a claim never +harness_selftest 350-the-pytest-corpus-must-be an undocumented file is caught along with the tests inside it never +harness_selftest 350-the-pytest-corpus-must-be an undocumented test is named rather than passed over never +harness_selftest 350-the-pytest-corpus-must-be and the same comparison agrees on the fixture that is right never +harness_selftest 350-the-pytest-corpus-must-be control: a document naming only what exists is clean never +harness_selftest 350-the-pytest-corpus-must-be control: a fully documented corpus reports nothing missing never +harness_selftest 350-the-pytest-corpus-must-be control: distinct names in the same corpus report no duplicate never +harness_selftest 350-the-pytest-corpus-must-be every test file and every test in the corpus is named in TESTS.md never +harness_selftest 350-the-pytest-corpus-must-be every test the document names exists in the corpus never +harness_selftest 350-the-pytest-corpus-must-be no test name is defined twice in the corpus never +harness_selftest 350-the-pytest-corpus-must-be premise: the corpus carries the documentation this part polices never +harness_selftest 350-the-pytest-corpus-must-be premise: the counting rule finds modes at all never +harness_selftest 350-the-pytest-corpus-must-be premise: the mode inventory is where this part thinks it is never +harness_selftest 350-the-pytest-corpus-must-be premise: the pytest corpus is where this part thinks it is never +harness_selftest 350-the-pytest-corpus-must-be premise: the reader still finds a totals line when one is there never +harness_selftest 350-the-pytest-corpus-must-be premise: the reverse sweep reads backticked names at all never +harness_selftest 350-the-pytest-corpus-must-be premise: the sweep found the corpus rather than an empty glob never +harness_selftest 350-the-pytest-corpus-must-be section 1a's document total is the sum of its two sections never +harness_selftest 350-the-pytest-corpus-must-be section 1a's not-refused total is the count of ids in section 3 never +harness_selftest 350-the-pytest-corpus-must-be section 1a's refused total is the count of ids in section 2 never +harness_selftest 350-the-pytest-corpus-must-be section 2's opening states the counted number of refused modes never +harness_selftest 350-the-pytest-corpus-must-be the admitted gap is the run total minus what is written down never +harness_selftest 350-the-pytest-corpus-must-be the closing paragraph states the counted number too never +harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 2 never +harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 3 never +harness_selftest 350-the-pytest-corpus-must-be the counter stops at the next heading never +harness_selftest 350-the-pytest-corpus-must-be the row's value is read, not a digit inside its label never +harness_selftest 350-the-pytest-corpus-must-be the sweep counts the fixture's tests and files never +harness_selftest 360-an-unrunnable-pytest-test-must a comparison on the exit status is not counted as an assignment never +harness_selftest 360-an-unrunnable-pytest-test-must a drifted exit code is visible rather than absorbed never +harness_selftest 360-an-unrunnable-pytest-test-must a write-only unrunnable field is caught never +harness_selftest 360-an-unrunnable-pytest-test-must an unconditional exit override is caught by the dominance arm never +harness_selftest 360-an-unrunnable-pytest-test-must and only ever moves a run off zero, so a failure still dominates never +harness_selftest 360-an-unrunnable-pytest-test-must and something READS it, rather than only writing it never +harness_selftest 360-an-unrunnable-pytest-test-must premise: and that same fixture does show the write, so the arm is not blind never +harness_selftest 360-an-unrunnable-pytest-test-must premise: lib.sh states an INCOMPLETE exit code this part could read never +harness_selftest 360-an-unrunnable-pytest-test-must premise: the harness library is where this part thinks it is never +harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer is where this part thinks it is never +harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer states one too never +harness_selftest 360-an-unrunnable-pytest-test-must premise: while a real assignment on the same line shape IS counted never +harness_selftest 360-an-unrunnable-pytest-test-must premise: while the real layer satisfies that same arm never +harness_selftest 360-an-unrunnable-pytest-test-must the layer ends a session by setting its exit status never +harness_selftest 360-an-unrunnable-pytest-test-must the layer prints the unrunnable reason in lib.sh's shape never +harness_selftest 360-an-unrunnable-pytest-test-must the layer still writes the unrunnable state never +harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses agree on the INCOMPLETE exit code never +harness_selftest 370-the-plan-marker-guard-must a neutered absent arm is caught never +harness_selftest 370-the-plan-marker-guard-must a neutered empty-plan refusal is caught never +harness_selftest 370-the-plan-marker-guard-must a neutered present arm is caught never +harness_selftest 370-the-plan-marker-guard-must and that refusal is a VacuityError, not an ordinary assertion never +harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is absent never +harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is present never +harness_selftest 370-the-plan-marker-guard-must plan_marker refuses a plan with no nodes at all never +harness_selftest 370-the-plan-marker-guard-must premise: both line numbers were found, so the ordering arm can mean something never +harness_selftest 370-the-plan-marker-guard-must premise: plan_marker's body was actually cut out of the file never +harness_selftest 370-the-plan-marker-guard-must premise: the pytest layer is where this part thinks it is never +harness_selftest 370-the-plan-marker-guard-must premise: while the real body satisfies all three, so the greps work never +harness_selftest 370-the-plan-marker-guard-must the empty-plan refusal precedes the arm it protects never +harness_selftest 380-the-pytest-cluster-helpers a caller that reimplements the digest is caught never +harness_selftest 380-the-pytest-cluster-helpers a fingerprint that reads src only is caught never +harness_selftest 380-the-pytest-cluster-helpers a make_cluster with no cleanup is caught never +harness_selftest 380-the-pytest-cluster-helpers an import from the pytest tree is caught never +harness_selftest 380-the-pytest-cluster-helpers and a hard-coded module list is caught by the name arm never +harness_selftest 380-the-pytest-cluster-helpers and it catches BaseException, so an interrupt cleans up too never +harness_selftest 380-the-pytest-cluster-helpers and it names no module directory, so it is a derivation and not a list never +harness_selftest 380-the-pytest-cluster-helpers and it stops a partially started cluster before removing the tree never +harness_selftest 380-the-pytest-cluster-helpers and no longer mixes in the bare filename never +harness_selftest 380-the-pytest-cluster-helpers and the original error is re-raised rather than swallowed never +harness_selftest 380-the-pytest-cluster-helpers and the shell keeps none either never +harness_selftest 380-the-pytest-cluster-helpers make_cluster removes its tree when setup raises never +harness_selftest 380-the-pytest-cluster-helpers premise: and the real helper still carries its cleanup never +harness_selftest 380-the-pytest-cluster-helpers premise: make_cluster's body was actually cut out of the file never +harness_selftest 380-the-pytest-cluster-helpers premise: the one fingerprint implementation is where this part thinks it is never +harness_selftest 380-the-pytest-cluster-helpers premise: the pytest cluster helper is where this part thinks it is never +harness_selftest 380-the-pytest-cluster-helpers premise: while the real module satisfies the derivation arm never +harness_selftest 380-the-pytest-cluster-helpers the fingerprint derives its build directories from a Makefile on disk never +harness_selftest 380-the-pytest-cluster-helpers the hash mixes in each file's path relative to the tree, not its name never +harness_selftest 380-the-pytest-cluster-helpers the module imports nothing from the pytest tree never +harness_selftest 380-the-pytest-cluster-helpers the pytest helper keeps no private fingerprint implementation never +harness_selftest 390-a-registered-suite-must-account a NEW unaccounted suite fails even while the known debt is excused never +harness_selftest 390-a-registered-suite-must-account a comment mentioning pgc_summary is not a declaration never +harness_selftest 390-a-registered-suite-must-account a declared suite that produced no accounting is caught never +harness_selftest 390-a-registered-suite-must-account a declared suite the driver never dispatched reconciles never +harness_selftest 390-a-registered-suite-must-account a file that does not exist is reported absent, not exempt never +harness_selftest 390-a-registered-suite-must-account a hash inside a word does not hide the call after it never +harness_selftest 390-a-registered-suite-must-account a log carrying lib.sh's accounting line is accounted never +harness_selftest 390-a-registered-suite-must-account a log carrying neither is not accounted never +harness_selftest 390-a-registered-suite-must-account a log claiming PASSED without the accounting line shows none never +harness_selftest 390-a-registered-suite-must-account a long suite that calls pgc_summary still declares accounting never +harness_selftest 390-a-registered-suite-must-account a longer name containing pgc_summary is not a declaration never +harness_selftest 390-a-registered-suite-must-account a passing log shows accounting never +harness_selftest 390-a-registered-suite-must-account a registered suite that is accounted by nothing FAILS never +harness_selftest 390-a-registered-suite-must-account a suite recorded as known debt passes never +harness_selftest 390-a-registered-suite-must-account a suite recorded as never dispatched that DID account is caught never +harness_selftest 390-a-registered-suite-must-account a suite that accounted passes never +harness_selftest 390-a-registered-suite-must-account a suite that calls pgc_summary declares accounting never +harness_selftest 390-a-registered-suite-must-account a suite that never calls it does not never +harness_selftest 390-a-registered-suite-must-account a suite that now accounts but is still listed as debt is reported never +harness_selftest 390-a-registered-suite-must-account a suite the driver never dispatched passes never +harness_selftest 390-a-registered-suite-must-account a trailing comment after the call does not hide it never +harness_selftest 390-a-registered-suite-must-account an absent log shows no accounting rather than erroring never +harness_selftest 390-a-registered-suite-must-account an accounting line that does not start its line is refused never +harness_selftest 390-a-registered-suite-must-account an indented comment is still a comment never +harness_selftest 390-a-registered-suite-must-account an undeclared suite that DID account is caught too never +harness_selftest 390-a-registered-suite-must-account and a failed population reconciliation fails the major never +harness_selftest 390-a-registered-suite-must-account and a failed reconciliation sets the per-major failure flag never +harness_selftest 390-a-registered-suite-must-account and a log carrying only its OWN checks-run line is accounted too never +harness_selftest 390-a-registered-suite-must-account and a reworded producer line is refused, so the arm can fail never +harness_selftest 390-a-registered-suite-must-account and a skip, which reached the summary and counted zero never +harness_selftest 390-a-registered-suite-must-account and absent is distinguishable from a present file that does not declare never +harness_selftest 390-a-registered-suite-must-account and an incomplete never +harness_selftest 390-a-registered-suite-must-account and debt naming a suite that is not registered is reported too never +harness_selftest 390-a-registered-suite-must-account and it agrees with the real reader on a SHORT file, which is why it survived review never +harness_selftest 390-a-registered-suite-must-account and it is NAMED, so the reader does not have to diff two lists never +harness_selftest 390-a-registered-suite-must-account and it is named as that fault, not as one of the other two never +harness_selftest 390-a-registered-suite-must-account and it is named as the opposite fault, not the same one never +harness_selftest 390-a-registered-suite-must-account and it is named, which the symmetry check could never do never +harness_selftest 390-a-registered-suite-must-account and prose containing the word does not count as the line never +harness_selftest 390-a-registered-suite-must-account and so does a failing one, which is the point never +harness_selftest 390-a-registered-suite-must-account and the excused one is not named as a failure never +harness_selftest 390-a-registered-suite-must-account and the reader answers no on it, which is the wrong answer the arm catches never +harness_selftest 390-a-registered-suite-must-account and the real function reconciles the same input, so the arm is not noise never +harness_selftest 390-a-registered-suite-must-account and the reconciliation is given that record never +harness_selftest 390-a-registered-suite-must-account and the registered file is written from the SUITES array itself never +harness_selftest 390-a-registered-suite-must-account and without that record the same run is still caught never +harness_selftest 390-a-registered-suite-must-account equal sets reconcile never +harness_selftest 390-a-registered-suite-must-account every registered suite has a file never +harness_selftest 390-a-registered-suite-must-account nor no for every one of them never +harness_selftest 390-a-registered-suite-must-account opposite errors do not cancel: both directions are reported never +harness_selftest 390-a-registered-suite-must-account premise: and produced exactly one accounting line to be read never +harness_selftest 390-a-registered-suite-must-account premise: and that count excludes the definition line, which mentions it never +harness_selftest 390-a-registered-suite-must-account premise: and the accounted reader that feeds it never +harness_selftest 390-a-registered-suite-must-account premise: and the real function still does never +harness_selftest 390-a-registered-suite-must-account premise: pipefail is on, which is the condition the bug needs never +harness_selftest 390-a-registered-suite-must-account premise: the declaration reader evalled out of the runner is callable never +harness_selftest 390-a-registered-suite-must-account premise: the drift changed the line the reader looks for never +harness_selftest 390-a-registered-suite-must-account premise: the fixture carries a well-formed accounting line, just indented never +harness_selftest 390-a-registered-suite-must-account premise: the fixture is long enough to lose the race never +harness_selftest 390-a-registered-suite-must-account premise: the fixture really does hide its call from the stripper never +harness_selftest 390-a-registered-suite-must-account premise: the mutation applied -- the twin no longer sorts its inputs never +harness_selftest 390-a-registered-suite-must-account premise: the observation reader evalled out of the runner is callable never +harness_selftest 390-a-registered-suite-must-account premise: the population reconciliation is callable never +harness_selftest 390-a-registered-suite-must-account premise: the real suite ran and reached its summary never +harness_selftest 390-a-registered-suite-must-account premise: the reconciliation evalled out of the runner is callable never +harness_selftest 390-a-registered-suite-must-account premise: the registered list is not empty, so the partition means something never +harness_selftest 390-a-registered-suite-must-account premise: the runner defines the declaration reader this part evals never +harness_selftest 390-a-registered-suite-must-account premise: the runner defines the observation reader this part evals never +harness_selftest 390-a-registered-suite-must-account premise: the runner defines the population reconciliation never +harness_selftest 390-a-registered-suite-must-account premise: the runner defines the reconciliation this part evals never +harness_selftest 390-a-registered-suite-must-account premise: the twin script was written and is runnable never +harness_selftest 390-a-registered-suite-must-account premise: the unsorted twin is callable never +harness_selftest 390-a-registered-suite-must-account the debt file is in the tree never +harness_selftest 390-a-registered-suite-must-account the grep -q shape is the one that gets this wrong under pipefail never +harness_selftest 390-a-registered-suite-must-account the identity catches comm reading unsorted input never +harness_selftest 390-a-registered-suite-must-account the partition over the real suite list adds up never +harness_selftest 390-a-registered-suite-must-account the population partitions, and prints inputs == sum(buckets) never +harness_selftest 390-a-registered-suite-must-account the reader accepts the line the producer actually emits never +harness_selftest 390-a-registered-suite-must-account the reader does not answer yes for every registered suite never +harness_selftest 390-a-registered-suite-must-account the reconciliation prints inputs == sum(buckets) never +harness_selftest 390-a-registered-suite-must-account the record cannot introduce a suite the source never declared never +harness_selftest 390-a-registered-suite-must-account the runner calls the population reconciliation never +harness_selftest 390-a-registered-suite-must-account the runner calls the reconciliation, not merely defines it never +harness_selftest 390-a-registered-suite-must-account the skip branch records the suite it did not dispatch never +harness_selftest 390-a-registered-suite-must-account the stripper hides no pgc_summary call in any registered suite never +harness_selftest 400-a-check-result-must-be-machine a bogus reason code records FAIL, not UNRUN never +harness_selftest 400-a-check-result-must-be-machine a failing check emits exactly one record never +harness_selftest 400-a-check-result-must-be-machine a failing check still prints its old line never +harness_selftest 400-a-check-result-must-be-machine a log that never stated a count is not silently accepted never +harness_selftest 400-a-check-result-must-be-machine a log whose records match its stated count reconciles never +harness_selftest 400-a-check-result-must-be-machine a log with fewer records than it claims is caught never +harness_selftest 400-a-check-result-must-be-machine a log with more records than it claims is caught too never +harness_selftest 400-a-check-result-must-be-machine a passing check emits exactly one record never +harness_selftest 400-a-check-result-must-be-machine a passing check still prints its old line never +harness_selftest 400-a-check-result-must-be-machine an unrunnable check emits exactly one record never +harness_selftest 400-a-check-result-must-be-machine an unrunnable check still prints its old line never +harness_selftest 400-a-check-result-must-be-machine and its name field is the check's name, spaces intact never +harness_selftest 400-a-check-result-must-be-machine and its verdict field says FAIL never +harness_selftest 400-a-check-result-must-be-machine and its verdict field says PASS never +harness_selftest 400-a-check-result-must-be-machine and its verdict field says UNRUN, which is neither of the other two never +harness_selftest 400-a-check-result-must-be-machine and records FAIL never +harness_selftest 400-a-check-result-must-be-machine and records FAIL, because nothing was compared never +harness_selftest 400-a-check-result-must-be-machine and records PASS when the ratio is inside the bound never +harness_selftest 400-a-check-result-must-be-machine and that place is pgc_record never +harness_selftest 400-a-check-result-must-be-machine and the REASON_CODE travels in the reason field, not in prose never +harness_selftest 400-a-check-result-must-be-machine and the two numbers are named, not just the verdict never +harness_selftest 400-a-check-result-must-be-machine check_num on a non-number emits one record never +harness_selftest 400-a-check-result-must-be-machine check_num's non-measurement line is unchanged never +harness_selftest 400-a-check-result-must-be-machine check_ratio on a non-number emits one record never +harness_selftest 400-a-check-result-must-be-machine check_ratio that forms a ratio emits one record never +harness_selftest 400-a-check-result-must-be-machine check_ratio with a zero side emits one record never +harness_selftest 400-a-check-result-must-be-machine check_text on an empty side emits one record never +harness_selftest 400-a-check-result-must-be-machine check_text's empty-side line is unchanged never +harness_selftest 400-a-check-result-must-be-machine lib.sh bumps PGC_CHECKS in exactly one place never +harness_selftest 400-a-check-result-must-be-machine pgc_fail emits one record never +harness_selftest 400-a-check-result-must-be-machine pgc_pass emits one record never +harness_selftest 400-a-check-result-must-be-machine premise: and that is this fragment, not the suite never +harness_selftest 400-a-check-result-must-be-machine premise: it is callable never +harness_selftest 400-a-check-result-must-be-machine premise: lib.sh is where the check helpers live never +harness_selftest 400-a-check-result-must-be-machine premise: the probe ran every helper shape once never +harness_selftest 400-a-check-result-must-be-machine the record count equals the counter the summary reports never +harness_selftest 400-a-check-result-must-be-machine the record names the part the check was asked from never +harness_selftest 400-a-check-result-must-be-machine the runner calls the record reconciliation, not merely defines it never +harness_selftest 400-a-check-result-must-be-machine the runner defines the record reconciliation never +harness_selftest 410-a-check-must-have-been-red a check merely added is not reported as a rename never +harness_selftest 410-a-check-must-have-been-red a check merely removed is not reported as a rename either never +harness_selftest 410-a-check-must-have-been-red a check observed red gains the date it was seen never +harness_selftest 410-a-check-must-have-been-red a check the ledger has never seen is refused, not absorbed never +harness_selftest 410-a-check-must-have-been-red a duplicated check name is reported by name never +harness_selftest 410-a-check-must-have-been-red a later green run does not erase an observation never +harness_selftest 410-a-check-must-have-been-red a merge that names its mutation records it against the check that reddened never +harness_selftest 410-a-check-must-have-been-red a name that appeared while another disappeared is reported as a rename never +harness_selftest 410-a-check-must-have-been-red a run whose debt is within budget passes the gate never +harness_selftest 410-a-check-must-have-been-red and a check that stayed green keeps its debt never +harness_selftest 410-a-check-must-have-been-red and a unique one is not never +harness_selftest 410-a-check-must-have-been-red and it is empty when nothing named a mutation never +harness_selftest 410-a-check-must-have-been-red and it is named, so the author knows which one never +harness_selftest 410-a-check-must-have-been-red and names the suite and the check, not just a count never +harness_selftest 410-a-check-must-have-been-red and not against one that stayed green never +harness_selftest 410-a-check-must-have-been-red and one over budget does not never +harness_selftest 410-a-check-must-have-been-red and records neither as ever having been red never +harness_selftest 410-a-check-must-have-been-red and the gate says which number was exceeded, by how much never +harness_selftest 410-a-check-must-have-been-red and the stable check is not reported never +harness_selftest 410-a-check-must-have-been-red every ledger row carries four fields, the fourth being the mutation never +harness_selftest 410-a-check-must-have-been-red merging a green run records both checks never +harness_selftest 410-a-check-must-have-been-red premise: the budget is a tracked file too never +harness_selftest 410-a-check-must-have-been-red premise: the check has history before the rename never +harness_selftest 410-a-check-must-have-been-red premise: the ledger is not empty, so the partition means something never +harness_selftest 410-a-check-must-have-been-red premise: the ledger itself is a tracked file, not a variable never +harness_selftest 410-a-check-must-have-been-red premise: the ledger tool exists never +harness_selftest 410-a-check-must-have-been-red the census reads a run's records never +harness_selftest 410-a-check-must-have-been-red the committed budget matches the committed ledger's debt never +harness_selftest 410-a-check-must-have-been-red the committed budget names both debts never +harness_selftest 410-a-check-must-have-been-red the ledger partitions into observed and never never +harness_selftest 410-a-check-must-have-been-red the two collapse to one row, which is the loss being reported never diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index c7e49a16..f0f52dcf 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -24,5 +24,5 @@ # invisible to everything above: the gate cannot refuse a new check in a suite # it has never seen. Counted separately so that "we ledger 605 checks" cannot # read as "we ledger the corpus". -checks_never_observed_red 608 +checks_never_observed_red 613 suites_not_covered 250 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index cdab1ac0..772bfecc 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -35,10 +35,14 @@ ------ Tab separated, one row per check, sorted: - suite check name last observed red mutation + suite part check name last observed red mutation `last observed red` is a date, or the literal `never`. `mutation` is free text or empty. Both are written by this tool, never by hand. + +The row is keyed on (suite, part, name). The part matters because harness_selftest +sources 40-odd parts into one shell and phrases its premises to be COPIED, so a +name-only key is a key of check NAMES rather than of checks. """ import argparse @@ -60,14 +64,23 @@ def read_records(paths): if not line.startswith("RESULT\t"): continue f = line.split("\t") - if len(f) < 4: + if len(f) < 5: continue - out.append((f[1], f[2], f[3])) + # suite, part, name, verdict + out.append((f[1], f[2], f[3], f[4])) return out def read_ledger(path): - """{(suite, name): [last_red, mutation]} from a ledger file.""" + """{(suite, part, name): [last_red, mutation]} from a ledger file. + + KEYED ON THE PART AS WELL AS THE NAME. harness_selftest sources 40-odd parts + into one shell and phrases its premises to be copied -- "premise: the pytest + layer is where THIS PART thinks it is" works verbatim in any of them -- so + (suite, name) is a key of check NAMES rather than of checks, and one sharer + going red would mark them all. Measured over a real run: 583 records give 579 + distinct (suite, name) and 582 distinct (suite, part, name). + """ rows = {} p = pathlib.Path(path) if not p.exists(): @@ -76,30 +89,30 @@ def read_ledger(path): if not line.strip() or line.startswith("#"): continue f = line.split("\t") - while len(f) < 4: + while len(f) < 5: f.append("") - rows[(f[0], f[1])] = [f[2] or NEVER, f[3]] + rows[(f[0], f[1], f[2])] = [f[3] or NEVER, f[4]] return rows def write_ledger(path, rows): lines = [ - "\t".join((suite, name, v[0], v[1])) - for (suite, name), v in sorted(rows.items()) + "\t".join((suite, part, name, v[0], v[1])) + for (suite, part, name), v in sorted(rows.items()) ] pathlib.Path(path).write_text("\n".join(lines) + ("\n" if lines else "")) def cmd_census(args): - for suite, name, verdict in read_records(args.logs): - print(f"{suite}\t{name}\t{verdict}") + for suite, part, name, verdict in read_records(args.logs): + print(f"{suite}\t{part}\t{name}\t{verdict}") return 0 def cmd_merge(args): rows = read_ledger(args.ledger) - for suite, name, verdict in read_records(args.logs): - key = (suite, name) + for suite, part, name, verdict in read_records(args.logs): + key = (suite, part, name) if key not in rows: # A check this ledger has never seen enters as DEBT. A green run has # observed nothing go red, so merging one must never record a red @@ -116,16 +129,18 @@ def cmd_merge(args): # BOTH as observed red -- a claim about a check nothing attacked, which is # precisely what this ledger must not make. It cannot be fixed by keying # harder without a synthetic id someone would maintain, so it is reported. + # A duplicate WITHIN one part still shares a row -- the part fixed the + # convention collisions, not genuine repeats. One survives in the real + # corpus, and naming it precisely is the point of keying on the part. records = read_records(args.logs) counts = {} - for suite, name, _ in records: - counts[(suite, name)] = counts.get((suite, name), 0) + 1 - dupes = sorted(k for k, c in counts.items() if c > 1) - for suite, name in dupes: + for suite, part, name, _ in records: + counts[(suite, part, name)] = counts.get((suite, part, name), 0) + 1 + for suite, part, name in sorted(k for k, c in counts.items() if c > 1): print(f" duplicate check name, so one ledger row covers " - f"{counts[(suite, name)]}: {suite}\t{name}") + f"{counts[(suite, part, name)]}: {suite}\t{part}\t{name}") - seen = len({(s, n) for s, n, _ in read_records(args.logs)}) + seen = len({(s, p_, n) for s, p_, n, _ in records}) red = sum(1 for v in rows.values() if v[0] != NEVER) print(f" ledger: rows={len(rows)} | seen this run={seen}, " f"observed red ever={red}, never={len(rows) - red}") @@ -146,21 +161,24 @@ def cmd_rename_scan(args): thing ignored. """ rows = read_ledger(args.ledger) - now = {(s, n) for s, n, _ in read_records(args.logs)} - suites = {s for s, _ in now} - known = {(s, n) for (s, n) in rows if s in suites} + now = {(s, p_, n) for s, p_, n, _ in read_records(args.logs)} + parts = {(s, p_) for s, p_, _ in now} + known = {k for k in rows if (k[0], k[1]) in parts} appeared = sorted(now - known) vanished = sorted(known - now) rc = 0 if appeared and vanished: - # Only pair within a suite, and only report while both sides remain. - for (s_a, n_a), (s_v, n_v) in zip(appeared, vanished): - if s_a != s_v: + # Only pair within a PART. Keying on the part also fixes a blind spot the + # name-only key had: a premise moving between parts was indistinguishable + # from a rename, and now it is a disappearance and an appearance in two + # different parts, which this does not pair. + for (s_a, p_a, n_a), (s_v, p_v, n_v) in zip(appeared, vanished): + if (s_a, p_a) != (s_v, p_v): continue - was = rows.get((s_v, n_v), [NEVER, ""])[0] + was = rows.get((s_v, p_v, n_v), [NEVER, ""])[0] print(f" possible rename: {n_v} -> {n_a} " - f"(in {s_a}, history: last red {was})") + f"(in {s_a}/{p_a}, history: last red {was})") rc = 1 print(f" rename scan: appeared={len(appeared)}, vanished={len(vanished)}") return rc @@ -184,15 +202,15 @@ def _read_budget(path): def cmd_gate(args): rows = read_ledger(args.ledger) budget = _read_budget(args.budget) - seen = {(s, n) for s, n, _ in read_records(args.logs)} + seen = {(s, p_, n) for s, p_, n, _ in read_records(args.logs)} # A check the ledger has never heard of is NEW. The allowlist exists so a # gate that fails on 3,762 unledgered sites is not what lands -- but a new # one must not enter as silent debt either. unknown = sorted(seen - set(rows)) rc = 0 - for suite, name in unknown: - print(f" not in the ledger: {suite}\t{name}") + for suite, part, name in unknown: + print(f" not in the ledger: {suite}\t{part}\t{name}") rc = 1 never = sum(1 for v in rows.values() if v[0] == NEVER) @@ -214,7 +232,7 @@ def cmd_gate(args): if args.registered: registered = {l.strip() for l in pathlib.Path(args.registered).read_text().split() if l.strip()} - covered = {s for s, _ in rows} + covered = {k[0] for k in rows} uncovered = sorted(registered - covered) want_s = budget.get("suites_not_covered") print(f" ledger coverage: registered={len(registered)} | " diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 226698de..b4100572 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1242,7 +1242,8 @@ run is the deliberate accelerator, not the only source. suite check name last observed red mutation ``` -`last observed red` is a date or the literal `never`. The **mutation column exists from +`last observed red` is a date or the literal `never`. The row is keyed on the first +three fields. The **mutation column exists from v1 with nothing filling it automatically**, because adding a column later means rewriting every entry — and if an entry can record *which* mutation reddened a check, the catalogue a mutation gate would need builds itself out of work people already do by @@ -1279,10 +1280,22 @@ whole thing gets ignored. ### `test_a_duplicated_check_name_shares_one_row_and_is_reported` -Two checks with the same name in one suite share a ledger row, so one going red marks -**both** as observed red — a claim about a check nothing attacked. Reported rather than -prevented, for the same reason as the rename. The real corpus carries four today, which -is how this was noticed: 609 records reduced to 605 rows. +Two checks with the same name **in one part** share a ledger row, so one going red marks +**both** as observed red — a claim about a check nothing attacked. + +The key is `(suite, part, name)`, not `(suite, name)`. `harness_selftest` sources 40-odd +parts into one shell and phrases its premises to be **copied** — *"premise: the pytest +layer is where **this part** thinks it is"* works verbatim in any of them — so a +name-only key is a key of check *names*, and the collision count grows with every part +anyone writes. Measured over a real run of 583 records: 579 distinct `(suite, name)` +against 582 distinct `(suite, part, name)`. The part is derived from `BASH_SOURCE` +rather than from a convention, so the next part written the same way is keyed correctly +without anyone remembering. + +One duplicate survives that, and it is a genuine one: +`340-the-binary-must-be-built-from` asks `premise: the fixture fingerprints at all` +twice within the same part. That is the kind of thing the ledger can now name precisely +instead of losing among convention artifacts. ### `test_the_gate_refuses_a_check_the_ledger_has_never_seen` diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index 4760012f..b4a6ff00 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -48,11 +48,11 @@ def _rows(path): return [l.split("\t") for l in pathlib.Path(path).read_text().splitlines() if l] -GREEN = ("RESULT\tdemo\tfirst check\tPASS\t\n" - "RESULT\tdemo\tsecond check\tPASS\t\n" +GREEN = ("RESULT\tdemo\tpart1\tfirst check\tPASS\t\n" + "RESULT\tdemo\tpart1\tsecond check\tPASS\t\n" "checks run: 2\n") -RED = ("RESULT\tdemo\tfirst check\tFAIL\t\n" - "RESULT\tdemo\tsecond check\tPASS\t\n" +RED = ("RESULT\tdemo\tpart1\tfirst check\tFAIL\t\n" + "RESULT\tdemo\tpart1\tsecond check\tPASS\t\n" "checks run: 2\n") @@ -68,7 +68,7 @@ def test_a_green_run_records_debt_and_never_a_red_observation(tmp_path, expect): _run("merge", "--ledger", ledger, log) rows = _rows(ledger) expect.num(len(rows), 2, "merging a green run records both checks") - expect.text(",".join(sorted({r[2] for r in rows})), "never", + expect.text(",".join(sorted({r[3] for r in rows})), "never", "and records neither as ever having been red") @@ -78,7 +78,7 @@ def test_a_red_observation_is_dated_and_survives_a_later_green_run(tmp_path, exp _run("merge", "--ledger", ledger, _write(tmp_path, "g.log", GREEN)) _run("merge", "--ledger", ledger, "--date", "2026-09-10", _write(tmp_path, "r.log", RED)) - by = {r[1]: r[2] for r in _rows(ledger)} + by = {r[2]: r[3] for r in _rows(ledger)} expect.text(by["first check"], "2026-09-10", "a check observed red gains the date it was seen") expect.text(by["second check"], "never", @@ -86,7 +86,7 @@ def test_a_red_observation_is_dated_and_survives_a_later_green_run(tmp_path, exp _run("merge", "--ledger", ledger, "--date", "2026-09-11", _write(tmp_path, "g2.log", GREEN)) - expect.text({r[1]: r[2] for r in _rows(ledger)}["first check"], "2026-09-10", + expect.text({r[2]: r[3] for r in _rows(ledger)}["first check"], "2026-09-10", "a later green run does not erase an observation") @@ -100,14 +100,14 @@ def test_the_mutation_column_exists_from_v1(tmp_path, expect): ledger = _write(tmp_path, "l.tsv", "") _run("merge", "--ledger", ledger, "--date", "2026-09-10", _write(tmp_path, "r.log", RED)) - expect.num(len([r for r in _rows(ledger) if len(r) != 4]), 0, + expect.num(len([r for r in _rows(ledger) if len(r) != 5]), 0, "every row carries four fields, the fourth being the mutation") - expect.text("[" + {r[1]: r[3] for r in _rows(ledger)}["first check"] + "]", "[]", + expect.text("[" + {r[2]: r[4] for r in _rows(ledger)}["first check"] + "]", "[]", "and it is empty when nothing named a mutation") _run("merge", "--ledger", ledger, "--date", "2026-09-10", "--mutation", "SAOP limit 128 -> 0", _write(tmp_path, "r2.log", RED)) - by = {r[1]: r[3] for r in _rows(ledger)} + by = {r[2]: r[4] for r in _rows(ledger)} expect.text(by["first check"], "SAOP limit 128 -> 0", "a named mutation is recorded against the check that reddened") expect.text("[" + by["second check"] + "]", "[]", @@ -123,27 +123,27 @@ def test_a_rename_is_reported_rather_than_silently_resetting_history(tmp_path, e rename, and reporting one on every new check is noise that gets it ignored. """ ledger = _write(tmp_path, "l.tsv", "") - before = ("RESULT\tdemo\tthe old name\tFAIL\t\n" - "RESULT\tdemo\ta stable check\tPASS\t\nchecks run: 2\n") + before = ("RESULT\tdemo\tpart1\tthe old name\tFAIL\t\n" + "RESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 2\n") _run("merge", "--ledger", ledger, "--date", "2026-09-01", _write(tmp_path, "b.log", before)) - after = ("RESULT\tdemo\tthe new name\tPASS\t\n" - "RESULT\tdemo\ta stable check\tPASS\t\nchecks run: 2\n") + after = ("RESULT\tdemo\tpart1\tthe new name\tPASS\t\n" + "RESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 2\n") out, _ = _run("rename-scan", "--ledger", ledger, _write(tmp_path, "a.log", after)) expect.num(out.count("possible rename: the old name -> the new name"), 1, "a name that appeared while another disappeared is reported") expect.num(out.count("a stable check"), 0, "and the stable check is not") added = after.replace("the new name", "the old name") + "" - added = ("RESULT\tdemo\tthe old name\tPASS\t\n" - "RESULT\tdemo\ta stable check\tPASS\t\n" - "RESULT\tdemo\ta genuinely new check\tPASS\t\nchecks run: 3\n") + added = ("RESULT\tdemo\tpart1\tthe old name\tPASS\t\n" + "RESULT\tdemo\tpart1\ta stable check\tPASS\t\n" + "RESULT\tdemo\tpart1\ta genuinely new check\tPASS\t\nchecks run: 3\n") out, _ = _run("rename-scan", "--ledger", ledger, _write(tmp_path, "add.log", added)) expect.num(out.count("possible rename"), 0, "a check merely added is not reported as a rename") - removed = "RESULT\tdemo\ta stable check\tPASS\t\nchecks run: 1\n" + removed = "RESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 1\n" out, _ = _run("rename-scan", "--ledger", ledger, _write(tmp_path, "rm.log", removed)) expect.num(out.count("possible rename"), 0, "nor is one merely removed") @@ -159,13 +159,13 @@ def test_a_duplicated_check_name_shares_one_row_and_is_reported(tmp_path, expect this was noticed: 609 records reduced to 605 rows. """ ledger = _write(tmp_path, "l.tsv", "") - dupe = ("RESULT\tdemo\tthe same name\tPASS\t\n" - "RESULT\tdemo\tthe same name\tFAIL\t\n" - "RESULT\tdemo\ta unique name\tPASS\t\nchecks run: 3\n") + dupe = ("RESULT\tdemo\tpart1\tthe same name\tPASS\t\n" + "RESULT\tdemo\tpart1\tthe same name\tFAIL\t\n" + "RESULT\tdemo\tpart1\ta unique name\tPASS\t\nchecks run: 3\n") out, _ = _run("merge", "--ledger", ledger, "--date", "2026-09-10", _write(tmp_path, "d.log", dupe)) expect.num(out.count("duplicate check name, so one ledger row covers 2: " - "demo\tthe same name"), 1, + "demo\tpart1\tthe same name"), 1, "a duplicated check name is reported by name") expect.num(out.count("a unique name"), 0, "and a unique one is not") expect.num(len(_rows(ledger)), 2, @@ -190,10 +190,10 @@ def test_the_gate_refuses_a_check_the_ledger_has_never_seen(tmp_path, expect): expect.num(out.count("checks_never_observed_red: 2 exceeds the budget of 1"), 1, "and the gate says which number was exceeded, by how much") - newer = _write(tmp_path, "n.log", GREEN + "RESULT\tdemo\tbrand new\tPASS\t\n") + newer = _write(tmp_path, "n.log", GREEN + "RESULT\tdemo\tpart1\tbrand new\tPASS\t\n") out, rc = _run("gate", "--ledger", ledger, "--budget", budget, newer) expect.num(rc, 1, "a check the ledger has never seen is refused") - expect.num(out.count("not in the ledger: demo\tbrand new"), 1, + expect.num(out.count("not in the ledger: demo\tpart1\tbrand new"), 1, "and it is named, so the author knows which one") @@ -207,8 +207,8 @@ def test_the_committed_ledger_and_budget_agree(expect): expect.text("yes" if budget.exists() else "no", "yes", "the budget is in the tree") rows = [l.split("\t") for l in ledger.read_text().splitlines() if l] - never = [r for r in rows if r[2] == "never"] - red = [r for r in rows if r[2] != "never"] + never = [r for r in rows if r[3] == "never"] + red = [r for r in rows if r[3] != "never"] print(f" ledger: inputs={len(rows)} | observed red={len(red)}, " f"never={len(never)} | sum={len(red) + len(never)}") expect.num(len(red) + len(never), len(rows), "the ledger partitions") diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 5ad3051a..18d4874e 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -35,15 +35,15 @@ _led_run() { python3 "$_led" "$@" 2>&1; } # ---- the census comes out of a run, not out of a list ----------------------- cat > "$_lw/green.log" <<'LOG' -RESULT demo first check PASS -RESULT demo second check PASS +RESULT demo part1 first check PASS +RESULT demo part1 second check PASS checks run: 2 LOG check "the census reads a run's records" \ "$(_led_run census "$_lw/green.log" | wc -l)" "2" check "and names the suite and the check, not just a count" \ - "$(_led_run census "$_lw/green.log" | head -1)" "demo first check PASS" + "$(_led_run census "$_lw/green.log" | head -1)" "demo part1 first check PASS" # ---- merging a green run adds the checks as DEBT, not as proven ------------- # @@ -56,27 +56,27 @@ _led_run merge --ledger "$_lw/ledger.tsv" "$_lw/green.log" >/dev/null check "merging a green run records both checks" \ "$(grep -c . "$_lw/ledger.tsv")" "2" check "and records neither as ever having been red" \ - "$(cut -f3 "$_lw/ledger.tsv" | sort -u | tr '\n' ' ')" "never " + "$(cut -f4 "$_lw/ledger.tsv" | sort -u | tr '\n' ' ')" "never " # ---- merging a run that DID go red records the observation ------------------ cat > "$_lw/red.log" <<'LOG' -RESULT demo first check FAIL -RESULT demo second check PASS +RESULT demo part1 first check FAIL +RESULT demo part1 second check PASS checks run: 2 LOG _led_run merge --ledger "$_lw/ledger.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null check "a check observed red gains the date it was seen" \ - "$(awk -F'\t' '$2=="first check"{print $3}' "$_lw/ledger.tsv")" "2026-09-10" + "$(awk -F'\t' '$3=="first check"{print $4}' "$_lw/ledger.tsv")" "2026-09-10" check "and a check that stayed green keeps its debt" \ - "$(awk -F'\t' '$2=="second check"{print $3}' "$_lw/ledger.tsv")" "never" + "$(awk -F'\t' '$3=="second check"{print $4}' "$_lw/ledger.tsv")" "never" # An observation is not undone by a later green run. The ledger records that the # check WAS seen red, which stays true. _led_run merge --ledger "$_lw/ledger.tsv" --date 2026-09-11 "$_lw/green.log" >/dev/null check "a later green run does not erase an observation" \ - "$(awk -F'\t' '$2=="first check"{print $3}' "$_lw/ledger.tsv")" "2026-09-10" + "$(awk -F'\t' '$3=="first check"{print $4}' "$_lw/ledger.tsv")" "2026-09-10" # ---- the gate: new checks must not be added to the debt silently ------------ # @@ -101,9 +101,9 @@ check "and the gate says which number was exceeded, by how much" \ # allowlist exists for: it is NEW, and it must not enter as silent debt. printf 'suites_not_covered 0\nchecks_never_observed_red 1\n' > "$_lw/budget.txt" cat > "$_lw/newcheck.log" <<'LOG' -RESULT demo first check PASS -RESULT demo second check PASS -RESULT demo a brand new check PASS +RESULT demo part1 first check PASS +RESULT demo part1 second check PASS +RESULT demo part1 a brand new check PASS checks run: 3 LOG check "a check the ledger has never seen is refused, not absorbed" \ @@ -111,7 +111,7 @@ check "a check the ledger has never seen is refused, not absorbed" \ && echo ok || echo refused)" "refused" check "and it is named, so the author knows which one" \ "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/newcheck.log" 2>&1 \ - | grep -c 'not in the ledger: demo a brand new check')" "1" + | grep -c 'not in the ledger: demo part1 a brand new check')" "1" # ---- the budget may only go DOWN -------------------------------------------- # @@ -124,8 +124,8 @@ check "the committed budget names both debts" \ # ---- inputs == sum(buckets), over the real ledger --------------------------- _l_total="$(grep -c . "$_ledger" || true)" -_l_red="$(awk -F'\t' '$3!="never"' "$_ledger" | grep -c . || true)" -_l_never="$(awk -F'\t' '$3=="never"' "$_ledger" | grep -c . || true)" +_l_red="$(awk -F'\t' '$4!="never"' "$_ledger" | grep -c . || true)" +_l_never="$(awk -F'\t' '$4=="never"' "$_ledger" | grep -c . || true)" echo " ledger: inputs=$_l_total | observed red=$_l_red, never=$_l_never | sum=$((_l_red + _l_never))" check "the ledger partitions into observed and never" \ "$((_l_red + _l_never))" "$_l_total" @@ -154,17 +154,17 @@ check "the committed budget matches the committed ledger's debt" \ : > "$_lw/ren.tsv" cat > "$_lw/before.log" <<'LOG' -RESULT demo the old name FAIL -RESULT demo a stable check PASS +RESULT demo part1 the old name FAIL +RESULT demo part1 a stable check PASS checks run: 2 LOG _led_run merge --ledger "$_lw/ren.tsv" --date 2026-09-01 "$_lw/before.log" >/dev/null check "premise: the check has history before the rename" \ - "$(awk -F'\t' '$2=="the old name"{print $3}' "$_lw/ren.tsv")" "2026-09-01" + "$(awk -F'\t' '$3=="the old name"{print $4}' "$_lw/ren.tsv")" "2026-09-01" cat > "$_lw/after.log" <<'LOG' -RESULT demo the new name PASS -RESULT demo a stable check PASS +RESULT demo part1 the new name PASS +RESULT demo part1 a stable check PASS checks run: 2 LOG check "a name that appeared while another disappeared is reported as a rename" \ @@ -177,9 +177,9 @@ check "and the stable check is not reported" \ # The detector must not fire when a check is simply ADDED. Without this it names # a rename on every new check, which is noise that gets it ignored. cat > "$_lw/added.log" <<'LOG' -RESULT demo the old name PASS -RESULT demo a stable check PASS -RESULT demo a genuinely new check PASS +RESULT demo part1 the old name PASS +RESULT demo part1 a stable check PASS +RESULT demo part1 a genuinely new check PASS checks run: 3 LOG check "a check merely added is not reported as a rename" \ @@ -188,7 +188,7 @@ check "a check merely added is not reported as a rename" \ # Nor when one is simply REMOVED. cat > "$_lw/removed.log" <<'LOG' -RESULT demo a stable check PASS +RESULT demo part1 a stable check PASS checks run: 1 LOG check "a check merely removed is not reported as a rename either" \ @@ -209,16 +209,16 @@ check "a check merely removed is not reported as a rename either" \ : > "$_lw/mut.tsv" _led_run merge --ledger "$_lw/mut.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null check "every ledger row carries four fields, the fourth being the mutation" \ - "$(awk -F'\t' 'NF!=4' "$_lw/mut.tsv" | grep -c . || true)" "0" + "$(awk -F'\t' 'NF!=5' "$_lw/mut.tsv" | grep -c . || true)" "0" check "and it is empty when nothing named a mutation" \ - "$(awk -F'\t' '$2=="first check"{print "[" $4 "]"}' "$_lw/mut.tsv")" "[]" + "$(awk -F'\t' '$3=="first check"{print "[" $5 "]"}' "$_lw/mut.tsv")" "[]" _led_run merge --ledger "$_lw/mut.tsv" --date 2026-09-10 \ --mutation 'PGCOLUMNAR_SAOP_ELEMENT_LIMIT 128 -> 0' "$_lw/red.log" >/dev/null check "a merge that names its mutation records it against the check that reddened" \ - "$(awk -F'\t' '$2=="first check"{print $4}' "$_lw/mut.tsv")" "PGCOLUMNAR_SAOP_ELEMENT_LIMIT 128 -> 0" + "$(awk -F'\t' '$3=="first check"{print $5}' "$_lw/mut.tsv")" "PGCOLUMNAR_SAOP_ELEMENT_LIMIT 128 -> 0" check "and not against one that stayed green" \ - "$(awk -F'\t' '$2=="second check"{print "[" $4 "]"}' "$_lw/mut.tsv")" "[]" + "$(awk -F'\t' '$3=="second check"{print "[" $5 "]"}' "$_lw/mut.tsv")" "[]" # ---- a duplicated check name shares one ledger row -------------------------- # @@ -233,15 +233,15 @@ check "and not against one that stayed green" \ # 609 records reduced to 605 rows. cat > "$_lw/dupe.log" <<'LOG' -RESULT demo the same name PASS -RESULT demo the same name FAIL -RESULT demo a unique name PASS +RESULT demo part1 the same name PASS +RESULT demo part1 the same name FAIL +RESULT demo part1 a unique name PASS checks run: 3 LOG : > "$_lw/dupe.tsv" check "a duplicated check name is reported by name" \ "$(_led_run merge --ledger "$_lw/dupe.tsv" --date 2026-09-10 "$_lw/dupe.log" \ - | grep -c 'duplicate check name, so one ledger row covers 2: demo the same name')" "1" + | grep -c 'duplicate check name, so one ledger row covers 2: demo part1 the same name')" "1" check "and a unique one is not" \ "$(_led_run merge --ledger "$_lw/dupe.tsv" --date 2026-09-10 "$_lw/dupe.log" \ | grep -c 'a unique name')" "0" From 671a946059ddaf88b3289d28ee891c20ab8fdb0d Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 07:56:25 -0600 Subject: [PATCH 06/27] test: two different premises were wearing one sentence, and the detector caught the rename (#918) OffgridwithJD looked at the collision that survived the (suite, part, name) key and found it is not a repeat of one question. At 340:268 and 340:894, `premise: the fixture fingerprints at all` is asked once of the SOURCE-PARTITION fixture and once of the LOCALE SWEEP: 268 _fs_before="$(pgc_source_fingerprint "$_fs_rp")" -> did the a.c/b.c partition fixture produce a fingerprint at all 894 _lc_vals="$_lc_vals $(LC_ALL=... pgc_source_fingerprint "$_lc/tree")" -> did every locale in the sweep produce one Two questions about two different code paths, sharing a sentence. That is the case where a shared ledger row is WRONG rather than merely imprecise: one failing marks the other's premise as observed red, and they are not about the same code. I FIXED IT RATHER THAN REPORTING IT, which departs from this PR's stance, and the distinction is worth stating. "Report rather than prevent" is right about the CONVENTION -- `premise: the pytest layer is where THIS PART thinks it is` is phrased to be copied, rewriting the corpus's shared premises is churn, and the tool exists to surface them. This is not that. It is one file, two lines, and the two premises are about different subjects. Naming them apart takes the corpus to ZERO collisions: 614 records, 614 distinct keys. AND IT IS THE RENAME DETECTOR'S FIRST TEST ON REAL DATA rather than fixtures. Run against the ledger seeded before the rename: possible rename: premise: the fixture fingerprints at all -> premise: every locale produced a fingerprint (in harness_selftest/340-the-binary-must-be-built-from, history: last red never) rename scan: appeared=2, vanished=1 appeared=2 against vanished=1 is honest: the two premises shared ONE row before, so one of the new names has no predecessor to pair with. Re-seeded: 614 rows, none ever observed red. Evidence: selftest exit 0, 614 checks, 0 failures, 0 collisions; gate rc=0 against the committed files; shellcheck rc=0. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/check_ledger.tsv | 3 ++- test/check_ledger_budget.txt | 2 +- test/pytest/TESTS.md | 23 +++++++++++++++---- .../340-the-binary-must-be-built-from.sh | 4 ++-- 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 1fe07ffc..e3542d36 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -340,9 +340,10 @@ harness_selftest 340-the-binary-must-be-built-from one tree, one fingerprint, wh harness_selftest 340-the-binary-must-be-built-from premise: and the stamp really was not written, so the arm is not vacuous never harness_selftest 340-the-binary-must-be-built-from premise: at least two locales are installed to compare never harness_selftest 340-the-binary-must-be-built-from premise: both fake configs report the same major, which is the whole point never +harness_selftest 340-the-binary-must-be-built-from premise: every locale produced a fingerprint never harness_selftest 340-the-binary-must-be-built-from premise: the argument parser reads the second argument at all never harness_selftest 340-the-binary-must-be-built-from premise: the build path ran to completion, so a stamp was due never -harness_selftest 340-the-binary-must-be-built-from premise: the fixture fingerprints at all never +harness_selftest 340-the-binary-must-be-built-from premise: the partition fixture fingerprints at all never harness_selftest 340-the-binary-must-be-built-from premise: the same function returns a fingerprint for a real tree never harness_selftest 340-the-binary-must-be-built-from premise: the spelling fixture fingerprints at all never harness_selftest 340-the-binary-must-be-built-from premise: the sweep finds the call sites it is meant to police never diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index f0f52dcf..7ce2882f 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -24,5 +24,5 @@ # invisible to everything above: the gate cannot refuse a new check in a suite # it has never seen. Counted separately so that "we ledger 605 checks" cannot # read as "we ledger the corpus". -checks_never_observed_red 613 +checks_never_observed_red 614 suites_not_covered 250 diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index b4100572..77ef1531 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1292,10 +1292,25 @@ against 582 distinct `(suite, part, name)`. The part is derived from `BASH_SOURC rather than from a convention, so the next part written the same way is keyed correctly without anyone remembering. -One duplicate survives that, and it is a genuine one: -`340-the-binary-must-be-built-from` asks `premise: the fixture fingerprints at all` -twice within the same part. That is the kind of thing the ledger can now name precisely -instead of losing among convention artifacts. +One duplicate survived that, and inspecting it showed it was worse than a repeat: at +`340:268` and `340:894`, `premise: the fixture fingerprints at all` was asked once of the +**source-partition fixture** and once of the **locale sweep** — two different questions +about two different code paths, wearing one sentence. A shared row there is not merely +imprecise: one failing would mark the other's premise as observed red. + +Both now name their subject, `premise: the partition fixture fingerprints at all` and +`premise: every locale produced a fingerprint`, and the corpus has **zero** collisions: +614 records, 614 distinct keys. + +That rename is also the detector's first test on real data rather than fixtures. Run +against the ledger seeded before it: + +``` +possible rename: premise: the fixture fingerprints at all + -> premise: every locale produced a fingerprint + (in harness_selftest/340-the-binary-must-be-built-from, history: last red never) +rename scan: appeared=2, vanished=1 +``` ### `test_the_gate_refuses_a_check_the_ledger_has_never_seen` diff --git a/test/selftest/340-the-binary-must-be-built-from.sh b/test/selftest/340-the-binary-must-be-built-from.sh index 584a01ca..cce17f81 100644 --- a/test/selftest/340-the-binary-must-be-built-from.sh +++ b/test/selftest/340-the-binary-must-be-built-from.sh @@ -265,7 +265,7 @@ printf 'static int x=1;\n' > "$_fs_rp/src/a.c" printf 'static int x=2;\n' > "$_fs_rp/src/b.c" _fs_before="$(pgc_source_fingerprint "$_fs_rp")" -check "premise: the fixture fingerprints at all" \ +check "premise: the partition fixture fingerprints at all" \ "$([ -n "$_fs_before" ] && echo yes || echo no)" "yes" # The same bytes, a different partition: a.c gains b.c's line and b.c is emptied. @@ -891,7 +891,7 @@ if [ "$_lc_count" -ge 2 ]; then for _lc_l in $_lc_have; do _lc_vals="$_lc_vals $(LC_ALL="$_lc_l" LANG="$_lc_l" pgc_source_fingerprint "$_lc/tree")" done - check "premise: the fixture fingerprints at all" \ + check "premise: every locale produced a fingerprint" \ "$([ -n "$(printf '%s' $_lc_vals)" ] && echo yes || echo empty)" "yes" check "one tree, one fingerprint, whatever the locale" \ "$(printf '%s\n' $_lc_vals | sort -u | grep -c .)" "1" From 0a6b2d32ca120fdfbc9aaf99c5266d778a235d42 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 09:06:39 -0600 Subject: [PATCH 07/27] test: a skipped timing check is an outcome, so count it and record it (#917) Blocking review by @linuxhikerpm at the exact head, and the finding is a hole in this change's own argument. check_timing and check_ratio_needs_quiet_machine, under PGC_SKIP_TIMING=1, printed a human SKIP line and returned. Driven before the fix: SKIP a timing check (PGC_SKIP_TIMING: wall-clock measurement) SKIP a ratio check (PGC_SKIP_TIMING: wall-clock ratio) -> PGC_CHECKS=0 PGC_PASSED=0 PGC_FAILED=0 PGC_UNRUN=0 Two outcomes a reader sees, invisible to the count and to the records both, in the change whose whole argument is that those are one operation. And nothing reached those branches: no arm in any harness mentioned either helper, so removing both emitters left everything green -- which is how they found it. SKIP IS A FOURTH OUTCOME, counted like the other three. `checks run:` now reports the checks a suite ENCOUNTERED rather than the ones it managed to evaluate, and pgc_summary reconciles four counters against that count instead of three -- the same shape one term wider, preserving the property its own comment argues for. IT IS DELIBERATELY NOT check_unrunnable. That third state exists for a check whose INPUT was absent and it exits the suite INCOMPLETE. CI sets PGC_SKIP_TIMING on every run, so routing these through it would turn every run red. A wall-clock check deliberately not asked on a shared runner is a different thing from one that could not be answered, and the ledger should be able to tell them apart. THE ALL-SKIPPED SUITE WOULD HAVE REPORTED PASSED. Before the fourth counter a skipped check left PGC_CHECKS at zero, so `if PGC_CHECKS = 0` caught that case by accident; counting it would have made such a suite report PASSED with nothing behind it. The condition now says what it always meant: PASSED + FAILED + UNRUN. REMOVAL PROOF, which is the thing their finding asked for. With both emitters deleted -- the mutation they applied, which used to leave everything green -- NINE named arms redden across both harnesses, and lib.sh restored byte-identical (md5 684272397ed5bb8a21a8b819a3066c59 before and after). THE DOCUMENTATION MISMATCH IS REAL AND NARROWER THAN REPORTED. The record has FIVE columns after the RESULT marker -- suite, part, name, verdict, reason -- and lib.sh, selftest 400 and TESTS.md all still said four, missing `part`. Fixed in all three. There is no `mutation` column in a record; that belongs to the LEDGER (#918), which keys on (suite, part, name) and records which mutation reddened a check. A record is one observation, not a history, and the docs now say so. The accounting line changing shape means both readers on main move with it -- the two regexes in run_all_versions.sh -- along with eleven fixtures across selftest 320, selftest 390 and test_suite_accounting.py. That coupling is exactly what #916's producer-versus-reader arm exists to catch, and it would have caught it. Evidence: selftest exit 0, 598 checks, 598 records, 0 failures; 23 pytest; shellcheck rc=0; docs_style PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/lib.sh | 50 +++++++-- test/pytest/TESTS.md | 47 +++++++- ...test_check_results_are_machine_readable.py | 87 ++++++++++++++ test/pytest/test_suite_accounting.py | 14 +-- test/run_all_versions.sh | 4 +- .../320-a-check-that-could-not-run.sh | 16 +-- .../390-a-registered-suite-must-account.sh | 14 +-- .../400-a-check-result-must-be-machine.sh | 106 +++++++++++++++++- 8 files changed, 294 insertions(+), 44 deletions(-) diff --git a/test/lib.sh b/test/lib.sh index b57965e1..3228f904 100755 --- a/test/lib.sh +++ b/test/lib.sh @@ -33,6 +33,12 @@ PGC_FAIL=0 # 3,762 forks a suite does not need. PGC_SUITE="$(basename "$0" .sh)" PGC_CHECKS=0 +# A FOURTH OUTCOME. A check deliberately not asked -- a wall-clock measurement on +# a shared runner under PGC_SKIP_TIMING -- is not a pass, not a failure, and not +# unrunnable. It is counted, so `checks run:` reports the checks a suite +# ENCOUNTERED rather than the ones it managed to evaluate, and pgc_summary +# reconciles four counters against that count instead of three. +PGC_SKIPPED=0 # The status pgc_summary uses for "ran no checks". # @@ -989,10 +995,18 @@ psql_file() { # stays byte-identical: suites, selftests and CI all grep `^PASS` and `^FAIL`, # and 3,762 call sites is far past what a careful refactor can be trusted on. # -# The record is tab separated -- suite, name, verdict, reason -- so a check name -# containing spaces survives, and the reason carries the REASON_CODE #915 -# introduced. That is what makes this more than a reformat: an unrunnable check -# is distinguishable from a passing one without parsing prose. +# The record is tab separated, five columns after the RESULT marker: +# +# RESULT suite part name verdict reason +# +# so a check name containing spaces survives. The reason carries the REASON_CODE +# #915 introduced, which is what makes this more than a reformat: an unrunnable +# check is distinguishable from a passing one without parsing prose. The verdict +# is one of PASS, FAIL, UNRUN or SKIP. +# +# There is no mutation column here. That one belongs to the LEDGER (#918), which +# keys on (suite, part, name) and records which mutation reddened a check; a +# record is one observation, not a history. pgc_record() { # pgc_record VERDICT NAME DISPLAY [REASON] local _v="$1" _name="$2" _display="$3" _reason="${4:-}" PGC_CHECKS=$((PGC_CHECKS + 1)) @@ -1000,12 +1014,13 @@ pgc_record() { # pgc_record VERDICT NAME DISPLAY [REASON] PASS) PGC_PASSED=$((PGC_PASSED + 1)) ;; FAIL) PGC_FAILED=$((PGC_FAILED + 1)); PGC_FAIL=1 ;; UNRUN) PGC_UNRUN=$((PGC_UNRUN + 1)) ;; + SKIP) PGC_SKIPPED=$((PGC_SKIPPED + 1)) ;; *) # An unknown verdict is a failure of the harness, not a check to # drop. Dropping it would leave PGC_CHECKS bumped with no outcome # recorded, which is the reconciliation failure pgc_summary refuses. PGC_FAILED=$((PGC_FAILED + 1)); PGC_FAIL=1 - _display="FAIL $_name: pgc_record was given the verdict [$_v], which is not PASS, FAIL or UNRUN" + _display="FAIL $_name: pgc_record was given the verdict [$_v], which is not PASS, FAIL, UNRUN or SKIP" _v=FAIL ;; esac @@ -1242,7 +1257,12 @@ check_timing() { local name="$1" got="$2" want="$3" if [ "${PGC_SKIP_TIMING:-0}" = 1 ]; then - echo "SKIP $name (PGC_SKIP_TIMING: wall-clock measurement)" + # Counted and recorded, like every other outcome. It printed a line a + # reader sees; leaving PGC_CHECKS at zero made that outcome invisible to + # the count and to the records both (#917, found by @linuxhikerpm). + pgc_record SKIP "$name" \ + "SKIP $name (PGC_SKIP_TIMING: wall-clock measurement)" \ + "PGC_SKIP_TIMING" return 0 fi check "$name" "$got" "$want" @@ -1284,7 +1304,9 @@ check_timing() { # same-run ratio) and nothing said it here, where it is decided (#787). check_ratio_needs_quiet_machine() { # if [ "${PGC_SKIP_TIMING:-0}" = 1 ]; then - echo "SKIP $1 (PGC_SKIP_TIMING: wall-clock ratio)" + pgc_record SKIP "$1" \ + "SKIP $1 (PGC_SKIP_TIMING: wall-clock ratio)" \ + "PGC_SKIP_TIMING" return 0 fi check_ratio "$@" @@ -1561,7 +1583,7 @@ pgc_skip() { # pgc_skip # did one level up for how many VERSIONS actually ran. pgc_summary() { local _failed=$PGC_FAILED - local _sum=$((PGC_PASSED + PGC_FAILED + PGC_UNRUN)) + local _sum=$((PGC_PASSED + PGC_FAILED + PGC_UNRUN + PGC_SKIPPED)) echo echo "checks run: $PGC_CHECKS" echo "checks unrunnable: $PGC_UNRUN" @@ -1569,7 +1591,7 @@ pgc_summary() { # go missing, and this harness has 3,762 check sites -- far past what anyone # notices by reading. If this line does not add up the harness is lying about # its own arithmetic, so it is a failure rather than a note. - echo "accounting: $PGC_PASSED passed + $_failed failed + $PGC_UNRUN unrunnable = $PGC_CHECKS" + echo "accounting: $PGC_PASSED passed + $_failed failed + $PGC_UNRUN unrunnable + $PGC_SKIPPED skipped = $PGC_CHECKS" # A MEASUREMENT, not an identity. The failed count is its own counter rather # than CHECKS - PASSED - UNRUN, because a derived third term makes # P + (N-P-U) + U = N true for ANY values: a helper that counts a check and @@ -1579,9 +1601,9 @@ pgc_summary() { # line could not see it. Three counters maintained independently, reconciled # against a fourth, is the only version of it that can fail. if [ "$_sum" != "$PGC_CHECKS" ]; then - echo "FAIL the summary does not reconcile: $PGC_PASSED passed + $PGC_FAILED failed + $PGC_UNRUN unrunnable = $_sum, but $PGC_CHECKS checks ran" + echo "FAIL the summary does not reconcile: $PGC_PASSED passed + $PGC_FAILED failed + $PGC_UNRUN unrunnable + $PGC_SKIPPED skipped = $_sum, but $PGC_CHECKS checks ran" echo " A check was counted whose outcome nothing recorded. Find the helper" - echo " that bumps PGC_CHECKS without touching PGC_PASSED, PGC_FAILED or PGC_UNRUN." + echo " that bumps PGC_CHECKS without touching PGC_PASSED, PGC_FAILED, PGC_UNRUN or PGC_SKIPPED." PGC_FAIL=1 fi if [ "$PGC_FAIL" != "0" ]; then @@ -1617,7 +1639,11 @@ pgc_summary() { fi exit 1 fi - if [ "$PGC_CHECKS" = "0" ]; then + # EVALUATED nothing, not ENCOUNTERED nothing. Before the fourth counter a + # skipped check left PGC_CHECKS at zero, so this branch caught the all-skipped + # suite by accident; now it is counted, and the suite would report PASSED with + # nothing behind it. The condition is what it always meant. + if [ "$((PGC_PASSED + PGC_FAILED + PGC_UNRUN))" = "0" ]; then echo "$(basename "$0"): SKIPPED (ran no checks)" exit $PGC_EXIT_SKIPPED fi diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index a6648765..e683c53c 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1168,10 +1168,20 @@ cannot report an outcome without being counted, and cannot be counted without reporting one, because no code path does either alone. `checks run: N` and the N record lines are the same increment seen twice. -The record is tab separated -- suite, name, verdict, reason -- so a check name with -spaces survives, and the reason carries the `REASON_CODE` from #915. That is what -makes it more than a reformat: an unrunnable check is distinguishable from a passing -one without parsing prose. +The record is tab separated, five columns after the `RESULT` marker: + +``` +RESULT suite part name verdict reason +``` + +so a check name with spaces survives. The verdict is one of `PASS`, `FAIL`, `UNRUN` or +`SKIP`, and the reason carries the `REASON_CODE` from #915 — which is what makes this +more than a reformat: an unrunnable check is distinguishable from a passing one without +parsing prose. + +There is **no mutation column here**. That belongs to the ledger (#918), which keys on +`(suite, part, name)` and records which mutation reddened a check. A record is one +observation, not a history. ### `test_lib_sh_counts_a_check_in_exactly_one_place` @@ -1201,6 +1211,35 @@ pinned rather than the refactor trusted. One operation, so it cannot fail by drifting. It can fail if a helper is added that prints an outcome without recording it, which is the `expect_fail` shape. +### `test_a_skipped_timing_check_is_counted_and_recorded` + +`check_timing` and `check_ratio_needs_quiet_machine` under `PGC_SKIP_TIMING=1` printed a +human `SKIP` line and returned — **no count, no record**. Two outcomes a reader sees, +invisible to both, inside the part whose whole argument is that counting and recording +are one operation. Nothing reached those branches either: removing both emitters left +every other arm green. + +`SKIP` is now a fourth outcome, counted like the other three, so `checks run:` reports +the checks a suite **encountered** rather than the ones it managed to evaluate. + +It is deliberately **not** `check_unrunnable`. That third state exits the suite +`INCOMPLETE`, and CI sets `PGC_SKIP_TIMING` on every run — so every run would go red. A +wall-clock check deliberately not asked on a shared runner is a different thing from one +that could not be answered. + +### `test_the_accounting_line_reconciles_four_outcomes` + +Four counters against the count, which is the same shape as three against it. The +skipped term is printed even when zero: a term that disappears when empty is one a +reader cannot tell from a term that was never there. + +### `test_a_suite_that_evaluated_nothing_did_not_pass` + +Before the fourth counter, a skipped check left `PGC_CHECKS` at zero, so an all-skipped +suite hit the "ran no checks" branch **by accident**. Counting it would have made that +suite report `PASSED` with nothing behind it, so the condition now says what it always +meant: `PASSED + FAILED + UNRUN`, not `CHECKS`. + ### `test_the_runner_reconciles_records_against_the_stated_count` A suite's log states `checks run: N` and carries N records. Those are two artifacts of diff --git a/test/pytest/test_check_results_are_machine_readable.py b/test/pytest/test_check_results_are_machine_readable.py index bded933d..c8e5f08f 100644 --- a/test/pytest/test_check_results_are_machine_readable.py +++ b/test/pytest/test_check_results_are_machine_readable.py @@ -193,3 +193,90 @@ def run(text): # from a miscount and must not read as a clean reconciliation. expect.num(run("RESULT\ts\ta\tPASS\t\n")[1], 1, "a log that never stated a count is not silently accepted") + + +# ---- the timing helpers reported an outcome that nothing counted ------------ + + +def _timing(skip, call): + """Run one timing helper with PGC_SKIP_TIMING set or clear.""" + body = (f'PGC_SKIP_TIMING={skip}\n' + 'PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0; PGC_SKIPPED=0\n' + f'{call}\n' + 'echo "COUNTS $PGC_CHECKS/$PGC_PASSED/$PGC_SKIPPED"') + out = _sh(body) + recs = [l for l in out.splitlines() if l.startswith("RESULT\t")] + human = [l for l in out.splitlines() + if not l.startswith("RESULT\t") and not l.startswith("COUNTS ")] + counts = next((l.split()[1] for l in out.splitlines() if l.startswith("COUNTS ")), "") + return recs, human, counts + + +def test_a_skipped_timing_check_is_counted_and_recorded(expect): + """`check_timing` and `check_ratio_needs_quiet_machine` under PGC_SKIP_TIMING=1 + printed a human SKIP line and returned -- no count, no record. + + Two outcomes a reader sees, invisible to both the count and the records, in the + part whose whole argument is that those are one operation. Nothing reached those + branches either: removing both emitters left every other arm green. Found by + @linuxhikerpm. + + SKIP is a fourth outcome, counted like the other three, so `checks run:` reports + the checks a suite ENCOUNTERED rather than the ones it managed to evaluate. It is + deliberately not `check_unrunnable`: that state exits the suite INCOMPLETE, and CI + sets PGC_SKIP_TIMING on every run, so every run would go red. A wall-clock check + not asked on a shared runner is a different thing from one that could not be + answered. + """ + for call, tail in ((' check_timing "a timing check" 1 1', + "wall-clock measurement"), + (' check_ratio_needs_quiet_machine "a ratio check" 1 1 2', + "wall-clock ratio")): + name = call.split('"')[1] + recs, human, counts = _timing(1, call) + expect.num(len(recs), 1, f"{name}: skipped, emits exactly one record") + expect.text(recs[0].split("\t")[4], "SKIP", f"{name}: and its verdict is SKIP") + expect.text(counts, "1/0/1", f"{name}: and it is counted as a skip") + expect.text("\n".join(human), f"SKIP {name} (PGC_SKIP_TIMING: {tail})", + f"{name}: and its human line is unchanged") + + recs, _, counts = _timing(0, call) + expect.num(len(recs), 1, f"{name}: enabled, emits exactly one record") + expect.text(recs[0].split("\t")[4], "PASS", f"{name}: and passes") + expect.text(counts, "1/1/0", f"{name}: counted as a pass, not a skip") + + +def test_the_accounting_line_reconciles_four_outcomes(expect): + """Four counters against the count, which is the same shape as three against it. + + The skipped term is printed even when zero: a term that disappears when empty is + a term a reader cannot tell from a term that was never there. + """ + def acct(skip): + out = _sh(f'PGC_SKIP_TIMING={skip}\n' + ' check "an ordinary check" x x\n' + ' check_timing "a timing check" 1 1\n' + ' pgc_summary') + return next((l for l in out.splitlines() if l.startswith("accounting: ")), "") + + expect.text(acct(1), "accounting: 1 passed + 0 failed + 0 unrunnable + 1 skipped = 2", + "a skipped check appears in the accounting identity") + expect.text(acct(0), "accounting: 2 passed + 0 failed + 0 unrunnable + 0 skipped = 2", + "and the term is printed when zero, not omitted") + + +def test_a_suite_that_evaluated_nothing_did_not_pass(expect): + """Before the fourth counter a skipped check left PGC_CHECKS at zero, so the + all-skipped suite hit the "ran no checks" branch by accident. Counting it would + have made the suite report PASSED with nothing behind it, so the condition now + says what it always meant: PASSED + FAILED + UNRUN, not CHECKS. + """ + out = _sh('PGC_SKIP_TIMING=1\n' + ' check_timing "a timing check" 1 1\n' + ' pgc_summary') + verdicts = [l for l in out.splitlines() + if l.endswith(("PASSED", "FAILED", "INCOMPLETE")) + or l.endswith("SKIPPED (ran no checks)")] + expect.text("\n".join(v.split(": ", 1)[1] for v in verdicts), + "SKIPPED (ran no checks)", + "a suite whose every check was skipped did not pass") diff --git a/test/pytest/test_suite_accounting.py b/test/pytest/test_suite_accounting.py index 764e5d63..8b75119c 100644 --- a/test/pytest/test_suite_accounting.py +++ b/test/pytest/test_suite_accounting.py @@ -111,10 +111,10 @@ def test_the_accounting_line_is_read_on_every_exit_path(tmp_path, expect): for PASSED: a suite that failed still reached its summary and still accounted. """ shapes = { - "pass": "accounting: 3 passed + 0 failed + 0 unrunnable = 3\nx.sh: PASSED\n", - "fail": "accounting: 1 passed + 2 failed + 0 unrunnable = 3\nx.sh: FAILED\n", - "skip": "accounting: 0 passed + 0 failed + 0 unrunnable = 0\nx.sh: SKIPPED (ran no checks)\n", - "inc": "accounting: 2 passed + 0 failed + 1 unrunnable = 3\nx.sh: INCOMPLETE\n", + "pass": "accounting: 3 passed + 0 failed + 0 unrunnable + 0 skipped = 3\nx.sh: PASSED\n", + "fail": "accounting: 1 passed + 2 failed + 0 unrunnable + 0 skipped = 3\nx.sh: FAILED\n", + "skip": "accounting: 0 passed + 0 failed + 0 unrunnable + 0 skipped = 0\nx.sh: SKIPPED (ran no checks)\n", + "inc": "accounting: 2 passed + 0 failed + 1 unrunnable + 0 skipped = 3\nx.sh: INCOMPLETE\n", } for shape, text in shapes.items(): log = _write(tmp_path, f"{shape}.log", text) @@ -135,9 +135,9 @@ def test_the_accounting_line_is_read_on_every_exit_path(tmp_path, expect): # arm here green. Reported by OffgridwithJD. The distinguishing input is a # well-formed accounting line that does not start its line. indented = _write(tmp_path, "indented.log", - " accounting: 3 passed + 0 failed + 0 unrunnable = 3\nx.sh: PASSED\n") + " accounting: 3 passed + 0 failed + 0 unrunnable + 0 skipped = 3\nx.sh: PASSED\n") expect.num(pathlib.Path(indented).read_text() - .count("accounting: 3 passed + 0 failed + 0 unrunnable = 3"), 1, + .count("accounting: 3 passed + 0 failed + 0 unrunnable + 0 skipped = 3"), 1, "premise: the fixture carries a well-formed line, just indented") expect.text(_call("pgc_log_shows_accounting", indented)[0].strip(), "no", "an accounting line that does not start its line is refused") @@ -368,7 +368,7 @@ def test_the_accounted_reader_takes_either_runtime_mechanism(tmp_path, expect): that adopts either leaves the debt bucket on its own. """ lib = _write(tmp_path, "lib.log", - "accounting: 1 passed + 0 failed + 0 unrunnable = 1\nx.sh: PASSED\n") + "accounting: 1 passed + 0 failed + 0 unrunnable + 0 skipped = 1\nx.sh: PASSED\n") own = _write(tmp_path, "own.log", "checks run: 9\ndocs_style.sh: PASSED\n") neither = _write(tmp_path, "none.log", "some output\nPASSED\n") expect.text(_call("pgc_log_shows_any_accounting", lib)[0].strip(), "yes", diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index 6ef2e84f..dd3c16e5 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -933,7 +933,7 @@ pgc_log_shows_accounting() { # pgc_log_shows_accounting LOGFILE -> yes|no # of thing that reads as an oversight later. local _log="$1" [ -f "$_log" ] || { echo no; return 0; } - if grep -qE '^accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable = [0-9]+$' "$_log"; then + if grep -qE '^accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable \+ [0-9]+ skipped = [0-9]+$' "$_log"; then echo yes else echo no @@ -981,7 +981,7 @@ pgc_log_shows_any_accounting() { # pgc_log_shows_any_accounting LOGFILE -> yes|n # the property that keeps the debt file from becoming a permission slip. local _log="$1" [ -f "$_log" ] || { echo no; return 0; } - if [ "$(grep -cE '^accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable = [0-9]+$' "$_log" || true)" != 0 ] \ + if [ "$(grep -cE '^accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable \+ [0-9]+ skipped = [0-9]+$' "$_log" || true)" != 0 ] \ || [ "$(grep -cE '^checks run: [0-9]+$' "$_log" || true)" != 0 ]; then echo yes else diff --git a/test/selftest/320-a-check-that-could-not-run.sh b/test/selftest/320-a-check-that-could-not-run.sh index 3a6a325c..ee6f23ce 100644 --- a/test/selftest/320-a-check-that-could-not-run.sh +++ b/test/selftest/320-a-check-that-could-not-run.sh @@ -87,8 +87,8 @@ check "one unrunnable check makes the suite INCOMPLETE, not passed" \ # the feature from its absence. Assert the accounting line instead, which only a # suite that recorded BOTH states can print. check "a failure outranks an unrunnable check, and both are still counted" \ - "$(_cur_out failunrun | grep -oE 'accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable')" \ - "accounting: 0 passed + 1 failed + 1 unrunnable" + "$(_cur_out failunrun | grep -oE 'accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable \+ [0-9]+ skipped')" \ + "accounting: 0 passed + 1 failed + 1 unrunnable + 0 skipped" # The distinction 66 cannot carry: this suite RAN a check. Reporting it as # "ran no checks" would merge "inert suite" with "could not evaluate one thing". @@ -125,12 +125,12 @@ check "an unrunnable reason outside the enum fails rather than being accepted" \ # Every state is in a total, or it is a state that can go missing. 3,762 check # sites is well past what anyone notices by reading. check "the summary reconciles the three states against the total" \ - "$(_cur_out passunrun | grep -oE 'accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable = [0-9]+')" \ - "accounting: 1 passed + 0 failed + 1 unrunnable = 2" + "$(_cur_out passunrun | grep -oE 'accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable \+ [0-9]+ skipped = [0-9]+')" \ + "accounting: 1 passed + 0 failed + 1 unrunnable + 0 skipped = 2" check "and a suite with no unrunnable checks reconciles too" \ - "$(_cur_out onlypass | grep -oE 'accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable = [0-9]+')" \ - "accounting: 1 passed + 0 failed + 0 unrunnable = 1" + "$(_cur_out onlypass | grep -oE 'accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable \+ [0-9]+ skipped = [0-9]+')" \ + "accounting: 1 passed + 0 failed + 0 unrunnable + 0 skipped = 1" # ---- the accounting must be a MEASUREMENT, not an identity ------------------ # @@ -155,8 +155,8 @@ _cur_make ratio 'check "a" ok ok check_ratio "a ratio well inside its bound" 10 100 1.0' check "a passing ratio check is counted as a pass, not a failure" \ - "$(_cur_out ratio | grep -oE 'accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable = [0-9]+')" \ - "accounting: 2 passed + 0 failed + 0 unrunnable = 2" + "$(_cur_out ratio | grep -oE 'accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable \+ [0-9]+ skipped = [0-9]+')" \ + "accounting: 2 passed + 0 failed + 0 unrunnable + 0 skipped = 2" check "and the suite that holds it still passes" \ "$(_cur_run ratio)" "0 PASSED" diff --git a/test/selftest/390-a-registered-suite-must-account.sh b/test/selftest/390-a-registered-suite-must-account.sh index 42321d4a..90f35a9b 100644 --- a/test/selftest/390-a-registered-suite-must-account.sh +++ b/test/selftest/390-a-registered-suite-must-account.sh @@ -164,18 +164,18 @@ check "and the reader answers no on it, which is the wrong answer the arm catche # on a pass, a failure, a skip and an incomplete alike. That is what makes it the # runtime twin of the declaration rather than a synonym for PASSED. -printf 'checks run: 3\nchecks unrunnable: 0\naccounting: 3 passed + 0 failed + 0 unrunnable = 3\nx.sh: PASSED\n' > "$_acc/pass.log" +printf 'checks run: 3\nchecks unrunnable: 0\naccounting: 3 passed + 0 failed + 0 unrunnable + 0 skipped = 3\nx.sh: PASSED\n' > "$_acc/pass.log" check "a passing log shows accounting" "$(pgc_log_shows_accounting "$_acc/pass.log")" "yes" -printf 'accounting: 1 passed + 2 failed + 0 unrunnable = 3\nx.sh: FAILED\n' > "$_acc/fail.log" +printf 'accounting: 1 passed + 2 failed + 0 unrunnable + 0 skipped = 3\nx.sh: FAILED\n' > "$_acc/fail.log" check "and so does a failing one, which is the point" \ "$(pgc_log_shows_accounting "$_acc/fail.log")" "yes" -printf 'accounting: 0 passed + 0 failed + 0 unrunnable = 0\nx.sh: SKIPPED (ran no checks)\n' > "$_acc/skip.log" +printf 'accounting: 0 passed + 0 failed + 0 unrunnable + 0 skipped = 0\nx.sh: SKIPPED (ran no checks)\n' > "$_acc/skip.log" check "and a skip, which reached the summary and counted zero" \ "$(pgc_log_shows_accounting "$_acc/skip.log")" "yes" -printf 'accounting: 2 passed + 0 failed + 1 unrunnable = 3\nx.sh: INCOMPLETE\n' > "$_acc/inc.log" +printf 'accounting: 2 passed + 0 failed + 1 unrunnable + 0 skipped = 3\nx.sh: INCOMPLETE\n' > "$_acc/inc.log" check "and an incomplete" "$(pgc_log_shows_accounting "$_acc/inc.log")" "yes" printf 'x.sh: PASSED\n' > "$_acc/bare.log" @@ -193,10 +193,10 @@ check "and prose containing the word does not count as the line" \ # suite run produces. Inert on real data today (0 non-line-start occurrences # across 246 PG17 logs and 244 PG18), so this closes a coverage gap rather than a # live defect. -printf ' accounting: 3 passed + 0 failed + 0 unrunnable = 3\nx.sh: PASSED\n' \ +printf ' accounting: 3 passed + 0 failed + 0 unrunnable + 0 skipped = 3\nx.sh: PASSED\n' \ > "$_acc/indented_acc.log" check "premise: the fixture carries a well-formed accounting line, just indented" \ - "$(grep -c 'accounting: 3 passed + 0 failed + 0 unrunnable = 3' "$_acc/indented_acc.log")" "1" + "$(grep -c 'accounting: 3 passed + 0 failed + 0 unrunnable + 0 skipped = 3' "$_acc/indented_acc.log")" "1" check "an accounting line that does not start its line is refused" \ "$(pgc_log_shows_accounting "$_acc/indented_acc.log")" "no" @@ -532,7 +532,7 @@ check "premise: the population reconciliation is callable" \ # ---- the accounted reader takes EITHER mechanism ---------------------------- -printf 'accounting: 1 passed + 0 failed + 0 unrunnable = 1\nx.sh: PASSED\n' > "$_acc/lib.log" +printf 'accounting: 1 passed + 0 failed + 0 unrunnable + 0 skipped = 1\nx.sh: PASSED\n' > "$_acc/lib.log" check "a log carrying lib.sh's accounting line is accounted" \ "$(pgc_log_shows_any_accounting "$_acc/lib.log")" "yes" diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index ac351999..55d44ff4 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -33,10 +33,14 @@ check "and that place is pgc_record" \ # ---- the record line itself ------------------------------------------------- # -# Tab separated, so a name containing spaces survives. Fields: suite, name, -# verdict, reason. The reason carries phase 1's REASON_CODE, which is what makes -# this more than a reformat: an unrunnable check is distinguishable from a -# passing one without parsing prose. +# Tab separated, so a name containing spaces survives. Five columns after the +# RESULT marker: suite, part, name, verdict, reason. The reason carries phase 1's +# REASON_CODE, which is what makes this more than a reformat: an unrunnable check +# is distinguishable from a passing one without parsing prose. The verdict is one +# of PASS, FAIL, UNRUN or SKIP. +# +# No mutation column: that is the LEDGER's (#918). A record is one observation, +# not a history. _rec() { # _rec HELPER ARGS... -> the RESULT lines that helper emitted ( PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0 @@ -184,3 +188,97 @@ check "a log that never stated a count is not silently accepted" \ check "the runner calls the record reconciliation, not merely defines it" \ "$(grep -c '[^_[:alnum:]]pgc_reconcile_records "' "$_rv")" "1" + +# ---- the timing helpers reported an outcome that nothing counted ------------- +# +# check_timing and check_ratio_needs_quiet_machine, under PGC_SKIP_TIMING=1, +# printed a human SKIP line and returned. Driven before the fix: +# +# SKIP a timing check (PGC_SKIP_TIMING: wall-clock measurement) +# SKIP a ratio check (PGC_SKIP_TIMING: wall-clock ratio) +# -> PGC_CHECKS=0 PGC_PASSED=0 PGC_FAILED=0 PGC_UNRUN=0 +# +# Two outcomes a reader sees, nothing counted, no record. That is the hole this +# part's whole argument cannot have, and NOTHING here reached those branches: +# removing both emitters left every arm above green. Found by @linuxhikerpm. +# +# SKIP IS A FOURTH OUTCOME, counted like the other three. `checks run: N` now +# reports the checks a suite ENCOUNTERED rather than the ones it managed to +# evaluate, and pgc_summary reconciles four counters against it instead of three +# -- the same shape, one term wider. +# +# It is NOT check_unrunnable. That third state exists for a check whose INPUT was +# absent and it exits the suite INCOMPLETE, which would turn every CI run red the +# moment PGC_SKIP_TIMING is set -- and CI sets it on every run. A wall-clock check +# on a shared runner is deliberately not asked, which is a different thing from a +# check that could not be answered. + +_tm() { # _tm SKIPFLAG HELPER ARGS... -> records emitted + ( PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0; PGC_SKIPPED=0 + PGC_SKIP_TIMING="$1"; shift; "$@" 2>/dev/null | grep '^RESULT' ) +} +_tmh() { # _tmh SKIPFLAG HELPER ARGS... -> the human lines + ( PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0; PGC_SKIPPED=0 + PGC_SKIP_TIMING="$1"; shift; "$@" 2>/dev/null | grep -v '^RESULT' ) +} +_tmc() { # _tmc SKIPFLAG HELPER ARGS... -> "CHECKS/PASSED/SKIPPED" + ( PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0; PGC_SKIPPED=0 + PGC_SKIP_TIMING="$1"; shift; "$@" >/dev/null 2>&1 + echo "$PGC_CHECKS/$PGC_PASSED/$PGC_SKIPPED" ) +} + +check "a skipped timing check emits exactly one record" \ + "$(_tm 1 check_timing "a timing check" 1 1 | wc -l)" "1" +check "and its verdict is SKIP" \ + "$(_tm 1 check_timing "a timing check" 1 1 | cut -f5)" "SKIP" +check "and it is counted, so checks run: reports it" \ + "$(_tmc 1 check_timing "a timing check" 1 1)" "1/0/1" +check "and its human line is unchanged" \ + "$(_tmh 1 check_timing "a timing check" 1 1)" \ + "SKIP a timing check (PGC_SKIP_TIMING: wall-clock measurement)" + +check "the same timing check, ENABLED, emits one record and passes" \ + "$(_tm 0 check_timing "a timing check" 1 1 | cut -f5)" "PASS" +check "and is counted as a pass, not a skip" \ + "$(_tmc 0 check_timing "a timing check" 1 1)" "1/1/0" + +check "a skipped ratio check emits exactly one record" \ + "$(_tm 1 check_ratio_needs_quiet_machine "a ratio check" 1 1 2 | wc -l)" "1" +check "and its verdict is SKIP" \ + "$(_tm 1 check_ratio_needs_quiet_machine "a ratio check" 1 1 2 | cut -f5)" "SKIP" +check "and it is counted" \ + "$(_tmc 1 check_ratio_needs_quiet_machine "a ratio check" 1 1 2)" "1/0/1" +check "and its human line is unchanged" \ + "$(_tmh 1 check_ratio_needs_quiet_machine "a ratio check" 1 1 2)" \ + "SKIP a ratio check (PGC_SKIP_TIMING: wall-clock ratio)" + +check "the same ratio check, ENABLED, emits one record and passes" \ + "$(_tm 0 check_ratio_needs_quiet_machine "a ratio check" 1 1 2 | cut -f5)" "PASS" +check "and is counted as a pass, not a skip" \ + "$(_tmc 0 check_ratio_needs_quiet_machine "a ratio check" 1 1 2)" "1/1/0" + +# ---- and the accounting line carries the fourth term ------------------------ + +_acct_line() { # _acct_line SKIPFLAG -> pgc_summary's accounting line + ( PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0; PGC_SKIPPED=0 + PGC_SKIP_TIMING="$1" + check "an ordinary check" x x + check_timing "a timing check" 1 1 + pgc_summary ) 2>/dev/null | sed -n 's/^\(accounting: .*\)$/\1/p' +} +check "the accounting line reconciles four outcomes against the count" \ + "$(_acct_line 1)" "accounting: 1 passed + 0 failed + 0 unrunnable + 1 skipped = 2" +check "and with timing enabled the skipped term is zero, not absent" \ + "$(_acct_line 0)" "accounting: 2 passed + 0 failed + 0 unrunnable + 0 skipped = 2" + +# A suite whose every check was skipped has evaluated nothing, so it must not +# report PASSED. Before the fourth counter it could not reach this state at all, +# because a skipped check left PGC_CHECKS at zero. +_allskip() { + ( PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0; PGC_SKIPPED=0 + PGC_SKIP_TIMING=1 + check_timing "a timing check" 1 1 + pgc_summary ) 2>/dev/null | grep -oE ': (PASSED|FAILED|SKIPPED \(ran no checks\)|INCOMPLETE)$' +} +check "a suite that skipped every check did not pass" \ + "$(_allskip)" ": SKIPPED (ran no checks)" From 2c03d3e2429adc3177005d5348b58edcdb901f90 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 09:15:38 -0600 Subject: [PATCH 08/27] test: the records/checks mismatch must name its cause, and the shape is swept (#917) Both from OffgridwithJD, re-swept against this head rather than carried over. THE RECONCILER CAUGHT THE DEFECT AND DESCRIBED THE BOOKKEEPING. A check inside a piped loop runs in a SUBSHELL, so its counter bump dies there while its outcome and its record both reach the parent's stdout. Driven: check "direct one"; check "direct two" printf 'three\nfour\n' | while IFS= read -r n; do check "piped $n" a a; done -> four PASS lines, four RESULT lines, PGC_CHECKS=2 pgc_reconcile_records reported `records=3 but the log states checks run: 1`, which is true and useless: a reader who has not met this has no route from two numbers to a pipeline. It now names the cause, and the two directions get different causes -- more records than counted is a lost subshell, fewer is a counter bumped without going through pgc_record. AND THE SHAPE IS SWEPT, the way selftest 080 sweeps its cousin. Latent today: four piped loops in the tree, none with a check inside, so the sweep reports zero and four fixtures prove it can fire -- a rule whose only evidence is that the corpus happens to be clean is not a rule. THE SWEEP WAS WRONG TWICE BEFORE IT WAS RIGHT, both found by running it over the real corpus instead of reading it. Requiring the closing `done` to be alone on its line left the scanner inside a loop for the rest of any file whose loop ended `done)"`, flagging every later check: 27 hits against a true zero. Then a loop written entirely on ONE line inside a command substitution opened a block that never closed -- twelve of those false hits were in the sweep's own file. So it opens only on a line that opens a loop and does not close it, and closes on a `done` token wherever it sits. Four fixtures now pin that: a check inside a piped loop is found, one in a process-substitution loop is not, one after a one-line loop is not, and a piped loop with no check in it is not. The diagnostic message is ASSEMBLED rather than written out, because spelling the shape made the sweep flag the line that warns about it -- selftest 080's control problem, third time tonight. ONE NOTE, NOT A DEFECT. pgc_log_shows_any_accounting's `checks run:` alternative answers yes to an accounting line of any shape, so it MASKS a change to that line: a producer moving without its readers would leave the population reconciliation green while the accounting one reddened. Two checks disagreeing about one log is a worse signal than either failing. It cannot happen inside one tree, so it is recorded where someone debugging a half-merge would look. Evidence: selftest exit 0, 608 checks, 0 failures; 23 pytest; shellcheck rc=0; docs_style PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/run_all_versions.sh | 25 ++++++ .../400-a-check-result-must-be-machine.sh | 88 +++++++++++++++++++ 2 files changed, 113 insertions(+) diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index dd3c16e5..0aa2853a 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -963,6 +963,20 @@ pgc_reconcile_records() { # pgc_reconcile_records LOGFILE -> 0 ok, 1 mismatch fi if [ "$_records" != "$_stated" ]; then echo " records=$_records but the log states checks run: $_stated" + # NAME THE CAUSE, not just the arithmetic. The two directions have + # different causes and a reader who has not met either has no route from + # a pair of numbers to the defect. Raised by OffgridwithJD. + if [ "$_records" -gt "$_stated" ]; then + echo " $((_records - _stated)) check(s) reported an outcome the count never saw:" + echo " a check ran in a subshell, so its counter bump died with it while its" + echo " outcome and record still reached the log. The usual shape is a check" + # The example is ASSEMBLED, not written out: spelling the shape here + # made the sweep in selftest 400 flag this very line. + printf ' inside a piped loop -- `cmd %s while read x; do check ...; done`.\n' '|' + else + echo " $((_stated - _records)) check(s) were counted without emitting a record:" + echo " something bumped PGC_CHECKS without going through pgc_record." + fi return 1 fi return 0 @@ -979,6 +993,17 @@ pgc_log_shows_any_accounting() { # pgc_log_shows_any_accounting LOGFILE -> yes|n # Both are runtime-observable and derived rather than declared, so a suite # that adopts either mechanism leaves the debt bucket on its own -- which is # the property that keeps the debt file from becoming a permission slip. + # A HALF-MERGE WOULD BE CONFUSING RATHER THAN LOUD, and it is worth knowing + # which way. The `checks run:` alternative below answers YES to an accounting + # line of ANY shape, so it MASKS a change to that line: if the producer ever + # moved without this file, the population reconciliation would stay green + # while pgc_log_shows_accounting broke and the accounting reconciliation + # reddened. Two checks disagreeing about the same log is a worse signal than + # either failing. + # + # It cannot happen inside one tree -- producer and both readers move in the + # same commit -- so this is a note about what to look for, not a defect. + # Raised by OffgridwithJD while verifying the four-term shape change. local _log="$1" [ -f "$_log" ] || { echo no; return 0; } if [ "$(grep -cE '^accounting: [0-9]+ passed \+ [0-9]+ failed \+ [0-9]+ unrunnable \+ [0-9]+ skipped = [0-9]+$' "$_log" || true)" != 0 ] \ diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index 55d44ff4..806fa491 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -282,3 +282,91 @@ _allskip() { } check "a suite that skipped every check did not pass" \ "$(_allskip)" ": SKIPPED (ran no checks)" + +# ---- a check inside a pipeline loses its count, and the message must say so -- +# +# `printf ... | while read n; do check "$n" a a; done` runs the loop body in a +# SUBSHELL, so the counter bump dies with it while the outcome and the record are +# both printed to the parent's stdout. Driven: +# +# four checks print PASS, four RESULT lines appear, PGC_CHECKS=2 +# +# pgc_reconcile_records catches it -- that is what it is for -- but it reported +# `records=3 but the log states checks run: 1`, which is the BOOKKEEPING rather +# than the cause. A reader who has not met this before has no way from that line +# to the pipeline. Raised by OffgridwithJD. +# +# Latent today: four piped loops in the tree, none with a check inside. So the +# sweep below reports zero, and a fixture proves it can fire -- a rule whose only +# evidence is that the corpus is currently clean is not a rule. + +eval "$(sed -n '/^pgc_reconcile_records()/,/^}/p' "$_rv")" +_pl="$PGC_WORKDIR/piped.log" +printf 'RESULT\ts\tp\ta\tPASS\t\nRESULT\ts\tp\tb\tPASS\t\nRESULT\ts\tp\tc\tPASS\t\nchecks run: 1\n' > "$_pl" +check "more records than counted checks names the cause, not just the arithmetic" \ + "$(pgc_reconcile_records "$_pl" 2>&1 | grep -c 'a check ran in a subshell')" "1" +check "and still reports the two numbers" \ + "$(pgc_reconcile_records "$_pl" 2>&1 | grep -c 'records=3 .*checks run: 1')" "1" + +# Fewer records than checks is the OPPOSITE fault -- a counted check that emitted +# no record -- and must not be described as a subshell. +printf 'RESULT\ts\tp\ta\tPASS\t\nchecks run: 3\n' > "$_pl" +check "fewer records than counted checks is not described as a subshell" \ + "$(pgc_reconcile_records "$_pl" 2>&1 | grep -c 'a check ran in a subshell')" "0" +check "and names its own cause instead" \ + "$(pgc_reconcile_records "$_pl" 2>&1 | grep -c 'counted without emitting a record')" "1" + +# ---- and the shape is swept, the way selftest 080 sweeps its cousin ---------- + +_pipeloop_sites() { # _pipeloop_sites FILE... -> file:line of a check inside a piped loop + # Two refinements, both from measuring rather than reading. Requiring the + # closing `done` to be alone on its line left the scanner inside a loop for + # the rest of any file whose loop ended `done)"` -- 27 hits against a true + # zero. And a loop written entirely on ONE line, inside a command + # substitution, opened a block that never closed, flagging every later check. + # + # So: open only on a line that opens the loop and does NOT close it, and close + # on a `done` token wherever it sits. Comments are stripped first, because the + # rule's own explanation necessarily spells the shape out. + awk ' + FNR == 1 { inloop = 0 } + { line = $0; sub(/^[[:space:]]*#.*/, "", line) } + line ~ /\|[[:space:]]*(while|for)[[:space:]]/ && line ~ /(^|[[:space:];])do([[:space:]]|$)/ \ + && line !~ /(^|[[:space:]();])done([[:space:]();]|$)/ { inloop = 1; next } + inloop && line ~ /(^|[[:space:]();])done([[:space:]();]|$)/ { inloop = 0; next } + inloop && line ~ /(^|[^_[:alnum:]])(check|check_num|check_text|check_ratio|check_unrunnable|pgc_pass|pgc_fail)[[:space:]]/ { + print FILENAME ":" FNR + } + ' "$@" +} + +_pl_fx="$PGC_WORKDIR/plfx"; mkdir -p "$_pl_fx" +_pl_call="$(printf '%s "$n" a a' check)" +{ printf 'printf "a\\nb\\n" | while IFS= read -r n; do\n'; printf '\t%s\n' "$_pl_call"; printf 'done\n'; } \ + > "$_pl_fx/bad.sh" +{ printf 'while IFS= read -r n; do\n'; printf '\t%s\n' "$_pl_call"; printf 'done < <(printf "a\\nb\\n")\n'; } \ + > "$_pl_fx/good.sh" +{ printf 'printf "a\\nb\\n" | while IFS= read -r n; do\n'; printf '\techo "$n"\n'; printf 'done\n'; } \ + > "$_pl_fx/nocheck.sh" + +check "premise: the fixtures carry the shapes this sweep is about" \ + "$(grep -lc 'while IFS= read' "$_pl_fx"/*.sh | grep -c .)" "3" +check "the sweep finds a check inside a PIPED loop" \ + "$(_pipeloop_sites "$_pl_fx/bad.sh" | grep -c .)" "1" +check "and not one inside a process-substitution loop, which keeps its shell" \ + "$(_pipeloop_sites "$_pl_fx/good.sh" | grep -c .)" "0" +check "and not a piped loop with no check in it" \ + "$(_pipeloop_sites "$_pl_fx/nocheck.sh" | grep -c .)" "0" + +# A loop written entirely on one line inside a command substitution opens and +# closes in the same place. The first version of this sweep opened a block there +# and never closed it, flagging every check after it -- which is how twelve of +# its twenty-seven false hits were in this very file. +{ printf 'x="$(printf "a\\n" | while IFS= read -r n; do echo "$n"; done)"\n' + printf '%s\n' "$_pl_call"; } > "$_pl_fx/oneline.sh" +check "and not a check after a one-line piped loop that already closed" \ + "$(_pipeloop_sites "$_pl_fx/oneline.sh" | grep -c .)" "0" + +_pl_hits="$(_pipeloop_sites "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh 2>/dev/null | grep -c . || true)" +[ "${_pl_hits:-0}" = 0 ] || _pipeloop_sites "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh | sed 's/^/ /' +check "no suite calls a check inside a piped loop" "${_pl_hits:-0}" "0" From dddee7fcb473723039c6cb815d4a33cbcfcc2dd6 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 09:22:00 -0600 Subject: [PATCH 09/27] docs: the CHANGELOG entry this change should have shipped with (#917) jd's standing rule is that no PR ships without its CHANGELOG and docs in the same PR. `git diff --name-only main HEAD -- CHANGELOG.md` was 0 files for this branch, which OffgridwithJD caught: between this and #918 the stack adds pgc_record, a machine-readable RESULT format across every call site, a fourth counted outcome, three selftest parts and two tracked data files, and none of it was recorded. The entry covers what a reader of the release notes needs: why one counter rather than eleven, why the record names the part and not just the suite, why a skipped wall-clock check is counted, and what the matrix now reconciles. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- CHANGELOG.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ee62d1c1..a0fc9251 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,41 @@ true until the next version shipped. ### Added +- Every check result is machine-readable, and counting a check is the same + operation as recording it (#917). + + `check`, `check_num` and `check_text` printed `PASS` or `FAIL` and nothing + else, so proving that a mutation reddened one NAMED check meant grepping prose + and retyping the result. That is also how a reverted guard once reported plain + green while the check count fell from 190 to 186: the suite passed, and the only + evidence anything had changed was a number nobody compared. + + `lib.sh` had eleven places that bumped `PGC_CHECKS`, each with its own outcome + line beside it, which is eleven chances to add a twelfth and forget the line. + `projections.sh` did exactly that with an `expect_fail` at ten call sites, for + as long as it existed. There is now one, `pgc_record`, so a helper cannot report + an outcome without being counted and cannot be counted without reporting one. + Every human line is byte-identical; 3,762 call sites is past what a careful + refactor can be trusted on, so both harnesses pin the exact strings. + + Each record names the suite, the part, the check, the verdict and the reason. + The part matters because `harness_selftest` sources 40-odd parts into one shell + and phrases its premises to be copied, so a key of suite and name is a key of + check NAMES rather than of checks: 583 records give 579 distinct pairs against + 582 distinct triples. It is derived from `BASH_SOURCE`, not from a convention. + + A skipped wall-clock check is a fourth counted outcome. Under + `PGC_SKIP_TIMING`, `check_timing` and `check_ratio_needs_quiet_machine` printed + a `SKIP` line a reader sees while leaving the count at zero and emitting no + record, in branches no arm reached. `checks run:` now reports the checks a suite + encountered rather than the ones it evaluated, and the summary reconciles four + counters against it. A suite that skipped every check reports `SKIPPED` rather + than `PASSED`, which the old zero-check condition caught only by accident. + + The matrix reconciles each suite's records against the count its log states, and + names the cause rather than the arithmetic: more records than counted is a check + that ran in a subshell, fewer is a counter bumped outside `pgc_record`. + - Exact zone-map boundary coverage now lives in matching shell and pytest tests (#831). From 2289f10df115161881db456b22ba0fa7f5433e55 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 10:10:14 -0600 Subject: [PATCH 10/27] test: redesign the ledger so it is fed, so it can refuse, and so it cannot deadlock (#918) jd's direction: shipping the recording half alone would be shipping something that does nothing. A ledger nothing feeds and nothing reads is not a staged feature, it is a data file with no producer and no consumer -- the `defeated: 0` shape one level up. So the gate is fixed rather than dropped. Ten findings, from @linuxhikerpm and OffgridwithJD, every one reproduced first. IT IS NOW FED AND IT NOW REFUSES. Nothing in the repository called the tool: zero references in .github/, zero in the runner, so "the gate refuses a check the ledger has never seen" was false as written. run_all_versions.sh now runs it before it removes the build directory, which is the only place a matrix run can reach every suite's log. CI verifies; humans commit the ledger, because a ledger CI rewrote by itself would be a file nobody reads changing under everybody. THE DEADLOCK WAS THE DESIGN, NOT THE NUMBER. Bounding `checks_never_observed_red` means every added check breaks the gate: a new check enters as `never`, so the only way to land one was to raise a number the file says in capitals may only fall. It shipped at 614 rows, 614 never, ceiling 614. The two numbers are different kinds of thing and the file now says so. `checks_never_observed_red` is a CENSUS, asserted to match the ledger in both harnesses so it cannot drift. `suites_not_covered` IS a ceiling, because adding a check to a covered suite does not move it, and the gate compares it against the previously committed value and refuses an increase -- so "may only fall" is mechanism rather than prose. AND THE REFUSAL IS RESTRICTED TO SUITES THE LEDGER COVERS, which is the MEANING of that ceiling rather than a softening of the gate. Without it the gate refuses every check of all 250 uncovered suites and reddens the whole matrix on its first run, which is a gate somebody turns off within the week. It tightens on its own as suites are seeded, and an arm pins that a new check in a NOW-covered suite is refused again. FAIL CLOSED. A nonexistent log, an empty one and a record missing its verdict each returned rc=0. An integrity failure that reads as a clean run is worse than no gate because it certifies. They return 2, distinguishable from a real refusal at 1, and --registered is required rather than silently skipped. THE MUTATION COLUMN ACCUMULATES a set rather than overwriting, because keeping only the last one records the most recent attack rather than the catalogue the column exists to become. One --mutation cannot be attributed across several logs at once. MULTI-LOG HANDLING was wrong in two directions. The same check in two logs is two RUNS and was reported as a duplicate; the same name twice in ONE log is the duplicate, and is what is reported now. Renames are grouped by (suite, part) before pairing, because a global positional zip misses a real rename whenever unrelated movement elsewhere shifts the ordering -- and a before-log and an after-log together are refused rather than silently finding nothing, since the vanished name is present in the union. 614 rows ended in a tab, because an empty mutation was an empty last field. An absent mutation is now `-`; `git diff --check` reports nothing. Evidence: selftest exit 0, 638 checks, 0 failures; 9 pytest; shellcheck rc=0; docs_style PASSED; git diff --check clean. Both harnesses carry red-and-control arms for every integrity boundary above, including the deadlock as its own arm. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- CHANGELOG.md | 38 + test/check_ledger.tsv | 1252 +++++++++-------- test/check_ledger_budget.txt | 55 +- test/pgc_ledger.py | 395 ++++-- test/pytest/TESTS.md | 137 +- test/pytest/test_mutation_ledger.py | 401 +++--- test/run_all_versions.sh | 39 + .../410-a-check-must-have-been-red.sh | 421 +++--- 8 files changed, 1507 insertions(+), 1231 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ee62d1c1..0cb74d69 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,44 @@ true until the next version shipped. ### Added +- A ledger of which checks have ever been seen red, and under what (#918). + + Nothing recorded it. That is the gap that let 39 checks across 35 suites ship + unable to fail, three of them inside the suite whose whole purpose is to stop + exactly that: the gate answered "did anything print FAIL" and had never + answered "could anything print FAIL". + + It records that a named check WAS OBSERVED RED in a recorded run. It does not + claim the check is proven able to fail, which needs a named mutation applied + deliberately; conflating the two would put a claim in the ledger that nothing + measured. It is fed by every real failure, not only by deliberate mutation + runs. + + `run_all_versions.sh` runs the gate before it removes the build directory, + which is the only place a matrix run can reach every suite's log. What the gate + refuses is a check the committed ledger has never seen, in a suite the ledger + covers. Regenerating the ledger is the intended fix and a reviewable diff. + + The two tracked numbers are different kinds of thing, and the first design + treated both as ceilings and deadlocked. `checks_never_observed_red` is a + CENSUS: every new check enters as `never`, so bounding it means every added + check breaks the gate and the only way to land one is to raise a number the + design says may only fall. It shipped that way once, at 614 rows, 614 never, + ceiling 614. `suites_not_covered` IS a ceiling, because adding a check to a + covered suite does not move it, and the gate refuses to see it raised above its + previously committed value rather than leaving that to review. + + The row is keyed on suite, part and check name. The part matters because + `harness_selftest` sources 40-odd parts into one shell and phrases its premises + to be copied, so a name-only key is a key of check NAMES: 583 records give 579 + distinct pairs against 582 distinct triples. A rename is detected and named + rather than silently resetting a check's history to `never`, and the mutation + column accumulates a set rather than keeping only the most recent attack. + + Bad input fails closed. An unreadable log, an empty one, and a record missing + its verdict each returned success before, which is worse than no gate because + it certifies. + - Exact zone-map boundary coverage now lives in matching shell and pytest tests (#831). diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index e3542d36..77868e8a 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1,614 +1,638 @@ -harness_selftest 030-assertions nothing leaked into the squatter never -harness_selftest 030-assertions pgc_port_free says the squatter's port is busy never -harness_selftest 030-assertions squatter survived untouched never -harness_selftest 030-assertions suite did not settle on the squatter's port never -harness_selftest 030-assertions suite's cluster is its own never -harness_selftest 030-assertions suite's own objects are visible to it never -harness_selftest 040-the-detection-primitive-itself detection distinguishes it from ours never -harness_selftest 040-the-detection-primitive-itself detection reports a foreign cluster's directory never -harness_selftest 040-the-detection-primitive-itself guard accepts our own cluster never -harness_selftest 040-the-detection-primitive-itself guard rejects a foreign cluster never -harness_selftest 040-the-detection-primitive-itself premise: the runner answered --list-suites, so the two checks below mean something never -harness_selftest 050-the-list-must-be-read-the a name after the array's closing paren is not read as a registered suite never -harness_selftest 050-the-list-must-be-read-the and the mistake empties the whole array rather than appending to it never -harness_selftest 050-the-list-must-be-read-the positive control: and it is a whole list, not one lucky line never -harness_selftest 050-the-list-must-be-read-the positive control: the real runner's list is read, and contains isolation never -harness_selftest 050-the-list-must-be-read-the premise: the fixture really does carry the stray name never -harness_selftest 060-the-list-stays-sorted-which-is premise: C collation puts sort_status before sorted_projection never -harness_selftest 060-the-list-stays-sorted-which-is the suite list is sorted in C order, so two new suites land in different places never -harness_selftest 070-and-comm-s-two-inputs-must a file that uses comm pins the collation of every sort feeding it never -harness_selftest 070-and-comm-s-two-inputs-must and a prefix of a registered name is not treated as registered never -harness_selftest 070-and-comm-s-two-inputs-must every registered suite has a file never -harness_selftest 070-and-comm-s-two-inputs-must every suite is registered in run_all_versions.sh never -harness_selftest 070-and-comm-s-two-inputs-must negative control: and does not find one that is not never -harness_selftest 070-and-comm-s-two-inputs-must positive control: the membership test finds a name that is registered never -harness_selftest 070-and-comm-s-two-inputs-must premise: some suite still uses comm, or the check below is vacuous never -harness_selftest 080-no-suite-pipes-a-captured-string and bench/ was in the scan, which is the hole this rule had never -harness_selftest 080-no-suite-pipes-a-captured-string and does not cover the live one above it never -harness_selftest 080-no-suite-pipes-a-captured-string and selftest/ was in the scan, which is the hole that reddened #923 never -harness_selftest 080-no-suite-pipes-a-captured-string and the heredoc exemption covers the one inside the heredoc, not the other never -harness_selftest 080-no-suite-pipes-a-captured-string and the scan examined the suites rather than finding nothing to read never -harness_selftest 080-no-suite-pipes-a-captured-string control: piping a large string into grep -q reports a match as absent never -harness_selftest 080-no-suite-pipes-a-captured-string no suite pipes a captured string into an early-exit reader never -harness_selftest 080-no-suite-pipes-a-captured-string premise: and the exemption covers a minority of them, not the corpus never -harness_selftest 080-no-suite-pipes-a-captured-string premise: and the line really does hold the reader it must not flag never -harness_selftest 080-no-suite-pipes-a-captured-string premise: and the old echo/printf pattern did NOT catch it never -harness_selftest 080-no-suite-pipes-a-captured-string premise: the heredoc exemption found heredoc lines to exempt never -harness_selftest 080-no-suite-pipes-a-captured-string premise: the sweep read lines to classify never -harness_selftest 080-no-suite-pipes-a-captured-string the sweep catches a producer that is neither echo nor printf never -harness_selftest 080-no-suite-pipes-a-captured-string the sweep does not mistake the || operator for a pipe never -harness_selftest 080-no-suite-pipes-a-captured-string the sweep's pattern sees both lines of the probe never -harness_selftest 090-no-suite-hands-every-run-the no suite hands every run the same default port never -harness_selftest 090-no-suite-hands-every-run-the no test picks a port from inside the ephemeral range never -harness_selftest 100-the-assertions-that-refuse-an-empty check compares two empty strings and passes, which is why the rest exist never -harness_selftest 100-the-assertions-that-refuse-an-empty check_num accepts a decimal and a sign never -harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses a psql error message never -harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses an md5, which is why check_text exists never -harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses the word a yes/no check would produce never -harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses two empty strings never -harness_selftest 100-the-assertions-that-refuse-an-empty check_num still compares two real numbers never -harness_selftest 100-the-assertions-that-refuse-an-empty check_num still fails two unequal numbers never -harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio fails a ratio outside its bound never -harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio passes a ratio inside its bound never -harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio refuses a zero denominator rather than dividing by it never -harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio refuses a zero numerator, which is inside every bound never -harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio refuses an empty measurement never -harness_selftest 100-the-assertions-that-refuse-an-empty check_text compares two md5 hashes, which check_num cannot never -harness_selftest 100-the-assertions-that-refuse-an-empty check_text refuses one empty side never -harness_selftest 100-the-assertions-that-refuse-an-empty check_text refuses two empty strings, where plain check passes never -harness_selftest 100-the-assertions-that-refuse-an-empty check_text still fails two different strings never -harness_selftest 100-the-assertions-that-refuse-an-empty pgc_require_tools fails on one that does not never -harness_selftest 100-the-assertions-that-refuse-an-empty pgc_require_tools passes on tools that exist never -harness_selftest 110-the-harness-must-say-which-binary pgc_setup reports the installed .so never -harness_selftest 110-the-harness-must-say-which-binary the installed .so is the one this run built never -harness_selftest 120-a-failing-suite-must-surface-the a failing suite names the first fatal event in its log never -harness_selftest 120-a-failing-suite-must-surface-the premise: the 40-line tail is filler, not the marker never -harness_selftest 120-a-failing-suite-must-surface-the premise: the sub-suite failed, so its summary ran never -harness_selftest 130-the-sanitizer-subset-must-cover-the premise: at least one suite drives the C-level encoding selftest never -harness_selftest 130-the-sanitizer-subset-must-cover-the premise: run_san.sh's default subset was found and is non-empty never -harness_selftest 130-the-sanitizer-subset-must-cover-the the sanitizer subset runs every suite that drives the encoding selftest never -harness_selftest 140-a-cluster-that-will-not-start and still matches a PANIC never -harness_selftest 140-a-cluster-that-will-not-start and still matches a signal death never -harness_selftest 140-a-cluster-that-will-not-start and still matches an AddressSanitizer report never -harness_selftest 140-a-cluster-that-will-not-start but not a routine statement error never -harness_selftest 140-a-cluster-that-will-not-start nor an ordinary log line never -harness_selftest 140-a-cluster-that-will-not-start premise: the harness exposes its fatal pattern to be judged never -harness_selftest 140-a-cluster-that-will-not-start the fatal pattern matches a library that will not load never -harness_selftest 150-the-verdict-must-not-assert-a a mixed run reports both causes and neither as the whole story never -harness_selftest 150-the-verdict-must-not-assert-a and is NOT made when our own postmaster died, which is the #537 case never -harness_selftest 150-the-verdict-must-not-assert-a and so is the attempt count never -harness_selftest 150-the-verdict-must-not-assert-a and the no-squatter verdict points at the server log never -harness_selftest 150-the-verdict-must-not-assert-a premise: the verdict is composed somewhere it can be judged never -harness_selftest 150-the-verdict-must-not-assert-a the ownership claim is made when a squatter held the port every time never -harness_selftest 150-the-verdict-must-not-assert-a the port is named either way never -harness_selftest 160-and-the-log-report-must-show and a log with no fatal line still reports rather than staying silent never -harness_selftest 160-and-the-log-report-must-show premise: the log report is a function that can be fed a fixture never -harness_selftest 160-and-the-log-report-must-show the report names the symbol that was actually missing never -harness_selftest 170-and-lib-sh-must-ask-these and asks pgc_start_failure_message for the verdict never -harness_selftest 170-and-lib-sh-must-ask-these and the old start-failure verdict is not echoed inline anywhere never -harness_selftest 170-and-lib-sh-must-ask-these and the start path asks pgc_start_fatal_pattern, its deliberately wider one never -harness_selftest 170-and-lib-sh-must-ask-these premise: lib.sh is readable, or every grep below approves nothing never -harness_selftest 170-and-lib-sh-must-ask-these premise: the start-failure path still exists to be judged never -harness_selftest 170-and-lib-sh-must-ask-these the failure path asks pgc_start_log_report for the reason never -harness_selftest 170-and-lib-sh-must-ask-these the summary path asks pgc_fatal_pattern rather than hardcoding it never -harness_selftest 180-the-port-walk-must-wrap-not a free port beyond the old 300-probe bound is still found never -harness_selftest 180-the-port-walk-must-wrap-not a seed at the ceiling wraps past a busy top and still finds a port never -harness_selftest 180-the-port-walk-must-wrap-not an entirely busy band reports itself full and terminates never -harness_selftest 180-the-port-walk-must-wrap-not and the port it found is below the busy region, which is where wrapping lands never -harness_selftest 180-the-port-walk-must-wrap-not premise: and really does allow the bottom never -harness_selftest 180-the-port-walk-must-wrap-not premise: the auxiliary band has a width to wrap within never -harness_selftest 180-the-port-walk-must-wrap-not premise: the real prober was restored, or every check after this lies never -harness_selftest 180-the-port-walk-must-wrap-not premise: the stub frees exactly one port, 500 past the floor never -harness_selftest 180-the-port-walk-must-wrap-not premise: the stub really does refuse the top of the band never -harness_selftest 190-an-in-tree-build-must-not an unknown provenance is not reported as a major never -harness_selftest 190-an-in-tree-build-must-not an unparseable stamp cleans rather than guessing never -harness_selftest 190-an-in-tree-build-must-not and an empty WANT is refused rather than compared never -harness_selftest 190-an-in-tree-build-must-not and in the other direction too never -harness_selftest 190-an-in-tree-build-must-not and it is 3 bytes, not an escaped literal never -harness_selftest 190-an-in-tree-build-must-not and it says plainly that no major was recorded never -harness_selftest 190-an-in-tree-build-must-not building a DIFFERENT major needs a clean, which is the #536 case never -harness_selftest 190-an-in-tree-build-must-not building the same major again needs no clean never -harness_selftest 190-an-in-tree-build-must-not but a tree with no objects at all needs nothing, stamp or not never -harness_selftest 190-an-in-tree-build-must-not objects with NO stamp are unknown provenance and must be cleaned never -harness_selftest 190-an-in-tree-build-must-not premise: the build-stamp decision is exposed to be judged never -harness_selftest 190-an-in-tree-build-must-not premise: the stamp writer is a function that can be exercised never -harness_selftest 190-an-in-tree-build-must-not the build path asks pgc_build_needs_clean rather than merely naming it never -harness_selftest 190-an-in-tree-build-must-not the stamp lib.sh writes is exactly the major never -harness_selftest 200-additions-go-in-their-own-file premise: the parts directory exists and was sourced never -harness_selftest 200-additions-go-in-their-own-file the driver holds no checks; they all live in parts never -harness_selftest 200-additions-go-in-their-own-file the driver sources the parts by glob, not by a list never -harness_selftest 210-no-suite-assigns-a-bash-special control: reads, longer names, and the deliberate RANDOM/SECONDS seeds are not flagged never -harness_selftest 210-no-suite-assigns-a-bash-special control: the sweep catches an assignment to a bash special never -harness_selftest 210-no-suite-assigns-a-bash-special no suite assigns to a bash special variable never -harness_selftest 220-an-opt-in-upgrade-guard-must CI runs the extension-upgrade guard somewhere (#741) never -harness_selftest 220-an-opt-in-upgrade-guard-must every PGC_RUN_UPGRADE-gated suite is excluded from the coverage runner (#741) never -harness_selftest 220-an-opt-in-upgrade-guard-must premise: both not_a_suite definitions were found never -harness_selftest 220-an-opt-in-upgrade-guard-must premise: not_a_suite says no to an ordinary suite, so its yes means something never -harness_selftest 220-an-opt-in-upgrade-guard-must premise: run_coverage.sh defines not_a_suite and calls it never -harness_selftest 220-an-opt-in-upgrade-guard-must premise: the PGC_RUN_UPGRADE block was found and names at least one suite never -harness_selftest 220-an-opt-in-upgrade-guard-must premise: the population is the real test directory, not an empty glob never -harness_selftest 220-an-opt-in-upgrade-guard-must premise: there are workflow files to search never -harness_selftest 220-an-opt-in-upgrade-guard-must the coverage runner's not_a_suite agrees with the selftest's, both ways (#741) never -harness_selftest 230-a-suite-connecting-by-socket-must every suite that connects by socket path sets unix_socket_directories never -harness_selftest 230-a-suite-connecting-by-socket-must premise: at least one suite connects by a socket path, so this is not vacuous never -harness_selftest 230-a-suite-connecting-by-socket-must premise: no TCP suite is counted as a socket user never -harness_selftest 240-the-nightly-enumeration-must-not every nightly job is named in docs/testing.md (#741) never -harness_selftest 240-the-nightly-enumeration-must-not premise: at least three nightly jobs were parsed, so the list is real never -harness_selftest 240-the-nightly-enumeration-must-not premise: the nightly paragraph was located and is not empty never -harness_selftest 240-the-nightly-enumeration-must-not premise: the nightly workflow and the testing doc are both present never -harness_selftest 250-the-coverage-runner-must-refuse GCOV_PREFIX is exported before the suites run (#740) never -harness_selftest 250-the-coverage-runner-must-refuse premise: both stray-counter probes were located never -harness_selftest 250-the-coverage-runner-must-refuse premise: both the counter refusal and the lcov capture were located never -harness_selftest 250-the-coverage-runner-must-refuse premise: the containment and the copy were both located never -harness_selftest 250-the-coverage-runner-must-refuse premise: the coverage runner is present and parses never -harness_selftest 250-the-coverage-runner-must-refuse premise: the guard's count directory and the capture's were both located never -harness_selftest 250-the-coverage-runner-must-refuse premise: the redirect, the suite invocation and the copy-back were located never -harness_selftest 250-the-coverage-runner-must-refuse the copy-back refuses a destination outside the tree (#740) never -harness_selftest 250-the-coverage-runner-must-refuse the counters are returned beside their objects before the refusal (#740) never -harness_selftest 250-the-coverage-runner-must-refuse the coverage runner refuses zero counters before it calls lcov (#740) never -harness_selftest 250-the-coverage-runner-must-refuse the refusal looks in GCOV_PREFIX before the tree-wide walk (#740) never -harness_selftest 250-the-coverage-runner-must-refuse the zero-counter guard counts the directory lcov captures (#740) never -harness_selftest 260-an-ordered-comparison-must-use-the and it is the same suites, not merely the same count never -harness_selftest 260-an-ordered-comparison-must-use-the every diff_query_ordered site actually names an ORDER BY never -harness_selftest 260-an-ordered-comparison-must-use-the every suite using the ordered oracle asserts its premise never -harness_selftest 260-an-ordered-comparison-must-use-the no diff_query site names an ORDER BY it cannot test (use diff_query_ordered) never -harness_selftest 260-an-ordered-comparison-must-use-the premise: both comparison helpers are present never -harness_selftest 260-an-ordered-comparison-must-use-the premise: both oracles are present never -harness_selftest 260-an-ordered-comparison-must-use-the premise: some suite uses the ordered oracle, or the next check is vacuous never -harness_selftest 260-an-ordered-comparison-must-use-the premise: the tree really contains continued diff_query calls to join never -harness_selftest 260-an-ordered-comparison-must-use-the sorted_projection's two comparisons are ordered, its subject being order never -harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle keeps the empty-result sentinel never -harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle keeps the unique query-error sentinel never -harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle numbers the rows as they arrive never -harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle orders by row_number, so it keeps the query's order never -harness_selftest 260-an-ordered-comparison-must-use-the the set oracle orders by the rendered row, so it is order-blind never -harness_selftest 270-a-set-options-call-must-use-values no suite calls set_options with a value it will reject never -harness_selftest 270-a-set-options-call-must-use-values premise: and does NOT fire when the error is the point, across a continuation never -harness_selftest 270-a-set-options-call-must-use-values premise: every file containing a set_options call is in the sweep never -harness_selftest 270-a-set-options-call-must-use-values premise: the detector fires on an out-of-range value that is NOT expect_error never -harness_selftest 270-a-set-options-call-must-use-values premise: the set_options sweep read a substantial number of calls never -harness_selftest 280-the-shared-cluster-config-must-not premise: and it is the right block (it sets the port and the preload) never -harness_selftest 280-the-shared-cluster-config-must-not premise: the cluster-config block was located in lib.sh never -harness_selftest 280-the-shared-cluster-config-must-not premise: the detector fires on the line that caused #799 never -harness_selftest 280-the-shared-cluster-config-must-not the per-suite escape hatch PGC_EXTRA_CONF is still applied to the config never -harness_selftest 280-the-shared-cluster-config-must-not the shared cluster config sets no pgcolumnar.* GUC never -harness_selftest 290-a-preflight-that-built-nothing a preflight that built nothing does not report PASSED never -harness_selftest 290-a-preflight-that-built-nothing a preflight that built nothing exits non-zero never -harness_selftest 290-a-preflight-that-built-nothing a preflight that built nothing says how many it built never -harness_selftest 290-a-preflight-that-built-nothing premise: and it built none of them never -harness_selftest 290-a-preflight-that-built-nothing premise: the probe run skipped every major never -harness_selftest 300-a-test-script-must-be-runnable and every executable script declares one never -harness_selftest 300-a-test-script-must-be-runnable and every script a document names exists never -harness_selftest 300-a-test-script-must-be-runnable and every script a document names is executable never -harness_selftest 300-a-test-script-must-be-runnable control: an interpreter declared without the bit is caught never -harness_selftest 300-a-test-script-must-be-runnable control: and a sourced fragment, with neither, is correct never -harness_selftest 300-a-test-script-must-be-runnable control: and the same file with the bit is not never -harness_selftest 300-a-test-script-must-be-runnable control: cp -a preserves the execute bit, so a staged tree reads the same never -harness_selftest 300-a-test-script-must-be-runnable control: the bit without an interpreter is caught too never -harness_selftest 300-a-test-script-must-be-runnable every script that declares an interpreter is executable never -harness_selftest 300-a-test-script-must-be-runnable premise: a runnable script one level down is inside the population never -harness_selftest 300-a-test-script-must-be-runnable premise: and bench/ is in the population never -harness_selftest 300-a-test-script-must-be-runnable premise: and so are the fixture host tools never -harness_selftest 300-a-test-script-must-be-runnable premise: and they name at least one command in every swept directory never -harness_selftest 300-a-test-script-must-be-runnable premise: the documents name a population of commands, not none never -harness_selftest 300-a-test-script-must-be-runnable premise: the sourced parts are inside the population, not pruned never -harness_selftest 300-a-test-script-must-be-runnable premise: the sweep reads a population of scripts, not an empty find never -harness_selftest 310-a-compiled-artifact-must-not-be and a compiled artifact written beside its source never -harness_selftest 310-a-compiled-artifact-must-not-be and the tracked list names none of them never -harness_selftest 310-a-compiled-artifact-must-not-be and the tree ignores the directory Python writes them to never -harness_selftest 310-a-compiled-artifact-must-not-be no compiled Python artifact is tracked never -harness_selftest 310-a-compiled-artifact-must-not-be premise: and git ls-files sees the harness it is being asked about never -harness_selftest 310-a-compiled-artifact-must-not-be premise: and that a tracked source file is not never -harness_selftest 310-a-compiled-artifact-must-not-be premise: check-ignore agrees a build object is already ignored never -harness_selftest 310-a-compiled-artifact-must-not-be premise: the source tree is a git checkout never -harness_selftest 320-a-check-that-could-not-run 67 without its line is a failure, not an INCOMPLETE taken on trust never -harness_selftest 320-a-check-that-could-not-run a counter that drifts is caught rather than absorbed never -harness_selftest 320-a-check-that-could-not-run a failure outranks an unrunnable check, and both are still counted never -harness_selftest 320-a-check-that-could-not-run a passing ratio check is counted as a pass, not a failure never -harness_selftest 320-a-check-that-could-not-run a suite of nothing but unrunnable checks is INCOMPLETE, not SKIPPED never -harness_selftest 320-a-check-that-could-not-run a suite whose checks all passed still exits 0 PASSED never -harness_selftest 320-a-check-that-could-not-run a suite with none says so as zero rather than staying silent never -harness_selftest 320-a-check-that-could-not-run an INCOMPLETE suite fails its major never -harness_selftest 320-a-check-that-could-not-run an unrunnable check counts toward checks run never -harness_selftest 320-a-check-that-could-not-run an unrunnable reason outside the enum fails rather than being accepted never -harness_selftest 320-a-check-that-could-not-run and 66 with its line a skip never -harness_selftest 320-a-check-that-could-not-run and 67 with its line INCOMPLETE, which is not a pass never -harness_selftest 320-a-check-that-could-not-run and a failing suite still does never -harness_selftest 320-a-check-that-could-not-run and a skip does not, which is the one that must stay true never -harness_selftest 320-a-check-that-could-not-run and a suite with no unrunnable checks reconciles too never -harness_selftest 320-a-check-that-could-not-run and allows one that does, which is what it was written to allow never -harness_selftest 320-a-check-that-could-not-run and an ordinary failure is still a failure never -harness_selftest 320-a-check-that-could-not-run and both exempt a file that keeps its own counter without lib.sh never -harness_selftest 320-a-check-that-could-not-run and it is not reported as having run no checks never -harness_selftest 320-a-check-that-could-not-run and no write-only failure flag survives in the runner never -harness_selftest 320-a-check-that-could-not-run and the suite holding it fails rather than reporting PASSED never -harness_selftest 320-a-check-that-could-not-run and the suite that holds it still passes never -harness_selftest 320-a-check-that-could-not-run and the unrunnable ones are reported as their own count never -harness_selftest 320-a-check-that-could-not-run every direct write to PGC_CHECKS records an outcome too never -harness_selftest 320-a-check-that-could-not-run lib.sh defines check_unrunnable never -harness_selftest 320-a-check-that-could-not-run lib.sh defines the INCOMPLETE exit status never -harness_selftest 320-a-check-that-could-not-run no non-zero status is classified as a pass never -harness_selftest 320-a-check-that-could-not-run no suite that uses lib.sh's accounting writes PGC_CHECKS directly never -harness_selftest 320-a-check-that-could-not-run one unrunnable check makes the suite INCOMPLETE, not passed never -harness_selftest 320-a-check-that-could-not-run premise: the classifier evalled out of the runner is callable never -harness_selftest 320-a-check-that-could-not-run premise: the fixtures carry the shapes these rules are about never -harness_selftest 320-a-check-that-could-not-run premise: the harness library is where this part thinks it is never -harness_selftest 320-a-check-that-could-not-run premise: the major-verdict mapping evalled out of the runner is callable never -harness_selftest 320-a-check-that-could-not-run premise: the runner defines the classifier this part is about to eval never -harness_selftest 320-a-check-that-could-not-run premise: the sweep read the corpus and found sites to classify never -harness_selftest 320-a-check-that-could-not-run the original rule flags a bump that records no outcome never -harness_selftest 320-a-check-that-could-not-run the runner calls a clean exit a pass never -harness_selftest 320-a-check-that-could-not-run the runner's INCOMPLETE branch calls the mapping rather than a local flag never -harness_selftest 320-a-check-that-could-not-run the stronger rule flags that same allowed bump, which is the change never -harness_selftest 320-a-check-that-could-not-run the summary reconciles the three states against the total never -harness_selftest 320-a-check-that-could-not-run the unrunnable check names itself, its reason code and its detail never -harness_selftest 320-a-check-that-could-not-run while a pass does not never -harness_selftest 330-the-incomplete-path-must-run-whole a suite with an unrunnable check exits 67 never -harness_selftest 330-the-incomplete-path-must-run-whole an INCOMPLETE suite sets the flag the major verdict actually reads never -harness_selftest 330-the-incomplete-path-must-run-whole and appears in the results string as INCOMPLETE never -harness_selftest 330-the-incomplete-path-must-run-whole and counted as incomplete, so the tally can say so never -harness_selftest 330-the-incomplete-path-must-run-whole and counts both suites as having run never -harness_selftest 330-the-incomplete-path-must-run-whole and exactly one of them as incomplete never -harness_selftest 330-the-incomplete-path-must-run-whole and is counted as having run never -harness_selftest 330-the-incomplete-path-must-run-whole and is not counted as skipped, nor is the skip count disturbed never -harness_selftest 330-the-incomplete-path-must-run-whole and its log carries the INCOMPLETE line the classifier needs never -harness_selftest 330-the-incomplete-path-must-run-whole and neither as skipped never -harness_selftest 330-the-incomplete-path-must-run-whole and no longer counts incompletes inline beside it never -harness_selftest 330-the-incomplete-path-must-run-whole and records each suite's own verdict in the results string never -harness_selftest 330-the-incomplete-path-must-run-whole and reprints the suite's own UNRUN line beneath it never -harness_selftest 330-the-incomplete-path-must-run-whole and the UNRUN line the runner prints into the matrix output never -harness_selftest 330-the-incomplete-path-must-run-whole and the run's overall status is failure never -harness_selftest 330-the-incomplete-path-must-run-whole and the summary line carries the incomplete count a reader needs never -harness_selftest 330-the-incomplete-path-must-run-whole and the tally announces it, with the reason lifted from the log never -harness_selftest 330-the-incomplete-path-must-run-whole control fixture: a suite whose checks all ran exits 0 never -harness_selftest 330-the-incomplete-path-must-run-whole control: and leaves the run's overall status alone never -harness_selftest 330-the-incomplete-path-must-run-whole control: and still announces it never -harness_selftest 330-the-incomplete-path-must-run-whole control: and still records that it ran, and how never -harness_selftest 330-the-incomplete-path-must-run-whole control: and the major reports PASS never -harness_selftest 330-the-incomplete-path-must-run-whole control: the same loop leaves a passing suite passing never -harness_selftest 330-the-incomplete-path-must-run-whole premise: all three runner functions were extracted, not empty ranges never -harness_selftest 330-the-incomplete-path-must-run-whole premise: and all three are callable never -harness_selftest 330-the-incomplete-path-must-run-whole premise: and each extraction ends at its own closing brace never -harness_selftest 330-the-incomplete-path-must-run-whole premise: the major-verdict branch was extracted, not an empty range never -harness_selftest 330-the-incomplete-path-must-run-whole premise: the runner's collect loop was extracted, not an empty range never -harness_selftest 330-the-incomplete-path-must-run-whole premise: the selftest has a workdir to build fixtures in never -harness_selftest 330-the-incomplete-path-must-run-whole running the real loop over both fixtures fails the major never -harness_selftest 330-the-incomplete-path-must-run-whole the loop delegates each verdict to pgc_tally_suite never -harness_selftest 330-the-incomplete-path-must-run-whole the runner classifies the file that suite actually produced never -harness_selftest 330-the-incomplete-path-must-run-whole with an incomplete suite in the tally the major reports FAIL never -harness_selftest 340-the-binary-must-be-built-from PREMISE and the target really holds sources find would otherwise hash never -harness_selftest 340-the-binary-must-be-built-from PREMISE the Makefile's recursion was actually parsed never -harness_selftest 340-the-binary-must-be-built-from PREMISE the copy discovers the same build directories as the real tree never -harness_selftest 340-the-binary-must-be-built-from PREMISE the fingerprint covers at least src never -harness_selftest 340-the-binary-must-be-built-from PREMISE the fixture's src really is a symlink never -harness_selftest 340-the-binary-must-be-built-from a /./ segment hashes the same tree the same way never -harness_selftest 340-the-binary-must-be-built-from a /src/.. segment hashes the same tree the same way never -harness_selftest 340-the-binary-must-be-built-from a caller passing a major is caught never -harness_selftest 340-the-binary-must-be-built-from a file that is not a build input does not move it never -harness_selftest 340-the-binary-must-be-built-from a fingerprint different from the record is stale never -harness_selftest 340-the-binary-must-be-built-from a fingerprint equal to the record is fresh never -harness_selftest 340-the-binary-must-be-built-from a fingerprint is 12 hex characters never -harness_selftest 340-the-binary-must-be-built-from a library newer than the running server is REFUSED never -harness_selftest 340-the-binary-must-be-built-from a library older than the running server is accepted never -harness_selftest 340-the-binary-must-be-built-from a missing binary timestamp is unknown, not predates never -harness_selftest 340-the-binary-must-be-built-from a missing postmaster timestamp is unknown, not predates never -harness_selftest 340-the-binary-must-be-built-from a new source file under objstore moves the fingerprint never -harness_selftest 340-the-binary-must-be-built-from a relative path hashes the same tree the same way never -harness_selftest 340-the-binary-must-be-built-from a server older than the binary predates it never -harness_selftest 340-the-binary-must-be-built-from a server started after the binary is fresh never -harness_selftest 340-the-binary-must-be-built-from a server started at the same second is fresh never -harness_selftest 340-the-binary-must-be-built-from a symlink to the tree hashes it the same way never -harness_selftest 340-the-binary-must-be-built-from a symlinked src contributes nothing, as find -P contributes nothing never -harness_selftest 340-the-binary-must-be-built-from a trailing slash hashes the same tree the same way never -harness_selftest 340-the-binary-must-be-built-from a tree with no hashable file yields no fingerprint never -harness_selftest 340-the-binary-must-be-built-from adding a source file moves it never -harness_selftest 340-the-binary-must-be-built-from an added file appears in the manifest by name never -harness_selftest 340-the-binary-must-be-built-from an added file shows up in the report never -harness_selftest 340-the-binary-must-be-built-from an empty manifest is reported as empty, not as silence never -harness_selftest 340-the-binary-must-be-built-from an unhashable tree has an empty manifest never -harness_selftest 340-the-binary-must-be-built-from an unreadable b.c yields no fingerprint, not a wrong one never -harness_selftest 340-the-binary-must-be-built-from an unreadable c.c yields no fingerprint, not a wrong one never -harness_selftest 340-the-binary-must-be-built-from an unreadable library is not a failure never -harness_selftest 340-the-binary-must-be-built-from and a non-numeric timestamp is unknown rather than compared as text never -harness_selftest 340-the-binary-must-be-built-from and an uncomputable current fingerprint is unknown, not stale never -harness_selftest 340-the-binary-must-be-built-from and comparing two manifests names it rather than saying 'changed' never -harness_selftest 340-the-binary-must-be-built-from and it says so rather than staying silent never -harness_selftest 340-the-binary-must-be-built-from and removing it restores the fingerprint never -harness_selftest 340-the-binary-must-be-built-from and restoring it restores the fingerprint never -harness_selftest 340-the-binary-must-be-built-from and restoring the partition restores the fingerprint never -harness_selftest 340-the-binary-must-be-built-from and the major is still readable in the name never -harness_selftest 340-the-binary-must-be-built-from and the reader reads back the fingerprint the writer recorded never -harness_selftest 340-the-binary-must-be-built-from and the refusal says the server must be restarted never -harness_selftest 340-the-binary-must-be-built-from and two pg_configs for one prefix share a stamp, keyed on pkglibdir never -harness_selftest 340-the-binary-must-be-built-from but it says which question went unanswered never -harness_selftest 340-the-binary-must-be-built-from control: a caller passing a pg_config is not flagged never -harness_selftest 340-the-binary-must-be-built-from control: a readable run still reads fresh never -harness_selftest 340-the-binary-must-be-built-from control: a real content change still moves the fingerprint never -harness_selftest 340-the-binary-must-be-built-from control: a real src directory is still hashed never -harness_selftest 340-the-binary-must-be-built-from control: and it still succeeds on a writable one never -harness_selftest 340-the-binary-must-be-built-from control: and restoring the content restores the fingerprint never -harness_selftest 340-the-binary-must-be-built-from control: and the tree fingerprints again once it is readable never -harness_selftest 340-the-binary-must-be-built-from control: the same pg_config twice gives the same path never -harness_selftest 340-the-binary-must-be-built-from control: writing the value it was given never -harness_selftest 340-the-binary-must-be-built-from each manifest line is a tree-relative path and a digest never -harness_selftest 340-the-binary-must-be-built-from editing a source file moves the fingerprint never -harness_selftest 340-the-binary-must-be-built-from every directory the Makefile builds from is in the fingerprint never -harness_selftest 340-the-binary-must-be-built-from moving bytes between files moves the fingerprint never -harness_selftest 340-the-binary-must-be-built-from no caller passes a major where a pg_config belongs never -harness_selftest 340-the-binary-must-be-built-from no record at all is unknown, not fresh never -harness_selftest 340-the-binary-must-be-built-from one tree, one fingerprint, whatever the locale never -harness_selftest 340-the-binary-must-be-built-from premise: and the stamp really was not written, so the arm is not vacuous never -harness_selftest 340-the-binary-must-be-built-from premise: at least two locales are installed to compare never -harness_selftest 340-the-binary-must-be-built-from premise: both fake configs report the same major, which is the whole point never -harness_selftest 340-the-binary-must-be-built-from premise: every locale produced a fingerprint never -harness_selftest 340-the-binary-must-be-built-from premise: the argument parser reads the second argument at all never -harness_selftest 340-the-binary-must-be-built-from premise: the build path ran to completion, so a stamp was due never -harness_selftest 340-the-binary-must-be-built-from premise: the partition fixture fingerprints at all never -harness_selftest 340-the-binary-must-be-built-from premise: the same function returns a fingerprint for a real tree never -harness_selftest 340-the-binary-must-be-built-from premise: the spelling fixture fingerprints at all never -harness_selftest 340-the-binary-must-be-built-from premise: the sweep finds the call sites it is meant to police never -harness_selftest 340-the-binary-must-be-built-from premise: the tree fingerprints to something when it is readable never -harness_selftest 340-the-binary-must-be-built-from premise: the unprivileged read agrees while everything is readable never -harness_selftest 340-the-binary-must-be-built-from premise: the writer wrote a stamp at all never -harness_selftest 340-the-binary-must-be-built-from renaming a source file moves the fingerprint too never -harness_selftest 340-the-binary-must-be-built-from so the tree still fingerprints from its root files alone never -harness_selftest 340-the-binary-must-be-built-from so the verdict is fresh, not unknown never -harness_selftest 340-the-binary-must-be-built-from so the verdict is unknown -- UNVERIFIED -- and never stale never -harness_selftest 340-the-binary-must-be-built-from the fingerprint is the hash of the manifest never -harness_selftest 340-the-binary-must-be-built-from the fixed fingerprint equals what the previous implementation produced never -harness_selftest 340-the-binary-must-be-built-from the manifest is tree-relative, never absolute never -harness_selftest 340-the-binary-must-be-built-from the manifest names every file the fingerprint hashes never -harness_selftest 340-the-binary-must-be-built-from the probe is written outside the live source tree never -harness_selftest 340-the-binary-must-be-built-from the report names each hashed file never -harness_selftest 340-the-binary-must-be-built-from the report states how many files it hashed never -harness_selftest 340-the-binary-must-be-built-from the same tree fingerprints the same twice never -harness_selftest 340-the-binary-must-be-built-from the stamp writer reports failure on an unwritable target never -harness_selftest 340-the-binary-must-be-built-from the writer writes the file the reader looks for never -harness_selftest 340-the-binary-must-be-built-from two installations of one major get different stamp paths never -harness_selftest 340-the-binary-must-be-built-from two unreadable pg_configs do not alias onto one stamp never -harness_selftest 350-the-pytest-corpus-must-be README.md quotes the number of modes the inventory names as refused never -harness_selftest 350-the-pytest-corpus-must-be TESTS.md states no totals line for a merge to get wrong never -harness_selftest 350-the-pytest-corpus-must-be TESTS.md states the counted number as well never -harness_selftest 350-the-pytest-corpus-must-be a documented file that does not exist is named never -harness_selftest 350-the-pytest-corpus-must-be a documented test that does not exist is named, not passed over never -harness_selftest 350-the-pytest-corpus-must-be a name defined in two files is named, not passed over never -harness_selftest 350-the-pytest-corpus-must-be a prose total that disagrees with the ids is visible never -harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with disk is visible never -harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with the ids is visible never -harness_selftest 350-the-pytest-corpus-must-be an absent prose total is empty rather than a stray number never -harness_selftest 350-the-pytest-corpus-must-be an absent total is empty rather than a number that happens to match never -harness_selftest 350-the-pytest-corpus-must-be an id named twice counts once never -harness_selftest 350-the-pytest-corpus-must-be an id of fewer than three words is not counted as a mode never -harness_selftest 350-the-pytest-corpus-must-be an unbackticked name in prose is not treated as a claim never -harness_selftest 350-the-pytest-corpus-must-be an undocumented file is caught along with the tests inside it never -harness_selftest 350-the-pytest-corpus-must-be an undocumented test is named rather than passed over never -harness_selftest 350-the-pytest-corpus-must-be and the same comparison agrees on the fixture that is right never -harness_selftest 350-the-pytest-corpus-must-be control: a document naming only what exists is clean never -harness_selftest 350-the-pytest-corpus-must-be control: a fully documented corpus reports nothing missing never -harness_selftest 350-the-pytest-corpus-must-be control: distinct names in the same corpus report no duplicate never -harness_selftest 350-the-pytest-corpus-must-be every test file and every test in the corpus is named in TESTS.md never -harness_selftest 350-the-pytest-corpus-must-be every test the document names exists in the corpus never -harness_selftest 350-the-pytest-corpus-must-be no test name is defined twice in the corpus never -harness_selftest 350-the-pytest-corpus-must-be premise: the corpus carries the documentation this part polices never -harness_selftest 350-the-pytest-corpus-must-be premise: the counting rule finds modes at all never -harness_selftest 350-the-pytest-corpus-must-be premise: the mode inventory is where this part thinks it is never -harness_selftest 350-the-pytest-corpus-must-be premise: the pytest corpus is where this part thinks it is never -harness_selftest 350-the-pytest-corpus-must-be premise: the reader still finds a totals line when one is there never -harness_selftest 350-the-pytest-corpus-must-be premise: the reverse sweep reads backticked names at all never -harness_selftest 350-the-pytest-corpus-must-be premise: the sweep found the corpus rather than an empty glob never -harness_selftest 350-the-pytest-corpus-must-be section 1a's document total is the sum of its two sections never -harness_selftest 350-the-pytest-corpus-must-be section 1a's not-refused total is the count of ids in section 3 never -harness_selftest 350-the-pytest-corpus-must-be section 1a's refused total is the count of ids in section 2 never -harness_selftest 350-the-pytest-corpus-must-be section 2's opening states the counted number of refused modes never -harness_selftest 350-the-pytest-corpus-must-be the admitted gap is the run total minus what is written down never -harness_selftest 350-the-pytest-corpus-must-be the closing paragraph states the counted number too never -harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 2 never -harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 3 never -harness_selftest 350-the-pytest-corpus-must-be the counter stops at the next heading never -harness_selftest 350-the-pytest-corpus-must-be the row's value is read, not a digit inside its label never -harness_selftest 350-the-pytest-corpus-must-be the sweep counts the fixture's tests and files never -harness_selftest 360-an-unrunnable-pytest-test-must a comparison on the exit status is not counted as an assignment never -harness_selftest 360-an-unrunnable-pytest-test-must a drifted exit code is visible rather than absorbed never -harness_selftest 360-an-unrunnable-pytest-test-must a write-only unrunnable field is caught never -harness_selftest 360-an-unrunnable-pytest-test-must an unconditional exit override is caught by the dominance arm never -harness_selftest 360-an-unrunnable-pytest-test-must and only ever moves a run off zero, so a failure still dominates never -harness_selftest 360-an-unrunnable-pytest-test-must and something READS it, rather than only writing it never -harness_selftest 360-an-unrunnable-pytest-test-must premise: and that same fixture does show the write, so the arm is not blind never -harness_selftest 360-an-unrunnable-pytest-test-must premise: lib.sh states an INCOMPLETE exit code this part could read never -harness_selftest 360-an-unrunnable-pytest-test-must premise: the harness library is where this part thinks it is never -harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer is where this part thinks it is never -harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer states one too never -harness_selftest 360-an-unrunnable-pytest-test-must premise: while a real assignment on the same line shape IS counted never -harness_selftest 360-an-unrunnable-pytest-test-must premise: while the real layer satisfies that same arm never -harness_selftest 360-an-unrunnable-pytest-test-must the layer ends a session by setting its exit status never -harness_selftest 360-an-unrunnable-pytest-test-must the layer prints the unrunnable reason in lib.sh's shape never -harness_selftest 360-an-unrunnable-pytest-test-must the layer still writes the unrunnable state never -harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses agree on the INCOMPLETE exit code never -harness_selftest 370-the-plan-marker-guard-must a neutered absent arm is caught never -harness_selftest 370-the-plan-marker-guard-must a neutered empty-plan refusal is caught never -harness_selftest 370-the-plan-marker-guard-must a neutered present arm is caught never -harness_selftest 370-the-plan-marker-guard-must and that refusal is a VacuityError, not an ordinary assertion never -harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is absent never -harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is present never -harness_selftest 370-the-plan-marker-guard-must plan_marker refuses a plan with no nodes at all never -harness_selftest 370-the-plan-marker-guard-must premise: both line numbers were found, so the ordering arm can mean something never -harness_selftest 370-the-plan-marker-guard-must premise: plan_marker's body was actually cut out of the file never -harness_selftest 370-the-plan-marker-guard-must premise: the pytest layer is where this part thinks it is never -harness_selftest 370-the-plan-marker-guard-must premise: while the real body satisfies all three, so the greps work never -harness_selftest 370-the-plan-marker-guard-must the empty-plan refusal precedes the arm it protects never -harness_selftest 380-the-pytest-cluster-helpers a caller that reimplements the digest is caught never -harness_selftest 380-the-pytest-cluster-helpers a fingerprint that reads src only is caught never -harness_selftest 380-the-pytest-cluster-helpers a make_cluster with no cleanup is caught never -harness_selftest 380-the-pytest-cluster-helpers an import from the pytest tree is caught never -harness_selftest 380-the-pytest-cluster-helpers and a hard-coded module list is caught by the name arm never -harness_selftest 380-the-pytest-cluster-helpers and it catches BaseException, so an interrupt cleans up too never -harness_selftest 380-the-pytest-cluster-helpers and it names no module directory, so it is a derivation and not a list never -harness_selftest 380-the-pytest-cluster-helpers and it stops a partially started cluster before removing the tree never -harness_selftest 380-the-pytest-cluster-helpers and no longer mixes in the bare filename never -harness_selftest 380-the-pytest-cluster-helpers and the original error is re-raised rather than swallowed never -harness_selftest 380-the-pytest-cluster-helpers and the shell keeps none either never -harness_selftest 380-the-pytest-cluster-helpers make_cluster removes its tree when setup raises never -harness_selftest 380-the-pytest-cluster-helpers premise: and the real helper still carries its cleanup never -harness_selftest 380-the-pytest-cluster-helpers premise: make_cluster's body was actually cut out of the file never -harness_selftest 380-the-pytest-cluster-helpers premise: the one fingerprint implementation is where this part thinks it is never -harness_selftest 380-the-pytest-cluster-helpers premise: the pytest cluster helper is where this part thinks it is never -harness_selftest 380-the-pytest-cluster-helpers premise: while the real module satisfies the derivation arm never -harness_selftest 380-the-pytest-cluster-helpers the fingerprint derives its build directories from a Makefile on disk never -harness_selftest 380-the-pytest-cluster-helpers the hash mixes in each file's path relative to the tree, not its name never -harness_selftest 380-the-pytest-cluster-helpers the module imports nothing from the pytest tree never -harness_selftest 380-the-pytest-cluster-helpers the pytest helper keeps no private fingerprint implementation never -harness_selftest 390-a-registered-suite-must-account a NEW unaccounted suite fails even while the known debt is excused never -harness_selftest 390-a-registered-suite-must-account a comment mentioning pgc_summary is not a declaration never -harness_selftest 390-a-registered-suite-must-account a declared suite that produced no accounting is caught never -harness_selftest 390-a-registered-suite-must-account a declared suite the driver never dispatched reconciles never -harness_selftest 390-a-registered-suite-must-account a file that does not exist is reported absent, not exempt never -harness_selftest 390-a-registered-suite-must-account a hash inside a word does not hide the call after it never -harness_selftest 390-a-registered-suite-must-account a log carrying lib.sh's accounting line is accounted never -harness_selftest 390-a-registered-suite-must-account a log carrying neither is not accounted never -harness_selftest 390-a-registered-suite-must-account a log claiming PASSED without the accounting line shows none never -harness_selftest 390-a-registered-suite-must-account a long suite that calls pgc_summary still declares accounting never -harness_selftest 390-a-registered-suite-must-account a longer name containing pgc_summary is not a declaration never -harness_selftest 390-a-registered-suite-must-account a passing log shows accounting never -harness_selftest 390-a-registered-suite-must-account a registered suite that is accounted by nothing FAILS never -harness_selftest 390-a-registered-suite-must-account a suite recorded as known debt passes never -harness_selftest 390-a-registered-suite-must-account a suite recorded as never dispatched that DID account is caught never -harness_selftest 390-a-registered-suite-must-account a suite that accounted passes never -harness_selftest 390-a-registered-suite-must-account a suite that calls pgc_summary declares accounting never -harness_selftest 390-a-registered-suite-must-account a suite that never calls it does not never -harness_selftest 390-a-registered-suite-must-account a suite that now accounts but is still listed as debt is reported never -harness_selftest 390-a-registered-suite-must-account a suite the driver never dispatched passes never -harness_selftest 390-a-registered-suite-must-account a trailing comment after the call does not hide it never -harness_selftest 390-a-registered-suite-must-account an absent log shows no accounting rather than erroring never -harness_selftest 390-a-registered-suite-must-account an accounting line that does not start its line is refused never -harness_selftest 390-a-registered-suite-must-account an indented comment is still a comment never -harness_selftest 390-a-registered-suite-must-account an undeclared suite that DID account is caught too never -harness_selftest 390-a-registered-suite-must-account and a failed population reconciliation fails the major never -harness_selftest 390-a-registered-suite-must-account and a failed reconciliation sets the per-major failure flag never -harness_selftest 390-a-registered-suite-must-account and a log carrying only its OWN checks-run line is accounted too never -harness_selftest 390-a-registered-suite-must-account and a reworded producer line is refused, so the arm can fail never -harness_selftest 390-a-registered-suite-must-account and a skip, which reached the summary and counted zero never -harness_selftest 390-a-registered-suite-must-account and absent is distinguishable from a present file that does not declare never -harness_selftest 390-a-registered-suite-must-account and an incomplete never -harness_selftest 390-a-registered-suite-must-account and debt naming a suite that is not registered is reported too never -harness_selftest 390-a-registered-suite-must-account and it agrees with the real reader on a SHORT file, which is why it survived review never -harness_selftest 390-a-registered-suite-must-account and it is NAMED, so the reader does not have to diff two lists never -harness_selftest 390-a-registered-suite-must-account and it is named as that fault, not as one of the other two never -harness_selftest 390-a-registered-suite-must-account and it is named as the opposite fault, not the same one never -harness_selftest 390-a-registered-suite-must-account and it is named, which the symmetry check could never do never -harness_selftest 390-a-registered-suite-must-account and prose containing the word does not count as the line never -harness_selftest 390-a-registered-suite-must-account and so does a failing one, which is the point never -harness_selftest 390-a-registered-suite-must-account and the excused one is not named as a failure never -harness_selftest 390-a-registered-suite-must-account and the reader answers no on it, which is the wrong answer the arm catches never -harness_selftest 390-a-registered-suite-must-account and the real function reconciles the same input, so the arm is not noise never -harness_selftest 390-a-registered-suite-must-account and the reconciliation is given that record never -harness_selftest 390-a-registered-suite-must-account and the registered file is written from the SUITES array itself never -harness_selftest 390-a-registered-suite-must-account and without that record the same run is still caught never -harness_selftest 390-a-registered-suite-must-account equal sets reconcile never -harness_selftest 390-a-registered-suite-must-account every registered suite has a file never -harness_selftest 390-a-registered-suite-must-account nor no for every one of them never -harness_selftest 390-a-registered-suite-must-account opposite errors do not cancel: both directions are reported never -harness_selftest 390-a-registered-suite-must-account premise: and produced exactly one accounting line to be read never -harness_selftest 390-a-registered-suite-must-account premise: and that count excludes the definition line, which mentions it never -harness_selftest 390-a-registered-suite-must-account premise: and the accounted reader that feeds it never -harness_selftest 390-a-registered-suite-must-account premise: and the real function still does never -harness_selftest 390-a-registered-suite-must-account premise: pipefail is on, which is the condition the bug needs never -harness_selftest 390-a-registered-suite-must-account premise: the declaration reader evalled out of the runner is callable never -harness_selftest 390-a-registered-suite-must-account premise: the drift changed the line the reader looks for never -harness_selftest 390-a-registered-suite-must-account premise: the fixture carries a well-formed accounting line, just indented never -harness_selftest 390-a-registered-suite-must-account premise: the fixture is long enough to lose the race never -harness_selftest 390-a-registered-suite-must-account premise: the fixture really does hide its call from the stripper never -harness_selftest 390-a-registered-suite-must-account premise: the mutation applied -- the twin no longer sorts its inputs never -harness_selftest 390-a-registered-suite-must-account premise: the observation reader evalled out of the runner is callable never -harness_selftest 390-a-registered-suite-must-account premise: the population reconciliation is callable never -harness_selftest 390-a-registered-suite-must-account premise: the real suite ran and reached its summary never -harness_selftest 390-a-registered-suite-must-account premise: the reconciliation evalled out of the runner is callable never -harness_selftest 390-a-registered-suite-must-account premise: the registered list is not empty, so the partition means something never -harness_selftest 390-a-registered-suite-must-account premise: the runner defines the declaration reader this part evals never -harness_selftest 390-a-registered-suite-must-account premise: the runner defines the observation reader this part evals never -harness_selftest 390-a-registered-suite-must-account premise: the runner defines the population reconciliation never -harness_selftest 390-a-registered-suite-must-account premise: the runner defines the reconciliation this part evals never -harness_selftest 390-a-registered-suite-must-account premise: the twin script was written and is runnable never -harness_selftest 390-a-registered-suite-must-account premise: the unsorted twin is callable never -harness_selftest 390-a-registered-suite-must-account the debt file is in the tree never -harness_selftest 390-a-registered-suite-must-account the grep -q shape is the one that gets this wrong under pipefail never -harness_selftest 390-a-registered-suite-must-account the identity catches comm reading unsorted input never -harness_selftest 390-a-registered-suite-must-account the partition over the real suite list adds up never -harness_selftest 390-a-registered-suite-must-account the population partitions, and prints inputs == sum(buckets) never -harness_selftest 390-a-registered-suite-must-account the reader accepts the line the producer actually emits never -harness_selftest 390-a-registered-suite-must-account the reader does not answer yes for every registered suite never -harness_selftest 390-a-registered-suite-must-account the reconciliation prints inputs == sum(buckets) never -harness_selftest 390-a-registered-suite-must-account the record cannot introduce a suite the source never declared never -harness_selftest 390-a-registered-suite-must-account the runner calls the population reconciliation never -harness_selftest 390-a-registered-suite-must-account the runner calls the reconciliation, not merely defines it never -harness_selftest 390-a-registered-suite-must-account the skip branch records the suite it did not dispatch never -harness_selftest 390-a-registered-suite-must-account the stripper hides no pgc_summary call in any registered suite never -harness_selftest 400-a-check-result-must-be-machine a bogus reason code records FAIL, not UNRUN never -harness_selftest 400-a-check-result-must-be-machine a failing check emits exactly one record never -harness_selftest 400-a-check-result-must-be-machine a failing check still prints its old line never -harness_selftest 400-a-check-result-must-be-machine a log that never stated a count is not silently accepted never -harness_selftest 400-a-check-result-must-be-machine a log whose records match its stated count reconciles never -harness_selftest 400-a-check-result-must-be-machine a log with fewer records than it claims is caught never -harness_selftest 400-a-check-result-must-be-machine a log with more records than it claims is caught too never -harness_selftest 400-a-check-result-must-be-machine a passing check emits exactly one record never -harness_selftest 400-a-check-result-must-be-machine a passing check still prints its old line never -harness_selftest 400-a-check-result-must-be-machine an unrunnable check emits exactly one record never -harness_selftest 400-a-check-result-must-be-machine an unrunnable check still prints its old line never -harness_selftest 400-a-check-result-must-be-machine and its name field is the check's name, spaces intact never -harness_selftest 400-a-check-result-must-be-machine and its verdict field says FAIL never -harness_selftest 400-a-check-result-must-be-machine and its verdict field says PASS never -harness_selftest 400-a-check-result-must-be-machine and its verdict field says UNRUN, which is neither of the other two never -harness_selftest 400-a-check-result-must-be-machine and records FAIL never -harness_selftest 400-a-check-result-must-be-machine and records FAIL, because nothing was compared never -harness_selftest 400-a-check-result-must-be-machine and records PASS when the ratio is inside the bound never -harness_selftest 400-a-check-result-must-be-machine and that place is pgc_record never -harness_selftest 400-a-check-result-must-be-machine and the REASON_CODE travels in the reason field, not in prose never -harness_selftest 400-a-check-result-must-be-machine and the two numbers are named, not just the verdict never -harness_selftest 400-a-check-result-must-be-machine check_num on a non-number emits one record never -harness_selftest 400-a-check-result-must-be-machine check_num's non-measurement line is unchanged never -harness_selftest 400-a-check-result-must-be-machine check_ratio on a non-number emits one record never -harness_selftest 400-a-check-result-must-be-machine check_ratio that forms a ratio emits one record never -harness_selftest 400-a-check-result-must-be-machine check_ratio with a zero side emits one record never -harness_selftest 400-a-check-result-must-be-machine check_text on an empty side emits one record never -harness_selftest 400-a-check-result-must-be-machine check_text's empty-side line is unchanged never -harness_selftest 400-a-check-result-must-be-machine lib.sh bumps PGC_CHECKS in exactly one place never -harness_selftest 400-a-check-result-must-be-machine pgc_fail emits one record never -harness_selftest 400-a-check-result-must-be-machine pgc_pass emits one record never -harness_selftest 400-a-check-result-must-be-machine premise: and that is this fragment, not the suite never -harness_selftest 400-a-check-result-must-be-machine premise: it is callable never -harness_selftest 400-a-check-result-must-be-machine premise: lib.sh is where the check helpers live never -harness_selftest 400-a-check-result-must-be-machine premise: the probe ran every helper shape once never -harness_selftest 400-a-check-result-must-be-machine the record count equals the counter the summary reports never -harness_selftest 400-a-check-result-must-be-machine the record names the part the check was asked from never -harness_selftest 400-a-check-result-must-be-machine the runner calls the record reconciliation, not merely defines it never -harness_selftest 400-a-check-result-must-be-machine the runner defines the record reconciliation never -harness_selftest 410-a-check-must-have-been-red a check merely added is not reported as a rename never -harness_selftest 410-a-check-must-have-been-red a check merely removed is not reported as a rename either never -harness_selftest 410-a-check-must-have-been-red a check observed red gains the date it was seen never -harness_selftest 410-a-check-must-have-been-red a check the ledger has never seen is refused, not absorbed never -harness_selftest 410-a-check-must-have-been-red a duplicated check name is reported by name never -harness_selftest 410-a-check-must-have-been-red a later green run does not erase an observation never -harness_selftest 410-a-check-must-have-been-red a merge that names its mutation records it against the check that reddened never -harness_selftest 410-a-check-must-have-been-red a name that appeared while another disappeared is reported as a rename never -harness_selftest 410-a-check-must-have-been-red a run whose debt is within budget passes the gate never -harness_selftest 410-a-check-must-have-been-red and a check that stayed green keeps its debt never -harness_selftest 410-a-check-must-have-been-red and a unique one is not never -harness_selftest 410-a-check-must-have-been-red and it is empty when nothing named a mutation never -harness_selftest 410-a-check-must-have-been-red and it is named, so the author knows which one never -harness_selftest 410-a-check-must-have-been-red and names the suite and the check, not just a count never -harness_selftest 410-a-check-must-have-been-red and not against one that stayed green never -harness_selftest 410-a-check-must-have-been-red and one over budget does not never -harness_selftest 410-a-check-must-have-been-red and records neither as ever having been red never -harness_selftest 410-a-check-must-have-been-red and the gate says which number was exceeded, by how much never -harness_selftest 410-a-check-must-have-been-red and the stable check is not reported never -harness_selftest 410-a-check-must-have-been-red every ledger row carries four fields, the fourth being the mutation never -harness_selftest 410-a-check-must-have-been-red merging a green run records both checks never -harness_selftest 410-a-check-must-have-been-red premise: the budget is a tracked file too never -harness_selftest 410-a-check-must-have-been-red premise: the check has history before the rename never -harness_selftest 410-a-check-must-have-been-red premise: the ledger is not empty, so the partition means something never -harness_selftest 410-a-check-must-have-been-red premise: the ledger itself is a tracked file, not a variable never -harness_selftest 410-a-check-must-have-been-red premise: the ledger tool exists never -harness_selftest 410-a-check-must-have-been-red the census reads a run's records never -harness_selftest 410-a-check-must-have-been-red the committed budget matches the committed ledger's debt never -harness_selftest 410-a-check-must-have-been-red the committed budget names both debts never -harness_selftest 410-a-check-must-have-been-red the ledger partitions into observed and never never -harness_selftest 410-a-check-must-have-been-red the two collapse to one row, which is the loss being reported never +harness_selftest 030-assertions nothing leaked into the squatter never - +harness_selftest 030-assertions pgc_port_free says the squatter's port is busy never - +harness_selftest 030-assertions squatter survived untouched never - +harness_selftest 030-assertions suite did not settle on the squatter's port never - +harness_selftest 030-assertions suite's cluster is its own never - +harness_selftest 030-assertions suite's own objects are visible to it never - +harness_selftest 040-the-detection-primitive-itself detection distinguishes it from ours never - +harness_selftest 040-the-detection-primitive-itself detection reports a foreign cluster's directory never - +harness_selftest 040-the-detection-primitive-itself guard accepts our own cluster never - +harness_selftest 040-the-detection-primitive-itself guard rejects a foreign cluster never - +harness_selftest 040-the-detection-primitive-itself premise: the runner answered --list-suites, so the two checks below mean something never - +harness_selftest 050-the-list-must-be-read-the a name after the array's closing paren is not read as a registered suite never - +harness_selftest 050-the-list-must-be-read-the and the mistake empties the whole array rather than appending to it never - +harness_selftest 050-the-list-must-be-read-the positive control: and it is a whole list, not one lucky line never - +harness_selftest 050-the-list-must-be-read-the positive control: the real runner's list is read, and contains isolation never - +harness_selftest 050-the-list-must-be-read-the premise: the fixture really does carry the stray name never - +harness_selftest 060-the-list-stays-sorted-which-is premise: C collation puts sort_status before sorted_projection never - +harness_selftest 060-the-list-stays-sorted-which-is the suite list is sorted in C order, so two new suites land in different places never - +harness_selftest 070-and-comm-s-two-inputs-must a file that uses comm pins the collation of every sort feeding it never - +harness_selftest 070-and-comm-s-two-inputs-must and a prefix of a registered name is not treated as registered never - +harness_selftest 070-and-comm-s-two-inputs-must every registered suite has a file never - +harness_selftest 070-and-comm-s-two-inputs-must every suite is registered in run_all_versions.sh never - +harness_selftest 070-and-comm-s-two-inputs-must negative control: and does not find one that is not never - +harness_selftest 070-and-comm-s-two-inputs-must positive control: the membership test finds a name that is registered never - +harness_selftest 070-and-comm-s-two-inputs-must premise: some suite still uses comm, or the check below is vacuous never - +harness_selftest 080-no-suite-pipes-a-captured-string and bench/ was in the scan, which is the hole this rule had never - +harness_selftest 080-no-suite-pipes-a-captured-string and does not cover the live one above it never - +harness_selftest 080-no-suite-pipes-a-captured-string and selftest/ was in the scan, which is the hole that reddened #923 never - +harness_selftest 080-no-suite-pipes-a-captured-string and the heredoc exemption covers the one inside the heredoc, not the other never - +harness_selftest 080-no-suite-pipes-a-captured-string and the scan examined the suites rather than finding nothing to read never - +harness_selftest 080-no-suite-pipes-a-captured-string control: piping a large string into grep -q reports a match as absent never - +harness_selftest 080-no-suite-pipes-a-captured-string no suite pipes a captured string into an early-exit reader never - +harness_selftest 080-no-suite-pipes-a-captured-string premise: and the exemption covers a minority of them, not the corpus never - +harness_selftest 080-no-suite-pipes-a-captured-string premise: and the line really does hold the reader it must not flag never - +harness_selftest 080-no-suite-pipes-a-captured-string premise: and the old echo/printf pattern did NOT catch it never - +harness_selftest 080-no-suite-pipes-a-captured-string premise: the heredoc exemption found heredoc lines to exempt never - +harness_selftest 080-no-suite-pipes-a-captured-string premise: the sweep read lines to classify never - +harness_selftest 080-no-suite-pipes-a-captured-string the sweep catches a producer that is neither echo nor printf never - +harness_selftest 080-no-suite-pipes-a-captured-string the sweep does not mistake the || operator for a pipe never - +harness_selftest 080-no-suite-pipes-a-captured-string the sweep's pattern sees both lines of the probe never - +harness_selftest 090-no-suite-hands-every-run-the no suite hands every run the same default port never - +harness_selftest 090-no-suite-hands-every-run-the no test picks a port from inside the ephemeral range never - +harness_selftest 100-the-assertions-that-refuse-an-empty check compares two empty strings and passes, which is why the rest exist never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_num accepts a decimal and a sign never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses a psql error message never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses an md5, which is why check_text exists never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses the word a yes/no check would produce never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_num refuses two empty strings never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_num still compares two real numbers never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_num still fails two unequal numbers never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio fails a ratio outside its bound never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio passes a ratio inside its bound never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio refuses a zero denominator rather than dividing by it never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio refuses a zero numerator, which is inside every bound never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_ratio refuses an empty measurement never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_text compares two md5 hashes, which check_num cannot never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_text refuses one empty side never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_text refuses two empty strings, where plain check passes never - +harness_selftest 100-the-assertions-that-refuse-an-empty check_text still fails two different strings never - +harness_selftest 100-the-assertions-that-refuse-an-empty pgc_require_tools fails on one that does not never - +harness_selftest 100-the-assertions-that-refuse-an-empty pgc_require_tools passes on tools that exist never - +harness_selftest 110-the-harness-must-say-which-binary pgc_setup reports the installed .so never - +harness_selftest 110-the-harness-must-say-which-binary the installed .so is the one this run built never - +harness_selftest 120-a-failing-suite-must-surface-the a failing suite names the first fatal event in its log never - +harness_selftest 120-a-failing-suite-must-surface-the premise: the 40-line tail is filler, not the marker never - +harness_selftest 120-a-failing-suite-must-surface-the premise: the sub-suite failed, so its summary ran never - +harness_selftest 130-the-sanitizer-subset-must-cover-the premise: at least one suite drives the C-level encoding selftest never - +harness_selftest 130-the-sanitizer-subset-must-cover-the premise: run_san.sh's default subset was found and is non-empty never - +harness_selftest 130-the-sanitizer-subset-must-cover-the the sanitizer subset runs every suite that drives the encoding selftest never - +harness_selftest 140-a-cluster-that-will-not-start and still matches a PANIC never - +harness_selftest 140-a-cluster-that-will-not-start and still matches a signal death never - +harness_selftest 140-a-cluster-that-will-not-start and still matches an AddressSanitizer report never - +harness_selftest 140-a-cluster-that-will-not-start but not a routine statement error never - +harness_selftest 140-a-cluster-that-will-not-start nor an ordinary log line never - +harness_selftest 140-a-cluster-that-will-not-start premise: the harness exposes its fatal pattern to be judged never - +harness_selftest 140-a-cluster-that-will-not-start the fatal pattern matches a library that will not load never - +harness_selftest 150-the-verdict-must-not-assert-a a mixed run reports both causes and neither as the whole story never - +harness_selftest 150-the-verdict-must-not-assert-a and is NOT made when our own postmaster died, which is the #537 case never - +harness_selftest 150-the-verdict-must-not-assert-a and so is the attempt count never - +harness_selftest 150-the-verdict-must-not-assert-a and the no-squatter verdict points at the server log never - +harness_selftest 150-the-verdict-must-not-assert-a premise: the verdict is composed somewhere it can be judged never - +harness_selftest 150-the-verdict-must-not-assert-a the ownership claim is made when a squatter held the port every time never - +harness_selftest 150-the-verdict-must-not-assert-a the port is named either way never - +harness_selftest 160-and-the-log-report-must-show and a log with no fatal line still reports rather than staying silent never - +harness_selftest 160-and-the-log-report-must-show premise: the log report is a function that can be fed a fixture never - +harness_selftest 160-and-the-log-report-must-show the report names the symbol that was actually missing never - +harness_selftest 170-and-lib-sh-must-ask-these and asks pgc_start_failure_message for the verdict never - +harness_selftest 170-and-lib-sh-must-ask-these and the old start-failure verdict is not echoed inline anywhere never - +harness_selftest 170-and-lib-sh-must-ask-these and the start path asks pgc_start_fatal_pattern, its deliberately wider one never - +harness_selftest 170-and-lib-sh-must-ask-these premise: lib.sh is readable, or every grep below approves nothing never - +harness_selftest 170-and-lib-sh-must-ask-these premise: the start-failure path still exists to be judged never - +harness_selftest 170-and-lib-sh-must-ask-these the failure path asks pgc_start_log_report for the reason never - +harness_selftest 170-and-lib-sh-must-ask-these the summary path asks pgc_fatal_pattern rather than hardcoding it never - +harness_selftest 180-the-port-walk-must-wrap-not a free port beyond the old 300-probe bound is still found never - +harness_selftest 180-the-port-walk-must-wrap-not a seed at the ceiling wraps past a busy top and still finds a port never - +harness_selftest 180-the-port-walk-must-wrap-not an entirely busy band reports itself full and terminates never - +harness_selftest 180-the-port-walk-must-wrap-not and the port it found is below the busy region, which is where wrapping lands never - +harness_selftest 180-the-port-walk-must-wrap-not premise: and really does allow the bottom never - +harness_selftest 180-the-port-walk-must-wrap-not premise: the auxiliary band has a width to wrap within never - +harness_selftest 180-the-port-walk-must-wrap-not premise: the real prober was restored, or every check after this lies never - +harness_selftest 180-the-port-walk-must-wrap-not premise: the stub frees exactly one port, 500 past the floor never - +harness_selftest 180-the-port-walk-must-wrap-not premise: the stub really does refuse the top of the band never - +harness_selftest 190-an-in-tree-build-must-not an unknown provenance is not reported as a major never - +harness_selftest 190-an-in-tree-build-must-not an unparseable stamp cleans rather than guessing never - +harness_selftest 190-an-in-tree-build-must-not and an empty WANT is refused rather than compared never - +harness_selftest 190-an-in-tree-build-must-not and in the other direction too never - +harness_selftest 190-an-in-tree-build-must-not and it is 3 bytes, not an escaped literal never - +harness_selftest 190-an-in-tree-build-must-not and it says plainly that no major was recorded never - +harness_selftest 190-an-in-tree-build-must-not building a DIFFERENT major needs a clean, which is the #536 case never - +harness_selftest 190-an-in-tree-build-must-not building the same major again needs no clean never - +harness_selftest 190-an-in-tree-build-must-not but a tree with no objects at all needs nothing, stamp or not never - +harness_selftest 190-an-in-tree-build-must-not objects with NO stamp are unknown provenance and must be cleaned never - +harness_selftest 190-an-in-tree-build-must-not premise: the build-stamp decision is exposed to be judged never - +harness_selftest 190-an-in-tree-build-must-not premise: the stamp writer is a function that can be exercised never - +harness_selftest 190-an-in-tree-build-must-not the build path asks pgc_build_needs_clean rather than merely naming it never - +harness_selftest 190-an-in-tree-build-must-not the stamp lib.sh writes is exactly the major never - +harness_selftest 200-additions-go-in-their-own-file premise: the parts directory exists and was sourced never - +harness_selftest 200-additions-go-in-their-own-file the driver holds no checks; they all live in parts never - +harness_selftest 200-additions-go-in-their-own-file the driver sources the parts by glob, not by a list never - +harness_selftest 210-no-suite-assigns-a-bash-special control: reads, longer names, and the deliberate RANDOM/SECONDS seeds are not flagged never - +harness_selftest 210-no-suite-assigns-a-bash-special control: the sweep catches an assignment to a bash special never - +harness_selftest 210-no-suite-assigns-a-bash-special no suite assigns to a bash special variable never - +harness_selftest 220-an-opt-in-upgrade-guard-must CI runs the extension-upgrade guard somewhere (#741) never - +harness_selftest 220-an-opt-in-upgrade-guard-must every PGC_RUN_UPGRADE-gated suite is excluded from the coverage runner (#741) never - +harness_selftest 220-an-opt-in-upgrade-guard-must premise: both not_a_suite definitions were found never - +harness_selftest 220-an-opt-in-upgrade-guard-must premise: not_a_suite says no to an ordinary suite, so its yes means something never - +harness_selftest 220-an-opt-in-upgrade-guard-must premise: run_coverage.sh defines not_a_suite and calls it never - +harness_selftest 220-an-opt-in-upgrade-guard-must premise: the PGC_RUN_UPGRADE block was found and names at least one suite never - +harness_selftest 220-an-opt-in-upgrade-guard-must premise: the population is the real test directory, not an empty glob never - +harness_selftest 220-an-opt-in-upgrade-guard-must premise: there are workflow files to search never - +harness_selftest 220-an-opt-in-upgrade-guard-must the coverage runner's not_a_suite agrees with the selftest's, both ways (#741) never - +harness_selftest 230-a-suite-connecting-by-socket-must every suite that connects by socket path sets unix_socket_directories never - +harness_selftest 230-a-suite-connecting-by-socket-must premise: at least one suite connects by a socket path, so this is not vacuous never - +harness_selftest 230-a-suite-connecting-by-socket-must premise: no TCP suite is counted as a socket user never - +harness_selftest 240-the-nightly-enumeration-must-not every nightly job is named in docs/testing.md (#741) never - +harness_selftest 240-the-nightly-enumeration-must-not premise: at least three nightly jobs were parsed, so the list is real never - +harness_selftest 240-the-nightly-enumeration-must-not premise: the nightly paragraph was located and is not empty never - +harness_selftest 240-the-nightly-enumeration-must-not premise: the nightly workflow and the testing doc are both present never - +harness_selftest 250-the-coverage-runner-must-refuse GCOV_PREFIX is exported before the suites run (#740) never - +harness_selftest 250-the-coverage-runner-must-refuse premise: both stray-counter probes were located never - +harness_selftest 250-the-coverage-runner-must-refuse premise: both the counter refusal and the lcov capture were located never - +harness_selftest 250-the-coverage-runner-must-refuse premise: the containment and the copy were both located never - +harness_selftest 250-the-coverage-runner-must-refuse premise: the coverage runner is present and parses never - +harness_selftest 250-the-coverage-runner-must-refuse premise: the guard's count directory and the capture's were both located never - +harness_selftest 250-the-coverage-runner-must-refuse premise: the redirect, the suite invocation and the copy-back were located never - +harness_selftest 250-the-coverage-runner-must-refuse the copy-back refuses a destination outside the tree (#740) never - +harness_selftest 250-the-coverage-runner-must-refuse the counters are returned beside their objects before the refusal (#740) never - +harness_selftest 250-the-coverage-runner-must-refuse the coverage runner refuses zero counters before it calls lcov (#740) never - +harness_selftest 250-the-coverage-runner-must-refuse the refusal looks in GCOV_PREFIX before the tree-wide walk (#740) never - +harness_selftest 250-the-coverage-runner-must-refuse the zero-counter guard counts the directory lcov captures (#740) never - +harness_selftest 260-an-ordered-comparison-must-use-the and it is the same suites, not merely the same count never - +harness_selftest 260-an-ordered-comparison-must-use-the every diff_query_ordered site actually names an ORDER BY never - +harness_selftest 260-an-ordered-comparison-must-use-the every suite using the ordered oracle asserts its premise never - +harness_selftest 260-an-ordered-comparison-must-use-the no diff_query site names an ORDER BY it cannot test (use diff_query_ordered) never - +harness_selftest 260-an-ordered-comparison-must-use-the premise: both comparison helpers are present never - +harness_selftest 260-an-ordered-comparison-must-use-the premise: both oracles are present never - +harness_selftest 260-an-ordered-comparison-must-use-the premise: some suite uses the ordered oracle, or the next check is vacuous never - +harness_selftest 260-an-ordered-comparison-must-use-the premise: the tree really contains continued diff_query calls to join never - +harness_selftest 260-an-ordered-comparison-must-use-the sorted_projection's two comparisons are ordered, its subject being order never - +harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle keeps the empty-result sentinel never - +harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle keeps the unique query-error sentinel never - +harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle numbers the rows as they arrive never - +harness_selftest 260-an-ordered-comparison-must-use-the the ordered oracle orders by row_number, so it keeps the query's order never - +harness_selftest 260-an-ordered-comparison-must-use-the the set oracle orders by the rendered row, so it is order-blind never - +harness_selftest 270-a-set-options-call-must-use-values no suite calls set_options with a value it will reject never - +harness_selftest 270-a-set-options-call-must-use-values premise: and does NOT fire when the error is the point, across a continuation never - +harness_selftest 270-a-set-options-call-must-use-values premise: every file containing a set_options call is in the sweep never - +harness_selftest 270-a-set-options-call-must-use-values premise: the detector fires on an out-of-range value that is NOT expect_error never - +harness_selftest 270-a-set-options-call-must-use-values premise: the set_options sweep read a substantial number of calls never - +harness_selftest 280-the-shared-cluster-config-must-not premise: and it is the right block (it sets the port and the preload) never - +harness_selftest 280-the-shared-cluster-config-must-not premise: the cluster-config block was located in lib.sh never - +harness_selftest 280-the-shared-cluster-config-must-not premise: the detector fires on the line that caused #799 never - +harness_selftest 280-the-shared-cluster-config-must-not the per-suite escape hatch PGC_EXTRA_CONF is still applied to the config never - +harness_selftest 280-the-shared-cluster-config-must-not the shared cluster config sets no pgcolumnar.* GUC never - +harness_selftest 290-a-preflight-that-built-nothing a preflight that built nothing does not report PASSED never - +harness_selftest 290-a-preflight-that-built-nothing a preflight that built nothing exits non-zero never - +harness_selftest 290-a-preflight-that-built-nothing a preflight that built nothing says how many it built never - +harness_selftest 290-a-preflight-that-built-nothing premise: and it built none of them never - +harness_selftest 290-a-preflight-that-built-nothing premise: the probe run skipped every major never - +harness_selftest 300-a-test-script-must-be-runnable and every executable script declares one never - +harness_selftest 300-a-test-script-must-be-runnable and every script a document names exists never - +harness_selftest 300-a-test-script-must-be-runnable and every script a document names is executable never - +harness_selftest 300-a-test-script-must-be-runnable control: an interpreter declared without the bit is caught never - +harness_selftest 300-a-test-script-must-be-runnable control: and a sourced fragment, with neither, is correct never - +harness_selftest 300-a-test-script-must-be-runnable control: and the same file with the bit is not never - +harness_selftest 300-a-test-script-must-be-runnable control: cp -a preserves the execute bit, so a staged tree reads the same never - +harness_selftest 300-a-test-script-must-be-runnable control: the bit without an interpreter is caught too never - +harness_selftest 300-a-test-script-must-be-runnable every script that declares an interpreter is executable never - +harness_selftest 300-a-test-script-must-be-runnable premise: a runnable script one level down is inside the population never - +harness_selftest 300-a-test-script-must-be-runnable premise: and bench/ is in the population never - +harness_selftest 300-a-test-script-must-be-runnable premise: and so are the fixture host tools never - +harness_selftest 300-a-test-script-must-be-runnable premise: and they name at least one command in every swept directory never - +harness_selftest 300-a-test-script-must-be-runnable premise: the documents name a population of commands, not none never - +harness_selftest 300-a-test-script-must-be-runnable premise: the sourced parts are inside the population, not pruned never - +harness_selftest 300-a-test-script-must-be-runnable premise: the sweep reads a population of scripts, not an empty find never - +harness_selftest 310-a-compiled-artifact-must-not-be and a compiled artifact written beside its source never - +harness_selftest 310-a-compiled-artifact-must-not-be and the tracked list names none of them never - +harness_selftest 310-a-compiled-artifact-must-not-be and the tree ignores the directory Python writes them to never - +harness_selftest 310-a-compiled-artifact-must-not-be no compiled Python artifact is tracked never - +harness_selftest 310-a-compiled-artifact-must-not-be premise: and git ls-files sees the harness it is being asked about never - +harness_selftest 310-a-compiled-artifact-must-not-be premise: and that a tracked source file is not never - +harness_selftest 310-a-compiled-artifact-must-not-be premise: check-ignore agrees a build object is already ignored never - +harness_selftest 310-a-compiled-artifact-must-not-be premise: the source tree is a git checkout never - +harness_selftest 320-a-check-that-could-not-run 67 without its line is a failure, not an INCOMPLETE taken on trust never - +harness_selftest 320-a-check-that-could-not-run a counter that drifts is caught rather than absorbed never - +harness_selftest 320-a-check-that-could-not-run a failure outranks an unrunnable check, and both are still counted never - +harness_selftest 320-a-check-that-could-not-run a passing ratio check is counted as a pass, not a failure never - +harness_selftest 320-a-check-that-could-not-run a suite of nothing but unrunnable checks is INCOMPLETE, not SKIPPED never - +harness_selftest 320-a-check-that-could-not-run a suite whose checks all passed still exits 0 PASSED never - +harness_selftest 320-a-check-that-could-not-run a suite with none says so as zero rather than staying silent never - +harness_selftest 320-a-check-that-could-not-run an INCOMPLETE suite fails its major never - +harness_selftest 320-a-check-that-could-not-run an unrunnable check counts toward checks run never - +harness_selftest 320-a-check-that-could-not-run an unrunnable reason outside the enum fails rather than being accepted never - +harness_selftest 320-a-check-that-could-not-run and 66 with its line a skip never - +harness_selftest 320-a-check-that-could-not-run and 67 with its line INCOMPLETE, which is not a pass never - +harness_selftest 320-a-check-that-could-not-run and a failing suite still does never - +harness_selftest 320-a-check-that-could-not-run and a skip does not, which is the one that must stay true never - +harness_selftest 320-a-check-that-could-not-run and a suite with no unrunnable checks reconciles too never - +harness_selftest 320-a-check-that-could-not-run and allows one that does, which is what it was written to allow never - +harness_selftest 320-a-check-that-could-not-run and an ordinary failure is still a failure never - +harness_selftest 320-a-check-that-could-not-run and both exempt a file that keeps its own counter without lib.sh never - +harness_selftest 320-a-check-that-could-not-run and it is not reported as having run no checks never - +harness_selftest 320-a-check-that-could-not-run and no write-only failure flag survives in the runner never - +harness_selftest 320-a-check-that-could-not-run and the suite holding it fails rather than reporting PASSED never - +harness_selftest 320-a-check-that-could-not-run and the suite that holds it still passes never - +harness_selftest 320-a-check-that-could-not-run and the unrunnable ones are reported as their own count never - +harness_selftest 320-a-check-that-could-not-run every direct write to PGC_CHECKS records an outcome too never - +harness_selftest 320-a-check-that-could-not-run lib.sh defines check_unrunnable never - +harness_selftest 320-a-check-that-could-not-run lib.sh defines the INCOMPLETE exit status never - +harness_selftest 320-a-check-that-could-not-run no non-zero status is classified as a pass never - +harness_selftest 320-a-check-that-could-not-run no suite that uses lib.sh's accounting writes PGC_CHECKS directly never - +harness_selftest 320-a-check-that-could-not-run one unrunnable check makes the suite INCOMPLETE, not passed never - +harness_selftest 320-a-check-that-could-not-run premise: the classifier evalled out of the runner is callable never - +harness_selftest 320-a-check-that-could-not-run premise: the fixtures carry the shapes these rules are about never - +harness_selftest 320-a-check-that-could-not-run premise: the harness library is where this part thinks it is never - +harness_selftest 320-a-check-that-could-not-run premise: the major-verdict mapping evalled out of the runner is callable never - +harness_selftest 320-a-check-that-could-not-run premise: the runner defines the classifier this part is about to eval never - +harness_selftest 320-a-check-that-could-not-run premise: the sweep read the corpus and found sites to classify never - +harness_selftest 320-a-check-that-could-not-run the original rule flags a bump that records no outcome never - +harness_selftest 320-a-check-that-could-not-run the runner calls a clean exit a pass never - +harness_selftest 320-a-check-that-could-not-run the runner's INCOMPLETE branch calls the mapping rather than a local flag never - +harness_selftest 320-a-check-that-could-not-run the stronger rule flags that same allowed bump, which is the change never - +harness_selftest 320-a-check-that-could-not-run the summary reconciles the three states against the total never - +harness_selftest 320-a-check-that-could-not-run the unrunnable check names itself, its reason code and its detail never - +harness_selftest 320-a-check-that-could-not-run while a pass does not never - +harness_selftest 330-the-incomplete-path-must-run-whole a suite with an unrunnable check exits 67 never - +harness_selftest 330-the-incomplete-path-must-run-whole an INCOMPLETE suite sets the flag the major verdict actually reads never - +harness_selftest 330-the-incomplete-path-must-run-whole and appears in the results string as INCOMPLETE never - +harness_selftest 330-the-incomplete-path-must-run-whole and counted as incomplete, so the tally can say so never - +harness_selftest 330-the-incomplete-path-must-run-whole and counts both suites as having run never - +harness_selftest 330-the-incomplete-path-must-run-whole and exactly one of them as incomplete never - +harness_selftest 330-the-incomplete-path-must-run-whole and is counted as having run never - +harness_selftest 330-the-incomplete-path-must-run-whole and is not counted as skipped, nor is the skip count disturbed never - +harness_selftest 330-the-incomplete-path-must-run-whole and its log carries the INCOMPLETE line the classifier needs never - +harness_selftest 330-the-incomplete-path-must-run-whole and neither as skipped never - +harness_selftest 330-the-incomplete-path-must-run-whole and no longer counts incompletes inline beside it never - +harness_selftest 330-the-incomplete-path-must-run-whole and records each suite's own verdict in the results string never - +harness_selftest 330-the-incomplete-path-must-run-whole and reprints the suite's own UNRUN line beneath it never - +harness_selftest 330-the-incomplete-path-must-run-whole and the UNRUN line the runner prints into the matrix output never - +harness_selftest 330-the-incomplete-path-must-run-whole and the run's overall status is failure never - +harness_selftest 330-the-incomplete-path-must-run-whole and the summary line carries the incomplete count a reader needs never - +harness_selftest 330-the-incomplete-path-must-run-whole and the tally announces it, with the reason lifted from the log never - +harness_selftest 330-the-incomplete-path-must-run-whole control fixture: a suite whose checks all ran exits 0 never - +harness_selftest 330-the-incomplete-path-must-run-whole control: and leaves the run's overall status alone never - +harness_selftest 330-the-incomplete-path-must-run-whole control: and still announces it never - +harness_selftest 330-the-incomplete-path-must-run-whole control: and still records that it ran, and how never - +harness_selftest 330-the-incomplete-path-must-run-whole control: and the major reports PASS never - +harness_selftest 330-the-incomplete-path-must-run-whole control: the same loop leaves a passing suite passing never - +harness_selftest 330-the-incomplete-path-must-run-whole premise: all three runner functions were extracted, not empty ranges never - +harness_selftest 330-the-incomplete-path-must-run-whole premise: and all three are callable never - +harness_selftest 330-the-incomplete-path-must-run-whole premise: and each extraction ends at its own closing brace never - +harness_selftest 330-the-incomplete-path-must-run-whole premise: the major-verdict branch was extracted, not an empty range never - +harness_selftest 330-the-incomplete-path-must-run-whole premise: the runner's collect loop was extracted, not an empty range never - +harness_selftest 330-the-incomplete-path-must-run-whole premise: the selftest has a workdir to build fixtures in never - +harness_selftest 330-the-incomplete-path-must-run-whole running the real loop over both fixtures fails the major never - +harness_selftest 330-the-incomplete-path-must-run-whole the loop delegates each verdict to pgc_tally_suite never - +harness_selftest 330-the-incomplete-path-must-run-whole the runner classifies the file that suite actually produced never - +harness_selftest 330-the-incomplete-path-must-run-whole with an incomplete suite in the tally the major reports FAIL never - +harness_selftest 340-the-binary-must-be-built-from PREMISE and the target really holds sources find would otherwise hash never - +harness_selftest 340-the-binary-must-be-built-from PREMISE the Makefile's recursion was actually parsed never - +harness_selftest 340-the-binary-must-be-built-from PREMISE the copy discovers the same build directories as the real tree never - +harness_selftest 340-the-binary-must-be-built-from PREMISE the fingerprint covers at least src never - +harness_selftest 340-the-binary-must-be-built-from PREMISE the fixture's src really is a symlink never - +harness_selftest 340-the-binary-must-be-built-from a /./ segment hashes the same tree the same way never - +harness_selftest 340-the-binary-must-be-built-from a /src/.. segment hashes the same tree the same way never - +harness_selftest 340-the-binary-must-be-built-from a caller passing a major is caught never - +harness_selftest 340-the-binary-must-be-built-from a file that is not a build input does not move it never - +harness_selftest 340-the-binary-must-be-built-from a fingerprint different from the record is stale never - +harness_selftest 340-the-binary-must-be-built-from a fingerprint equal to the record is fresh never - +harness_selftest 340-the-binary-must-be-built-from a fingerprint is 12 hex characters never - +harness_selftest 340-the-binary-must-be-built-from a library newer than the running server is REFUSED never - +harness_selftest 340-the-binary-must-be-built-from a library older than the running server is accepted never - +harness_selftest 340-the-binary-must-be-built-from a missing binary timestamp is unknown, not predates never - +harness_selftest 340-the-binary-must-be-built-from a missing postmaster timestamp is unknown, not predates never - +harness_selftest 340-the-binary-must-be-built-from a new source file under objstore moves the fingerprint never - +harness_selftest 340-the-binary-must-be-built-from a relative path hashes the same tree the same way never - +harness_selftest 340-the-binary-must-be-built-from a server older than the binary predates it never - +harness_selftest 340-the-binary-must-be-built-from a server started after the binary is fresh never - +harness_selftest 340-the-binary-must-be-built-from a server started at the same second is fresh never - +harness_selftest 340-the-binary-must-be-built-from a symlink to the tree hashes it the same way never - +harness_selftest 340-the-binary-must-be-built-from a symlinked src contributes nothing, as find -P contributes nothing never - +harness_selftest 340-the-binary-must-be-built-from a trailing slash hashes the same tree the same way never - +harness_selftest 340-the-binary-must-be-built-from a tree with no hashable file yields no fingerprint never - +harness_selftest 340-the-binary-must-be-built-from adding a source file moves it never - +harness_selftest 340-the-binary-must-be-built-from an added file appears in the manifest by name never - +harness_selftest 340-the-binary-must-be-built-from an added file shows up in the report never - +harness_selftest 340-the-binary-must-be-built-from an empty manifest is reported as empty, not as silence never - +harness_selftest 340-the-binary-must-be-built-from an unhashable tree has an empty manifest never - +harness_selftest 340-the-binary-must-be-built-from an unreadable b.c yields no fingerprint, not a wrong one never - +harness_selftest 340-the-binary-must-be-built-from an unreadable c.c yields no fingerprint, not a wrong one never - +harness_selftest 340-the-binary-must-be-built-from an unreadable library is not a failure never - +harness_selftest 340-the-binary-must-be-built-from and a non-numeric timestamp is unknown rather than compared as text never - +harness_selftest 340-the-binary-must-be-built-from and an uncomputable current fingerprint is unknown, not stale never - +harness_selftest 340-the-binary-must-be-built-from and comparing two manifests names it rather than saying 'changed' never - +harness_selftest 340-the-binary-must-be-built-from and it says so rather than staying silent never - +harness_selftest 340-the-binary-must-be-built-from and removing it restores the fingerprint never - +harness_selftest 340-the-binary-must-be-built-from and restoring it restores the fingerprint never - +harness_selftest 340-the-binary-must-be-built-from and restoring the partition restores the fingerprint never - +harness_selftest 340-the-binary-must-be-built-from and the major is still readable in the name never - +harness_selftest 340-the-binary-must-be-built-from and the reader reads back the fingerprint the writer recorded never - +harness_selftest 340-the-binary-must-be-built-from and the refusal says the server must be restarted never - +harness_selftest 340-the-binary-must-be-built-from and two pg_configs for one prefix share a stamp, keyed on pkglibdir never - +harness_selftest 340-the-binary-must-be-built-from but it says which question went unanswered never - +harness_selftest 340-the-binary-must-be-built-from control: a caller passing a pg_config is not flagged never - +harness_selftest 340-the-binary-must-be-built-from control: a readable run still reads fresh never - +harness_selftest 340-the-binary-must-be-built-from control: a real content change still moves the fingerprint never - +harness_selftest 340-the-binary-must-be-built-from control: a real src directory is still hashed never - +harness_selftest 340-the-binary-must-be-built-from control: and it still succeeds on a writable one never - +harness_selftest 340-the-binary-must-be-built-from control: and restoring the content restores the fingerprint never - +harness_selftest 340-the-binary-must-be-built-from control: and the tree fingerprints again once it is readable never - +harness_selftest 340-the-binary-must-be-built-from control: the same pg_config twice gives the same path never - +harness_selftest 340-the-binary-must-be-built-from control: writing the value it was given never - +harness_selftest 340-the-binary-must-be-built-from each manifest line is a tree-relative path and a digest never - +harness_selftest 340-the-binary-must-be-built-from editing a source file moves the fingerprint never - +harness_selftest 340-the-binary-must-be-built-from every directory the Makefile builds from is in the fingerprint never - +harness_selftest 340-the-binary-must-be-built-from moving bytes between files moves the fingerprint never - +harness_selftest 340-the-binary-must-be-built-from no caller passes a major where a pg_config belongs never - +harness_selftest 340-the-binary-must-be-built-from no record at all is unknown, not fresh never - +harness_selftest 340-the-binary-must-be-built-from one tree, one fingerprint, whatever the locale never - +harness_selftest 340-the-binary-must-be-built-from premise: and the stamp really was not written, so the arm is not vacuous never - +harness_selftest 340-the-binary-must-be-built-from premise: at least two locales are installed to compare never - +harness_selftest 340-the-binary-must-be-built-from premise: both fake configs report the same major, which is the whole point never - +harness_selftest 340-the-binary-must-be-built-from premise: every locale produced a fingerprint never - +harness_selftest 340-the-binary-must-be-built-from premise: the argument parser reads the second argument at all never - +harness_selftest 340-the-binary-must-be-built-from premise: the build path ran to completion, so a stamp was due never - +harness_selftest 340-the-binary-must-be-built-from premise: the partition fixture fingerprints at all never - +harness_selftest 340-the-binary-must-be-built-from premise: the same function returns a fingerprint for a real tree never - +harness_selftest 340-the-binary-must-be-built-from premise: the spelling fixture fingerprints at all never - +harness_selftest 340-the-binary-must-be-built-from premise: the sweep finds the call sites it is meant to police never - +harness_selftest 340-the-binary-must-be-built-from premise: the tree fingerprints to something when it is readable never - +harness_selftest 340-the-binary-must-be-built-from premise: the unprivileged read agrees while everything is readable never - +harness_selftest 340-the-binary-must-be-built-from premise: the writer wrote a stamp at all never - +harness_selftest 340-the-binary-must-be-built-from renaming a source file moves the fingerprint too never - +harness_selftest 340-the-binary-must-be-built-from so the tree still fingerprints from its root files alone never - +harness_selftest 340-the-binary-must-be-built-from so the verdict is fresh, not unknown never - +harness_selftest 340-the-binary-must-be-built-from so the verdict is unknown -- UNVERIFIED -- and never stale never - +harness_selftest 340-the-binary-must-be-built-from the fingerprint is the hash of the manifest never - +harness_selftest 340-the-binary-must-be-built-from the fixed fingerprint equals what the previous implementation produced never - +harness_selftest 340-the-binary-must-be-built-from the manifest is tree-relative, never absolute never - +harness_selftest 340-the-binary-must-be-built-from the manifest names every file the fingerprint hashes never - +harness_selftest 340-the-binary-must-be-built-from the probe is written outside the live source tree never - +harness_selftest 340-the-binary-must-be-built-from the report names each hashed file never - +harness_selftest 340-the-binary-must-be-built-from the report states how many files it hashed never - +harness_selftest 340-the-binary-must-be-built-from the same tree fingerprints the same twice never - +harness_selftest 340-the-binary-must-be-built-from the stamp writer reports failure on an unwritable target never - +harness_selftest 340-the-binary-must-be-built-from the writer writes the file the reader looks for never - +harness_selftest 340-the-binary-must-be-built-from two installations of one major get different stamp paths never - +harness_selftest 340-the-binary-must-be-built-from two unreadable pg_configs do not alias onto one stamp never - +harness_selftest 350-the-pytest-corpus-must-be README.md quotes the number of modes the inventory names as refused never - +harness_selftest 350-the-pytest-corpus-must-be TESTS.md states no totals line for a merge to get wrong never - +harness_selftest 350-the-pytest-corpus-must-be TESTS.md states the counted number as well never - +harness_selftest 350-the-pytest-corpus-must-be a documented file that does not exist is named never - +harness_selftest 350-the-pytest-corpus-must-be a documented test that does not exist is named, not passed over never - +harness_selftest 350-the-pytest-corpus-must-be a name defined in two files is named, not passed over never - +harness_selftest 350-the-pytest-corpus-must-be a prose total that disagrees with the ids is visible never - +harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with disk is visible never - +harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with the ids is visible never - +harness_selftest 350-the-pytest-corpus-must-be an absent prose total is empty rather than a stray number never - +harness_selftest 350-the-pytest-corpus-must-be an absent total is empty rather than a number that happens to match never - +harness_selftest 350-the-pytest-corpus-must-be an id named twice counts once never - +harness_selftest 350-the-pytest-corpus-must-be an id of fewer than three words is not counted as a mode never - +harness_selftest 350-the-pytest-corpus-must-be an unbackticked name in prose is not treated as a claim never - +harness_selftest 350-the-pytest-corpus-must-be an undocumented file is caught along with the tests inside it never - +harness_selftest 350-the-pytest-corpus-must-be an undocumented test is named rather than passed over never - +harness_selftest 350-the-pytest-corpus-must-be and the same comparison agrees on the fixture that is right never - +harness_selftest 350-the-pytest-corpus-must-be control: a document naming only what exists is clean never - +harness_selftest 350-the-pytest-corpus-must-be control: a fully documented corpus reports nothing missing never - +harness_selftest 350-the-pytest-corpus-must-be control: distinct names in the same corpus report no duplicate never - +harness_selftest 350-the-pytest-corpus-must-be every test file and every test in the corpus is named in TESTS.md never - +harness_selftest 350-the-pytest-corpus-must-be every test the document names exists in the corpus never - +harness_selftest 350-the-pytest-corpus-must-be no test name is defined twice in the corpus never - +harness_selftest 350-the-pytest-corpus-must-be premise: the corpus carries the documentation this part polices never - +harness_selftest 350-the-pytest-corpus-must-be premise: the counting rule finds modes at all never - +harness_selftest 350-the-pytest-corpus-must-be premise: the mode inventory is where this part thinks it is never - +harness_selftest 350-the-pytest-corpus-must-be premise: the pytest corpus is where this part thinks it is never - +harness_selftest 350-the-pytest-corpus-must-be premise: the reader still finds a totals line when one is there never - +harness_selftest 350-the-pytest-corpus-must-be premise: the reverse sweep reads backticked names at all never - +harness_selftest 350-the-pytest-corpus-must-be premise: the sweep found the corpus rather than an empty glob never - +harness_selftest 350-the-pytest-corpus-must-be section 1a's document total is the sum of its two sections never - +harness_selftest 350-the-pytest-corpus-must-be section 1a's not-refused total is the count of ids in section 3 never - +harness_selftest 350-the-pytest-corpus-must-be section 1a's refused total is the count of ids in section 2 never - +harness_selftest 350-the-pytest-corpus-must-be section 2's opening states the counted number of refused modes never - +harness_selftest 350-the-pytest-corpus-must-be the admitted gap is the run total minus what is written down never - +harness_selftest 350-the-pytest-corpus-must-be the closing paragraph states the counted number too never - +harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 2 never - +harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 3 never - +harness_selftest 350-the-pytest-corpus-must-be the counter stops at the next heading never - +harness_selftest 350-the-pytest-corpus-must-be the row's value is read, not a digit inside its label never - +harness_selftest 350-the-pytest-corpus-must-be the sweep counts the fixture's tests and files never - +harness_selftest 360-an-unrunnable-pytest-test-must a comparison on the exit status is not counted as an assignment never - +harness_selftest 360-an-unrunnable-pytest-test-must a drifted exit code is visible rather than absorbed never - +harness_selftest 360-an-unrunnable-pytest-test-must a write-only unrunnable field is caught never - +harness_selftest 360-an-unrunnable-pytest-test-must an unconditional exit override is caught by the dominance arm never - +harness_selftest 360-an-unrunnable-pytest-test-must and only ever moves a run off zero, so a failure still dominates never - +harness_selftest 360-an-unrunnable-pytest-test-must and something READS it, rather than only writing it never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: and that same fixture does show the write, so the arm is not blind never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: lib.sh states an INCOMPLETE exit code this part could read never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: the harness library is where this part thinks it is never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer is where this part thinks it is never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer states one too never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: while a real assignment on the same line shape IS counted never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: while the real layer satisfies that same arm never - +harness_selftest 360-an-unrunnable-pytest-test-must the layer ends a session by setting its exit status never - +harness_selftest 360-an-unrunnable-pytest-test-must the layer prints the unrunnable reason in lib.sh's shape never - +harness_selftest 360-an-unrunnable-pytest-test-must the layer still writes the unrunnable state never - +harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses agree on the INCOMPLETE exit code never - +harness_selftest 370-the-plan-marker-guard-must a neutered absent arm is caught never - +harness_selftest 370-the-plan-marker-guard-must a neutered empty-plan refusal is caught never - +harness_selftest 370-the-plan-marker-guard-must a neutered present arm is caught never - +harness_selftest 370-the-plan-marker-guard-must and that refusal is a VacuityError, not an ordinary assertion never - +harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is absent never - +harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is present never - +harness_selftest 370-the-plan-marker-guard-must plan_marker refuses a plan with no nodes at all never - +harness_selftest 370-the-plan-marker-guard-must premise: both line numbers were found, so the ordering arm can mean something never - +harness_selftest 370-the-plan-marker-guard-must premise: plan_marker's body was actually cut out of the file never - +harness_selftest 370-the-plan-marker-guard-must premise: the pytest layer is where this part thinks it is never - +harness_selftest 370-the-plan-marker-guard-must premise: while the real body satisfies all three, so the greps work never - +harness_selftest 370-the-plan-marker-guard-must the empty-plan refusal precedes the arm it protects never - +harness_selftest 380-the-pytest-cluster-helpers a caller that reimplements the digest is caught never - +harness_selftest 380-the-pytest-cluster-helpers a fingerprint that reads src only is caught never - +harness_selftest 380-the-pytest-cluster-helpers a make_cluster with no cleanup is caught never - +harness_selftest 380-the-pytest-cluster-helpers an import from the pytest tree is caught never - +harness_selftest 380-the-pytest-cluster-helpers and a hard-coded module list is caught by the name arm never - +harness_selftest 380-the-pytest-cluster-helpers and it catches BaseException, so an interrupt cleans up too never - +harness_selftest 380-the-pytest-cluster-helpers and it names no module directory, so it is a derivation and not a list never - +harness_selftest 380-the-pytest-cluster-helpers and it stops a partially started cluster before removing the tree never - +harness_selftest 380-the-pytest-cluster-helpers and no longer mixes in the bare filename never - +harness_selftest 380-the-pytest-cluster-helpers and the original error is re-raised rather than swallowed never - +harness_selftest 380-the-pytest-cluster-helpers and the shell keeps none either never - +harness_selftest 380-the-pytest-cluster-helpers make_cluster removes its tree when setup raises never - +harness_selftest 380-the-pytest-cluster-helpers premise: and the real helper still carries its cleanup never - +harness_selftest 380-the-pytest-cluster-helpers premise: make_cluster's body was actually cut out of the file never - +harness_selftest 380-the-pytest-cluster-helpers premise: the one fingerprint implementation is where this part thinks it is never - +harness_selftest 380-the-pytest-cluster-helpers premise: the pytest cluster helper is where this part thinks it is never - +harness_selftest 380-the-pytest-cluster-helpers premise: while the real module satisfies the derivation arm never - +harness_selftest 380-the-pytest-cluster-helpers the fingerprint derives its build directories from a Makefile on disk never - +harness_selftest 380-the-pytest-cluster-helpers the hash mixes in each file's path relative to the tree, not its name never - +harness_selftest 380-the-pytest-cluster-helpers the module imports nothing from the pytest tree never - +harness_selftest 380-the-pytest-cluster-helpers the pytest helper keeps no private fingerprint implementation never - +harness_selftest 390-a-registered-suite-must-account a NEW unaccounted suite fails even while the known debt is excused never - +harness_selftest 390-a-registered-suite-must-account a comment mentioning pgc_summary is not a declaration never - +harness_selftest 390-a-registered-suite-must-account a declared suite that produced no accounting is caught never - +harness_selftest 390-a-registered-suite-must-account a declared suite the driver never dispatched reconciles never - +harness_selftest 390-a-registered-suite-must-account a file that does not exist is reported absent, not exempt never - +harness_selftest 390-a-registered-suite-must-account a hash inside a word does not hide the call after it never - +harness_selftest 390-a-registered-suite-must-account a log carrying lib.sh's accounting line is accounted never - +harness_selftest 390-a-registered-suite-must-account a log carrying neither is not accounted never - +harness_selftest 390-a-registered-suite-must-account a log claiming PASSED without the accounting line shows none never - +harness_selftest 390-a-registered-suite-must-account a long suite that calls pgc_summary still declares accounting never - +harness_selftest 390-a-registered-suite-must-account a longer name containing pgc_summary is not a declaration never - +harness_selftest 390-a-registered-suite-must-account a passing log shows accounting never - +harness_selftest 390-a-registered-suite-must-account a registered suite that is accounted by nothing FAILS never - +harness_selftest 390-a-registered-suite-must-account a suite recorded as known debt passes never - +harness_selftest 390-a-registered-suite-must-account a suite recorded as never dispatched that DID account is caught never - +harness_selftest 390-a-registered-suite-must-account a suite that accounted passes never - +harness_selftest 390-a-registered-suite-must-account a suite that calls pgc_summary declares accounting never - +harness_selftest 390-a-registered-suite-must-account a suite that never calls it does not never - +harness_selftest 390-a-registered-suite-must-account a suite that now accounts but is still listed as debt is reported never - +harness_selftest 390-a-registered-suite-must-account a suite the driver never dispatched passes never - +harness_selftest 390-a-registered-suite-must-account a trailing comment after the call does not hide it never - +harness_selftest 390-a-registered-suite-must-account an absent log shows no accounting rather than erroring never - +harness_selftest 390-a-registered-suite-must-account an accounting line that does not start its line is refused never - +harness_selftest 390-a-registered-suite-must-account an indented comment is still a comment never - +harness_selftest 390-a-registered-suite-must-account an undeclared suite that DID account is caught too never - +harness_selftest 390-a-registered-suite-must-account and a failed population reconciliation fails the major never - +harness_selftest 390-a-registered-suite-must-account and a failed reconciliation sets the per-major failure flag never - +harness_selftest 390-a-registered-suite-must-account and a log carrying only its OWN checks-run line is accounted too never - +harness_selftest 390-a-registered-suite-must-account and a reworded producer line is refused, so the arm can fail never - +harness_selftest 390-a-registered-suite-must-account and a skip, which reached the summary and counted zero never - +harness_selftest 390-a-registered-suite-must-account and absent is distinguishable from a present file that does not declare never - +harness_selftest 390-a-registered-suite-must-account and an incomplete never - +harness_selftest 390-a-registered-suite-must-account and debt naming a suite that is not registered is reported too never - +harness_selftest 390-a-registered-suite-must-account and it agrees with the real reader on a SHORT file, which is why it survived review never - +harness_selftest 390-a-registered-suite-must-account and it is NAMED, so the reader does not have to diff two lists never - +harness_selftest 390-a-registered-suite-must-account and it is named as that fault, not as one of the other two never - +harness_selftest 390-a-registered-suite-must-account and it is named as the opposite fault, not the same one never - +harness_selftest 390-a-registered-suite-must-account and it is named, which the symmetry check could never do never - +harness_selftest 390-a-registered-suite-must-account and prose containing the word does not count as the line never - +harness_selftest 390-a-registered-suite-must-account and so does a failing one, which is the point never - +harness_selftest 390-a-registered-suite-must-account and the excused one is not named as a failure never - +harness_selftest 390-a-registered-suite-must-account and the reader answers no on it, which is the wrong answer the arm catches never - +harness_selftest 390-a-registered-suite-must-account and the real function reconciles the same input, so the arm is not noise never - +harness_selftest 390-a-registered-suite-must-account and the reconciliation is given that record never - +harness_selftest 390-a-registered-suite-must-account and the registered file is written from the SUITES array itself never - +harness_selftest 390-a-registered-suite-must-account and without that record the same run is still caught never - +harness_selftest 390-a-registered-suite-must-account equal sets reconcile never - +harness_selftest 390-a-registered-suite-must-account every registered suite has a file never - +harness_selftest 390-a-registered-suite-must-account nor no for every one of them never - +harness_selftest 390-a-registered-suite-must-account opposite errors do not cancel: both directions are reported never - +harness_selftest 390-a-registered-suite-must-account premise: and produced exactly one accounting line to be read never - +harness_selftest 390-a-registered-suite-must-account premise: and that count excludes the definition line, which mentions it never - +harness_selftest 390-a-registered-suite-must-account premise: and the accounted reader that feeds it never - +harness_selftest 390-a-registered-suite-must-account premise: and the real function still does never - +harness_selftest 390-a-registered-suite-must-account premise: pipefail is on, which is the condition the bug needs never - +harness_selftest 390-a-registered-suite-must-account premise: the declaration reader evalled out of the runner is callable never - +harness_selftest 390-a-registered-suite-must-account premise: the drift changed the line the reader looks for never - +harness_selftest 390-a-registered-suite-must-account premise: the fixture carries a well-formed accounting line, just indented never - +harness_selftest 390-a-registered-suite-must-account premise: the fixture is long enough to lose the race never - +harness_selftest 390-a-registered-suite-must-account premise: the fixture really does hide its call from the stripper never - +harness_selftest 390-a-registered-suite-must-account premise: the mutation applied -- the twin no longer sorts its inputs never - +harness_selftest 390-a-registered-suite-must-account premise: the observation reader evalled out of the runner is callable never - +harness_selftest 390-a-registered-suite-must-account premise: the population reconciliation is callable never - +harness_selftest 390-a-registered-suite-must-account premise: the real suite ran and reached its summary never - +harness_selftest 390-a-registered-suite-must-account premise: the reconciliation evalled out of the runner is callable never - +harness_selftest 390-a-registered-suite-must-account premise: the registered list is not empty, so the partition means something never - +harness_selftest 390-a-registered-suite-must-account premise: the runner defines the declaration reader this part evals never - +harness_selftest 390-a-registered-suite-must-account premise: the runner defines the observation reader this part evals never - +harness_selftest 390-a-registered-suite-must-account premise: the runner defines the population reconciliation never - +harness_selftest 390-a-registered-suite-must-account premise: the runner defines the reconciliation this part evals never - +harness_selftest 390-a-registered-suite-must-account premise: the twin script was written and is runnable never - +harness_selftest 390-a-registered-suite-must-account premise: the unsorted twin is callable never - +harness_selftest 390-a-registered-suite-must-account the debt file is in the tree never - +harness_selftest 390-a-registered-suite-must-account the grep -q shape is the one that gets this wrong under pipefail never - +harness_selftest 390-a-registered-suite-must-account the identity catches comm reading unsorted input never - +harness_selftest 390-a-registered-suite-must-account the partition over the real suite list adds up never - +harness_selftest 390-a-registered-suite-must-account the population partitions, and prints inputs == sum(buckets) never - +harness_selftest 390-a-registered-suite-must-account the reader accepts the line the producer actually emits never - +harness_selftest 390-a-registered-suite-must-account the reader does not answer yes for every registered suite never - +harness_selftest 390-a-registered-suite-must-account the reconciliation prints inputs == sum(buckets) never - +harness_selftest 390-a-registered-suite-must-account the record cannot introduce a suite the source never declared never - +harness_selftest 390-a-registered-suite-must-account the runner calls the population reconciliation never - +harness_selftest 390-a-registered-suite-must-account the runner calls the reconciliation, not merely defines it never - +harness_selftest 390-a-registered-suite-must-account the skip branch records the suite it did not dispatch never - +harness_selftest 390-a-registered-suite-must-account the stripper hides no pgc_summary call in any registered suite never - +harness_selftest 400-a-check-result-must-be-machine a bogus reason code records FAIL, not UNRUN never - +harness_selftest 400-a-check-result-must-be-machine a failing check emits exactly one record never - +harness_selftest 400-a-check-result-must-be-machine a failing check still prints its old line never - +harness_selftest 400-a-check-result-must-be-machine a log that never stated a count is not silently accepted never - +harness_selftest 400-a-check-result-must-be-machine a log whose records match its stated count reconciles never - +harness_selftest 400-a-check-result-must-be-machine a log with fewer records than it claims is caught never - +harness_selftest 400-a-check-result-must-be-machine a log with more records than it claims is caught too never - +harness_selftest 400-a-check-result-must-be-machine a passing check emits exactly one record never - +harness_selftest 400-a-check-result-must-be-machine a passing check still prints its old line never - +harness_selftest 400-a-check-result-must-be-machine an unrunnable check emits exactly one record never - +harness_selftest 400-a-check-result-must-be-machine an unrunnable check still prints its old line never - +harness_selftest 400-a-check-result-must-be-machine and its name field is the check's name, spaces intact never - +harness_selftest 400-a-check-result-must-be-machine and its verdict field says FAIL never - +harness_selftest 400-a-check-result-must-be-machine and its verdict field says PASS never - +harness_selftest 400-a-check-result-must-be-machine and its verdict field says UNRUN, which is neither of the other two never - +harness_selftest 400-a-check-result-must-be-machine and records FAIL never - +harness_selftest 400-a-check-result-must-be-machine and records FAIL, because nothing was compared never - +harness_selftest 400-a-check-result-must-be-machine and records PASS when the ratio is inside the bound never - +harness_selftest 400-a-check-result-must-be-machine and that place is pgc_record never - +harness_selftest 400-a-check-result-must-be-machine and the REASON_CODE travels in the reason field, not in prose never - +harness_selftest 400-a-check-result-must-be-machine and the two numbers are named, not just the verdict never - +harness_selftest 400-a-check-result-must-be-machine check_num on a non-number emits one record never - +harness_selftest 400-a-check-result-must-be-machine check_num's non-measurement line is unchanged never - +harness_selftest 400-a-check-result-must-be-machine check_ratio on a non-number emits one record never - +harness_selftest 400-a-check-result-must-be-machine check_ratio that forms a ratio emits one record never - +harness_selftest 400-a-check-result-must-be-machine check_ratio with a zero side emits one record never - +harness_selftest 400-a-check-result-must-be-machine check_text on an empty side emits one record never - +harness_selftest 400-a-check-result-must-be-machine check_text's empty-side line is unchanged never - +harness_selftest 400-a-check-result-must-be-machine lib.sh bumps PGC_CHECKS in exactly one place never - +harness_selftest 400-a-check-result-must-be-machine pgc_fail emits one record never - +harness_selftest 400-a-check-result-must-be-machine pgc_pass emits one record never - +harness_selftest 400-a-check-result-must-be-machine premise: and that is this fragment, not the suite never - +harness_selftest 400-a-check-result-must-be-machine premise: it is callable never - +harness_selftest 400-a-check-result-must-be-machine premise: lib.sh is where the check helpers live never - +harness_selftest 400-a-check-result-must-be-machine premise: the probe ran every helper shape once never - +harness_selftest 400-a-check-result-must-be-machine the record count equals the counter the summary reports never - +harness_selftest 400-a-check-result-must-be-machine the record names the part the check was asked from never - +harness_selftest 400-a-check-result-must-be-machine the runner calls the record reconciliation, not merely defines it never - +harness_selftest 400-a-check-result-must-be-machine the runner defines the record reconciliation never - +harness_selftest 410-a-check-must-have-been-red a before-log and an after-log together are refused, not silently empty never - +harness_selftest 410-a-check-must-have-been-red a check in an UNCOVERED suite is not refused never - +harness_selftest 410-a-check-must-have-been-red a check merely added is not reported as a rename never - +harness_selftest 410-a-check-must-have-been-red a check observed red gains the date it was seen never - +harness_selftest 410-a-check-must-have-been-red a check the ledger has never seen is refused never - +harness_selftest 410-a-check-must-have-been-red a gate over a nonexistent log is an integrity failure, not a pass never - +harness_selftest 410-a-check-must-have-been-red a later green run does not erase an observation never - +harness_selftest 410-a-check-must-have-been-red a name that appeared while another disappeared is reported as a rename never - +harness_selftest 410-a-check-must-have-been-red a named mutation is recorded against the check that reddened never - +harness_selftest 410-a-check-must-have-been-red a real refusal is a different status from an integrity failure never - +harness_selftest 410-a-check-must-have-been-red a rename in one part survives an addition in another never - +harness_selftest 410-a-check-must-have-been-red a run whose checks are all ledgered passes the gate never - +harness_selftest 410-a-check-must-have-been-red a second mutation ACCUMULATES rather than replacing the first never - +harness_selftest 410-a-check-must-have-been-red an empty log is one too, because there is nothing to reconcile never - +harness_selftest 410-a-check-must-have-been-red and a record missing its verdict never - +harness_selftest 410-a-check-must-have-been-red and a refused gate fails the major never - +harness_selftest 410-a-check-must-have-been-red and an empty mutation is a placeholder, not an empty last field never - +harness_selftest 410-a-check-must-have-been-red and it entered as debt, not as an observation nothing made never - +harness_selftest 410-a-check-must-have-been-red and it is named, so the author knows which one never - +harness_selftest 410-a-check-must-have-been-red and it is the new one that is named, not the one already ledgered never - +harness_selftest 410-a-check-must-have-been-red and it runs before the build directory is removed, which is the only place it can never - +harness_selftest 410-a-check-must-have-been-red and none of them ends in a tab never - +harness_selftest 410-a-check-must-have-been-red and not against one that stayed green never - +harness_selftest 410-a-check-must-have-been-red and one that stayed green keeps its debt never - +harness_selftest 410-a-check-must-have-been-red and records neither as ever having been red never - +harness_selftest 410-a-check-must-have-been-red and the addition in the other part is not called a rename never - +harness_selftest 410-a-check-must-have-been-red and the history it is about to lose travels with it never - +harness_selftest 410-a-check-must-have-been-red and the message says how to fix it, because regenerating is the intended action never - +harness_selftest 410-a-check-must-have-been-red and the refusal names both values never - +harness_selftest 410-a-check-must-have-been-red but that suite is counted as not covered, which is the debt never - +harness_selftest 410-a-check-must-have-been-red each says what was wrong with the input never - +harness_selftest 410-a-check-must-have-been-red every committed row has five fields never - +harness_selftest 410-a-check-must-have-been-red every row has five fields and no trailing tab never - +harness_selftest 410-a-check-must-have-been-red lowering it is allowed, which is the direction the burn-down goes never - +harness_selftest 410-a-check-must-have-been-red merging a green run records both checks never - +harness_selftest 410-a-check-must-have-been-red nor is one merely removed never - +harness_selftest 410-a-check-must-have-been-red once the suite is covered, a new check in it IS refused never - +harness_selftest 410-a-check-must-have-been-red one --mutation cannot be attributed across several runs at once never - +harness_selftest 410-a-check-must-have-been-red premise: the budget is a tracked file too never - +harness_selftest 410-a-check-must-have-been-red premise: the check has history before the rename never - +harness_selftest 410-a-check-must-have-been-red premise: the ledger is not empty, so the partition means something never - +harness_selftest 410-a-check-must-have-been-red premise: the ledger itself is a tracked file, not a variable never - +harness_selftest 410-a-check-must-have-been-red premise: the ledger tool exists never - +harness_selftest 410-a-check-must-have-been-red premise: the scratch repo has a prior ceiling committed never - +harness_selftest 410-a-check-must-have-been-red raising the ceiling above its committed value is refused never - +harness_selftest 410-a-check-must-have-been-red regenerating the ledger lets the new check through never - +harness_selftest 410-a-check-must-have-been-red the budget names a ceiling and a census, and says which is which never - +harness_selftest 410-a-check-must-have-been-red the ceiling refuses being exceeded never - +harness_selftest 410-a-check-must-have-been-red the committed census matches the committed ledger never - +harness_selftest 410-a-check-must-have-been-red the gate refuses to run without the registered suite list never - +harness_selftest 410-a-check-must-have-been-red the ledger partitions into observed and never never - +harness_selftest 410-a-check-must-have-been-red the runner invokes the ledger gate never - +harness_selftest 410-a-check-must-have-been-red the same check in two logs is two runs, not a duplicate never - +harness_selftest 410-a-check-must-have-been-red the same name twice in ONE log is a duplicate, and is named never - +harness_selftest 410-a-check-must-have-been-red the stable check is not reported never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 7ce2882f..f0815846 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -1,28 +1,37 @@ -# How much of the check corpus has never been seen red -- DEBT, in a tracked -# file so a change to it is a diff a reviewer sees. +# The ledger's debt, in a tracked file so a change to it is a diff a reviewer +# sees. An environment variable would not be: PGC_SKIP_TIMING is the precedent, +# set in two workflow files, suppressing whole suites for months with no diff +# ever showing it. # -# An environment variable would not be. PGC_SKIP_TIMING is the precedent: set in -# two workflow files, it suppressed whole suites for months and no diff ever -# showed it. +# THE TWO NUMBERS ARE DIFFERENT KINDS OF THING. The first version of this file +# treated both as ceilings and deadlocked, so the distinction is written down. # -# BOTH NUMBERS MAY ONLY GO DOWN. A change that raises either is a change that -# adds debt, and it must read as exactly that in review rather than as a passing -# gate. +# suites_not_covered -- A CEILING, and monotone. +# Registered suites with no rows in the ledger at all. Their checks are +# invisible to the gate: it cannot refuse a new check in a suite it has never +# seen. It falls as suites are seeded, and it MAY ONLY FALL -- the gate compares +# this value against the previously committed one and refuses an increase, so +# widening the debt is an edit a reviewer sees AND a gate refuses. # -# checks_never_observed_red -# Checks in test/check_ledger.tsv that no recorded run has ever seen go red. -# It does NOT mean they cannot fail -- that is a stronger claim needing a named -# mutation, and this ledger does not make it. It means nothing has attacked -# them yet, which is worth knowing on its own. +# Adding a check to a suite that is already covered does not move it, which is +# what makes it safe to bound. +suites_not_covered 250 # -# It drains from the project's actual failures as well as from deliberate -# mutation runs: every real CI red, every flake, every bisect fills the ledger, -# and those arrive whether anyone remembers to run something or not. +# checks_never_observed_red -- A CENSUS. NOT a ceiling, and it must not become +# one. +# Every new check enters the ledger as `never`, so bounding this number means +# every added check breaks the gate, and the only way to land one is to raise a +# number the design says may only fall. That is a deadlock rather than a budget, +# and the first version of this file shipped it: 614 rows, 614 never, ceiling +# 614. # -# suites_not_covered -# Registered suites with no rows in the ledger at all. Their checks are -# invisible to everything above: the gate cannot refuse a new check in a suite -# it has never seen. Counted separately so that "we ledger 605 checks" cannot -# read as "we ledger the corpus". -checks_never_observed_red 614 -suites_not_covered 250 +# It is reported by the gate on every run and falls as checks are attacked. What +# the gate REFUSES is a check the ledger has never seen, which regenerating the +# ledger fixes -- a reviewable one-line diff, and the intended action rather +# than a forbidden one. +# +# The number here is ASSERTED to match the committed ledger, in both harnesses. +# Without that it is a hand-maintained count that drifts, which is the failure +# this repository has spent a day proving. It is not a ceiling; it is a +# measurement that must be true. +checks_never_observed_red 638 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 772bfecc..1a5cf763 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -6,103 +6,155 @@ whole purpose is to stop exactly that. The gate answered "did anything print FAIL" and had never answered "could anything print FAIL". -WHAT THIS LEDGER CLAIMS, AND WHAT IT DOES NOT ---------------------------------------------- +WHAT THIS RECORDS, AND WHAT IT DOES NOT +--------------------------------------- It records that a named check WAS OBSERVED RED in a recorded run. It does NOT -claim the check is proven able to fail: that is a stronger statement, it needs a -named mutation applied deliberately, and conflating the two would put a claim in -the ledger that nothing measured -- the `defeated: 0` shape from VACUITY_MODES -section 1, a number that reads as evidence and is not. - -So v1 fills the observed column honestly and leaves the rest as debt, counted. A -ledger whose entries all read `never` is a measurement of how much of the suite -has never been attacked, and that measurement is worth having on day one. - -WHAT FILLS IT +claim the check is proven able to fail: that needs a named mutation applied +deliberately, and conflating the two would put a claim in the ledger that nothing +measured -- the `defeated: 0` shape from VACUITY_MODES section 1. + +THE TWO NUMBERS ARE DIFFERENT KINDS OF THING, and the first design got this wrong +in a way that deadlocked. +------------------------------------------------------------------------------- +`checks_never_observed_red` is a CENSUS. It cannot be a ceiling: every new check +enters the ledger as `never`, so bounding it means every added check breaks the +gate, and the only way to land one is to raise a number the design says may only +fall. That is a deadlock, not a budget. It is reported, and it falls as checks are +attacked. + +`suites_not_covered` IS a ceiling, and a monotone one, because adding a check to a +covered suite does not move it. It falls as suites are seeded, and it may only +fall: the gate compares the working value against the previously committed one and +refuses an increase, so widening the debt is an edit a reviewer sees AND a gate +refuses, rather than either alone. + +WHAT THE GATE REFUSES +--------------------- +A check the committed ledger has never seen. That is the allowlist the issue asks +for -- existing checks are grandfathered, a new one is named and refused until the +ledger is regenerated, which is a reviewable one-line diff and the INTENDED action +rather than a forbidden one. + +WHAT FEEDS IT ------------- -Not only deliberate mutation runs. Every real CI red fills it, every flake, every -bisect -- and those arrive whether anyone remembers or not. A mutation run is the -deliberate accelerator, not the only source. +`run_all_versions.sh` merges every suite's log before it removes the build +directory, so every matrix run feeds a ledger -- locally and in CI. The committed +ledger is updated deliberately, by running `merge` against a real run and +committing the diff. CI verifies; humans commit. A ledger that CI rewrote by +itself would be a file nobody reads changing under everybody. + +It is not only mutation runs. Every real CI red fills it, every flake, every +bisect. A mutation run is the deliberate accelerator. -THE MUTATION COLUMN -------------------- -Present from v1 with nothing filling it automatically, because adding a column -later means rewriting every entry. If an entry can record WHICH mutation reddened -a check, the catalogue a mutation gate would need builds itself out of work people -already do by hand. +FAIL CLOSED +----------- +An unreadable file, an empty one, or a record with too few fields is an ERROR. +Silently skipping them made every integrity failure indistinguishable from a clean +run: a gate over a nonexistent log returned success. FORMAT ------ -Tab separated, one row per check, sorted: +Tab separated, five columns, keyed on the first three: - suite part check name last observed red mutation + suite part check name last observed red mutations -`last observed red` is a date, or the literal `never`. `mutation` is free text or -empty. Both are written by this tool, never by hand. - -The row is keyed on (suite, part, name). The part matters because harness_selftest -sources 40-odd parts into one shell and phrases its premises to be COPIED, so a -name-only key is a key of check NAMES rather than of checks. +`last observed red` is a date or `never`. `mutations` is `-`, or a `;`-separated +SET of the mutations that have reddened this check -- accumulated, not overwritten, +because a column that keeps only the last one records the most recent attack +rather than the catalogue it exists to become. """ import argparse import pathlib +import subprocess import sys NEVER = "never" +NONE = "-" +FIELDS = 5 + +class LedgerError(Exception): + """An integrity failure. Never silently skipped.""" -def read_records(paths): - """(suite, name, verdict) for every RESULT line in the given logs.""" + +def read_records(paths, *, require_nonempty=True): + """[(suite, part, name, verdict)] for every RESULT line in the given logs. + + Fails closed. A path that cannot be read, a log with no records, or a record + with too few fields is an error -- silently skipping them is how a gate over a + nonexistent log returned success. + """ out = [] for p in paths: + path = pathlib.Path(p) try: - text = pathlib.Path(p).read_text(errors="replace") - except OSError: - continue - for line in text.splitlines(): + text = path.read_text(errors="replace") + except OSError as e: + raise LedgerError(f"cannot read {p}: {e}") from e + found = 0 + for n, line in enumerate(text.splitlines(), 1): if not line.startswith("RESULT\t"): continue f = line.split("\t") if len(f) < 5: - continue - # suite, part, name, verdict + raise LedgerError( + f"{p}:{n}: a record needs suite, part, name and verdict; got {len(f) - 1} field(s)") out.append((f[1], f[2], f[3], f[4])) + found += 1 + if require_nonempty and found == 0: + raise LedgerError(f"{p}: no RESULT records, so there is nothing to reconcile") return out def read_ledger(path): - """{(suite, part, name): [last_red, mutation]} from a ledger file. - - KEYED ON THE PART AS WELL AS THE NAME. harness_selftest sources 40-odd parts - into one shell and phrases its premises to be copied -- "premise: the pytest - layer is where THIS PART thinks it is" works verbatim in any of them -- so - (suite, name) is a key of check NAMES rather than of checks, and one sharer - going red would mark them all. Measured over a real run: 583 records give 579 - distinct (suite, name) and 582 distinct (suite, part, name). + """{(suite, part, name): [last_red, {mutations}]}. + + Keyed on the part as well as the name: harness_selftest sources 40-odd parts + into one shell and phrases its premises to be COPIED, so a name-only key is a + key of check NAMES rather than of checks. """ rows = {} p = pathlib.Path(path) if not p.exists(): return rows - for line in p.read_text(errors="replace").splitlines(): + for n, line in enumerate(p.read_text(errors="replace").splitlines(), 1): if not line.strip() or line.startswith("#"): continue f = line.split("\t") - while len(f) < 5: - f.append("") - rows[(f[0], f[1], f[2])] = [f[3] or NEVER, f[4]] + if len(f) != FIELDS: + raise LedgerError(f"{path}:{n}: a ledger row needs {FIELDS} fields, got {len(f)}") + muts = set() if f[4] == NONE else {m for m in f[4].split(";") if m} + rows[(f[0], f[1], f[2])] = [f[3] or NEVER, muts] return rows def write_ledger(path, rows): - lines = [ - "\t".join((suite, part, name, v[0], v[1])) - for (suite, part, name), v in sorted(rows.items()) - ] + lines = [] + for (suite, part, name), (red, muts) in sorted(rows.items()): + # No trailing tab. An empty last field is trailing whitespace on every + # row, which `git diff --check` reports and which made 614 of them. + lines.append("\t".join((suite, part, name, red, + ";".join(sorted(muts)) if muts else NONE))) pathlib.Path(path).write_text("\n".join(lines) + ("\n" if lines else "")) +def _by_run(paths): + """[(path, {(suite, part, name): [verdicts]})] -- one entry per LOG. + + Per log, because the same check appearing in two logs is two RUNS of it, while + twice in one log is a duplicate name sharing a ledger row. Merging the logs + first cannot tell those apart, and reported the first as the second. + """ + runs = [] + for p in paths: + seen = {} + for suite, part, name, verdict in read_records([p]): + seen.setdefault((suite, part, name), []).append(verdict) + runs.append((p, seen)) + return runs + + def cmd_census(args): for suite, part, name, verdict in read_records(args.logs): print(f"{suite}\t{part}\t{name}\t{verdict}") @@ -111,85 +163,84 @@ def cmd_census(args): def cmd_merge(args): rows = read_ledger(args.ledger) - for suite, part, name, verdict in read_records(args.logs): - key = (suite, part, name) - if key not in rows: - # A check this ledger has never seen enters as DEBT. A green run has - # observed nothing go red, so merging one must never record a red - # observation -- otherwise an ordinary CI run retires the debt it - # exists to count. - rows[key] = [NEVER, ""] - if verdict == "FAIL": - rows[key][0] = args.date - if args.mutation: - rows[key][1] = args.mutation - write_ledger(args.ledger, rows) + runs = _by_run(args.logs) + + if args.mutation and len(runs) > 1: + raise LedgerError( + "--mutation names one deliberate change, so it cannot be attributed across " + f"{len(runs)} logs at once: merge them one run at a time") + + for path, seen in runs: + for key, verdicts in sorted(seen.items()): + if key not in rows: + # A check this ledger has never seen enters as DEBT. A green run + # has observed nothing go red, so merging one must never record a + # red observation. + rows[key] = [NEVER, set()] + if "FAIL" in verdicts: + rows[key][0] = args.date + if args.mutation: + # A SET. Keeping only the last one records the most recent + # attack rather than the catalogue this column exists to + # become. + rows[key][1].add(args.mutation) + for key, verdicts in sorted(seen.items()): + if len(verdicts) > 1: + print(f" duplicate check name in one run, so one ledger row covers " + f"{len(verdicts)}: {key[0]}\t{key[1]}\t{key[2]}") - # A DUPLICATED NAME SHARES ONE LEDGER ROW, so one of the two going red marks - # BOTH as observed red -- a claim about a check nothing attacked, which is - # precisely what this ledger must not make. It cannot be fixed by keying - # harder without a synthetic id someone would maintain, so it is reported. - # A duplicate WITHIN one part still shares a row -- the part fixed the - # convention collisions, not genuine repeats. One survives in the real - # corpus, and naming it precisely is the point of keying on the part. - records = read_records(args.logs) - counts = {} - for suite, part, name, _ in records: - counts[(suite, part, name)] = counts.get((suite, part, name), 0) + 1 - for suite, part, name in sorted(k for k, c in counts.items() if c > 1): - print(f" duplicate check name, so one ledger row covers " - f"{counts[(suite, part, name)]}: {suite}\t{part}\t{name}") - - seen = len({(s, p_, n) for s, p_, n, _ in records}) + write_ledger(args.ledger, rows) + seen_all = {k for _, s in runs for k in s} red = sum(1 for v in rows.values() if v[0] != NEVER) - print(f" ledger: rows={len(rows)} | seen this run={seen}, " + print(f" ledger: rows={len(rows)} | runs={len(runs)}, distinct checks this merge={len(seen_all)}, " f"observed red ever={red}, never={len(rows) - red}") return 0 def cmd_rename_scan(args): - """A name that appeared while another disappeared is probably a rename. + """A name that appeared while another disappeared, WITHIN ONE PART, is a rename. - Keyed by the display string, a rename loses the check's history and reads - exactly like a brand-new check that has never been red -- the one state this - ledger exists to distinguish. It cannot be prevented without a synthetic id - that someone would have to maintain, so it is DETECTED and named instead of - silently resetting a count to `never`. + Keyed by the display string, a rename loses history and reads exactly like a + brand-new check that has never been red -- the one state this ledger exists to + distinguish. Detected and named rather than silently reset. - Both directions are required. A check merely added, or merely removed, is not - a rename, and reporting one on every new check is noise that gets the whole - thing ignored. + Grouped by (suite, part) before pairing. A global positional zip misses a real + rename whenever unrelated movement in another part shifts the ordering. + + Scanned against ONE run. Given a before-log and an after-log together, the + vanished name is present in the union and nothing appears to have gone. """ + runs = _by_run(args.logs) + if len(runs) > 1: + raise LedgerError( + f"rename-scan compares ONE run against the ledger, but got {len(runs)} logs: " + "the union of a before-log and an after-log hides the disappearance") rows = read_ledger(args.ledger) - now = {(s, p_, n) for s, p_, n, _ in read_records(args.logs)} - parts = {(s, p_) for s, p_, _ in now} + now = set(runs[0][1]) + + parts = {(s, p) for s, p, _ in now} known = {k for k in rows if (k[0], k[1]) in parts} - appeared = sorted(now - known) - vanished = sorted(known - now) rc = 0 - if appeared and vanished: - # Only pair within a PART. Keying on the part also fixes a blind spot the - # name-only key had: a premise moving between parts was indistinguishable - # from a rename, and now it is a disappearance and an appearance in two - # different parts, which this does not pair. - for (s_a, p_a, n_a), (s_v, p_v, n_v) in zip(appeared, vanished): - if (s_a, p_a) != (s_v, p_v): - continue - was = rows.get((s_v, p_v, n_v), [NEVER, ""])[0] - print(f" possible rename: {n_v} -> {n_a} " - f"(in {s_a}/{p_a}, history: last red {was})") + n_app = n_van = 0 + for part in sorted(parts): + app = sorted(k[2] for k in now - known if (k[0], k[1]) == part) + van = sorted(k[2] for k in known - now if (k[0], k[1]) == part) + n_app += len(app) + n_van += len(van) + for new, old in zip(app, van): + was = rows.get((part[0], part[1], old), [NEVER, set()])[0] + print(f" possible rename: {old} -> {new} " + f"(in {part[0]}/{part[1]}, history: last red {was})") rc = 1 - print(f" rename scan: appeared={len(appeared)}, vanished={len(vanished)}") + print(f" rename scan: appeared={n_app}, vanished={n_van}") return rc -def _read_budget(path): +def read_budget(path): out = {} - p = pathlib.Path(path) - if not p.exists(): - return out - for line in p.read_text().splitlines(): + text = pathlib.Path(path).read_text() + for line in text.splitlines(): line = line.strip() if not line or line.startswith("#"): continue @@ -199,51 +250,91 @@ def _read_budget(path): return out +def _committed_budget(path, ref): + """The budget as of `ref`, or None when it does not exist there.""" + try: + blob = subprocess.run(["git", "show", f"{ref}:{path}"], + capture_output=True, text=True, check=True).stdout + except (subprocess.CalledProcessError, OSError): + return None + out = {} + for line in blob.splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + p = line.split() + if len(p) == 2 and p[1].isdigit(): + out[p[0]] = int(p[1]) + return out + + def cmd_gate(args): rows = read_ledger(args.ledger) - budget = _read_budget(args.budget) - seen = {(s, p_, n) for s, p_, n, _ in read_records(args.logs)} + budget = read_budget(args.budget) + seen = {k for k, _, _ in ((k, None, None) for k in + {(s, p, n) for s, p, n, _ in read_records(args.logs)})} - # A check the ledger has never heard of is NEW. The allowlist exists so a - # gate that fails on 3,762 unledgered sites is not what lands -- but a new - # one must not enter as silent debt either. - unknown = sorted(seen - set(rows)) rc = 0 + + # THE REFUSAL: a check the committed ledger has never seen, IN A SUITE THE + # LEDGER COVERS. + # + # The suite restriction is not a softening, it is the meaning of + # suites_not_covered: the gate cannot refuse a new check in a suite it has + # never seen, because it has no idea which of that suite's checks are new. + # Without it the gate refuses every check of all 250 uncovered suites and + # reddens the whole matrix on the first run -- which is a gate somebody turns + # off, the failure mode this issue family exists to prevent. + # + # It tightens on its own as suites are seeded, and the ceiling is what forces + # that direction. + covered_suites = {k[0] for k in rows} + unknown = sorted(k for k in seen - set(rows) if k[0] in covered_suites) for suite, part, name in unknown: print(f" not in the ledger: {suite}\t{part}\t{name}") + if unknown: + print(f" {len(unknown)} check(s) the ledger has never seen. Regenerate it with:") + print(f" python3 test/pgc_ledger.py merge --ledger {args.ledger} --date ") rc = 1 + # A CENSUS, not a ceiling. Bounding it deadlocks: every new check enters as + # `never`, so the only way to land one would be to raise a number the design + # says may only fall. never = sum(1 for v in rows.values() if v[0] == NEVER) - want = budget.get("checks_never_observed_red") - print(f" ledger gate: rows={len(rows)} | never observed red={never}, " - f"budget={want if want is not None else 'unset'}, new={len(unknown)}") + print(f" ledger census: rows={len(rows)} | never observed red={never}, " + f"ever red={len(rows) - never}, new this run={len(unknown)}") + + if not args.registered: + raise LedgerError( + "--registered is required: without the registered suite list the coverage " + "claim cannot be made, and skipping it silently is how a gate reports success " + "for a question it never asked") + registered = {w for w in pathlib.Path(args.registered).read_text().split() if w} + if not registered: + raise LedgerError(f"{args.registered}: no registered suites listed") + uncovered = sorted(registered - {k[0] for k in rows}) + want = budget.get("suites_not_covered") + print(f" ledger coverage: registered={len(registered)} | covered={len(registered) - len(uncovered)}, " + f"not covered={len(uncovered)}, ceiling={want if want is not None else 'unset'}") if want is None: - print(" the budget file names no checks_never_observed_red, so nothing bounds the debt") + print(" the budget names no suites_not_covered, so nothing bounds the coverage debt") return 1 - if never > want: - print(f" checks_never_observed_red: {never} exceeds the budget of {want}") + if len(uncovered) > want: + print(f" suites_not_covered: {len(uncovered)} exceeds the ceiling of {want}") rc = 1 - # The SECOND debt, and the one that is easy to forget: a suite with no rows - # in the ledger is not covered at all, and its checks are invisible to - # everything above -- the gate cannot refuse a new check in a suite it has - # never seen. Counting it separately keeps "we ledger 605 checks" from - # reading as "we ledger the corpus". - if args.registered: - registered = {l.strip() for l in pathlib.Path(args.registered).read_text().split() - if l.strip()} - covered = {k[0] for k in rows} - uncovered = sorted(registered - covered) - want_s = budget.get("suites_not_covered") - print(f" ledger coverage: registered={len(registered)} | " - f"covered={len(registered) - len(uncovered)}, not covered={len(uncovered)}, " - f"budget={want_s if want_s is not None else 'unset'}") - if want_s is None: - print(" the budget file names no suites_not_covered, so nothing bounds the coverage") - return 1 - if len(uncovered) > want_s: - print(f" suites_not_covered: {len(uncovered)} exceeds the budget of {want_s}") - rc = 1 + # MONOTONE, mechanically. The tracked file says the ceiling may only fall; + # without this that sentence is prose and raising the number passes. + if args.against: + prior = _committed_budget(args.budget, args.against) + if prior is None: + print(f" no budget at {args.against}, so there is no prior ceiling to compare") + else: + p_want = prior.get("suites_not_covered") + if p_want is not None and want > p_want: + print(f" suites_not_covered was raised from {p_want} to {want}: " + f"the ceiling may only fall") + rc = 1 return rc @@ -251,7 +342,7 @@ def main(argv=None): ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) sub = ap.add_subparsers(dest="cmd", required=True) - c = sub.add_parser("census", help="print suite/name/verdict for each RESULT line") + c = sub.add_parser("census", help="print suite/part/name/verdict for each record") c.add_argument("logs", nargs="+") c.set_defaults(fn=cmd_census) @@ -267,16 +358,22 @@ def main(argv=None): r.add_argument("logs", nargs="+") r.set_defaults(fn=cmd_rename_scan) - g = sub.add_parser("gate", help="refuse new checks and debt over budget") + g = sub.add_parser("gate", help="refuse a check the ledger has never seen") g.add_argument("--ledger", required=True) g.add_argument("--budget", required=True) g.add_argument("--registered", default="", - help="file listing every registered suite, for the coverage debt") + help="file listing every registered suite (required)") + g.add_argument("--against", default="", + help="git ref whose budget is the prior ceiling, for the monotone check") g.add_argument("logs", nargs="+") g.set_defaults(fn=cmd_gate) args = ap.parse_args(argv) - return args.fn(args) + try: + return args.fn(args) + except LedgerError as e: + print(f" ledger integrity failure: {e}") + return 2 if __name__ == "__main__": diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 77ef1531..ca318f84 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1212,117 +1212,86 @@ reconciliation. ## 16. test_mutation_ledger.py: which checks have ever been red -Nothing recorded whether a check had ever been red. That is the gap that let **39 -checks across 35 suites** ship unable to fail, three of them inside the suite whose -whole purpose is to stop exactly that. The gate answered *did anything print FAIL* and -had never answered *could anything print FAIL*. +Nothing recorded whether a check had ever been red. That is the gap that let **39 checks +across 35 suites** ship unable to fail, three of them inside the suite whose whole +purpose is to stop exactly that. -### What this ledger claims, and what it does not +It records that a named check **was observed red in a recorded run**. Not that it is +proven able to fail: that needs a named mutation applied deliberately, and conflating +the two would put a claim in the ledger nothing measured. -It records that a named check **was observed red in a recorded run**. It does **not** -claim the check is proven able to fail — that is a stronger statement, it needs a named -mutation applied deliberately, and conflating the two would put a claim in the ledger -that nothing measured. That is the `defeated: 0` shape from `VACUITY_MODES` section 1: a -number that reads as evidence and is not. +### The first design deadlocked, and the fix is the distinction -So v1 fills the observed column honestly and leaves the rest as debt, counted. **Every -entry currently reads `never`**, and that is the finding rather than an embarrassment: a -ledger of all-`never` is a measurement of how much of the corpus has never been -attacked. +Bounding `checks_never_observed_red` means **every added check breaks the gate**, because +a new check enters as `never` — so the only way to land one was to raise a number the +design said may only fall. It shipped at 614 rows, 614 `never`, ceiling 614. -### What fills it +| number | kind | why | +| --- | --- | --- | +| `suites_not_covered` | **ceiling**, monotone | adding a check to a covered suite does not move it | +| `checks_never_observed_red` | **census**, asserted | every new check enters as `never`, so bounding it deadlocks | -Not only deliberate mutation runs. Every real CI red fills it, every flake, every -bisect — and those arrive whether anyone remembers to run something or not. A mutation -run is the deliberate accelerator, not the only source. +What the gate refuses is a check the committed ledger has never seen, **in a suite the +ledger covers**. Regenerating the ledger is the intended fix and a reviewable diff. -### The format +The format is five tab-separated columns keyed on the first three: +`suite`, `part`, `check name`, `last observed red`, `mutations` — the last a `-` or a +`;`-separated **set**, accumulated rather than overwritten. -``` -suite check name last observed red mutation -``` +`run_all_versions.sh` invokes the gate before it removes the build directory, which is +the only place a matrix run can reach every suite's log. -`last observed red` is a date or the literal `never`. The row is keyed on the first -three fields. The **mutation column exists from -v1 with nothing filling it automatically**, because adding a column later means -rewriting every entry — and if an entry can record *which* mutation reddened a check, -the catalogue a mutation gate would need builds itself out of work people already do by -hand. +### `test_bad_input_is_an_integrity_failure_not_a_clean_run` -Two tracked files carry the debt, `test/check_ledger.tsv` and -`test/check_ledger_budget.txt`, so a change to either is a diff a reviewer sees. -`PGC_SKIP_TIMING` is the precedent for why it is not an environment variable: set in two -workflow files, it suppressed whole suites for months and no diff ever showed it. +A nonexistent log, an empty one and a record missing its verdict all returned **rc=0**. +An integrity failure that reads as a clean run is worse than no gate, because it +certifies. They now return 2, distinguishable from a real refusal at 1, and +`--registered` is required rather than silently skipped. ### `test_a_green_run_records_debt_and_never_a_red_observation` -The arm that matters most. A green run has seen nothing go red, so merging one must -never record a red observation — otherwise an ordinary CI run retires the debt the -ledger exists to count. - -### `test_a_red_observation_is_dated_and_survives_a_later_green_run` +A green run has observed nothing go red, so merging one must never record a red +observation — otherwise an ordinary CI run retires the debt the ledger exists to count. -The ledger records that a check **was** seen red, which stays true. +### `test_the_mutation_column_accumulates_rather_than_overwriting` -### `test_the_mutation_column_exists_from_v1` +Last-write-wins records the most recent attack rather than the catalogue the column +exists to become. One `--mutation` copied across several logs attributes a deliberate +change to failures it had nothing to do with, and is refused. -Empty when nothing named a mutation; recorded against the check that reddened and not -against one that stayed green. +### `test_two_runs_of_a_check_are_not_a_duplicate_of_it` -### `test_a_rename_is_reported_rather_than_silently_resetting_history` +Merging logs first cannot tell *the same check in two runs* from *the same name twice in +one run*, and reported the first as the second. -Keyed by the display string, a rename loses the check's history and reads exactly like a -brand-new check that has never been red — the one state this ledger exists to -distinguish. It cannot be prevented without a synthetic id someone would have to -maintain, so it is **detected**: a name that appeared while another disappeared is -named. Both directions are required, or every new check is reported as a rename and the -whole thing gets ignored. +### `test_renames_are_grouped_by_part_and_scanned_against_one_run` -### `test_a_duplicated_check_name_shares_one_row_and_is_reported` +A global positional pairing misses a real rename whenever unrelated movement in another +part shifts the ordering. Given a before-log and an after-log together the vanished name +is present in the union, so the scan **refuses** rather than silently finding nothing. -Two checks with the same name **in one part** share a ledger row, so one going red marks -**both** as observed red — a claim about a check nothing attacked. +### `test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers` -The key is `(suite, part, name)`, not `(suite, name)`. `harness_selftest` sources 40-odd -parts into one shell and phrases its premises to be **copied** — *"premise: the pytest -layer is where **this part** thinks it is"* works verbatim in any of them — so a -name-only key is a key of check *names*, and the collision count grows with every part -anyone writes. Measured over a real run of 583 records: 579 distinct `(suite, name)` -against 582 distinct `(suite, part, name)`. The part is derived from `BASH_SOURCE` -rather than from a convention, so the next part written the same way is keyed correctly -without anyone remembering. +The suite restriction is the *meaning* of `suites_not_covered`, not a softening: without +it the gate refuses every check of all 250 uncovered suites and reddens the whole matrix +on its first run. It tightens on its own as suites are seeded, and the deadlock that +shipped is pinned as its own arm — regenerating the ledger lets a new check through. -One duplicate survived that, and inspecting it showed it was worse than a repeat: at -`340:268` and `340:894`, `premise: the fixture fingerprints at all` was asked once of the -**source-partition fixture** and once of the **locale sweep** — two different questions -about two different code paths, wearing one sentence. A shared row there is not merely -imprecise: one failing would mark the other's premise as observed red. +### `test_the_ceiling_may_only_fall_and_that_is_enforced` -Both now name their subject, `premise: the partition fixture fingerprints at all` and -`premise: every locale produced a fingerprint`, and the corpus has **zero** collisions: -614 records, 614 distinct keys. - -That rename is also the detector's first test on real data rather than fixtures. Run -against the ledger seeded before it: - -``` -possible rename: premise: the fixture fingerprints at all - -> premise: every locale produced a fingerprint - (in harness_selftest/340-the-binary-must-be-built-from, history: last red never) -rename scan: appeared=2, vanished=1 -``` +The tracked file says the ceiling may only fall. Without a mechanism that is prose, and +raising the number passed. The gate compares against the previously committed value. -### `test_the_gate_refuses_a_check_the_ledger_has_never_seen` +### `test_the_runner_invokes_the_gate_before_it_removes_the_logs` -A gate that fails on 3,762 unledgered sites is one somebody disables under deadline, and -then we are back at `PGC_SKIP_TIMING` with extra steps. The budget grandfathers what -exists; a new check must not enter as silent debt. +A gate nothing runs is a comment. Nothing in the repository called this tool: zero +references in `.github/`, zero in the runner. ### `test_the_committed_ledger_and_budget_agree` -If they disagree, one was edited by hand. `suites_not_covered` is counted separately so -that "we ledger 605 checks" cannot read as "we ledger the corpus" — 250 of 251 suites -have no rows at all. +If they disagree, one was edited by hand. `suites_not_covered` is 250 of 251, so the +gate cannot refuse a new check in 250 suites — a real limit, counted rather than hidden, +which falls as suites are seeded. ## 17. Adding a test diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index b4a6ff00..9098a512 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -2,43 +2,46 @@ Nothing recorded whether a check had ever been red. That is the gap that let 39 checks across 35 suites ship unable to fail, three of them inside the suite whose -whole purpose is to stop exactly that. The gate answered *did anything print FAIL* -and had never answered *could anything print FAIL*. - -**What this ledger claims, and what it does not.** It records that a named check was -OBSERVED RED in a recorded run. It does not claim the check is proven able to fail: -that is a stronger statement, it needs a named mutation applied deliberately, and -conflating the two would put a claim in the ledger that nothing measured -- the -`defeated: 0` shape from `VACUITY_MODES` section 1, a number that reads as evidence -and is not. - -So v1 fills the observed column honestly and leaves the rest as debt, counted. A -ledger whose entries all read `never` is a measurement of how much of the corpus has -never been attacked, and that measurement is worth having on day one. - -**What fills it.** Not only deliberate mutation runs. Every real CI red fills it, -every flake, every bisect -- and those arrive whether anyone remembers or not. A -mutation run is the deliberate accelerator, not the only source. - -These tests drive the real tool, for the same reason the other two files drive the -real shell: a Python twin of a Python tool would agree with itself. +whole purpose is to stop exactly that. + +**What this records.** That a named check *was observed red in a recorded run*. Not +that it is proven able to fail: that needs a named mutation applied deliberately, and +conflating them would put a claim in the ledger nothing measured. + +**The first design deadlocked.** Bounding `checks_never_observed_red` means every +added check breaks the gate, because a new check enters as `never` -- so the only way +to land one was to raise a number the design said may only fall. It shipped at 614 +rows, 614 never, ceiling 614. It is now a CENSUS, asserted to match the ledger. The +CEILING is `suites_not_covered`, which adding a check does not move, and which the +gate refuses to see raised. + +**What the gate refuses** is a check the committed ledger has never seen, in a suite +the ledger covers. Regenerating the ledger is the intended fix and a reviewable diff. + +These tests drive the real tool, for the same reason the other files drive the real +shell: a Python twin of a Python tool would agree with itself. """ import pathlib import subprocess REPO = pathlib.Path(__file__).resolve().parents[2] -LEDGER_TOOL = REPO / "test" / "pgc_ledger.py" +TOOL = REPO / "test" / "pgc_ledger.py" RUNNER = REPO / "test" / "run_all_versions.sh" +GREEN = ("RESULT\tdemo\tpart1\tfirst check\tPASS\t\n" + "RESULT\tdemo\tpart1\tsecond check\tPASS\t\nchecks run: 2\n") +RED = ("RESULT\tdemo\tpart1\tfirst check\tFAIL\t\n" + "RESULT\tdemo\tpart1\tsecond check\tPASS\t\nchecks run: 2\n") + -def _run(*args): - r = subprocess.run(["python3", str(LEDGER_TOOL), *args], - capture_output=True, text=True) +def _run(*args, cwd=None): + r = subprocess.run(["python3", str(TOOL), *args], + capture_output=True, text=True, cwd=cwd) return r.stdout + r.stderr, r.returncode -def _write(tmp_path, name, text): +def _w(tmp_path, name, text): p = tmp_path / name p.write_text(text) return str(p) @@ -48,159 +51,221 @@ def _rows(path): return [l.split("\t") for l in pathlib.Path(path).read_text().splitlines() if l] -GREEN = ("RESULT\tdemo\tpart1\tfirst check\tPASS\t\n" - "RESULT\tdemo\tpart1\tsecond check\tPASS\t\n" - "checks run: 2\n") -RED = ("RESULT\tdemo\tpart1\tfirst check\tFAIL\t\n" - "RESULT\tdemo\tpart1\tsecond check\tPASS\t\n" - "checks run: 2\n") +def test_bad_input_is_an_integrity_failure_not_a_clean_run(tmp_path, expect): + """Every one of these returned rc=0 before. + + `read_records` ignored unreadable files, empty ones and short records, so a gate + over a NONEXISTENT log reported success. An integrity failure that reads as a + clean run is worse than no gate, because it certifies. Reported by @linuxhikerpm. + """ + ledger = _w(tmp_path, "l.tsv", "") + budget = _w(tmp_path, "b.txt", "suites_not_covered 0\n") + reg = _w(tmp_path, "reg", "demo\n") + empty = _w(tmp_path, "empty.log", "") + short = _w(tmp_path, "short.log", "RESULT\tdemo\tpart1\tname\n") + + for label, log in (("nonexistent", str(tmp_path / "nope.log")), + ("empty", empty), ("malformed", short)): + out, rc = _run("gate", "--ledger", ledger, "--budget", budget, + "--registered", reg, log) + expect.num(rc, 2, f"a {label} log is an integrity failure") + expect.num(out.count("ledger integrity failure"), 1, + f"and the {label} log says what was wrong with it") + + # It must stay distinguishable from a real refusal, or fail-closed just renames + # every outcome to the same thing. + good = _w(tmp_path, "g.log", GREEN) + expect.num(_run("gate", "--ledger", ledger, "--budget", budget, + "--registered", reg, good)[1], 1, + "a real refusal is a different status from an integrity failure") + + # --registered is required: skipping it silently is how a gate reports success + # for a question it never asked. + expect.num(_run("gate", "--ledger", ledger, "--budget", budget, good)[1], 2, + "the gate refuses to run without the registered suite list") def test_a_green_run_records_debt_and_never_a_red_observation(tmp_path, expect): - """The arm that matters most. - - A green run has seen nothing go red, so merging one must never record a red - observation -- otherwise an ordinary CI run retires the debt the ledger exists - to count. - """ - ledger = _write(tmp_path, "l.tsv", "") - log = _write(tmp_path, "green.log", GREEN) - _run("merge", "--ledger", ledger, log) + """A green run has seen nothing go red, so merging one must never record a red + observation -- otherwise an ordinary CI run retires the debt it exists to count.""" + ledger = _w(tmp_path, "l.tsv", "") + _run("merge", "--ledger", ledger, _w(tmp_path, "g.log", GREEN)) rows = _rows(ledger) expect.num(len(rows), 2, "merging a green run records both checks") expect.text(",".join(sorted({r[3] for r in rows})), "never", "and records neither as ever having been red") + expect.num(len([r for r in rows if len(r) != 5]), 0, "every row has five fields") + expect.num(pathlib.Path(ledger).read_text().count("\t\n"), 0, + "and no row ends in a tab, which was 614 of them") -def test_a_red_observation_is_dated_and_survives_a_later_green_run(tmp_path, expect): - """The ledger records that a check WAS seen red, which stays true.""" - ledger = _write(tmp_path, "l.tsv", "") - _run("merge", "--ledger", ledger, _write(tmp_path, "g.log", GREEN)) - _run("merge", "--ledger", ledger, "--date", "2026-09-10", - _write(tmp_path, "r.log", RED)) - by = {r[2]: r[3] for r in _rows(ledger)} - expect.text(by["first check"], "2026-09-10", - "a check observed red gains the date it was seen") - expect.text(by["second check"], "never", - "and one that stayed green keeps its debt") - - _run("merge", "--ledger", ledger, "--date", "2026-09-11", - _write(tmp_path, "g2.log", GREEN)) - expect.text({r[2]: r[3] for r in _rows(ledger)}["first check"], "2026-09-10", - "a later green run does not erase an observation") - +def test_the_mutation_column_accumulates_rather_than_overwriting(tmp_path, expect): + """Last-write-wins records the most recent attack rather than the catalogue the + column exists to become, which defeats its purpose rather than limiting it. -def test_the_mutation_column_exists_from_v1(tmp_path, expect): - """Present with nothing filling it automatically, because adding a column later - means rewriting every entry. - - If an entry can record WHICH mutation reddened a check, the catalogue a mutation - gate would need builds itself out of work people already do by hand. + And one `--mutation` copied across several logs attributes a deliberate change to + failures it had nothing to do with. Both reported by @linuxhikerpm. """ - ledger = _write(tmp_path, "l.tsv", "") - _run("merge", "--ledger", ledger, "--date", "2026-09-10", - _write(tmp_path, "r.log", RED)) - expect.num(len([r for r in _rows(ledger) if len(r) != 5]), 0, - "every row carries four fields, the fourth being the mutation") - expect.text("[" + {r[2]: r[4] for r in _rows(ledger)}["first check"] + "]", "[]", - "and it is empty when nothing named a mutation") - - _run("merge", "--ledger", ledger, "--date", "2026-09-10", - "--mutation", "SAOP limit 128 -> 0", _write(tmp_path, "r2.log", RED)) + ledger = _w(tmp_path, "l.tsv", "") + red = _w(tmp_path, "r.log", RED) + _run("merge", "--ledger", ledger, "--date", "D", "--mutation", "SAOP 128 -> 0", red) by = {r[2]: r[4] for r in _rows(ledger)} - expect.text(by["first check"], "SAOP limit 128 -> 0", + expect.text(by["first check"], "SAOP 128 -> 0", "a named mutation is recorded against the check that reddened") - expect.text("[" + by["second check"] + "]", "[]", - "and not against one that stayed green") - - -def test_a_rename_is_reported_rather_than_silently_resetting_history(tmp_path, expect): - """Keyed by the display string, a rename loses history and reads exactly like a - brand-new check that has never been red -- the one state this ledger exists to - distinguish. It is detected and named instead. - - Both directions are required: a check merely added, or merely removed, is not a - rename, and reporting one on every new check is noise that gets it ignored. + expect.text(by["second check"], "-", "and not against one that stayed green") + + _run("merge", "--ledger", ledger, "--date", "D", "--mutation", "bloom neutered", red) + expect.text({r[2]: r[4] for r in _rows(ledger)}["first check"], + "SAOP 128 -> 0;bloom neutered", + "a second mutation accumulates rather than replacing the first") + + expect.num(_run("merge", "--ledger", ledger, "--date", "D", "--mutation", "X", + red, _w(tmp_path, "g.log", GREEN))[1], 2, + "one mutation cannot be attributed across several runs at once") + + +def test_two_runs_of_a_check_are_not_a_duplicate_of_it(tmp_path, expect): + """Merging the logs first cannot tell "the same check in two runs" from "the same + name twice in one run", and reported the first as the second.""" + ledger = _w(tmp_path, "l.tsv", "") + g = _w(tmp_path, "g.log", GREEN) + out, _ = _run("merge", "--ledger", ledger, "--date", "D", g, g) + expect.num(out.count("duplicate"), 0, + "the same check in two logs is two runs, not a duplicate") + + twice = _w(tmp_path, "twice.log", + "RESULT\tdemo\tpart1\tsame\tPASS\t\n" + "RESULT\tdemo\tpart1\tsame\tFAIL\t\nchecks run: 2\n") + out, _ = _run("merge", "--ledger", _w(tmp_path, "l2.tsv", ""), "--date", "D", twice) + expect.num(out.count("duplicate check name in one run, so one ledger row covers 2: " + "demo\tpart1\tsame"), 1, + "the same name twice in ONE log is a duplicate, and is named") + + +def test_renames_are_grouped_by_part_and_scanned_against_one_run(tmp_path, expect): + """A global positional pairing misses a real rename whenever unrelated movement in + another part shifts the ordering. + + And given a before-log and an after-log together, the vanished name is present in + the union and nothing appears to have gone -- a scan that silently finds nothing is + worse than one that refuses. + """ + ledger = _w(tmp_path, "l.tsv", "") + before = _w(tmp_path, "b.log", + "RESULT\tdemo\tpartA\told A\tFAIL\t\n" + "RESULT\tdemo\tpartB\tstable B\tPASS\t\nchecks run: 2\n") + after = _w(tmp_path, "a.log", + "RESULT\tdemo\tpartA\tnew A\tPASS\t\n" + "RESULT\tdemo\tpartB\tstable B\tPASS\t\n" + "RESULT\tdemo\tpartB\tadded B\tPASS\t\nchecks run: 3\n") + _run("merge", "--ledger", ledger, "--date", "2026-09-01", before) + + out, rc = _run("rename-scan", "--ledger", ledger, after) + expect.num(out.count("possible rename: old A -> new A"), 1, + "a rename in one part survives an addition in another") + expect.num(out.count("last red 2026-09-01"), 1, + "and the history it is about to lose travels with it") + expect.num(out.count("added B"), 0, + "the addition in the other part is not called a rename") + expect.num(rc, 1, "and a detected rename is reported as a nonzero status") + + expect.num(_run("rename-scan", "--ledger", ledger, before, after)[1], 2, + "a before-log and an after-log together are refused, not silently empty") + + +def test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers(tmp_path, expect): + """The suite restriction is the MEANING of `suites_not_covered`, not a softening. + + Without it the gate refuses every check of all 250 uncovered suites and reddens the + whole matrix on its first run -- a gate somebody turns off within the week, which is + the failure this issue family exists to prevent. It tightens on its own as suites + are seeded. """ - ledger = _write(tmp_path, "l.tsv", "") - before = ("RESULT\tdemo\tpart1\tthe old name\tFAIL\t\n" - "RESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 2\n") - _run("merge", "--ledger", ledger, "--date", "2026-09-01", - _write(tmp_path, "b.log", before)) - - after = ("RESULT\tdemo\tpart1\tthe new name\tPASS\t\n" - "RESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 2\n") - out, _ = _run("rename-scan", "--ledger", ledger, _write(tmp_path, "a.log", after)) - expect.num(out.count("possible rename: the old name -> the new name"), 1, - "a name that appeared while another disappeared is reported") - expect.num(out.count("a stable check"), 0, "and the stable check is not") - - added = after.replace("the new name", "the old name") + "" - added = ("RESULT\tdemo\tpart1\tthe old name\tPASS\t\n" - "RESULT\tdemo\tpart1\ta stable check\tPASS\t\n" - "RESULT\tdemo\tpart1\ta genuinely new check\tPASS\t\nchecks run: 3\n") - out, _ = _run("rename-scan", "--ledger", ledger, _write(tmp_path, "add.log", added)) - expect.num(out.count("possible rename"), 0, - "a check merely added is not reported as a rename") - - removed = "RESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 1\n" - out, _ = _run("rename-scan", "--ledger", ledger, _write(tmp_path, "rm.log", removed)) - expect.num(out.count("possible rename"), 0, - "nor is one merely removed") - - -def test_a_duplicated_check_name_shares_one_row_and_is_reported(tmp_path, expect): - """Two checks with the same name in one suite share a ledger row, so one going - red marks BOTH as observed red -- a claim about a check nothing attacked, which - is exactly what this ledger must not make. - - It cannot be fixed by keying harder without a synthetic id someone would - maintain, so it is reported. The real corpus carries four today, which is how - this was noticed: 609 records reduced to 605 rows. + ledger = _w(tmp_path, "l.tsv", "") + reg = _w(tmp_path, "reg", "demo\nother\n") + _run("merge", "--ledger", ledger, "--date", "D", _w(tmp_path, "g.log", GREEN)) + + other = _w(tmp_path, "o.log", "RESULT\tother\tpartX\tsomething\tPASS\t\nchecks run: 1\n") + b1 = _w(tmp_path, "b1.txt", "suites_not_covered 1\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", b1, "--registered", reg, other) + expect.num(rc, 0, "a check in an uncovered suite is not refused") + expect.num(out.count("not covered=1"), 1, "but that suite is counted as debt") + + _run("merge", "--ledger", ledger, "--date", "D", other) + other2 = _w(tmp_path, "o2.log", + "RESULT\tother\tpartX\tsomething\tPASS\t\n" + "RESULT\tother\tpartX\tnewly added\tPASS\t\nchecks run: 2\n") + b0 = _w(tmp_path, "b0.txt", "suites_not_covered 0\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", b0, "--registered", reg, other2) + expect.num(rc, 1, "once the suite is covered, a new check in it IS refused") + expect.num(out.count("not in the ledger: other\tpartX\tnewly added"), 1, + "and it is the new one that is named") + expect.num(out.count("Regenerate it with"), 1, + "and the message says how to fix it, because that is the intended action") + + # THE DEADLOCK THAT SHIPPED, as its own arm: adding a check must not require an + # edit the design forbids. + _run("merge", "--ledger", ledger, "--date", "D", other2) + expect.num(_run("gate", "--ledger", ledger, "--budget", b0, + "--registered", reg, other2)[1], 0, + "regenerating the ledger lets the new check through") + expect.text({r[2]: r[3] for r in _rows(ledger)}["newly added"], "never", + "and it entered as debt, not as an observation nothing made") + + +def test_the_ceiling_may_only_fall_and_that_is_enforced(tmp_path, expect): + """The tracked file says the ceiling may only fall. Without a mechanism that + sentence is prose, and raising the number passed.""" + repo = tmp_path / "repo" + repo.mkdir() + for cmd in (["git", "init", "-q", "."], ["git", "config", "user.email", "t@t"], + ["git", "config", "user.name", "t"]): + subprocess.run(cmd, cwd=repo, capture_output=True) + (repo / "b.txt").write_text("suites_not_covered 5\n") + subprocess.run(["git", "add", "b.txt"], cwd=repo, capture_output=True) + subprocess.run(["git", "commit", "-qm", "base"], cwd=repo, capture_output=True) + + prior = subprocess.run(["git", "show", "HEAD:b.txt"], cwd=repo, + capture_output=True, text=True).stdout + expect.num(prior.count("suites_not_covered 5"), 1, + "premise: the scratch repo has a prior ceiling committed") + + ledger = _w(tmp_path, "l.tsv", "") + reg = _w(tmp_path, "reg", "demo\n") + _run("merge", "--ledger", ledger, "--date", "D", _w(tmp_path, "g.log", GREEN)) + log = _w(tmp_path, "g2.log", GREEN) + + (repo / "b.txt").write_text("suites_not_covered 9\n") + out, rc = _run("gate", "--ledger", ledger, "--budget", "b.txt", + "--registered", reg, "--against", "HEAD", log, cwd=repo) + expect.num(rc, 1, "raising the ceiling above its committed value is refused") + expect.num(out.count("was raised from 5 to 9"), 1, "and the refusal names both values") + + (repo / "b.txt").write_text("suites_not_covered 3\n") + expect.num(_run("gate", "--ledger", ledger, "--budget", "b.txt", + "--registered", reg, "--against", "HEAD", log, cwd=repo)[1], 0, + "lowering it is allowed, which is the direction the burn-down goes") + + +def test_the_runner_invokes_the_gate_before_it_removes_the_logs(expect): + """A gate nothing runs is a comment -- selftest 350's phrasing about its own + subject. Nothing in the repository called this tool: zero references in `.github/`, + zero in the runner. Reported by @linuxhikerpm and by OffgridwithJD independently. """ - ledger = _write(tmp_path, "l.tsv", "") - dupe = ("RESULT\tdemo\tpart1\tthe same name\tPASS\t\n" - "RESULT\tdemo\tpart1\tthe same name\tFAIL\t\n" - "RESULT\tdemo\tpart1\ta unique name\tPASS\t\nchecks run: 3\n") - out, _ = _run("merge", "--ledger", ledger, "--date", "2026-09-10", - _write(tmp_path, "d.log", dupe)) - expect.num(out.count("duplicate check name, so one ledger row covers 2: " - "demo\tpart1\tthe same name"), 1, - "a duplicated check name is reported by name") - expect.num(out.count("a unique name"), 0, "and a unique one is not") - expect.num(len(_rows(ledger)), 2, - "the two collapse to one row, which is the loss being reported") - - -def test_the_gate_refuses_a_check_the_ledger_has_never_seen(tmp_path, expect): - """A gate that fails on 3,762 unledgered sites is one somebody disables under - deadline, and then we are back at PGC_SKIP_TIMING with extra steps. So the - budget grandfathers what exists -- but a NEW check must not enter as silent - debt either.""" - ledger = _write(tmp_path, "l.tsv", "") - _run("merge", "--ledger", ledger, _write(tmp_path, "g.log", GREEN)) - budget = _write(tmp_path, "b.txt", "checks_never_observed_red 2\n") - log = _write(tmp_path, "g2.log", GREEN) - expect.num(_run("gate", "--ledger", ledger, "--budget", budget, log)[1], 0, - "a run whose debt is within budget passes") - - over = _write(tmp_path, "b2.txt", "checks_never_observed_red 1\n") - out, rc = _run("gate", "--ledger", ledger, "--budget", over, log) - expect.num(rc, 1, "and one over budget does not") - expect.num(out.count("checks_never_observed_red: 2 exceeds the budget of 1"), 1, - "and the gate says which number was exceeded, by how much") - - newer = _write(tmp_path, "n.log", GREEN + "RESULT\tdemo\tpart1\tbrand new\tPASS\t\n") - out, rc = _run("gate", "--ledger", ledger, "--budget", budget, newer) - expect.num(rc, 1, "a check the ledger has never seen is refused") - expect.num(out.count("not in the ledger: demo\tpart1\tbrand new"), 1, - "and it is named, so the author knows which one") + text = RUNNER.read_text().splitlines() + call = [i for i, l in enumerate(text) if 'pgc_ledger.py" gate' in l] + teardown = [i for i, l in enumerate(text) if 'rm -rf "$builddir"' in l] + expect.num(len(call), 1, "the runner invokes the ledger gate exactly once") + expect.at_least(len(teardown), 1, "premise: the runner removes the build directory") + expect.text("before" if call[0] < teardown[-1] else "after", "before", + "and it runs before the logs are removed, the only place it can") + window = "\n".join(text[call[0]:call[0] + 8]) + expect.num(window.count("verfail=1"), 1, "and a refused gate fails the major") def test_the_committed_ledger_and_budget_agree(expect): - """Both are tracked files, so a change to either is a diff a reviewer sees. If - they disagree, one of them was edited by hand -- the failure this whole design - refuses.""" + """Both are tracked, so a change to either is a diff a reviewer sees. If they + disagree, one was edited by hand -- the failure this design refuses.""" ledger = REPO / "test" / "check_ledger.tsv" budget = REPO / "test" / "check_ledger_budget.txt" expect.text("yes" if ledger.exists() else "no", "yes", "the ledger is in the tree") @@ -209,21 +274,21 @@ def test_the_committed_ledger_and_budget_agree(expect): rows = [l.split("\t") for l in ledger.read_text().splitlines() if l] never = [r for r in rows if r[3] == "never"] red = [r for r in rows if r[3] != "never"] - print(f" ledger: inputs={len(rows)} | observed red={len(red)}, " - f"never={len(never)} | sum={len(red) + len(never)}") + print(f" ledger: inputs={len(rows)} | observed red={len(red)}, never={len(never)}") expect.num(len(red) + len(never), len(rows), "the ledger partitions") - expect.at_least(len(rows), 1, "premise: it is not empty") + expect.num(len([r for r in rows if len(r) != 5]), 0, "every committed row has five fields") + expect.num(ledger.read_text().count("\t\n"), 0, "and none ends in a tab") nums = {} for line in budget.read_text().splitlines(): - parts = line.split() - if len(parts) == 2 and parts[1].isdigit() and not line.startswith("#"): - nums[parts[0]] = int(parts[1]) + p = line.split() + if len(p) == 2 and p[1].isdigit() and not line.startswith("#"): + nums[p[0]] = int(p[1]) expect.num(nums.get("checks_never_observed_red", -1), len(never), - "the committed budget matches the committed ledger's debt") + "the committed census matches the committed ledger") listed = subprocess.run(["bash", str(RUNNER), "--list-suites"], capture_output=True, text=True).stdout.split() covered = {r[0] for r in rows} expect.num(nums.get("suites_not_covered", -1), len(set(listed) - covered), - "and the coverage debt matches the suites with no rows") + "and the ceiling matches the suites with no rows") diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index 6ef2e84f..43b367e4 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -1304,6 +1304,45 @@ pgc_tally_suite() { # pgc_tally_suite NAME VERDICT LOGFILE verfail=1 fi + # THE LEDGER GATE (#918). Every suite's log is here and the build directory is + # about to be removed, so this is the only place a matrix run can feed it. + # + # The gate runs against the COMMITTED ledger: a check it has never seen is + # named and refused, which is the allowlist the issue asks for. Regenerating + # the ledger is the intended fix and a reviewable diff, so this cannot + # deadlock the way a ceiling on `never` rows did. + # + # CI verifies; humans commit. A ledger that CI rewrote by itself would be a + # file nobody reads changing under everybody. + # + # Only logs that CARRY records are passed. The twelve suites outside lib.sh's + # accounting produce none, and the tool fails closed on an empty input -- + # correctly, since a caller asking it to reconcile nothing is a caller with a + # bug. + _led_logs="" + for s in "${SUITES[@]}"; do + [ -s "$builddir/${s}.log" ] || continue + [ "$(grep -c '^RESULT ' "$builddir/${s}.log" || true)" != 0 ] \ + && _led_logs="$_led_logs $builddir/${s}.log" + done + if [ -z "$_led_logs" ]; then + echo " no suite emitted a check record on PG$major, so the ledger has nothing to gate" + verfail=1 + elif [ ! -f "$builddir/test/check_ledger.tsv" ]; then + echo " the ledger is missing from the tree under test, which is not a pass" + verfail=1 + else + # shellcheck disable=SC2086 + if ! python3 "$builddir/test/pgc_ledger.py" gate \ + --ledger "$builddir/test/check_ledger.tsv" \ + --budget "$builddir/test/check_ledger_budget.txt" \ + --registered "$_acc_registered" \ + $_led_logs; then + echo " PG$major has a check the ledger has never seen, which is not a pass" + verfail=1 + fi + fi + # How many of the suites counted as having RUN actually accounted for their # checks (#916). Ten registered suites exit 0 having never called pgc_summary; # counting them among the suites that ran is the overcount #447 added this diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 18d4874e..5918fc28 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -1,22 +1,25 @@ # ---- a check must have been seen red, or be counted as debt ----------------- # -# Nothing records whether a check has ever been red. That is the gap that let 39 +# Nothing recorded whether a check had ever been red. That is the gap that let 39 # checks across 35 suites ship unable to fail, three of them inside this very -# suite. The gate answers "did anything print FAIL" and has never answered "could -# anything print FAIL". +# suite. The gate answered "did anything print FAIL" and had never answered +# "could anything print FAIL". # -# An audit fixes today; only a ledger keeps it fixed. And a ledger somebody -# maintains is a hand-maintained count, which this repository has spent a day -# proving the cost of: nine collisions on one written number, both sides wrong -# every time. So the ledger is DERIVED FROM RUNS. The only hand-written numbers -# are the two budgets, and they may only go down. +# WHAT THIS RECORDS, AND WHAT IT DOES NOT. It records that a named check WAS +# OBSERVED RED in a recorded run. It does NOT claim the check is proven able to +# fail: that needs a named mutation applied deliberately, and conflating the two +# would put a claim in the ledger that nothing measured. # -# WHAT THIS LEDGER CLAIMS, AND WHAT IT DOES NOT. It records that a named check -# WAS OBSERVED RED in a recorded run. It does NOT claim the check is proven able -# to fail: that is a stronger statement, it needs a named mutation applied -# deliberately, and conflating the two would put a claim in the ledger that -# nothing measured. v1 fills the observed column honestly and leaves the rest as -# debt, counted. +# THE FIRST DESIGN DEADLOCKED AND THE SECOND DOES NOT. Bounding +# `checks_never_observed_red` means every added check breaks the gate, because a +# new check enters as `never` -- so the only way to land one was to raise a number +# the design said may only fall. It shipped at 614 rows, 614 never, ceiling 614. +# It is now a CENSUS, asserted to match the ledger; the CEILING is +# `suites_not_covered`, which adding a check does not move. +# +# WHAT THE GATE REFUSES is a check the committed ledger has never seen. Existing +# checks are grandfathered; a new one is named, and regenerating the ledger is the +# INTENDED fix rather than a forbidden edit. # --------------------------------------------------------------------------- _led="$PGC_TESTDIR/pgc_ledger.py" @@ -31,219 +34,251 @@ check "premise: the budget is a tracked file too" \ _lw="$PGC_WORKDIR/ledger"; mkdir -p "$_lw" _led_run() { python3 "$_led" "$@" 2>&1; } +_led_rc() { python3 "$_led" "$@" >/dev/null 2>&1; echo $?; } + +printf 'RESULT\tdemo\tpart1\tfirst check\tPASS\t\nRESULT\tdemo\tpart1\tsecond check\tPASS\t\nchecks run: 2\n' > "$_lw/green.log" +printf 'RESULT\tdemo\tpart1\tfirst check\tFAIL\t\nRESULT\tdemo\tpart1\tsecond check\tPASS\t\nchecks run: 2\n' > "$_lw/red.log" +printf 'demo\n' > "$_lw/registered" +printf 'suites_not_covered 0\n' > "$_lw/budget.txt" -# ---- the census comes out of a run, not out of a list ----------------------- +# ---- fail closed. Every one of these returned rc=0 before ------------------- +# +# read_records ignored unreadable files, empty ones and short records, so a gate +# over a NONEXISTENT log reported success. An integrity failure that reads as a +# clean run is worse than no gate, because it certifies. Reported by @linuxhikerpm. -cat > "$_lw/green.log" <<'LOG' -RESULT demo part1 first check PASS -RESULT demo part1 second check PASS -checks run: 2 -LOG +: > "$_lw/empty.log" +printf 'RESULT\tdemo\tpart1\tname\n' > "$_lw/short.log" +: > "$_lw/l.tsv" +check "a gate over a nonexistent log is an integrity failure, not a pass" \ + "$(_led_rc gate --ledger "$_lw/l.tsv" --budget "$_lw/budget.txt" --registered "$_lw/registered" "$_lw/nope.log")" "2" +check "an empty log is one too, because there is nothing to reconcile" \ + "$(_led_rc gate --ledger "$_lw/l.tsv" --budget "$_lw/budget.txt" --registered "$_lw/registered" "$_lw/empty.log")" "2" +check "and a record missing its verdict" \ + "$(_led_rc gate --ledger "$_lw/l.tsv" --budget "$_lw/budget.txt" --registered "$_lw/registered" "$_lw/short.log")" "2" +check "each says what was wrong with the input" \ + "$(_led_run gate --ledger "$_lw/l.tsv" --budget "$_lw/budget.txt" --registered "$_lw/registered" "$_lw/short.log" \ + | grep -c 'a record needs suite, part, name and verdict')" "1" -check "the census reads a run's records" \ - "$(_led_run census "$_lw/green.log" | wc -l)" "2" -check "and names the suite and the check, not just a count" \ - "$(_led_run census "$_lw/green.log" | head -1)" "demo part1 first check PASS" +# The three must be distinguishable from a REAL refusal, or fail-closed just +# renames every outcome. +check "a real refusal is a different status from an integrity failure" \ + "$(_led_rc gate --ledger "$_lw/l.tsv" --budget "$_lw/budget.txt" --registered "$_lw/registered" "$_lw/green.log")" "1" -# ---- merging a green run adds the checks as DEBT, not as proven ------------- -# -# The arm that matters. A green run has seen nothing go red, so merging one must -# never record a red observation. Anything else would let an ordinary CI run -# retire the debt it exists to count. +# --registered is required. Skipping it silently is how a gate reports success +# for a question it never asked. +check "the gate refuses to run without the registered suite list" \ + "$(_led_rc gate --ledger "$_lw/l.tsv" --budget "$_lw/budget.txt" "$_lw/green.log")" "2" + +# ---- the census: a green run records debt and never a red observation ------- : > "$_lw/ledger.tsv" -_led_run merge --ledger "$_lw/ledger.tsv" "$_lw/green.log" >/dev/null -check "merging a green run records both checks" \ - "$(grep -c . "$_lw/ledger.tsv")" "2" +_led_run merge --ledger "$_lw/ledger.tsv" --date 2026-09-10 "$_lw/green.log" >/dev/null +check "merging a green run records both checks" "$(grep -c . "$_lw/ledger.tsv")" "2" check "and records neither as ever having been red" \ "$(cut -f4 "$_lw/ledger.tsv" | sort -u | tr '\n' ' ')" "never " - -# ---- merging a run that DID go red records the observation ------------------ - -cat > "$_lw/red.log" <<'LOG' -RESULT demo part1 first check FAIL -RESULT demo part1 second check PASS -checks run: 2 -LOG +check "every row has five fields and no trailing tab" \ + "$(awk -F'\t' 'NF!=5' "$_lw/ledger.tsv" | grep -c . || true)" "0" +check "and an empty mutation is a placeholder, not an empty last field" \ + "$(grep -cP '\t$' "$_lw/ledger.tsv" || true)" "0" _led_run merge --ledger "$_lw/ledger.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null check "a check observed red gains the date it was seen" \ "$(awk -F'\t' '$3=="first check"{print $4}' "$_lw/ledger.tsv")" "2026-09-10" -check "and a check that stayed green keeps its debt" \ +check "and one that stayed green keeps its debt" \ "$(awk -F'\t' '$3=="second check"{print $4}' "$_lw/ledger.tsv")" "never" - -# An observation is not undone by a later green run. The ledger records that the -# check WAS seen red, which stays true. _led_run merge --ledger "$_lw/ledger.tsv" --date 2026-09-11 "$_lw/green.log" >/dev/null check "a later green run does not erase an observation" \ "$(awk -F'\t' '$3=="first check"{print $4}' "$_lw/ledger.tsv")" "2026-09-10" -# ---- the gate: new checks must not be added to the debt silently ------------ -# -# A gate that fails on 3,762 unledgered checks is a gate somebody disables under -# deadline, and then we are back at PGC_SKIP_TIMING with extra steps. So the -# budget grandfathers what exists and refuses to grow. - -printf 'suites_not_covered 0\nchecks_never_observed_red 1\n' > "$_lw/budget.txt" -check "a run whose debt is within budget passes the gate" \ - "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/green.log" >/dev/null 2>&1 \ - && echo ok || echo over)" "ok" - -printf 'suites_not_covered 0\nchecks_never_observed_red 0\n' > "$_lw/budget.txt" -check "and one over budget does not" \ - "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/green.log" >/dev/null 2>&1 \ - && echo ok || echo over)" "over" -check "and the gate says which number was exceeded, by how much" \ - "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/green.log" 2>&1 \ - | grep -c 'checks_never_observed_red: 1 exceeds the budget of 0')" "1" - -# A check the run produced that the ledger has never heard of is the case the -# allowlist exists for: it is NEW, and it must not enter as silent debt. -printf 'suites_not_covered 0\nchecks_never_observed_red 1\n' > "$_lw/budget.txt" -cat > "$_lw/newcheck.log" <<'LOG' -RESULT demo part1 first check PASS -RESULT demo part1 second check PASS -RESULT demo part1 a brand new check PASS -checks run: 3 -LOG -check "a check the ledger has never seen is refused, not absorbed" \ - "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/newcheck.log" >/dev/null 2>&1 \ - && echo ok || echo refused)" "refused" -check "and it is named, so the author knows which one" \ - "$(_led_run gate --ledger "$_lw/ledger.tsv" --budget "$_lw/budget.txt" "$_lw/newcheck.log" 2>&1 \ - | grep -c 'not in the ledger: demo part1 a brand new check')" "1" - -# ---- the budget may only go DOWN -------------------------------------------- +# ---- the mutation column ACCUMULATES ---------------------------------------- # -# Both numbers are debt. A change that raises either is a change that adds debt, -# and it must be visible in a diff as exactly that rather than as a passing gate. - -check "the committed budget names both debts" \ - "$(grep -cE '^(suites_not_covered|checks_never_observed_red) [0-9]+$' "$_budget")" "2" +# Last-write-wins records the most recent attack rather than the catalogue the +# column exists to become, which defeats its stated purpose rather than limiting +# it. And one --mutation value copied across several logs attributes a deliberate +# change to failures it had nothing to do with. Both reported by @linuxhikerpm. -# ---- inputs == sum(buckets), over the real ledger --------------------------- +: > "$_lw/mut.tsv" +_led_run merge --ledger "$_lw/mut.tsv" --date D --mutation 'SAOP limit 128 -> 0' "$_lw/red.log" >/dev/null +check "a named mutation is recorded against the check that reddened" \ + "$(awk -F'\t' '$3=="first check"{print $5}' "$_lw/mut.tsv")" "SAOP limit 128 -> 0" +check "and not against one that stayed green" \ + "$(awk -F'\t' '$3=="second check"{print $5}' "$_lw/mut.tsv")" "-" +_led_run merge --ledger "$_lw/mut.tsv" --date D --mutation 'bloom neutered' "$_lw/red.log" >/dev/null +check "a second mutation ACCUMULATES rather than replacing the first" \ + "$(awk -F'\t' '$3=="first check"{print $5}' "$_lw/mut.tsv")" "SAOP limit 128 -> 0;bloom neutered" +check "one --mutation cannot be attributed across several runs at once" \ + "$(_led_rc merge --ledger "$_lw/mut.tsv" --date D --mutation X "$_lw/red.log" "$_lw/green.log")" "2" -_l_total="$(grep -c . "$_ledger" || true)" -_l_red="$(awk -F'\t' '$4!="never"' "$_ledger" | grep -c . || true)" -_l_never="$(awk -F'\t' '$4=="never"' "$_ledger" | grep -c . || true)" -echo " ledger: inputs=$_l_total | observed red=$_l_red, never=$_l_never | sum=$((_l_red + _l_never))" -check "the ledger partitions into observed and never" \ - "$((_l_red + _l_never))" "$_l_total" -check "premise: the ledger is not empty, so the partition means something" \ - "$([ "$_l_total" -gt 0 ] && echo yes || echo no)" "yes" +# ---- two runs of a check are not a duplicate of it -------------------------- +# +# Merging the logs first cannot tell "the same check in two runs" from "the same +# name twice in one run", and reported the first as the second. -# The committed budget must match the committed ledger. If it does not, one of -# the two was edited by hand -- which is the failure this whole design refuses. -check "the committed budget matches the committed ledger's debt" \ - "$(sed -n 's/^checks_never_observed_red //p' "$_budget")" "$_l_never" +: > "$_lw/dup.tsv" +check "the same check in two logs is two runs, not a duplicate" \ + "$(_led_run merge --ledger "$_lw/dup.tsv" --date D "$_lw/green.log" "$_lw/green.log" | grep -c 'duplicate')" "0" +printf 'RESULT\tdemo\tpart1\tsame\tPASS\t\nRESULT\tdemo\tpart1\tsame\tFAIL\t\nchecks run: 2\n' > "$_lw/twice.log" +: > "$_lw/dup2.tsv" +check "the same name twice in ONE log is a duplicate, and is named" \ + "$(_led_run merge --ledger "$_lw/dup2.tsv" --date D "$_lw/twice.log" \ + | grep -c 'duplicate check name in one run, so one ledger row covers 2: demo part1 same')" "1" -# ---- a rename is not a new check, and must not look like one ---------------- -# -# The ledger is keyed by check NAME, and check names in this harness are prose -- -# they are renamed freely, which is most of why #917 exists. So a rename loses -# the check's history and reads exactly like a brand-new check that has never -# been red, which is the ONE state the ledger exists to distinguish. -# -# Raised by OffgridwithJD, who also named the detector: a name appearing with no -# history in the same run another disappears is a rename, and the ledger should -# SAY so rather than quietly resetting a count to `never`. It is the same -# both-directions set comparison as the suite reconciliation, over check names. +# ---- renames, grouped by part and scanned against ONE run ------------------- # -# The alternative -- a synthetic stable id -- would have to be maintained, and -# this repository removed a hand-maintained list today for that exact reason. +# A global positional pairing misses a real rename whenever unrelated movement in +# another part shifts the ordering. And given a before-log and an after-log +# together, the vanished name is present in the union and nothing appears to have +# gone -- a scan that silently finds nothing is worse than one that refuses. : > "$_lw/ren.tsv" -cat > "$_lw/before.log" <<'LOG' -RESULT demo part1 the old name FAIL -RESULT demo part1 a stable check PASS -checks run: 2 -LOG +printf 'RESULT\tdemo\tpart1\tthe old name\tFAIL\t\nRESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 2\n' > "$_lw/before.log" +printf 'RESULT\tdemo\tpart1\tthe new name\tPASS\t\nRESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 2\n' > "$_lw/after.log" _led_run merge --ledger "$_lw/ren.tsv" --date 2026-09-01 "$_lw/before.log" >/dev/null check "premise: the check has history before the rename" \ "$(awk -F'\t' '$3=="the old name"{print $4}' "$_lw/ren.tsv")" "2026-09-01" - -cat > "$_lw/after.log" <<'LOG' -RESULT demo part1 the new name PASS -RESULT demo part1 a stable check PASS -checks run: 2 -LOG check "a name that appeared while another disappeared is reported as a rename" \ - "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/after.log" 2>&1 \ + "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/after.log" \ | grep -c 'possible rename: the old name -> the new name')" "1" -check "and the stable check is not reported" \ - "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/after.log" 2>&1 \ - | grep -c 'a stable check')" "0" - -# The detector must not fire when a check is simply ADDED. Without this it names -# a rename on every new check, which is noise that gets it ignored. -cat > "$_lw/added.log" <<'LOG' -RESULT demo part1 the old name PASS -RESULT demo part1 a stable check PASS -RESULT demo part1 a genuinely new check PASS -checks run: 3 -LOG +check "and the history it is about to lose travels with it" \ + "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/after.log" | grep -c 'last red 2026-09-01')" "1" +check "the stable check is not reported" \ + "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/after.log" | grep -c 'a stable check')" "0" +check "a before-log and an after-log together are refused, not silently empty" \ + "$(_led_rc rename-scan --ledger "$_lw/ren.tsv" "$_lw/before.log" "$_lw/after.log")" "2" + +# Movement in ANOTHER part must not consume this part's pairing. That is what a +# global positional zip gets wrong, and it fails silently. +: > "$_lw/ren2.tsv" +printf 'RESULT\tdemo\tpartA\told A\tPASS\t\nRESULT\tdemo\tpartB\tstable B\tPASS\t\nchecks run: 2\n' > "$_lw/b2.log" +printf 'RESULT\tdemo\tpartA\tnew A\tPASS\t\nRESULT\tdemo\tpartB\tstable B\tPASS\t\nRESULT\tdemo\tpartB\tadded B\tPASS\t\nchecks run: 3\n' > "$_lw/a2.log" +_led_run merge --ledger "$_lw/ren2.tsv" --date D "$_lw/b2.log" >/dev/null +check "a rename in one part survives an addition in another" \ + "$(_led_run rename-scan --ledger "$_lw/ren2.tsv" "$_lw/a2.log" \ + | grep -c 'possible rename: old A -> new A')" "1" +check "and the addition in the other part is not called a rename" \ + "$(_led_run rename-scan --ledger "$_lw/ren2.tsv" "$_lw/a2.log" | grep -c 'added B')" "0" + +# A check merely added, or merely removed, is not a rename. +printf 'RESULT\tdemo\tpart1\tthe old name\tPASS\t\nRESULT\tdemo\tpart1\ta stable check\tPASS\t\nRESULT\tdemo\tpart1\tbrand new\tPASS\t\nchecks run: 3\n' > "$_lw/added.log" check "a check merely added is not reported as a rename" \ - "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/added.log" 2>&1 \ - | grep -c 'possible rename')" "0" - -# Nor when one is simply REMOVED. -cat > "$_lw/removed.log" <<'LOG' -RESULT demo part1 a stable check PASS -checks run: 1 -LOG -check "a check merely removed is not reported as a rename either" \ - "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/removed.log" 2>&1 \ - | grep -c 'possible rename')" "0" - -# ---- the mutation field, present from v1 even though nothing fills it ------- + "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/added.log" | grep -c 'possible rename')" "0" +printf 'RESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 1\n' > "$_lw/removed.log" +check "nor is one merely removed" \ + "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/removed.log" | grep -c 'possible rename')" "0" + +# ---- the gate refuses a check the ledger has never seen --------------------- + +: > "$_lw/g.tsv" +_led_run merge --ledger "$_lw/g.tsv" --date D "$_lw/green.log" >/dev/null +printf 'suites_not_covered 0\n' > "$_lw/gb.txt" +check "a run whose checks are all ledgered passes the gate" \ + "$(_led_rc gate --ledger "$_lw/g.tsv" --budget "$_lw/gb.txt" --registered "$_lw/registered" "$_lw/green.log")" "0" +printf 'RESULT\tdemo\tpart1\tfirst check\tPASS\t\nRESULT\tdemo\tpart1\tsecond check\tPASS\t\nRESULT\tdemo\tpart1\tbrand new\tPASS\t\nchecks run: 3\n' > "$_lw/new.log" +check "a check the ledger has never seen is refused" \ + "$(_led_rc gate --ledger "$_lw/g.tsv" --budget "$_lw/gb.txt" --registered "$_lw/registered" "$_lw/new.log")" "1" +check "and it is named, so the author knows which one" \ + "$(_led_run gate --ledger "$_lw/g.tsv" --budget "$_lw/gb.txt" --registered "$_lw/registered" "$_lw/new.log" \ + | grep -c 'not in the ledger: demo part1 brand new')" "1" +check "and the message says how to fix it, because regenerating is the intended action" \ + "$(_led_run gate --ledger "$_lw/g.tsv" --budget "$_lw/gb.txt" --registered "$_lw/registered" "$_lw/new.log" \ + | grep -c 'Regenerate it with')" "1" + +# THE DEADLOCK THAT SHIPPED, as its own arm. Adding a check must not require an +# edit the design forbids. +_led_run merge --ledger "$_lw/g.tsv" --date D "$_lw/new.log" >/dev/null +check "regenerating the ledger lets the new check through" \ + "$(_led_rc gate --ledger "$_lw/g.tsv" --budget "$_lw/gb.txt" --registered "$_lw/registered" "$_lw/new.log")" "0" +check "and it entered as debt, not as an observation nothing made" \ + "$(awk -F'\t' '$3=="brand new"{print $4}' "$_lw/g.tsv")" "never" + +# ---- the ceiling is monotone, mechanically ---------------------------------- # -# If an entry can record WHICH mutation reddened a check, the mutation catalogue -# builds itself out of work people already do by hand -- the vacuity branches are -# writing nine to eleven per change tonight, each chosen to revert one property. -# OffgridwithJD's point, and the reason the column exists now: adding it later -# means rewriting every entry. +# The file says the ceiling may only fall. Without this the sentence is prose: +# raising the number passed. Measured against a prior value from git rather than +# taken on trust. + +check "the ceiling refuses being exceeded" \ + "$(printf 'suites_not_covered 0\n' > "$_lw/gb0.txt" + printf 'other\ndemo\n' > "$_lw/reg2" + _led_rc gate --ledger "$_lw/g.tsv" --budget "$_lw/gb0.txt" --registered "$_lw/reg2" "$_lw/new.log")" "1" + +_lg="$_lw/repo"; rm -rf "$_lg"; mkdir -p "$_lg" +( cd "$_lg" && git init -q . && git config user.email t@t && git config user.name t + printf 'suites_not_covered 5\n' > b.txt && git add b.txt && git commit -qm base ) >/dev/null 2>&1 +check "premise: the scratch repo has a prior ceiling committed" \ + "$(cd "$_lg" && git show HEAD:b.txt | grep -c 'suites_not_covered 5')" "1" +printf 'suites_not_covered 9\n' > "$_lg/b.txt" +check "raising the ceiling above its committed value is refused" \ + "$(cd "$_lg" && _led_rc gate --ledger "$_lw/g.tsv" --budget b.txt \ + --registered "$_lw/registered" --against HEAD "$_lw/new.log")" "1" +check "and the refusal names both values" \ + "$(cd "$_lg" && _led_run gate --ledger "$_lw/g.tsv" --budget b.txt \ + --registered "$_lw/registered" --against HEAD "$_lw/new.log" \ + | grep -c 'was raised from 5 to 9')" "1" +printf 'suites_not_covered 3\n' > "$_lg/b.txt" +check "lowering it is allowed, which is the direction the burn-down goes" \ + "$(cd "$_lg" && _led_rc gate --ledger "$_lw/g.tsv" --budget b.txt \ + --registered "$_lw/registered" --against HEAD "$_lw/new.log")" "0" + +# ---- and the RUNNER must invoke it ------------------------------------------ # -# NOTHING FILLS IT AUTOMATICALLY YET, and the arms say so rather than implying a -# capability that does not exist. +# A gate nothing runs is a comment, which is selftest 350's phrasing about its own +# subject. Nothing in the repository called this tool: zero references in +# .github/, zero in the runner. Reported by @linuxhikerpm and by OffgridwithJD +# independently. -: > "$_lw/mut.tsv" -_led_run merge --ledger "$_lw/mut.tsv" --date 2026-09-10 "$_lw/red.log" >/dev/null -check "every ledger row carries four fields, the fourth being the mutation" \ - "$(awk -F'\t' 'NF!=5' "$_lw/mut.tsv" | grep -c . || true)" "0" -check "and it is empty when nothing named a mutation" \ - "$(awk -F'\t' '$3=="first check"{print "[" $5 "]"}' "$_lw/mut.tsv")" "[]" - -_led_run merge --ledger "$_lw/mut.tsv" --date 2026-09-10 \ - --mutation 'PGCOLUMNAR_SAOP_ELEMENT_LIMIT 128 -> 0' "$_lw/red.log" >/dev/null -check "a merge that names its mutation records it against the check that reddened" \ - "$(awk -F'\t' '$3=="first check"{print $5}' "$_lw/mut.tsv")" "PGCOLUMNAR_SAOP_ELEMENT_LIMIT 128 -> 0" -check "and not against one that stayed green" \ - "$(awk -F'\t' '$3=="second check"{print "[" $5 "]"}' "$_lw/mut.tsv")" "[]" +check "the runner invokes the ledger gate" \ + "$(grep -c 'pgc_ledger.py" gate' "$_rv")" "1" +check "and it runs before the build directory is removed, which is the only place it can" \ + "$([ "$(grep -n 'pgc_ledger.py" gate' "$_rv" | cut -d: -f1)" -lt \ + "$(grep -n 'rm -rf "\$builddir"' "$_rv" | tail -1 | cut -d: -f1)" ] && echo before || echo after)" "before" +check "and a refused gate fails the major" \ + "$(grep -A8 'pgc_ledger.py" gate' "$_rv" | grep -c 'verfail=1')" "1" -# ---- a duplicated check name shares one ledger row -------------------------- +# ---- the committed files agree ---------------------------------------------- + +_l_total="$(grep -c . "$_ledger" || true)" +_l_red="$(awk -F'\t' '$4!="never"' "$_ledger" | grep -c . || true)" +_l_never="$(awk -F'\t' '$4=="never"' "$_ledger" | grep -c . || true)" +echo " ledger: inputs=$_l_total | observed red=$_l_red, never=$_l_never | sum=$((_l_red + _l_never))" +check "the ledger partitions into observed and never" "$((_l_red + _l_never))" "$_l_total" +check "premise: the ledger is not empty, so the partition means something" \ + "$([ "$_l_total" -gt 0 ] && echo yes || echo no)" "yes" +check "every committed row has five fields" \ + "$(awk -F'\t' 'NF!=5' "$_ledger" | grep -c . || true)" "0" +check "and none of them ends in a tab" "$(grep -cP '\t$' "$_ledger" || true)" "0" +check "the committed census matches the committed ledger" \ + "$(sed -n 's/^checks_never_observed_red //p' "$_budget")" "$_l_never" +check "the budget names a ceiling and a census, and says which is which" \ + "$(grep -cE '^(suites_not_covered|checks_never_observed_red) [0-9]+$' "$_budget")" "2" + +# ---- the gate cannot refuse a check in a suite it has never seen ------------- # -# The ledger is keyed by (suite, name). Two checks with the same name in one -# suite therefore share a row, so ONE of them going red marks BOTH as observed -# red -- a claim about a check nothing attacked, which is exactly what this -# ledger must not make. +# The suite restriction is the MEANING of suites_not_covered, not a softening of +# the refusal. Without it the gate refuses every check of all 250 uncovered +# suites and reddens the whole matrix on its first run -- a gate somebody turns +# off within the week, which is the failure this issue family exists to prevent. # -# It cannot be fixed by keying harder without a synthetic id someone would have -# to maintain. So it is REPORTED, and the number is printed rather than assumed: -# the selftest corpus carries some today, which is how this was noticed at all -- -# 609 records reduced to 605 rows. - -cat > "$_lw/dupe.log" <<'LOG' -RESULT demo part1 the same name PASS -RESULT demo part1 the same name FAIL -RESULT demo part1 a unique name PASS -checks run: 3 -LOG -: > "$_lw/dupe.tsv" -check "a duplicated check name is reported by name" \ - "$(_led_run merge --ledger "$_lw/dupe.tsv" --date 2026-09-10 "$_lw/dupe.log" \ - | grep -c 'duplicate check name, so one ledger row covers 2: demo part1 the same name')" "1" -check "and a unique one is not" \ - "$(_led_run merge --ledger "$_lw/dupe.tsv" --date 2026-09-10 "$_lw/dupe.log" \ - | grep -c 'a unique name')" "0" -check "the two collapse to one row, which is the loss being reported" \ - "$(grep -c . "$_lw/dupe.tsv")" "2" +# It tightens on its own as suites are seeded, and the ceiling forces that +# direction. + +printf 'RESULT\tother\tpartX\tsomething\tPASS\t\nchecks run: 1\n' > "$_lw/othersuite.log" +printf 'demo\nother\n' > "$_lw/reg_both" +printf 'suites_not_covered 1\n' > "$_lw/gb1.txt" +check "a check in an UNCOVERED suite is not refused" \ + "$(_led_rc gate --ledger "$_lw/g.tsv" --budget "$_lw/gb1.txt" --registered "$_lw/reg_both" "$_lw/othersuite.log")" "0" +check "but that suite is counted as not covered, which is the debt" \ + "$(_led_run gate --ledger "$_lw/g.tsv" --budget "$_lw/gb1.txt" --registered "$_lw/reg_both" "$_lw/othersuite.log" \ + | grep -c 'not covered=1')" "1" + +# And once the suite IS covered, a new check in it is refused again -- the +# restriction tightens rather than exempting the suite forever. +_led_run merge --ledger "$_lw/g.tsv" --date D "$_lw/othersuite.log" >/dev/null +printf 'RESULT\tother\tpartX\tsomething\tPASS\t\nRESULT\tother\tpartX\tnewly added\tPASS\t\nchecks run: 2\n' > "$_lw/other2.log" +printf 'suites_not_covered 0\n' > "$_lw/gb2.txt" +check "once the suite is covered, a new check in it IS refused" \ + "$(_led_rc gate --ledger "$_lw/g.tsv" --budget "$_lw/gb2.txt" --registered "$_lw/reg_both" "$_lw/other2.log")" "1" +check "and it is the new one that is named, not the one already ledgered" \ + "$(_led_run gate --ledger "$_lw/g.tsv" --budget "$_lw/gb2.txt" --registered "$_lw/reg_both" "$_lw/other2.log" \ + | grep -c 'not in the ledger: other partX newly added')" "1" From fb872370462f795c7472d088076716815dbe8865 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 10:27:12 -0600 Subject: [PATCH 11/27] test: the monotone ceiling was true of the tool and false of the wiring (#918) Reported by OffgridwithJD, measured on the shipped form. I wrote that the gate "now refuses to see the ceiling raised above its previously committed value". That was true of pgc_ledger.py and false of run_all_versions.sh: the runner's exact invocation, no --against rc=0 the raise is not refused --against HEAD, absolute path rc=0 "no prior ceiling to compare" --against HEAD, repo-relative path rc=1 correctly refused THE MIDDLE LINE IS THE ONE THAT MATTERS. `git show REF:PATH` needs a repo-relative path and the runner passes an absolute one inside a copied build directory, so _committed_budget returned None and the gate printed a note that READS LIKE A PASS while the ceiling it was asked to enforce went unchecked. Asked to compare, unable to compare, is not the same as nothing to compare -- and that is the fail-open shape this whole change is about, in the code that closes it. So the tool resolves the path itself, through the budget's own git toplevel, and every failure to resolve it is an ERROR. The caller no longer has to know. WHICH REF IS NOW A DECISION RATHER THAN A DEFAULT. `--against HEAD` compares a committed file against ITSELF: for any change already committed the working budget and HEAD's are identical, so it catches only an uncommitted raise. The property that matters is that a branch may not raise the ceiling relative to MAIN. The runner prefers origin/main, falls back to HEAD, and PRINTS the fallback and what it costs, because a silent fallback is a gate quietly enforcing less than it claims. THE SCRATCH-REPO ARMS WERE NECESSARY AND NOT SUFFICIENT, which is the gate-nothing-invokes finding one level down: they proved the tool while no wired invocation exercised it. There are now arms in the REAL tree at the REAL path -- an absolute path resolves rather than shrugs, a budget git has never seen is an integrity failure rather than a note, and raising the tracked ceiling in place is refused, with the file restored byte-exact. Also in this commit: the merge of #923's base, whose CHANGELOG entry conflicted with this one. Both entries are kept, #917 then #918, since they describe two changes under one heading. #925 was CONFLICTING against its base, which is why no CI had run on it. Evidence: selftest exit 0, 678 checks, 0 failures; 145 pytest passed (the 35 errors are /usr/local/pg18a absent on this host, identical on main); shellcheck rc=0; docs_style PASSED; git diff --check clean. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/selftest/410-a-check-must-have-been-red.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 15dba48f..33de4187 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -352,8 +352,12 @@ check "premise: the budget was restored byte-exact" \ # ---- and the RUNNER must pass --against, or none of the above is wired ------- +# The ref is a variable, chosen above the call and pinned by the three arms at +# the end of this part. What matters here is that the call site passes one at all: +# without --against the monotone block never runs, which is how the tool was right +# and the wiring was not. check "the runner passes --against to the gate" \ - "$(grep -A5 'pgc_ledger.py" gate' "$_rv" | grep -c -- '--against HEAD')" "1" + "$(grep -A5 'pgc_ledger.py" gate' "$_rv" | grep -c -- '--against')" "1" # ---- and WHICH ref the runner compares against is a decision, not a default -- # From 3d880f49d5d1b9656dfe6dcc7955ccbe9b4b7b75 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 10:37:42 -0600 Subject: [PATCH 12/27] test: the ceiling's prior is resolved, never named after a remote (#918) Reported by OffgridwithJD, and it is my own sentence one level up. `origin` is not a fixed thing. In a contributor's clone it is their FORK -- they measured theirs 446 commits behind upstream -- so `--against origin/main` compared the ceiling against a 16-day-old main and printed that it had compared. THE DIRECTION IS WHAT MAKES IT FAIL OPEN. The ceiling may only fall, so an older main carries one that is higher or equal, and a raise passes whenever the stale prior is high enough. Never falsely red, silently weaker, with a line that reads like the enforcement happened. "Asked to compare, unable to compare, printing a note that reads like a pass" was the previous finding; this is the same shape with "compared against the wrong thing" in place of "could not compare". AND THE FALLBACK WAS THE SAME SHAPE AGAIN. When origin/main did not resolve the runner printed the cost and proceeded with `--against HEAD`, which compares a committed file against itself and therefore catches nothing for any change under review. A fallback that enforces less while saying so is still a gate enforcing less, and one level down I had already made an unresolvable prior an error. SO THE PRIOR IS RESOLVED, AND NEVER GUESSED. `--against auto` takes GITHUB_BASE_REF, which in CI names the PR's target and IS the prior by definition, or the local main's configured upstream outside CI, which is the per-clone answer to "which main is mine". Neither available is rc=2. The ref used is printed, so a reader can see which prior the comparison actually made. AND CI MUST FETCH THAT BASE. actions/checkout takes one ref at depth 1 and the suites job set no fetch-depth, so the base branch is absent and `auto` would stop the run -- correctly, but for a reason the workflow owns rather than the author. The suites job now fetches it at depth 1, guarded on github.base_ref so a push build does not fail on it. Only the file at that commit is read. Arms for all of it, including the two that would have caught me: no base ref and no upstream is an integrity failure with its reason named, and a GITHUB_BASE_REF whose ref is absent says the checkout needs to fetch it rather than falling back. Three arms testing the design this replaces were deleted rather than left to pass against nothing. Evidence: selftest exit 0, 686 checks, 0 failures; 9 pytest; shellcheck rc=0; docs_style PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- .github/workflows/ci.yml | 15 ++++ test/check_ledger.tsv | 14 ++- test/check_ledger_budget.txt | 2 +- test/pgc_ledger.py | 67 +++++++++++++-- test/run_all_versions.sh | 31 +++---- .../410-a-check-must-have-been-red.sh | 86 +++++++++++++++---- 6 files changed, 172 insertions(+), 43 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 52c3178b..45f44b89 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -335,6 +335,21 @@ jobs: steps: - uses: actions/checkout@v4 + # THE LEDGER GATE NEEDS THE BASE BRANCH TO EXIST (#918). It compares the + # tracked ceiling in `test/check_ledger_budget.txt` against the one already + # on the PR's target, and it FAILS CLOSED when no trustworthy prior can be + # read rather than quietly enforcing less. + # + # actions/checkout fetches one ref at depth 1, so without this the base is + # absent and the gate stops the run -- correctly, but for a reason that is + # this workflow's to fix rather than the author's. Depth 1 is enough: only + # the file at that commit is read. + - name: fetch the PR base, for the ledger ceiling comparison + if: github.base_ref != '' + run: | + git fetch --depth=1 origin \ + "+refs/heads/${{ github.base_ref }}:refs/remotes/origin/${{ github.base_ref }}" + # The repository and the package lists first, on their own, because the # cache key below is derived from the versions apt resolves and cannot be # computed before the lists exist. diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 70aa9106..b46e9b4e 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -619,26 +619,31 @@ harness_selftest 410-a-check-must-have-been-red a run whose checks are all ledge harness_selftest 410-a-check-must-have-been-red a second mutation ACCUMULATES rather than replacing the first never - harness_selftest 410-a-check-must-have-been-red an absolute budget path resolves against git rather than shrugging never - harness_selftest 410-a-check-must-have-been-red an empty log is one too, because there is nothing to reconcile never - +harness_selftest 410-a-check-must-have-been-red and a raise against that base ref is refused never - harness_selftest 410-a-check-must-have-been-red and a record missing its verdict never - harness_selftest 410-a-check-must-have-been-red and a refused gate fails the major never - harness_selftest 410-a-check-must-have-been-red and an empty mutation is a placeholder, not an empty last field never - -harness_selftest 410-a-check-must-have-been-red and falls back to HEAD only after saying so never - harness_selftest 410-a-check-must-have-been-red and it entered as debt, not as an observation nothing made never - harness_selftest 410-a-check-must-have-been-red and it is named, so the author knows which one never - harness_selftest 410-a-check-must-have-been-red and it is the new one that is named, not the one already ledgered never - harness_selftest 410-a-check-must-have-been-red and it runs before the build directory is removed, which is the only place it can never - harness_selftest 410-a-check-must-have-been-red and it says it was asked to compare and could not never - +harness_selftest 410-a-check-must-have-been-red and it says why, rather than falling back to something weaker never - harness_selftest 410-a-check-must-have-been-red and none of them ends in a tab never - harness_selftest 410-a-check-must-have-been-red and not against one that stayed green never - harness_selftest 410-a-check-must-have-been-red and one that stayed green keeps its debt never - +harness_selftest 410-a-check-must-have-been-red and only when there is a base, so a push build does not fail on it never - harness_selftest 410-a-check-must-have-been-red and records neither as ever having been red never - harness_selftest 410-a-check-must-have-been-red and the addition in the other part is not called a rename never - harness_selftest 410-a-check-must-have-been-red and the comparison passes when the ceiling did not rise never - -harness_selftest 410-a-check-must-have-been-red and the gate is given that ref rather than a literal never - harness_selftest 410-a-check-must-have-been-red and the history it is about to lose travels with it never - harness_selftest 410-a-check-must-have-been-red and the message says how to fix it, because regenerating is the intended action never - harness_selftest 410-a-check-must-have-been-red and the refusal names both values never - harness_selftest 410-a-check-must-have-been-red and the refusal names the raise never - +harness_selftest 410-a-check-must-have-been-red and the runner names no remote at that call site never - +harness_selftest 410-a-check-must-have-been-red auto refuses when GITHUB_BASE_REF names a ref that is not here never - +harness_selftest 410-a-check-must-have-been-red auto uses the base ref when it resolves, and names it never - +harness_selftest 410-a-check-must-have-been-red auto with no base ref and no upstream is an integrity failure never - harness_selftest 410-a-check-must-have-been-red but that suite is counted as not covered, which is the debt never - harness_selftest 410-a-check-must-have-been-red each says what was wrong with the input never - harness_selftest 410-a-check-must-have-been-red every committed row has five fields never - @@ -658,7 +663,9 @@ harness_selftest 410-a-check-must-have-been-red premise: the ledger itself is a harness_selftest 410-a-check-must-have-been-red premise: the ledger tool exists never - harness_selftest 410-a-check-must-have-been-red premise: the raised copy really does carry a higher ceiling never - harness_selftest 410-a-check-must-have-been-red premise: the real budget is inside a git repository never - +harness_selftest 410-a-check-must-have-been-red premise: the scratch repo has a committed ceiling and no upstream never - harness_selftest 410-a-check-must-have-been-red premise: the scratch repo has a prior ceiling committed never - +harness_selftest 410-a-check-must-have-been-red premise: the workflow file is where this part thinks it is never - harness_selftest 410-a-check-must-have-been-red raising the ceiling above its committed value is refused never - harness_selftest 410-a-check-must-have-been-red raising the ceiling in the tracked file is refused never - harness_selftest 410-a-check-must-have-been-red regenerating the ledger lets the new check through never - @@ -667,9 +674,10 @@ harness_selftest 410-a-check-must-have-been-red the ceiling refuses being exceed harness_selftest 410-a-check-must-have-been-red the committed census matches the committed ledger never - harness_selftest 410-a-check-must-have-been-red the gate refuses to run without the registered suite list never - harness_selftest 410-a-check-must-have-been-red the ledger partitions into observed and never never - +harness_selftest 410-a-check-must-have-been-red the runner asks the tool to resolve the prior rather than naming one never - harness_selftest 410-a-check-must-have-been-red the runner invokes the ledger gate never - harness_selftest 410-a-check-must-have-been-red the runner passes --against to the gate never - -harness_selftest 410-a-check-must-have-been-red the runner prefers origin/main as the ceiling's reference never - harness_selftest 410-a-check-must-have-been-red the same check in two logs is two runs, not a duplicate never - harness_selftest 410-a-check-must-have-been-red the same name twice in ONE log is a duplicate, and is named never - harness_selftest 410-a-check-must-have-been-red the stable check is not reported never - +harness_selftest 410-a-check-must-have-been-red the suites job fetches the PR base for the ceiling comparison never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 754361fe..1c063273 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 675 +checks_never_observed_red 683 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 1841a31e..215b655f 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -65,6 +65,7 @@ """ import argparse +import os import pathlib import subprocess import sys @@ -250,6 +251,60 @@ def read_budget(path): return out +def _resolve_against(path, spec): + """Which ref carries the prior ceiling. Fails closed rather than guessing. + + NOT a hardcoded remote name. `origin` is per-clone: in a contributor's setup it + is their fork, and OffgridwithJD measured theirs 446 commits behind upstream. + Comparing against a stale main makes this check WEAKER, never falsely red -- + the ceiling may only fall, so an older main carries a higher one, and a raise + passes whenever the stale prior is high enough. It fails open while printing a + line that reads like the enforcement happened, which is the same shape as the + absolute-path bug one level down: compared against the wrong thing, rather than + could not compare. + + So: + + GITHUB_BASE_REF in CI this names the PR's target branch, which IS the prior + by definition. Its remote-tracking ref must exist -- if the + checkout did not fetch it, that is an error, not a fallback. + main@{upstream} outside CI, ask git rather than a convention. The configured + upstream of the local main is the answer to "which main is + mine", per clone. + + Anything else is an error. A gate that quietly enforces less than it claims is + the thing this whole change exists to refuse, and a fallback that says so is + still a gate enforcing less. + """ + if spec != "auto": + return spec + repo = pathlib.Path(path).resolve().parent + + def _rev(ref): + r = subprocess.run(["git", "-C", str(repo), "rev-parse", "--verify", "-q", ref], + capture_output=True, text=True) + return ref if r.returncode == 0 else None + + base = os.environ.get("GITHUB_BASE_REF", "").strip() + if base: + for cand in (f"refs/remotes/origin/{base}", base): + if _rev(cand): + return cand + raise LedgerError( + f"GITHUB_BASE_REF is {base!r} but no ref for it resolves here, so the prior " + "ceiling cannot be read. The checkout needs to fetch the base branch") + + r = subprocess.run(["git", "-C", str(repo), "rev-parse", "--abbrev-ref", "main@{upstream}"], + capture_output=True, text=True) + if r.returncode == 0 and r.stdout.strip(): + return r.stdout.strip() + raise LedgerError( + "no trustworthy prior ceiling: GITHUB_BASE_REF is unset and the local main has no " + "configured upstream. Naming a remote would compare against whatever `origin` " + "happens to be in this clone, which is how a fork 446 commits stale gets treated " + "as the prior") + + def _committed_budget(path, ref): """The budget as of `ref`. Raises rather than returning None. @@ -354,13 +409,14 @@ def cmd_gate(args): # MONOTONE, mechanically. The tracked file says the ceiling may only fall; # without this that sentence is prose and raising the number passes. if args.against: - p_want = _committed_budget(args.budget, args.against)["suites_not_covered"] + ref = _resolve_against(args.budget, args.against) + p_want = _committed_budget(args.budget, ref)["suites_not_covered"] if want > p_want: - print(f" suites_not_covered was raised from {p_want} to {want}: " - f"the ceiling may only fall") + print(f" suites_not_covered was raised from {p_want} to {want} " + f"(against {ref}): the ceiling may only fall") rc = 1 else: - print(f" ceiling against {args.against}: {p_want} -> {want}, which does not rise") + print(f" ceiling against {ref}: {p_want} -> {want}, which does not rise") return rc @@ -390,7 +446,8 @@ def main(argv=None): g.add_argument("--registered", default="", help="file listing every registered suite (required)") g.add_argument("--against", default="", - help="git ref whose budget is the prior ceiling, for the monotone check") + help="'auto' to resolve the prior from GITHUB_BASE_REF or main@{upstream}, " + "or an explicit git ref. Fails closed when no trustworthy prior exists") g.add_argument("logs", nargs="+") g.set_defaults(fn=cmd_gate) diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index f3e4c6a4..72845389 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -1357,31 +1357,24 @@ pgc_tally_suite() { # pgc_tally_suite NAME VERDICT LOGFILE echo " the ledger is missing from the tree under test, which is not a pass" verfail=1 else - # WHICH REF THE CEILING IS COMPARED AGAINST, chosen here and printed, - # because the two candidates enforce different things and a silent - # fallback would be the shape this gate exists to refuse. + # WHICH REF CARRIES THE PRIOR CEILING is resolved by the tool, from + # GITHUB_BASE_REF in CI or the local main's configured upstream outside + # it, and it FAILS CLOSED when neither gives a trustworthy answer. # - # origin/main is the real property: a branch may not RAISE the ceiling - # relative to what is already on main. HEAD only catches a raise made in - # the working tree since the last commit, which is worth having locally - # and is much weaker. - # - # The tool fails closed when it cannot read the prior, so an unresolvable - # ref would redden rather than pass. Choosing here means the fallback is a - # decision someone can see rather than an error someone has to diagnose. - if git -C "$builddir" rev-parse --verify -q origin/main >/dev/null 2>&1; then - _led_ref=origin/main - else - _led_ref=HEAD - echo " origin/main does not resolve here, so the ledger ceiling is" - echo " compared against HEAD: that catches an uncommitted raise only" - fi + # It used to be chosen here, preferring origin/main with a printed + # fallback to HEAD. Both halves were wrong. `origin` is per-clone -- in a + # contributor's setup it is their fork, measured 446 commits stale -- and + # comparing against an older main makes the check WEAKER rather than + # falsely red, because the ceiling may only fall. And the fallback to HEAD + # compares a committed file against itself, so it caught nothing for any + # change under review while printing that it had compared. A gate that + # quietly enforces less than it claims is what this whole change refuses. # shellcheck disable=SC2086 if ! python3 "$builddir/test/pgc_ledger.py" gate \ --ledger "$builddir/test/check_ledger.tsv" \ --budget "$builddir/test/check_ledger_budget.txt" \ --registered "$_acc_registered" \ - --against "$_led_ref" \ + --against auto \ $_led_logs; then echo " PG$major has a check the ledger has never seen, which is not a pass" verfail=1 diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 33de4187..c9733999 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -359,20 +359,76 @@ check "premise: the budget was restored byte-exact" \ check "the runner passes --against to the gate" \ "$(grep -A5 'pgc_ledger.py" gate' "$_rv" | grep -c -- '--against')" "1" -# ---- and WHICH ref the runner compares against is a decision, not a default -- + + +# ---- the prior is resolved, never named ------------------------------------ +# +# `origin` is not a fixed thing. In a contributor's clone it is their FORK, and +# OffgridwithJD measured theirs 446 commits behind upstream. Comparing the ceiling +# against a stale main makes this check WEAKER rather than falsely red -- the +# ceiling may only fall, so an older main carries a higher one, and a raise passes +# whenever the stale prior is high enough. # -# `--against HEAD` compares a committed file against itself: for any change that -# is already committed, the working budget and HEAD's are identical, so it catches -# only an UNCOMMITTED raise. The property that matters is that a branch may not -# raise the ceiling relative to MAIN, which needs origin/main as the ref. +# It fails open while printing a line that reads like the enforcement happened. +# That is the same shape as the absolute-path bug this part already carries, with +# "compared against the wrong thing" in place of "could not compare". # -# The runner therefore chooses, and prints the choice when it falls back. A silent -# fallback to the weaker ref would be a gate quietly enforcing less than it says, -# which is the shape this whole change exists to refuse. - -check "the runner prefers origin/main as the ceiling's reference" \ - "$(grep -c '_led_ref=origin/main' "$_rv")" "1" -check "and falls back to HEAD only after saying so" \ - "$(grep -A3 '_led_ref=HEAD' "$_rv" | grep -c 'catches an uncommitted raise only')" "1" -check "and the gate is given that ref rather than a literal" \ - "$(grep -c -- '--against "\$_led_ref"' "$_rv")" "1" +# So the tool resolves it: GITHUB_BASE_REF in CI, which names the PR's target and +# IS the prior by definition; the local main's configured upstream outside CI, +# which is the per-clone answer to "which main is mine". Neither available is an +# ERROR, because a fallback that enforces less while saying so is still a gate +# enforcing less. + +check "the runner asks the tool to resolve the prior rather than naming one" \ + "$(grep -A5 'pgc_ledger.py" gate' "$_rv" | grep -c -- '--against auto')" "1" +check "and the runner names no remote at that call site" \ + "$(grep -A6 'pgc_ledger.py" gate' "$_rv" | grep -c 'origin/main')" "0" + +_res="$_lw/resolve"; rm -rf "$_res"; mkdir -p "$_res" +( cd "$_res" && git init -q . && git config user.email t@t && git config user.name t + printf 'suites_not_covered 7\n' > b.txt && git add b.txt && git commit -qm base ) >/dev/null 2>&1 +printf 'suites_not_covered 7\n' > "$_res/b.txt" + +check "premise: the scratch repo has a committed ceiling and no upstream" \ + "$(cd "$_res" && git rev-parse --abbrev-ref main@{upstream} 2>/dev/null || echo none)" "none" +check "auto with no base ref and no upstream is an integrity failure" \ + "$(cd "$_res" && env -u GITHUB_BASE_REF python3 "$_led" gate --ledger "$_lw/g.tsv" \ + --budget b.txt --registered "$_lw/registered" --against auto "$_lw/new.log" \ + >/dev/null 2>&1; echo $?)" "2" +check "and it says why, rather than falling back to something weaker" \ + "$(cd "$_res" && env -u GITHUB_BASE_REF python3 "$_led" gate --ledger "$_lw/g.tsv" \ + --budget b.txt --registered "$_lw/registered" --against auto "$_lw/new.log" 2>&1 \ + | grep -c 'no trustworthy prior ceiling')" "1" + +# GITHUB_BASE_REF names a branch whose ref must actually exist. A CI checkout that +# did not fetch the base is an error the workflow fixes, not one the author works +# around. +check "auto refuses when GITHUB_BASE_REF names a ref that is not here" \ + "$(cd "$_res" && GITHUB_BASE_REF=nosuchbranch python3 "$_led" gate --ledger "$_lw/g.tsv" \ + --budget b.txt --registered "$_lw/registered" --against auto "$_lw/new.log" 2>&1 \ + | grep -ci 'the checkout needs to fetch the base branch')" "1" + +# And when it IS here, that is the ref used -- named in the output, so a reader +# can see which prior the comparison actually made. +( cd "$_res" && git branch -q basebranch && git update-ref refs/remotes/origin/basebranch \ + "$(git rev-parse HEAD)" ) >/dev/null 2>&1 +check "auto uses the base ref when it resolves, and names it" \ + "$(cd "$_res" && GITHUB_BASE_REF=basebranch python3 "$_led" gate --ledger "$_lw/g.tsv" \ + --budget b.txt --registered "$_lw/registered" --against auto "$_lw/new.log" 2>&1 \ + | grep -c 'ceiling against refs/remotes/origin/basebranch')" "1" +printf 'suites_not_covered 99\n' > "$_res/b.txt" +check "and a raise against that base ref is refused" \ + "$(cd "$_res" && GITHUB_BASE_REF=basebranch python3 "$_led" gate --ledger "$_lw/g.tsv" \ + --budget b.txt --registered "$_lw/registered" --against auto "$_lw/new.log" 2>&1 \ + | grep -c 'was raised from 7 to 99')" "1" + +# ---- and CI must fetch that base, or the gate stops the run ---------------- + +_ci="$PGC_SRCDIR/.github/workflows/ci.yml" +check "premise: the workflow file is where this part thinks it is" \ + "$([ -f "$_ci" ] && echo yes || echo no)" "yes" +check "the suites job fetches the PR base for the ceiling comparison" \ + "$(grep -c 'fetch the PR base, for the ledger ceiling comparison' "$_ci")" "1" +check "and only when there is a base, so a push build does not fail on it" \ + "$(grep -A1 'fetch the PR base, for the ledger ceiling comparison' "$_ci" \ + | grep -c "github.base_ref != ''")" "1" From 380f9f68e7b47ffe44ad25edc2b2c532b1e72ca3 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 10:44:13 -0600 Subject: [PATCH 13/27] test: say how far behind the ceiling's prior is, because the route changed and the destination did not (#918) Reported by OffgridwithJD, who measured it in their own clone rather than predicting it. `main@{upstream}` is the per-clone answer to "which main is mine", and in a contributor's setup it resolves to their FORK -- `git push -u origin main` is what sets that config. Theirs is 446 commits behind upstream, so `--against auto` outside CI lands on exactly the ref the hardcoded `origin/main` did. The route changed; the destination did not. It is not the same defect: the ref used is printed, so a reader can see `origin/main`. But a reader cannot see that it is 446 commits stale, and the direction still fails open -- the ceiling may only fall, so an older main carries a higher one and a raise passes whenever the stale prior is high enough. There is no better ref to pick that does not guess, and I am not going to guess. So the weakness is made VISIBLE instead: ceiling against origin/main (446 commits behind HEAD): 250 -> 250 Naming the ref told a reader WHICH prior was used. This tells them what the comparison is worth. It costs nothing when the number is zero, where the label is omitted entirely. AND THE CI HALF IS NOW MEASURED RATHER THAN CONSTRUCTED, from my own run 34502198282 on fb872370: origin/main does not resolve here, so the ledger ceiling is ceiling against HEAD: 250 -> 250, which does not rise The fallback fired, the prior was HEAD, and the comparison was a committed file against itself -- catching nothing for any change under review, in the only place this gate runs for real, with a line above it that reads like enforcement. The previous commit removed that fallback and made CI fetch the base; this is the evidence it needed to be removed rather than kept with a warning. Evidence: selftest exit 0, 690 checks, 0 failures; 9 pytest; shellcheck rc=0; docs_style PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/check_ledger.tsv | 4 ++ test/check_ledger_budget.txt | 2 +- test/pgc_ledger.py | 32 +++++++++++++++- .../410-a-check-must-have-been-red.sh | 38 +++++++++++++++++++ 4 files changed, 73 insertions(+), 3 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index b46e9b4e..ca4c2926 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -617,8 +617,10 @@ harness_selftest 410-a-check-must-have-been-red a real refusal is a different st harness_selftest 410-a-check-must-have-been-red a rename in one part survives an addition in another never - harness_selftest 410-a-check-must-have-been-red a run whose checks are all ledgered passes the gate never - harness_selftest 410-a-check-must-have-been-red a second mutation ACCUMULATES rather than replacing the first never - +harness_selftest 410-a-check-must-have-been-red a stale prior is named WITH its distance from HEAD never - harness_selftest 410-a-check-must-have-been-red an absolute budget path resolves against git rather than shrugging never - harness_selftest 410-a-check-must-have-been-red an empty log is one too, because there is nothing to reconcile never - +harness_selftest 410-a-check-must-have-been-red and a level prior carries no distance, so zero is silent never - harness_selftest 410-a-check-must-have-been-red and a raise against that base ref is refused never - harness_selftest 410-a-check-must-have-been-red and a record missing its verdict never - harness_selftest 410-a-check-must-have-been-red and a refused gate fails the major never - @@ -663,6 +665,7 @@ harness_selftest 410-a-check-must-have-been-red premise: the ledger itself is a harness_selftest 410-a-check-must-have-been-red premise: the ledger tool exists never - harness_selftest 410-a-check-must-have-been-red premise: the raised copy really does carry a higher ceiling never - harness_selftest 410-a-check-must-have-been-red premise: the real budget is inside a git repository never - +harness_selftest 410-a-check-must-have-been-red premise: the scratch prior really is three commits behind never - harness_selftest 410-a-check-must-have-been-red premise: the scratch repo has a committed ceiling and no upstream never - harness_selftest 410-a-check-must-have-been-red premise: the scratch repo has a prior ceiling committed never - harness_selftest 410-a-check-must-have-been-red premise: the workflow file is where this part thinks it is never - @@ -672,6 +675,7 @@ harness_selftest 410-a-check-must-have-been-red regenerating the ledger lets the harness_selftest 410-a-check-must-have-been-red the budget names a ceiling and a census, and says which is which never - harness_selftest 410-a-check-must-have-been-red the ceiling refuses being exceeded never - harness_selftest 410-a-check-must-have-been-red the committed census matches the committed ledger never - +harness_selftest 410-a-check-must-have-been-red the distance travels with a refusal too, not only with a pass never - harness_selftest 410-a-check-must-have-been-red the gate refuses to run without the registered suite list never - harness_selftest 410-a-check-must-have-been-red the ledger partitions into observed and never never - harness_selftest 410-a-check-must-have-been-red the runner asks the tool to resolve the prior rather than naming one never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 1c063273..7a92b339 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 683 +checks_never_observed_red 687 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 215b655f..d027b1bb 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -305,6 +305,22 @@ def _rev(ref): "as the prior") +def _behind(path, ref): + """" (N commits behind HEAD)" when the prior lags, "" when it does not. + + A reader can see WHICH ref was compared against. Without this they cannot see + that it is 446 commits stale, which is the difference between knowing the + comparison happened and knowing what it was worth. + """ + repo = pathlib.Path(path).resolve().parent + r = subprocess.run(["git", "-C", str(repo), "rev-list", "--count", f"{ref}..HEAD"], + capture_output=True, text=True) + if r.returncode != 0 or not r.stdout.strip().isdigit(): + return " (distance from HEAD unknown)" + n = int(r.stdout.strip()) + return "" if n == 0 else f" ({n} commit{'s' if n != 1 else ''} behind HEAD)" + + def _committed_budget(path, ref): """The budget as of `ref`. Raises rather than returning None. @@ -410,13 +426,25 @@ def cmd_gate(args): # without this that sentence is prose and raising the number passes. if args.against: ref = _resolve_against(args.budget, args.against) + # HOW FAR BEHIND THE PRIOR IS, printed beside it. + # + # `main@{upstream}` is the per-clone answer to "which main is mine", and + # in a contributor's setup it resolves to their FORK -- OffgridwithJD's is + # 446 commits behind upstream. Naming the ref told a reader WHICH prior + # was used; it did not tell them what the comparison was worth. The + # direction still fails open: an older main carries a higher ceiling, so a + # raise passes whenever the stale prior is high enough. + # + # There is no better ref to pick that does not guess, so the weakness is + # made visible instead. It costs nothing when the number is 0. + shown = f"{ref}{_behind(args.budget, ref)}" p_want = _committed_budget(args.budget, ref)["suites_not_covered"] if want > p_want: print(f" suites_not_covered was raised from {p_want} to {want} " - f"(against {ref}): the ceiling may only fall") + f"(against {shown}): the ceiling may only fall") rc = 1 else: - print(f" ceiling against {ref}: {p_want} -> {want}, which does not rise") + print(f" ceiling against {shown}: {p_want} -> {want}, which does not rise") return rc diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index c9733999..6c52eedf 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -432,3 +432,41 @@ check "the suites job fetches the PR base for the ceiling comparison" \ check "and only when there is a base, so a push build does not fail on it" \ "$(grep -A1 'fetch the PR base, for the ledger ceiling comparison' "$_ci" \ | grep -c "github.base_ref != ''")" "1" + +# ---- and how far behind the prior is, printed beside it --------------------- +# +# `main@{upstream}` is the per-clone answer to "which main is mine", and in a +# contributor's setup it resolves to their FORK. OffgridwithJD measured theirs 446 +# commits behind upstream -- and `git push -u origin main` is what sets that +# config, so `auto` outside CI lands on exactly the ref the hardcoded version did. +# The route changed; the destination did not. +# +# There is no better ref to pick that does not guess, so the weakness is made +# VISIBLE rather than removed. Naming the ref told a reader WHICH prior was used; +# it did not tell them what the comparison was worth. The direction still fails +# open: an older main carries a higher ceiling, so a raise passes whenever the +# stale prior is high enough. +# +# Their suggestion, and it costs nothing when the number is 0. + +_dist="$_lw/dist"; rm -rf "$_dist"; mkdir -p "$_dist" +( cd "$_dist" && git init -q . && git config user.email t@t && git config user.name t + printf 'suites_not_covered 7\n' > b.txt && git add b.txt && git commit -qm base + git branch -q oldbase + for i in 1 2 3; do echo "x$i" > f; git add f; git commit -qm "c$i"; done ) >/dev/null 2>&1 +printf 's\tp\ta\tnever\t-\n' > "$_dist/led" +printf 'RESULT\ts\tp\ta\tPASS\t\n' > "$_dist/log" +printf 's\n' > "$_dist/reg" + +check "premise: the scratch prior really is three commits behind" \ + "$(cd "$_dist" && git rev-list --count oldbase..HEAD)" "3" +check "a stale prior is named WITH its distance from HEAD" \ + "$(cd "$_dist" && python3 "$_led" gate --ledger led --budget b.txt --registered reg \ + --against oldbase log 2>&1 | grep -c 'ceiling against oldbase (3 commits behind HEAD)')" "1" +check "and a level prior carries no distance, so zero is silent" \ + "$(cd "$_dist" && python3 "$_led" gate --ledger led --budget b.txt --registered reg \ + --against HEAD log 2>&1 | grep -c 'ceiling against HEAD: ')" "1" +check "the distance travels with a refusal too, not only with a pass" \ + "$(cd "$_dist" && sed -i 's/^suites_not_covered 7$/suites_not_covered 99/' b.txt + python3 "$_led" gate --ledger led --budget b.txt --registered reg \ + --against oldbase log 2>&1 | grep -c 'against oldbase (3 commits behind HEAD)')" "1" From 3c067deb052ad7faee8d3c49d3d403447a2712e0 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 11:09:58 -0600 Subject: [PATCH 14/27] test: a file absent at the prior has no ceiling to violate, and the two failure kinds are different (#918) THE GATE CAUGHT ITS OWN BOOTSTRAP the first time it ran in CI, which is the right outcome for a rule that could not be satisfied. From run 34503924812: ledger integrity failure: --against refs/remotes/origin/feat/917-machine-readable-results was given, but test/check_ledger_budget.txt does not exist at that ref #925's base is #923's branch, where the budget does not exist because THIS change adds it. So `auto` resolved the base correctly, fetched it, found no prior, failed closed, and reddened the matrix -- and a PR introducing the file could never pass its own gate. OffgridwithJD hit the identical failure independently in their own clone, resolved to their fork, where the file is also absent. THREE STATES, NOT TWO, which is the distinction this change already draws one level down: the prior ref does not resolve ERROR. Asked to compare, unable to. the ref resolves, file absent there NO PRIOR. Nothing could have been raised relative to a file that did not exist. the ref resolves, file present COMPARE. The middle one is the first-landing case and it is genuinely "nothing to compare". It is expressed as the PROPERTY -- absent at the prior -- rather than as a flag or a date, so it clears itself: once the file is on main every future base carries it, and there is no exemption left for anyone to forget to remove. It is not a hole. Deleting the budget on a branch and re-adding it higher does not reach it, because the file still exists at the prior and the comparison happens. AND THE RUNNER COLLAPSED THE TWO FAILURE KINDS. The gate distinguishes rc=1, a real refusal whose fix is to regenerate the ledger, from rc=2, the gate unable to do its job at all. The runner reported both as "has a check the ledger has never seen" -- sending the reader at a repair that cannot help, three lines below the gate's own "new this run=0", which says the opposite. Also OffgridwithJD, from the CI log of this branch. It now branches, and both arms still fail the major. THE ARMS FOR IT WERE WRONG TWICE, both times in the same way. They counted occurrences inside `grep -A6`, `-A8` and `-A12` windows, and every one broke the moment the call site gained a comment: a window's size is a fact about formatting. The block is now EXTRACTED and tested, as selftest 320 already does with the runner's classifier. Then the extraction counted the block's own explanation as a second occurrence of the sentences it was counting, so comments are stripped -- selftest 080's control problem, met in a fifth file tonight. One arm was deleted rather than fixed: it read a variable defined ninety lines below it, and the block that defines the variable already asserts the same thing. Evidence: selftest exit 0, 696 checks, 0 failures; 145 pytest passed; shellcheck rc=0; docs_style PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/check_ledger.tsv | 12 +++- test/check_ledger_budget.txt | 2 +- test/pgc_ledger.py | 37 +++++++++++-- test/pytest/test_mutation_ledger.py | 18 +++++- test/run_all_versions.sh | 23 ++++++-- .../410-a-check-must-have-been-red.sh | 55 +++++++++++++++++-- 6 files changed, 124 insertions(+), 23 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index ca4c2926..78bd9ee3 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -604,32 +604,35 @@ harness_selftest 400-a-check-result-must-be-machine the same ratio check, ENABLE harness_selftest 400-a-check-result-must-be-machine the same timing check, ENABLED, emits one record and passes never - harness_selftest 400-a-check-result-must-be-machine the sweep finds a check inside a PIPED loop never - harness_selftest 410-a-check-must-have-been-red a before-log and an after-log together are refused, not silently empty never - -harness_selftest 410-a-check-must-have-been-red a budget git does not know is an integrity failure, not a note never - +harness_selftest 410-a-check-must-have-been-red a budget that does not exist at the prior is a note, not a refusal never - harness_selftest 410-a-check-must-have-been-red a check in an UNCOVERED suite is not refused never - harness_selftest 410-a-check-must-have-been-red a check merely added is not reported as a rename never - harness_selftest 410-a-check-must-have-been-red a check observed red gains the date it was seen never - harness_selftest 410-a-check-must-have-been-red a check the ledger has never seen is refused never - +harness_selftest 410-a-check-must-have-been-red a clean status says nothing and does not fail the major never - harness_selftest 410-a-check-must-have-been-red a gate over a nonexistent log is an integrity failure, not a pass never - harness_selftest 410-a-check-must-have-been-red a later green run does not erase an observation never - harness_selftest 410-a-check-must-have-been-red a name that appeared while another disappeared is reported as a rename never - harness_selftest 410-a-check-must-have-been-red a named mutation is recorded against the check that reddened never - harness_selftest 410-a-check-must-have-been-red a real refusal is a different status from an integrity failure never - +harness_selftest 410-a-check-must-have-been-red a refusal keeps the regenerate-the-ledger wording never - harness_selftest 410-a-check-must-have-been-red a rename in one part survives an addition in another never - harness_selftest 410-a-check-must-have-been-red a run whose checks are all ledgered passes the gate never - harness_selftest 410-a-check-must-have-been-red a second mutation ACCUMULATES rather than replacing the first never - harness_selftest 410-a-check-must-have-been-red a stale prior is named WITH its distance from HEAD never - harness_selftest 410-a-check-must-have-been-red an absolute budget path resolves against git rather than shrugging never - harness_selftest 410-a-check-must-have-been-red an empty log is one too, because there is nothing to reconcile never - +harness_selftest 410-a-check-must-have-been-red an integrity failure says regenerating will not help never - harness_selftest 410-a-check-must-have-been-red and a level prior carries no distance, so zero is silent never - harness_selftest 410-a-check-must-have-been-red and a raise against that base ref is refused never - harness_selftest 410-a-check-must-have-been-red and a record missing its verdict never - -harness_selftest 410-a-check-must-have-been-red and a refused gate fails the major never - harness_selftest 410-a-check-must-have-been-red and an empty mutation is a placeholder, not an empty last field never - harness_selftest 410-a-check-must-have-been-red and it entered as debt, not as an observation nothing made never - +harness_selftest 410-a-check-must-have-been-red and it is a different sentence from the refusal, not the same one twice never - harness_selftest 410-a-check-must-have-been-red and it is named, so the author knows which one never - harness_selftest 410-a-check-must-have-been-red and it is the new one that is named, not the one already ledgered never - harness_selftest 410-a-check-must-have-been-red and it runs before the build directory is removed, which is the only place it can never - -harness_selftest 410-a-check-must-have-been-red and it says it was asked to compare and could not never - +harness_selftest 410-a-check-must-have-been-red and it says the change introduces the file rather than raising anything never - harness_selftest 410-a-check-must-have-been-red and it says why, rather than falling back to something weaker never - harness_selftest 410-a-check-must-have-been-red and none of them ends in a tab never - harness_selftest 410-a-check-must-have-been-red and not against one that stayed green never - @@ -646,6 +649,7 @@ harness_selftest 410-a-check-must-have-been-red and the runner names no remote a harness_selftest 410-a-check-must-have-been-red auto refuses when GITHUB_BASE_REF names a ref that is not here never - harness_selftest 410-a-check-must-have-been-red auto uses the base ref when it resolves, and names it never - harness_selftest 410-a-check-must-have-been-red auto with no base ref and no upstream is an integrity failure never - +harness_selftest 410-a-check-must-have-been-red both failure arms fail the major never - harness_selftest 410-a-check-must-have-been-red but that suite is counted as not covered, which is the debt never - harness_selftest 410-a-check-must-have-been-red each says what was wrong with the input never - harness_selftest 410-a-check-must-have-been-red every committed row has five fields never - @@ -660,6 +664,7 @@ harness_selftest 410-a-check-must-have-been-red premise: the budget is a tracked harness_selftest 410-a-check-must-have-been-red premise: the budget was restored byte-exact never - harness_selftest 410-a-check-must-have-been-red premise: the check has history before the rename never - harness_selftest 410-a-check-must-have-been-red premise: the fixture log names checks the real ledger already knows never - +harness_selftest 410-a-check-must-have-been-red premise: the gate's status block was found in the runner never - harness_selftest 410-a-check-must-have-been-red premise: the ledger is not empty, so the partition means something never - harness_selftest 410-a-check-must-have-been-red premise: the ledger itself is a tracked file, not a variable never - harness_selftest 410-a-check-must-have-been-red premise: the ledger tool exists never - @@ -679,6 +684,7 @@ harness_selftest 410-a-check-must-have-been-red the distance travels with a refu harness_selftest 410-a-check-must-have-been-red the gate refuses to run without the registered suite list never - harness_selftest 410-a-check-must-have-been-red the ledger partitions into observed and never never - harness_selftest 410-a-check-must-have-been-red the runner asks the tool to resolve the prior rather than naming one never - +harness_selftest 410-a-check-must-have-been-red the runner captures the gate's status rather than only its success never - harness_selftest 410-a-check-must-have-been-red the runner invokes the ledger gate never - harness_selftest 410-a-check-must-have-been-red the runner passes --against to the gate never - harness_selftest 410-a-check-must-have-been-red the same check in two logs is two runs, not a duplicate never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 7a92b339..74df0f47 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 687 +checks_never_observed_red 693 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index d027b1bb..635566d4 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -322,7 +322,12 @@ def _behind(path, ref): def _committed_budget(path, ref): - """The budget as of `ref`. Raises rather than returning None. + """The budget as of `ref`, or None when the file does not exist there. + + None means "this change introduces the file", which is not a raise. Every + OTHER failure -- an unresolvable ref, a path outside its repository, a budget + naming no ceiling -- raises, because asked to compare and unable is a different + thing from nothing to compare. `git show REF:PATH` needs a REPO-RELATIVE path. The production caller passes an absolute one inside a copied build directory, so the first version returned @@ -347,13 +352,28 @@ def _committed_budget(path, ref): rel = abspath.relative_to(pathlib.Path(top).resolve()) except ValueError as e: raise LedgerError(f"{path} resolves outside its own repository at {top}") from e + # A FILE THAT DOES NOT EXIST AT THE PRIOR HAS NO CEILING TO VIOLATE. This + # returns None and the caller notes it, rather than erroring, and the + # distinction is the whole of it: introducing the budget is not raising it. + # + # The gate caught its own bootstrap the first time it ran in CI -- #925's base + # is #923's branch, where check_ledger_budget.txt does not exist because this + # PR adds it, so `auto` resolved the base correctly, found no prior, failed + # closed, and reddened the matrix. Correct behaviour for a rule that could not + # be satisfied: a PR introducing the file could never pass its own gate. + # + # It is not a hole. Deleting the budget on a branch and re-adding it with a + # higher ceiling does not reach here, because the file still exists at the + # prior and the comparison happens. Only a genuinely new file gets the note, + # and a genuinely new budget file is reviewed as a new file. + # + # An unresolvable REF stays an error above, and a malformed budget stays one + # below. Asked to compare and unable is different from nothing to compare. try: blob = subprocess.run(["git", "-C", top, "show", f"{ref}:{rel.as_posix()}"], capture_output=True, text=True, check=True).stdout - except (subprocess.CalledProcessError, OSError) as e: - raise LedgerError( - f"--against {ref} was given, but {rel.as_posix()} does not exist at {ref}, " - "so there is no prior ceiling to compare against") from e + except (subprocess.CalledProcessError, OSError): + return None out = {} for line in blob.splitlines(): line = line.strip() @@ -438,7 +458,12 @@ def cmd_gate(args): # There is no better ref to pick that does not guess, so the weakness is # made visible instead. It costs nothing when the number is 0. shown = f"{ref}{_behind(args.budget, ref)}" - p_want = _committed_budget(args.budget, ref)["suites_not_covered"] + prior = _committed_budget(args.budget, ref) + if prior is None: + print(f" no budget at {shown}: this change introduces it, so there is no " + f"prior ceiling it could have raised") + return rc + p_want = prior["suites_not_covered"] if want > p_want: print(f" suites_not_covered was raised from {p_want} to {want} " f"(against {shown}): the ceiling may only fall") diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index 9098a512..25231500 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -259,8 +259,22 @@ def test_the_runner_invokes_the_gate_before_it_removes_the_logs(expect): expect.at_least(len(teardown), 1, "premise: the runner removes the build directory") expect.text("before" if call[0] < teardown[-1] else "after", "before", "and it runs before the logs are removed, the only place it can") - window = "\n".join(text[call[0]:call[0] + 8]) - expect.num(window.count("verfail=1"), 1, "and a refused gate fails the major") + # The block is extracted, not a fixed-size window: a window's size is a fact + # about formatting, and the first version measured 8 lines and broke the moment + # the call site gained a comment. + src = RUNNER.read_text() + block = src[src.index("\t\t_led_rc=$?"):] + block = block[:block.index("\t\tesac") + len("\t\tesac")] + # Comments stripped: the block's own explanation quotes the sentences counted + # below, so an unstripped extraction counts the documentation as an occurrence. + block = "\n".join(l for l in block.splitlines() if not l.strip().startswith("#")) + expect.num(block.count("verfail=1"), 2, + "both failure arms fail the major") + expect.num(block.count("has a check the ledger has never seen"), 1, + "a refusal keeps the regenerate-the-ledger wording") + expect.num(block.count("could not run the ledger gate at all"), 1, + "and an integrity failure gets its own sentence, since regenerating " + "the ledger would not help") def test_the_committed_ledger_and_budget_agree(expect): diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index 72845389..15e2331c 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -1370,15 +1370,28 @@ pgc_tally_suite() { # pgc_tally_suite NAME VERDICT LOGFILE # change under review while printing that it had compared. A gate that # quietly enforces less than it claims is what this whole change refuses. # shellcheck disable=SC2086 - if ! python3 "$builddir/test/pgc_ledger.py" gate \ + python3 "$builddir/test/pgc_ledger.py" gate \ --ledger "$builddir/test/check_ledger.tsv" \ --budget "$builddir/test/check_ledger_budget.txt" \ --registered "$_acc_registered" \ --against auto \ - $_led_logs; then - echo " PG$major has a check the ledger has never seen, which is not a pass" - verfail=1 - fi + $_led_logs + _led_rc=$? + # BRANCH ON THE STATUS THE TOOL WENT TO THE TROUBLE OF DISTINGUISHING. + # rc=1 is a real refusal and the fix is to regenerate the ledger; rc=2 is + # the gate unable to do its job at all, where regenerating helps nothing. + # Collapsing them printed "has a check the ledger has never seen" three + # lines below the gate's own "new this run=0", which contradicts it and + # sends the reader at the wrong repair. Reported by OffgridwithJD. + case "$_led_rc" in + 0) ;; + 1) echo " PG$major has a check the ledger has never seen, which is not a pass" + verfail=1 ;; + *) echo " PG$major could not run the ledger gate at all, which is not a pass:" + echo " the input or the prior ceiling was unusable, and regenerating the" + echo " ledger will not help. The failure above says which." + verfail=1 ;; + esac fi # How many of the suites counted as having RUN actually accounted for their diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 6c52eedf..390683d2 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -233,8 +233,6 @@ check "the runner invokes the ledger gate" \ check "and it runs before the build directory is removed, which is the only place it can" \ "$([ "$(grep -n 'pgc_ledger.py" gate' "$_rv" | cut -d: -f1)" -lt \ "$(grep -n 'rm -rf "\$builddir"' "$_rv" | tail -1 | cut -d: -f1)" ] && echo before || echo after)" "before" -check "and a refused gate fails the major" \ - "$(grep -A8 'pgc_ledger.py" gate' "$_rv" | grep -c 'verfail=1')" "1" # ---- the committed files agree ---------------------------------------------- @@ -327,12 +325,24 @@ check "premise: the raised copy really does carry a higher ceiling" \ "$(sed -n 's/^suites_not_covered //p' "$_mono_raised")" "9999" check "premise: and it is a file git has never seen, which is the case that used to fail open" \ "$(git -C "$PGC_TESTDIR" show "HEAD:test/check_ledger_budget.txt.raised" >/dev/null 2>&1 && echo tracked || echo untracked)" "untracked" -check "a budget git does not know is an integrity failure, not a note" \ +# A FILE THAT DOES NOT EXIST AT THE PRIOR HAS NO CEILING TO VIOLATE, so it is a +# NOTE rather than an error. Introducing the budget is not raising it. +# +# The first version made it an error and the gate caught its own bootstrap the +# first time it ran in CI: #925's base is #923's branch, where the budget does not +# exist because this change adds it, so `auto` resolved the base correctly, found +# no prior, failed closed, and reddened the matrix. A PR introducing the file could +# never pass its own gate. +# +# It is not a hole: deleting the budget on a branch and re-adding it higher does +# not reach here, because the file still exists at the prior and the comparison +# happens. Only a genuinely new file gets the note. +check "a budget that does not exist at the prior is a note, not a refusal" \ "$(_led_rc gate --ledger "$_ledger" --budget "$_mono_raised" --registered "$_mono_reg" \ - --against HEAD "$_mono_log")" "2" -check "and it says it was asked to compare and could not" \ + --against HEAD "$_mono_log")" "0" +check "and it says the change introduces the file rather than raising anything" \ "$(_led_run gate --ledger "$_ledger" --budget "$_mono_raised" --registered "$_mono_reg" \ - --against HEAD "$_mono_log" | grep -c 'does not exist at HEAD')" "1" + --against HEAD "$_mono_log" | grep -c 'this change introduces it')" "1" rm -f "$_mono_raised" # And the raise itself, on the tracked path, by rewriting it in place and putting @@ -470,3 +480,36 @@ check "the distance travels with a refusal too, not only with a pass" \ "$(cd "$_dist" && sed -i 's/^suites_not_covered 7$/suites_not_covered 99/' b.txt python3 "$_led" gate --ledger led --budget b.txt --registered reg \ --against oldbase log 2>&1 | grep -c 'against oldbase (3 commits behind HEAD)')" "1" + +# ---- the runner must not collapse the two failure kinds --------------------- +# +# The gate distinguishes rc=1, a real refusal whose fix is to regenerate the +# ledger, from rc=2, the gate unable to do its job at all. The runner reported +# both as "has a check the ledger has never seen", which sends the reader at a +# repair that cannot help -- and printed it three lines below the gate's own +# "new this run=0", which says the opposite. Reported by OffgridwithJD from the +# CI log of this very branch. + +# The block is EXTRACTED and tested, not counted inside a -A window. A window's +# size is a fact about formatting: the first version of these arms measured 6, 8 +# and 12 lines, and every one of them went wrong the moment the call site gained +# a comment. Selftest 320 already takes this approach with the runner's classifier. +# Comments stripped. The block's own explanation quotes the sentences these arms +# count, so an unstripped extraction counts the documentation as a second +# occurrence -- selftest 080's control problem, which this session has now met +# five times in five different files. +_ledcase="$(sed -n '/^\t\t_led_rc=\$?$/,/^\t\tesac$/p' "$_rv" | grep -vE '^[[:space:]]*#')" +check "premise: the gate's status block was found in the runner" \ + "$(printf '%s' "$_ledcase" | grep -c 'case "\$_led_rc" in')" "1" +check "the runner captures the gate's status rather than only its success" \ + "$(grep -c '_led_rc=\$?' "$_rv")" "1" +check "a refusal keeps the regenerate-the-ledger wording" \ + "$(printf '%s' "$_ledcase" | grep -c 'has a check the ledger has never seen')" "1" +check "an integrity failure says regenerating will not help" \ + "$(printf '%s' "$_ledcase" | grep -c 'will not help')" "1" +check "and it is a different sentence from the refusal, not the same one twice" \ + "$(printf '%s' "$_ledcase" | grep -c 'could not run the ledger gate at all')" "1" +check "a clean status says nothing and does not fail the major" \ + "$(printf '%s' "$_ledcase" | grep -cE '^[[:space:]]+0\)[[:space:]]*;;[[:space:]]*$')" "1" +check "both failure arms fail the major" \ + "$(printf '%s' "$_ledcase" | grep -c 'verfail=1')" "2" From 71a7c9bcd02a57201d9e010bbfb94abd11835071 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 11:22:34 -0600 Subject: [PATCH 15/27] test: an unresolvable ref is not the bootstrap case (#918) Reported by OffgridwithJD, who scoped it precisely rather than leading with the headline: they checked the production path FIRST and confirmed it was sound, so "the fail-open hole is back" would have been wrong. The three states this round exists to draw had collapsed back to two on one path. The ref-resolution check lived inside the `auto` resolver, so it covered the production call site and nothing else. An EXPLICIT ref that did not resolve fell through to the file-absent branch: --against refs/heads/no-such-ref-xyz -> rc=0 "no budget at refs/heads/no-such-ref-xyz (distance from HEAD unknown): this change introduces it, so there is no prior ceiling it could have raised" Two things wrong in one line. The verdict is "nothing to compare" where the truth is "could not compare". And it asserts a change introduces a file at a ref that does not exist, one clause after saying the distance from HEAD was unknown -- the code knew it could not resolve the ref and contradicted itself inside one sentence. Resolving now belongs to the reader, where every caller passes through, so the file-absent branch describes only what it claims: a file missing at a ref that IS there. All three states are pinned by arms against one scratch repo carrying a branch with the budget and a branch without it -- unresolvable is rc=2 and never says "introduces", absent-at-an-existing-ref is rc=0 and does, and present compares. Without the last two the fix would be satisfied by refusing everything. Not reachable from the runner, which always passes `auto`. It was a trap for these arms, for anyone driving the tool by hand, and for whoever later passes a concrete ref because `auto` was inconvenient. AND THE PREVIOUS HEAD WENT 12/12 IN CI, with the gate's own output confirming every piece of the redesign: ledger census: rows=693 | never observed red=693, ever red=0, new this run=0 ledger coverage: registered=251 | covered=1, not covered=250, ceiling=250 no budget at refs/remotes/origin/feat/917-machine-readable-results (1 commit behind HEAD): this change introduces it Fed from real matrix logs, refusing nothing because nothing is unledgered, the prior's distance printed beside it, and the bootstrap recognised rather than fatal. Evidence: selftest exit 0, 704 checks, 0 failures; 9 pytest; shellcheck rc=0; docs_style PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/check_ledger.tsv | 8 +++ test/check_ledger_budget.txt | 2 +- test/pgc_ledger.py | 19 +++++++ .../410-a-check-must-have-been-red.sh | 55 +++++++++++++++++++ 4 files changed, 83 insertions(+), 1 deletion(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 78bd9ee3..49f89012 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -615,6 +615,9 @@ harness_selftest 410-a-check-must-have-been-red a later green run does not erase harness_selftest 410-a-check-must-have-been-red a name that appeared while another disappeared is reported as a rename never - harness_selftest 410-a-check-must-have-been-red a named mutation is recorded against the check that reddened never - harness_selftest 410-a-check-must-have-been-red a real refusal is a different status from an integrity failure never - +harness_selftest 410-a-check-must-have-been-red a ref that does not resolve is an integrity failure, not a bootstrap never - +harness_selftest 410-a-check-must-have-been-red a ref that exists with the budget still compares never - +harness_selftest 410-a-check-must-have-been-red a ref that exists without the budget is still the bootstrap case never - harness_selftest 410-a-check-must-have-been-red a refusal keeps the regenerate-the-ledger wording never - harness_selftest 410-a-check-must-have-been-red a rename in one part survives an addition in another never - harness_selftest 410-a-check-must-have-been-red a run whose checks are all ledgered passes the gate never - @@ -630,10 +633,13 @@ harness_selftest 410-a-check-must-have-been-red and an empty mutation is a place harness_selftest 410-a-check-must-have-been-red and it entered as debt, not as an observation nothing made never - harness_selftest 410-a-check-must-have-been-red and it is a different sentence from the refusal, not the same one twice never - harness_selftest 410-a-check-must-have-been-red and it is named, so the author knows which one never - +harness_selftest 410-a-check-must-have-been-red and it is that case that says the change introduces the file never - harness_selftest 410-a-check-must-have-been-red and it is the new one that is named, not the one already ledgered never - harness_selftest 410-a-check-must-have-been-red and it runs before the build directory is removed, which is the only place it can never - harness_selftest 410-a-check-must-have-been-red and it says the change introduces the file rather than raising anything never - +harness_selftest 410-a-check-must-have-been-red and it says the ref does not resolve, rather than claiming the file is new never - harness_selftest 410-a-check-must-have-been-red and it says why, rather than falling back to something weaker never - +harness_selftest 410-a-check-must-have-been-red and never says a change introduces a file at a ref that is not there never - harness_selftest 410-a-check-must-have-been-red and none of them ends in a tab never - harness_selftest 410-a-check-must-have-been-red and not against one that stayed green never - harness_selftest 410-a-check-must-have-been-red and one that stayed green keeps its debt never - @@ -660,11 +666,13 @@ harness_selftest 410-a-check-must-have-been-red nor is one merely removed never harness_selftest 410-a-check-must-have-been-red once the suite is covered, a new check in it IS refused never - harness_selftest 410-a-check-must-have-been-red one --mutation cannot be attributed across several runs at once never - harness_selftest 410-a-check-must-have-been-red premise: and it is a file git has never seen, which is the case that used to fail open never - +harness_selftest 410-a-check-must-have-been-red premise: and the nobudget branch does not, which is the bootstrap shape never - harness_selftest 410-a-check-must-have-been-red premise: the budget is a tracked file too never - harness_selftest 410-a-check-must-have-been-red premise: the budget was restored byte-exact never - harness_selftest 410-a-check-must-have-been-red premise: the check has history before the rename never - harness_selftest 410-a-check-must-have-been-red premise: the fixture log names checks the real ledger already knows never - harness_selftest 410-a-check-must-have-been-red premise: the gate's status block was found in the runner never - +harness_selftest 410-a-check-must-have-been-red premise: the hasbudget branch carries the budget never - harness_selftest 410-a-check-must-have-been-red premise: the ledger is not empty, so the partition means something never - harness_selftest 410-a-check-must-have-been-red premise: the ledger itself is a tracked file, not a variable never - harness_selftest 410-a-check-must-have-been-red premise: the ledger tool exists never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 74df0f47..1c3d8cca 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 693 +checks_never_observed_red 701 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index 635566d4..67f47d07 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -352,6 +352,25 @@ def _committed_budget(path, ref): rel = abspath.relative_to(pathlib.Path(top).resolve()) except ValueError as e: raise LedgerError(f"{path} resolves outside its own repository at {top}") from e + + # DOES THE REF EXIST, asked before anything is read from it. This check used to + # live inside the `auto` resolver, so it covered the production call site and + # nothing else: an EXPLICIT ref that did not resolve fell through to the + # file-absent branch and was reported as the bootstrap case -- rc=0, with a + # message asserting "this change introduces it" about a ref that does not + # exist, one clause after saying the distance from HEAD was unknown. The code + # knew it could not resolve the ref and contradicted itself in one sentence. + # + # Resolving belongs here, where every caller passes through, so that the + # file-absent branch below describes only what it claims: a file missing at a + # ref that IS there. Reported by OffgridwithJD, who scoped it precisely -- + # unreachable from the runner, which always passes `auto`, and a trap for the + # harness arms and anyone driving the tool by hand. + if subprocess.run(["git", "-C", top, "rev-parse", "--verify", "-q", f"{ref}^{{commit}}"], + capture_output=True, text=True).returncode != 0: + raise LedgerError( + f"--against {ref} was given, but that ref does not resolve here, so the prior " + "ceiling cannot be read") # A FILE THAT DOES NOT EXIST AT THE PRIOR HAS NO CEILING TO VIOLATE. This # returns None and the caller notes it, rather than erroring, and the # distinction is the whole of it: introducing the budget is not raising it. diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index 390683d2..dc7a6ea6 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -513,3 +513,58 @@ check "a clean status says nothing and does not fail the major" \ "$(printf '%s' "$_ledcase" | grep -cE '^[[:space:]]+0\)[[:space:]]*;;[[:space:]]*$')" "1" check "both failure arms fail the major" \ "$(printf '%s' "$_ledcase" | grep -c 'verfail=1')" "2" + +# ---- an unresolvable ref is not the bootstrap case -------------------------- +# +# The three states are the point of this part, and one path had collapsed two of +# them. The ref-resolution check lived inside the `auto` resolver, so it covered +# the production call site and nothing else: an EXPLICIT ref that did not resolve +# fell through to the file-absent branch and was reported as rc=0, with a message +# asserting "this change introduces it" about a ref that does not exist -- one +# clause after saying the distance from HEAD was unknown. The code knew it could +# not resolve the ref and contradicted itself inside one sentence. +# +# Unreachable from the runner, which always passes `auto`. A trap for these arms, +# for anyone driving the tool by hand, and for whoever later passes a concrete ref +# because `auto` was inconvenient. Scoped exactly that way by OffgridwithJD, who +# checked the production path first rather than leading with the headline. + +_rr="$_lw/refres"; rm -rf "$_rr"; mkdir -p "$_rr" +( cd "$_rr" && git init -q . && git config user.email t@t && git config user.name t + printf 'suites_not_covered 7\n' > b.txt && git add b.txt && git commit -qm base + git branch -q hasbudget + git rm -q b.txt && git commit -qm "a commit without the budget" + git branch -q nobudget + git checkout -q hasbudget ) >/dev/null 2>&1 +printf 's\tp\ta\tnever\t-\n' > "$_rr/led" +printf 'RESULT\ts\tp\ta\tPASS\t\n' > "$_rr/log" +printf 's\n' > "$_rr/reg" + +check "premise: the hasbudget branch carries the budget" \ + "$(cd "$_rr" && git show hasbudget:b.txt >/dev/null 2>&1 && echo yes || echo no)" "yes" +check "premise: and the nobudget branch does not, which is the bootstrap shape" \ + "$(cd "$_rr" && git show nobudget:b.txt >/dev/null 2>&1 && echo yes || echo no)" "no" + +check "a ref that does not resolve is an integrity failure, not a bootstrap" \ + "$(cd "$_rr" && python3 "$_led" gate --ledger led --budget b.txt --registered reg \ + --against refs/heads/no-such-ref-xyz log >/dev/null 2>&1; echo $?)" "2" +check "and it says the ref does not resolve, rather than claiming the file is new" \ + "$(cd "$_rr" && python3 "$_led" gate --ledger led --budget b.txt --registered reg \ + --against refs/heads/no-such-ref-xyz log 2>&1 | grep -c 'that ref does not resolve here')" "1" +check "and never says a change introduces a file at a ref that is not there" \ + "$(cd "$_rr" && python3 "$_led" gate --ledger led --budget b.txt --registered reg \ + --against refs/heads/no-such-ref-xyz log 2>&1 | grep -c 'this change introduces it')" "0" + +# The bootstrap case must still be the bootstrap case: a ref that EXISTS, without +# the file. Without this arm the fix above is satisfied by refusing everything. +check "a ref that exists without the budget is still the bootstrap case" \ + "$(cd "$_rr" && python3 "$_led" gate --ledger led --budget b.txt --registered reg \ + --against nobudget log >/dev/null 2>&1; echo $?)" "0" +check "and it is that case that says the change introduces the file" \ + "$(cd "$_rr" && python3 "$_led" gate --ledger led --budget b.txt --registered reg \ + --against nobudget log 2>&1 | grep -c 'this change introduces it')" "1" + +# And a ref that exists WITH the file still compares, so the third state is intact. +check "a ref that exists with the budget still compares" \ + "$(cd "$_rr" && python3 "$_led" gate --ledger led --budget b.txt --registered reg \ + --against hasbudget log 2>&1 | grep -c 'ceiling against hasbudget')" "1" From 51b73fdb4b777e30f81a141abf4e40439be86069 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 11:29:23 -0600 Subject: [PATCH 16/27] test: selftest 330 ran a collect loop whose reconciliation could not execute (#917) Two findings OffgridwithJD posted after approving, both real, and the first is the one worth fixing before merge. SELFTEST 330 CLAIMS TO RUN "THE RUNNER'S OWN COLLECT LOOP". It evalled three of the functions that loop calls and not the two #917 added -- pgc_log_shows_accounting and pgc_reconcile_records. The guard is a COMMAND SUBSTITUTION, so the undefined function yielded "", the reconciliation branch never ran, and the first absence hid the second. Measured: a real run printed `command not found` THREE TIMES and still reported PASSED, 608 checks, 0 failures. The part testing the loop end to end was testing a loop with a branch that could not execute. Both are now evalled, with the same non-empty and closing-brace premises the part already applies to the other three. AND THE NEXT CALLER CANNOT REPEAT IT. A premise naming five would go stale the same way, so the loop's own text is swept for the functions it invokes and every one must be defined. A sixth caller added later fails here rather than silently disappearing into a command substitution. THE SECOND FINDING: the pytest mirror's two `part` arms could not fail. The field is `${_part:-${PGC_SUITE:-unknown}}`, so deleting the derivation still yields a non-blank single value -- "exactly one, and not blank" is satisfied by the fallback. They now use the case the field EXISTS for: an outer script sourcing an inner one, where the suite and the part differ. Removing the BASH_SOURCE walk reddens them, lib.sh restored byte-identical. My own sweep read a variable defined 87 lines below it on the first attempt, which is the third time tonight I have written an arm against something not yet in scope. It is placed after the extraction now. Evidence: selftest exit 0, 610 checks, 0 failures, zero `command not found`; pytest; shellcheck rc=0. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- ...test_check_results_are_machine_readable.py | 28 +++++++++-- .../330-the-incomplete-path-must-run-whole.sh | 48 +++++++++++++++---- 2 files changed, 62 insertions(+), 14 deletions(-) diff --git a/test/pytest/test_check_results_are_machine_readable.py b/test/pytest/test_check_results_are_machine_readable.py index c8e5f08f..20ba213f 100644 --- a/test/pytest/test_check_results_are_machine_readable.py +++ b/test/pytest/test_check_results_are_machine_readable.py @@ -72,7 +72,7 @@ def test_lib_sh_counts_a_check_in_exactly_one_place(expect): # ---- the record line -------------------------------------------------------- -def test_each_verdict_emits_one_record_carrying_its_fields(expect): +def test_each_verdict_emits_one_record_carrying_its_fields(tmp_path, expect): """Tab separated -- suite, name, verdict, reason -- so a name with spaces survives. The reason carries the REASON_CODE, which is what makes this more than a reformat: @@ -93,10 +93,28 @@ def test_each_verdict_emits_one_record_carrying_its_fields(expect): # check NAMES, not of checks, and one sharer going red would mark them all. # Derived from BASH_SOURCE rather than from a convention. Found by # OffgridwithJD, whose own six branches were each adding more. - parts = {r.split("\t")[2] for r in _records(' check "a name" x x')} - expect.num(len(parts), 1, "the record names exactly one part") - expect.text("nonempty" if parts and next(iter(parts)) else "empty", "nonempty", - "and the part field is not blank") + # THESE TWO ARMS COULD NOT FAIL, and OffgridwithJD said so. The field is + # `${_part:-${PGC_SUITE:-unknown}}`, so deleting the derivation still yields a + # non-blank single value and both arms stayed green. "Exactly one, and not + # blank" is satisfied by the fallback. + # + # The property needs a case where the part and the suite DIFFER, which is the + # case the field exists for: harness_selftest sources 40-odd parts into one + # shell. So a fixture does the same -- an outer script that sources an inner + # one which calls the check -- and the record must name the INNER file. + outer = tmp_path / "outer_suite.sh" + inner = tmp_path / "inner_part.sh" + inner.write_text('check "a name" x x\n') + outer.write_text(f'. "{LIB}"\n. "{inner}"\n') + r = subprocess.run(["bash", "-c", f'set -uo pipefail\nbash "{outer}"'], + capture_output=True, text=True) + recs = [l for l in r.stdout.splitlines() if l.startswith("RESULT\t")] + expect.num(len(recs), 1, "premise: the sourced part emitted exactly one record") + suite, part = recs[0].split("\t")[1], recs[0].split("\t")[2] + expect.text(suite, "outer_suite", "the suite is the script that ran") + expect.text(part, "inner_part", "and the part is the file the check was asked from") + expect.text("different" if suite != part else "same", "different", + "which are not the same thing, and the fallback would make them so") recs = _records(' check_unrunnable "a name" MISSING_DEPENDENCY "no jq"') expect.num(len(recs), 1, "an unrunnable check emits exactly one record") diff --git a/test/selftest/330-the-incomplete-path-must-run-whole.sh b/test/selftest/330-the-incomplete-path-must-run-whole.sh index ea51103f..6e2075c1 100644 --- a/test/selftest/330-the-incomplete-path-must-run-whole.sh +++ b/test/selftest/330-the-incomplete-path-must-run-whole.sh @@ -92,26 +92,40 @@ check "control fixture: a suite whose checks all ran exits 0" \ _e2e_txt_classify="$(sed -n '/^pgc_classify_suite_rc()/,/^}/p' "$_e2e_rv")" _e2e_txt_fails="$(sed -n '/^pgc_verdict_fails_major()/,/^}/p' "$_e2e_rv")" _e2e_txt_tally="$(sed -n '/^pgc_tally_suite()/,/^}/p' "$_e2e_rv")" +# TWO MORE, because the collect loop grew two callers and this part did not follow. +# +# #917 added a records-versus-count reconciliation inside the loop, guarded by +# pgc_log_shows_accounting. Neither was evalled here, and the guard is a COMMAND +# SUBSTITUTION: an undefined function yields "", the branch never runs, and the +# first absence hides the second. A real run printed `command not found` three +# times and still reported PASSED, 608 checks, 0 failures. +# +# So the part claiming to run "the runner's OWN collect loop" was running a loop +# whose reconciliation could not execute. Reported by OffgridwithJD. +_e2e_txt_anyacct="$(sed -n '/^pgc_log_shows_accounting()/,/^}/p' "$_e2e_rv")" +_e2e_txt_records="$(sed -n '/^pgc_reconcile_records()/,/^}/p' "$_e2e_rv")" -check "premise: all three runner functions were extracted, not empty ranges" \ - "$([ -n "$_e2e_txt_classify" ] && echo y || echo n)$([ -n "$_e2e_txt_fails" ] && echo y || echo n)$([ -n "$_e2e_txt_tally" ] && echo y || echo n)" \ - "yyy" +check "premise: all five runner functions were extracted, not empty ranges" \ + "$([ -n "$_e2e_txt_classify" ] && echo y || echo n)$([ -n "$_e2e_txt_fails" ] && echo y || echo n)$([ -n "$_e2e_txt_tally" ] && echo y || echo n)$([ -n "$_e2e_txt_anyacct" ] && echo y || echo n)$([ -n "$_e2e_txt_records" ] && echo y || echo n)" \ + "yyyyy" # A truncated extraction evals to a syntax error, not to nothing, so the closing # brace is asserted rather than assumed: the sed range stops at the first line # beginning with `}`, and a body containing one would yield a fragment. check "premise: and each extraction ends at its own closing brace" \ - "$(printf '%s\n%s\n%s\n' "$_e2e_txt_classify" "$_e2e_txt_fails" "$_e2e_txt_tally" | grep -c '^}$')" \ - "3" + "$(printf '%s\n%s\n%s\n%s\n%s\n' "$_e2e_txt_classify" "$_e2e_txt_fails" "$_e2e_txt_tally" \ + "$_e2e_txt_anyacct" "$_e2e_txt_records" | grep -c '^}$')" \ + "5" eval "$_e2e_txt_classify" eval "$_e2e_txt_fails" eval "$_e2e_txt_tally" +eval "$_e2e_txt_anyacct" +eval "$_e2e_txt_records" -check "premise: and all three are callable" \ - "$(type -t pgc_classify_suite_rc)/$(type -t pgc_verdict_fails_major)/$(type -t pgc_tally_suite)" \ - "function/function/function" - +check "premise: and all five are callable" \ + "$(type -t pgc_classify_suite_rc)/$(type -t pgc_verdict_fails_major)/$(type -t pgc_tally_suite)/$(type -t pgc_log_shows_accounting)/$(type -t pgc_reconcile_records)" \ + "function/function/function/function/function" check "the runner classifies the file that suite actually produced" \ "$(pgc_classify_suite_rc "$(cat "$_e2e_dir/e2e_incomplete.rc")" "$_e2e_dir/e2e_incomplete.log")" \ "INCOMPLETE" @@ -165,6 +179,22 @@ check "and reprints the suite's own UNRUN line beneath it" \ _e2e_txt_loop="$(awk '/^\tsuites_incomplete=/{f=1} f{print} f&&/^\tdone$/{exit}' "$_e2e_rv")" +# AND NOTHING THE LOOP CALLS MAY BE MISSING. The premise above names five because +# five is what the loop calls today; a sixth caller added later would reproduce +# exactly the failure this fixes, silently. So the loop's own text is swept for +# the functions it invokes, and every one must be defined here. +_e2e_loop_calls="$(printf '%s\n' "$_e2e_txt_loop" | grep -oE '(^|[^_[:alnum:]])pgc_[a-z_]+' \ + | grep -oE 'pgc_[a-z_]+' | sort -u)" +_e2e_undef=0 +while IFS= read -r _e2e_fn; do + [ -n "$_e2e_fn" ] || continue + [ "$(type -t "$_e2e_fn" 2>/dev/null)" = function ] || { + _e2e_undef=$((_e2e_undef + 1)); echo " the collect loop calls $_e2e_fn, which this part never evalled"; } +done <<<"$_e2e_loop_calls" +check "premise: the sweep found the loop's callers to check" \ + "$([ "$(printf '%s' "$_e2e_loop_calls" | grep -c .)" -ge 3 ] && echo yes || echo no)" "yes" +check "every function the collect loop calls is defined here" "$_e2e_undef" "0" + check "premise: the runner's collect loop was extracted, not an empty range" \ "$([ -n "$_e2e_txt_loop" ] && echo yes || echo no)" "yes" From 595d701e716fdd73f5c0282399ffd3f4f16574f2 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 13:56:44 -0600 Subject: [PATCH 17/27] test: the record count is not the record schema (#917) @linuxhikerpm's finding 1: pgc_reconcile_records counted lines beginning `RESULT` and compared that to `checks run:`. It never looked inside a record, so five malformed shapes reconciled cleanly. Measured, each against a well-formed control that reconciled the same way, so the function could not tell them apart: rc=0 ACCEPTED a well-formed record (control) rc=0 ACCEPTED NF=4: two fields missing rc=0 ACCEPTED NF=8: extra fields rc=0 ACCEPTED verdict BOGUS rc=0 ACCEPTED empty check name rc=0 ACCEPTED every field empty Now one awk pass validates the schema: exactly five fields after the marker, a non-empty suite, part and name, and a verdict from the emitter's own list. One pass rather than a loop, because a matrix run carries thousands of these and the emitter next door already paid for that lesson at 331x. pgc_record also strips newlines and CR, not only tabs. The comment beside the tab strip already gave the reason -- "a tab in a field would split it" -- and a newline splits the record more completely, ending the line so the remainder becomes text no reader can key. Measured before the fix: a newline in the name produced a 4-field record plus two stray lines; after it, 6 fields and one line. THE RECONCILIATION FIXTURES WERE STALE AND THAT IS WHY THE ARMS WENT RED FIRST. Lines 169-185 built `RESULT\ts\ta\tPASS\t`, a four-field record from before this branch added `part`, while the population fixtures thirty lines below already used five. They asserted on a shape pgc_record cannot emit. Both harnesses' fixtures now use the emitter's shape. Nine new arms, and the control is asserted FIRST and deliberately: five arms that all report "mismatch" prove nothing if the function has started refusing everything, which is the shape this suite exists to catch. Proved both ways: validation never fires -> 4 red, exactly the malformed shapes validation always fires -> the control and all four verdict arms red selftest 400 is 64 checks to 73. On the composed tree: 350 53/53, 080 15/15, the driver-free job 10 files 170 passed, membership_report []. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/lib.sh | 17 +++++-- ...test_check_results_are_machine_readable.py | 10 ++-- test/run_all_versions.sh | 32 +++++++++++++ .../400-a-check-result-must-be-machine.sh | 46 +++++++++++++++++-- 4 files changed, 92 insertions(+), 13 deletions(-) diff --git a/test/lib.sh b/test/lib.sh index 3228f904..012d795c 100755 --- a/test/lib.sh +++ b/test/lib.sh @@ -1049,8 +1049,14 @@ pgc_record() { # pgc_record VERDICT NAME DISPLAY [REASON] done printf '%s\n' "$_display" - # Tabs in a field would split it. Nothing in the tree puts one in a check - # name, and this makes that true rather than assumed. + # Tabs in a field would split it, and a NEWLINE splits the whole record just + # as completely -- it ends the line, so what follows becomes a second line the + # reader cannot key. Nothing in the tree puts either in a check name, and this + # makes that true rather than assumed. The tab was stripped here from the + # first version; the newline was not, which @linuxhikerpm named on #917: the + # reason already written for the tab is the reason for both. Measured before + # the fix, a newline in the name gave a record of 4 fields plus two stray + # lines; after it, 6 fields and one line. # # PARAMETER EXPANSION, not `printf | tr` in a command substitution. The first # version paid four forks per record -- two subshells and two tr processes -- @@ -1059,12 +1065,15 @@ pgc_record() { # pgc_record VERDICT NAME DISPLAY [REASON] # an idle box, 2,000 calls, identical output on every input including a real # tab: 3.1577 ms per call against 0.0096 ms, 331x, or 11.9 seconds of pure # fork overhead across a full suite against 36 ms. Reported by OffgridwithJD. + local _nl_name="${_name//$'\t'/ }" _nl_reason="${_reason//$'\t'/ }" + _nl_name="${_nl_name//$'\n'/ }"; _nl_reason="${_nl_reason//$'\n'/ }" + _nl_name="${_nl_name//$'\r'/ }"; _nl_reason="${_nl_reason//$'\r'/ }" printf 'RESULT\t%s\t%s\t%s\t%s\t%s\n' \ "${PGC_SUITE:-unknown}" \ "${_part:-${PGC_SUITE:-unknown}}" \ - "${_name//$'\t'/ }" \ + "${_nl_name}" \ "$_v" \ - "${_reason//$'\t'/ }" + "${_nl_reason}" } pgc_pass() { # pgc_pass NAME diff --git a/test/pytest/test_check_results_are_machine_readable.py b/test/pytest/test_check_results_are_machine_readable.py index 20ba213f..21dc1de0 100644 --- a/test/pytest/test_check_results_are_machine_readable.py +++ b/test/pytest/test_check_results_are_machine_readable.py @@ -196,20 +196,20 @@ def run(text): r = subprocess.run(["bash", "-c", script], capture_output=True, text=True) return r.stdout, r.returncode - ok = "RESULT\ts\ta\tPASS\t\nRESULT\ts\tb\tPASS\t\nchecks run: 2\n" + ok = "RESULT\ts\tp\ta\tPASS\t\nRESULT\ts\tp\tb\tPASS\t\nchecks run: 2\n" expect.num(run(ok)[1], 0, "a log whose records match its stated count reconciles") - out, rc = run("RESULT\ts\ta\tPASS\t\nchecks run: 2\n") + out, rc = run("RESULT\ts\tp\ta\tPASS\t\nchecks run: 2\n") expect.num(rc, 1, "a log with fewer records than it claims is caught") expect.num(out.count("records=1"), 1, "and both numbers are named, not just the verdict") - expect.num(run("RESULT\ts\ta\tPASS\t\nRESULT\ts\tb\tPASS\t\n" - "RESULT\ts\tc\tPASS\t\nchecks run: 2\n")[1], 1, + expect.num(run("RESULT\ts\tp\ta\tPASS\t\nRESULT\ts\tp\tb\tPASS\t\n" + "RESULT\ts\tp\tc\tPASS\t\nchecks run: 2\n")[1], 1, "a log with more records than it claims is caught too") # A log with no count at all never reached its summary. That is a different fault # from a miscount and must not read as a clean reconciliation. - expect.num(run("RESULT\ts\ta\tPASS\t\n")[1], 1, + expect.num(run("RESULT\ts\tp\ta\tPASS\t\n")[1], 1, "a log that never stated a count is not silently accepted") diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index 0aa2853a..db5c99c9 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -956,6 +956,38 @@ pgc_reconcile_records() { # pgc_reconcile_records LOGFILE -> 0 ok, 1 mismatch return 1 fi _records="$(grep -c '^RESULT ' "$_log" || true)" + + # THE COUNT IS NOT THE SCHEMA, and counting alone let six malformed shapes + # reconcile cleanly: a record missing two fields, one carrying extra fields, + # a verdict outside the vocabulary, an empty check name, and every field + # empty. All measured returning 0 before this arm, against a well-formed + # control that also returned 0 -- so the function could not tell them apart. + # Reported by @linuxhikerpm on #917. + # + # ONE awk PASS, not a loop with a fork per record: a full matrix run carries + # thousands of these, and the emitter next door already paid for that lesson + # at 331x. The verdict list is the emitter's own, so the two cannot drift + # without this going red. + local _bad + _bad="$(awk -F'\t' ' + /^RESULT / { + n++ + if (NF != 6) { why[n] = "has " NF-1 " fields, want 5"; bad++; next } + if ($2 == "" || $3 == "" || $4 == "") { why[n] = "has an empty suite, part or name"; bad++; next } + if ($5 != "PASS" && $5 != "FAIL" && $5 != "UNRUN" && $5 != "SKIP") { + why[n] = "has verdict \"" $5 "\", which pgc_record cannot emit"; bad++; next + } + } + END { + if (bad) { for (i = 1; i <= n; i++) if (i in why) print " record " i " " why[i] } + exit 0 + }' "$_log")" + if [ -n "$_bad" ]; then + echo " $_records record(s) present but at least one does not parse:" + printf '%s\n' "$_bad" | head -5 + return 1 + fi + _stated="$(sed -n 's/^checks run: \([0-9][0-9]*\)$/\1/p' "$_log" | tail -1)" if [ -z "$_stated" ]; then echo " records=$_records but the log never stated a count, so it did not reach its summary" diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index 806fa491..28973c1e 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -166,29 +166,67 @@ eval "$(sed -n '/^pgc_reconcile_records()/,/^}/p' "$_rv")" check "premise: it is callable" "$(type -t pgc_reconcile_records)" "function" _rl="$PGC_WORKDIR/rec.log" -printf 'RESULT\ts\ta\tPASS\t\nRESULT\ts\tb\tPASS\t\nchecks run: 2\n' > "$_rl" +printf 'RESULT\ts\tp\ta\tPASS\t\nRESULT\ts\tp\tb\tPASS\t\nchecks run: 2\n' > "$_rl" check "a log whose records match its stated count reconciles" \ "$(pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch)" "ok" -printf 'RESULT\ts\ta\tPASS\t\nchecks run: 2\n' > "$_rl" +printf 'RESULT\ts\tp\ta\tPASS\t\nchecks run: 2\n' > "$_rl" check "a log with fewer records than it claims is caught" \ "$(pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch)" "mismatch" check "and the two numbers are named, not just the verdict" \ "$(pgc_reconcile_records "$_rl" 2>&1 | grep -c 'records=1 .*checks run: 2')" "1" -printf 'RESULT\ts\ta\tPASS\t\nRESULT\ts\tb\tPASS\t\nRESULT\ts\tc\tPASS\t\nchecks run: 2\n' > "$_rl" +printf 'RESULT\ts\tp\ta\tPASS\t\nRESULT\ts\tp\tb\tPASS\t\nRESULT\ts\tp\tc\tPASS\t\nchecks run: 2\n' > "$_rl" check "a log with more records than it claims is caught too" \ "$(pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch)" "mismatch" # A log with no `checks run:` line at all did not reach its summary. That is a # different fault from a miscount and must not read as a clean reconciliation. -printf 'RESULT\ts\ta\tPASS\t\n' > "$_rl" +printf 'RESULT\ts\tp\ta\tPASS\t\n' > "$_rl" check "a log that never stated a count is not silently accepted" \ "$(pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch)" "mismatch" check "the runner calls the record reconciliation, not merely defines it" \ "$(grep -c '[^_[:alnum:]]pgc_reconcile_records "' "$_rv")" "1" +# THE COUNT IS NOT THE SCHEMA. Counting `^RESULT` lines and comparing to +# `checks run:` accepted every malformed record below, each against a well-formed +# control that reconciled the same way -- so the function returned 0 whether the +# record parsed or not. Named by @linuxhikerpm on #917. +# +# The control comes FIRST and is asserted, because five arms that all say +# "mismatch" prove nothing if the function has simply started refusing +# everything. That is the shape this suite exists to catch. +printf 'RESULT\ts\tp\ta\tPASS\t\nchecks run: 1\n' > "$_rl" +check "control: a well-formed record still reconciles" \ + "$(pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch)" "ok" + +_rq_bad() { # _rq_bad RECORD -> ok|mismatch, with the count always matching + printf '%s\nchecks run: 1\n' "$1" > "$_rl" + pgc_reconcile_records "$_rl" >/dev/null 2>&1 && echo ok || echo mismatch +} + +check "a record missing fields does not reconcile" \ + "$(_rq_bad "$(printf 'RESULT\ts\tp\ta')")" "mismatch" + +check "a record carrying extra fields does not reconcile" \ + "$(_rq_bad "$(printf 'RESULT\ts\tp\tn\tPASS\tr\textra\tmore')")" "mismatch" + +check "a verdict pgc_record cannot emit does not reconcile" \ + "$(_rq_bad "$(printf 'RESULT\ts\tp\tn\tBOGUS\t')")" "mismatch" + +check "an empty check name does not reconcile" \ + "$(_rq_bad "$(printf 'RESULT\ts\tp\t\tPASS\t')")" "mismatch" + +# The four verdicts are the emitter's own list. If pgc_record grows a fifth and +# this one does not, this arm goes red rather than the vocabulary drifting. +for _rq_v in PASS FAIL UNRUN SKIP; do + check "the reconciliation accepts the verdict $_rq_v, which pgc_record emits" \ + "$(_rq_bad "$(printf 'RESULT\ts\tp\tn\t%s\t' "$_rq_v")")" "ok" +done +unset -f _rq_bad +unset _rq_v + # ---- the timing helpers reported an outcome that nothing counted ------------- # # check_timing and check_ratio_needs_quiet_machine, under PGC_SKIP_TIMING=1, From 54130531258a7a7ed7228c81c3c1fc17e4275c97 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 14:22:28 -0600 Subject: [PATCH 18/27] test: a named SKIP is an outcome, so it is counted and recorded (#917) @linuxhikerpm's finding 2. `echo "SKIP ..."` printed a line a reader sees and left PGC_CHECKS alone, so the outcome existed for a human and for nobody else: no record, no count, nothing for pgc_reconcile_records to reconcile. The tree already said why this mattered, at native_index_projection.sh: "the skip must be visible: a check that reports nothing is indistinguishable from a check that passes". That was TRUE while the human line WAS the record. It stopped being true when the RESULT stream became the machine-readable one, and these sites were left on the wrong side of the change. 25 sites converted to a new `check_skip NAME DISPLAY [REASON]`, which is pgc_record with the verdict fixed: 22 in suites, the two in selftest 010, and lib.sh's own pgc_skip waiver -- where the unwaived branch records a FAIL and the waived one printed and vanished, in the function whose subject is "a missing dependency is not a pass". DISPLAY is passed whole, so every human line is byte-identical. These messages are individually worded and people grep them; recomposing them would change what a reader sees for no gain. Measured on one: counted 1, skipped 1, one record of five fields with verdict SKIP, and the printed line unchanged. FIVE SITES ARE DELIBERATELY LEFT, and counting them as offenders was my error before I read them. Four in sorted_mark_rename.sh are `check` ARGUMENTS whose value is the string "SKIPPED (gate fired wrongly)" -- already counted checks, not skips. The fifth is run_all_versions.sh declining a whole PostgreSQL major before any suite exists, so there is no PGC_CHECKS for it to belong to. The anti-drift arm is derived rather than a filename list: a file that calls `check` is a suite or a part and its skips are check outcomes; the runner calls `check` zero times. So a new runner or a new suite is classified without editing the arm. It carries a premise that it classified 50+ files, a fixture proving it names a check-calling file that echoes a SKIP, and a fixture proving it leaves a non-suite alone. Reverting ONE site reddens it: got [1] want [0]. selftest 400 is 73 checks to 77. 350 53/53, 080 15/15, the driver-free corpus 10 files 170 passed, all 22 changed shell files parse. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/advisory_lock_class.sh | 2 +- test/analyze_differential.sh | 2 +- test/analyze_function.sh | 2 +- test/lib.sh | 27 +++++++++- test/logical_subscriber.sh | 2 +- test/native_groupagg.sh | 2 +- test/native_groupagg_batch.sh | 2 +- test/native_index_projection.sh | 2 +- test/native_parquet_flba.sh | 4 +- test/native_parquet_multifile.sh | 2 +- test/native_parquet_pushdown.sh | 4 +- test/native_parquet_schema.sh | 2 +- test/native_repack.sh | 2 +- test/objstore_module.sh | 2 +- test/objstore_stash_recovery.sh | 2 +- test/parquet_nested_import.sh | 2 +- test/pg19_vacuum_options.sh | 2 +- test/selftest/010-stand-up-a-squatter-on-a.sh | 4 +- .../340-the-binary-must-be-built-from.sh | 2 +- .../400-a-check-result-must-be-machine.sh | 52 +++++++++++++++++++ test/sorted_pathkeys.sh | 2 +- test/unique_conc.sh | 4 +- 22 files changed, 102 insertions(+), 25 deletions(-) diff --git a/test/advisory_lock_class.sh b/test/advisory_lock_class.sh index f69659af..a6c17bd1 100755 --- a/test/advisory_lock_class.sh +++ b/test/advisory_lock_class.sh @@ -127,7 +127,7 @@ if [ -n "$LK_CLASSID" ] && [ "$LK_CLASSID" -le 2147483647 ] && [ "$LK_OBJID" -le sleep 0.2 done else - echo "SKIP classid $LK_CLASSID or objid $LK_OBJID exceeds int4, so the SQL form cannot address it" + check_skip "the SQL form of the advisory lock" "SKIP classid $LK_CLASSID or objid $LK_OBJID exceeds int4, so the SQL form cannot address it" "classid or objid exceeds int4" fi # --------------------------------------------------------------------------- diff --git a/test/analyze_differential.sh b/test/analyze_differential.sh index 443e09a8..a2e66de3 100755 --- a/test/analyze_differential.sh +++ b/test/analyze_differential.sh @@ -59,7 +59,7 @@ if ! pgc_is_number "${PGC_MAJOR:-}"; then pgc_summary fi if [ "$PGC_MAJOR" -lt 18 ]; then - echo "SKIP pgcolumnar.analyze() needs pg_restore_attribute_stats (PG18+); this server is $PGC_MAJOR" + check_skip "the differential analyze path" "SKIP pgcolumnar.analyze() needs pg_restore_attribute_stats (PG18+); this server is $PGC_MAJOR" "needs pg_restore_attribute_stats, PG18+" pgc_summary fi diff --git a/test/analyze_function.sh b/test/analyze_function.sh index 10a133f4..63efdb1d 100755 --- a/test/analyze_function.sh +++ b/test/analyze_function.sh @@ -68,7 +68,7 @@ if ! pgc_is_number "${PGC_MAJOR:-}"; then pgc_summary fi if [ "$PGC_MAJOR" -lt 18 ]; then - echo "SKIP pgcolumnar.analyze() needs pg_restore_attribute_stats (PG18+); this server is $PGC_MAJOR" + check_skip "pgcolumnar.analyze()" "SKIP pgcolumnar.analyze() needs pg_restore_attribute_stats (PG18+); this server is $PGC_MAJOR" "needs pg_restore_attribute_stats, PG18+" pgc_summary fi diff --git a/test/lib.sh b/test/lib.sh index 012d795c..ca1abdee 100755 --- a/test/lib.sh +++ b/test/lib.sh @@ -1262,6 +1262,26 @@ pgc_require_tools() { # So the ratio is skipped and the rest of the suite runs. A skip is announced # rather than silent, and it is not counted as a pass, because a count that # includes checks nobody ran is the thing this project keeps having to unlearn. +# A named check that could not run HERE, for a reason the suite knows. +# +# WHY THIS EXISTS. `echo "SKIP ..."` printed a line a reader sees and left +# PGC_CHECKS alone, so the outcome existed for a human and for nobody else: no +# record, no count, and nothing for `pgc_reconcile_records` to reconcile. The +# tree's own comment at native_index_projection.sh said why that mattered -- "the +# skip must be visible: a check that reports nothing is indistinguishable from a +# check that passes" -- and that was TRUE while the human line WAS the record. It +# stopped being true when the RESULT stream became the machine-readable one, and +# 22 sites were left behind on the wrong side of the change. Named by +# @linuxhikerpm on #917; the owner asked for every site, not the three examples. +# +# DISPLAY IS PASSED WHOLE, exactly as pgc_record takes it, so every existing +# human line stays byte-identical. These messages are individually worded and a +# reader greps them; recomposing them here would change what people see for no +# gain, which is the same reason pgc_record does not compose PASS lines either. +check_skip() { # check_skip NAME DISPLAY [REASON] + pgc_record SKIP "$1" "$2" "${3:-}" +} + check_timing() { local name="$1" got="$2" want="$3" @@ -1564,7 +1584,12 @@ pgc_skip() { # pgc_skip cap="$(printf '%s' "$1" | tr '[:lower:]-' '[:upper:]_')" allow_one="PGC_ALLOW_MISSING_$cap" if [ "${PGC_ALLOW_MISSING:-0}" = 1 ] || [ "${!allow_one:-0}" = 1 ]; then - echo "SKIP $2 (waived by $allow_one or PGC_ALLOW_MISSING)" + # The unwaived branch below records a FAIL. This one printed and left + # PGC_CHECKS at zero, so waiving a dependency also erased the outcome -- + # the same asymmetry check_timing had, in the function whose whole + # subject is "a missing dependency is not a pass". + check_skip "$2" "SKIP $2 (waived by $allow_one or PGC_ALLOW_MISSING)" \ + "waived by $allow_one or PGC_ALLOW_MISSING" pgc_summary fi pgc_record FAIL "$2" "FAIL $2" diff --git a/test/logical_subscriber.sh b/test/logical_subscriber.sh index 045f93c9..40b35088 100755 --- a/test/logical_subscriber.sh +++ b/test/logical_subscriber.sh @@ -50,7 +50,7 @@ pick_port() { } SUB_PORT="$(pick_port)" if [ "$SUB_PORT" = 0 ]; then - echo "SKIP no free port for the subscriber cluster" + check_skip "the logical subscriber round trip" "SKIP no free port for the subscriber cluster" "no free port for the subscriber cluster" pgc_summary; exit 0 fi diff --git a/test/native_groupagg.sh b/test/native_groupagg.sh index 85b3dbbf..ee055ef7 100755 --- a/test/native_groupagg.sh +++ b/test/native_groupagg.sh @@ -263,7 +263,7 @@ if [ "$(q "SELECT 1 FROM pg_collation WHERE collname = 'ci'")" = "1" ]; then "$(q "SELECT count(*) FROM (SELECT k FROM t_ci GROUP BY k) s")" \ "$(q "SELECT count(DISTINCT lower(k)) FROM t_ci")" else - echo "SKIP non-deterministic collation (ICU unavailable)" + check_skip "the non-deterministic collation case" "SKIP non-deterministic collation (ICU unavailable)" "ICU unavailable" fi # an output expression built on a group key (not a bare key) -> falls back diff --git a/test/native_groupagg_batch.sh b/test/native_groupagg_batch.sh index 72aa57d7..ecf5fdc9 100755 --- a/test/native_groupagg_batch.sh +++ b/test/native_groupagg_batch.sh @@ -176,7 +176,7 @@ for pair in "text key:$Q_TEXTKEY" "expression key:$Q_EXPRKEY" \ else # The node itself declined the shape; there is no fold line to read and # nothing for this suite to gate. Say so rather than assert a missing line. - echo "SKIP $label: the grouped node is not planned for this shape" + check_skip "$label" "SKIP $label: the grouped node is not planned for this shape" "the grouped node is not planned for this shape" fi agree "$label: answers match the heap mirror" "$tmpl" done diff --git a/test/native_index_projection.sh b/test/native_index_projection.sh index 77a16640..29616459 100755 --- a/test/native_index_projection.sh +++ b/test/native_index_projection.sh @@ -194,7 +194,7 @@ if psql_run "CREATE EXTENSION IF NOT EXISTS amcheck;" >/dev/null 2>&1 && "$(grep -qE 'ERROR' <<<"$out" && echo bad || echo ok)" "ok" done else - echo "SKIP amcheck is not installed on this build; the seq-scan oracle above still ran" + check_skip "the amcheck oracle for $idx" "SKIP amcheck is not installed on this build; the seq-scan oracle above still ran" "amcheck is not installed on this build" fi pgc_summary diff --git a/test/native_parquet_flba.sh b/test/native_parquet_flba.sh index 2db4d09b..3e5719d3 100755 --- a/test/native_parquet_flba.sh +++ b/test/native_parquet_flba.sh @@ -208,7 +208,7 @@ if phys != {"i32": "INT32", "i64": "INT64"}: sys.exit("unexpected physical types: %s" % phys) PYINT if [ $? -ne 0 ]; then - echo "SKIP this pyarrow does not store decimals as integers as expected" + check_skip "the integer-backed decimal case" "SKIP this pyarrow does not store decimals as integers as expected" "this pyarrow does not store decimals as integers" else check "INT32-backed DECIMAL reads" \ "$(q "SELECT string_agg(d::text, ',' ORDER BY d) FROM pgcolumnar.read_parquet('$W/dec_i32.parquet') AS t(d numeric);")" \ @@ -231,7 +231,7 @@ PYINT "-3500000,0,1250000" fi else - echo "SKIP pyarrow not available; foreign-producer FLBA cases skipped" + check_skip "the foreign-producer FLBA cases" "SKIP pyarrow not available; foreign-producer FLBA cases skipped" "pyarrow not available" fi pgc_summary diff --git a/test/native_parquet_multifile.sh b/test/native_parquet_multifile.sh index 038918c6..4bad5fb2 100755 --- a/test/native_parquet_multifile.sh +++ b/test/native_parquet_multifile.sh @@ -172,7 +172,7 @@ if mkfifo "$DIR/pipe.parquet" 2>/dev/null; then check "FIFO named *.parquet is skipped, does not block" "$fifo_out" "3000" rm -f "$DIR/pipe.parquet" else - echo "SKIP mkfifo unavailable; FIFO case not exercised" + check_skip "the FIFO case" "SKIP mkfifo unavailable; FIFO case not exercised" "mkfifo unavailable" fi # ---- recursive walk -------------------------------------------------------- diff --git a/test/native_parquet_pushdown.sh b/test/native_parquet_pushdown.sh index 72e7403f..7ab4ab15 100755 --- a/test/native_parquet_pushdown.sh +++ b/test/native_parquet_pushdown.sh @@ -211,7 +211,7 @@ if f.metadata.num_row_groups != 4: sys.exit("expected 4 row groups") PYDEC if [ $? -ne 0 ]; then - echo "SKIP could not build the integer-DECIMAL pushdown file" + check_skip "the integer-DECIMAL pushdown case" "SKIP could not build the integer-DECIMAL pushdown file" "could not build the fixture file" else psql_run "CREATE FOREIGN TABLE ftdec (d numeric) SERVER pq OPTIONS (path '$PGC_WORKDIR/dec_push.parquet');" @@ -225,7 +225,7 @@ PYDEC "$(skipped_for_t ftdec 'd >= 0')" "0" fi else - echo "SKIP pyarrow not available; integer-DECIMAL pushdown case skipped" + check_skip "the integer-DECIMAL pushdown case" "SKIP pyarrow not available; integer-DECIMAL pushdown case skipped" "pyarrow not available" fi pgc_summary diff --git a/test/native_parquet_schema.sh b/test/native_parquet_schema.sh index d49ede6e..d552273c 100755 --- a/test/native_parquet_schema.sh +++ b/test/native_parquet_schema.sh @@ -133,7 +133,7 @@ PY "$(q "SELECT count(*) FILTER (WHERE field_id IS NULL) || '/' || count(*) FROM pgcolumnar.parquet_schema('$REQ');")" \ "2/2" else - echo "SKIP pyarrow not available; REQUIRED-column and field-id checks skipped" + check_skip "the REQUIRED-column and field-id checks" "SKIP pyarrow not available; REQUIRED-column and field-id checks skipped" "pyarrow not available" fi pgc_summary diff --git a/test/native_repack.sh b/test/native_repack.sh index ef671131..7ea82479 100755 --- a/test/native_repack.sh +++ b/test/native_repack.sh @@ -58,7 +58,7 @@ if ! pgc_is_number "$srv"; then pgc_summary fi if [ "$srv" -lt 190000 ]; then - echo "SKIP REPACK requires PostgreSQL 19 (server_version_num=$srv)" + check_skip "REPACK" "SKIP REPACK requires PostgreSQL 19 (server_version_num=$srv)" "requires PostgreSQL 19" pgc_summary fi diff --git a/test/objstore_module.sh b/test/objstore_module.sh index 3aea7f2b..4f03b08b 100755 --- a/test/objstore_module.sh +++ b/test/objstore_module.sh @@ -172,7 +172,7 @@ objstore_teardown() { restore_module; pgc_teardown; } trap objstore_teardown EXIT INT TERM if ! mv "$MOD" "$MOD.probe" 2>/dev/null; then - echo "SKIP cannot move $MOD, so the absent and broken paths are untested here" + check_skip "the absent and broken module paths" "SKIP cannot move $MOD, so the absent and broken paths are untested here" "cannot move $MOD" pgc_summary exit 0 fi diff --git a/test/objstore_stash_recovery.sh b/test/objstore_stash_recovery.sh index 81842b4e..1def6b88 100755 --- a/test/objstore_stash_recovery.sh +++ b/test/objstore_stash_recovery.sh @@ -89,7 +89,7 @@ cleanup() { trap cleanup EXIT INT TERM if ! mod_is_valid "$MOD"; then - echo "SKIP no valid module installed at $MOD, so there is no state to arrange" + check_skip "stash recovery" "SKIP no valid module installed at $MOD, so there is no state to arrange" "no valid module installed at $MOD" pgc_summary fi SAFE="$(mktemp /tmp/pgc-objstore-safe.XXXXXX)" diff --git a/test/parquet_nested_import.sh b/test/parquet_nested_import.sh index 3be75c41..6ab49426 100755 --- a/test/parquet_nested_import.sh +++ b/test/parquet_nested_import.sh @@ -86,7 +86,7 @@ then else rc=$? if [ "$rc" = 3 ]; then - echo "SKIP: pyarrow not available for the reference-writer case" + check_skip "the reference-writer case" "SKIP: pyarrow not available for the reference-writer case" "pyarrow not available" else echo "FAIL: pyarrow nested file generation errored (rc=$rc)" PGC_FAIL=1 diff --git a/test/pg19_vacuum_options.sh b/test/pg19_vacuum_options.sh index 3ee7f6d0..1e281f25 100755 --- a/test/pg19_vacuum_options.sh +++ b/test/pg19_vacuum_options.sh @@ -44,7 +44,7 @@ if ! pgc_is_number "$srv"; then pgc_summary fi if [ "$srv" -lt 190000 ]; then - echo "SKIP parallel autovacuum requires PostgreSQL 19 (server_version_num=$srv)" + check_skip "parallel autovacuum" "SKIP parallel autovacuum requires PostgreSQL 19 (server_version_num=$srv)" "requires PostgreSQL 19" pgc_summary fi diff --git a/test/selftest/010-stand-up-a-squatter-on-a.sh b/test/selftest/010-stand-up-a-squatter-on-a.sh index cd56d170..d49330aa 100644 --- a/test/selftest/010-stand-up-a-squatter-on-a.sh +++ b/test/selftest/010-stand-up-a-squatter-on-a.sh @@ -16,7 +16,7 @@ for _try in $(seq 1 20); do fi done if [ "$SQ_PORT" = 0 ]; then - echo "SKIP could not find a free port for the squatter cluster" + check_skip "the squatter cluster" "SKIP could not find a free port for the squatter cluster" "no free port" rm -rf "$SQ_DIR" exit 0 fi @@ -50,7 +50,7 @@ sq_datadir() { } if [ -z "$(sq_datadir)" ]; then - echo "SKIP could not stand up a squatter cluster to test against" + check_skip "the squatter cluster" "SKIP could not stand up a squatter cluster to test against" "could not stand it up" squatter_down exit 0 fi diff --git a/test/selftest/340-the-binary-must-be-built-from.sh b/test/selftest/340-the-binary-must-be-built-from.sh index 584a01ca..3daa5c62 100644 --- a/test/selftest/340-the-binary-must-be-built-from.sh +++ b/test/selftest/340-the-binary-must-be-built-from.sh @@ -557,7 +557,7 @@ _fp_as() { # _fp_as EXPR -> stdout } if [ -z "$_fp_user" ]; then - echo "SKIP no non-root user to read as; root ignores chmod 000" + check_skip "the unreadable-source refusal" "SKIP no non-root user to read as; root ignores chmod 000" "no non-root user to read as" else _fp_base="$(_fp_as "pgc_source_fingerprint \"$_fp/tree\"")" check "premise: the tree fingerprints to something when it is readable" \ diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index 28973c1e..25165865 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -227,6 +227,58 @@ done unset -f _rq_bad unset _rq_v +# ---- a named SKIP is an outcome, so it is counted and recorded -------------- +# +# `echo "SKIP ..."` printed a line a reader sees and left PGC_CHECKS alone, so +# 22 sites across 20 files reported an outcome that no count and no record ever +# saw. The tree's own comment said why that mattered -- "the skip must be +# visible: a check that reports nothing is indistinguishable from a check that +# passes" -- and it was true while the human line WAS the record. +# +# THE EXEMPTION IS DERIVED, NOT A FILENAME LIST. A file that calls `check` is a +# suite or a part, and its skips are check outcomes. run_all_versions.sh calls +# `check` zero times because it is the runner: its one `echo "SKIP"` declines a +# whole PostgreSQL major before any suite exists, so there is no PGC_CHECKS for +# it to belong to. That distinction is read off the files rather than written +# here, so a new runner or a new suite is classified without editing this arm. +_sk_offenders="" +for _sk_f in "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh; do + [ -e "$_sk_f" ] || continue + # A comment is not a statement, so the pattern anchors to the line start + # after whitespace only. Measured: this arm's own prose above says + # echo "SKIP" and is not matched, which a looser pattern would flag. + [ "$(grep -cE '^[[:space:]]*check(_[a-z_]+)? ' "$_sk_f")" -gt 0 ] || continue + if [ "$(grep -cE '^[[:space:]]*echo "SKIP' "$_sk_f")" -gt 0 ]; then + _sk_offenders="$_sk_offenders ${_sk_f##*/}" + fi +done +check "no file that calls check prints a SKIP outcome the count cannot see" \ + "$(printf '%s' "$_sk_offenders" | wc -w | tr -d ' ')" "0" +[ -z "$_sk_offenders" ] || printf ' %s\n' $_sk_offenders + +# The sweep has to be looking at something, and it has to be able to find one. +_sk_seen=0 +for _sk_f in "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh; do + [ -e "$_sk_f" ] || continue + [ "$(grep -cE '^[[:space:]]*check(_[a-z_]+)? ' "$_sk_f")" -gt 0 ] && _sk_seen=$((_sk_seen + 1)) +done +check "premise: the sweep classified a corpus of check-calling files" \ + "$([ "$_sk_seen" -ge 50 ] && echo yes || echo "no ($_sk_seen)")" "yes" + +_sk_fix="$PGC_WORKDIR/skipsweep"; rm -rf "$_sk_fix"; mkdir -p "$_sk_fix" +printf 'check "x" a a\necho "SKIP a bare skip"\n' > "$_sk_fix/offender.sh" +check "premise: and it would name a file that calls check and echoes a SKIP" \ + "$([ "$(grep -cE '^[[:space:]]*check(_[a-z_]+)? ' "$_sk_fix/offender.sh")" -gt 0 ] \ + && [ "$(grep -cE '^[[:space:]]*echo "SKIP' "$_sk_fix/offender.sh")" -gt 0 ] \ + && echo caught || echo missed)" "caught" + +printf 'echo "SKIP a bare skip"\n' > "$_sk_fix/runner.sh" +check "premise: while a file that calls no check is not its business" \ + "$([ "$(grep -cE '^[[:space:]]*check(_[a-z_]+)? ' "$_sk_fix/runner.sh")" -gt 0 ] \ + && echo caught || echo "not a suite")" "not a suite" + +unset _sk_offenders _sk_f _sk_seen _sk_fix + # ---- the timing helpers reported an outcome that nothing counted ------------- # # check_timing and check_ratio_needs_quiet_machine, under PGC_SKIP_TIMING=1, diff --git a/test/sorted_pathkeys.sh b/test/sorted_pathkeys.sh index aa4090cb..a5c29c70 100755 --- a/test/sorted_pathkeys.sh +++ b/test/sorted_pathkeys.sh @@ -275,7 +275,7 @@ ansp "and it answers in C order, matching heap" colh colc \ ALTCOLL="$(q "SELECT collname FROM pg_collation WHERE collname IN ('en_US.utf8','en_US.UTF-8','en_US','und-x-icu') ORDER BY 1 LIMIT 1;")" if [ -z "$ALTCOLL" ] || \ [ "$(q "SELECT (min(k) COLLATE \"C\") = (SELECT min(k COLLATE \"$ALTCOLL\") FROM colh) FROM colh;" 2>/dev/null)" != "f" ]; then - echo "SKIP the collation-change demonstration: this server has no collation that" + check_skip "the collation-change demonstration" "SKIP the collation-change demonstration: this server has no collation that" "this server has no suitable collation" echo " disagrees with C on ASCII, so the arm could not fail and is not run." echo " The refusal it demonstrates is asserted above on COLLATE \"C\"." else diff --git a/test/unique_conc.sh b/test/unique_conc.sh index a09c03cb..cb539911 100755 --- a/test/unique_conc.sh +++ b/test/unique_conc.sh @@ -390,7 +390,7 @@ if [ "$CITEXT" = 1 ]; then check "2c post-fix: exactly one row equal to 'abc'" \ "$(ctl_q "SELECT count(*) FROM s_ci WHERE v = 'abc';")" "1" else - echo "SKIP 2c citext case test (citext extension not available)" + check_skip "the 2c citext case" "SKIP 2c citext case test (citext extension not available)" "citext extension not available" fi # =========================================================================== @@ -522,7 +522,7 @@ if [ "$PG_MAJOR" -ge 15 ]; then check "6 NULLS NOT DISTINCT: exactly one NULL row" \ "$(ctl_q "SELECT count(*) FROM s_nn WHERE k IS NULL;")" "1" else - echo "SKIP 6 NULLS NOT DISTINCT test (PostgreSQL < 15)" + check_skip "the NULLS NOT DISTINCT case" "SKIP 6 NULLS NOT DISTINCT test (PostgreSQL < 15)" "PostgreSQL < 15" fi # =========================================================================== From ee49fe690e9f7ab5a4914635edb33f2a809a9167 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 14:45:45 -0600 Subject: [PATCH 19/27] test: a check_skip must not read a name its own file never assigns (#917) I shipped this in 54130531. Converting the skips, I wrote check_skip "the amcheck oracle for $idx" ... into native_index_projection.sh. The loop variable is `ix`, in the OTHER branch, and `$idx` appeared nowhere else in the file. Every suite runs under `set -u`, so on a box without amcheck the else branch died: native_index_projection.sh: line 197: idx: unbound variable rc=1, before pgc_summary and before any `checks run:` line CI could not have shown it. The PGDG packages carry amcheck, so CI always takes the then branch. Found by @OffgridwithJD on a container without it, and reproduced here in isolation: unset, rc=1 and the next statement never runs; assigned, rc=0 and it does. THE FIX IS THE LOOPING ONE, not just deleting `$idx`. Deleting it leaves eleven records on a machine with amcheck and one without, and nobody comparing two logs can explain the ten-check difference from either. One list now feeds both branches, so the names the else branch declines are the names the then branch checks, and they cannot drift when an index is added to one. The else branch prints ELEVEN lines, not one. The first version kept the old single summary line and recorded eleven, and selftest 400's own sweep refused it -- correctly: a line reading `SKIP ...` that no record backs is the exact shape this change exists to remove, and a reader cannot tell it from a recorded one. The new arm is static because nothing dynamic can catch this: `bash -n` sees valid syntax, and any arm that runs only the branch CI takes sees nothing. It reads every check_skip line and requires each name it expands to be assigned in that same file, as a variable, a for-loop variable or a local, or to be a harness global. Three fixtures pin it: one unassigned name is found, a loop variable is not flagged, an assigned name is not flagged. Reintroducing the exact defect reddens it and names the file, line and variable. Scanned the whole class rather than the instance: 25 check_skip sites across 21 files, 0 reading an unassigned name, and the scan finds it when reintroduced. selftest 400 is 77 checks to 81. 350 53/53, 080 15/15. The else branch now records 11 and returns 0 under set -u. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/native_index_projection.sh | 21 ++++++- .../400-a-check-result-must-be-machine.sh | 56 +++++++++++++++++++ 2 files changed, 75 insertions(+), 2 deletions(-) diff --git a/test/native_index_projection.sh b/test/native_index_projection.sh index 29616459..4fd1a79e 100755 --- a/test/native_index_projection.sh +++ b/test/native_index_projection.sh @@ -185,16 +185,33 @@ check "parallel-built index returns each row once" \ # amcheck where available. The skip must be visible: a check that reports nothing is # indistinguishable from a check that passes, which is what this file is about. +# ONE LIST, BOTH BRANCHES. The names the else branch declines have to be the +# names the then branch checks, or two logs from two boxes differ by ten checks +# and nobody reading them can say why. It also stops the two drifting when an +# index is added to one branch and not the other. +_ixs="w_k w_k12 w_c18 w_expr w_part w_len w_par w_par2 w_pare w_parp w_ser" if psql_run "CREATE EXTENSION IF NOT EXISTS amcheck;" >/dev/null 2>&1 && [ "$(q "SELECT count(*) FROM pg_proc WHERE proname='bt_index_check'")" != "0" ]; then - for ix in w_k w_k12 w_c18 w_expr w_part w_len w_par w_par2 w_pare w_parp w_ser; do + for ix in $_ixs; do out=$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ -d "$PGC_DB" -c "SELECT bt_index_check('$ix'::regclass)" 2>&1) check "bt_index_check($ix)" \ "$(grep -qE 'ERROR' <<<"$out" && echo bad || echo ok)" "ok" done else - check_skip "the amcheck oracle for $idx" "SKIP amcheck is not installed on this build; the seq-scan oracle above still ran" "amcheck is not installed on this build" + # ELEVEN LINES, NOT ONE. The first version printed the old single summary + # line and recorded eleven, which selftest 400 refused and was right to: a + # line reading `SKIP ...` that no record backs is exactly the shape this + # branch of the change exists to remove, and a reader cannot tell it from a + # recorded one. The then branch prints eleven PASS lines; this prints eleven + # SKIP lines, and the reason travels on each. + echo " amcheck is not installed on this build; the seq-scan oracle above still ran" + for ix in $_ixs; do + check_skip "bt_index_check($ix)" \ + "SKIP bt_index_check($ix): amcheck is not installed on this build" \ + "amcheck is not installed on this build" + done fi +unset _ixs pgc_summary diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index 25165865..0927f02e 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -279,6 +279,62 @@ check "premise: while a file that calls no check is not its business" \ unset _sk_offenders _sk_f _sk_seen _sk_fix +# ---- a check_skip must not read a name its own file never assigns ---------- +# +# I SHIPPED THIS DEFECT AND THIS ARM IS WHY IT CANNOT COME BACK. Converting the +# skips, I wrote `check_skip "the amcheck oracle for $idx"` into +# native_index_projection.sh. The loop variable is `ix`, in the OTHER branch, and +# `$idx` appeared nowhere else in the file. Every suite runs under `set -u`, so on +# any box without amcheck the else branch died with "idx: unbound variable" at +# rc=1, before pgc_summary and before any `checks run:` line. CI never saw it: the +# PGDG packages carry amcheck, so CI always takes the then branch. Found by +# @OffgridwithJD on a container without it. +# +# `bash -n` cannot see this -- the syntax is fine -- and neither can any arm that +# only runs the branch CI happens to take. So the check is static and reads the +# text: every name a check_skip line expands must be assigned somewhere in that +# same file, or be one of the harness globals lib.sh exports. +_us_globals=" PGC_MAJOR PGC_SUITE PGC_DB PGC_PORT PGC_BINDIR PGC_TESTDIR PGC_WORKDIR PGC_SRCDIR PGC_ALLOW_MISSING " +_us_unbound() { # _us_unbound FILE -> lines naming a variable the file never assigns + local _f="$1" _line _v _n + grep -nE '^[[:space:]]*check_skip ' "$_f" 2>/dev/null | while IFS= read -r _line; do + _n="${_line%%:*}" + for _v in $(printf '%s' "${_line#*:}" | grep -oE '\$\{?[A-Za-z_][A-Za-z0-9_]*' | tr -d '${'); do + case "$_us_globals" in *" $_v "*) continue ;; esac + grep -qE "(^|[[:space:]]|;)$_v=" "$_f" && continue + grep -qE "for[[:space:]]+$_v[[:space:]]+in[[:space:]]" "$_f" && continue + grep -qE "local[[:space:]][^#]*\b$_v\b" "$_f" && continue + printf '%s:%s:%s\n' "${_f##*/}" "$_n" "$_v" + done + done +} + +_us_bad="" +for _us_f in "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh; do + [ -e "$_us_f" ] || continue + _us_bad="$_us_bad$(_us_unbound "$_us_f")" +done +check "no check_skip reads a name its own file never assigns" \ + "$(printf '%s' "$_us_bad" | grep -c . || true)" "0" +[ -z "$_us_bad" ] || printf ' %s\n' $_us_bad + +# The sweep must be able to find one, and must not flag a name that IS assigned. +_us_fix="$PGC_WORKDIR/unbound"; rm -rf "$_us_fix"; mkdir -p "$_us_fix" +printf 'check_skip "the oracle for $idx" "SKIP x" "y"\n' > "$_us_fix/bad.sh" +check "premise: it names a check_skip reading an unassigned variable" \ + "$(_us_unbound "$_us_fix/bad.sh" | grep -c . || true)" "1" + +printf 'for ix in a b; do\ncheck_skip "the oracle for $ix" "SKIP x" "y"\ndone\n' > "$_us_fix/good.sh" +check "and leaves one whose variable is the loop it sits in" \ + "$(_us_unbound "$_us_fix/good.sh" | grep -c . || true)" "0" + +printf 'idx=w_k\ncheck_skip "the oracle for $idx" "SKIP x" "y"\n' > "$_us_fix/assigned.sh" +check "and leaves one whose variable is assigned earlier" \ + "$(_us_unbound "$_us_fix/assigned.sh" | grep -c . || true)" "0" + +unset -f _us_unbound +unset _us_bad _us_f _us_fix _us_globals + # ---- the timing helpers reported an outcome that nothing counted ------------- # # check_timing and check_ratio_needs_quiet_machine, under PGC_SKIP_TIMING=1, From 89a0df38f9d761ec3ca2258e6c361f446146c6fb Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 14:51:24 -0600 Subject: [PATCH 20/27] test: braces, because $_v[ reads as an array expansion (#917) The arm added in ee49fe69 failed the shellcheck job, which runs `shellcheck -S error -s bash test/*.sh test/selftest/*.sh`: test/selftest/400-a-check-result-must-be-machine.sh line 305: grep -qE "for[[:space:]]+$_v[[:space:]]+in[[:space:]]" "$_f" && continue ^-- SC1087 (error): Use braces when expanding arrays `$_v[` parses as an array subscript. `${_v}[` is what was meant, and the comment now says why so the braces are not tidied away later. I did not run shellcheck before pushing, which is the whole content of local-build-hides-what-ci-fails-on: `bash -n` was clean and the arm passed, and neither of those is the gate. Reproduced in the container and re-run after the fix: rc=0 over the whole harness. The brace change is on the line that classifies a for-loop variable, so it could have broken the one fixture that depends on it. Re-checked: 400 is 81/81, the loop-variable fixture is still not flagged, and reintroducing the $idx defect still reds the arm naming file, line and variable. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/selftest/400-a-check-result-must-be-machine.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index 0927f02e..672654ab 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -302,7 +302,9 @@ _us_unbound() { # _us_unbound FILE -> lines naming a variable the file never ass for _v in $(printf '%s' "${_line#*:}" | grep -oE '\$\{?[A-Za-z_][A-Za-z0-9_]*' | tr -d '${'); do case "$_us_globals" in *" $_v "*) continue ;; esac grep -qE "(^|[[:space:]]|;)$_v=" "$_f" && continue - grep -qE "for[[:space:]]+$_v[[:space:]]+in[[:space:]]" "$_f" && continue + # BRACES, because `$_v[` reads as an array expansion: shellcheck + # SC1087 at -S error, which is the CI gate for this harness. + grep -qE "for[[:space:]]+${_v}[[:space:]]+in[[:space:]]" "$_f" && continue grep -qE "local[[:space:]][^#]*\b$_v\b" "$_f" && continue printf '%s:%s:%s\n' "${_f##*/}" "$_n" "$_v" done From 272084e55fbe80a72938cee84f5a4d457829de4c Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 14:52:57 -0600 Subject: [PATCH 21/27] test: say why the braces are there, since the arm is about exactly this (#917) @OffgridwithJD measured that `$_v[` and `${_v}[` expand identically -- bash cannot take `[` as part of a name -- so SC1087 was lint-only here and the arm worked before the fix. Worth recording, because otherwise the next reader sees braces with no reason and tidies them away. And it is the tidiest example of what the arm is for: the guard against a variable-expansion mistake carried one, in the very regex that looks for the for-loop assigning the name. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/selftest/400-a-check-result-must-be-machine.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index 672654ab..15c1789a 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -303,7 +303,13 @@ _us_unbound() { # _us_unbound FILE -> lines naming a variable the file never ass case "$_us_globals" in *" $_v "*) continue ;; esac grep -qE "(^|[[:space:]]|;)$_v=" "$_f" && continue # BRACES, because `$_v[` reads as an array expansion: shellcheck - # SC1087 at -S error, which is the CI gate for this harness. + # SC1087 at -S error, which is the CI gate for this harness. It is + # lint-only -- bash cannot take `[` as part of a name, so both + # spellings expand identically and this arm worked before the fix -- + # but it is the tidiest example of what the arm is for: the guard + # against a variable-expansion mistake carried one, in the very + # regex that looks for the for-loop assigning the name. Noticed by + # @OffgridwithJD, who also measured that the two expansions match. grep -qE "for[[:space:]]+${_v}[[:space:]]+in[[:space:]]" "$_f" && continue grep -qE "local[[:space:]][^#]*\b$_v\b" "$_f" && continue printf '%s:%s:%s\n' "${_f##*/}" "$_n" "$_v" From aa07d47c5e4ea1808207c9d65b251e979a5dca4b Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 15:39:30 -0600 Subject: [PATCH 22/27] test: drop the pytest twin that reaches into the shell harness (#917) @linuxhikerpm's blocking finding, and they are right on a point I had got wrong. I had listed test_check_results_are_machine_readable.py in CONTEXT.md's debt inventory as "arrives with PR #923". A file that has not landed is not pre-existing debt: the harness-independence rule is already on main, and this PR would have introduced a fresh violation of it. The file sources the real test/lib.sh, executes a fixture that sources it, and extracts pgc_reconcile_records from run_all_versions.sh. Under the owner's ruling -- "each harness is independent and should only parallel test functionality" -- that is the coupling rather than the twin: it agrees with the shell by construction and can never report it wrong. DELETED RATHER THAN REWRITTEN, deliberately. The remedy the review asks for is a pytest-native equivalent, and the pytest harness has no per-assertion record stream to be native to: Expect counts assertions, and user_properties carries only the unrunnable state. So "independently implement identity, verdict, reason, sanitization, SKIP accounting and count reconciliation" is a FEATURE in the pytest layer, not a port of this file. Filed separately rather than grown onto a change that is already large. The precedent is #927, where the shell part whose subject was a python module's source text was deleted rather than repaired. What this PR keeps is the shell work, which is what the shell harness owns: strict six-field RESULT validation with the emitter's own verdict list, tab, CR and newline sanitisation, 25 named SKIP sites routed through pgc_record, the derived anti-drift sweep, and the static arm for a check_skip reading a name its file never assigns. Removed the NO_CLUSTER entry, the TESTS.md section and its TOC entry, and the CONTEXT.md debt line that named a file which will no longer arrive. Checked rather than eyeballed: 21 headings against 21 TOC entries, contiguous 1..21, every anchor equal to GitHub's derivation; membership_report []; the derived job 9 files, 161 passed. selftest 350 53/53, 400 81/81, 080 15/15, shellcheck rc=0. And the property the review turns on, measured rather than asserted: of the python files this branch touches, none now reaches into shell. test_suite_accounting.py still does, but it landed on main in #922, it is named in the CONTEXT.md inventory, and this branch's seven changed lines there add zero shell references. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- CONTEXT.md | 5 +- test/pytest/TESTS.md | 99 ------ ...test_check_results_are_machine_readable.py | 300 ------------------ test/pytest/test_harness_deps.py | 2 - 4 files changed, 1 insertion(+), 405 deletions(-) delete mode 100644 test/pytest/test_check_results_are_machine_readable.py diff --git a/CONTEXT.md b/CONTEXT.md index c43f816e..a92126b3 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -209,8 +209,7 @@ this was first counted. The file is the stable unit, so each file below is named with the mechanism that makes it a reference -- which is also what has to change for it to stop being one. -**The debt this starts with, on 2026-09-10: 3 python files and 7 shell files**, -with a fourth python file arriving in PR #923. +**The debt this starts with, on 2026-09-10: 3 python files and 7 shell files.** Python that reaches into shell: @@ -222,8 +221,6 @@ Python that reaches into shell: - `test_suite_accounting.py` -- reads `run_all_versions.sh`'s text, sources the real `lib.sh` from a suite it writes, and executes the real runner. - `pgc_cluster.py` -- sources the real `test/lib.sh`. -- `test_check_results_are_machine_readable.py` -- sources `./lib.sh`. Arrives - with PR #923; not on `main` yet. Shell whose subject is python: `lib.sh`, and `selftest/030`, `040`, `350`, `360`, `370`, `380`. diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 5c5e7bc5..ddf8752f 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -67,7 +67,6 @@ behaviour, the source of that number is named. - [19. Traps this corpus records](#19-traps-this-corpus-records) - [20. test_raises_sqlstate.py: which error, and which statement](#20-test_raises_sqlstatepy-which-error-and-which-statement) - [21. test_failed_query_sentinel.py: a failed query is not a comparison](#21-test_failed_query_sentinelpy-a-failed-query-is-not-a-comparison) -- [22. test_check_results_are_machine_readable.py: one counter, one record](#22-test_check_results_are_machine_readablepy-one-counter-one-record) ## 1. How to read a test in here @@ -1613,101 +1612,3 @@ mutations leave the file the same size and Python reuses the stale bytecode. Wit `rm -rf __pycache__` between runs, mutations 3, 4 and 5 report the same failure and the table reads as though two refusals did not bite. -## 22. test_check_results_are_machine_readable.py: one counter, one record - -Check results were prose. `check`, `check_num` and `check_text` printed `PASS` or -`FAIL` and nothing else, so proving that a mutation reddened one **named** check meant -grepping text. That is how a reverted guard once reported plain green while the check -count fell from 190 to 186 -- the suite passed, and the only evidence anything had -changed was a number nobody was comparing. - -The fix is not a second emitter beside the counters. A second source of truth for how -many checks ran is the defect this family of issues exists to close, and `lib.sh` had -**eleven** places that bumped `PGC_CHECKS` -- eleven chances to add a twelfth and -forget the line beside it, which is exactly what `projections.sh`'s `expect_fail` did -with ten call sites for as long as it existed. - -So counting a check and recording it are **one operation**, `pgc_record`. A helper -cannot report an outcome without being counted, and cannot be counted without -reporting one, because no code path does either alone. `checks run: N` and the N -record lines are the same increment seen twice. - -The record is tab separated, five columns after the `RESULT` marker: - -``` -RESULT suite part name verdict reason -``` - -so a check name with spaces survives. The verdict is one of `PASS`, `FAIL`, `UNRUN` or -`SKIP`, and the reason carries the `REASON_CODE` from #915 — which is what makes this -more than a reformat: an unrunnable check is distinguishable from a passing one without -parsing prose. - -There is **no mutation column here**. That belongs to the ledger (#918), which keys on -`(suite, part, name)` and records which mutation reddened a check. A record is one -observation, not a history. - -### `test_lib_sh_counts_a_check_in_exactly_one_place` - -The structural arm, and the one that matters most. It stops the next `expect_fail` -from being written rather than catching it after a year of silent miscounting. - -### `test_each_verdict_emits_one_record_carrying_its_fields` - -PASS, FAIL and UNRUN each emit one record with the right verdict, and the name field -keeps its spaces. A reason code the enum does not hold is already a failure, and must -record the verdict it produced rather than the one it was asked for. - -### `test_every_helper_records_exactly_once` - -`check_text`, `check_num`, `check_ratio`, `pgc_pass` and `pgc_fail`, not a sample of -them. Each had its own counter bump and its own outcome line, and each was one place -the pair could come apart. - -### `test_the_human_lines_are_byte_identical` - -3,762 call sites, and suites, selftests and CI all grep `^PASS` and `^FAIL`. Adding a -record beside them is only safe if the prose did not move, so the exact strings are -pinned rather than the refactor trusted. - -### `test_the_record_count_equals_the_counter_the_summary_reports` - -One operation, so it cannot fail by drifting. It can fail if a helper is added that -prints an outcome without recording it, which is the `expect_fail` shape. - -### `test_a_skipped_timing_check_is_counted_and_recorded` - -`check_timing` and `check_ratio_needs_quiet_machine` under `PGC_SKIP_TIMING=1` printed a -human `SKIP` line and returned — **no count, no record**. Two outcomes a reader sees, -invisible to both, inside the part whose whole argument is that counting and recording -are one operation. Nothing reached those branches either: removing both emitters left -every other arm green. - -`SKIP` is now a fourth outcome, counted like the other three, so `checks run:` reports -the checks a suite **encountered** rather than the ones it managed to evaluate. - -It is deliberately **not** `check_unrunnable`. That third state exits the suite -`INCOMPLETE`, and CI sets `PGC_SKIP_TIMING` on every run — so every run would go red. A -wall-clock check deliberately not asked on a shared runner is a different thing from one -that could not be answered. - -### `test_the_accounting_line_reconciles_four_outcomes` - -Four counters against the count, which is the same shape as three against it. The -skipped term is printed even when zero: a term that disappears when empty is one a -reader cannot tell from a term that was never there. - -### `test_a_suite_that_evaluated_nothing_did_not_pass` - -Before the fourth counter, a skipped check left `PGC_CHECKS` at zero, so an all-skipped -suite hit the "ran no checks" branch **by accident**. Counting it would have made that -suite report `PASSED` with nothing behind it, so the condition now says what it always -meant: `PASSED + FAILED + UNRUN`, not `CHECKS`. - -### `test_the_runner_reconciles_records_against_the_stated_count` - -A suite's log states `checks run: N` and carries N records. Those are two artifacts of -the same run and they can genuinely disagree: a suite killed mid-way, a truncated log, -a helper that prints an outcome without recording it. A log with no count at all never -reached its summary -- a different fault from a miscount, and not a clean -reconciliation. diff --git a/test/pytest/test_check_results_are_machine_readable.py b/test/pytest/test_check_results_are_machine_readable.py deleted file mode 100644 index 21dc1de0..00000000 --- a/test/pytest/test_check_results_are_machine_readable.py +++ /dev/null @@ -1,300 +0,0 @@ -"""A check result must be machine-readable, and counted in ONE place. - -Check results were prose. `check`, `check_num` and `check_text` printed PASS or FAIL -and nothing else, so proving a mutation reddened one NAMED check meant grepping text. -That is how a reverted guard once reported plain green while the check count fell from -190 to 186: the suite passed, and the only evidence anything had changed was a number -nobody was comparing. - -The fix is not a second emitter beside the counters. A second source of truth for how -many checks ran is the defect this issue family exists to close, and `lib.sh` had -ELEVEN places that bumped `PGC_CHECKS` -- eleven chances to add a twelfth and forget -the line beside it, which is exactly what `projections.sh`'s `expect_fail` did with ten -call sites for as long as it existed. - -So counting a check and recording it are ONE operation, `pgc_record`. `checks run: N` -and the N record lines are the same increment seen twice. - -These tests drive the shell out of `lib.sh` rather than reimplementing it, for the same -reason `test_suite_accounting.py` does: a Python twin would agree with itself. -""" - -import pathlib -import subprocess - -REPO = pathlib.Path(__file__).resolve().parents[2] -LIB = REPO / "test" / "lib.sh" -RUNNER = REPO / "test" / "run_all_versions.sh" - - -def _sh(body): - """Run a snippet with lib.sh sourced, under the shell options the suites use.""" - script = f'set -uo pipefail\ncd "{LIB.parent}"\n. ./lib.sh >/dev/null 2>&1\n{body}\n' - return subprocess.run(["bash", "-c", script], capture_output=True, text=True).stdout - - -def _records(call): - return [l for l in _sh(call).splitlines() if l.startswith("RESULT\t")] - - -def _human(call): - return [l for l in _sh(call).splitlines() if not l.startswith("RESULT\t")] - - -def _extract(path, name): - out, keep = [], False - for line in path.read_text().splitlines(): - if line.startswith(f"{name}() "): - keep = True - if keep: - out.append(line) - if line == "}": - break - return "\n".join(out) - - -# ---- the structural arm ----------------------------------------------------- - - -def test_lib_sh_counts_a_check_in_exactly_one_place(expect): - """Eleven bump sites were eleven chances to add a twelfth and forget the outcome. - - This is the arm that stops the next `expect_fail` from being written, rather than - catching it after it has been silently miscounting for a year. - """ - text = LIB.read_text() - expect.num(text.count("PGC_CHECKS=$((PGC_CHECKS"), 1, - "lib.sh bumps PGC_CHECKS in exactly one place") - expect.num(_extract(LIB, "pgc_record").count("PGC_CHECKS=$((PGC_CHECKS"), 1, - "and that place is pgc_record") - - -# ---- the record line -------------------------------------------------------- - - -def test_each_verdict_emits_one_record_carrying_its_fields(tmp_path, expect): - """Tab separated -- suite, name, verdict, reason -- so a name with spaces survives. - - The reason carries the REASON_CODE, which is what makes this more than a reformat: - an unrunnable check is distinguishable from a passing one without parsing prose. - """ - for call, verdict in ((' check "a name" x x', "PASS"), - (' check "a name" x y', "FAIL")): - recs = _records(call) - expect.num(len(recs), 1, f"a {verdict} check emits exactly one record") - expect.text(recs[0].split("\t")[4], verdict, f"and its verdict field says {verdict}") - expect.text(_records(' check "a name" x x')[0].split("\t")[3], "a name", - "and the name field keeps its spaces") - - # WHICH PART asked it. The suite is not enough: harness_selftest sources - # 40-odd parts into one shell and its premises are phrased to be COPIED -- - # "premise: the pytest layer is where THIS PART thinks it is" says "this part" - # so the same sentence works in any of them. So (suite, name) is a key of - # check NAMES, not of checks, and one sharer going red would mark them all. - # Derived from BASH_SOURCE rather than from a convention. Found by - # OffgridwithJD, whose own six branches were each adding more. - # THESE TWO ARMS COULD NOT FAIL, and OffgridwithJD said so. The field is - # `${_part:-${PGC_SUITE:-unknown}}`, so deleting the derivation still yields a - # non-blank single value and both arms stayed green. "Exactly one, and not - # blank" is satisfied by the fallback. - # - # The property needs a case where the part and the suite DIFFER, which is the - # case the field exists for: harness_selftest sources 40-odd parts into one - # shell. So a fixture does the same -- an outer script that sources an inner - # one which calls the check -- and the record must name the INNER file. - outer = tmp_path / "outer_suite.sh" - inner = tmp_path / "inner_part.sh" - inner.write_text('check "a name" x x\n') - outer.write_text(f'. "{LIB}"\n. "{inner}"\n') - r = subprocess.run(["bash", "-c", f'set -uo pipefail\nbash "{outer}"'], - capture_output=True, text=True) - recs = [l for l in r.stdout.splitlines() if l.startswith("RESULT\t")] - expect.num(len(recs), 1, "premise: the sourced part emitted exactly one record") - suite, part = recs[0].split("\t")[1], recs[0].split("\t")[2] - expect.text(suite, "outer_suite", "the suite is the script that ran") - expect.text(part, "inner_part", "and the part is the file the check was asked from") - expect.text("different" if suite != part else "same", "different", - "which are not the same thing, and the fallback would make them so") - - recs = _records(' check_unrunnable "a name" MISSING_DEPENDENCY "no jq"') - expect.num(len(recs), 1, "an unrunnable check emits exactly one record") - expect.text(recs[0].split("\t")[4], "UNRUN", - "and its verdict is UNRUN, which is neither of the other two") - expect.text(recs[0].split("\t")[5], "MISSING_DEPENDENCY", - "and the REASON_CODE travels in the reason field, not in prose") - - # A reason the enum does not hold is already a FAIL. It must record the verdict it - # produced, not the one it was asked for. - expect.text(_records(' check_unrunnable "n" NOT_A_REASON "x"')[0].split("\t")[4], - "FAIL", "a bogus reason code records FAIL, not UNRUN") - - -def test_every_helper_records_exactly_once(expect): - """Not a sample. Each of these had its own counter bump and its own outcome line, - and each was one place the pair could come apart.""" - cases = { - 'check_text "n" "" "x"': "FAIL", - 'check_num "n" abc 1': "FAIL", - 'check_ratio "n" abc 1 2': "FAIL", - 'check_ratio "n" 0 1 2': "FAIL", - 'check_ratio "n" 1 1 2': "PASS", - 'pgc_pass "n"': "PASS", - 'pgc_fail "n" "d"': "FAIL", - } - for call, verdict in cases.items(): - recs = _records(" " + call) - expect.num(len(recs), 1, f"{call.split()[0]} emits exactly one record") - expect.text(recs[0].split("\t")[4], verdict, f"and records {verdict}") - - -def test_the_human_lines_are_byte_identical(expect): - """3,762 call sites, and suites, selftests and CI all grep `^PASS` and `^FAIL`. - - Adding a record beside them is only safe if the prose did not move, so the exact - strings are pinned rather than the refactor trusted. - """ - cases = { - ' check "a name" x x': "PASS a name", - ' check "a name" x y': "FAIL a name: got [x] want [y]", - ' check_unrunnable "a name" MISSING_DEPENDENCY "no jq"': - "UNRUN a name: MISSING_DEPENDENCY: no jq", - ' check_text "n" "" "x"': - "FAIL n: a side is empty, so nothing was compared: got [] want [x]", - ' check_num "n" abc 1': - "FAIL n: not a measurement, so nothing was compared: got [abc] want [1]", - } - for call, want in cases.items(): - expect.text("\n".join(_human(call)), want, f"{call.strip().split()[0]} prints its old line") - - -def test_the_record_count_equals_the_counter_the_summary_reports(expect): - """One operation, so it cannot fail by drifting -- but it CAN fail if a helper is - added that prints an outcome without recording it, which is the expect_fail shape.""" - out = _sh(' check a x x; check b x y; check_text c "" x; check_num d abc 1\n' - ' check_ratio e 1 1 2; pgc_pass f; check_unrunnable g MISSING_DEPENDENCY h\n' - ' echo "COUNTED $PGC_CHECKS"') - records = len([l for l in out.splitlines() if l.startswith("RESULT\t")]) - counted = int(next(l.split()[1] for l in out.splitlines() if l.startswith("COUNTED "))) - expect.num(records, 7, "premise: the probe ran every helper shape once") - expect.num(records, counted, "the record count equals the counter the summary reports") - - -# ---- the runner reconciles the two -------------------------------------------- - - -def test_the_runner_reconciles_records_against_the_stated_count(tmp_path, expect): - """A log states `checks run: N` and carries N records. Those are two artifacts of - the same run, and they can genuinely disagree: a suite killed mid-way, a truncated - log, a helper that prints an outcome without recording it.""" - body = _extract(RUNNER, "pgc_reconcile_records") - expect.at_least(len(body), 1, "premise: the runner defines the reconciliation") - - def run(text): - log = tmp_path / "s.log" - log.write_text(text) - script = f'set -uo pipefail\n{body}\npgc_reconcile_records "{log}"\n' - r = subprocess.run(["bash", "-c", script], capture_output=True, text=True) - return r.stdout, r.returncode - - ok = "RESULT\ts\tp\ta\tPASS\t\nRESULT\ts\tp\tb\tPASS\t\nchecks run: 2\n" - expect.num(run(ok)[1], 0, "a log whose records match its stated count reconciles") - - out, rc = run("RESULT\ts\tp\ta\tPASS\t\nchecks run: 2\n") - expect.num(rc, 1, "a log with fewer records than it claims is caught") - expect.num(out.count("records=1"), 1, "and both numbers are named, not just the verdict") - - expect.num(run("RESULT\ts\tp\ta\tPASS\t\nRESULT\ts\tp\tb\tPASS\t\n" - "RESULT\ts\tp\tc\tPASS\t\nchecks run: 2\n")[1], 1, - "a log with more records than it claims is caught too") - - # A log with no count at all never reached its summary. That is a different fault - # from a miscount and must not read as a clean reconciliation. - expect.num(run("RESULT\ts\tp\ta\tPASS\t\n")[1], 1, - "a log that never stated a count is not silently accepted") - - -# ---- the timing helpers reported an outcome that nothing counted ------------ - - -def _timing(skip, call): - """Run one timing helper with PGC_SKIP_TIMING set or clear.""" - body = (f'PGC_SKIP_TIMING={skip}\n' - 'PGC_FAIL=0; PGC_CHECKS=0; PGC_PASSED=0; PGC_FAILED=0; PGC_UNRUN=0; PGC_SKIPPED=0\n' - f'{call}\n' - 'echo "COUNTS $PGC_CHECKS/$PGC_PASSED/$PGC_SKIPPED"') - out = _sh(body) - recs = [l for l in out.splitlines() if l.startswith("RESULT\t")] - human = [l for l in out.splitlines() - if not l.startswith("RESULT\t") and not l.startswith("COUNTS ")] - counts = next((l.split()[1] for l in out.splitlines() if l.startswith("COUNTS ")), "") - return recs, human, counts - - -def test_a_skipped_timing_check_is_counted_and_recorded(expect): - """`check_timing` and `check_ratio_needs_quiet_machine` under PGC_SKIP_TIMING=1 - printed a human SKIP line and returned -- no count, no record. - - Two outcomes a reader sees, invisible to both the count and the records, in the - part whose whole argument is that those are one operation. Nothing reached those - branches either: removing both emitters left every other arm green. Found by - @linuxhikerpm. - - SKIP is a fourth outcome, counted like the other three, so `checks run:` reports - the checks a suite ENCOUNTERED rather than the ones it managed to evaluate. It is - deliberately not `check_unrunnable`: that state exits the suite INCOMPLETE, and CI - sets PGC_SKIP_TIMING on every run, so every run would go red. A wall-clock check - not asked on a shared runner is a different thing from one that could not be - answered. - """ - for call, tail in ((' check_timing "a timing check" 1 1', - "wall-clock measurement"), - (' check_ratio_needs_quiet_machine "a ratio check" 1 1 2', - "wall-clock ratio")): - name = call.split('"')[1] - recs, human, counts = _timing(1, call) - expect.num(len(recs), 1, f"{name}: skipped, emits exactly one record") - expect.text(recs[0].split("\t")[4], "SKIP", f"{name}: and its verdict is SKIP") - expect.text(counts, "1/0/1", f"{name}: and it is counted as a skip") - expect.text("\n".join(human), f"SKIP {name} (PGC_SKIP_TIMING: {tail})", - f"{name}: and its human line is unchanged") - - recs, _, counts = _timing(0, call) - expect.num(len(recs), 1, f"{name}: enabled, emits exactly one record") - expect.text(recs[0].split("\t")[4], "PASS", f"{name}: and passes") - expect.text(counts, "1/1/0", f"{name}: counted as a pass, not a skip") - - -def test_the_accounting_line_reconciles_four_outcomes(expect): - """Four counters against the count, which is the same shape as three against it. - - The skipped term is printed even when zero: a term that disappears when empty is - a term a reader cannot tell from a term that was never there. - """ - def acct(skip): - out = _sh(f'PGC_SKIP_TIMING={skip}\n' - ' check "an ordinary check" x x\n' - ' check_timing "a timing check" 1 1\n' - ' pgc_summary') - return next((l for l in out.splitlines() if l.startswith("accounting: ")), "") - - expect.text(acct(1), "accounting: 1 passed + 0 failed + 0 unrunnable + 1 skipped = 2", - "a skipped check appears in the accounting identity") - expect.text(acct(0), "accounting: 2 passed + 0 failed + 0 unrunnable + 0 skipped = 2", - "and the term is printed when zero, not omitted") - - -def test_a_suite_that_evaluated_nothing_did_not_pass(expect): - """Before the fourth counter a skipped check left PGC_CHECKS at zero, so the - all-skipped suite hit the "ran no checks" branch by accident. Counting it would - have made the suite report PASSED with nothing behind it, so the condition now - says what it always meant: PASSED + FAILED + UNRUN, not CHECKS. - """ - out = _sh('PGC_SKIP_TIMING=1\n' - ' check_timing "a timing check" 1 1\n' - ' pgc_summary') - verdicts = [l for l in out.splitlines() - if l.endswith(("PASSED", "FAILED", "INCOMPLETE")) - or l.endswith("SKIPPED (ran no checks)")] - expect.text("\n".join(v.split(": ", 1)[1] for v in verdicts), - "SKIPPED (ran no checks)", - "a suite whose every check was skipped did not pass") diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index a7ce3474..59054dde 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -86,8 +86,6 @@ # driver-dependent, 9 passed in a venv with no driver. Note for whoever pays # down the CONTEXT.md harness-independence debt -- this is one of the files on # that list, so the job now runs a file the rule says should not have this - # shape. Declaring it is still right; the fix belongs to the debt, not here. - "test_check_results_are_machine_readable.py", ] From c7334e45e0117696fd5c7307b7d45fc1651708a8 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 16:05:03 -0600 Subject: [PATCH 23/27] test: the deleted entry left six orphaned comment lines (#917) @OffgridwithJD found the fourth place. aa07d47c removed test_check_results_are_machine_readable.py from NO_CLUSTER, its TESTS.md section, its TOC entry and the CONTEXT.md debt line -- and left its six-line comment dangling before the closing bracket, still saying "this branch's own new file" about a file that is gone, and cut off mid-phrase at "should not have this". My removal walked back to the nearest `\n #` before the entry, which is the LAST line of the comment block rather than its first. Nothing executes a comment, which is why this survived every gate: the module parses, NO_CLUSTER holds the right ten entries, and no arm reads it. AND A CORRECTION TO THAT COMMIT'S CLOSING CLAIM, which was too broad. I wrote "of the python files this branch touches, none now reaches into shell". That is false: this branch touches test_suite_accounting.py, and that file does reach into shell. What I had measured was that none of the files this branch INTRODUCES does, which is the distinction @linuxhikerpm's finding turns on, and the sentence should have said so. As written it reads as "this branch is clean of shell-reaching python" and makes a tracked debt file look paid off. The counting is its own lesson. My pattern reported 5 references there, a broader one 19, and executable-only 12 -- three numbers for one file, which is the reason CONTEXT.md's own rule says to count FILES and name the mechanism. I did not follow the rule I wrote. 350 53/53, 400 81/81, the driver-free job 10 files 174 passed, membership_report []. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/pytest/test_harness_deps.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index bf0f8d46..da263f08 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -81,12 +81,6 @@ # declaration has to say so. The third time this arm has caught a merge-order # consequence rather than a mistake. "test_failed_query_sentinel.py", - # This branch's own new file, named by the same arm when main gained the - # classifier. It drives `test/lib.sh` by subprocess, so what it needs is bash - # and the tree rather than psycopg or a cluster: cluster-free, not - # driver-dependent, 9 passed in a venv with no driver. Note for whoever pays - # down the CONTEXT.md harness-independence debt -- this is one of the files on - # that list, so the job now runs a file the rule says should not have this ] From f5a908efbd4a2fc3c4ce4d08b5a999f81f66245c Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 18:49:44 -0600 Subject: [PATCH 24/27] test: four string-compared grep -c counts this branch added (#929) #941 landed the rule and its sweep while this branch was open, and the sweep reads every part -- so these four went from "the old idiom" to an offence the selftest refuses, without anyone editing them. They are MINE, not pre-existing. On main, `320-a-check-that-could-not-run.sh` holds zero string-compared `grep -c` sites; this branch added four: 254 grep -c 'pgc_summary' "$_cnt_f" 283 grep -c 'pgc_summary' "$_f" 295 grep -c 'pgc_summary' "$_f" 296 grep -c 'PGC_CHECKS=\$((PGC_CHECKS' "$_f" So this branch reintroduced #929's defect four times while #941 was closing it: `grep -c` prints NOTHING on a pattern that does not compile, and `[ "" != 0 ]` is TRUE, so each of these answers "present" for a question grep never managed to ask -- and three of the four are premise arms phrased to want present, which is the direction that turns green when the instrument breaks. All four pass exactly ONE input to grep, checked rather than assumed, which is #941's condition for the conversion being behaviour-preserving: `grep -c` over several files prints `file:count` lines, which a numeric comparison would reject where the string form tolerated it. No site here does that. #941's sweep on the composed tree, before 4 after 0 and still a live zero: one planted back 1 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- test/selftest/320-a-check-that-could-not-run.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/test/selftest/320-a-check-that-could-not-run.sh b/test/selftest/320-a-check-that-could-not-run.sh index 84c084d2..f7dc7e7b 100644 --- a/test/selftest/320-a-check-that-could-not-run.sh +++ b/test/selftest/320-a-check-that-could-not-run.sh @@ -251,7 +251,7 @@ _cnt_lib=0; _cnt_lib_bad="" for _cnt_l in "${_cnt_sites[@]}"; do _cnt_f="${_cnt_l%%:*}" _cnt_ln="$(printf '%s' "$_cnt_l" | cut -d: -f2)" - [ "$(grep -c 'pgc_summary' "$_cnt_f" || true)" != 0 ] || continue + [ "$(grep -c 'pgc_summary' "$_cnt_f" || true)" -ne 0 ] || continue _cnt_lib=$((_cnt_lib + 1)) [ "$_cnt_lib" -le 5 ] && _cnt_lib_bad="$_cnt_lib_bad ${_cnt_f##*/}:$_cnt_ln" done @@ -280,7 +280,7 @@ printf '%s\necho "checks run: 1"\n' "$_cnt_bump" > "$_cnt_fx/private.sh" _cnt_fx_old() { # the ORIGINAL rule, applied to one file local _f="$1" _l - [ "$(grep -c 'pgc_summary' "$_f" || true)" != 0 ] || { echo exempt; return; } + [ "$(grep -c 'pgc_summary' "$_f" || true)" -ne 0 ] || { echo exempt; return; } _l="$(grep -n 'PGC_CHECKS=\$((PGC_CHECKS' "$_f" | head -1 | cut -d: -f1)" [ -n "$_l" ] || { echo none; return; } if [ "$(sed -n "$((_l > 3 ? _l - 3 : 1)),$((_l + 6))p" "$_f" \ @@ -292,8 +292,8 @@ _cnt_fx_old() { # the ORIGINAL rule, applied to one file } _cnt_fx_new() { # the STRONGER rule, applied to one file local _f="$1" - [ "$(grep -c 'pgc_summary' "$_f" || true)" != 0 ] || { echo exempt; return; } - [ "$(grep -c 'PGC_CHECKS=\$((PGC_CHECKS' "$_f" || true)" != 0 ] && echo flagged || echo none + [ "$(grep -c 'pgc_summary' "$_f" || true)" -ne 0 ] || { echo exempt; return; } + [ "$(grep -c 'PGC_CHECKS=\$((PGC_CHECKS' "$_f" || true)" -ne 0 ] && echo flagged || echo none } check "premise: the fixtures carry the shapes these rules are about" \ From 54cb94c09a59dc0f0f4b7eef50307f6c5a2d5e01 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 18:51:57 -0600 Subject: [PATCH 25/27] test: the manifest and one more grep -c the base move brought under new rules (#918) Merging the updated #923 base, which now carries #939, #940 and #941, puts this branch under two rules that did not exist when it was written. Neither is a conflict git could have shown me: both are arms that read the WHOLE tree, so a file this branch never touched changed what this branch has to satisfy. THE MANIFEST, which is this PR's own arm firing correctly. Part 420 compares `parts.manifest` against the glob in both directions. #940 and #941 added parts 430 and 440 to main, so the manifest was two short and the arm would have gone red -- which is the guard working rather than a merge problem. @OffgridwithJD called this exact ordering on both of their PRs before either landed. on disk 44, listed 42, only-on-disk: 430-..., 440-... after, on disk 44, listed 44, both directions empty still live: drop 430 from the comparison and it is named again Added in sorted position rather than appended, because the file is sorted and a manifest that stops being sorted is a diff nobody can read. AND ONE MORE STRING-COMPARED `grep -c`, MINE, at `run_all_versions.sh:1382`: [ "$(grep -c '^RESULT\t' "$builddir/${s}.log" || true)" != 0 ] Zero such sites on main and zero on the #923 base, so this branch added it -- #929's defect, reintroduced while #941 was closing it. `grep -c` prints nothing on a pattern that does not compile and `[ "" != 0 ]` is TRUE, so this answers "the log has RESULT records" for a question grep never asked. One input to grep, checked, which is #941's condition for the conversion being behaviour-preserving. #941's sweep on the merged tree, before 1 after 0 still live: one planted back 1 Checked on the merged tree, not on this branch alone: #940's exit-0 and bare-exit sweep 0 the driver handed a bad pg_config rc=2, no summary ledger control / BOGUS verdict / one rc=0 / rc=2 / rc=2 --mutation against two failures driver-free pytest job 195 passed Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- test/run_all_versions.sh | 2 +- test/selftest/parts.manifest | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index c9c1fb36..8edb93e2 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -1379,7 +1379,7 @@ pgc_tally_suite() { # pgc_tally_suite NAME VERDICT LOGFILE _led_logs="" for s in "${SUITES[@]}"; do [ -s "$builddir/${s}.log" ] || continue - [ "$(grep -c '^RESULT ' "$builddir/${s}.log" || true)" != 0 ] \ + [ "$(grep -c '^RESULT ' "$builddir/${s}.log" || true)" -ne 0 ] \ && _led_logs="$_led_logs $builddir/${s}.log" done if [ -z "$_led_logs" ]; then diff --git a/test/selftest/parts.manifest b/test/selftest/parts.manifest index 7606ba99..1a5f64d9 100644 --- a/test/selftest/parts.manifest +++ b/test/selftest/parts.manifest @@ -40,3 +40,5 @@ 400-a-check-result-must-be-machine.sh 410-a-check-must-have-been-red.sh 420-a-deleted-part-must-be-visible.sh +430-the-self-test-must-not-report.sh +440-a-count-grep-never-produced.sh From d01241145df46341e398e3513457a46f9de3ef46 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 19:08:29 -0600 Subject: [PATCH 26/27] test: the ledger is the OTHER tree-wide artifact new parts invalidate (#918) CI refused this branch, and the refusal was this PR's own gate working. Every suite passed; the gate stopped the run because parts 430 and 440 -- #940 and #941, which landed on main after this ledger was generated -- contribute checks the ledger has never seen: ledger census: rows=794 | never observed red=794, ever red=0, new this run=32 PG17 has a check the ledger has never seen, which is not a pass I HALF-LEARNED THE LESSON @OffgridwithJD ALREADY GAVE ME. They called the parts.manifest ordering before either of their PRs landed, and I updated the manifest for 430 and 440 -- and did not then ask what ELSE in this PR is a tree-wide artifact that a new part invalidates. There are two, and I fixed one. The manifest lists parts; the ledger lists CHECKS, so it moves whenever any part gains or loses one, which is far more often. REGENERATED FROM A REAL RUN, not from the CI log. The failing job prints all 32 as `not in the ledger: suitepartname`, so they could have been parsed out -- but a ledger whose rows came from someone reading a log is the artifact this ledger exists to replace, and the merge path validates a log against its own `checks run:` count, which scraped text would not have. bash test/harness_selftest.sh /usr/local/pg17/bin/pg_config rc=0, checks run: 827, accounting: 827 passed + 0 failed + 0 unrunnable + 0 skipped = 827 of those, 32 are from parts 430 and 440 -- the number CI named MY FIRST RUN OF THAT WAS UNUSABLE AND I NEARLY MERGED IT. I copied the tree without `.git`, and 15 checks failed on "premise: the source tree is a git checkout: got [no]". Merging that log would have written 15 environment-induced reds into a ledger whose entire subject is which checks have ever been red -- poisoning the record with the one kind of entry it must never contain. Re-run from a real checkout: 0 failures. The merge is additive and nothing existing moved: ledger 794 -> 826 rows, delta 32 added: 18 from 430-the-self-test-must-not-report, 14 from 440-a-count-grep-never-produced rows removed or altered: 0 And the comparison that failed CI, run both ways so it can fail: checks in the run absent from the OLD ledger 32 (18/14, as CI said) checks in the run absent from the NEW ledger 0 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- test/check_ledger.tsv | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 40d57778..7f10fbaa 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -792,3 +792,35 @@ harness_selftest 420-a-deleted-part-must-be-visible premise: and a part added wi harness_selftest 420-a-deleted-part-must-be-visible premise: the manifest names something, so the comparison has two sides never - harness_selftest 420-a-deleted-part-must-be-visible the parts manifest exists, because without it a deletion is invisible never - harness_selftest 420-a-deleted-part-must-be-visible the three buckets account for every part on disk never - +harness_selftest 430-the-self-test-must-not-report a pg_config whose --bindir is empty is refused too, not run with a broken PATH never - +harness_selftest 430-the-self-test-must-not-report and it does not pretend to have run checks never - +harness_selftest 430-the-self-test-must-not-report and pairs each with the marker the runners require beside the status never - +harness_selftest 430-the-self-test-must-not-report and part 010 exits with exactly that status on its skip paths never - +harness_selftest 430-the-self-test-must-not-report and the refusal is the guard's own, naming the path it could not use never - +harness_selftest 430-the-self-test-must-not-report control: and the second, so a good pg_config is not refused never - +harness_selftest 430-the-self-test-must-not-report control: the pg_config this run was handed satisfies the first predicate never - +harness_selftest 430-the-self-test-must-not-report handed a pg_config that does not exist, the self-test refuses instead of exiting 0 never - +harness_selftest 430-the-self-test-must-not-report no selftest part exits 0, which would exit the driver before its summary never - +harness_selftest 430-the-self-test-must-not-report part 010's skip status is the one lib.sh calls PGC_EXIT_SKIPPED never - +harness_selftest 430-the-self-test-must-not-report premise: and a BARE exit too, which ends with the last status and is usually 0 never - +harness_selftest 430-the-self-test-must-not-report premise: and it answers --bindir with nothing never - +harness_selftest 430-the-self-test-must-not-report premise: and it finds a planted exit 0 outside a heredoc never - +harness_selftest 430-the-self-test-must-not-report premise: the driver this part is about is where it is expected never - +harness_selftest 430-the-self-test-must-not-report premise: the stub is executable, so an -x test alone would accept it never - +harness_selftest 430-the-self-test-must-not-report premise: the sweep reads every selftest part never - +harness_selftest 430-the-self-test-must-not-report premise: while a deliberate non-zero exit is not an offence never - +harness_selftest 430-the-self-test-must-not-report premise: while a fixture script ending in exit 0 inside a heredoc is not an offence never - +harness_selftest 440-a-count-grep-never-produced an empty count is not 'present' under a numeric comparison never - +harness_selftest 440-a-count-grep-never-produced and the numeric form says so on stderr rather than silently never - +harness_selftest 440-a-count-grep-never-produced and the string comparison it replaces WOULD have said present never - +harness_selftest 440-a-count-grep-never-produced no count from grep -c is compared as a string, which answers present when grep could not answer never - +harness_selftest 440-a-count-grep-never-produced premise: a real count compares the same both ways, so the conversion is behaviour-preserving never - +harness_selftest 440-a-count-grep-never-produced premise: and a zero count does too never - +harness_selftest 440-a-count-grep-never-produced premise: and it finds a planted string comparison on a grep -c never - +harness_selftest 440-a-count-grep-never-produced premise: and it finds one whose PATTERN contains a parenthesis, which the first version could not never - +harness_selftest 440-a-count-grep-never-produced premise: and the = 0 spelling too, which fails the same way never - +harness_selftest 440-a-count-grep-never-produced premise: grep -c prints nothing at all on a pattern that does not compile never - +harness_selftest 440-a-count-grep-never-produced premise: nor is one inside a generated fixture script never - +harness_selftest 440-a-count-grep-never-produced premise: the sweep has a corpus to read never - +harness_selftest 440-a-count-grep-never-produced premise: while a numeric comparison is not an offence never - +harness_selftest 440-a-count-grep-never-produced premise: while a valid pattern prints a number never - From f3fb7b627f7cb6fbcdc7071766fa4313ab138d97 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Thu, 10 Sep 2026 19:13:48 -0600 Subject: [PATCH 27/27] test: the census is the THIRD artifact, and this time I looked for all of them (#918) Regenerating the ledger moved a number the budget asserts, so the pytest guard went red: `the committed census matches the committed ledger: got 794 want 826`. checks_never_observed_red 794 -> 826 THIS IS THE THIRD TREE-WIDE ARTIFACT IN A ROW, and the first two I found by being told. @OffgridwithJD called the parts.manifest before their PRs landed; CI caught the ledger; this one I found by running the guard. So this time I enumerated instead of fixing what broke and pushing: every occurrence of 794 in the tree ONE, check_ledger_budget.txt:37 part 410, the shell twin DERIVES it, `sed -n 's/^checks_never...'` compared against the ledger -- no copy So there is no fourth. The shell twin cannot drift because it does not hold the number, which is the design the budget file's own comment argues for. THE CEILING IS NOT TOUCHED, and that distinction is the whole point of the file. `suites_not_covered` is monotone and may only FALL; the gate refuses an increase. `checks_never_observed_red` is a CENSUS -- a measurement that must be true, not a bound -- because every new check enters the ledger as `never`, so bounding it would deadlock. Both re-derived rather than assumed: census committed 826 vs ledger 826 agree ceiling committed 250 vs computed 250 agree, unmoved Adding checks to `harness_selftest`, which already has rows, cannot move the ceiling -- exactly as the file predicts. Proved by removal, both directions, against a green control: revert the census to 794 test_the_committed_ledger_and_budget_agree FAILS raise the ceiling to 9999 the same arm FAILS control 12 passed Both harnesses, on the merged tree: pytest driver-free job 11 files, 195 passed harness_selftest PG17 rc=0, 827 passed + 0 failed + 0 unrunnable + 0 skipped = 827, part 410's census arm PASS Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- test/check_ledger_budget.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index eef7dab8..01a8c04f 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 794 +checks_never_observed_red 826