From 8ae8865ea2a591af12d3bc68ce92c184d6f116c6 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 14 Sep 2026 08:46:30 -0600 Subject: [PATCH 1/3] test: the piped-loop sweep needs a population, not just a detector (#1033) `selftest/400` proves its detector FIRES -- a fixture with a check inside a piped loop gives 1, at :548 -- and nothing proved it had EXAMINED anything. Without `nullglob` a wrong `$PGC_TESTDIR` leaves both globs LITERAL, awk opens no file, `grep -c .` over no input prints 0, and the arm compares that 0 against 0 and passes. Measured with the identical expression: PGC_TESTDIR= hits=1 detector fires PGC_TESTDIR=/nonexistent hits=0 ARM PASSES, nothing read THE SAME FILE ALREADY GETS THIS RIGHT 260 LINES ABOVE, where a different sweep carries `[ -e "$_sk_f" ] || continue` and a `premise: the sweep classified a corpus of check-calling files` arm. One sweep was premised and the other was not, which is why this reads as an omission rather than a missing idea. COUNTED BY WHAT AWK OPENED, not by `ls`. `FNR == 1` fires once per file awk reads, so a file that exists and cannot be read is a miss here rather than an invisible one -- and it is the same mechanism the detector itself uses, so the premise and the thing it premises cannot drift apart. RECONCILED, NOT FLOORED. The read count is compared against what the globs offered rather than against a constant, so there is nothing to maintain: a literal glob offers 2 words and reads 0, a mismatch, while the real corpus offers and reads the same 313. A second arm floors the population at 200 so a stray directory that happens to reconcile is still caught; only a mass deletion of suites approaches it. REMOVAL PROOF, pointed at a nonexistent tree: PASS no suite calls a check inside a piped loop <- the arm is BLIND FAIL premise: the piped-loop sweep read every file it was offered: got [0] want [2] FAIL premise: and that population is the suite corpus: got [no] want [yes] The arm passing on an empty population IS the defect, and it is what the premises now stop. Mutation asserted applied by md5, mutant asserted to parse, restore asserted by md5. harness_selftest.sh PASSED, 956 checks, 956 passed + 0 failed + 0 unrunnable + 0 skipped, run from a `git clone --shared` rather than a tar without `.git` -- the first attempt used tar and produced 19 false `no-repo` failures, which is the recorded trap and not a result. No pytest twin: `test/selftest/` is the BASH harness checking itself, as `test/pytest/test_harness_deps.py` and its neighbours are the pytest harness checking itself. Each harness self-checks; a cross-harness twin here would be the coupling the independence rule forbids. Closes the third of #1033's three gaps. `ae008c2` closed the other two. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- CHANGELOG.md | 25 ++++++++++++++++++ .../400-a-check-result-must-be-machine.sh | 26 +++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f9c8c061..931b2f15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,31 @@ true until the next version shipped. ### Added +- The piped-loop sweep reported a clean tree without reading one (#1033). + + `selftest/400` proves its detector FIRES -- a fixture with a check inside a piped + loop gives 1 -- and nothing proved it had EXAMINED anything. Without `nullglob` a + wrong `$PGC_TESTDIR` leaves both globs LITERAL, `awk` opens no file, `grep -c .` + over no input prints `0`, and the arm compares that `0` against `0` and passes: + + PGC_TESTDIR= hits=1 detector fires + PGC_TESTDIR=/nonexistent hits=0 ARM PASSES, nothing read + + The same file already gets this right 260 lines above, where a different sweep + carries `[ -e "$_sk_f" ] || continue` and a `premise: the sweep classified a corpus + of check-calling files` arm. One sweep was premised and the other was not. + + The population is now counted by what `awk` actually OPENED -- `FNR == 1` fires once + per file it reads -- rather than by `ls`, so a file that exists and cannot be read is + a miss rather than an invisible one, and the premise uses the same mechanism as the + detector so the two cannot drift apart. It is RECONCILED against what the globs + offered rather than floored at a constant: a literal glob offers 2 words and reads 0. + + Removal proof: pointed at a nonexistent tree, `no suite calls a check inside a piped + loop` still PASSES -- which is the defect -- while both premises redden. + + Closes the third of #1033's three gaps; `ae008c2` closed the other two. + - The grader could not read a port's own name when a `for` loop supplied it (#1045 class 2). diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index b5c32293..39f62dca 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -561,6 +561,32 @@ check "and not a piped loop with no check in it" \ check "and not a check after a one-line piped loop that already closed" \ "$(_pipeloop_sites "$_pl_fx/oneline.sh" | grep -c .)" "0" +# THE POPULATION, because the arm below can report a clean tree having read nothing. +# The detector is proven to FIRE, at :548 against a fixture. Nothing proved it had +# EXAMINED anything. Without `nullglob` a wrong `$PGC_TESTDIR` leaves both globs +# LITERAL, awk opens no file, `grep -c .` over no input prints 0, and the arm compares +# that 0 against 0 and passes. Measured with the identical expression: +# +# PGC_TESTDIR= hits=1 detector fires +# PGC_TESTDIR=/nonexistent hits=0 ARM PASSES, nothing read +# +# Counted by what awk actually OPENED -- `FNR == 1` fires once per file it reads -- and +# not by `ls`, so a file that exists and cannot be read is a miss here rather than an +# invisible one. That is also the same mechanism the detector uses, so the premise and +# the thing it premises cannot drift apart. +# +# RECONCILED against what the globs offered rather than floored at a number, so there is +# no constant to maintain: a literal glob offers 2 words and reads 0, which is a +# mismatch, while the real corpus offers and reads the same 313. +_pl_offered="$(set -- "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh; echo $#)" +_pl_read="$(awk 'FNR == 1 { n++ } END { print n+0 }' \ + "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh 2>/dev/null)" +check_num "premise: the piped-loop sweep read every file it was offered" \ + "${_pl_read:-0}" "${_pl_offered:-0}" +# And that the population is the suite corpus rather than a stray directory that +# happens to reconcile. Only a mass deletion of suites can approach this floor. +check "premise: and that population is the suite corpus" \ + "$([ "${_pl_read:-0}" -ge 200 ] && echo yes || echo no)" "yes" _pl_hits="$(_pipeloop_sites "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh 2>/dev/null | grep -c . || true)" [ "${_pl_hits:-0}" = 0 ] || _pipeloop_sites "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh | sed 's/^/ /' check "no suite calls a check inside a piped loop" "${_pl_hits:-0}" "0" From 1ef034e21cbb6f53dd811e34d6e1692853e405a7 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 14 Sep 2026 09:09:26 -0600 Subject: [PATCH 2/3] test: the two new part-400 rows claim every major they were observed on (#1033) CI's `suites` legs failed on 8ae8865 with `harness_selftest=PASS` in both. The only failing condition was the ledger gate: not in the ledger: harness_selftest 400-a-check-result-must-be-machine premise: and that population is the suite corpus (on major 17) premise: the piped-loop sweep read every file it was offered (on major 17) Two checks the committed ledger has never seen, which is what that gate is for. NOT FIXED BY WRITING THE FIELD. Setting 15;16;17;18;19 to match the 1217 uniform neighbours produces a value that is consistent, plausible and UNOBSERVED, and that field is what the gate compares against -- a hand-written row is a claim the gate can never catch being wrong. That is 3a640b0's reasoning, which I proposed the shortcut for and withdrew on; it would be worse to take it now because the red branch is mine. The stronger reason is that this answers whether the two new premises pass on 15 and 19 AT ALL. The removal proof for them was PG17 only, and a defect in the major field is invisible there by construction. So harness_selftest was run on all five majors and all five logs merged. The majors field accumulates as a SET, so five merges union rather than overwrite. Agreed with @OffgridwithJD BEFORE the runs: if the five disagreed on the (part, name) set, that is a finding about part 400 and gets reported rather than unioned away. They did not disagree. PG15 rc=0 956 checks 0 FAIL major stamped 15 PG16 rc=0 956 checks 0 FAIL 16 PG17 rc=0 956 checks 0 FAIL 17 PG18 rc=0 956 checks 0 FAIL 18 PG19 rc=0 956 checks 0 FAIL 19 merge: rows=1219 | runs=5, distinct checks this merge=956 majors: uniform, all 1219 rows carry 15;16;17;18;19 rows 1219 = sum of buckets printed 1219 ledger 1217 -> 1219, purely additive: 0 lines removed, 2 added both new rows carry all five majors, observed Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- test/check_ledger.tsv | 2 ++ .../400-a-check-result-must-be-machine.sh | 22 +++++++++++++++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index f7515224..619c23b1 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -842,10 +842,12 @@ harness_selftest 400-a-check-result-must-be-machine pgc_fail emits one record 15 harness_selftest 400-a-check-result-must-be-machine pgc_pass emits one record 15;16;17;18;19 never - harness_selftest 400-a-check-result-must-be-machine premise: and it would name a file that calls check and echoes a SKIP 15;16;17;18;19 never - harness_selftest 400-a-check-result-must-be-machine premise: and that is this fragment, not the suite 15;16;17;18;19 never - +harness_selftest 400-a-check-result-must-be-machine premise: and that population is the suite corpus 15;16;17;18;19 never - harness_selftest 400-a-check-result-must-be-machine premise: it is callable 15;16;17;18;19 never - harness_selftest 400-a-check-result-must-be-machine premise: it names a check_skip reading an unassigned variable 15;16;17;18;19 never - harness_selftest 400-a-check-result-must-be-machine premise: lib.sh is where the check helpers live 15;16;17;18;19 never - harness_selftest 400-a-check-result-must-be-machine premise: the fixtures carry the shapes this sweep is about 15;16;17;18;19 never - +harness_selftest 400-a-check-result-must-be-machine premise: the piped-loop sweep read every file it was offered 15;16;17;18;19 never - harness_selftest 400-a-check-result-must-be-machine premise: the probe ran every helper shape once 15;16;17;18;19 never - harness_selftest 400-a-check-result-must-be-machine premise: the sweep classified a corpus of check-calling files 15;16;17;18;19 never - harness_selftest 400-a-check-result-must-be-machine premise: while a file that calls no check is not its business 15;16;17;18;19 never - diff --git a/test/selftest/400-a-check-result-must-be-machine.sh b/test/selftest/400-a-check-result-must-be-machine.sh index 39f62dca..510c689a 100644 --- a/test/selftest/400-a-check-result-must-be-machine.sh +++ b/test/selftest/400-a-check-result-must-be-machine.sh @@ -564,8 +564,8 @@ check "and not a check after a one-line piped loop that already closed" \ # THE POPULATION, because the arm below can report a clean tree having read nothing. # The detector is proven to FIRE, at :548 against a fixture. Nothing proved it had # EXAMINED anything. Without `nullglob` a wrong `$PGC_TESTDIR` leaves both globs -# LITERAL, awk opens no file, `grep -c .` over no input prints 0, and the arm compares -# that 0 against 0 and passes. Measured with the identical expression: +# LITERAL, `_pipeloop_sites` opens no file, `grep -c .` over no input prints 0, and the +# arm compares that 0 against 0 and passes. Measured with the identical expression: # # PGC_TESTDIR= hits=1 detector fires # PGC_TESTDIR=/nonexistent hits=0 ARM PASSES, nothing read @@ -578,6 +578,24 @@ check "and not a check after a one-line piped loop that already closed" \ # RECONCILED against what the globs offered rather than floored at a number, so there is # no constant to maintain: a literal glob offers 2 words and reads 0, which is a # mismatch, while the real corpus offers and reads the same 313. +# +# `${_pl_read:-0}` IS LOAD-BEARING HERE, NOT BELT-AND-BRACES. On the literal-glob path +# awk exits 2 WITHOUT reaching END, so `print n+0` never runs and the substitution is +# EMPTY, not `0`. Measured: +# +# awk against an unopenable file stdout=[] rc=2 END never runs +# awk against a real EMPTY file stdout=[0] rc=0 END runs, FNR==1 does not +# +# So `:-0` is what turns that silence into the number the mismatch is computed from. +# It is the very shape #1033 is about -- a broken probe's silence becoming a value -- +# and it is safe ONLY because the reconciliation against `_pl_offered` is the next +# line. Do not remove it as redundant; without it `check_num` would refuse the empty +# string as "not a measurement", which is a correct refusal but a worse message. +# +# AND AN EMPTY FILE READS AS UNREAD, because `FNR == 1` never fires for one. A +# `touch test/new_suite.sh` before it is written gives offered=313 read=312 and reddens +# this arm with a message pointing nowhere near the cause. Zero empty `.sh` in the tree +# today, and it fails CLOSED, so it is recorded rather than worked around. _pl_offered="$(set -- "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh; echo $#)" _pl_read="$(awk 'FNR == 1 { n++ } END { print n+0 }' \ "$PGC_TESTDIR"/*.sh "$PGC_TESTDIR"/selftest/*.sh 2>/dev/null)" From 251ef6e751c3a612116b96c89845618ca0d3e099 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 14 Sep 2026 09:38:37 -0600 Subject: [PATCH 3/3] test: re-derive the ledger census after adding the two rows (#1033) @linuxhikerpm caught this: the two new `never` rows landed in `test/check_ledger.tsv` and the census in `test/check_ledger_budget.txt` was not re-derived, so `test_the_committed_ledger_and_budget_agree` got 1209 and wanted 1211. `checks_never_observed_red` is a CENSUS, not a ceiling -- the budget file says so in its own comment -- so it moves with the rows rather than being held down. RE-DERIVED BY COUNTING, not by adding 2 to 1209: awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l -> 1211 The arithmetic would have landed on the same number here, which is exactly the case where it looks safe. It is not a method: the ledger gained two rows in this branch, and any other change to a row's verdict moves this count without changing the row count at all. test_the_committed_ledger_and_budget_agree 1 passed, 7 checks test_mutation_ledger.py 30 passed, 163 checks, 0 fail `suites_not_covered` is untouched at 249: it is a CEILING and adding checks to a covered suite does not move it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- test/check_ledger_budget.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 1a00d366..c82ee2f5 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -54,4 +54,4 @@ suites_not_covered 249 # the number of attacked checks -- and it overcounts from the first moment this # ledger does the job it exists for. The gate prints both quantities side by side # (`rows=N | never observed red=M`) because they are different questions. -checks_never_observed_red 1209 +checks_never_observed_red 1211