From 72edbab4d92baa6ccfe4f146da12836424a1cd45 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 18 Sep 2026 18:32:34 +0000 Subject: [PATCH] test: two guards read the prose describing their subject as code (#1123) selftest/320 counted `MAJOR_FAIL=` over run_all_versions.sh, which carries 874 comment lines, and expected zero. MEASURED, not supposed: adding one comment saying the old spelling was MAJOR_FAIL= before #967 renamed it, and changing no code, took harness_selftest to `1080 passed + 1 failed`. The arm exists to keep that name retired, so the sentence recording the retirement is what breaks it. selftest/080 counted the bare word `grep` over harness_selftest.sh, a file of 61 comment lines about readers. It passes today only because that file happens to contain the word zero times, which is luck rather than a property. Both now strip comments into a variable and read the variable, with a herestring rather than a pipe -- the form part 080 itself requires (#486). PROVED IN FOUR ARMS, because a guard that stops flagging prose looks exactly like one that stopped working: clean tree 1081 passed + 0 failed the comment that reddened 320 1081 passed + 0 failed (was 1080 + 1) a comment mentioning `grep` 1081 passed + 0 failed a REAL reader put back in the subject 1080 passed + 1 failed <- still caught THE COUNT I COULD NOT PRODUCE WHEN I FILED #1123, produced. The first attempt extracted the surrounding `"$(grep ...` rather than the grep's own pattern and reported 54 "exposed" including obviously immune cases, so nothing was published. Parsing each `$(...)` body with shlex instead of regexing the line fixes it: 126 sweeps over shell source 34 anchored at ^, so a comment line cannot match 92 unanchored 2 unanchored AND over a commented file AND the pattern is a bare identifier Both of those two were real and both are fixed here. THE WIDER 92 IS A MEASUREMENT AND NOT A BUDGET, and no gate is added on it. `/pbt/run\.sh$` is unanchored and cannot appear in prose; deciding which patterns plausibly can is a judgment a guard should not pretend to make. CONTEXT.md carries the rule, the three decisions and the numbers. The justification in 080 also carried a stale count -- "60 lines, zero" for a file that is now 90 lines. Corrected, with the growth noted, because the exposure surface grew with it. No check names change, so no ledger row moves and the census stays at 1391. Verified: harness_selftest 1081 passed + 0 failed; docs_style.sh 47 checks, PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_012RSw4qMHS7ByE7PY8Ns4cs --- CHANGELOG.md | 45 +++++++++++++++++ CONTEXT.md | 49 +++++++++++++++++++ .../080-no-suite-pipes-a-captured-string.sh | 10 +++- .../320-a-check-that-could-not-run.sh | 10 +++- 4 files changed, 111 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 94c596e7..17cc2d00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -110,6 +110,51 @@ true until the next version shipped. NOT A REGRESSION FROM #1107: the old constant `0.5` also beat the base for this query and the planner also chose the projection. What changed is how confidently. +- Two guards read the prose describing their subject as if it were the subject + (#1123). One of them reddened on a comment. + + `selftest/320` counted `MAJOR_FAIL=` over `run_all_versions.sh`, which carries 874 + comment lines, and expected zero. MEASURED: adding one comment saying the old + spelling was `MAJOR_FAIL=` before #967 renamed it, and changing no code, took + `harness_selftest` to `1080 passed + 1 failed`. The arm exists to keep that name + retired, so the sentence that records the retirement is exactly what breaks it. + + `selftest/080` counted the bare word `grep` over `harness_selftest.sh`, a file of + 61 comment lines about readers. It passes today only because that file happens to + contain the word zero times. + + Both now strip comments into a variable and read the variable, with a herestring + rather than a pipe, which is the form part 080 itself requires (#486). + + PROVED IN FOUR ARMS, because a guard that stops flagging prose looks identical to + one that stopped working: + + clean tree 1081 passed + 0 failed + the comment that reddened 320 1081 passed + 0 failed (was 1080 + 1) + a comment mentioning `grep` 1081 passed + 0 failed + a REAL reader put back in the subject 1080 passed + 1 failed <- still caught + + THE COUNT I COULD NOT PRODUCE WHEN I FILED #1123, produced. The first attempt + extracted the surrounding `"$(grep ...` rather than the grep's own pattern and + reported 54 "exposed" including obviously immune cases, so nothing was published. + Parsing each `$(...)` body with `shlex` instead of regexing the line fixes it: + + 126 sweeps over shell source + 34 anchored at ^, so a comment line cannot match + 92 unanchored + 2 unanchored AND over a commented file AND the pattern is a bare identifier + + Both of those two were real and both are fixed here. The wider 92 is a + MEASUREMENT AND NOT A BUDGET: `/pbt/run\.sh$` is unanchored and cannot appear in + prose, and deciding which patterns plausibly can is a judgment a guard should not + pretend to make. `CONTEXT.md` carries the rule and the numbers; no gate is added + on the 92. + + The justification in 080 also carried a stale count -- "60 lines, zero" for a file + that is now 90 lines. Corrected, with the growth noted, because the exposure + surface grew with it. + + No check names change, so no ledger row moves and the census stays at 1391. - The union-merge page did not say why rebasing works where merging does not (#1116 follow-up). diff --git a/CONTEXT.md b/CONTEXT.md index d9390ec7..5c0d4a68 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -261,6 +261,55 @@ Python that reaches into shell: reported "one ledger row covers 2" -- and the right response to a name that already exists is to ask why, not to rename it. Anchor a check sweep at `^[[:space:]]*`. +- **A sweep over shell source cannot tell code from the prose describing it.** + Anchoring is only half the decision. Four guards got this wrong in one week, in + both directions, and every one was silent (#1123). + + ``` + selftest/070 flagged its own subject's comment, which quoted the pattern it greps for + grep -l pgc_setup counted six comments saying "skipped deliberately" as CALLS + the same trap as an EXCLUSION dropped those six, and with them 233 records + a record-pattern matched the word `check` inside run_all_versions.sh's echo strings + ``` + + Over-inclusion reads as a strict guard; under-inclusion reads as a clean tree. + Neither announces itself. + + **Three decisions, and the third removes the question where it applies:** + + 1. **Strip comments.** `grep -vE '^[[:space:]]*#'` into a variable, then read the + variable. A herestring, not a pipe -- part 080 forbids the pipe (#486). + 2. **Decide whether string literals count, and say which.** A pattern that + matches inside an `echo` is a different sweep from one that does not, and + stripping comments does not settle it. + 3. **Prefer a population that cannot contain prose about itself.** Sweeping the + REGISTERED SUITES rather than `test/*.sh` dropped `run_all_versions.sh` out of + scope, because the runner is not a suite. That fixed one of the four with no + pattern work at all. + + **Measured over this tree**, with each grep's own pattern extracted by `shlex` + from the `$(...)` body rather than by a regex over the line -- which is the part + that was wrong the first time and produced a number too bad to publish: + + ``` + 126 sweeps over shell source + 34 anchored at ^, so a comment line cannot match + 92 unanchored + 2 unanchored AND over a commented file AND the pattern is a bare identifier + ``` + + That last class is the one that bites, and both instances were real. In + `selftest/320` the pattern was `MAJOR_FAIL=` over a runner carrying 874 comment + lines: adding one comment saying the old name was `MAJOR_FAIL=`, changing no + code, took `harness_selftest` to `1080 passed + 1 failed`. In `selftest/080` the + pattern was the bare word `grep` over a file of 61 comment lines about readers. + Both now read code only. + + **The wider count is a measurement, not a budget.** "Unanchored" is not + "exposed": a pattern like `/pbt/run\.sh$` cannot plausibly appear in prose, and + whether one can is a judgment a guard should not pretend to make. The narrow + class above is the part worth checking by hand when a sweep is written. + - **The one permitted cross-reference, named as the rule asks.** `test_the_two_fingerprint_implementations_cover_the_same_inputs` asserts that the shell path and the Python path give the same value, which is to say that neither diff --git a/test/selftest/080-no-suite-pipes-a-captured-string.sh b/test/selftest/080-no-suite-pipes-a-captured-string.sh index 49bbfdc4..5eb8ca76 100644 --- a/test/selftest/080-no-suite-pipes-a-captured-string.sh +++ b/test/selftest/080-no-suite-pipes-a-captured-string.sh @@ -301,15 +301,21 @@ _epipe_scanned="$(printf '%s' "$_epipe_files" | grep -c . || true)" # '"'"'/harness_selftest.sh:'"'"'` entered with the rule itself (23c96c7, 2026-08-07), when # harness_selftest.sh was the monolith and held 25 occurrences of `grep` inside its # own explanation of the forbidden shape. #554 split that file into the parts in -# this directory three days later, and it has held none since: 60 lines, zero. +# this directory three days later, and it has held none since: 90 lines, zero. +# (60 when that sentence was written; the file has grown and the count with it, +# which is why the arm below now reads code rather than the whole file.) # # What remains is the DERIVED exemption -- a line inside a quoted heredoc is text, # whatever file it sits in -- which is the form the comment above already argues # for. The arm below keeps the removal honest. Put a reader back into # harness_selftest.sh and the sweep will flag it, which it should: that file runs # its pipelines like any other. +# CODE ONLY (#1123). The pattern here is the bare word `grep`, which is the single +# most likely word to appear in a comment in a file about readers, and that file +# carries 61 comment lines. Counting the prose would redden this arm for a sentence. +_hs_code="$(grep -vE '^[[:space:]]*#' "$TESTDIR/harness_selftest.sh")" check "premise: the file the old filename exclusion named holds no reader to exclude" \ - "$(grep -c 'grep' "$TESTDIR/harness_selftest.sh" || true)" "0" + "$(grep -c 'grep' <<<"$_hs_code" || true)" "0" check "and the scan examined the suites rather than finding nothing to read" \ "$([ "${_epipe_scanned:-0}" -ge 20 ] && echo yes || echo "no (scanned $_epipe_scanned)")" "yes" diff --git a/test/selftest/320-a-check-that-could-not-run.sh b/test/selftest/320-a-check-that-could-not-run.sh index f7dc7e7b..c8eb57b9 100644 --- a/test/selftest/320-a-check-that-could-not-run.sh +++ b/test/selftest/320-a-check-that-could-not-run.sh @@ -391,8 +391,16 @@ check "and a skip does not, which is the one that must stay true" \ check "the runner's INCOMPLETE branch calls the mapping rather than a local flag" \ "$(grep -c 'pgc_verdict_fails_major "\$_verdict"' "$_rv")" "1" +# CODE ONLY, NOT THE PROSE THAT DESCRIBES IT (#1123). `$_rv` carries 874 comment +# lines, and one of them writing `MAJOR_FAIL=` -- a historical note about the very +# name this arm exists to keep retired -- reddens it against a correct runner. +# MEASURED, not supposed: adding that one comment line and changing nothing else +# took harness_selftest to `1080 passed + 1 failed`. +# +# A herestring, not a pipe, because this file forbids the pipe (#486, part 080). +_rv_code="$(grep -vE '^[[:space:]]*#' "$_rv")" check "and no write-only failure flag survives in the runner" \ - "$(grep -c 'MAJOR_FAIL=' "$_rv")" "0" + "$(grep -c 'MAJOR_FAIL=' <<<"$_rv_code" || true)" "0" unset -f pgc_verdict_fails_major unset _rv _rvlog _rvrc