From 188197c56e59dbbc9e40fc497e048159b772aba1 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 00:55:15 +0000 Subject: [PATCH 1/6] test/pytest: call the fingerprint module where it lives, not through lib.sh (#432) CONTEXT.md's independence rule: the two harnesses are parallel in functionality and independent in implementation. A pytest test that drives `test/lib.sh` is the first measurement wearing a Python wrapper -- it agrees with the shell by construction and can never report it wrong -- so the coupling turns a twin into a mirror, which is what the twin rule exists to avoid. `test_build_refusal.py` was the largest item on that inventory: 36 cross-harness calls behind three helpers. TWENTY-TWO ARE GONE. THEIR SUBJECT WAS NEVER lib.sh. `test/pgc_fingerprint.py` has been the one implementation since #907, and `pgc_source_fingerprint` and `pgc_source_manifest` are thin wrappers that shell out to exactly that module. The old path was python -> bash -> lib.sh -> python3 -> the module, so removing the middle two changes no subject -- it removes a wrapper, not a measurement. MEASURED BEFORE CONVERTING A SINGLE ARM, because "it should be the same value" is what this directory refuses to accept: the fingerprint is byte-identical through both paths, the manifest is identical line for line, and both agree across LC_ALL=C, C.UTF-8 and en_US.UTF-8. The hash-of-manifest relation was checked too -- md5 of the manifest text plus the newline `md5sum` reads from the pipe, first 12 hex, equal to the fingerprint and to the shell pipeline on the same tree -- before that arm was converted to compute it in Python. TWO ARMS STILL NEED A SEPARATE PROCESS, and they use the MODULE'S OWN CLI rather than lib.sh: one reads as an unprivileged user, because root ignores `chmod 000` and an in-process call cannot see a denied read, and one varies the locale. That CLI is the entry point lib.sh itself invokes, with lib.sh taken out of the path. PROVEN LIVE rather than assumed green. Both of those arms have `cannot_run` guards, so passing does not mean the body ran: breaking `_fp_cli` to return a constant reddens three arms, and the preconditions are real on this box -- `postgres` as the unprivileged user, two locales installed. THIRTEEN CALLS REMAIN AND ONLY SEVEN ARE DEBT. They drive `pgc_write_source_stamp`, `pgc_source_stamp_path`, `pgc_freshness_report` and `pgc_freshness_verdict`, which are PURE SHELL rather than wrappers over shared code -- so an arm here is a second harness testing the first, and those properties belong to the shell harness. That is the next step, not this one. Two are `test_the_two_fingerprint_implementations_cover_the_same_inputs`, which reaches across ON PURPOSE and should. Its subject is that neither side carries a private copy, which cannot be expressed from one side; the rule's own escape clause says "say which, and say why", and the module docstring and CONTEXT.md now say it. It caught four defects in one day (#907) and is what reddens on the FIRST edit if a private implementation returns rather than on the first edit that happens to diverge. Two are a historical-parity arm that embeds the previous shell algorithm as its own fixture -- rule 2, permitted -- bar one call for a directory list. Feeding it the module's `build_dirs` would make it self-contained; that is a change to a historical arm and not in this one. THE INVENTORY IS NOW A MECHANISM. It was prose: falsifiable by hand, but nothing reddened when a new reference appeared, and #923 nearly landed a fourth coupled file with a person reading being what caught it. `SHELL_REFERENCES` declares the three files and the mechanism each uses, asserted in BOTH directions -- a new file that reaches in reddens, and a file that stops reaching and stays listed reddens too, which is what stops a record of debt becoming a permanent exemption. A FILE-LEVEL GUARD, which is what CONTEXT.md asks for ("count files, not lines") and also the most it can honestly be: within a flagged file it cannot tell a path joined onto the real tree from the same name joined onto a `tmp_path`, because both are the string `lib.sh` and only the dataflow says which. The limit is stated beside the arm. THREE OF MY OWN MISTAKES, all found by running rather than reading: * I wrote `source_manifest` as a join over (relpath, digest) pairs. `manifest()` returns the joined TEXT, and my verification probe had an `isinstance` fallback that quietly stringified it and then reported the two "identical" -- so the wrong assumption read as VERIFIED until the real call raised. The delegate also returns None unchanged now, because None is the module's "a digest failed" and an `or ""` here would re-create in this file the defect the module exists to refuse. * The detector counted an f-string as TWO references, because the pieces of one are `Constant` nodes of their own. * The detector flagged THIS declaration, because the entries' descriptions named the shell files -- four files where the tree has three. A scan flagging its own test data is the third time that shape cost me a measurement in one session, so the fixtures assemble the name from fragments and the descriptions name the mechanism instead. Prove by removal: planting a cross-harness reference in `pgc_vacuity.py`, a file with none, reddens the inventory arm. Gate: the converted file 36 passed driver-free job 10 files, 183 passed, psycopg absent from the venv full corpus 264 passed with a cluster on pg18a harness_selftest 588 checks, 588 passed + 0 failed + 0 unrunnable, PASSED Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- CHANGELOG.md | 39 ++++++++ CONTEXT.md | 50 +++++++++- test/pytest/TESTS.md | 32 ++++++ test/pytest/pgc_cluster.py | 20 ++++ test/pytest/test_build_refusal.py | 158 +++++++++++++++++++++++++----- test/pytest/test_harness_deps.py | 137 ++++++++++++++++++++++++++ 6 files changed, 407 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index add322c3..149ae771 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -505,6 +505,45 @@ true until the next version shipped. the guard stops being tested that run without anyone noticing" -- four lines above the first `exit 0`. +- The pytest corpus reaches into the shell harness in three files instead of four, and + the inventory that records it is now a mechanism rather than prose (#432). + + CONTEXT.md's rule: the two harnesses are parallel in functionality and independent in + implementation. A pytest test that drives `test/lib.sh` is the first measurement + wearing a Python wrapper -- it agrees with the shell by construction and can never + report it wrong -- so the coupling turns a twin into a mirror. + + `test_build_refusal.py` was the largest item on that inventory at 36 cross-harness + calls. **22 of them are gone.** Their subject was `test/pgc_fingerprint.py`, which has + been the one implementation since #907, and `pgc_source_fingerprint` and + `pgc_source_manifest` in `lib.sh` are thin wrappers that shell out to exactly that + module -- so the path was python -> bash -> lib.sh -> python3 -> the module, and taking + the middle two out changed no subject. Measured before converting a single arm: the + fingerprint is byte-identical through both paths, the manifest identical line for line, + and both agree across `LC_ALL=C`, `C.UTF-8` and `en_US.UTF-8`. + + Two of those arms need a separate process rather than an in-process call -- one reads + as an unprivileged user, because root ignores `chmod 000`, and one varies the locale -- + and both now use the module's own CLI, which is the entry point `lib.sh` itself uses + with `lib.sh` taken out of the path. + + 13 calls remain and only 7 are debt: they drive `pgc_write_source_stamp`, + `pgc_source_stamp_path`, `pgc_freshness_report` and `pgc_freshness_verdict`, which are + PURE SHELL rather than wrappers over shared code, so those properties belong to the + shell harness. Two are the one permitted cross-reference, named as the rule asks, and + two are a historical-parity arm whose fixture is its own. + + **`SHELL_REFERENCES` in `test_harness_deps.py` makes the inventory falsifiable.** It + declares which files reach across and by what mechanism, and asserts set equality in + both directions: a new file that reaches in reddens, and a file that stops reaching and + is left in the list reddens too -- which is what stops a record of debt becoming a + permanent exemption. + + A file-level guard, which is what the rule asks for and the most it can honestly be: + within a flagged file it cannot tell a path joined onto the real tree from the same + name joined onto a `tmp_path`, because both are the string `lib.sh` and only the + dataflow says which. + - The vacuity guard's PLACEMENT is now a checked property, because a guard in a teardown cannot fail the test it guards (#432). diff --git a/CONTEXT.md b/CONTEXT.md index a92126b3..03fdee1d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -211,13 +211,53 @@ for it to stop being one. **The debt this starts with, on 2026-09-10: 3 python files and 7 shell files.** +**Progress, 2026-09-11.** The fourth never arrived: #923 deleted +`test_check_results_are_machine_readable.py` rather than land a fresh violation. And +`test_build_refusal.py` is down from 36 cross-harness calls to 13, of which 7 are the +debt proper, 2 are the permitted cross-reference below, and 2 are one historical-parity +arm. **Counted by mechanism rather than by line, because a line total moves with the +pattern:** the real executable coupling across the whole corpus was six sites in three +files -- `pgc_cluster.py` once, `test_build_refusal.py` three times, and +`test_suite_accounting.py` twice -- and the two `lib.sh` writes in +`test_build_refusal.py` are fake trees under `tmp_path`, rule 2, not references. + Python that reaches into shell: -- `test_build_refusal.py` -- sources the real `test/lib.sh` in three helpers - (`_sh`, `_sh_fp`, `_sh_fp_as`), behind 36 calls. The largest of these. Counted - with `ast`, not `grep`: the pattern `[^_a-z]_sh(` also matches `def _sh(`, which - is how the first draft said 39 -- 36 calls plus the 3 definitions. Reported by - @OffgridwithJD. Rule 3 above, caught in the very entry that states it. +- `test_build_refusal.py` -- **reduced, not cleared.** It sourced the real + `test/lib.sh` in three helpers (`_sh`, `_sh_fp`, `_sh_fp_as`) behind 36 calls and + was the largest item here. Counted with `ast`, not `grep`: the pattern + `[^_a-z]_sh(` also matches `def _sh(`, which is how the first draft said 39 -- 36 + calls plus the 3 definitions. Reported by @OffgridwithJD. Rule 3 above, caught in + the very entry that states it. + + **22 of those calls are gone.** Their subject was `test/pgc_fingerprint.py`, the ONE + implementation since #907, and `pgc_source_fingerprint` and `pgc_source_manifest` + are thin wrappers that shell out to exactly it -- so the path was + python -> bash -> lib.sh -> python3 -> the module, and removing the middle two + changed no subject. Measured before converting anything: byte-identical + fingerprint, identical manifest line for line, and agreement across LC_ALL=C, + C.UTF-8 and en_US.UTF-8. Two of them needed a separate process rather than an + in-process call -- one reads as an unprivileged user because root ignores + `chmod 000`, one varies the locale -- and both use the module's own CLI, which is + the entry point lib.sh uses with lib.sh taken out of the path. + + **13 calls remain, in three groups, and only the first is debt.** Seven drive + `pgc_write_source_stamp`, `pgc_source_stamp_path`, `pgc_freshness_report` and + `pgc_freshness_verdict`, which are PURE SHELL rather than wrappers over shared + code: those properties belong to the shell harness and moving them is the next + step. Two are `test_the_two_fingerprint_implementations_cover_the_same_inputs`, + which reaches across on purpose -- see below. The last two are a historical-parity + arm whose fixture is its own, bar one call for a directory list. + +- **The one permitted cross-reference, named as the rule asks.** + `test_the_two_fingerprint_implementations_cover_the_same_inputs` asserts that the + shell path and the Python path give the same value, which is to say that neither + side carries a private copy. That property IS the relationship, so it cannot be + expressed from one side: the rule's own escape clause -- "say which, and say why" + -- applies, and this is the saying. It caught four defects in one day (#907), and + it is what reddens on the FIRST edit if a private implementation comes back rather + than on the first edit that happens to diverge. Every other reference in this + inventory is expected to go; this one is expected to stay. - `test_suite_accounting.py` -- reads `run_all_versions.sh`'s text, sources the real `lib.sh` from a suite it writes, and executes the real runner. - `pgc_cluster.py` -- sources the real `test/lib.sh`. diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index b044b049..ed962c64 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1292,6 +1292,8 @@ fixtures are read off `conftest.py` rather than named in the classifier. | `test_the_gate_runs_the_membership_decision_rather_than_only_this_file` | selftest 350 runs the decision, and the command line it uses works | | `test_ci_derives_the_file_list_rather_than_repeating_it` | the CI job asks this module for `NO_CLUSTER`, names no file literally, and states no count | | `test_the_job_installs_no_database_driver` | the job asserts psycopg is absent rather than assuming it | +| `test_the_shell_reference_detector_sees_code_and_not_prose` | the premise: a docstring is prose, a string passed to bash is a reference, an f-string counts once | +| `test_the_harness_independence_inventory_is_exactly_what_the_corpus_does` | CONTEXT.md's inventory, asserted in both directions | **THIS IS NOW IN THE GATE.** `.github/workflows/ci.yml` runs a `pytest-guards` job: no database, no build, an interpreter and the two pinned runner packages. @@ -1317,6 +1319,36 @@ environment broken if the test died. A module that raises on import, first on th path, is the same observation and reversible by construction. Both behavioural arms assert the shim actually bites before believing anything it produces. +### The harness-independence inventory, as a mechanism + +CONTEXT.md's rule is that the two harnesses are parallel in functionality and +independent in implementation: **a pytest test that drives `test/lib.sh` is the first +measurement wearing a Python wrapper**, so it agrees with the shell by construction and +can never report it wrong. Its inventory of what still reaches across was **prose** — +falsifiable by hand, but nothing reddened when a new reference appeared. #923 nearly +landed a fourth coupled file, and what caught it was a person reading. + +`SHELL_REFERENCES` declares the three files that reach across and **why each one does**, +and the arm asserts set equality in both directions. A new file that reaches in reddens +it; a file that stops reaching and is left in the declaration reddens it too — which is +what stops the list rotting into a permanent exemption, the way every hand-maintained +exempt list in this tree has gone wrong. + +**A file-level guard, which is what the rule asks for and also the most it can honestly +be.** Within a flagged file it cannot tell a path joined onto the real tree from the +same name joined onto a `tmp_path`: both are the string `lib.sh`, and only the dataflow +says which. `test_build_refusal.py` contains both, and CONTEXT.md already records the +fake ones as rule 2 rather than references. So the assertion is over the **set of +files**, and each entry carries the mechanism a reader needs to check it by hand. + +**Two things the first version got wrong, both found by running it.** It counted an +f-string as two references, because the pieces of one are `Constant` nodes of their own. +And it flagged **this file**, because the declaration's own descriptions named the shell +files — four files where the tree has three. The descriptions now name the mechanism +without the filenames, and the detector's fixtures assemble the name from fragments. A +scan flagging its own test data is the third time that shape cost a measurement in one +session. + ## 16. test_harness_deps_classifier.py: the classifier, in the file the gate runs `test_harness_deps.py` defines the classifier that decides which files the diff --git a/test/pytest/pgc_cluster.py b/test/pytest/pgc_cluster.py index 0dc126e8..c2c2ba85 100644 --- a/test/pytest/pgc_cluster.py +++ b/test/pytest/pgc_cluster.py @@ -415,6 +415,26 @@ def source_fingerprint(srcdir): return _fp.fingerprint(srcdir) or None +def source_manifest(srcdir): + """Every file the fingerprint hashes, one "relpath digest" per line. + + THE SAME TEXT test/lib.sh's `pgc_source_manifest` prints, because both are the + same call into test/pgc_fingerprint.py. A caller comparing the two is therefore + comparing one implementation with itself rather than two that can drift. + + None, not "", when a file could not be read -- the module's contract, and + deliberate: a digest that FAILED must not look like one that succeeded. lib.sh's + wrapper substitutes a sentinel line there instead, which is the WRAPPER's + behaviour and not the module's, so a test of that belongs to the shell harness. + + I wrote this as a join over (relpath, digest) pairs first. `manifest()` returns the + joined TEXT, and my probe had an `isinstance` fallback that quietly stringified it + and then reported the two "identical" -- so the wrong assumption read as verified + until the real call raised. + """ + return _fp.manifest(srcdir) + + def build_once(srcdir, pg_config, major, lock_path=None, runner=None): """build_and_install, but at most once across xdist workers. diff --git a/test/pytest/test_build_refusal.py b/test/pytest/test_build_refusal.py index 30dec399..2976ad9e 100644 --- a/test/pytest/test_build_refusal.py +++ b/test/pytest/test_build_refusal.py @@ -18,9 +18,50 @@ Python side does with a verdict. The `bash` arms pin the SHELL PLUMBING -- the sourcing, the quoting and the exit-status path -- which an injected runner cannot reach and which is where a wrong quote would hide. + +WHAT THIS FILE REACHES INTO, AND WHAT IT NO LONGER DOES (#432). + +CONTEXT.md's independence rule: the two harnesses are parallel in functionality and +independent in implementation, and a pytest test that drives `test/lib.sh` is the +first measurement wearing a Python wrapper -- it agrees with the shell by +construction and can never report it wrong. This file was the largest item on that +inventory. + +The fingerprint and manifest arms no longer go through it. Their subject is +`test/pgc_fingerprint.py`, which has been the ONE implementation since #907; +`pgc_source_fingerprint` and `pgc_source_manifest` in lib.sh are thin wrappers that +shell out to exactly that module, so the old path was +python -> bash -> lib.sh -> python3 -> the module. They call the module now, in +process where the property allows and through the module's own CLI where it needs a +different user or locale. Measured before converting a single arm: the fingerprint is +byte-identical through both paths, the manifest is identical line for line, and both +agree across LC_ALL=C, C.UTF-8 and en_US.UTF-8. + +WHAT STILL GOES THROUGH bash, and why each one is not a wrapper to be removed: + + * `pgc_write_source_stamp`, `pgc_source_stamp_path`, `pgc_freshness_report` and + `pgc_freshness_verdict` are pure shell. They are the shell harness's OWN + implementation, not a wrapper over shared code, so an arm here is a second + harness testing the first. Those belong to the shell harness and are the next + step, not this one. + * `test_the_two_fingerprint_implementations_cover_the_same_inputs` reaches across + ON PURPOSE and is the one arm that should. Its subject is that neither side + carries a private copy, which cannot be expressed without touching both. + CONTEXT.md's rule allows exactly this -- "say which, and say why" -- so this is + the saying: it caught four defects in one day (#907), it is the guard that + reddens on the first edit if a private implementation returns, and the property + is the relationship rather than either side. + * `test_the_fix_does_not_rebaseline_stamps_already_on_disk` embeds the PREVIOUS + shell algorithm as a fixture and compares today's value against it. The fixture + is a file the test builds itself, which the rule permits, but it calls + `pgc_source_build_dirs` out of the real lib.sh for its directory list. Feeding it + the module's own `build_dirs` would make it self-contained; that is a change to + a historical-parity arm and it is not in this one. """ +import hashlib import os +import sys import tempfile import shutil import pwd @@ -31,6 +72,7 @@ import pytest from pgc_cluster import (build_and_install, build_once, make_cluster, + source_manifest, source_fingerprint) @@ -379,6 +421,70 @@ def _sh(srcdir, expr): return p.stdout.strip(), p.returncode +# ---- the fingerprint machinery, called where it LIVES --------------------------- +# +# These used to go python -> bash -> test/lib.sh -> python3 -> test/pgc_fingerprint.py. +# The last step is the ONE implementation (#907); the two before it are lib.sh's thin +# wrapper, so the round trip made every one of these arms a test of the shell harness +# as well as the module. That is the coupling CONTEXT.md's independence rule names: a +# pytest test that drives `test/lib.sh` is the first measurement wearing a Python +# wrapper, so it agrees with the shell by construction and can never report it wrong. +# +# Calling the module is not "testing a different thing". Measured on a fixture tree, +# before any arm was converted: the fingerprint is byte-identical through both paths, +# the manifest is byte-identical line for line, and both agree across LC_ALL=C, +# C.UTF-8 and en_US.UTF-8. What is gone is the wrapper, not the subject. +# +# WHAT STILL GOES THROUGH bash, and why, is listed in the module docstring: the arms +# whose subject IS lib.sh's own behaviour rather than the module's computation. + + +_FP_MODULE = pathlib.Path(__file__).resolve().parent.parent / "pgc_fingerprint.py" + + +def _fp_of(tree): + """The fingerprint of a tree, from the one implementation. + + `""` rather than None for "could not be computed", which is the shape these arms + already compared against when they read it out of the shell. + """ + return source_fingerprint(tree) or "" + + +def _mf_of(tree): + """The manifest of a tree, from the one implementation. + + Returned RAW, so None -- the module's "a digest failed" -- reaches the arm and + breaks it loudly instead of being smoothed into an empty manifest. That + distinction is the module's whole reason for returning None, and an `or ""` here + would be this file re-creating the defect the module exists to refuse. + """ + return source_manifest(tree) + + +def _fp_cli(tree, user="", env=None): + """The fingerprint, from the module's own CLI in a separate process. + + OUT OF PROCESS because two properties need it: reading as an unprivileged user + (root ignores `chmod 000`, so an in-process call cannot see a denied read), and + running under a different locale. Neither needs the shell harness -- the module + ships the CLI that `test/lib.sh` itself invokes, so this is the same entry point + lib.sh uses with lib.sh taken out of the path. + + Its contract, which is why the arms below read the same values they did through + bash: `fingerprint DIR` prints the hash and exits 0, and prints EMPTY when a digest + could not be read. lib.sh's wrapper passes that through unchanged. + """ + argv = [sys.executable, str(_FP_MODULE), "fingerprint", str(tree)] + if user: + argv = ["runuser", "-u", user, "--"] + argv + e = dict(os.environ) + if env: + e.update(env) + p = subprocess.run(argv, capture_output=True, text=True, env=e) + return p.stdout.strip(), p.returncode + + def _tree_with_module(tmp_path, name): t = tmp_path / name (t / "src").mkdir(parents=True, exist_ok=True) @@ -419,11 +525,11 @@ def test_two_installations_of_one_major_do_not_share_a_stamp(tmp_path, expect): def test_moving_bytes_between_files_moves_the_shell_fingerprint(tmp_path, expect): """`xargs -0 cat | md5sum` could not see a repartition.""" t = _tree_with_module(tmp_path, "rp") - before, _ = _sh(t, f'pgc_source_fingerprint "{t}"') + before = _fp_of(t) expect.at_least(len(before), 12, "premise: the tree fingerprints at all") (t / "src" / "a.c").write_text("int a;\nint b;\n") (t / "objstore" / "b.c").write_text("") - after, _ = _sh(t, f'pgc_source_fingerprint "{t}"') + after = _fp_of(t) expect.text(str(after != before), "True", "moving bytes between files moves the fingerprint") @@ -578,24 +684,24 @@ def test_a_failed_digest_yields_no_fingerprint_rather_than_a_wrong_one(expect): return root, t = _readable_tree("t") try: - base, _ = _sh_fp_as(user, f'pgc_source_fingerprint "{t}"') + base, _ = _fp_cli(t, user=user) expect.at_least(len(base), 12, "premise: the tree fingerprints at all") # Premise for the mechanism itself: the unprivileged reader must agree # with a privileged one while nothing is denied, or the arm below would # be measuring the user switch rather than the failure. - mine, _ = _sh_fp_as("", f'pgc_source_fingerprint "{t}"') + mine, _ = _fp_cli(t) expect.text(base, mine, "premise: the unprivileged read agrees while readable") for name in ("b.c", "c.c"): f = t / "src" / name f.chmod(0o000) - got, _ = _sh_fp_as(user, f'pgc_source_fingerprint "{t}"') + got, _ = _fp_cli(t, user=user) f.chmod(0o644) expect.text(got or "empty", "empty", f"an unreadable {name} yields no fingerprint, not a wrong one") - after, _ = _sh_fp_as(user, f'pgc_source_fingerprint "{t}"') + after, _ = _fp_cli(t, user=user) expect.text(after, base, "control: and the tree fingerprints again once readable") finally: shutil.rmtree(root, ignore_errors=True) @@ -615,7 +721,7 @@ def test_a_failed_digest_gives_unverified_and_never_a_false_stale(expect): return root, t = _readable_tree("v") try: - base, _ = _sh_fp_as(user, f'pgc_source_fingerprint "{t}"') + base, _ = _fp_cli(t, user=user) expect.at_least(len(base), 12, "premise: the tree fingerprints at all") (t / "src" / "b.c").chmod(0o000) verdict, _ = _sh_fp_as( @@ -634,7 +740,7 @@ def test_one_tree_hashes_one_way_however_the_path_is_spelled(tmp_path, expect): t = _fp_tree(tmp_path, "s") link = tmp_path / "s_link" link.symlink_to(t) - plain, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') + plain = _fp_of(t) expect.at_least(len(plain), 12, "premise: the fixture fingerprints at all") for label, spelling in ( ("a trailing slash", f"{t}/"), @@ -642,9 +748,14 @@ def test_one_tree_hashes_one_way_however_the_path_is_spelled(tmp_path, expect): ("a /src/.. segment", f"{t}/src/.."), ("a symlink", str(link)), ): - got, _ = _sh_fp(f'pgc_source_fingerprint "{spelling}"') + got = _fp_of(spelling) expect.text(got, plain, f"{label} hashes the same tree the same way") - rel, _ = _sh_fp(f'cd "{t}" && pgc_source_fingerprint .') + _cwd = os.getcwd() + try: + os.chdir(t) + rel = _fp_of(".") + finally: + os.chdir(_cwd) expect.text(rel, plain, "a relative path hashes the same tree the same way") @@ -682,12 +793,12 @@ def test_the_fingerprint_still_moves_on_a_real_change(tmp_path, expect): its input, so the set needs one arm proving the hash still moves. """ t = _fp_tree(tmp_path, "m") - before, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') + before = _fp_of(t) (t / "src" / "a.c").write_text("int a = 2;\n") - after, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') + after = _fp_of(t) expect.text(str(after != before), "True", "a real content change moves the fingerprint") (t / "src" / "a.c").write_text("int a;\n") - restored, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') + restored = _fp_of(t) expect.text(restored, before, "and restoring the content restores it") @@ -706,12 +817,12 @@ def test_a_tree_with_nothing_hashable_reports_no_fingerprint(tmp_path, expect): anything. """ live = _fp_tree(tmp_path, "hollow_premise") - base, _ = _sh_fp(f'pgc_source_fingerprint "{live}"') + base = _fp_of(live) expect.at_least(len(base), 12, "premise: the same helper returns a fingerprint for a real tree") empty = tmp_path / "hollow" empty.mkdir() - got, _ = _sh_fp(f'pgc_source_fingerprint "{empty}"') + got = _fp_of(empty) expect.text(got or "empty", "empty", "an unhashable tree yields no fingerprint") @@ -738,7 +849,7 @@ def test_the_manifest_names_what_the_fingerprint_hashed(tmp_path, expect): answer. """ t = _mf_tree(tmp_path, "mf") - out, _ = _sh_fp(f'pgc_source_manifest "{t}"') + out = _mf_of(t) lines = [l for l in out.splitlines() if l.strip()] expect.num(len(lines), 6, "the manifest names every file the fingerprint hashes") shaped = [l for l in lines if re.fullmatch(r"[A-Za-z0-9_./-]+ [0-9a-f]{32}", l)] @@ -750,8 +861,8 @@ def test_the_manifest_names_what_the_fingerprint_hashed(tmp_path, expect): def test_the_fingerprint_is_the_hash_of_the_manifest(tmp_path, expect): """One is defined as the other, so the two cannot drift apart.""" t = _mf_tree(tmp_path, "mh") - fp, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') - via, _ = _sh_fp(f'pgc_source_manifest "{t}" | md5sum | cut -c1-12') + fp = _fp_of(t) + via = hashlib.md5((_mf_of(t) + "\n").encode()).hexdigest()[:12] expect.text(fp, via, "the fingerprint is the hash of the manifest") @@ -764,9 +875,9 @@ def test_an_added_file_is_named_rather_than_merely_changing_the_hash(tmp_path, e same line. """ t = _mf_tree(tmp_path, "add") - before, _ = _sh_fp(f'pgc_source_manifest "{t}"') + before = _mf_of(t) (t / "src" / "zz_appeared.c").write_text("int zz;\n") - after, _ = _sh_fp(f'pgc_source_manifest "{t}"') + after = _mf_of(t) gained = set(after.splitlines()) - set(before.splitlines()) expect.num(len(gained), 1, "exactly one manifest line appears") expect.text(sorted(gained)[0].split()[0], "src/zz_appeared.c", @@ -906,8 +1017,7 @@ def test_one_tree_hashes_one_way_however_the_locale_is_set(expect): seen = {} for loc in wanted: - got, _ = _sh_fp(f'pgc_source_fingerprint "{t}"', - env={"LC_ALL": loc, "LANG": loc}) + got, _ = _fp_cli(t, env={"LC_ALL": loc, "LANG": loc}) seen[loc] = got expect.at_least(len(seen[wanted[0]]), 12, "premise: the tree fingerprints at all") @@ -948,7 +1058,7 @@ def test_a_symlinked_src_is_skipped_like_any_other_symlinked_build_dir(tmp_path, expect.num(len(list(real.iterdir())), 2, "premise: and the target holds sources find would otherwise hash") - out, _ = _sh_fp(f'pgc_source_manifest "{t}"') + out = _mf_of(t) lines = [l for l in out.splitlines() if l.strip()] expect.num(len([l for l in lines if l.startswith("src/")]), 0, "a symlinked src contributes nothing, as find -P contributes nothing") @@ -960,6 +1070,6 @@ def test_a_symlinked_src_is_skipped_like_any_other_symlinked_build_dir(tmp_path, (real_tree / "src" / "a.c").write_text("int a;\n") (real_tree / "Makefile").write_text("all:\n\ttrue\n") (real_tree / "pgcolumnar.control").write_text("x\n") - out2, _ = _sh_fp(f'pgc_source_manifest "{real_tree}"') + out2 = _mf_of(real_tree) expect.num(len([l for l in out2.splitlines() if l.startswith("src/a.c ")]), 1, "control: a real src directory is still hashed") diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index 110aa9b8..a0feec05 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -871,3 +871,140 @@ def test_the_job_installs_no_database_driver(expect): if __name__ == "__main__": sys.exit(_main(sys.argv[1:])) + +# ---- the harness-independence inventory, as a mechanism (#432) ------------------ +# +# CONTEXT.md: the two harnesses are parallel in functionality and independent in +# implementation, and they must not call, import or reference each other. Its inventory +# of what still does was PROSE -- falsifiable by hand, but nothing reddened when a new +# reference appeared. #923 nearly landed a fourth coupled file, and the review that +# caught it was a person reading, not an arm. +# +# So the inventory is declared here and asserted in BOTH directions. A new file that +# reaches into the shell harness reddens this; a file that stops reaching and is not +# removed from the declaration reddens it too, which is what stops the list rotting +# into a permanent exemption. +# +# A FILE-LEVEL GUARD, which is what CONTEXT.md's rule asks for -- "count files, not +# lines" -- and also the most this can honestly be. Within a flagged file it cannot tell +# a path joined onto the REAL tree from the same name joined onto a `tmp_path`: both are +# the string "lib.sh" and only the dataflow says which. test_build_refusal.py contains +# both, and CONTEXT.md already records that the fake ones are rule 2 rather than +# references. So the assertion is over the SET OF FILES, and the per-file entry carries +# the mechanism a reader needs to check it by hand. +# +# THE COUNTING RULE IS CONTEXT.md'S, and the three exclusions are the ones it names, +# in the order it says they get confused: prose is not a reference, a file the test +# BUILDS ITSELF under tmp_path is not a reference even with the same name, and a word +# that merely looks like a filename is not one. Docstrings are dropped by AST position +# rather than by pattern, which is the only way to tell the first from a string the +# code actually passes to bash. + +# THE DESCRIPTIONS NAME NO FILE, and that is not squeamishness: the first version +# spelled the helper library's path in them, and the detector flagged THIS file for its +# own inventory -- four files where the tree has three. The mechanism is what the entry +# has to carry; the filenames are in CONTEXT.md, which is prose and may name anything. +SHELL_REFERENCES = { + "pgc_cluster.py": + "sources the shell harness's helper library to call its build-and-install " + "function, so the build refusal has one implementation rather than two", + "test_build_refusal.py": + "sources that library for the pure-shell stamp and freshness helpers, and " + "for the one permitted cross-implementation arm; see its module docstring", + "test_suite_accounting.py": + "reads the matrix runner's text and executes the real runner", +} + +_SHELL_NAMES = re.compile( + r"(^|/)(lib\.sh|run_all_versions\.sh|portlib\.sh|harness_selftest\.sh)" + r"|(^|/)test/[A-Za-z0-9_]+\.sh" + r"|(^|/)selftest/") + + +def _docstring_nodes(tree): + """Every Constant that is a docstring, by position rather than by content.""" + out = set() + for node in ast.walk(tree): + if isinstance(node, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, + ast.ClassDef)): + body = getattr(node, "body", None) + if body and isinstance(body[0], ast.Expr) \ + and isinstance(body[0].value, ast.Constant) \ + and isinstance(body[0].value.value, str): + out.add(id(body[0].value)) + return out + + +def _shell_reference_sites(source): + """-> [(lineno, text)] for executable strings naming a shell file in this tree. + + An f-string's PIECES are Constants of their own, so counting both the JoinedStr and + its parts reported one reference as two. Measured: the fixture below asserted 1 and + the first version answered 2. + """ + tree = ast.parse(source) + skip = _docstring_nodes(tree) + for node in ast.walk(tree): + if isinstance(node, ast.JoinedStr): + for piece in node.values: + if isinstance(piece, ast.Constant): + skip.add(id(piece)) + hits = [] + for node in ast.walk(tree): + if isinstance(node, ast.Constant) and isinstance(node.value, str): + if id(node) in skip: + continue + if _SHELL_NAMES.search(node.value): + hits.append((node.lineno, node.value[:60])) + elif isinstance(node, ast.JoinedStr): + txt = "".join(v.value for v in node.values + if isinstance(v, ast.Constant) and isinstance(v.value, str)) + if _SHELL_NAMES.search(txt): + hits.append((node.lineno, txt[:60])) + return sorted(set(hits)) + + +# THE FIXTURES ARE ASSEMBLED, never written out, and the first version of this arm got +# that wrong: spelling the shell file's name in its own test data made THIS FILE a +# referencing file, and the inventory arm below reported four files where the tree has +# three. A scan flagging its own fixtures is the third time that shape has cost me a +# measurement today, so the name is built from fragments no single string contains. +_LIB = "test/" + "lib" + ".sh" + + +def test_the_shell_reference_detector_sees_code_and_not_prose(expect): + """The premise, and the distinction the whole arm below rests on.""" + expect.num(len(_shell_reference_sites( + 'def f():\n """This drives ' + _LIB + ' and says so."""\n return 1\n')), 0, + "a docstring naming a shell file is prose, not a reference") + expect.num(len(_shell_reference_sites( + 'import subprocess\nsubprocess.run(["bash", "-c", ". ' + _LIB + '"])\n')), 1, + "a string the code passes to bash is a reference") + expect.num(len(_shell_reference_sites( + 'srcdir = "/x"\nscript = f\'. "{srcdir}/' + _LIB + '" || exit 1\'\n')), 1, + "and so is one built with an f-string, counted ONCE rather than per piece") + expect.num(len(_shell_reference_sites('sharedir = "/usr/share"\n')), 0, + "a word that merely looks like a path is not a reference") + + +def test_the_harness_independence_inventory_is_exactly_what_the_corpus_does(expect): + """CONTEXT.md's inventory, asserted in both directions. + + A file that starts reaching into the shell harness reddens this. A file that stops + and is left in the declaration reddens it too -- otherwise the list becomes a + permanent exemption that outlives the coupling it was written for, which is how + every hand-maintained exempt list in this tree has gone wrong. + """ + here = pathlib.Path(__file__).parent + files = sorted(here.glob("*.py")) + expect.at_least(len(files), 10, "premise: the scan has a corpus to read") + found = {} + for f in files: + sites = _shell_reference_sites(f.read_text(encoding="utf-8")) + if sites: + found[f.name] = sites + expect.text(repr(sorted(found)), repr(sorted(SHELL_REFERENCES)), + "the files that reach into the shell harness are exactly the declared ones") + for name in sorted(found): + expect.at_least(len(SHELL_REFERENCES.get(name, "")), 20, + f"{name}'s entry says by what mechanism it reaches across") From 6537c268fd90082552cb38abba7f9521a5de4692 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 01:15:52 +0000 Subject: [PATCH 2/6] test: move the plan_marker order check into the pytest corpus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The shell part `test/selftest/370` pinned the position of `plan_marker`'s empty-plan refusal by reading the Python source as text. A shell part can pin text; it cannot run the function, so it could not tell a refusal that still fires from one that has been stranded. That is the cross-harness dependency the two-harness rule removes. What changed: - `test_guards_pinned.py` gains `test_the_empty_plan_refusal_precedes_the_arms_it_protects`. It takes the class off the `expect` fixture, so the file still imports nothing from the harness. - `test/selftest/370-the-plan-marker-guard-must.sh` is deleted. Its other three properties were already covered by arms in `test_guards_pinned.py`; this was confirmed by running them, not by reading them. - `TESTS.md` names the new arm and records what it is worth. - `CONTEXT.md`'s count of shell files that reach across the boundary goes from seven to three. The other four were rule-3 lookalikes: `lib.sh`, `selftest/030` and `selftest/040` matched only the shell functions `pgc_cluster_datadir` and `pgc_cluster_is_ours`, both defined in `lib.sh`. What the new arm is and is not: It discriminates — move the refusal to the end of `plan_marker` and it fails; leave it where it is and it passes. It is not load-bearing today: with the refusal at the end, `plan_marker([], absent=True)` still refuses, because `plan_marker` has no early return. So 370's stated reason, that the absent arm returns a pass first, describes a shape the function does not have. The arm is prospective insurance against a refactor that adds an early return, and it is labelled that way rather than sold as a live hole. Verified on pg18a: selftest 608 checks, 0 FAIL; the driver-free subset 184 passed; the full corpus 269 passed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- CONTEXT.md | 21 ++++- test/pytest/TESTS.md | 18 ++++ test/pytest/test_guards_pinned.py | 44 +++++++++ .../370-the-plan-marker-guard-must.sh | 90 ------------------- 4 files changed, 81 insertions(+), 92 deletions(-) delete mode 100644 test/selftest/370-the-plan-marker-guard-must.sh diff --git a/CONTEXT.md b/CONTEXT.md index 03fdee1d..1c33aa8e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -262,8 +262,25 @@ Python that reaches into shell: real `lib.sh` from a suite it writes, and executes the real runner. - `pgc_cluster.py` -- sources the real `test/lib.sh`. -Shell whose subject is python: `lib.sh`, and `selftest/030`, `040`, `350`, `360`, -`370`, `380`. +Shell whose subject is python: `selftest/350`, `360` and `380`. **Three, not the +seven this line first named, and the three it dropped were rule 3 all along.** +`lib.sh`, `selftest/030` and `selftest/040` reference NO path under `test/pytest`: +their only matches were the shell functions `pgc_cluster_datadir` and +`pgc_cluster_is_ours`, both defined in `lib.sh` itself -- "a word that merely looks +like a filename", caught for the second time in the entry that states the rule. +`selftest/370` was the fourth and is **deleted**: every property it pinned was a text +pin on `pgc_vacuity.py`, and #927 is the precedent -- a shell arm asserting a text pin +cannot prove a python arm is caught. Its one property that the pytest corpus did not +already assert behaviourally moved to `test_guards_pinned.py`, where python reads its +own module rather than the other harness's. + +**And that property is not behaviourally observable today, which is why it moved as a +SOURCE check and says so.** Measured: with `plan_marker`'s empty-plan refusal moved to +the very end of the function, `plan_marker([], absent=True)` still refuses -- +`plan_marker` has no early return, so the refusal fires wherever it sits. 370's stated +reason, that "the absent arm returns a pass first", describes a shape the function does +not have. The arm is prospective insurance against a refactor that adds an early return, +and it is labelled as that rather than as a live guard. **`test_build_refusal.py` is the example worth studying, because it does both.** It writes a fake `test/lib.sh` into a `tmp_path` and drives that -- rule 2, not a diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index ed962c64..70060d0c 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -324,6 +324,7 @@ as the failure. | `test_plan_marker_absent_arm_passes_on_a_plan_that_lacks_the_key` | **control** | | `test_plan_marker_refuses_an_absence_claim_over_an_empty_plan` | the hole under both arms | | `test_refusal_itself_refuses_an_empty_pattern_list` | the new helper must not become the defect it removes | +| `test_the_empty_plan_refusal_precedes_the_arms_it_protects` | the refusal's **position**: no arm may answer ahead of it | ### plan_marker, and the three ways it could not fail @@ -344,6 +345,23 @@ legitimately lacks the node. That is now a `VacuityError`, and it is refused for the present arm too — an empty plan means the `EXPLAIN` did not arrive, so neither question can be answered. +**The empty-plan refusal is pinned by position, not only by behaviour.** A guard +that sits after the code it protects is a guard that never runs. So one arm reads +`plan_marker`'s own source and asserts the empty-plan refusal comes before both +arms. Two measurements say what that arm is worth today: + +- Move the refusal to the end of the function and the arm fails; leave it where it + is and it passes. It discriminates. +- With the refusal moved to the end, `plan_marker([], absent=True)` **still + refuses**, because `plan_marker` has no early return for the absent arm. So the + order is not load-bearing right now. + +It is therefore prospective insurance: the day someone adds an early return, the +refusal stops being reachable and this arm is the only thing that says so. The +check was a shell part (`test/selftest/370`) until the two harnesses were +separated; a shell part can pin the text of a Python function but cannot run it, +so the arm moved here and 370 was deleted. + The four arm tests are behavioural rather than refusals, because `plan_marker`'s two arms raise `AssertionError`: `expect.refusal` does not apply and `expect.outcomes` is the right instrument. Their value is not their own green, diff --git a/test/pytest/test_guards_pinned.py b/test/pytest/test_guards_pinned.py index e69655d6..4840670f 100644 --- a/test/pytest/test_guards_pinned.py +++ b/test/pytest/test_guards_pinned.py @@ -292,3 +292,47 @@ def test_empty_directions(expect): expect.ordering_observable([], [], "no rows either way") '''), "ordering_observable names its own both-empty refusal", "both directions are empty") + +# ---- plan_marker's refusal must stay ahead of the arms it protects -------------- +# +# MOVED HERE FROM test/selftest/370 (#432). That part read `pgc_vacuity.py`'s TEXT from +# the shell harness, which is the coupling CONTEXT.md's independence rule forbids: a +# shell arm asserting a text pin cannot prove a python arm is caught, which is why #927 +# deleted the part whose subject was a python module's source rather than repairing it. +# Python reading its OWN module is not a cross-harness reference -- it is one harness +# checking itself -- so the property keeps its home and loses the boundary crossing. +# +# IT IS A SOURCE CHECK BECAUSE THE ORDERING IS NOT OBSERVABLE TODAY, and that is worth +# stating rather than implying. Measured: with the refusal moved to the very END of +# `plan_marker`, `plan_marker([], absent=True)` STILL refuses. `plan_marker` has no early +# return -- the passing case falls off the end -- so the refusal fires wherever it sits, +# and no input distinguishes the orders. +# +# 370's stated reason was that "an empty plan leaves found=False and the absent arm +# returns a pass first". The absent arm does not return; it falls through. So the defect +# that comment describes cannot arise in this shape, and the behavioural arm above +# (`test_plan_marker_refuses_an_absence_claim_over_an_empty_plan`) does not redden when +# the order changes -- I checked, expecting it to, and it did not. +# +# The arm is therefore PROSPECTIVE: it is insurance against a refactor that adds an early +# return, after which the order would decide whether an empty plan passes. That is a real +# risk and cheap to pin, but it is a claim about source order and it says so. + + +def test_the_empty_plan_refusal_precedes_the_arms_it_protects(expect): + """The refusal must sit before the arms, so a future early return cannot skip it.""" + import inspect + + # THE CLASS COMES OFF THE FIXTURE, so this file still imports nothing: the + # `expect` object IS an instance of the layer being read. + body = inspect.getsource(type(expect).plan_marker) + lines = body.split("\n") + i_refusal = next((n for n, l in enumerate(lines) if "if not nodes:" in l), None) + i_absent = next((n for n, l in enumerate(lines) if "if absent and found:" in l), None) + i_present = next((n for n, l in enumerate(lines) + if "if not absent and not found:" in l), None) + expect.text(f"{i_refusal is not None} {i_absent is not None} {i_present is not None}", + "True True True", + "premise: all three were found, so the ordering can mean something") + expect.num(int(i_refusal < i_absent and i_refusal < i_present), 1, + "the empty-plan refusal precedes both arms it protects") diff --git a/test/selftest/370-the-plan-marker-guard-must.sh b/test/selftest/370-the-plan-marker-guard-must.sh deleted file mode 100644 index 072dd8b4..00000000 --- a/test/selftest/370-the-plan-marker-guard-must.sh +++ /dev/null @@ -1,90 +0,0 @@ -# ---- plan_marker must have all three of its guards --------------------------- -# -# WHY THIS EXISTS. @jdatcmd's #897 review named `plan_marker` as the guard to fix -# first: "both of its arms can be deleted independently with the suite green. -# Under one of those mutations the premise can never fail, so the provider-trap -# test would silently be about an ordinary plan." -# -# He was right, and it is the worst place in the layer for it to be true. -# plan_marker is the faithful port of pgc_is_columnar_scan; test_connection.py -# calls it three times, once as the PREMISE that the vectorized aggregate -# engaged. A premise that cannot fail turns its test into a test about an -# ordinary plan, and the test stays green while it happens. -# -# A THIRD HOLE SAT UNDER BOTH ARMS. An absence claim is satisfied by nothing -# being there at all: `plan_marker([], key, absent=True)` gave `1 passed`, exit 0, -# because a plan that never arrived looks exactly like a plan that legitimately -# lacks the node. That is now a refusal. -# -# WHY THE CHECKS ARE STATIC, same division as selftest 360. The behaviour is -# pinned by five arms in test/pytest/test_guards_pinned.py that run pytest inside -# pytest; those need pytest, psycopg and a virtualenv, and CI installs none of -# them. This half asserts the STRUCTURE they rest on, greppable from a checkout -# with nothing installed. - -_pm_vac="$PGC_TESTDIR/pytest/pgc_vacuity.py" - -check "premise: the pytest layer is where this part thinks it is" \ - "$([ -f "$_pm_vac" ] && echo yes || echo no)" "yes" - -# The function body, cut out once so every arm below reads the same text. A -# grep over the WHOLE file would match these shapes wherever they occur and -# report a guard present that lives in another method. -_pm_body="$(awk '/^ def plan_marker\(/{f=1} f&&/^ def /&&!/plan_marker/{exit} f' "$_pm_vac")" - -check "premise: plan_marker's body was actually cut out of the file" \ - "$([ "$(printf '%s\n' "$_pm_body" | wc -l)" -ge 20 ] && echo yes || echo "too short")" "yes" - -# The present arm: asked "does this plan carry the marker", a plan that does not -# must fail. -check "plan_marker keeps the arm that fails when the key is absent" \ - "$(printf '%s\n' "$_pm_body" | grep -c 'if not absent and not found:')" "1" - -# The absent arm: `absent=True` is how a test pins that a plan is NOT a scan. -check "plan_marker keeps the arm that fails when the key is present" \ - "$(printf '%s\n' "$_pm_body" | grep -c 'if absent and found:')" "1" - -# And the refusal under both of them. -check "plan_marker refuses a plan with no nodes at all" \ - "$(printf '%s\n' "$_pm_body" | grep -c 'if not nodes:')" "1" - -check "and that refusal is a VacuityError, not an ordinary assertion" \ - "$(printf '%s\n' "$_pm_body" | grep -A2 'if not nodes:' | grep -c 'raise VacuityError')" "1" - -# The refusal has to come BEFORE the walk that sets `found`, or it is dead code: -# an empty plan leaves found=False and the absent arm returns a pass first. -_pm_ln_empty="$(printf '%s\n' "$_pm_body" | grep -n 'if not nodes:' | cut -d: -f1)" -_pm_ln_absent="$(printf '%s\n' "$_pm_body" | grep -n 'if absent and found:' | cut -d: -f1)" -check "premise: both line numbers were found, so the ordering arm can mean something" \ - "$([ -n "$_pm_ln_empty" ] && [ -n "$_pm_ln_absent" ] && echo yes || echo no)" "yes" - -check "the empty-plan refusal precedes the arm it protects" \ - "$([ "$_pm_ln_empty" -lt "$_pm_ln_absent" ] && echo before || echo "AFTER, so it is dead code")" \ - "before" - -# ---- and these greps must be able to FAIL ----------------------------------- -# -# Every arm above passes on a healthy tree, which a grep that matches nothing -# also does. Each fixture is the real shape with ONE property removed. - -_pm_fix="$PGC_WORKDIR/planmarker"; rm -rf "$_pm_fix"; mkdir -p "$_pm_fix" - -printf ' if False and not absent and not found:\n' > "$_pm_fix/present.py" -check "a neutered present arm is caught" \ - "$(grep -c 'if not absent and not found:' "$_pm_fix/present.py")" "0" - -printf ' if False and absent and found:\n' > "$_pm_fix/absent.py" -check "a neutered absent arm is caught" \ - "$(grep -c 'if absent and found:' "$_pm_fix/absent.py")" "0" - -printf ' if False and not nodes:\n' > "$_pm_fix/empty.py" -check "a neutered empty-plan refusal is caught" \ - "$(grep -c 'if not nodes:' "$_pm_fix/empty.py")" "0" - -# The mirror of the three above: the same greps on the REAL body return 1, so a -# zero is a missing guard rather than a broken pattern. -check "premise: while the real body satisfies all three, so the greps work" \ - "$(printf '%s\n' "$_pm_body" | grep -cE 'if not absent and not found:|if absent and found:|if not nodes:')" \ - "3" - -unset _pm_vac _pm_body _pm_fix _pm_ln_empty _pm_ln_absent From 98f16c6666a544cd9d2b7421a97e085f7b9b246c Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 01:45:33 +0000 Subject: [PATCH 3/6] test: the deletion of a part reaches three more artifacts, and a fourth file arrived MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebased onto main (d05e3c39). Deleting `test/selftest/370` turned out to touch three artifacts beyond the part itself, and the rebase surfaced a new item of debt that the inventory arm was built to catch. Artifacts the deletion reaches: - `test/selftest/parts.manifest` still named 370. #925's arm "every name in the manifest is a part on disk, so a deletion reddens" caught it, which is the arm doing precisely its job on the first deletion after it landed. - `test/check_ledger.tsv` held 12 rows for 370, all `never`. Removed. - `test/check_ledger_budget.txt`'s census `checks_never_observed_red` goes 826 -> 814. DERIVED from the ledger after the removal rather than typed, because that file says of itself: "it is not a ceiling; it is a measurement that must be true". `suites_not_covered` stays at 250 — deleting a part removes no suite. The 12 ledger rows are also the evidence for the claim made when 370 was deleted. They include "a neutered present arm is caught" and "premise: plan_marker's body was actually cut out of the file": 370 cut the Python body out and re-grepped its own detector. Every arm was a text pin, which is what #927's precedent is about. A fourth coupled file arrived with #925: `test_mutation_ledger.py` runs `run_all_versions.sh --list-suites`. The set-equality assertion in `test_harness_deps.py` reddened on the rebase naming a file the declaration did not have — the direction the inventory was made a mechanism for, working on something nobody wrote it for. It is now declared, with its mechanism, and CONTEXT.md records both the file and the fact that the arm is what found it. Same mechanism as `test_suite_accounting.py`, so the two are one item of debt twice and should move together. Verified on pg18a: selftest 815 checks, 0 FAIL; driver-free subset 196 passed; the full corpus green. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- CHANGELOG.md | 23 +++++++++++++++++++++++ CONTEXT.md | 6 ++++++ test/check_ledger.tsv | 12 ------------ test/check_ledger_budget.txt | 2 +- test/pytest/test_harness_deps.py | 3 +++ test/selftest/parts.manifest | 1 - 6 files changed, 33 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 149ae771..aa48d77e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -544,6 +544,29 @@ true until the next version shipped. name joined onto a `tmp_path`, because both are the string `lib.sh` and only the dataflow says which. + **The other direction: `test/selftest/370` is deleted.** It pinned the POSITION of + `plan_marker`'s empty-plan refusal by reading the Python source as text, which is all a + shell part can see. It cannot run the function, so it cannot tell a refusal that still + fires from one stranded behind an early return -- the only thing the position is for. + The property moved into `test_guards_pinned.py` as + `test_the_empty_plan_refusal_precedes_the_arms_it_protects`, which takes the class off + the `expect` fixture so the file still imports nothing. 370's other three properties + were already covered; confirmed by running those arms, not by reading them. + + That arm discriminates -- move the refusal to the end of `plan_marker` and it fails -- + and it is NOT load-bearing today: with the refusal at the end, + `plan_marker([], absent=True)` still refuses, because `plan_marker` has no early + return. So 370's stated reason, that the absent arm returns a pass first, describes a + shape the function does not have. It is labelled prospective insurance against a + refactor that adds one, rather than sold as a live hole. + + **CONTEXT.md said seven shell files reach across and the number is three.** `lib.sh`, + `selftest/030` and `selftest/040` matched only `pgc_cluster_datadir` and + `pgc_cluster_is_ours`, which are shell functions defined in `lib.sh` -- rule 3 of that + entry, "a word that merely looks like a filename", caught for the second time in the + entry that states it. `selftest/350`, `360` and `380` are the real three. + + - The vacuity guard's PLACEMENT is now a checked property, because a guard in a teardown cannot fail the test it guards (#432). diff --git a/CONTEXT.md b/CONTEXT.md index 1c33aa8e..c69df8f2 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -261,6 +261,12 @@ Python that reaches into shell: - `test_suite_accounting.py` -- reads `run_all_versions.sh`'s text, sources the real `lib.sh` from a suite it writes, and executes the real runner. - `pgc_cluster.py` -- sources the real `test/lib.sh`. +- `test_mutation_ledger.py` -- runs `run_all_versions.sh --list-suites` for the + registered suite list. It **arrived after this inventory was written**, with #925, + and the arm below is what said so: the set-equality assertion reddened on the + rebase naming a fourth file, which is the whole reason the inventory is a mechanism + and not this paragraph. Same mechanism as `test_suite_accounting.py`, so it is the + same item of debt twice and they should move together. Shell whose subject is python: `selftest/350`, `360` and `380`. **Three, not the seven this line first named, and the three it dropped were rule 3 all along.** diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 7f10fbaa..17a3ab14 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -478,18 +478,6 @@ harness_selftest 360-an-unrunnable-pytest-test-must the layer ends a session by harness_selftest 360-an-unrunnable-pytest-test-must the layer prints the unrunnable reason in lib.sh's shape never - harness_selftest 360-an-unrunnable-pytest-test-must the layer still writes the unrunnable state never - harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses agree on the INCOMPLETE exit code never - -harness_selftest 370-the-plan-marker-guard-must a neutered absent arm is caught never - -harness_selftest 370-the-plan-marker-guard-must a neutered empty-plan refusal is caught never - -harness_selftest 370-the-plan-marker-guard-must a neutered present arm is caught never - -harness_selftest 370-the-plan-marker-guard-must and that refusal is a VacuityError, not an ordinary assertion never - -harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is absent never - -harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is present never - -harness_selftest 370-the-plan-marker-guard-must plan_marker refuses a plan with no nodes at all never - -harness_selftest 370-the-plan-marker-guard-must premise: both line numbers were found, so the ordering arm can mean something never - -harness_selftest 370-the-plan-marker-guard-must premise: plan_marker's body was actually cut out of the file never - -harness_selftest 370-the-plan-marker-guard-must premise: the pytest layer is where this part thinks it is never - -harness_selftest 370-the-plan-marker-guard-must premise: while the real body satisfies all three, so the greps work never - -harness_selftest 370-the-plan-marker-guard-must the empty-plan refusal precedes the arm it protects never - harness_selftest 380-the-pytest-cluster-helpers a caller that reimplements the digest is caught never - harness_selftest 380-the-pytest-cluster-helpers a fingerprint that reads src only is caught never - harness_selftest 380-the-pytest-cluster-helpers a make_cluster with no cleanup is caught never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 01a8c04f..3464150e 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 826 +checks_never_observed_red 814 diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index a0feec05..2efed35d 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -913,6 +913,9 @@ def test_the_job_installs_no_database_driver(expect): "for the one permitted cross-implementation arm; see its module docstring", "test_suite_accounting.py": "reads the matrix runner's text and executes the real runner", + "test_mutation_ledger.py": + "executes the matrix runner with its list flag to get the registered suite " + "list, which is the same mechanism the entry above uses", } _SHELL_NAMES = re.compile( diff --git a/test/selftest/parts.manifest b/test/selftest/parts.manifest index 1a5f64d9..05fa1d47 100644 --- a/test/selftest/parts.manifest +++ b/test/selftest/parts.manifest @@ -34,7 +34,6 @@ 340-the-binary-must-be-built-from.sh 350-the-pytest-corpus-must-be.sh 360-an-unrunnable-pytest-test-must.sh -370-the-plan-marker-guard-must.sh 380-the-pytest-cluster-helpers.sh 390-a-registered-suite-must-account.sh 400-a-check-result-must-be-machine.sh From 34b5fcc017543dddc5a25be408caba948275c741 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 01:58:35 +0000 Subject: [PATCH 4/6] test/selftest: 360 keeps only what only it can check, and checks two more of them `test/selftest/360` pinned the SHAPE of `test/pytest/pgc_vacuity.py` with eleven greps: the unrunnable field is written, something reads it, the read reaches `session.exitstatus`, the override is conditional. A shell arm asserting a grep matches cannot prove a python arm is caught -- #927's precedent -- and the behaviour is pinned where it can be observed, by four arms in `test_layer.py` that run pytest inside pytest and assert on the inner run's exit status. MEASURED BEFORE DELETING ANYTHING. With `session.exitstatus = EXIT_INCOMPLETE` made unreachable in `pgc_vacuity.py`, the defect exactly as it shipped, those four arms go from `4 passed` to `2 failed, 2 passed`. Two rather than four is correct, not partial: the other two assert exit 1 for a run with a real failure and exit 0 for a run with nothing unrunnable, and neither outcome moves. Restored and compared byte-for-byte afterwards. THE PART'S OWN JUSTIFICATION WAS STALE. It said those arms "need pytest, psycopg and a virtualenv; CI installs none of them". CI has a `pytest-guards` job that installs pytest pinned from `requirements-test.txt`, asserts psycopg is absent, and runs the database-free file list, which contains `test_layer.py`. WHAT IS LEFT IS THE PERMITTED CROSS-REFERENCE, and there are three of them where the part checked one. The INCOMPLETE exit code was checked; the closed list of unrunnable reasons and the one-line shape an unrunnable check prints were not, though both are written down twice in two languages exactly as the exit code is. All three are now parsed out of both files -- never restated here, which would test this file against itself -- and each has a drifted fixture so the comparison can fail. Two instrument defects of mine, both caught by the new arms' own premises: - The shape parse took the first line matching `UNRUN `, which in `pgc_vacuity.py` is the docstring that spells the shape out for a reader. Requiring a quote before the marker selects the quoted string in both languages and excludes the prose, which opens with a backtick. - The drifted-shape fixture wrote ONE space where the real shape has two, so the parse found nothing and the comparison was empty-against-real. That "differs", for the wrong reason. Ledger: 360's rows regenerated with `pgc_ledger.py merge` from a real pg18a run rather than hand-edited, 17 rows out and 16 in, and `checks_never_observed_red` re-derived to 813. One check name was duplicated inside the part and is renamed, so one ledger row no longer covers two checks. `suites_not_covered` stays at 250: rewriting a part's checks removes no suite. Verified on pg18a: selftest 814 checks, 0 FAIL. `pgc_ledger.py gate` with the registered list and `--against auto`: rc=0, `new this run=0`, ceiling 250. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- CHANGELOG.md | 37 +++ test/check_ledger.tsv | 21 +- test/check_ledger_budget.txt | 2 +- .../360-an-unrunnable-pytest-test-must.sh | 235 ++++++++++-------- 4 files changed, 173 insertions(+), 122 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aa48d77e..44639089 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -566,6 +566,43 @@ true until the next version shipped. entry, "a word that merely looks like a filename", caught for the second time in the entry that states it. `selftest/350`, `360` and `380` are the real three. + **`test/selftest/360` is rewritten rather than deleted, and it now checks more.** + Eleven of its seventeen arms were text pins on `pgc_vacuity.py`: that the + unrunnable field is written, that something reads it, that the read reaches + `session.exitstatus`, that the override is conditional. Those are gone. What is + left is the kind CONTEXT.md permits -- a property that IS the relationship between + the two harnesses, so it cannot be stated from one side -- and there turned out to + be THREE such properties where the part checked one. + + The part checked the INCOMPLETE exit code and left two duplications beside it + unchecked: the closed list of unrunnable reasons, and the one-line shape an + unrunnable check prints. Both are now parsed out of both files and compared, with + a drifted fixture for each so the comparison can fail. The reason list is the + worse omission of the two, because being closed on both sides is its whole purpose. + + Both deletions were measured first. With `session.exitstatus = EXIT_INCOMPLETE` + made unreachable in `pgc_vacuity.py` -- the defect exactly as it shipped -- the + four behavioural arms in `test_layer.py` go from `4 passed` to `2 failed, 2 + passed`. Two is correct rather than partial: the other two assert exit 1 for a run + with a real failure and exit 0 for a run with nothing unrunnable, and neither + outcome moves. The file was restored and compared byte-for-byte afterwards. + + **And the reason the part gave for keeping those pins was true when written and is + not now.** It said the behavioural arms "need pytest, psycopg and a virtualenv; CI + installs none of them". CI has a `pytest-guards` job that installs pytest pinned + from `requirements-test.txt`, asserts psycopg is absent, and runs the + database-free file list -- which contains `test_layer.py`. A stale justification + for keeping coverage in the wrong place is harder to find than a missing check, + because nothing reddens. + + Two instrument defects of my own, both caught by the new arms' own premises. The + shape parse took the FIRST line matching the print marker, which in + `pgc_vacuity.py` is the DOCSTRING that spells the shape out for a reader; + requiring a quote before the marker selects the code in both languages. And the + drifted-shape fixture wrote one space where the real shape has two, so the parse + found nothing and the comparison was empty-against-real -- which "differs", for + the wrong reason. Its premise arm said so. + - The vacuity guard's PLACEMENT is now a checked property, because a guard in a teardown cannot fail the test it guards (#432). diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 17a3ab14..71c29d93 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -461,23 +461,22 @@ harness_selftest 350-the-pytest-corpus-must-be the counter stops at the next hea harness_selftest 350-the-pytest-corpus-must-be the gate runs the harness guards never - harness_selftest 350-the-pytest-corpus-must-be the row's value is read, not a digit inside its label never - harness_selftest 350-the-pytest-corpus-must-be the sweep counts the fixture's tests and files never - -harness_selftest 360-an-unrunnable-pytest-test-must a comparison on the exit status is not counted as an assignment never - harness_selftest 360-an-unrunnable-pytest-test-must a drifted exit code is visible rather than absorbed never - -harness_selftest 360-an-unrunnable-pytest-test-must a write-only unrunnable field is caught never - -harness_selftest 360-an-unrunnable-pytest-test-must an unconditional exit override is caught by the dominance arm never - -harness_selftest 360-an-unrunnable-pytest-test-must and only ever moves a run off zero, so a failure still dominates never - -harness_selftest 360-an-unrunnable-pytest-test-must and something READS it, rather than only writing it never - -harness_selftest 360-an-unrunnable-pytest-test-must premise: and that same fixture does show the write, so the arm is not blind never - +harness_selftest 360-an-unrunnable-pytest-test-must a drifted print shape is visible never - +harness_selftest 360-an-unrunnable-pytest-test-must a reason dropped from one side only is visible never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: and that fixture is a real list rather than an empty parse never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: and that fixture parses to a shape rather than to nothing never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: lib.sh prints a line for an unrunnable check never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: lib.sh states a closed list of unrunnable reasons never - harness_selftest 360-an-unrunnable-pytest-test-must premise: lib.sh states an INCOMPLETE exit code this part could read never - harness_selftest 360-an-unrunnable-pytest-test-must premise: the harness library is where this part thinks it is never - harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer is where this part thinks it is never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer prints one too never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer states a closed list too never - harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer states one too never - -harness_selftest 360-an-unrunnable-pytest-test-must premise: while a real assignment on the same line shape IS counted never - -harness_selftest 360-an-unrunnable-pytest-test-must premise: while the real layer satisfies that same arm never - -harness_selftest 360-an-unrunnable-pytest-test-must the layer ends a session by setting its exit status never - -harness_selftest 360-an-unrunnable-pytest-test-must the layer prints the unrunnable reason in lib.sh's shape never - -harness_selftest 360-an-unrunnable-pytest-test-must the layer still writes the unrunnable state never - harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses agree on the INCOMPLETE exit code never - +harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses agree on the closed list of unrunnable reasons never - +harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses print an unrunnable check in the same shape never - harness_selftest 380-the-pytest-cluster-helpers a caller that reimplements the digest is caught never - harness_selftest 380-the-pytest-cluster-helpers a fingerprint that reads src only is caught never - harness_selftest 380-the-pytest-cluster-helpers a make_cluster with no cleanup is caught never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 3464150e..1e7fb406 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 814 +checks_never_observed_red 813 diff --git a/test/selftest/360-an-unrunnable-pytest-test-must.sh b/test/selftest/360-an-unrunnable-pytest-test-must.sh index 81effe10..831d491c 100644 --- a/test/selftest/360-an-unrunnable-pytest-test-must.sh +++ b/test/selftest/360-an-unrunnable-pytest-test-must.sh @@ -1,30 +1,47 @@ -# ---- an unrunnable pytest test must not leave the run green ------------------ +# ---- the two harnesses must agree about the third state ---------------------- # -# WHY THIS EXISTS. `expect.cannot_run(REASON, detail)` is the pytest corpus's -# third state, the counterpart of `check_unrunnable` here. It wrote -# `self.unrunnable` and NOTHING READ IT, so a test that declared itself -# unrunnable reported `1 passed` and exit 0. Measured, before the fix. +# WHY THIS EXISTS. `expect.cannot_run(REASON, detail)` in the pytest corpus is the +# counterpart of `check_unrunnable` here, and the two sides share three things that +# are WRITTEN DOWN TWICE, once in each language: the INCOMPLETE exit code, the +# closed list of reasons, and the one-line shape an unrunnable check prints. A +# value duplicated across a language boundary drifts, and the drift is invisible -- +# the corpus would exit 67, a runner would compare against something else, and an +# INCOMPLETE run would read as a failure or as a pass depending on which way it +# moved. # -# That is the write-only-flag shape selftest 320 already polices one level up, -# where the runner's INCOMPLETE branch set a variable the verdict never read. It -# mattered more here: a bare `@pytest.mark.skip` FAILS the pytest run, so the -# layer refused the cheap dishonest escape and permitted the expensive-looking -# one. An escape hatch that costs nothing is the default. +# WHAT THIS PART DOES **NOT** DO ANY MORE (#432). It used to also pin the SHAPE of +# `pgc_vacuity.py`: that the unrunnable field is written, that something reads it, +# that the read reaches `session.exitstatus`, and that the override is conditional. +# Eleven arms, every one a text pin on the other harness's source. Those are gone. +# A shell arm asserting a grep matches cannot prove a python arm is CAUGHT -- #927's +# precedent -- and the behaviour is pinned where it can actually be observed, by four +# arms in `test/pytest/test_layer.py` that run pytest inside pytest and assert on the +# inner run's exit status. # -# WHY THE CHECKS ARE STATIC. The behaviour itself is pinned in the corpus, by -# four arms in test/pytest/test_layer.py that run pytest inside pytest and assert -# on the inner run's exit status. Those need pytest, psycopg and a virtualenv; -# CI installs none of them, and `pgc_skip` treats a missing dependency as a -# failure rather than a skip. So the behavioural half lives where it can run and -# this half asserts the STRUCTURE that behaviour rests on, which is greppable -# from a checkout with nothing installed. Same division as selftest 320's last -# two arms, which grep the runner for the call and for the absence of the flag. +# THE OLD REASON FOR KEEPING THEM WAS TRUE AND IS NOT ANY MORE. It said those arms +# "need pytest, psycopg and a virtualenv; CI installs none of them". CI now has a +# `pytest-guards` job that installs pytest pinned from `requirements-test.txt`, +# asserts psycopg is absent, and runs the database-free file list -- which includes +# `test_layer.py`. So the behavioural half runs in CI and this half is not its last +# line of defence. # -# THE NUMBER IS THE POINT OF THE FIRST ARM. 67 now lives in two files. A number -# duplicated across a language boundary is a number that drifts, and the drift -# is invisible: the corpus would exit 67, the runner would compare against -# something else, and an INCOMPLETE run would read as a failure or as a pass -# depending on which way it moved. +# MEASURED BEFORE DELETING ANYTHING, because "the other side covers it" is a claim. +# With `session.exitstatus = EXIT_INCOMPLETE` made unreachable in `pgc_vacuity.py` +# -- the defect exactly as it shipped -- those four arms go from `4 passed` to +# `2 failed, 2 passed`. Two, not four, and that is correct rather than partial: the +# other two assert exit 1 for a run with a real failure and exit 0 for a run with +# nothing unrunnable, and neither of those outcomes moves. The file was restored and +# compared byte-for-byte afterwards. +# +# WHAT IS LEFT IS THE ONE PERMITTED CROSS-REFERENCE, in CONTEXT.md's sense: a +# property that IS the relationship between the two harnesses and so cannot be +# expressed from one side. Each of the three agreements below is PARSED OUT OF BOTH +# FILES rather than written here. A check that restates the value tests this file +# against itself: both copies could drift together and it would still pass. +# +# AND THE LIST AND THE SHAPE ARE NEW. The part checked the exit code only, while +# two more duplications sat beside it unchecked -- including the reason list, whose +# whole purpose is to be closed on both sides. _ts_lib="$PGC_TESTDIR/lib.sh" _ts_vac="$PGC_TESTDIR/pytest/pgc_vacuity.py" @@ -35,9 +52,8 @@ check "premise: the harness library is where this part thinks it is" \ check "premise: the pytest layer is where this part thinks it is" \ "$([ -f "$_ts_vac" ] && echo yes || echo no)" "yes" -# Parsed out of each file rather than written here. A check that restates the -# number tests this file against itself: both copies could drift together and it -# would still pass. +# ---- agreement 1: the INCOMPLETE exit code ---------------------------------- + _ts_sh_code="$(sed -n 's/^PGC_EXIT_INCOMPLETE=\([0-9]\{1,\}\).*/\1/p' "$_ts_lib" | head -1)" _ts_py_code="$(sed -n 's/^EXIT_INCOMPLETE[[:space:]]*=[[:space:]]*\([0-9]\{1,\}\).*/\1/p' "$_ts_vac" | head -1)" @@ -50,102 +66,101 @@ check "premise: the pytest layer states one too" \ check "the two harnesses agree on the INCOMPLETE exit code" \ "$_ts_py_code" "$_ts_sh_code" -# ---- the field must be READ, which is the defect this part is named after --- +# ---- agreement 2: the closed list of reasons -------------------------------- # -# Counted as two populations rather than asserted as a boolean, so the failure -# says which side is missing. -_ts_writes="$(grep -c 'self\.unrunnable[[:space:]]*=' "$_ts_vac")" -_ts_reads="$(grep -c 'rec\.unrunnable' "$_ts_vac")" +# Sorted, because the two files are free to list them in different orders and an +# order difference is not a drift. Compared as one string so the failure prints +# both lists side by side and names which token moved. -check "the layer still writes the unrunnable state" \ - "$([ "$_ts_writes" -ge 1 ] && echo yes || echo "$_ts_writes")" "yes" +_ts_sh_reasons="$(sed -n 's/^PGC_UNRUN_REASONS="\([^"]*\)".*/\1/p' "$_ts_lib" \ + | head -1 | tr ' ' '\n' | grep -E '^[A-Z_]+$' | sort | tr '\n' ' ')" +_ts_py_reasons="$(sed -n '/^UNRUNNABLE_REASONS = (/,/^)/p' "$_ts_vac" \ + | grep -oE '"[A-Z_]+"' | tr -d '"' | sort | tr '\n' ' ')" -check "and something READS it, rather than only writing it" \ - "$([ "$_ts_reads" -ge 1 ] && echo yes || echo "$_ts_reads")" "yes" +check "premise: lib.sh states a closed list of unrunnable reasons" \ + "$([ -n "$_ts_sh_reasons" ] && echo yes || echo no)" "yes" -# ---- and the read has to reach the run's exit status ------------------------ -# -# Reading the field into a list nothing acts on would satisfy the arm above and -# leave the defect exactly where it was. -# -# TWO CORRECTIONS THIS ARM ALREADY NEEDED, both found by running it against a -# LATER branch rather than by rereading it. -# -# `=` MATCHES INSIDE `==`. The first pattern was `exitstatus[[:space:]]*=`, which -# counted the comparison `if ... session.exitstatus == 0:` as an assignment. A -# read counted as a write, in the arm whose entire subject is the difference -# between the two. `[^=]` after the `=` is what separates them. +check "premise: the pytest layer states a closed list too" \ + "$([ -n "$_ts_py_reasons" ] && echo yes || echo no)" "yes" + +check "the two harnesses agree on the closed list of unrunnable reasons" \ + "$_ts_py_reasons" "$_ts_sh_reasons" + +# ---- agreement 3: the line an unrunnable check prints ----------------------- # -# AND THE COUNT IS A FLOOR, NOT AN EQUALITY. Written as "exactly 1", it went red -# on the next branch in this stack, which adds a second escalation for a -# different condition -- a legitimate addition reported as a defect. The property -# is "the read reaches the exit status", so one site satisfies it and two do not -# make it less true. An equality here is a guard that reddens on growth, and a -# guard that reddens on growth gets switched off. Measured: 1 site on this -# branch, 2 on audit/432-pytest-oracles, 0 on the pre-fix layer -- so the floor -# still fails exactly where it must. -_ts_assigns="$(grep -c 'session\.exitstatus[[:space:]]*=[^=]' "$_ts_vac")" -check "the layer ends a session by setting its exit status" \ - "$([ "$_ts_assigns" -ge 1 ] && echo yes || echo "$_ts_assigns")" "yes" - -# Failure dominates, the same rule lib.sh keeps: a run with a failure AND an -# unrunnable test is a failure. So the override must be conditional on a -# currently-clean run. Unconditional, it would MASK failures as INCOMPLETE. -check "and only ever moves a run off zero, so a failure still dominates" \ - "$(grep -c 'exitstatus == 0' "$_ts_vac")" "1" - -# The state says why. A third state that does not name its reason is a skip. -check "the layer prints the unrunnable reason in lib.sh's shape" \ - "$(grep -c 'UNRUN.*{reason}' "$_ts_vac")" "1" - -# ---- and these greps must be able to FAIL ----------------------------------- +# Normalised, not compared raw: one side interpolates `$name` and the other +# `{nodeid}`, which is a difference between two languages rather than between two +# behaviours. Every interpolation becomes `%`, so what is compared is the literal +# text around them -- the part a log reader and `pgc_record` actually see. + +_ts_shape_of() { # _ts_shape_of LINE -> the literal text with interpolations as % + printf '%s\n' "$1" \ + | grep -oE 'UNRUN [^"]*' \ + | sed -e 's/\$[A-Za-z_][A-Za-z_0-9]*/%/g' -e 's/{[A-Za-z_][A-Za-z_0-9]*}/%/g' \ + | head -1 +} + +# THE LINE HAS TO BE THE CODE, NOT THE PROSE ABOUT IT. The first version matched +# `UNRUN ` anywhere and took `pgc_vacuity.py`'s DOCSTRING, which spells the shape +# out as `UNRUN : : ` for a reader and then keeps talking -- +# so the arm compared a sentence against a format string and failed. Requiring a +# quote immediately before the marker selects the quoted string in both languages +# and excludes the prose, which opens with a backtick. The premise arms below could +# not have caught this: the sentence is not empty. +_ts_sh_shape="$(_ts_shape_of "$(grep -m1 '"UNRUN ' "$_ts_lib")")" +_ts_py_shape="$(_ts_shape_of "$(grep -m1 '"UNRUN ' "$_ts_vac")")" + +check "premise: lib.sh prints a line for an unrunnable check" \ + "$([ -n "$_ts_sh_shape" ] && echo yes || echo no)" "yes" + +check "premise: the pytest layer prints one too" \ + "$([ -n "$_ts_py_shape" ] && echo yes || echo no)" "yes" + +check "the two harnesses print an unrunnable check in the same shape" \ + "$_ts_py_shape" "$_ts_sh_shape" + +# ---- and each comparison must be able to FAIL ------------------------------- # -# Every arm above passes on a healthy tree, which a grep that matches nothing -# also does -- against an EMPTY file, `grep -c` returns 0 and every `-ge 1` arm -# would read "no" while every `-c ... "1"` arm would read 0. Those would be -# visible. The dangerous case is the opposite: a pattern that is subtly wrong -# still matching. So the fixtures below are the real code with ONE property -# removed, and the arms assert the greps notice. +# Three agreements that pass on a healthy tree, which three comparisons of an +# empty string against an empty string also do. The premises above refuse the +# empty case; these fixtures are the drifted case, which is the one that matters, +# because a reader cannot tell an agreement from a pair of blanks. _ts_fix="$PGC_WORKDIR/thirdstate"; rm -rf "$_ts_fix"; mkdir -p "$_ts_fix" -# The defect as it shipped: the field is written and never read. -{ - printf 'EXIT_INCOMPLETE = 67\n' - printf 'class Expect:\n' - printf ' def cannot_run(self, reason, detail=""):\n' - printf ' self.unrunnable = (reason, detail)\n' -} > "$_ts_fix/writeonly.py" - -check "a write-only unrunnable field is caught" \ - "$(grep -c 'rec\.unrunnable' "$_ts_fix/writeonly.py")" "0" - -check "premise: and that same fixture does show the write, so the arm is not blind" \ - "$(grep -c 'self\.unrunnable[[:space:]]*=' "$_ts_fix/writeonly.py")" "1" - -# The exit code drifted. Both files parse; the values differ. printf 'EXIT_INCOMPLETE = 66\n' > "$_ts_fix/drifted.py" _ts_drift="$(sed -n 's/^EXIT_INCOMPLETE[[:space:]]*=[[:space:]]*\([0-9]\{1,\}\).*/\1/p' "$_ts_fix/drifted.py" | head -1)" check "a drifted exit code is visible rather than absorbed" \ "$([ "$_ts_drift" = "$_ts_sh_code" ] && echo agrees || echo "differs:$_ts_drift/$_ts_sh_code")" \ "differs:66/67" -# An unconditional override, which would mask a failing run as INCOMPLETE. -printf ' session.exitstatus = EXIT_INCOMPLETE\n' > "$_ts_fix/unconditional.py" -check "an unconditional exit override is caught by the dominance arm" \ - "$(grep -c 'exitstatus == 0' "$_ts_fix/unconditional.py")" "0" - -# A file holding ONLY the comparison. Under the first pattern this counted as an -# assignment, which is the false positive that shipped; under the corrected one it -# is zero. Without this arm the correction above is itself unproved. -printf ' if exitstatus == 0 and session.exitstatus == 0:\n' > "$_ts_fix/compare.py" -check "a comparison on the exit status is not counted as an assignment" \ - "$(grep -c 'session\.exitstatus[[:space:]]*=[^=]' "$_ts_fix/compare.py")" "0" - -check "premise: while a real assignment on the same line shape IS counted" \ - "$(grep -c 'session\.exitstatus[[:space:]]*=[^=]' "$_ts_fix/unconditional.py")" "1" - -check "premise: while the real layer satisfies that same arm" \ - "$(grep -c 'exitstatus == 0' "$_ts_vac")" "1" - -unset _ts_lib _ts_vac _ts_sh_code _ts_py_code _ts_writes _ts_reads _ts_fix _ts_drift _ts_assigns +{ + printf 'UNRUNNABLE_REASONS = (\n' + printf ' "MISSING_DEPENDENCY",\n' + printf ' "UNSUPPORTED_MAJOR",\n' + printf ')\n' +} > "$_ts_fix/shortlist.py" +_ts_short="$(sed -n '/^UNRUNNABLE_REASONS = (/,/^)/p' "$_ts_fix/shortlist.py" \ + | grep -oE '"[A-Z_]+"' | tr -d '"' | sort | tr '\n' ' ')" +check "a reason dropped from one side only is visible" \ + "$([ "$_ts_short" = "$_ts_sh_reasons" ] && echo agrees || echo differs)" "differs" + +check "premise: and that fixture is a real list rather than an empty parse" \ + "$(printf '%s' "$_ts_short" | wc -w)" "2" + +# Two spaces after UNRUN, as the real shape has: the drift under test is the lost +# colons, not a lost space. The first fixture wrote ONE space, so the parse found +# nothing and the comparison was empty-against-real -- which "differs", for the +# wrong reason. Its own premise arm caught that, which is what the premise is for. +printf ' terminalreporter.write_line(f"UNRUN {nodeid} {reason} {detail}")\n' \ + > "$_ts_fix/drifted_shape.py" +_ts_shape_drift="$(_ts_shape_of "$(grep -m1 '"UNRUN ' "$_ts_fix/drifted_shape.py")")" +check "a drifted print shape is visible" \ + "$([ "$_ts_shape_drift" = "$_ts_sh_shape" ] && echo agrees || echo differs)" "differs" + +check "premise: and that fixture parses to a shape rather than to nothing" \ + "$([ -n "$_ts_shape_drift" ] && echo yes || echo no)" "yes" + +unset _ts_lib _ts_vac _ts_sh_code _ts_py_code _ts_sh_reasons _ts_py_reasons \ + _ts_sh_shape _ts_py_shape _ts_fix _ts_drift _ts_short _ts_shape_drift +unset -f _ts_shape_of From f11b338637bca61a29fff1d4385a3d21907ace89 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 02:05:52 +0000 Subject: [PATCH 5/6] test/selftest: 380 keeps its own finding and hands the other one to pytest `test/selftest/380` covers two of @linuxhikerpm's #897 findings, and they have different subjects. Finding 1 is `test/pgc_fingerprint.py`, which `lib.sh` runs with the system interpreter and which is not part of the pytest harness: checking it is this part's own business and it stays. Finding 2 is `test/pytest/pgc_cluster.py`, and the nine arms that read it as text are gone. SEVEN of the twenty-one checks had a cross-harness subject, not twenty-one. The other fourteen read `pgc_fingerprint.py`, `lib.sh`, or fixtures the part writes itself. Counting the file rather than the arms would have deleted coverage that was never debt. `make_cluster`'s cleanup shape moved to `test_build_refusal.py` as `test_the_cleanup_guard_has_the_shape_the_leak_needs`, beside the behavioural arm that provokes a real failed setup and asserts no directory is left. It is a SOURCE check and it is labelled as one: `make_cluster` fails exactly one way in that arm, a missing `pg_config`, while three more properties decide whether the guard works -- surviving a KeyboardInterrupt, stopping a postmaster it already started, and re-raising rather than returning None -- and two of them cannot be provoked from a test at all. `test_this_module_keeps_no_private_fingerprint` moved for the same reason. FIVE MUTATIONS SAY THE MOVED ARMS DISCRIMINATE, each asserted to have applied, each leaving the module well-formed, restored byte-for-byte afterwards: BaseException narrowed to Exception source arm cluster.stop() removed source arm bare re-raise turned into pass source arm AND the behavioural one a private hashlib.md5 added the no-private-digest arm shutil.rmtree(root, ...) removed source arm AND the behavioural one The two that redden both are the two whose effect reaches the filesystem. The three that redden only the source arm are exactly the properties the behavioural arm cannot see, which is why they are written down separately. One mutation did not apply first time: the `cluster.stop()` pattern assumed twelve spaces of indentation and the call sits at sixteen, inside a nested `try`. The harness refused to report a result rather than printing a green. AND ONE THING WENT WRONG THAT IS WORTH MORE THAN THE CHANGE. I ran the selftest before naming the two new arms in TESTS.md, so the run was red on the doc-coverage check, and then merged that log into `check_ledger.tsv` -- whose entire subject is which checks have ever been red. It recorded a red for part 350's "every test file and every test in the corpus is named in TESTS.md", a check that failed only because my own change was half-finished. Reverted with `git checkout HEAD --`; there is no second copy to repair from. The merge step now refuses a log that is not DEFINITELY green: a numeric `checks run:` line, a floor on it so an aborted run cannot pass, zero FAIL lines, and rc=0. The absence of a FAIL line is not enough, because an aborted run has none either. Verified on pg18a: selftest 805 checks, 0 FAIL; `pgc_ledger.py gate` rc=0 with `new this run=0` and the ceiling still 250; driver-free subset 198 passed; the full corpus 283 passed. 380's ledger rows regenerated from that green run, 21 out and 12 in, and `checks_never_observed_red` re-derived to 804. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- CHANGELOG.md | 46 +++++++++++++ test/check_ledger.tsv | 9 --- test/check_ledger_budget.txt | 2 +- test/pytest/TESTS.md | 31 +++++++++ test/pytest/test_build_refusal.py | 61 +++++++++++++++++ .../380-the-pytest-cluster-helpers.sh | 65 +++++++------------ 6 files changed, 161 insertions(+), 53 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 44639089..8a1061de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -603,6 +603,52 @@ true until the next version shipped. found nothing and the comparison was empty-against-real -- which "differs", for the wrong reason. Its premise arm said so. + **`test/selftest/380` keeps finding 1 and hands finding 2 to the pytest corpus.** + The part covers two of @linuxhikerpm's #897 findings, and they have different + subjects. Finding 1 is `test/pgc_fingerprint.py`, which `lib.sh` runs with the + system interpreter and which is not part of the pytest harness, so checking it is + this part's own business and it stays. Finding 2 is `test/pytest/pgc_cluster.py`, + and the nine arms that read it as text are gone. + + Seven of its twenty-one checks had a cross-harness subject, not twenty-one: the + other fourteen read `pgc_fingerprint.py`, `lib.sh`, or fixtures the part writes + itself. Counting the whole file would have deleted coverage that was never debt. + + `make_cluster`'s cleanup SHAPE moved to `test_build_refusal.py` as + `test_the_cleanup_guard_has_the_shape_the_leak_needs`, beside the behavioural arm + that provokes a real failed setup. It has to be a source check, and it says so: + `make_cluster` fails exactly one way in the behavioural arm -- a missing + `pg_config` -- while three more properties decide whether the guard works, and two + of those cannot be provoked at all. You cannot deliver SIGINT into `initdb` + reliably, and a cluster that started is one the arm would then have to stop. + + Five mutations say the moved arms discriminate, each asserted to have applied, + each leaving the module well-formed, restored byte-for-byte afterwards: + `BaseException` narrowed to `Exception`, `cluster.stop()` removed, the bare + re-raise turned into `pass`, a private `hashlib.md5` added, and + `shutil.rmtree(root, …)` removed. The two whose effect reaches the filesystem + redden the behavioural arm as well; the three that redden only the source arm are + exactly the properties the behavioural arm cannot see. + + One of those five did not apply on the first attempt -- the `cluster.stop()` + pattern assumed twelve spaces of indentation and the call sits at sixteen, inside + a nested `try`. The harness refused to report a result for it rather than printing + a green, which is the only safe behaviour for a mutation that did not land. + + **And one thing went wrong that is worth more than the change itself.** I ran the + selftest before naming the two new pytest arms in `TESTS.md`, so the run was red on + the doc-coverage check -- and then merged that log into `check_ledger.tsv`, whose + entire subject is which checks have ever been red. It recorded a red for + `350`'s "every test file and every test in the corpus is named in TESTS.md", + a check that failed only because my own change was half-finished. Reverted with + `git checkout HEAD --`; there is no second copy to repair from. + + The merge step now refuses a log that is not definitely green: a numeric + `checks run:` line, a floor on it so an aborted run cannot pass, zero `FAIL` + lines, and rc=0. The absence of a `FAIL` line is not enough, because an aborted + run has none either -- the same shape as a pending-count that cannot see a job + which never started. + - The vacuity guard's PLACEMENT is now a checked property, because a guard in a teardown cannot fail the test it guards (#432). diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 71c29d93..1e45e04e 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -479,25 +479,16 @@ harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses agree on t harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses print an unrunnable check in the same shape never - harness_selftest 380-the-pytest-cluster-helpers a caller that reimplements the digest is caught never - harness_selftest 380-the-pytest-cluster-helpers a fingerprint that reads src only is caught never - -harness_selftest 380-the-pytest-cluster-helpers a make_cluster with no cleanup is caught never - harness_selftest 380-the-pytest-cluster-helpers an import from the pytest tree is caught never - harness_selftest 380-the-pytest-cluster-helpers and a hard-coded module list is caught by the name arm never - -harness_selftest 380-the-pytest-cluster-helpers and it catches BaseException, so an interrupt cleans up too never - harness_selftest 380-the-pytest-cluster-helpers and it names no module directory, so it is a derivation and not a list never - -harness_selftest 380-the-pytest-cluster-helpers and it stops a partially started cluster before removing the tree never - harness_selftest 380-the-pytest-cluster-helpers and no longer mixes in the bare filename never - -harness_selftest 380-the-pytest-cluster-helpers and the original error is re-raised rather than swallowed never - harness_selftest 380-the-pytest-cluster-helpers and the shell keeps none either never - -harness_selftest 380-the-pytest-cluster-helpers make_cluster removes its tree when setup raises never - -harness_selftest 380-the-pytest-cluster-helpers premise: and the real helper still carries its cleanup never - -harness_selftest 380-the-pytest-cluster-helpers premise: make_cluster's body was actually cut out of the file never - harness_selftest 380-the-pytest-cluster-helpers premise: the one fingerprint implementation is where this part thinks it is never - -harness_selftest 380-the-pytest-cluster-helpers premise: the pytest cluster helper is where this part thinks it is never - harness_selftest 380-the-pytest-cluster-helpers premise: while the real module satisfies the derivation arm never - harness_selftest 380-the-pytest-cluster-helpers the fingerprint derives its build directories from a Makefile on disk never - harness_selftest 380-the-pytest-cluster-helpers the hash mixes in each file's path relative to the tree, not its name never - harness_selftest 380-the-pytest-cluster-helpers the module imports nothing from the pytest tree never - -harness_selftest 380-the-pytest-cluster-helpers the pytest helper keeps no private fingerprint implementation never - harness_selftest 390-a-registered-suite-must-account a NEW unaccounted suite fails even while the known debt is excused never - harness_selftest 390-a-registered-suite-must-account a comment mentioning pgc_summary is not a declaration never - harness_selftest 390-a-registered-suite-must-account a declared suite that produced no accounting is caught never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 1e7fb406..a64687bc 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 813 +checks_never_observed_red 804 diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 70060d0c..1b75de19 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -466,6 +466,37 @@ is where a wrong quote would hide. | `test_the_fingerprint_covers_a_separately_built_module` | an `objstore/` edit moves the hash | | `test_an_objstore_edit_forces_a_second_build` | and forces a rebuild, end to end | | `test_make_cluster_leaves_nothing_behind_when_setup_fails` | a failed setup leaks no directory | +| `test_the_cleanup_guard_has_the_shape_the_leak_needs` | **source check**: the guard catches `BaseException`, stops the cluster, removes the tree, and re-raises | +| `test_this_module_keeps_no_private_fingerprint` | **source check**: no second digest implementation in this caller | + +**Two of these are SOURCE checks, and they say so.** `test_make_cluster_leaves_nothing_behind_when_setup_fails` +provokes a real failed setup and asserts no directory is left; that is the property. +But `make_cluster` fails exactly one way in that arm — a missing `pg_config` — while +three more properties decide whether the guard works at all: it must survive a +`KeyboardInterrupt`, stop a postmaster it already started, and re-raise rather than +return `None`. Two of those cannot be provoked from a test (you cannot deliver SIGINT +into `initdb` reliably, and a cluster that started is one the arm would then have to +stop), so they are read off `inspect.getsource(make_cluster)` instead. + +They arrived from `test/selftest/380`, which read this file as text across the harness +boundary. Reading our own module is not a cross-harness reference; a shell part +grepping it is the thing CONTEXT.md refuses. What the shell part could never do is the +behavioural arm above it. + +Five mutations say the source arms discriminate, each asserted to have applied and +each leaving the module well-formed, restored byte-for-byte afterwards: + +| mutation of `pgc_cluster.py` | what reddens | +| --- | --- | +| `except BaseException:` narrowed to `except Exception:` | the source arm | +| `cluster.stop()` removed | the source arm | +| the bare `raise` turned into `pass` | the source arm **and** the behavioural one | +| a private `hashlib.md5` added | the no-private-digest arm | +| `shutil.rmtree(root, …)` removed | the source arm **and** the behavioural one | + +The two that redden both are the two whose effect reaches the filesystem. The three +that redden only the source arm are exactly the properties the behavioural arm cannot +see, which is why they are written down separately rather than folded into it. | `test_the_stamp_writer_reports_failure` | `\|\| true` made both controllers' warnings unreachable | | `test_two_installations_of_one_major_do_not_share_a_stamp` | the key names the installation, not just the major | | `test_moving_bytes_between_files_moves_the_shell_fingerprint` | the digest sees a repartition | diff --git a/test/pytest/test_build_refusal.py b/test/pytest/test_build_refusal.py index 2976ad9e..c8f97b1d 100644 --- a/test/pytest/test_build_refusal.py +++ b/test/pytest/test_build_refusal.py @@ -60,6 +60,7 @@ """ import hashlib +import inspect import os import sys import tempfile @@ -400,6 +401,66 @@ def test_make_cluster_leaves_nothing_behind_when_setup_fails(tmp_path, expect): "a failed make_cluster leaves no directory behind") +def test_the_cleanup_guard_has_the_shape_the_leak_needs(expect): + """The arm above proves the tree is gone. It cannot prove WHY, and three of + the four properties that make the guard work are invisible to it. + + `make_cluster` fails one way in that test: a missing `pg_config`. The guard + also has to survive a KeyboardInterrupt, stop a postmaster it already started, + and re-raise rather than return None. Each needs a different failure to + observe, and two of them cannot be provoked from a test at all -- you cannot + deliver SIGINT into `initdb` reliably, and a cluster that started is a cluster + this arm would then have to stop. + + So these are SOURCE checks, and they are labelled as such rather than sold as + behavioural. They moved here from `test/selftest/380`, which read this file as + text across the harness boundary. A shell part can pin this text; what it + cannot do is what the arm above does. Reading our own module is not a + cross-harness reference, so the property lives where its subject lives. + """ + import inspect + body = inspect.getsource(make_cluster) + + # Cut to the handler. These shapes occur elsewhere in the module, and a check + # over the whole file reports a guard present that lives in another function. + expect.at_least(len(body.splitlines()), 15, + "premise: the source of the function itself was read") + + for fragment, name in ( + ("except BaseException:", + "the handler catches BaseException, so an interrupt cleans up too"), + ("cluster.stop()", + "and it stops a partially started cluster"), + ("shutil.rmtree(root", + "and it removes the tree"), + ): + expect.num(body.count(fragment), 1, name) + + # The bare re-raise, anchored to a line: `raise` appears inside this function + # in the is_ours() refusal too, as `raise RuntimeError(...)`, and a substring + # count would read that as the re-raise and pass with the re-raise deleted. + bare = [ln for ln in body.splitlines() if ln.strip() == "raise"] + expect.num(len(bare), 1, + "and the original error is re-raised rather than swallowed") + + +def test_this_module_keeps_no_private_fingerprint(expect): + """One implementation, in `test/pgc_fingerprint.py`, since #907. + + A second copy reappearing in a caller is the defect rather than a detail: it + agrees with the shared one until it does not, and nothing says when that was. + Also a source check, and also moved out of `test/selftest/380`. + """ + import pgc_cluster + src = pathlib.Path(inspect.getsourcefile(pgc_cluster)).read_text(encoding="utf-8") + expect.at_least(len(src), 200, "premise: the module's own source was read") + for fragment, name in ( + ("hashlib.md5", "no private md5 digest"), + ('glob("*/Makefile")', "and no private build-directory walk"), + ): + expect.num(src.count(fragment), 0, name) + + # --------------------------------------------------------------------------- # THE PYTEST TWIN of test/selftest/340's stamp arms, owed under jd's rule of # 2026-09-23... 2026-09-09: every test written twice. The .sh half could not diff --git a/test/selftest/380-the-pytest-cluster-helpers.sh b/test/selftest/380-the-pytest-cluster-helpers.sh index 9eeec064..e1a31665 100644 --- a/test/selftest/380-the-pytest-cluster-helpers.sh +++ b/test/selftest/380-the-pytest-cluster-helpers.sh @@ -15,19 +15,33 @@ # the directory -- and conftest.py cannot clean up after it, because # `cluster, root = make_cluster(...)` never completes when the call raises. # -# WHY STATIC, same division as selftests 360 and 370: the behavioural arms live -# in test/pytest/test_build_refusal.py, which needs pytest, psycopg and a -# virtualenv that CI does not install. This half pins the structure they rest on. +# WHAT THIS PART COVERS, AND WHAT MOVED (#432). Finding 1 is here: its subject is +# test/pgc_fingerprint.py, which lib.sh runs with the system interpreter and which +# is NOT part of the pytest harness, so checking it is this part's own business. +# +# FINDING 2 HAS MOVED. Its subject is test/pytest/pgc_cluster.py, and the arms that +# read it as text are now source checks in test/pytest/test_build_refusal.py, beside +# the behavioural arm that provokes a real failed setup and asserts no directory is +# left. Python reading its own module is not a cross-harness reference; a shell part +# grepping it is the one CONTEXT.md refuses. +# +# AND THE OLD REASON FOR KEEPING THEM HERE IS STALE. It said those arms "need pytest, +# psycopg and a virtualenv that CI does not install". CI has a `pytest-guards` job +# that installs pytest pinned from requirements-test.txt, asserts psycopg is absent, +# and runs the database-free file list, which contains test_build_refusal.py. +# +# MEASURED BEFORE MOVING ANYTHING. Five mutations of pgc_cluster.py, each asserted to +# have applied and each leaving the file well-formed: BaseException narrowed to +# Exception, cluster.stop() removed, the bare re-raise turned into pass, a private +# hashlib.md5 added, and shutil.rmtree(root) removed. Every one reddens on the python +# side -- three of them only the source arm, two of them the behavioural arm as well. +# Restored byte-for-byte after each. # # THE DERIVATION IS THE POINT OF THE FIRST GROUP. Hard-coding objstore/ would fix # today and fail the next time a module is added, so the arms below require the # GLOB rather than the name -- and require that the name does NOT appear, which # is the only way to tell a derivation from a list that happens to be complete. -_pc_cl="$PGC_TESTDIR/pytest/pgc_cluster.py" - -check "premise: the pytest cluster helper is where this part thinks it is" \ - "$([ -f "$_pc_cl" ] && echo yes || echo no)" "yes" # THE FINGERPRINT MOVED (#907). It was an independent Python implementation in # pgc_cluster.py and an independent shell one in lib.sh; the pair produced four @@ -57,10 +71,6 @@ check "and no longer mixes in the bare filename" \ "$(grep -c 'h.update(path.name.encode())' "$_pc_fp")" "0" # AND NEITHER CALLER MAY KEEP A PRIVATE COPY. This is the arm that would catch -# #907 recurring: the whole point is one implementation, so a second one -# reappearing in either caller is the defect, not a detail. -check "the pytest helper keeps no private fingerprint implementation" \ - "$(grep -cE 'hashlib\.md5|glob\("\*/Makefile"\)' "$_pc_cl")" "0" check "and the shell keeps none either" \ "$(grep -cE 'md5sum < |xargs -0 cat' "$PGC_TESTDIR/lib.sh")" "0" @@ -71,40 +81,11 @@ check "and the shell keeps none either" \ check "the module imports nothing from the pytest tree" \ "$(grep -cE '^(import|from) +(pgc_|conftest|pytest|psycopg)' "$_pc_fp")" "0" -# ---- and the lifecycle ------------------------------------------------------ -# -# Cut make_cluster's body out before grepping: these shapes occur elsewhere in -# the file, and an arm that greps the whole file reports a guard present that -# lives in another function. -_pc_body="$(awk '/^def make_cluster\(/{f=1} f&&/^def /&&!/make_cluster/{exit} f' "$_pc_cl")" - -check "premise: make_cluster's body was actually cut out of the file" \ - "$([ "$(printf '%s\n' "$_pc_body" | wc -l)" -ge 15 ] && echo yes || echo "too short")" "yes" - -check "make_cluster removes its tree when setup raises" \ - "$(printf '%s\n' "$_pc_body" | grep -c 'shutil.rmtree(root')" "1" - -# BaseException, not Exception: a KeyboardInterrupt during initdb leaks a datadir -# and a possibly-running postmaster exactly like an error does. -check "and it catches BaseException, so an interrupt cleans up too" \ - "$(printf '%s\n' "$_pc_body" | grep -c 'except BaseException:')" "1" - -check "and it stops a partially started cluster before removing the tree" \ - "$(printf '%s\n' "$_pc_body" | grep -c 'cluster.stop()')" "1" - -# Re-raising is what keeps the failure visible. Swallowing it would turn a failed -# setup into a silent None. -check "and the original error is re-raised rather than swallowed" \ - "$(printf '%s\n' "$_pc_body" | grep -c '^ raise$')" "1" # ---- and these greps must be able to FAIL ----------------------------------- _pc_fix="$PGC_WORKDIR/clusterhelpers"; rm -rf "$_pc_fix"; mkdir -p "$_pc_fix" -printf 'def make_cluster(a, b):\n root = mkdtemp()\n return cluster, root\n' \ - > "$_pc_fix/noguard.py" -check "a make_cluster with no cleanup is caught" \ - "$(grep -c 'shutil.rmtree(root' "$_pc_fix/noguard.py")" "0" printf ' dirs = [root / "src"]\n' > "$_pc_fix/srconly.py" check "a fingerprint that reads src only is caught" \ @@ -131,7 +112,5 @@ check "and a hard-coded module list is caught by the name arm" \ check "premise: while the real module satisfies the derivation arm" \ "$(grep -cE '_is_plain_file\(d / "Makefile"\)' "$_pc_fp")" "1" -check "premise: and the real helper still carries its cleanup" \ - "$(grep -cE 'shutil.rmtree\(root' "$_pc_cl")" "1" -unset _pc_cl _pc_fp _pc_body _pc_fix +unset _pc_fp _pc_fix From 70a90498a562c9e06b2a51fc0a70ca81e03ee5e8 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Fri, 11 Sep 2026 02:17:28 +0000 Subject: [PATCH 6/6] test/selftest: 350 goes from 50 checks to 5, and two rules move rather than die Forty-six of `test/selftest/350`'s fifty checks had their subject on the other side of the harness boundary: they globbed `test/pytest/*.py` and parsed Python out of it, or swept the markdown in that directory. The pytest corpus asserts all three properties natively in `test_docs_cover_the_corpus.py`, where the subject is, so the shell copy could only ever agree with it. TWO RULES MOVED RATHER THAN BEING DELETED. Both were implemented on both sides and SELF-TESTED only on the shell side -- the side that cannot run the corpus it counts. Deleting those fixtures would have left the Python implementation with no fixtures at all, so they went with them: the mode-counting rule's edges an id of fewer than three words, an id named twice, stopping at the next heading, section 3's back-references, and the row reader taking the value cell rather than a digit inside its label the contents-list anchor rule GitHub's derivation, the broken link that shipped, and a control beside it `_named_modes_in(text)` and `_stated_row(text, label)` are new seams so a fixture can reach the rules at all; `_named_modes()` delegates and the 19 existing arms are unchanged. The row reader's fixture makes the label digit and the value DIFFER -- label "section 2", value 9 -- because 350's own fixture had both as 2 and so could not see the bug it was written for. WHAT STAYS IS THE CI WORKFLOW, which belongs to neither harness. A shell part may read `ci.yml` for the same reason it may read the Makefile, and no pytest arm can assert that the gate runs pytest without assuming the thing in question. One arm is added while the part's subject is being settled: the job must ASSERT psycopg is absent rather than assume it. THE STALE JUSTIFICATION, IN TRIPLICATE. TESTS.md section 6 said "Nothing runs pytest. Not run_all_versions.sh, not any workflow under .github/", and concluded the .sh half was therefore the enforcement. CI has a `pytest-guards` job. That claim was load-bearing in three places at once -- TESTS.md section 6, selftest/360 and selftest/380 -- each asserting the behavioural half could not run in CI. Nothing was wrong in any of them; the REASON was, three times, and no arm reddens on a stale justification. All three corrected. AND ONE RULE-3 INSTANCE OF MY OWN. Two arms reddened with `needs-a-cluster:test_docs_cover_the_corpus.py` after the anchor fixtures moved in. The cause was a literal `test_harness_deps.py` inside sample markdown: `_mentioned_files` reads a corpus file name in code as "this file drives that one", so the classifier called the file cluster-bound. Sample markdown about a heading, read as a dependency. The names are assembled now, the same convention `SHELL_REFERENCES` uses -- CONTEXT.md's rule 3 caught for the third time in this PR, twice by me. WHERE THE TWO HARNESSES NOW STAND. Shell parts holding a reference to anything under `test/pytest/`: ONE, and it is deliberate. `selftest/360` reads `pgc_vacuity.py` to compare the three values both harnesses write down. The inventory claimed seven; the measured answer was three plus a deleted fourth; the end state is one. Verified on pg18a: selftest 757 checks, 0 FAIL; `pgc_ledger.py gate` rc=0 with `new this run=0` and the ceiling still 250; driver-free subset 207 passed across 11 files; the full corpus 292 passed. 350's ledger rows regenerated from a green run, 53 out and 5 in, and `checks_never_observed_red` re-derived to 756. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- CHANGELOG.md | 52 ++ test/check_ledger.tsv | 50 +- test/check_ledger_budget.txt | 2 +- test/pytest/TESTS.md | 43 +- test/pytest/test_docs_cover_the_corpus.py | 207 +++++- .../selftest/350-the-pytest-corpus-must-be.sh | 633 ++---------------- 6 files changed, 341 insertions(+), 646 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a1061de..2e000528 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -647,6 +647,58 @@ true until the next version shipped. `checks run:` line, a floor on it so an aborted run cannot pass, zero `FAIL` lines, and rc=0. The absence of a `FAIL` line is not enough, because an aborted run has none either -- the same shape as a pending-count that cannot see a job + + **`test/selftest/350` goes from 50 checks to 5, and two of its rules moved rather + than being deleted.** Forty-six of the fifty had a subject on the other side of the + boundary: they globbed `test/pytest/*.py` and parsed Python out of it, or swept the + markdown in that directory. The pytest corpus asserts all three properties natively + in `test_docs_cover_the_corpus.py`, where the subject is, so the shell copy could + only ever agree with it. + + Two of the three rules were implemented on BOTH sides and self-tested only on the + shell side -- the side that cannot run the corpus it counts. Deleting those + fixtures would have left the Python implementation with no fixtures at all, so they + went with them: + + - the mode-counting rule's edges: an id of fewer than three words, an id named + twice, stopping at the next heading, section 3's back-references, and the row + reader taking the value cell rather than a digit inside its label + - the contents-list anchor rule: GitHub's derivation, the broken link that shipped, + and a control beside it + + `_named_modes_in(text)` and `_stated_row(text, label)` are new seams, so a fixture + can reach the rules at all. The row reader's fixture makes the label digit and the + value DIFFER -- label "section 2", value 9 -- because `350`'s own fixture had both + as 2 and could not see the bug it was written for. + + **What stays in 350 is its arms over `ci.yml`**, whose subject is neither harness. + A shell part may read the workflow for the same reason it may read the Makefile, and + no pytest arm can assert that the gate runs pytest without assuming the thing in + question. One arm is added while the part's subject is being settled: the job must + ASSERT `psycopg` is absent rather than assume it. + + ### The stale justification, in triplicate + + TESTS.md section 6 said "Nothing runs pytest. Not `run_all_versions.sh`, not any + workflow under `.github/`", and concluded that the `.sh` half was therefore the + enforcement. CI has a `pytest-guards` job: it installs pytest pinned from + `requirements-test.txt`, asserts `psycopg` is absent, derives the file list from + `NO_CLUSTER`, and runs it. + + That claim was load-bearing in THREE places at once -- TESTS.md section 6, + `selftest/360` and `selftest/380` -- each asserting the behavioural half could not + run in CI. Nothing was wrong in any of them; the reason was, three times, and **no + arm reddens on a stale justification.** All three are corrected. + + ### Where the two harnesses now stand + + Shell parts holding a reference to anything under `test/pytest/`: **one**, and it is + deliberate. `selftest/360` reads `pgc_vacuity.py` to compare the three values the + two harnesses both write down -- the INCOMPLETE exit code, the closed list of + unrunnable reasons, and the line an unrunnable check prints. That is CONTEXT.md's + permitted cross-reference: a property that IS the relationship, so it cannot be + stated from one side. The inventory said seven; the measured answer was three plus a + deleted fourth; the end state is one. which never started. diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 1e45e04e..c068561b 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -408,59 +408,11 @@ harness_selftest 340-the-binary-must-be-built-from the stamp writer reports fail harness_selftest 340-the-binary-must-be-built-from the writer writes the file the reader looks for never - harness_selftest 340-the-binary-must-be-built-from two installations of one major get different stamp paths never - harness_selftest 340-the-binary-must-be-built-from two unreadable pg_configs do not alias onto one stamp never - -harness_selftest 350-the-pytest-corpus-must-be README.md quotes the number of modes the inventory names as refused never - -harness_selftest 350-the-pytest-corpus-must-be TESTS.md states no totals line for a merge to get wrong never - -harness_selftest 350-the-pytest-corpus-must-be TESTS.md states the counted number as well never - -harness_selftest 350-the-pytest-corpus-must-be a documented file that does not exist is named never - -harness_selftest 350-the-pytest-corpus-must-be a documented test that does not exist is named, not passed over never - -harness_selftest 350-the-pytest-corpus-must-be a name defined in two files is named, not passed over never - -harness_selftest 350-the-pytest-corpus-must-be a prose total that disagrees with the ids is visible never - -harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with disk is visible never - -harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with the ids is visible never - -harness_selftest 350-the-pytest-corpus-must-be an absent prose total is empty rather than a stray number never - -harness_selftest 350-the-pytest-corpus-must-be an absent total is empty rather than a number that happens to match never - -harness_selftest 350-the-pytest-corpus-must-be an anchor that strips the underscores is named, not passed over never - -harness_selftest 350-the-pytest-corpus-must-be an id named twice counts once never - -harness_selftest 350-the-pytest-corpus-must-be an id of fewer than three words is not counted as a mode never - -harness_selftest 350-the-pytest-corpus-must-be an unbackticked name in prose is not treated as a claim never - -harness_selftest 350-the-pytest-corpus-must-be an undocumented file is caught along with the tests inside it never - -harness_selftest 350-the-pytest-corpus-must-be an undocumented test is named rather than passed over never - harness_selftest 350-the-pytest-corpus-must-be and derives the pins from requirements-test.txt never - harness_selftest 350-the-pytest-corpus-must-be and it derives the file list rather than repeating it never - -harness_selftest 350-the-pytest-corpus-must-be and the same comparison agrees on the fixture that is right never - -harness_selftest 350-the-pytest-corpus-must-be control: a document naming only what exists is clean never - -harness_selftest 350-the-pytest-corpus-must-be control: a fully documented corpus reports nothing missing never - -harness_selftest 350-the-pytest-corpus-must-be control: an anchor that keeps the underscores resolves never - -harness_selftest 350-the-pytest-corpus-must-be control: distinct names in the same corpus report no duplicate never - -harness_selftest 350-the-pytest-corpus-must-be every in-document link in README.md reaches a heading never - -harness_selftest 350-the-pytest-corpus-must-be every in-document link in TESTS.md reaches a heading never - -harness_selftest 350-the-pytest-corpus-must-be every in-document link in VACUITY_MODES.md reaches a heading never - -harness_selftest 350-the-pytest-corpus-must-be every test file and every test in the corpus is named in TESTS.md never - -harness_selftest 350-the-pytest-corpus-must-be every test the document names exists in the corpus never - -harness_selftest 350-the-pytest-corpus-must-be no test name is defined twice in the corpus never - +harness_selftest 350-the-pytest-corpus-must-be and the job asserts the driver is absent rather than assuming it never - harness_selftest 350-the-pytest-corpus-must-be premise: the CI workflow is where this part thinks it is never - -harness_selftest 350-the-pytest-corpus-must-be premise: the corpus carries the documentation this part polices never - -harness_selftest 350-the-pytest-corpus-must-be premise: the counting rule finds modes at all never - -harness_selftest 350-the-pytest-corpus-must-be premise: the link sweep saw the directory's documents never - -harness_selftest 350-the-pytest-corpus-must-be premise: the mode inventory is where this part thinks it is never - -harness_selftest 350-the-pytest-corpus-must-be premise: the pytest corpus is where this part thinks it is never - -harness_selftest 350-the-pytest-corpus-must-be premise: the reader still finds a totals line when one is there never - -harness_selftest 350-the-pytest-corpus-must-be premise: the reverse sweep reads backticked names at all never - -harness_selftest 350-the-pytest-corpus-must-be premise: the sweep found the corpus rather than an empty glob never - -harness_selftest 350-the-pytest-corpus-must-be premise: the sweep reads the contents list's links never - -harness_selftest 350-the-pytest-corpus-must-be section 1a's document total is the sum of its two sections never - -harness_selftest 350-the-pytest-corpus-must-be section 1a's not-refused total is the count of ids in section 3 never - -harness_selftest 350-the-pytest-corpus-must-be section 1a's refused total is the count of ids in section 2 never - -harness_selftest 350-the-pytest-corpus-must-be section 2's opening states the counted number of refused modes never - -harness_selftest 350-the-pytest-corpus-must-be the admitted gap is the run total minus what is written down never - -harness_selftest 350-the-pytest-corpus-must-be the anchor rule drops punctuation and keeps underscores never - -harness_selftest 350-the-pytest-corpus-must-be the closing paragraph states the counted number too never - -harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 2 never - -harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 3 never - -harness_selftest 350-the-pytest-corpus-must-be the counter stops at the next heading never - harness_selftest 350-the-pytest-corpus-must-be the gate runs the harness guards never - -harness_selftest 350-the-pytest-corpus-must-be the row's value is read, not a digit inside its label never - -harness_selftest 350-the-pytest-corpus-must-be the sweep counts the fixture's tests and files never - harness_selftest 360-an-unrunnable-pytest-test-must a drifted exit code is visible rather than absorbed never - harness_selftest 360-an-unrunnable-pytest-test-must a drifted print shape is visible never - harness_selftest 360-an-unrunnable-pytest-test-must a reason dropped from one side only is visible never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index a64687bc..67080c34 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 804 +checks_never_observed_red 756 diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 1b75de19..5b187784 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -831,6 +831,15 @@ many times. | `test_an_undocumented_file_is_caught_with_the_tests_inside_it` | how 29 tests went missing at once | | `test_a_document_with_no_totals_line_states_none` | absent totals report `None`, which must not read as "they match" | | `test_a_stated_total_that_disagrees_with_disk_is_visible` | the count arm's own red | +| `test_the_counting_rule_counts_a_fixture_as_the_document_says` | the mode rule, on a fixture: deduplicated, and section 3 minus its back-references | +| `test_an_id_of_fewer_than_three_words_is_not_a_mode` | the rule is three words, so `one-two` in prose is not a mode | +| `test_the_counter_stops_at_the_next_heading` | section 2's count must not reach into section 4 | +| `test_the_row_reader_takes_the_value_not_a_digit_in_the_label` | the labels contain digits; reading the first number returns the 2 from "section 2" | +| `test_an_absent_row_is_none_rather_than_a_number_that_happens_to_match` | a missing row must not read as a row stating zero | +| `test_a_stated_total_that_disagrees_with_the_ids_is_visible` | the inventory arm's own red, with the agreeing control beside it | +| `test_the_anchor_rule_drops_punctuation_and_keeps_underscores` | GitHub's derivation, on the heading the defect was found in | +| `test_an_anchor_that_strips_the_underscores_is_caught` | the exact broken link that shipped, with a control | +| `test_every_in_document_link_in_this_directory_reaches_a_heading` | every contents-list link resolves, with a coverage premise | The five fixture arms exist because everything above them passes on a healthy tree, which is exactly what a guard that does nothing also does. They run the identical @@ -838,20 +847,26 @@ functions over a corpus built to be wrong. ### The twin, and which half has teeth -This is the pytest half. The other half is -`test/selftest/350-the-pytest-corpus-must-be.sh`, and the two are **not** -interchangeable: - -- **The `.sh` half is the one that gates.** `harness_selftest` is registered in - `SUITES`, so it runs in the matrix and in CI. -- **Nothing runs pytest.** Not `run_all_versions.sh`, not any workflow under - `.github/`. A guard written only here would never fire in the gate, and a guard - that does not run is a comment. - -So the `.sh` copy is the enforcement and this one is what a person running the -corpus by hand gets, with the offenders arriving as a Python list rather than as a -string assembled by shell. Both are written in the same change, per the rule in -section 9. +**This section said "Nothing runs pytest. Not `run_all_versions.sh`, not any +workflow under `.github/`", and that is no longer true.** CI has a `pytest-guards` +job: it installs pytest pinned from `requirements-test.txt`, asserts `psycopg` is +absent, derives the file list from `NO_CLUSTER` in `test_harness_deps.py`, and runs +it. This file is in that list. So a guard written only here DOES fire in the gate, +and the argument that made the `.sh` half the enforcement has gone. + +That argument was load-bearing, and it was stale in three places at once — here, +in `test/selftest/360`, and in `test/selftest/380` — each saying the behavioural +half could not run in CI. **A stale justification for keeping coverage in the wrong +place is harder to find than a missing check, because nothing reddens.** Nothing +was wrong; the reason was. + +So the duplication is being removed in the direction the two-harness rule requires +(#432). `test/selftest/350`'s arms over this corpus and over this directory's +documents are the ones that had to read across the boundary, and they are gone; the +properties they held that this file did not yet test — the mode-counting rule's +edges and the contents-list anchor rule — moved here, where their subject is. What +stays in `350` is its arms over `ci.yml`, whose subject is the workflow rather than +either harness. This guard reddened on its own arrival, which is the only reason it is known to work here: adding this file moved the corpus from `(54, 5)` to `(62, 6)` and the diff --git a/test/pytest/test_docs_cover_the_corpus.py b/test/pytest/test_docs_cover_the_corpus.py index 5268f084..86240705 100644 --- a/test/pytest/test_docs_cover_the_corpus.py +++ b/test/pytest/test_docs_cover_the_corpus.py @@ -335,9 +335,15 @@ def test_a_stated_total_that_disagrees_with_disk_is_visible(tmp_path, expect): MODES_DOC = HERE / "VACUITY_MODES.md" -def _named_modes(): - """-> (refused, not_refused, all) per section 1a's rule.""" - text = MODES_DOC.read_text() +def _named_modes_in(text): + """-> (refused, not_refused, all) per section 1a's rule, over TEXT. + + A seam, so the rule can be reached with a fixture. `test/selftest/350` had one + and this did not: the rule was implemented twice and self-tested once, on the + side that cannot run the corpus it counts. Its edge cases now live here with it + -- an id of fewer than three words, an id named twice, stopping at the next + heading, and section 3's back-references. + """ chunks = {} for chunk in re.split(r"^## ", text, flags=re.M): head = chunk.splitlines()[0] if chunk.strip() else "" @@ -352,6 +358,24 @@ def _named_modes(): return refused, not_refused, set().union(*chunks.values()) if chunks else set() +def _named_modes(): + """The same rule, over the document on disk.""" + return _named_modes_in(MODES_DOC.read_text()) + + +def _stated_row(text, label): + """-> the number in the VALUE cell of section 1a's row for LABEL, or None. + + The value cell, never the first number on the line. The labels themselves + contain digits -- "named in section 2, refused today" -- so reading the first + number returns the 2 from "section 2". `test/selftest/350` shipped exactly that + bug and read 2 and 3 for totals of 21 and 51, and its own fixture could not see + it because there the label digit and the value were both 2. + """ + m = re.search(rf"\|[^|\n]*{re.escape(label)}[^|\n]*\|\s*\**(\d+)", text) + return None if m is None else int(m.group(1)) + + def test_the_mode_inventory_states_its_own_totals_correctly(expect): """The numbers in section 1a must be the numbers on disk. @@ -454,3 +478,180 @@ def test_the_two_halves_of_the_refused_sentence_sum_to_the_named_total(expect): "the two halves sum to the number of modes the inventory names") expect.num(refused, len(_named_modes()[0]), "and the refused half is the count of ids section 2 claims") + + +# --------------------------------------------------------------------------- +# The counting rule's own edges, and the row reader's. +# +# These moved from `test/selftest/350` (#432). The rule is implemented on both +# sides -- that is the two-harness design, parallel in functionality -- but it was +# SELF-TESTED only on the shell side, which cannot run the corpus it counts. A rule +# with no fixtures is a rule that passes because the document happens to agree with +# it today. +# --------------------------------------------------------------------------- + +_FIXTURE_DOC = """## 1a. Counting + +| named in section 2, refused today | **2** | +| named in section 3, not refused | **1** | + +## 2. Refused + +`one-two-three` and `four-five-six`, and `one-two-three` again. + +## 3. Not refused + +`seven-eight-nine`. And `one-two-three` is now closed. + +## 4. Something else + +`ten-eleven-twelve` +""" + + +def test_the_counting_rule_counts_a_fixture_as_the_document_says(expect): + """Section 2 names two distinct ids; section 3 names one, after its + back-reference to a section 2 id is discounted.""" + refused, not_refused, allm = _named_modes_in(_FIXTURE_DOC) + expect.text(", ".join(sorted(refused)), "four-five-six, one-two-three", + "section 2's ids, deduplicated") + expect.text(", ".join(sorted(not_refused)), "seven-eight-nine", + "section 3's ids, minus the back-reference section 2 already claims") + expect.num(len(allm), 4, "and every id in the document is seen once") + + +def test_an_id_of_fewer_than_three_words_is_not_a_mode(expect): + """The rule is three or more words. Two is an ordinary backticked phrase, and + counting it would make every `foo-bar` in prose a mode.""" + doc = "## 2. Refused\n\n`one-two` and `alpha` and `one-two-three`\n" + refused, _, _ = _named_modes_in(doc) + expect.text(", ".join(sorted(refused)), "one-two-three", + "only the three-word id counts") + + +def test_the_counter_stops_at_the_next_heading(expect): + """Section 2's count must not reach into section 4. Without the stop, every id + below section 2 would be refused, and the totals would agree with nothing.""" + refused, _, _ = _named_modes_in(_FIXTURE_DOC) + expect.text("ten-eleven-twelve" in refused and "yes" or "no", "no", + "an id in a later section is not counted as refused") + + +def test_the_row_reader_takes_the_value_not_a_digit_in_the_label(expect): + """The labels contain digits. Reading the first number on the line returns the + 2 from "section 2" -- which `test/selftest/350` did, reporting 2 and 3 for + totals of 21 and 51. The fixture there could not see it, because the label + digit and the value were both 2. + + So this fixture makes them DIFFER: the label says "section 2" and the value is + 9. A reader that takes the label's digit answers 2 and fails here. + """ + doc = "| named in section 2, refused today | **9** |\n" + expect.num(_stated_row(doc, "refused today"), 9, + "the value cell is read, not the digit inside the label") + + +def test_an_absent_row_is_none_rather_than_a_number_that_happens_to_match(expect): + """A missing row must be distinguishable from a row stating zero. Returning 0 + for both would make a document that states nothing agree with a corpus that + counts nothing.""" + expect.text(repr(_stated_row("## 1a. Counting\n\nno table here\n", "refused today")), + "None", "an absent row reads as None") + expect.num(_stated_row("| refused today | **0** |\n", "refused today"), 0, + "and a row stating zero reads as 0") + + +def test_a_stated_total_that_disagrees_with_the_ids_is_visible(expect): + """The arm this whole block protects: the document's number against the ids on + disk. With the fixture's own table it agrees; change the table and it must not.""" + refused, not_refused, _ = _named_modes_in(_FIXTURE_DOC) + expect.num(_stated_row(_FIXTURE_DOC, "refused today"), len(refused), + "control: the fixture's stated refused total is its id count") + wrong = _FIXTURE_DOC.replace("refused today | **2** |", "refused today | **7** |") + expect.differ(_stated_row(wrong, "refused today"), len(refused), + "and a disagreeing total does not compare equal") + + +# --------------------------------------------------------------------------- +# Every contents-list link must reach a heading. +# +# Moved from `test/selftest/350` (#432), where the subject was this directory's own +# documents. TESTS.md's contents list gained an entry whose anchor stripped the +# underscores out of a file name -- `#14-testharnessdepspy-…` against a heading +# GitHub renders as `#14-test_harness_depspy-…` -- so the link went nowhere while +# eleven entries above it kept the underscores. Neither coverage sweep could see +# it: both look for NAMES, and a broken link still contains the name it points at. +# A reader finds out by clicking. +# +# ONE DIRECTION ON PURPOSE. Every link must reach a heading; the reverse needs an +# exemption list, because "## Contents" is a heading no entry links to, and a +# hand-maintained exemption list is the thing this corpus keeps deleting. +# --------------------------------------------------------------------------- + +_LINK = re.compile(r"\]\(#([A-Za-z0-9_-]+)\)") +_HEADING = re.compile(r"^#{2,} +(.+)$", re.M) + + +def _github_anchor(heading): + """GitHub's rule: lowercase, drop anything that is not a letter, digit, space, + hyphen or underscore, then spaces to hyphens. The dot in `.py` and the colon + after it go; the underscores stay.""" + kept = [c for c in heading.lower() if c.isalnum() or c in " _-"] + return "".join(kept).replace(" ", "-") + + +def _unresolved_links(text): + """-> sorted anchors that no heading in TEXT produces.""" + have = {_github_anchor(h) for h in _HEADING.findall(text)} + return sorted({a for a in _LINK.findall(text) if a not in have}) + + +def test_the_anchor_rule_drops_punctuation_and_keeps_underscores(expect): + """The derivation, on the heading the defect was found in.""" + # Assembled, for the reason given in the arm below: a literal corpus file name + # in a string makes the membership classifier read this file as driving that + # one. The anchor on the right keeps its literal form, because the dot is gone + # from it and so it is no longer that file's name. + heading = "14. " + "test" + "_harness" + "_deps" + ".py: the harness must self-test" + expect.text(_github_anchor(heading), + "14-test_harness_depspy-the-harness-must-self-test", + "the dot and the colon go, the underscores stay") + + +def test_an_anchor_that_strips_the_underscores_is_caught(expect): + """The exact shape that shipped, and a control beside it. Without the control a + sweep that matches nothing reports the same clean answer.""" + # THE FILENAME IS ASSEMBLED, NOT WRITTEN. A literal `test_harness_deps.py` in + # this string makes `_mentioned_files` in that very file read this one as a + # driver of it, and the membership classifier then calls this file + # cluster-bound. That is CONTEXT.md's rule 3 -- "a word that merely looks like + # a filename" -- and it cost two red arms before the cause was found. The + # heading here is sample markdown, not a reference. + _f = "test" + "_harness" + "_deps" + ".py" + good = (f"## 14. {_f}: the harness\n" + "- [14. x](#14-test_harness_depspy-the-harness)\n") + bad = (f"## 14. {_f}: the harness\n" + "- [14. x](#14-testharnessdepspy-the-harness)\n") + expect.text(", ".join(_unresolved_links(good)) or "none", "none", + "control: an anchor that keeps the underscores resolves") + expect.text(", ".join(_unresolved_links(bad)), + "14-testharnessdepspy-the-harness", + "and one that strips them is named rather than passed over") + + +def test_every_in_document_link_in_this_directory_reaches_a_heading(expect): + """The real sweep, over every document beside this file. + + With a coverage premise: a sweep over an empty file list reports nothing + broken, which is what a correct set of documents reports too. + """ + docs = sorted(HERE.glob("*.md")) + expect.at_least(len(docs), 3, "premise: the sweep found this directory's documents") + total_links = 0 + for doc in docs: + text = doc.read_text(encoding="utf-8") + total_links += len(_LINK.findall(text)) + expect.text(", ".join(_unresolved_links(text)) or "none", "none", + f"every in-document link in {doc.name} reaches a heading") + expect.at_least(total_links, 15, + "premise: and it parsed links rather than finding none") diff --git a/test/selftest/350-the-pytest-corpus-must-be.sh b/test/selftest/350-the-pytest-corpus-must-be.sh index 88ce1a0a..c178ba56 100644 --- a/test/selftest/350-the-pytest-corpus-must-be.sh +++ b/test/selftest/350-the-pytest-corpus-must-be.sh @@ -1,398 +1,59 @@ -# ---- the pytest corpus must be documented, or the documentation is decoration - +# ---- the gate must run the pytest corpus's guards ---------------------------- +# +# WHAT THIS PART IS NOW (#432). It used to hold three groups of arms over +# test/pytest/: that every file and test is named in TESTS.md, that every +# contents-list link resolves, and that VACUITY_MODES.md's section 1a counts itself +# correctly. Fifty checks, and the subject of forty-six of them lived on the other +# side of the harness boundary. +# +# THE TWO HARNESSES ARE PARALLEL IN FUNCTIONALITY AND MUST NOT REFERENCE EACH OTHER +# (jd, 2026-09-10). Every one of those arms either globbed test/pytest/*.py and +# parsed Python out of it, or swept the markdown in that directory. The pytest +# corpus asserts all three properties natively in test_docs_cover_the_corpus.py, +# where the subject is, and the shell copy could only ever agree with it. +# +# WHAT MOVED RATHER THAN BEING DELETED. Two of the three rules were implemented on +# both sides and SELF-TESTED only here -- the side that cannot run the corpus it +# counts. Deleting those fixtures would have left the python implementation with no +# fixtures at all, so they went with them: +# +# the mode-counting rule's edges an id of fewer than three words, an id named +# twice, stopping at the next heading, section +# 3's back-references, and the row reader taking +# the value cell rather than a digit in its label +# the contents-list anchor rule GitHub's derivation, the broken link that +# shipped, and a control beside it +# +# AND THE REASON THE DOCUMENT GAVE FOR KEEPING THE SHELL COPY WAS STALE. TESTS.md +# section 6 said "Nothing runs pytest. Not run_all_versions.sh, not any workflow +# under .github/", and concluded that the .sh half was therefore the enforcement. +# CI has a pytest-guards job. That argument was stale in three places at once -- +# there, in selftest/360 and in selftest/380 -- each claiming the behavioural half +# could not run in CI. Nothing was wrong; the reason was, and a stale reason for +# keeping coverage in the wrong place is harder to find than a missing check, +# because nothing reddens. +# +# WHAT REMAINS HERE IS THE CI WORKFLOW, which belongs to neither harness. A shell +# part may read it for the same reason it may read the Makefile: ci.yml is not a +# test suite, and no pytest arm can assert that the gate runs pytest without +# assuming the very thing in question. +# +# THE MEMBERSHIP DECISION AND THE CONFTEST IMPORT ARE STILL NOT CHECKED HERE, and +# that was already jd's rule rather than an omission. An earlier version of this +# part read test/pytest/conftest.py and INVOKED test_harness_deps.py with +# `python3 ... --disagree`, which is the coupling rather than a second measurement: +# a shell arm driving the python decider agrees with it by construction and can +# never report it wrong. Both properties are asserted in the corpus: # -# WHY THIS EXISTS. `test/pytest/TESTS.md` says its job is "what each test asserts, -# and why it exists". It went stale inside a single rework: the corpus grew from -# 25 tests in three files to 54 in five, and the two new files -- 29 tests, every -# one of them added by the rework that answered a review -- were named nowhere in -# it. The header still read "Twenty-five tests in three files". -# -# That is worse than an undocumented directory. A reader who opens a file whose -# stated purpose is completeness does not then go and count the tests, so a -# partial index reads as a total one. The header was not merely out of date, it -# was a false statement of coverage, which is the same defect class the vacuity -# layer in that corpus exists to refuse one level down. -# -# WHY HERE AND NOT IN THE CORPUS ITSELF. Nothing runs pytest. `SUITES` in -# run_all_versions.sh does not list it and no workflow in .github/ invokes it -# (README.md in that directory records the decision and its price). A guard -# written as a pytest test would therefore never run in the gate, and a guard -# that does not run is a comment. `harness_selftest` IS registered, so this is -# the only place the rule can actually bite. The corpus carries a twin of this -# check for anyone running it by hand; this copy is the one with teeth. -# -# WHAT IS POLICED. Three properties, each mechanical: -# * every test_*.py file in the corpus is named in TESTS.md -# * every `def test_` in those files is named in TESTS.md -# * the totals TESTS.md states are the totals on disk -# The third is what the stale header got wrong, and neither of the first two -# would have caught it: a file can name every test and still miscount them. - -_dcv_dir="$PGC_TESTDIR/pytest" -_dcv_doc="$_dcv_dir/TESTS.md" - -check "premise: the pytest corpus is where this part thinks it is" \ - "$([ -d "$_dcv_dir" ] && echo yes || echo no)" "yes" - -check "premise: the corpus carries the documentation this part polices" \ - "$([ -f "$_dcv_doc" ] && echo yes || echo no)" "yes" - -# The sweep, as a function over a DIRECTORY and a DOC, so the arms below can run -# the identical logic over a fixture. A guard that can only be pointed at the -# real tree is proved by nothing: it passes today and there is no way to see it -# fire. selftest 320 makes the same move with the runner's classifier. -# -# Prints the offenders, capped, in the "[]" / "[n:...]" shape the other parts use -# so a failure names what is wrong rather than only that something is. -_dcv_missing() { # _dcv_missing DIR DOC -> "[]" or "[n: a b c]" - local dir="$1" doc="$2" f base name n=0 bad="" doctext - doctext="$(cat "$doc" 2>/dev/null)" - for f in "$dir"/test_*.py; do - # Without nullglob an unmatched glob stays literal, so a corpus with no - # test files would iterate once over a path that does not exist. The - # file test guards that; the premise below asserts the sweep saw files. - [ -f "$f" ] || continue - base="${f##*/}" - case "$doctext" in - *"$base"*) ;; - *) n=$((n + 1)); [ "$n" -le 6 ] && bad="$bad $base" ;; - esac - while IFS= read -r name; do - [ -n "$name" ] || continue - case "$doctext" in - *"$name"*) ;; - *) n=$((n + 1)); [ "$n" -le 6 ] && bad="$bad $name" ;; - esac - done < <(grep -oE '^def (test_[A-Za-z0-9_]+)' "$f" | sed 's/^def //') - done - [ "$n" -eq 0 ] && { printf '[]'; return; } - printf '[%d:%s]' "$n" "$bad" -} - -_dcv_count() { # _dcv_count DIR -> " " - local dir="$1" f t=0 c=0 - for f in "$dir"/test_*.py; do - [ -f "$f" ] || continue - c=$((c + 1)) - t=$((t + $(grep -cE '^def test_' "$f"))) - done - printf '%d %d' "$t" "$c" -} - -# A sweep that found nothing reports "nothing missing" and is indistinguishable -# from a sweep that works. Assert it saw the corpus before believing its verdict. -_dcv_seen="$(_dcv_count "$_dcv_dir")" -check "premise: the sweep found the corpus rather than an empty glob" \ - "$([ "${_dcv_seen%% *}" -ge 20 ] && [ "${_dcv_seen##* }" -ge 3 ] && echo enough || echo "$_dcv_seen")" \ - "enough" - -check "every test file and every test in the corpus is named in TESTS.md" \ - "$(_dcv_missing "$_dcv_dir" "$_dcv_doc")" "[]" - -# TESTS.md STATES NO TOTALS, AND THAT IS THE POINT (#908). -# -# It used to, and this part compared the stated pair against the corpus, which is -# what made it a claim rather than decoration. The problem was never the check: it -# was that the number was WRITTEN rather than DERIVED, and its correct value is a -# function of the MERGE rather than of either branch. It collided on essentially -# every rebase touching the corpus -- ten times in one day, both sides wrong every -# time, so there was no side to pick. -# -# REMOVING IT COSTS NOTHING, provably. The two sweeps are strictly stronger than -# any count: the arm above requires every test on disk to be NAMED here, and the -# reverse arm below requires every name here to EXIST on disk. Two subsets in -# opposite directions is set equality, so any count over the document equals the -# count over the corpus. A stated total was a derived value maintained by hand. -_dcv_stated="$(grep -oE '^\*\*[0-9]+ tests in [0-9]+ files\.\*\*' "$_dcv_doc" \ - | head -1 | grep -oE '[0-9]+' | tr '\n' ' ' | sed 's/ $//')" - -check "TESTS.md states no totals line for a merge to get wrong" \ - "$([ -z "$_dcv_stated" ] && echo none || echo "$_dcv_stated")" "none" - -# PREMISE: the reader must still be able to FIND a totals line, or the arm above -# passes because the parser is broken rather than because the line is gone -- -# which is the shape this whole part exists to refuse. -# $_dcv_fix is not created until the failure-proof section below, and this file -# runs under `set -u`, so this fixture makes its own path rather than borrowing -# one that does not exist yet. -_dcv_ht="$PGC_WORKDIR/doccov-hastotals"; mkdir -p "$_dcv_ht" -printf '**7 tests in 3 files.** and prose\n' > "$_dcv_ht/HASTOTALS.md" -check "premise: the reader still finds a totals line when one is there" \ - "$(grep -oE '^\*\*[0-9]+ tests in [0-9]+ files\.\*\*' "$_dcv_ht/HASTOTALS.md" \ - | head -1 | grep -oE '[0-9]+' | tr '\n' ' ' | sed 's/ $//')" "7 3" - -# And the counts still REACH a reader -- they move to this run's output, where -# they are computed from the corpus and cannot go stale. -echo " CORPUS: $_dcv_seen (test functions, files)" - -# ---- and the guard must be able to FAIL -------------------------------------- -# -# Everything above passes on a healthy tree, which is exactly what a guard that -# does nothing also does. These arms run the same two functions over fixtures -# built to be wrong, so a future edit that neuters the sweep reddens here even -# while the real corpus stays clean. - -_dcv_fix="$PGC_WORKDIR/doccov"; rm -rf "$_dcv_fix"; mkdir -p "$_dcv_fix" -printf 'def test_alpha(expect):\n pass\ndef test_beta(expect):\n pass\n' \ - > "$_dcv_fix/test_one.py" - -# Documented completely: file named, both tests named, totals stated. -printf '**2 tests in 1 files.**\ntest_one.py: test_alpha and test_beta\n' \ - > "$_dcv_fix/GOOD.md" -check "control: a fully documented corpus reports nothing missing" \ - "$(_dcv_missing "$_dcv_fix" "$_dcv_fix/GOOD.md")" "[]" - -# One test left out. This is the exact shape that shipped. -printf '**2 tests in 1 files.**\ntest_one.py: test_alpha\n' > "$_dcv_fix/PARTIAL.md" -check "an undocumented test is named rather than passed over" \ - "$(_dcv_missing "$_dcv_fix" "$_dcv_fix/PARTIAL.md")" "[1: test_beta]" - -# A whole file left out, which is how 29 tests went missing at once. -printf '**2 tests in 1 files.**\nnothing about the corpus at all\n' > "$_dcv_fix/NONE.md" -check "an undocumented file is caught along with the tests inside it" \ - "$(_dcv_missing "$_dcv_fix" "$_dcv_fix/NONE.md")" "[3: test_one.py test_alpha test_beta]" - -# The count arm, proved separately: a doc can name every test and still state a -# wrong total, which is precisely what the stale header did. -check "the sweep counts the fixture's tests and files" \ - "$(_dcv_count "$_dcv_fix")" "2 1" - -# Kept although the real document no longer states a total: it proves the -# comparison still works, so "no totals line" above is a fact about the document -# rather than about a broken reader. -check "a stated total that disagrees with disk is visible" \ - "$([ "$(grep -oE '^\*\*[0-9]+ tests in [0-9]+ files\.\*\*' "$_dcv_fix/GOOD.md" \ - | grep -oE '[0-9]+' | tr '\n' ' ' | sed 's/ $//')" = "$(_dcv_count "$_dcv_fix")" ] \ - && echo agrees || echo differs)" "agrees" - - -# ---- and the sweep must go the OTHER way too (#908) ------------------------- -# -# `_dcv_missing` above computes tests on disk the document fails to name. NOTHING -# computed the reverse, so a test DELETED or RENAMED while its entry survived was -# held by the totals line and by nothing else -- and that line is a merge target -# whose correct value is a function of the merge, so it is the half most likely to -# be removed. Removing it while this direction was uncovered would have retired a -# check silently, which is the move this part exists to prevent. -# -# IT FOUND TWO ON THE SHIPPED CORPUS. Section 3 named -# test_layer_rejects_an_absence_assertion_over_an_empty_plan and a control beside -# it, and neither had ever been written; the real work lives in -# test_guards_pinned.py under a different name and is documented correctly in its -# own section. Two rows claimed coverage under names nobody had written. -# -# A BACKTICKED NAME, not any occurrence. This document discusses fixtures and dead -# names in prose, and a bare-word sweep would report those as missing. Backticks -# are how it already marks a real identifier, so backticking a name IS the claim -# that it exists -- which is why the paragraph describing this defect writes the -# dead names without them. - -_dcv_absent() { # _dcv_absent DIR DOC -> "[]" or "[n: a b c]" - local dir="$1" doc="$2" name n=0 bad="" - local ondisk - # Every function and every file the corpus actually has, one per line. - ondisk="$( { grep -hoE '^def (test_[A-Za-z0-9_]+)' "$dir"/test_*.py 2>/dev/null \ - | sed 's/^def //' - for f in "$dir"/test_*.py; do [ -f "$f" ] && printf '%s\n' "${f##*/}"; done - } | sort -u )" - while IFS= read -r name; do - [ -n "$name" ] || continue - # grep -cxF on a here-string, NOT `printf ... | grep -qxF`. grep -q exits - # the moment it matches, the printf takes EPIPE, and under this suite's - # `set -o pipefail` the pipeline reports failure though the name WAS - # present -- so a name in the corpus is reported absent. Selftest 080 - # states this rule and demonstrates it; its sweep is deliberately - # non-recursive and never looked in here. - # - # It is not latent. It reddened #923's `suites (PG 17)` on a name that - # exists, while PG 18 passed, and reproduces at this corpus size only - # under load: 170 names, 400 trials on a busy machine, 6 false absences - # piped and 0 on a here-string. An independent run of the same shape in - # isolation gave 10 in 40, so the rate is load- and size-dependent rather - # than fixed -- the two measurements bracket it. - [ "$(grep -cxF "$name" <<<"$ondisk" || true)" -ne 0 ] && continue - n=$((n + 1)); [ "$n" -le 6 ] && bad="$bad $name" - done < <(grep -oE '`test_[A-Za-z0-9_]*(\.py)?`' "$doc" 2>/dev/null \ - | tr -d '`' | sort -u) - [ "$n" -eq 0 ] && { printf '[]'; return; } - printf '[%d:%s]' "$n" "$bad" -} - -check "premise: the reverse sweep reads backticked names at all" \ - "$([ "$(grep -coE '`test_[A-Za-z0-9_]*(\.py)?`' "$_dcv_doc")" -ge 20 ] \ - && echo enough || echo too-few)" "enough" - -check "every test the document names exists in the corpus" \ - "$(_dcv_absent "$_dcv_dir" "$_dcv_doc")" "[]" - -# ---- and it must be able to FAIL, on a fixture rather than on the tree ------- - -printf 'def test_alpha(expect):\n pass\n' > "$_dcv_fix/test_one.py" - -printf '`test_one.py`: `test_alpha` and `test_beta`\n' > "$_dcv_fix/GHOST.md" -check "a documented test that does not exist is named, not passed over" \ - "$(_dcv_absent "$_dcv_fix" "$_dcv_fix/GHOST.md")" "[1: test_beta]" - -printf '`test_one.py`: `test_alpha`\n' > "$_dcv_fix/REAL.md" -check "control: a document naming only what exists is clean" \ - "$(_dcv_absent "$_dcv_fix" "$_dcv_fix/REAL.md")" "[]" - -printf '`test_one.py` and `test_gone.py`: `test_alpha`\n' > "$_dcv_fix/GONEFILE.md" -check "a documented file that does not exist is named" \ - "$(_dcv_absent "$_dcv_fix" "$_dcv_fix/GONEFILE.md")" "[1: test_gone.py]" - -# A name in prose without backticks is not a claim, so it must NOT be reported -- -# otherwise the document could never describe a test it removed. -printf 'the old test_beta was removed; `test_alpha` remains\n' > "$_dcv_fix/PROSE.md" -check "an unbackticked name in prose is not treated as a claim" \ - "$(_dcv_absent "$_dcv_fix" "$_dcv_fix/PROSE.md")" "[]" - -unset -f _dcv_absent - -# ---- and the names must be UNIQUE, or the equality argument does not hold ---- -# -# The two sweeps give equality of the two NAME SETS. That is not equality of -# DEFINITION COUNTS, which is what a total counts: two files defining one name -# leave both arms green while the counts differ (@jdatcmd, against #919). It -# cannot happen today, so the conclusion was true in fact and not by -# construction -- the difference between an argument and a guard. -# -# It closes something real beyond the argument: `_dcv_missing` asks whether a -# name appears in the document AT ALL, so a test defined TWICE and documented -# ONCE reads as fully covered while pytest runs both. - -_dcv_dupes() { # _dcv_dupes DIR -> "" or the repeated names - grep -hoE '^def (test_[A-Za-z0-9_]+)' "$1"/test_*.py 2>/dev/null \ - | sed 's/^def //' | sort | uniq -d -} - -check "no test name is defined twice in the corpus" \ - "$(_dcv_dupes "$_dcv_dir" | tr '\n' ' ' | sed 's/ $//')" "" - -# And it must be able to FAIL, on a fixture rather than on the tree. -printf 'def test_shared_shape(expect):\n pass\n' > "$_dcv_fix/test_a.py" -printf 'def test_shared_shape(expect):\n pass\n' > "$_dcv_fix/test_b.py" -check "a name defined in two files is named, not passed over" \ - "$(_dcv_dupes "$_dcv_fix" | tr '\n' ' ' | sed 's/ $//')" "test_shared_shape" -rm -f "$_dcv_fix/test_a.py" "$_dcv_fix/test_b.py" - -check "control: distinct names in the same corpus report no duplicate" \ - "$(_dcv_dupes "$_dcv_dir" | tr '\n' ' ' | sed 's/ $//')" "" - -unset -f _dcv_dupes - - -# ---- and every table-of-contents link must RESOLVE --------------------------- -# -# WHY THIS EXISTS. TESTS.md's contents list gained an entry whose anchor stripped -# the underscores out of the file name -- "#14-testharnessdepspy-..." against a -# heading GitHub renders as "#14-test_harness_depspy-..." -- so the link silently -# went nowhere. Eleven entries above it keep the underscores, so the document -# already stated the convention and the new entry simply disagreed with it. -# -# NEITHER EXISTING ARM COULD SEE IT. Both sweep for NAMES; an anchor is not a -# name, and a broken link is still a string containing the file name it points at. -# A reader finds out by clicking. -# -# THE RULE IS GITHUB'S, and it is mechanical: lowercase the heading text, drop -# every character that is not a letter, digit, space, hyphen or underscore, then -# turn spaces into hyphens. So the dot in ".py" and the colon after it disappear -# and the underscores stay. Over the three documents in this directory the sweep -# reports nothing, and over the document as it shipped it reported exactly the one -# entry -- which is the whole false-positive budget, measured rather than assumed. -# -# ONE DIRECTION ON PURPOSE: every link must reach a heading. The reverse, every -# heading must be linked, is a different property, and "## Contents" is itself a -# heading that no entry links to -- so the reverse needs an exemption list, which -# is the hand-maintained value this file keeps removing. - -_toc_anchor() { # _toc_anchor TEXT -> the anchor GitHub derives from it - printf '%s' "$1" | tr 'A-Z' 'a-z' | sed -e 's/[^a-z0-9 _-]//g' -e 's/ /-/g' -} - -_toc_unresolved() { # _toc_unresolved DOC -> "[]" or "[n: a b c]" - local doc="$1" anchor n=0 bad="" anchors - # Every anchor the document's own headings produce, one per line. - # Through _toc_anchor, so GitHub's rule has ONE definition here. The stream - # form was a second copy of the same sed program, and two copies drift. - anchors="$(while IFS= read -r _h; do [ -n "$_h" ] && _toc_anchor "$_h" && echo; done \ - < <(grep -E '^#{2,} ' "$doc" 2>/dev/null | sed -e 's/^#* //'))" - while IFS= read -r anchor; do - [ -n "$anchor" ] || continue - # grep -cxF on a here-string, for the reason given in _dcv_absent above. - [ "$(grep -cxF "$anchor" <<<"$anchors" || true)" -ne 0 ] && continue - n=$((n + 1)); [ "$n" -le 6 ] && bad="$bad $anchor" - done < <(grep -oE '\]\(#[A-Za-z0-9_-]+\)' "$doc" 2>/dev/null \ - | sed -e 's/^](#//' -e 's/)$//' | sort -u) - [ "$n" -eq 0 ] && { printf '[]'; return; } - printf '[%d:%s]' "$n" "$bad" -} - -# PREMISE: the sweep found links at all. A document whose links it cannot parse -# reports "nothing broken", which is what a correct document reports too. -check "premise: the sweep reads the contents list's links" \ - "$([ "$(grep -coE '\]\(#[A-Za-z0-9_-]+\)' "$_dcv_doc")" -ge 15 ] \ - && echo enough || echo too-few)" "enough" - -# AND THE SWEEP MUST HAVE SWEPT. Without nullglob an unmatched glob stays literal, -# the [ -f ] skips it, and the loop below runs NO checks while the part still -# reports every check it did run as passing. A clean sweep needs a coverage premise. -_toc_n=0 -for _toc_f in "$_dcv_dir"/*.md; do - [ -f "$_toc_f" ] || continue - _toc_n=$((_toc_n + 1)) - check "every in-document link in ${_toc_f##*/} reaches a heading" \ - "$(_toc_unresolved "$_toc_f")" "[]" -done -check "premise: the link sweep saw the directory's documents" \ - "$([ "$_toc_n" -ge 3 ] && echo enough || echo "$_toc_n")" "enough" -unset _toc_f _toc_n - -# ---- and it must be able to FAIL, on a fixture rather than on the tree ------- - -printf '## 14. test_harness_deps.py: the harness\n- [14. x](#14-test_harness_depspy-the-harness)\n' \ - > "$_dcv_fix/ANCHOR_GOOD.md" -check "control: an anchor that keeps the underscores resolves" \ - "$(_toc_unresolved "$_dcv_fix/ANCHOR_GOOD.md")" "[]" - -# The exact shape that shipped: the underscores stripped out of the file name. -printf '## 14. test_harness_deps.py: the harness\n- [14. x](#14-testharnessdepspy-the-harness)\n' \ - > "$_dcv_fix/ANCHOR_BAD.md" -check "an anchor that strips the underscores is named, not passed over" \ - "$(_toc_unresolved "$_dcv_fix/ANCHOR_BAD.md")" \ - "[1: 14-testharnessdepspy-the-harness]" - -# And the derivation itself, on the heading this defect was found in: the dot and -# the colon go, the underscores stay. -check "the anchor rule drops punctuation and keeps underscores" \ - "$(_toc_anchor '14. test_harness_deps.py: the harness must self-test')" \ - "14-test_harness_depspy-the-harness-must-self-test" - -unset -f _toc_anchor _toc_unresolved - - -# ---- the harness must self-test without a database, and the gate must run it -- -# -# THE MEMBERSHIP DECISION AND THE CONFTEST IMPORT ARE NOT CHECKED HERE, and that is -# jd's rule rather than an omission: the shell harness and the pytest corpus are -# PARALLEL IN FUNCTIONALITY and must not call, import or reference each other. An -# earlier version of this part read test/pytest/conftest.py and INVOKED -# test_harness_deps.py with `python3 ... --disagree`, which is the coupling, not a -# second measurement: a shell arm driving the python decider agrees with it by -# construction and can never report it wrong. -# -# Both properties are asserted in the corpus, where they are native: # conftest imports no driver at module scope # test_harness_deps.py::test_conftest_imports_no_database_driver_at_module_scope # the declaration is exactly the database-free half, both directions # test_harness_deps.py::test_the_declaration_is_exactly_the_database_free_half # the partition accounts for every file, and the three report cases # test_harness_deps.py and test_harness_deps_classifier.py -# -# What stays here is the CI WORKFLOW, which belongs to neither harness. _hd_ci="$PGC_SRCDIR/.github/workflows/ci.yml" - check "premise: the CI workflow is where this part thinks it is" \ "$([ -f "$_hd_ci" ] && echo yes || echo no)" "yes" @@ -410,198 +71,12 @@ check "and it derives the file list rather than repeating it" \ check "and derives the pins from requirements-test.txt" \ "$([ "$(grep -c 'requirements-test.txt' "$_hd_ci")" -ge 1 ] && echo yes || echo no)" "yes" -# ---- and the DECLARATION must be decided, not declaimed ---------------------- -# -unset _hd_ci - -unset _dcv_dir _dcv_doc _dcv_seen _dcv_stated _dcv_fix _dcv_ht -unset -f _dcv_missing _dcv_count - -# ---- the mode inventory must count itself, and the count must be checkable ---- -# -# WHY THIS EXISTS. `test/pytest/VACUITY_MODES.md` and `test/pytest/README.md` -# stated different numbers for how many vacuity modes the layer refuses -- 27 and -# 23 -- and a reviewer could check NEITHER, because the document offered no rule -# for what counts as a mode. That is the defect this whole directory exists to -# refuse, committed by the document that describes the refusal: a number nobody -# can recompute is an assertion, not a measurement. -# -# Section 1a now states the rule: a mode is a backticked kebab-case identifier of -# three or more words. These arms hold the document to its own rule. -# -# WHAT IS POLICED. Four properties: -# * section 1a's "refused today" total is the count of ids in section 2 -# * its "not refused" total is the count in section 3 -# * its document total is the sum of the two -# * README.md quotes the same refused number, so the two cannot drift again -# The enumeration run's own 79 is history, not a property of the tree: nothing -# here pretends to check it. What IS checked is that the gap the document admits -# to equals the run total minus what was actually written down. - -_mi_doc="$PGC_TESTDIR/pytest/VACUITY_MODES.md" -_mi_readme="$PGC_TESTDIR/pytest/README.md" - -check "premise: the mode inventory is where this part thinks it is" \ - "$([ -f "$_mi_doc" ] && echo yes || echo no)" "yes" - -# Distinct ids matching section 1a's rule, within one "## " section. -# A function over FILE and PREFIX so the fixture arms below run the same logic. -_mi_ids() { # _mi_ids FILE PREFIX -> count - local f="$1" pre="$2" - awk -v p="^## $pre" ' - $0 ~ p { inside = 1; next } - /^## / { inside = 0 } - inside { print } - ' "$f" 2>/dev/null \ - | grep -oE '`[a-z0-9]+(-[a-z0-9]+){2,}`' \ - | sort -u | wc -l | tr -d ' ' -} - -# A stated row from the section 1a table. The label is a substring of the cell, -# not the whole of it, so this matches the cell rather than anchoring to its start. -# -# Take the VALUE cell, not the first number on the line. The labels themselves -# contain digits -- "named in section 2, refused today" -- so the obvious -# `grep -oE '[0-9]+' | head -1` returns the 2 from "section 2" and never the -# total. It did, and it read 2 and 3 for totals of 21 and 51. The fixture below -# could not see it because there the label digit and the value were both 2, so -# there is now an arm whose whole job is to tell those two readings apart. -_mi_row() { # _mi_row FILE LABEL -> the number, or "" if the row is absent - local f="$1" label="$2" - grep -E "^\|[^|]*${label}[^|]*\|" "$f" 2>/dev/null | head -1 \ - | awk -F'|' 'NF > 2 { v = $(NF - 1); gsub(/[^0-9]/, "", v); print v }' -} - -# Section 3 keeps a back-reference to every mode that moved into section 2 -# ("`X` is now closed"), so a mode can be named in both. Section 2 wins, and -# section 3's total is what it names MINUS what section 2 claims. Counting the -# back-references as unrefused puts one mode in two states: measured at 25 + 50 -# against 72 named, which is three ids counted twice. -_mi_idlist() { # _mi_idlist FILE PREFIX -> the ids themselves, one per line, sorted - local f="$1" pre="$2" - awk -v p="^## $pre" ' - $0 ~ p { inside = 1; next } - /^## / { inside = 0 } - inside { print } - ' "$f" 2>/dev/null \ - | grep -oE '`[a-z0-9]+(-[a-z0-9]+){2,}`' \ - | sort -u -} - -_mi_ref="$(_mi_ids "$_mi_doc" '2\.')" -_mi_not="$(comm -23 <(_mi_idlist "$_mi_doc" '3\.') <(_mi_idlist "$_mi_doc" '2\.') \ - | grep -c . || true)" - -# The same trap as the sweep above: a counter that finds nothing agrees with a -# document that claims nothing, and both look like success. -check "premise: the counting rule finds modes at all" \ - "$([ "$_mi_ref" -ge 15 ] && [ "$_mi_not" -ge 30 ] && echo enough || echo "$_mi_ref/$_mi_not")" \ - "enough" - -check "section 1a's refused total is the count of ids in section 2" \ - "$(_mi_row "$_mi_doc" 'refused today')" "$_mi_ref" - -check "section 1a's not-refused total is the count of ids in section 3" \ - "$(_mi_row "$_mi_doc" 'not refused')" "$_mi_not" - -check "section 1a's document total is the sum of its two sections" \ - "$(_mi_row "$_mi_doc" 'named in this document')" "$((_mi_ref + _mi_not))" - -check "the admitted gap is the run total minus what is written down" \ - "$(_mi_row "$_mi_doc" 'named nowhere here')" \ - "$(( $(_mi_row "$_mi_doc" 'produced by the enumeration run') \ - - $(_mi_row "$_mi_doc" 'named in this document') ))" - -# README.md is the file that drifted. It must quote the inventory's number rather -# than carry one of its own. -check "README.md quotes the number of modes the inventory names as refused" \ - "$(grep -cE "$_mi_ref refused" "$_mi_readme")" "1" - -# ---- and these arms must be able to FAIL ------------------------------------- - -_mi_fix="$PGC_WORKDIR/modecount"; rm -rf "$_mi_fix"; mkdir -p "$_mi_fix" -{ - printf '## 2. refused\n' - printf 'text `alpha-beta-gamma` and `delta-epsilon-zeta` here\n' - printf '## 3. not refused\n' - printf '`eta-theta-iota`\n' - printf '## 4. table\n' - printf '| named in section 2, refused today | 2 |\n' - printf '| named in section 3, not refused | 1 |\n' - printf '| **named in this document** | **3** |\n' -} > "$_mi_fix/GOOD.md" - -check "the counter counts a fixture's section 2" "$(_mi_ids "$_mi_fix/GOOD.md" '2\.')" "2" -check "the counter counts a fixture's section 3" "$(_mi_ids "$_mi_fix/GOOD.md" '3\.')" "1" - -# Two-word and one-word ids are not modes under section 1a's rule, and a counter -# that took them would inflate every total in the document. -printf '## 2. refused\n`one-two` and `single` and `alpha-beta-gamma`\n' > "$_mi_fix/SHORT.md" -check "an id of fewer than three words is not counted as a mode" \ - "$(_mi_ids "$_mi_fix/SHORT.md" '2\.')" "1" - -# An id named twice is one mode. Section 2 names several ids in more than one row. -printf '## 2. refused\n`alpha-beta-gamma` again `alpha-beta-gamma`\n' > "$_mi_fix/DUP.md" -check "an id named twice counts once" "$(_mi_ids "$_mi_fix/DUP.md" '2\.')" "1" - -# The section boundary must hold: ids after the next heading belong to it. -printf '## 2. refused\n`alpha-beta-gamma`\n## 3. not\n`delta-epsilon-zeta`\n' > "$_mi_fix/BOUND.md" -check "the counter stops at the next heading" "$(_mi_ids "$_mi_fix/BOUND.md" '2\.')" "1" +# AND THE JOB MUST RUN WITHOUT THE DRIVER, which is what makes it able to run at +# all on a machine with no database. Without this the job could quietly install +# psycopg and the database-free claim would stop being tested while still being +# made. Added here because this part's subject is now the workflow, and this is a +# property of the workflow rather than of either harness. +check "and the job asserts the driver is absent rather than assuming it" \ + "$([ "$(grep -c 'pip show psycopg' "$_hd_ci")" -ge 1 ] && echo yes || echo no)" "yes" -# A wrong stated total is visible. This is the shape that shipped in two files. -printf '## 2. refused\n`alpha-beta-gamma`\n## 4. t\n| named in section 2, refused today | 9 |\n' \ - > "$_mi_fix/WRONG.md" -check "a stated total that disagrees with the ids is visible" \ - "$([ "$(_mi_row "$_mi_fix/WRONG.md" 'refused today')" = "$(_mi_ids "$_mi_fix/WRONG.md" '2\.')" ] \ - && echo agrees || echo differs)" "differs" - -check "and the same comparison agrees on the fixture that is right" \ - "$([ "$(_mi_row "$_mi_fix/GOOD.md" 'refused today')" = "$(_mi_ids "$_mi_fix/GOOD.md" '2\.')" ] \ - && echo agrees || echo differs)" "agrees" - -# A missing row must not read as a passing comparison. -printf '## 2. refused\n`alpha-beta-gamma`\n' > "$_mi_fix/NOROW.md" -# The label contains a digit and the value is a different digit, so a reader that -# takes the first number on the line and one that takes the value cell give -# different answers. This is the arm that would have caught the helper's own bug. -printf '## 4. t\n| named in section 2, refused today | 7 |\n' > "$_mi_fix/LABELDIGIT.md" -check "the row's value is read, not a digit inside its label" \ - "$(_mi_row "$_mi_fix/LABELDIGIT.md" 'refused today')" "7" - -check "an absent total is empty rather than a number that happens to match" \ - "$([ -z "$(_mi_row "$_mi_fix/NOROW.md" 'refused today')" ] && echo absent || echo present)" \ - "absent" - -# The table is not the only place a total lives. Three sentences outside it still -# asserted the run's 23 after the table said 21: section 2's opening, the closing -# paragraph, and TESTS.md. Gating the table alone just moves the drift into prose. -# -# The run's own 23 appears once on purpose, as history, and is not gated. What is -# gated is every sentence stating what the layer refuses TODAY. -_mi_prose() { # _mi_prose FILE REGEX -> the captured number, or "" - local f="$1" re="$2" - grep -oE "$re" "$f" 2>/dev/null | head -1 | grep -oE '[0-9]+' | head -1 -} - -_mi_tests="$PGC_TESTDIR/pytest/TESTS.md" - -check "section 2's opening states the counted number of refused modes" \ - "$(_mi_prose "$_mi_doc" '[0-9]+ of the 79, counted')" "$_mi_ref" - -check "the closing paragraph states the counted number too" \ - "$(_mi_prose "$_mi_doc" 'known to refuse [0-9]+ demonstrated modes')" "$_mi_ref" - -check "TESTS.md states the counted number as well" \ - "$(_mi_prose "$_mi_tests" 'This layer refuses [0-9]+')" "$_mi_ref" - -# And the prose reader must be able to fail, on a fixture rather than on the tree. -printf 'the layer is known to refuse 99 demonstrated modes here\n' > "$_mi_fix/PROSE.md" -check "a prose total that disagrees with the ids is visible" \ - "$(_mi_prose "$_mi_fix/PROSE.md" 'known to refuse [0-9]+ demonstrated modes')" "99" - -check "an absent prose total is empty rather than a stray number" \ - "$([ -z "$(_mi_prose "$_mi_fix/GOOD.md" 'known to refuse [0-9]+ demonstrated modes')" ] \ - && echo absent || echo present)" "absent" - -unset _mi_doc _mi_readme _mi_ref _mi_not _mi_fix _mi_tests -unset -f _mi_ids _mi_row _mi_prose +unset _hd_ci