diff --git a/CHANGELOG.md b/CHANGELOG.md index add322c3..2e000528 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -505,6 +505,203 @@ true until the next version shipped. the guard stops being tested that run without anyone noticing" -- four lines above the first `exit 0`. +- The pytest corpus reaches into the shell harness in three files instead of four, and + the inventory that records it is now a mechanism rather than prose (#432). + + CONTEXT.md's rule: the two harnesses are parallel in functionality and independent in + implementation. A pytest test that drives `test/lib.sh` is the first measurement + wearing a Python wrapper -- it agrees with the shell by construction and can never + report it wrong -- so the coupling turns a twin into a mirror. + + `test_build_refusal.py` was the largest item on that inventory at 36 cross-harness + calls. **22 of them are gone.** Their subject was `test/pgc_fingerprint.py`, which has + been the one implementation since #907, and `pgc_source_fingerprint` and + `pgc_source_manifest` in `lib.sh` are thin wrappers that shell out to exactly that + module -- so the path was python -> bash -> lib.sh -> python3 -> the module, and taking + the middle two out changed no subject. Measured before converting a single arm: the + fingerprint is byte-identical through both paths, the manifest identical line for line, + and both agree across `LC_ALL=C`, `C.UTF-8` and `en_US.UTF-8`. + + Two of those arms need a separate process rather than an in-process call -- one reads + as an unprivileged user, because root ignores `chmod 000`, and one varies the locale -- + and both now use the module's own CLI, which is the entry point `lib.sh` itself uses + with `lib.sh` taken out of the path. + + 13 calls remain and only 7 are debt: they drive `pgc_write_source_stamp`, + `pgc_source_stamp_path`, `pgc_freshness_report` and `pgc_freshness_verdict`, which are + PURE SHELL rather than wrappers over shared code, so those properties belong to the + shell harness. Two are the one permitted cross-reference, named as the rule asks, and + two are a historical-parity arm whose fixture is its own. + + **`SHELL_REFERENCES` in `test_harness_deps.py` makes the inventory falsifiable.** It + declares which files reach across and by what mechanism, and asserts set equality in + both directions: a new file that reaches in reddens, and a file that stops reaching and + is left in the list reddens too -- which is what stops a record of debt becoming a + permanent exemption. + + A file-level guard, which is what the rule asks for and the most it can honestly be: + within a flagged file it cannot tell a path joined onto the real tree from the same + name joined onto a `tmp_path`, because both are the string `lib.sh` and only the + dataflow says which. + + **The other direction: `test/selftest/370` is deleted.** It pinned the POSITION of + `plan_marker`'s empty-plan refusal by reading the Python source as text, which is all a + shell part can see. It cannot run the function, so it cannot tell a refusal that still + fires from one stranded behind an early return -- the only thing the position is for. + The property moved into `test_guards_pinned.py` as + `test_the_empty_plan_refusal_precedes_the_arms_it_protects`, which takes the class off + the `expect` fixture so the file still imports nothing. 370's other three properties + were already covered; confirmed by running those arms, not by reading them. + + That arm discriminates -- move the refusal to the end of `plan_marker` and it fails -- + and it is NOT load-bearing today: with the refusal at the end, + `plan_marker([], absent=True)` still refuses, because `plan_marker` has no early + return. So 370's stated reason, that the absent arm returns a pass first, describes a + shape the function does not have. It is labelled prospective insurance against a + refactor that adds one, rather than sold as a live hole. + + **CONTEXT.md said seven shell files reach across and the number is three.** `lib.sh`, + `selftest/030` and `selftest/040` matched only `pgc_cluster_datadir` and + `pgc_cluster_is_ours`, which are shell functions defined in `lib.sh` -- rule 3 of that + entry, "a word that merely looks like a filename", caught for the second time in the + entry that states it. `selftest/350`, `360` and `380` are the real three. + + **`test/selftest/360` is rewritten rather than deleted, and it now checks more.** + Eleven of its seventeen arms were text pins on `pgc_vacuity.py`: that the + unrunnable field is written, that something reads it, that the read reaches + `session.exitstatus`, that the override is conditional. Those are gone. What is + left is the kind CONTEXT.md permits -- a property that IS the relationship between + the two harnesses, so it cannot be stated from one side -- and there turned out to + be THREE such properties where the part checked one. + + The part checked the INCOMPLETE exit code and left two duplications beside it + unchecked: the closed list of unrunnable reasons, and the one-line shape an + unrunnable check prints. Both are now parsed out of both files and compared, with + a drifted fixture for each so the comparison can fail. The reason list is the + worse omission of the two, because being closed on both sides is its whole purpose. + + Both deletions were measured first. With `session.exitstatus = EXIT_INCOMPLETE` + made unreachable in `pgc_vacuity.py` -- the defect exactly as it shipped -- the + four behavioural arms in `test_layer.py` go from `4 passed` to `2 failed, 2 + passed`. Two is correct rather than partial: the other two assert exit 1 for a run + with a real failure and exit 0 for a run with nothing unrunnable, and neither + outcome moves. The file was restored and compared byte-for-byte afterwards. + + **And the reason the part gave for keeping those pins was true when written and is + not now.** It said the behavioural arms "need pytest, psycopg and a virtualenv; CI + installs none of them". CI has a `pytest-guards` job that installs pytest pinned + from `requirements-test.txt`, asserts psycopg is absent, and runs the + database-free file list -- which contains `test_layer.py`. A stale justification + for keeping coverage in the wrong place is harder to find than a missing check, + because nothing reddens. + + Two instrument defects of my own, both caught by the new arms' own premises. The + shape parse took the FIRST line matching the print marker, which in + `pgc_vacuity.py` is the DOCSTRING that spells the shape out for a reader; + requiring a quote before the marker selects the code in both languages. And the + drifted-shape fixture wrote one space where the real shape has two, so the parse + found nothing and the comparison was empty-against-real -- which "differs", for + the wrong reason. Its premise arm said so. + + **`test/selftest/380` keeps finding 1 and hands finding 2 to the pytest corpus.** + The part covers two of @linuxhikerpm's #897 findings, and they have different + subjects. Finding 1 is `test/pgc_fingerprint.py`, which `lib.sh` runs with the + system interpreter and which is not part of the pytest harness, so checking it is + this part's own business and it stays. Finding 2 is `test/pytest/pgc_cluster.py`, + and the nine arms that read it as text are gone. + + Seven of its twenty-one checks had a cross-harness subject, not twenty-one: the + other fourteen read `pgc_fingerprint.py`, `lib.sh`, or fixtures the part writes + itself. Counting the whole file would have deleted coverage that was never debt. + + `make_cluster`'s cleanup SHAPE moved to `test_build_refusal.py` as + `test_the_cleanup_guard_has_the_shape_the_leak_needs`, beside the behavioural arm + that provokes a real failed setup. It has to be a source check, and it says so: + `make_cluster` fails exactly one way in the behavioural arm -- a missing + `pg_config` -- while three more properties decide whether the guard works, and two + of those cannot be provoked at all. You cannot deliver SIGINT into `initdb` + reliably, and a cluster that started is one the arm would then have to stop. + + Five mutations say the moved arms discriminate, each asserted to have applied, + each leaving the module well-formed, restored byte-for-byte afterwards: + `BaseException` narrowed to `Exception`, `cluster.stop()` removed, the bare + re-raise turned into `pass`, a private `hashlib.md5` added, and + `shutil.rmtree(root, …)` removed. The two whose effect reaches the filesystem + redden the behavioural arm as well; the three that redden only the source arm are + exactly the properties the behavioural arm cannot see. + + One of those five did not apply on the first attempt -- the `cluster.stop()` + pattern assumed twelve spaces of indentation and the call sits at sixteen, inside + a nested `try`. The harness refused to report a result for it rather than printing + a green, which is the only safe behaviour for a mutation that did not land. + + **And one thing went wrong that is worth more than the change itself.** I ran the + selftest before naming the two new pytest arms in `TESTS.md`, so the run was red on + the doc-coverage check -- and then merged that log into `check_ledger.tsv`, whose + entire subject is which checks have ever been red. It recorded a red for + `350`'s "every test file and every test in the corpus is named in TESTS.md", + a check that failed only because my own change was half-finished. Reverted with + `git checkout HEAD --`; there is no second copy to repair from. + + The merge step now refuses a log that is not definitely green: a numeric + `checks run:` line, a floor on it so an aborted run cannot pass, zero `FAIL` + lines, and rc=0. The absence of a `FAIL` line is not enough, because an aborted + run has none either -- the same shape as a pending-count that cannot see a job + + **`test/selftest/350` goes from 50 checks to 5, and two of its rules moved rather + than being deleted.** Forty-six of the fifty had a subject on the other side of the + boundary: they globbed `test/pytest/*.py` and parsed Python out of it, or swept the + markdown in that directory. The pytest corpus asserts all three properties natively + in `test_docs_cover_the_corpus.py`, where the subject is, so the shell copy could + only ever agree with it. + + Two of the three rules were implemented on BOTH sides and self-tested only on the + shell side -- the side that cannot run the corpus it counts. Deleting those + fixtures would have left the Python implementation with no fixtures at all, so they + went with them: + + - the mode-counting rule's edges: an id of fewer than three words, an id named + twice, stopping at the next heading, section 3's back-references, and the row + reader taking the value cell rather than a digit inside its label + - the contents-list anchor rule: GitHub's derivation, the broken link that shipped, + and a control beside it + + `_named_modes_in(text)` and `_stated_row(text, label)` are new seams, so a fixture + can reach the rules at all. The row reader's fixture makes the label digit and the + value DIFFER -- label "section 2", value 9 -- because `350`'s own fixture had both + as 2 and could not see the bug it was written for. + + **What stays in 350 is its arms over `ci.yml`**, whose subject is neither harness. + A shell part may read the workflow for the same reason it may read the Makefile, and + no pytest arm can assert that the gate runs pytest without assuming the thing in + question. One arm is added while the part's subject is being settled: the job must + ASSERT `psycopg` is absent rather than assume it. + + ### The stale justification, in triplicate + + TESTS.md section 6 said "Nothing runs pytest. Not `run_all_versions.sh`, not any + workflow under `.github/`", and concluded that the `.sh` half was therefore the + enforcement. CI has a `pytest-guards` job: it installs pytest pinned from + `requirements-test.txt`, asserts `psycopg` is absent, derives the file list from + `NO_CLUSTER`, and runs it. + + That claim was load-bearing in THREE places at once -- TESTS.md section 6, + `selftest/360` and `selftest/380` -- each asserting the behavioural half could not + run in CI. Nothing was wrong in any of them; the reason was, three times, and **no + arm reddens on a stale justification.** All three are corrected. + + ### Where the two harnesses now stand + + Shell parts holding a reference to anything under `test/pytest/`: **one**, and it is + deliberate. `selftest/360` reads `pgc_vacuity.py` to compare the three values the + two harnesses both write down -- the INCOMPLETE exit code, the closed list of + unrunnable reasons, and the line an unrunnable check prints. That is CONTEXT.md's + permitted cross-reference: a property that IS the relationship, so it cannot be + stated from one side. The inventory said seven; the measured answer was three plus a + deleted fourth; the end state is one. + which never started. + + - The vacuity guard's PLACEMENT is now a checked property, because a guard in a teardown cannot fail the test it guards (#432). diff --git a/CONTEXT.md b/CONTEXT.md index a92126b3..c69df8f2 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -211,19 +211,82 @@ for it to stop being one. **The debt this starts with, on 2026-09-10: 3 python files and 7 shell files.** +**Progress, 2026-09-11.** The fourth never arrived: #923 deleted +`test_check_results_are_machine_readable.py` rather than land a fresh violation. And +`test_build_refusal.py` is down from 36 cross-harness calls to 13, of which 7 are the +debt proper, 2 are the permitted cross-reference below, and 2 are one historical-parity +arm. **Counted by mechanism rather than by line, because a line total moves with the +pattern:** the real executable coupling across the whole corpus was six sites in three +files -- `pgc_cluster.py` once, `test_build_refusal.py` three times, and +`test_suite_accounting.py` twice -- and the two `lib.sh` writes in +`test_build_refusal.py` are fake trees under `tmp_path`, rule 2, not references. + Python that reaches into shell: -- `test_build_refusal.py` -- sources the real `test/lib.sh` in three helpers - (`_sh`, `_sh_fp`, `_sh_fp_as`), behind 36 calls. The largest of these. Counted - with `ast`, not `grep`: the pattern `[^_a-z]_sh(` also matches `def _sh(`, which - is how the first draft said 39 -- 36 calls plus the 3 definitions. Reported by - @OffgridwithJD. Rule 3 above, caught in the very entry that states it. +- `test_build_refusal.py` -- **reduced, not cleared.** It sourced the real + `test/lib.sh` in three helpers (`_sh`, `_sh_fp`, `_sh_fp_as`) behind 36 calls and + was the largest item here. Counted with `ast`, not `grep`: the pattern + `[^_a-z]_sh(` also matches `def _sh(`, which is how the first draft said 39 -- 36 + calls plus the 3 definitions. Reported by @OffgridwithJD. Rule 3 above, caught in + the very entry that states it. + + **22 of those calls are gone.** Their subject was `test/pgc_fingerprint.py`, the ONE + implementation since #907, and `pgc_source_fingerprint` and `pgc_source_manifest` + are thin wrappers that shell out to exactly it -- so the path was + python -> bash -> lib.sh -> python3 -> the module, and removing the middle two + changed no subject. Measured before converting anything: byte-identical + fingerprint, identical manifest line for line, and agreement across LC_ALL=C, + C.UTF-8 and en_US.UTF-8. Two of them needed a separate process rather than an + in-process call -- one reads as an unprivileged user because root ignores + `chmod 000`, one varies the locale -- and both use the module's own CLI, which is + the entry point lib.sh uses with lib.sh taken out of the path. + + **13 calls remain, in three groups, and only the first is debt.** Seven drive + `pgc_write_source_stamp`, `pgc_source_stamp_path`, `pgc_freshness_report` and + `pgc_freshness_verdict`, which are PURE SHELL rather than wrappers over shared + code: those properties belong to the shell harness and moving them is the next + step. Two are `test_the_two_fingerprint_implementations_cover_the_same_inputs`, + which reaches across on purpose -- see below. The last two are a historical-parity + arm whose fixture is its own, bar one call for a directory list. + +- **The one permitted cross-reference, named as the rule asks.** + `test_the_two_fingerprint_implementations_cover_the_same_inputs` asserts that the + shell path and the Python path give the same value, which is to say that neither + side carries a private copy. That property IS the relationship, so it cannot be + expressed from one side: the rule's own escape clause -- "say which, and say why" + -- applies, and this is the saying. It caught four defects in one day (#907), and + it is what reddens on the FIRST edit if a private implementation comes back rather + than on the first edit that happens to diverge. Every other reference in this + inventory is expected to go; this one is expected to stay. - `test_suite_accounting.py` -- reads `run_all_versions.sh`'s text, sources the real `lib.sh` from a suite it writes, and executes the real runner. - `pgc_cluster.py` -- sources the real `test/lib.sh`. - -Shell whose subject is python: `lib.sh`, and `selftest/030`, `040`, `350`, `360`, -`370`, `380`. +- `test_mutation_ledger.py` -- runs `run_all_versions.sh --list-suites` for the + registered suite list. It **arrived after this inventory was written**, with #925, + and the arm below is what said so: the set-equality assertion reddened on the + rebase naming a fourth file, which is the whole reason the inventory is a mechanism + and not this paragraph. Same mechanism as `test_suite_accounting.py`, so it is the + same item of debt twice and they should move together. + +Shell whose subject is python: `selftest/350`, `360` and `380`. **Three, not the +seven this line first named, and the three it dropped were rule 3 all along.** +`lib.sh`, `selftest/030` and `selftest/040` reference NO path under `test/pytest`: +their only matches were the shell functions `pgc_cluster_datadir` and +`pgc_cluster_is_ours`, both defined in `lib.sh` itself -- "a word that merely looks +like a filename", caught for the second time in the entry that states the rule. +`selftest/370` was the fourth and is **deleted**: every property it pinned was a text +pin on `pgc_vacuity.py`, and #927 is the precedent -- a shell arm asserting a text pin +cannot prove a python arm is caught. Its one property that the pytest corpus did not +already assert behaviourally moved to `test_guards_pinned.py`, where python reads its +own module rather than the other harness's. + +**And that property is not behaviourally observable today, which is why it moved as a +SOURCE check and says so.** Measured: with `plan_marker`'s empty-plan refusal moved to +the very end of the function, `plan_marker([], absent=True)` still refuses -- +`plan_marker` has no early return, so the refusal fires wherever it sits. 370's stated +reason, that "the absent arm returns a pass first", describes a shape the function does +not have. The arm is prospective insurance against a refactor that adds an early return, +and it is labelled as that rather than as a live guard. **`test_build_refusal.py` is the example worth studying, because it does both.** It writes a fake `test/lib.sh` into a `tmp_path` and drives that -- rule 2, not a diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 7f10fbaa..c068561b 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -408,109 +408,39 @@ harness_selftest 340-the-binary-must-be-built-from the stamp writer reports fail harness_selftest 340-the-binary-must-be-built-from the writer writes the file the reader looks for never - harness_selftest 340-the-binary-must-be-built-from two installations of one major get different stamp paths never - harness_selftest 340-the-binary-must-be-built-from two unreadable pg_configs do not alias onto one stamp never - -harness_selftest 350-the-pytest-corpus-must-be README.md quotes the number of modes the inventory names as refused never - -harness_selftest 350-the-pytest-corpus-must-be TESTS.md states no totals line for a merge to get wrong never - -harness_selftest 350-the-pytest-corpus-must-be TESTS.md states the counted number as well never - -harness_selftest 350-the-pytest-corpus-must-be a documented file that does not exist is named never - -harness_selftest 350-the-pytest-corpus-must-be a documented test that does not exist is named, not passed over never - -harness_selftest 350-the-pytest-corpus-must-be a name defined in two files is named, not passed over never - -harness_selftest 350-the-pytest-corpus-must-be a prose total that disagrees with the ids is visible never - -harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with disk is visible never - -harness_selftest 350-the-pytest-corpus-must-be a stated total that disagrees with the ids is visible never - -harness_selftest 350-the-pytest-corpus-must-be an absent prose total is empty rather than a stray number never - -harness_selftest 350-the-pytest-corpus-must-be an absent total is empty rather than a number that happens to match never - -harness_selftest 350-the-pytest-corpus-must-be an anchor that strips the underscores is named, not passed over never - -harness_selftest 350-the-pytest-corpus-must-be an id named twice counts once never - -harness_selftest 350-the-pytest-corpus-must-be an id of fewer than three words is not counted as a mode never - -harness_selftest 350-the-pytest-corpus-must-be an unbackticked name in prose is not treated as a claim never - -harness_selftest 350-the-pytest-corpus-must-be an undocumented file is caught along with the tests inside it never - -harness_selftest 350-the-pytest-corpus-must-be an undocumented test is named rather than passed over never - harness_selftest 350-the-pytest-corpus-must-be and derives the pins from requirements-test.txt never - harness_selftest 350-the-pytest-corpus-must-be and it derives the file list rather than repeating it never - -harness_selftest 350-the-pytest-corpus-must-be and the same comparison agrees on the fixture that is right never - -harness_selftest 350-the-pytest-corpus-must-be control: a document naming only what exists is clean never - -harness_selftest 350-the-pytest-corpus-must-be control: a fully documented corpus reports nothing missing never - -harness_selftest 350-the-pytest-corpus-must-be control: an anchor that keeps the underscores resolves never - -harness_selftest 350-the-pytest-corpus-must-be control: distinct names in the same corpus report no duplicate never - -harness_selftest 350-the-pytest-corpus-must-be every in-document link in README.md reaches a heading never - -harness_selftest 350-the-pytest-corpus-must-be every in-document link in TESTS.md reaches a heading never - -harness_selftest 350-the-pytest-corpus-must-be every in-document link in VACUITY_MODES.md reaches a heading never - -harness_selftest 350-the-pytest-corpus-must-be every test file and every test in the corpus is named in TESTS.md never - -harness_selftest 350-the-pytest-corpus-must-be every test the document names exists in the corpus never - -harness_selftest 350-the-pytest-corpus-must-be no test name is defined twice in the corpus never - +harness_selftest 350-the-pytest-corpus-must-be and the job asserts the driver is absent rather than assuming it never - harness_selftest 350-the-pytest-corpus-must-be premise: the CI workflow is where this part thinks it is never - -harness_selftest 350-the-pytest-corpus-must-be premise: the corpus carries the documentation this part polices never - -harness_selftest 350-the-pytest-corpus-must-be premise: the counting rule finds modes at all never - -harness_selftest 350-the-pytest-corpus-must-be premise: the link sweep saw the directory's documents never - -harness_selftest 350-the-pytest-corpus-must-be premise: the mode inventory is where this part thinks it is never - -harness_selftest 350-the-pytest-corpus-must-be premise: the pytest corpus is where this part thinks it is never - -harness_selftest 350-the-pytest-corpus-must-be premise: the reader still finds a totals line when one is there never - -harness_selftest 350-the-pytest-corpus-must-be premise: the reverse sweep reads backticked names at all never - -harness_selftest 350-the-pytest-corpus-must-be premise: the sweep found the corpus rather than an empty glob never - -harness_selftest 350-the-pytest-corpus-must-be premise: the sweep reads the contents list's links never - -harness_selftest 350-the-pytest-corpus-must-be section 1a's document total is the sum of its two sections never - -harness_selftest 350-the-pytest-corpus-must-be section 1a's not-refused total is the count of ids in section 3 never - -harness_selftest 350-the-pytest-corpus-must-be section 1a's refused total is the count of ids in section 2 never - -harness_selftest 350-the-pytest-corpus-must-be section 2's opening states the counted number of refused modes never - -harness_selftest 350-the-pytest-corpus-must-be the admitted gap is the run total minus what is written down never - -harness_selftest 350-the-pytest-corpus-must-be the anchor rule drops punctuation and keeps underscores never - -harness_selftest 350-the-pytest-corpus-must-be the closing paragraph states the counted number too never - -harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 2 never - -harness_selftest 350-the-pytest-corpus-must-be the counter counts a fixture's section 3 never - -harness_selftest 350-the-pytest-corpus-must-be the counter stops at the next heading never - harness_selftest 350-the-pytest-corpus-must-be the gate runs the harness guards never - -harness_selftest 350-the-pytest-corpus-must-be the row's value is read, not a digit inside its label never - -harness_selftest 350-the-pytest-corpus-must-be the sweep counts the fixture's tests and files never - -harness_selftest 360-an-unrunnable-pytest-test-must a comparison on the exit status is not counted as an assignment never - harness_selftest 360-an-unrunnable-pytest-test-must a drifted exit code is visible rather than absorbed never - -harness_selftest 360-an-unrunnable-pytest-test-must a write-only unrunnable field is caught never - -harness_selftest 360-an-unrunnable-pytest-test-must an unconditional exit override is caught by the dominance arm never - -harness_selftest 360-an-unrunnable-pytest-test-must and only ever moves a run off zero, so a failure still dominates never - -harness_selftest 360-an-unrunnable-pytest-test-must and something READS it, rather than only writing it never - -harness_selftest 360-an-unrunnable-pytest-test-must premise: and that same fixture does show the write, so the arm is not blind never - +harness_selftest 360-an-unrunnable-pytest-test-must a drifted print shape is visible never - +harness_selftest 360-an-unrunnable-pytest-test-must a reason dropped from one side only is visible never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: and that fixture is a real list rather than an empty parse never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: and that fixture parses to a shape rather than to nothing never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: lib.sh prints a line for an unrunnable check never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: lib.sh states a closed list of unrunnable reasons never - harness_selftest 360-an-unrunnable-pytest-test-must premise: lib.sh states an INCOMPLETE exit code this part could read never - harness_selftest 360-an-unrunnable-pytest-test-must premise: the harness library is where this part thinks it is never - harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer is where this part thinks it is never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer prints one too never - +harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer states a closed list too never - harness_selftest 360-an-unrunnable-pytest-test-must premise: the pytest layer states one too never - -harness_selftest 360-an-unrunnable-pytest-test-must premise: while a real assignment on the same line shape IS counted never - -harness_selftest 360-an-unrunnable-pytest-test-must premise: while the real layer satisfies that same arm never - -harness_selftest 360-an-unrunnable-pytest-test-must the layer ends a session by setting its exit status never - -harness_selftest 360-an-unrunnable-pytest-test-must the layer prints the unrunnable reason in lib.sh's shape never - -harness_selftest 360-an-unrunnable-pytest-test-must the layer still writes the unrunnable state never - harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses agree on the INCOMPLETE exit code never - -harness_selftest 370-the-plan-marker-guard-must a neutered absent arm is caught never - -harness_selftest 370-the-plan-marker-guard-must a neutered empty-plan refusal is caught never - -harness_selftest 370-the-plan-marker-guard-must a neutered present arm is caught never - -harness_selftest 370-the-plan-marker-guard-must and that refusal is a VacuityError, not an ordinary assertion never - -harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is absent never - -harness_selftest 370-the-plan-marker-guard-must plan_marker keeps the arm that fails when the key is present never - -harness_selftest 370-the-plan-marker-guard-must plan_marker refuses a plan with no nodes at all never - -harness_selftest 370-the-plan-marker-guard-must premise: both line numbers were found, so the ordering arm can mean something never - -harness_selftest 370-the-plan-marker-guard-must premise: plan_marker's body was actually cut out of the file never - -harness_selftest 370-the-plan-marker-guard-must premise: the pytest layer is where this part thinks it is never - -harness_selftest 370-the-plan-marker-guard-must premise: while the real body satisfies all three, so the greps work never - -harness_selftest 370-the-plan-marker-guard-must the empty-plan refusal precedes the arm it protects never - +harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses agree on the closed list of unrunnable reasons never - +harness_selftest 360-an-unrunnable-pytest-test-must the two harnesses print an unrunnable check in the same shape never - harness_selftest 380-the-pytest-cluster-helpers a caller that reimplements the digest is caught never - harness_selftest 380-the-pytest-cluster-helpers a fingerprint that reads src only is caught never - -harness_selftest 380-the-pytest-cluster-helpers a make_cluster with no cleanup is caught never - harness_selftest 380-the-pytest-cluster-helpers an import from the pytest tree is caught never - harness_selftest 380-the-pytest-cluster-helpers and a hard-coded module list is caught by the name arm never - -harness_selftest 380-the-pytest-cluster-helpers and it catches BaseException, so an interrupt cleans up too never - harness_selftest 380-the-pytest-cluster-helpers and it names no module directory, so it is a derivation and not a list never - -harness_selftest 380-the-pytest-cluster-helpers and it stops a partially started cluster before removing the tree never - harness_selftest 380-the-pytest-cluster-helpers and no longer mixes in the bare filename never - -harness_selftest 380-the-pytest-cluster-helpers and the original error is re-raised rather than swallowed never - harness_selftest 380-the-pytest-cluster-helpers and the shell keeps none either never - -harness_selftest 380-the-pytest-cluster-helpers make_cluster removes its tree when setup raises never - -harness_selftest 380-the-pytest-cluster-helpers premise: and the real helper still carries its cleanup never - -harness_selftest 380-the-pytest-cluster-helpers premise: make_cluster's body was actually cut out of the file never - harness_selftest 380-the-pytest-cluster-helpers premise: the one fingerprint implementation is where this part thinks it is never - -harness_selftest 380-the-pytest-cluster-helpers premise: the pytest cluster helper is where this part thinks it is never - harness_selftest 380-the-pytest-cluster-helpers premise: while the real module satisfies the derivation arm never - harness_selftest 380-the-pytest-cluster-helpers the fingerprint derives its build directories from a Makefile on disk never - harness_selftest 380-the-pytest-cluster-helpers the hash mixes in each file's path relative to the tree, not its name never - harness_selftest 380-the-pytest-cluster-helpers the module imports nothing from the pytest tree never - -harness_selftest 380-the-pytest-cluster-helpers the pytest helper keeps no private fingerprint implementation never - harness_selftest 390-a-registered-suite-must-account a NEW unaccounted suite fails even while the known debt is excused never - harness_selftest 390-a-registered-suite-must-account a comment mentioning pgc_summary is not a declaration never - harness_selftest 390-a-registered-suite-must-account a declared suite that produced no accounting is caught never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 01a8c04f..67080c34 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 826 +checks_never_observed_red 756 diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index b044b049..5b187784 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -324,6 +324,7 @@ as the failure. | `test_plan_marker_absent_arm_passes_on_a_plan_that_lacks_the_key` | **control** | | `test_plan_marker_refuses_an_absence_claim_over_an_empty_plan` | the hole under both arms | | `test_refusal_itself_refuses_an_empty_pattern_list` | the new helper must not become the defect it removes | +| `test_the_empty_plan_refusal_precedes_the_arms_it_protects` | the refusal's **position**: no arm may answer ahead of it | ### plan_marker, and the three ways it could not fail @@ -344,6 +345,23 @@ legitimately lacks the node. That is now a `VacuityError`, and it is refused for the present arm too — an empty plan means the `EXPLAIN` did not arrive, so neither question can be answered. +**The empty-plan refusal is pinned by position, not only by behaviour.** A guard +that sits after the code it protects is a guard that never runs. So one arm reads +`plan_marker`'s own source and asserts the empty-plan refusal comes before both +arms. Two measurements say what that arm is worth today: + +- Move the refusal to the end of the function and the arm fails; leave it where it + is and it passes. It discriminates. +- With the refusal moved to the end, `plan_marker([], absent=True)` **still + refuses**, because `plan_marker` has no early return for the absent arm. So the + order is not load-bearing right now. + +It is therefore prospective insurance: the day someone adds an early return, the +refusal stops being reachable and this arm is the only thing that says so. The +check was a shell part (`test/selftest/370`) until the two harnesses were +separated; a shell part can pin the text of a Python function but cannot run it, +so the arm moved here and 370 was deleted. + The four arm tests are behavioural rather than refusals, because `plan_marker`'s two arms raise `AssertionError`: `expect.refusal` does not apply and `expect.outcomes` is the right instrument. Their value is not their own green, @@ -448,6 +466,37 @@ is where a wrong quote would hide. | `test_the_fingerprint_covers_a_separately_built_module` | an `objstore/` edit moves the hash | | `test_an_objstore_edit_forces_a_second_build` | and forces a rebuild, end to end | | `test_make_cluster_leaves_nothing_behind_when_setup_fails` | a failed setup leaks no directory | +| `test_the_cleanup_guard_has_the_shape_the_leak_needs` | **source check**: the guard catches `BaseException`, stops the cluster, removes the tree, and re-raises | +| `test_this_module_keeps_no_private_fingerprint` | **source check**: no second digest implementation in this caller | + +**Two of these are SOURCE checks, and they say so.** `test_make_cluster_leaves_nothing_behind_when_setup_fails` +provokes a real failed setup and asserts no directory is left; that is the property. +But `make_cluster` fails exactly one way in that arm — a missing `pg_config` — while +three more properties decide whether the guard works at all: it must survive a +`KeyboardInterrupt`, stop a postmaster it already started, and re-raise rather than +return `None`. Two of those cannot be provoked from a test (you cannot deliver SIGINT +into `initdb` reliably, and a cluster that started is one the arm would then have to +stop), so they are read off `inspect.getsource(make_cluster)` instead. + +They arrived from `test/selftest/380`, which read this file as text across the harness +boundary. Reading our own module is not a cross-harness reference; a shell part +grepping it is the thing CONTEXT.md refuses. What the shell part could never do is the +behavioural arm above it. + +Five mutations say the source arms discriminate, each asserted to have applied and +each leaving the module well-formed, restored byte-for-byte afterwards: + +| mutation of `pgc_cluster.py` | what reddens | +| --- | --- | +| `except BaseException:` narrowed to `except Exception:` | the source arm | +| `cluster.stop()` removed | the source arm | +| the bare `raise` turned into `pass` | the source arm **and** the behavioural one | +| a private `hashlib.md5` added | the no-private-digest arm | +| `shutil.rmtree(root, …)` removed | the source arm **and** the behavioural one | + +The two that redden both are the two whose effect reaches the filesystem. The three +that redden only the source arm are exactly the properties the behavioural arm cannot +see, which is why they are written down separately rather than folded into it. | `test_the_stamp_writer_reports_failure` | `\|\| true` made both controllers' warnings unreachable | | `test_two_installations_of_one_major_do_not_share_a_stamp` | the key names the installation, not just the major | | `test_moving_bytes_between_files_moves_the_shell_fingerprint` | the digest sees a repartition | @@ -782,6 +831,15 @@ many times. | `test_an_undocumented_file_is_caught_with_the_tests_inside_it` | how 29 tests went missing at once | | `test_a_document_with_no_totals_line_states_none` | absent totals report `None`, which must not read as "they match" | | `test_a_stated_total_that_disagrees_with_disk_is_visible` | the count arm's own red | +| `test_the_counting_rule_counts_a_fixture_as_the_document_says` | the mode rule, on a fixture: deduplicated, and section 3 minus its back-references | +| `test_an_id_of_fewer_than_three_words_is_not_a_mode` | the rule is three words, so `one-two` in prose is not a mode | +| `test_the_counter_stops_at_the_next_heading` | section 2's count must not reach into section 4 | +| `test_the_row_reader_takes_the_value_not_a_digit_in_the_label` | the labels contain digits; reading the first number returns the 2 from "section 2" | +| `test_an_absent_row_is_none_rather_than_a_number_that_happens_to_match` | a missing row must not read as a row stating zero | +| `test_a_stated_total_that_disagrees_with_the_ids_is_visible` | the inventory arm's own red, with the agreeing control beside it | +| `test_the_anchor_rule_drops_punctuation_and_keeps_underscores` | GitHub's derivation, on the heading the defect was found in | +| `test_an_anchor_that_strips_the_underscores_is_caught` | the exact broken link that shipped, with a control | +| `test_every_in_document_link_in_this_directory_reaches_a_heading` | every contents-list link resolves, with a coverage premise | The five fixture arms exist because everything above them passes on a healthy tree, which is exactly what a guard that does nothing also does. They run the identical @@ -789,20 +847,26 @@ functions over a corpus built to be wrong. ### The twin, and which half has teeth -This is the pytest half. The other half is -`test/selftest/350-the-pytest-corpus-must-be.sh`, and the two are **not** -interchangeable: - -- **The `.sh` half is the one that gates.** `harness_selftest` is registered in - `SUITES`, so it runs in the matrix and in CI. -- **Nothing runs pytest.** Not `run_all_versions.sh`, not any workflow under - `.github/`. A guard written only here would never fire in the gate, and a guard - that does not run is a comment. - -So the `.sh` copy is the enforcement and this one is what a person running the -corpus by hand gets, with the offenders arriving as a Python list rather than as a -string assembled by shell. Both are written in the same change, per the rule in -section 9. +**This section said "Nothing runs pytest. Not `run_all_versions.sh`, not any +workflow under `.github/`", and that is no longer true.** CI has a `pytest-guards` +job: it installs pytest pinned from `requirements-test.txt`, asserts `psycopg` is +absent, derives the file list from `NO_CLUSTER` in `test_harness_deps.py`, and runs +it. This file is in that list. So a guard written only here DOES fire in the gate, +and the argument that made the `.sh` half the enforcement has gone. + +That argument was load-bearing, and it was stale in three places at once — here, +in `test/selftest/360`, and in `test/selftest/380` — each saying the behavioural +half could not run in CI. **A stale justification for keeping coverage in the wrong +place is harder to find than a missing check, because nothing reddens.** Nothing +was wrong; the reason was. + +So the duplication is being removed in the direction the two-harness rule requires +(#432). `test/selftest/350`'s arms over this corpus and over this directory's +documents are the ones that had to read across the boundary, and they are gone; the +properties they held that this file did not yet test — the mode-counting rule's +edges and the contents-list anchor rule — moved here, where their subject is. What +stays in `350` is its arms over `ci.yml`, whose subject is the workflow rather than +either harness. This guard reddened on its own arrival, which is the only reason it is known to work here: adding this file moved the corpus from `(54, 5)` to `(62, 6)` and the @@ -1292,6 +1356,8 @@ fixtures are read off `conftest.py` rather than named in the classifier. | `test_the_gate_runs_the_membership_decision_rather_than_only_this_file` | selftest 350 runs the decision, and the command line it uses works | | `test_ci_derives_the_file_list_rather_than_repeating_it` | the CI job asks this module for `NO_CLUSTER`, names no file literally, and states no count | | `test_the_job_installs_no_database_driver` | the job asserts psycopg is absent rather than assuming it | +| `test_the_shell_reference_detector_sees_code_and_not_prose` | the premise: a docstring is prose, a string passed to bash is a reference, an f-string counts once | +| `test_the_harness_independence_inventory_is_exactly_what_the_corpus_does` | CONTEXT.md's inventory, asserted in both directions | **THIS IS NOW IN THE GATE.** `.github/workflows/ci.yml` runs a `pytest-guards` job: no database, no build, an interpreter and the two pinned runner packages. @@ -1317,6 +1383,36 @@ environment broken if the test died. A module that raises on import, first on th path, is the same observation and reversible by construction. Both behavioural arms assert the shim actually bites before believing anything it produces. +### The harness-independence inventory, as a mechanism + +CONTEXT.md's rule is that the two harnesses are parallel in functionality and +independent in implementation: **a pytest test that drives `test/lib.sh` is the first +measurement wearing a Python wrapper**, so it agrees with the shell by construction and +can never report it wrong. Its inventory of what still reaches across was **prose** — +falsifiable by hand, but nothing reddened when a new reference appeared. #923 nearly +landed a fourth coupled file, and what caught it was a person reading. + +`SHELL_REFERENCES` declares the three files that reach across and **why each one does**, +and the arm asserts set equality in both directions. A new file that reaches in reddens +it; a file that stops reaching and is left in the declaration reddens it too — which is +what stops the list rotting into a permanent exemption, the way every hand-maintained +exempt list in this tree has gone wrong. + +**A file-level guard, which is what the rule asks for and also the most it can honestly +be.** Within a flagged file it cannot tell a path joined onto the real tree from the +same name joined onto a `tmp_path`: both are the string `lib.sh`, and only the dataflow +says which. `test_build_refusal.py` contains both, and CONTEXT.md already records the +fake ones as rule 2 rather than references. So the assertion is over the **set of +files**, and each entry carries the mechanism a reader needs to check it by hand. + +**Two things the first version got wrong, both found by running it.** It counted an +f-string as two references, because the pieces of one are `Constant` nodes of their own. +And it flagged **this file**, because the declaration's own descriptions named the shell +files — four files where the tree has three. The descriptions now name the mechanism +without the filenames, and the detector's fixtures assemble the name from fragments. A +scan flagging its own test data is the third time that shape cost a measurement in one +session. + ## 16. test_harness_deps_classifier.py: the classifier, in the file the gate runs `test_harness_deps.py` defines the classifier that decides which files the diff --git a/test/pytest/pgc_cluster.py b/test/pytest/pgc_cluster.py index 0dc126e8..c2c2ba85 100644 --- a/test/pytest/pgc_cluster.py +++ b/test/pytest/pgc_cluster.py @@ -415,6 +415,26 @@ def source_fingerprint(srcdir): return _fp.fingerprint(srcdir) or None +def source_manifest(srcdir): + """Every file the fingerprint hashes, one "relpath digest" per line. + + THE SAME TEXT test/lib.sh's `pgc_source_manifest` prints, because both are the + same call into test/pgc_fingerprint.py. A caller comparing the two is therefore + comparing one implementation with itself rather than two that can drift. + + None, not "", when a file could not be read -- the module's contract, and + deliberate: a digest that FAILED must not look like one that succeeded. lib.sh's + wrapper substitutes a sentinel line there instead, which is the WRAPPER's + behaviour and not the module's, so a test of that belongs to the shell harness. + + I wrote this as a join over (relpath, digest) pairs first. `manifest()` returns the + joined TEXT, and my probe had an `isinstance` fallback that quietly stringified it + and then reported the two "identical" -- so the wrong assumption read as verified + until the real call raised. + """ + return _fp.manifest(srcdir) + + def build_once(srcdir, pg_config, major, lock_path=None, runner=None): """build_and_install, but at most once across xdist workers. diff --git a/test/pytest/test_build_refusal.py b/test/pytest/test_build_refusal.py index 30dec399..c8f97b1d 100644 --- a/test/pytest/test_build_refusal.py +++ b/test/pytest/test_build_refusal.py @@ -18,9 +18,51 @@ Python side does with a verdict. The `bash` arms pin the SHELL PLUMBING -- the sourcing, the quoting and the exit-status path -- which an injected runner cannot reach and which is where a wrong quote would hide. + +WHAT THIS FILE REACHES INTO, AND WHAT IT NO LONGER DOES (#432). + +CONTEXT.md's independence rule: the two harnesses are parallel in functionality and +independent in implementation, and a pytest test that drives `test/lib.sh` is the +first measurement wearing a Python wrapper -- it agrees with the shell by +construction and can never report it wrong. This file was the largest item on that +inventory. + +The fingerprint and manifest arms no longer go through it. Their subject is +`test/pgc_fingerprint.py`, which has been the ONE implementation since #907; +`pgc_source_fingerprint` and `pgc_source_manifest` in lib.sh are thin wrappers that +shell out to exactly that module, so the old path was +python -> bash -> lib.sh -> python3 -> the module. They call the module now, in +process where the property allows and through the module's own CLI where it needs a +different user or locale. Measured before converting a single arm: the fingerprint is +byte-identical through both paths, the manifest is identical line for line, and both +agree across LC_ALL=C, C.UTF-8 and en_US.UTF-8. + +WHAT STILL GOES THROUGH bash, and why each one is not a wrapper to be removed: + + * `pgc_write_source_stamp`, `pgc_source_stamp_path`, `pgc_freshness_report` and + `pgc_freshness_verdict` are pure shell. They are the shell harness's OWN + implementation, not a wrapper over shared code, so an arm here is a second + harness testing the first. Those belong to the shell harness and are the next + step, not this one. + * `test_the_two_fingerprint_implementations_cover_the_same_inputs` reaches across + ON PURPOSE and is the one arm that should. Its subject is that neither side + carries a private copy, which cannot be expressed without touching both. + CONTEXT.md's rule allows exactly this -- "say which, and say why" -- so this is + the saying: it caught four defects in one day (#907), it is the guard that + reddens on the first edit if a private implementation returns, and the property + is the relationship rather than either side. + * `test_the_fix_does_not_rebaseline_stamps_already_on_disk` embeds the PREVIOUS + shell algorithm as a fixture and compares today's value against it. The fixture + is a file the test builds itself, which the rule permits, but it calls + `pgc_source_build_dirs` out of the real lib.sh for its directory list. Feeding it + the module's own `build_dirs` would make it self-contained; that is a change to + a historical-parity arm and it is not in this one. """ +import hashlib +import inspect import os +import sys import tempfile import shutil import pwd @@ -31,6 +73,7 @@ import pytest from pgc_cluster import (build_and_install, build_once, make_cluster, + source_manifest, source_fingerprint) @@ -358,6 +401,66 @@ def test_make_cluster_leaves_nothing_behind_when_setup_fails(tmp_path, expect): "a failed make_cluster leaves no directory behind") +def test_the_cleanup_guard_has_the_shape_the_leak_needs(expect): + """The arm above proves the tree is gone. It cannot prove WHY, and three of + the four properties that make the guard work are invisible to it. + + `make_cluster` fails one way in that test: a missing `pg_config`. The guard + also has to survive a KeyboardInterrupt, stop a postmaster it already started, + and re-raise rather than return None. Each needs a different failure to + observe, and two of them cannot be provoked from a test at all -- you cannot + deliver SIGINT into `initdb` reliably, and a cluster that started is a cluster + this arm would then have to stop. + + So these are SOURCE checks, and they are labelled as such rather than sold as + behavioural. They moved here from `test/selftest/380`, which read this file as + text across the harness boundary. A shell part can pin this text; what it + cannot do is what the arm above does. Reading our own module is not a + cross-harness reference, so the property lives where its subject lives. + """ + import inspect + body = inspect.getsource(make_cluster) + + # Cut to the handler. These shapes occur elsewhere in the module, and a check + # over the whole file reports a guard present that lives in another function. + expect.at_least(len(body.splitlines()), 15, + "premise: the source of the function itself was read") + + for fragment, name in ( + ("except BaseException:", + "the handler catches BaseException, so an interrupt cleans up too"), + ("cluster.stop()", + "and it stops a partially started cluster"), + ("shutil.rmtree(root", + "and it removes the tree"), + ): + expect.num(body.count(fragment), 1, name) + + # The bare re-raise, anchored to a line: `raise` appears inside this function + # in the is_ours() refusal too, as `raise RuntimeError(...)`, and a substring + # count would read that as the re-raise and pass with the re-raise deleted. + bare = [ln for ln in body.splitlines() if ln.strip() == "raise"] + expect.num(len(bare), 1, + "and the original error is re-raised rather than swallowed") + + +def test_this_module_keeps_no_private_fingerprint(expect): + """One implementation, in `test/pgc_fingerprint.py`, since #907. + + A second copy reappearing in a caller is the defect rather than a detail: it + agrees with the shared one until it does not, and nothing says when that was. + Also a source check, and also moved out of `test/selftest/380`. + """ + import pgc_cluster + src = pathlib.Path(inspect.getsourcefile(pgc_cluster)).read_text(encoding="utf-8") + expect.at_least(len(src), 200, "premise: the module's own source was read") + for fragment, name in ( + ("hashlib.md5", "no private md5 digest"), + ('glob("*/Makefile")', "and no private build-directory walk"), + ): + expect.num(src.count(fragment), 0, name) + + # --------------------------------------------------------------------------- # THE PYTEST TWIN of test/selftest/340's stamp arms, owed under jd's rule of # 2026-09-23... 2026-09-09: every test written twice. The .sh half could not @@ -379,6 +482,70 @@ def _sh(srcdir, expr): return p.stdout.strip(), p.returncode +# ---- the fingerprint machinery, called where it LIVES --------------------------- +# +# These used to go python -> bash -> test/lib.sh -> python3 -> test/pgc_fingerprint.py. +# The last step is the ONE implementation (#907); the two before it are lib.sh's thin +# wrapper, so the round trip made every one of these arms a test of the shell harness +# as well as the module. That is the coupling CONTEXT.md's independence rule names: a +# pytest test that drives `test/lib.sh` is the first measurement wearing a Python +# wrapper, so it agrees with the shell by construction and can never report it wrong. +# +# Calling the module is not "testing a different thing". Measured on a fixture tree, +# before any arm was converted: the fingerprint is byte-identical through both paths, +# the manifest is byte-identical line for line, and both agree across LC_ALL=C, +# C.UTF-8 and en_US.UTF-8. What is gone is the wrapper, not the subject. +# +# WHAT STILL GOES THROUGH bash, and why, is listed in the module docstring: the arms +# whose subject IS lib.sh's own behaviour rather than the module's computation. + + +_FP_MODULE = pathlib.Path(__file__).resolve().parent.parent / "pgc_fingerprint.py" + + +def _fp_of(tree): + """The fingerprint of a tree, from the one implementation. + + `""` rather than None for "could not be computed", which is the shape these arms + already compared against when they read it out of the shell. + """ + return source_fingerprint(tree) or "" + + +def _mf_of(tree): + """The manifest of a tree, from the one implementation. + + Returned RAW, so None -- the module's "a digest failed" -- reaches the arm and + breaks it loudly instead of being smoothed into an empty manifest. That + distinction is the module's whole reason for returning None, and an `or ""` here + would be this file re-creating the defect the module exists to refuse. + """ + return source_manifest(tree) + + +def _fp_cli(tree, user="", env=None): + """The fingerprint, from the module's own CLI in a separate process. + + OUT OF PROCESS because two properties need it: reading as an unprivileged user + (root ignores `chmod 000`, so an in-process call cannot see a denied read), and + running under a different locale. Neither needs the shell harness -- the module + ships the CLI that `test/lib.sh` itself invokes, so this is the same entry point + lib.sh uses with lib.sh taken out of the path. + + Its contract, which is why the arms below read the same values they did through + bash: `fingerprint DIR` prints the hash and exits 0, and prints EMPTY when a digest + could not be read. lib.sh's wrapper passes that through unchanged. + """ + argv = [sys.executable, str(_FP_MODULE), "fingerprint", str(tree)] + if user: + argv = ["runuser", "-u", user, "--"] + argv + e = dict(os.environ) + if env: + e.update(env) + p = subprocess.run(argv, capture_output=True, text=True, env=e) + return p.stdout.strip(), p.returncode + + def _tree_with_module(tmp_path, name): t = tmp_path / name (t / "src").mkdir(parents=True, exist_ok=True) @@ -419,11 +586,11 @@ def test_two_installations_of_one_major_do_not_share_a_stamp(tmp_path, expect): def test_moving_bytes_between_files_moves_the_shell_fingerprint(tmp_path, expect): """`xargs -0 cat | md5sum` could not see a repartition.""" t = _tree_with_module(tmp_path, "rp") - before, _ = _sh(t, f'pgc_source_fingerprint "{t}"') + before = _fp_of(t) expect.at_least(len(before), 12, "premise: the tree fingerprints at all") (t / "src" / "a.c").write_text("int a;\nint b;\n") (t / "objstore" / "b.c").write_text("") - after, _ = _sh(t, f'pgc_source_fingerprint "{t}"') + after = _fp_of(t) expect.text(str(after != before), "True", "moving bytes between files moves the fingerprint") @@ -578,24 +745,24 @@ def test_a_failed_digest_yields_no_fingerprint_rather_than_a_wrong_one(expect): return root, t = _readable_tree("t") try: - base, _ = _sh_fp_as(user, f'pgc_source_fingerprint "{t}"') + base, _ = _fp_cli(t, user=user) expect.at_least(len(base), 12, "premise: the tree fingerprints at all") # Premise for the mechanism itself: the unprivileged reader must agree # with a privileged one while nothing is denied, or the arm below would # be measuring the user switch rather than the failure. - mine, _ = _sh_fp_as("", f'pgc_source_fingerprint "{t}"') + mine, _ = _fp_cli(t) expect.text(base, mine, "premise: the unprivileged read agrees while readable") for name in ("b.c", "c.c"): f = t / "src" / name f.chmod(0o000) - got, _ = _sh_fp_as(user, f'pgc_source_fingerprint "{t}"') + got, _ = _fp_cli(t, user=user) f.chmod(0o644) expect.text(got or "empty", "empty", f"an unreadable {name} yields no fingerprint, not a wrong one") - after, _ = _sh_fp_as(user, f'pgc_source_fingerprint "{t}"') + after, _ = _fp_cli(t, user=user) expect.text(after, base, "control: and the tree fingerprints again once readable") finally: shutil.rmtree(root, ignore_errors=True) @@ -615,7 +782,7 @@ def test_a_failed_digest_gives_unverified_and_never_a_false_stale(expect): return root, t = _readable_tree("v") try: - base, _ = _sh_fp_as(user, f'pgc_source_fingerprint "{t}"') + base, _ = _fp_cli(t, user=user) expect.at_least(len(base), 12, "premise: the tree fingerprints at all") (t / "src" / "b.c").chmod(0o000) verdict, _ = _sh_fp_as( @@ -634,7 +801,7 @@ def test_one_tree_hashes_one_way_however_the_path_is_spelled(tmp_path, expect): t = _fp_tree(tmp_path, "s") link = tmp_path / "s_link" link.symlink_to(t) - plain, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') + plain = _fp_of(t) expect.at_least(len(plain), 12, "premise: the fixture fingerprints at all") for label, spelling in ( ("a trailing slash", f"{t}/"), @@ -642,9 +809,14 @@ def test_one_tree_hashes_one_way_however_the_path_is_spelled(tmp_path, expect): ("a /src/.. segment", f"{t}/src/.."), ("a symlink", str(link)), ): - got, _ = _sh_fp(f'pgc_source_fingerprint "{spelling}"') + got = _fp_of(spelling) expect.text(got, plain, f"{label} hashes the same tree the same way") - rel, _ = _sh_fp(f'cd "{t}" && pgc_source_fingerprint .') + _cwd = os.getcwd() + try: + os.chdir(t) + rel = _fp_of(".") + finally: + os.chdir(_cwd) expect.text(rel, plain, "a relative path hashes the same tree the same way") @@ -682,12 +854,12 @@ def test_the_fingerprint_still_moves_on_a_real_change(tmp_path, expect): its input, so the set needs one arm proving the hash still moves. """ t = _fp_tree(tmp_path, "m") - before, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') + before = _fp_of(t) (t / "src" / "a.c").write_text("int a = 2;\n") - after, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') + after = _fp_of(t) expect.text(str(after != before), "True", "a real content change moves the fingerprint") (t / "src" / "a.c").write_text("int a;\n") - restored, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') + restored = _fp_of(t) expect.text(restored, before, "and restoring the content restores it") @@ -706,12 +878,12 @@ def test_a_tree_with_nothing_hashable_reports_no_fingerprint(tmp_path, expect): anything. """ live = _fp_tree(tmp_path, "hollow_premise") - base, _ = _sh_fp(f'pgc_source_fingerprint "{live}"') + base = _fp_of(live) expect.at_least(len(base), 12, "premise: the same helper returns a fingerprint for a real tree") empty = tmp_path / "hollow" empty.mkdir() - got, _ = _sh_fp(f'pgc_source_fingerprint "{empty}"') + got = _fp_of(empty) expect.text(got or "empty", "empty", "an unhashable tree yields no fingerprint") @@ -738,7 +910,7 @@ def test_the_manifest_names_what_the_fingerprint_hashed(tmp_path, expect): answer. """ t = _mf_tree(tmp_path, "mf") - out, _ = _sh_fp(f'pgc_source_manifest "{t}"') + out = _mf_of(t) lines = [l for l in out.splitlines() if l.strip()] expect.num(len(lines), 6, "the manifest names every file the fingerprint hashes") shaped = [l for l in lines if re.fullmatch(r"[A-Za-z0-9_./-]+ [0-9a-f]{32}", l)] @@ -750,8 +922,8 @@ def test_the_manifest_names_what_the_fingerprint_hashed(tmp_path, expect): def test_the_fingerprint_is_the_hash_of_the_manifest(tmp_path, expect): """One is defined as the other, so the two cannot drift apart.""" t = _mf_tree(tmp_path, "mh") - fp, _ = _sh_fp(f'pgc_source_fingerprint "{t}"') - via, _ = _sh_fp(f'pgc_source_manifest "{t}" | md5sum | cut -c1-12') + fp = _fp_of(t) + via = hashlib.md5((_mf_of(t) + "\n").encode()).hexdigest()[:12] expect.text(fp, via, "the fingerprint is the hash of the manifest") @@ -764,9 +936,9 @@ def test_an_added_file_is_named_rather_than_merely_changing_the_hash(tmp_path, e same line. """ t = _mf_tree(tmp_path, "add") - before, _ = _sh_fp(f'pgc_source_manifest "{t}"') + before = _mf_of(t) (t / "src" / "zz_appeared.c").write_text("int zz;\n") - after, _ = _sh_fp(f'pgc_source_manifest "{t}"') + after = _mf_of(t) gained = set(after.splitlines()) - set(before.splitlines()) expect.num(len(gained), 1, "exactly one manifest line appears") expect.text(sorted(gained)[0].split()[0], "src/zz_appeared.c", @@ -906,8 +1078,7 @@ def test_one_tree_hashes_one_way_however_the_locale_is_set(expect): seen = {} for loc in wanted: - got, _ = _sh_fp(f'pgc_source_fingerprint "{t}"', - env={"LC_ALL": loc, "LANG": loc}) + got, _ = _fp_cli(t, env={"LC_ALL": loc, "LANG": loc}) seen[loc] = got expect.at_least(len(seen[wanted[0]]), 12, "premise: the tree fingerprints at all") @@ -948,7 +1119,7 @@ def test_a_symlinked_src_is_skipped_like_any_other_symlinked_build_dir(tmp_path, expect.num(len(list(real.iterdir())), 2, "premise: and the target holds sources find would otherwise hash") - out, _ = _sh_fp(f'pgc_source_manifest "{t}"') + out = _mf_of(t) lines = [l for l in out.splitlines() if l.strip()] expect.num(len([l for l in lines if l.startswith("src/")]), 0, "a symlinked src contributes nothing, as find -P contributes nothing") @@ -960,6 +1131,6 @@ def test_a_symlinked_src_is_skipped_like_any_other_symlinked_build_dir(tmp_path, (real_tree / "src" / "a.c").write_text("int a;\n") (real_tree / "Makefile").write_text("all:\n\ttrue\n") (real_tree / "pgcolumnar.control").write_text("x\n") - out2, _ = _sh_fp(f'pgc_source_manifest "{real_tree}"') + out2 = _mf_of(real_tree) expect.num(len([l for l in out2.splitlines() if l.startswith("src/a.c ")]), 1, "control: a real src directory is still hashed") diff --git a/test/pytest/test_docs_cover_the_corpus.py b/test/pytest/test_docs_cover_the_corpus.py index 5268f084..86240705 100644 --- a/test/pytest/test_docs_cover_the_corpus.py +++ b/test/pytest/test_docs_cover_the_corpus.py @@ -335,9 +335,15 @@ def test_a_stated_total_that_disagrees_with_disk_is_visible(tmp_path, expect): MODES_DOC = HERE / "VACUITY_MODES.md" -def _named_modes(): - """-> (refused, not_refused, all) per section 1a's rule.""" - text = MODES_DOC.read_text() +def _named_modes_in(text): + """-> (refused, not_refused, all) per section 1a's rule, over TEXT. + + A seam, so the rule can be reached with a fixture. `test/selftest/350` had one + and this did not: the rule was implemented twice and self-tested once, on the + side that cannot run the corpus it counts. Its edge cases now live here with it + -- an id of fewer than three words, an id named twice, stopping at the next + heading, and section 3's back-references. + """ chunks = {} for chunk in re.split(r"^## ", text, flags=re.M): head = chunk.splitlines()[0] if chunk.strip() else "" @@ -352,6 +358,24 @@ def _named_modes(): return refused, not_refused, set().union(*chunks.values()) if chunks else set() +def _named_modes(): + """The same rule, over the document on disk.""" + return _named_modes_in(MODES_DOC.read_text()) + + +def _stated_row(text, label): + """-> the number in the VALUE cell of section 1a's row for LABEL, or None. + + The value cell, never the first number on the line. The labels themselves + contain digits -- "named in section 2, refused today" -- so reading the first + number returns the 2 from "section 2". `test/selftest/350` shipped exactly that + bug and read 2 and 3 for totals of 21 and 51, and its own fixture could not see + it because there the label digit and the value were both 2. + """ + m = re.search(rf"\|[^|\n]*{re.escape(label)}[^|\n]*\|\s*\**(\d+)", text) + return None if m is None else int(m.group(1)) + + def test_the_mode_inventory_states_its_own_totals_correctly(expect): """The numbers in section 1a must be the numbers on disk. @@ -454,3 +478,180 @@ def test_the_two_halves_of_the_refused_sentence_sum_to_the_named_total(expect): "the two halves sum to the number of modes the inventory names") expect.num(refused, len(_named_modes()[0]), "and the refused half is the count of ids section 2 claims") + + +# --------------------------------------------------------------------------- +# The counting rule's own edges, and the row reader's. +# +# These moved from `test/selftest/350` (#432). The rule is implemented on both +# sides -- that is the two-harness design, parallel in functionality -- but it was +# SELF-TESTED only on the shell side, which cannot run the corpus it counts. A rule +# with no fixtures is a rule that passes because the document happens to agree with +# it today. +# --------------------------------------------------------------------------- + +_FIXTURE_DOC = """## 1a. Counting + +| named in section 2, refused today | **2** | +| named in section 3, not refused | **1** | + +## 2. Refused + +`one-two-three` and `four-five-six`, and `one-two-three` again. + +## 3. Not refused + +`seven-eight-nine`. And `one-two-three` is now closed. + +## 4. Something else + +`ten-eleven-twelve` +""" + + +def test_the_counting_rule_counts_a_fixture_as_the_document_says(expect): + """Section 2 names two distinct ids; section 3 names one, after its + back-reference to a section 2 id is discounted.""" + refused, not_refused, allm = _named_modes_in(_FIXTURE_DOC) + expect.text(", ".join(sorted(refused)), "four-five-six, one-two-three", + "section 2's ids, deduplicated") + expect.text(", ".join(sorted(not_refused)), "seven-eight-nine", + "section 3's ids, minus the back-reference section 2 already claims") + expect.num(len(allm), 4, "and every id in the document is seen once") + + +def test_an_id_of_fewer_than_three_words_is_not_a_mode(expect): + """The rule is three or more words. Two is an ordinary backticked phrase, and + counting it would make every `foo-bar` in prose a mode.""" + doc = "## 2. Refused\n\n`one-two` and `alpha` and `one-two-three`\n" + refused, _, _ = _named_modes_in(doc) + expect.text(", ".join(sorted(refused)), "one-two-three", + "only the three-word id counts") + + +def test_the_counter_stops_at_the_next_heading(expect): + """Section 2's count must not reach into section 4. Without the stop, every id + below section 2 would be refused, and the totals would agree with nothing.""" + refused, _, _ = _named_modes_in(_FIXTURE_DOC) + expect.text("ten-eleven-twelve" in refused and "yes" or "no", "no", + "an id in a later section is not counted as refused") + + +def test_the_row_reader_takes_the_value_not_a_digit_in_the_label(expect): + """The labels contain digits. Reading the first number on the line returns the + 2 from "section 2" -- which `test/selftest/350` did, reporting 2 and 3 for + totals of 21 and 51. The fixture there could not see it, because the label + digit and the value were both 2. + + So this fixture makes them DIFFER: the label says "section 2" and the value is + 9. A reader that takes the label's digit answers 2 and fails here. + """ + doc = "| named in section 2, refused today | **9** |\n" + expect.num(_stated_row(doc, "refused today"), 9, + "the value cell is read, not the digit inside the label") + + +def test_an_absent_row_is_none_rather_than_a_number_that_happens_to_match(expect): + """A missing row must be distinguishable from a row stating zero. Returning 0 + for both would make a document that states nothing agree with a corpus that + counts nothing.""" + expect.text(repr(_stated_row("## 1a. Counting\n\nno table here\n", "refused today")), + "None", "an absent row reads as None") + expect.num(_stated_row("| refused today | **0** |\n", "refused today"), 0, + "and a row stating zero reads as 0") + + +def test_a_stated_total_that_disagrees_with_the_ids_is_visible(expect): + """The arm this whole block protects: the document's number against the ids on + disk. With the fixture's own table it agrees; change the table and it must not.""" + refused, not_refused, _ = _named_modes_in(_FIXTURE_DOC) + expect.num(_stated_row(_FIXTURE_DOC, "refused today"), len(refused), + "control: the fixture's stated refused total is its id count") + wrong = _FIXTURE_DOC.replace("refused today | **2** |", "refused today | **7** |") + expect.differ(_stated_row(wrong, "refused today"), len(refused), + "and a disagreeing total does not compare equal") + + +# --------------------------------------------------------------------------- +# Every contents-list link must reach a heading. +# +# Moved from `test/selftest/350` (#432), where the subject was this directory's own +# documents. TESTS.md's contents list gained an entry whose anchor stripped the +# underscores out of a file name -- `#14-testharnessdepspy-…` against a heading +# GitHub renders as `#14-test_harness_depspy-…` -- so the link went nowhere while +# eleven entries above it kept the underscores. Neither coverage sweep could see +# it: both look for NAMES, and a broken link still contains the name it points at. +# A reader finds out by clicking. +# +# ONE DIRECTION ON PURPOSE. Every link must reach a heading; the reverse needs an +# exemption list, because "## Contents" is a heading no entry links to, and a +# hand-maintained exemption list is the thing this corpus keeps deleting. +# --------------------------------------------------------------------------- + +_LINK = re.compile(r"\]\(#([A-Za-z0-9_-]+)\)") +_HEADING = re.compile(r"^#{2,} +(.+)$", re.M) + + +def _github_anchor(heading): + """GitHub's rule: lowercase, drop anything that is not a letter, digit, space, + hyphen or underscore, then spaces to hyphens. The dot in `.py` and the colon + after it go; the underscores stay.""" + kept = [c for c in heading.lower() if c.isalnum() or c in " _-"] + return "".join(kept).replace(" ", "-") + + +def _unresolved_links(text): + """-> sorted anchors that no heading in TEXT produces.""" + have = {_github_anchor(h) for h in _HEADING.findall(text)} + return sorted({a for a in _LINK.findall(text) if a not in have}) + + +def test_the_anchor_rule_drops_punctuation_and_keeps_underscores(expect): + """The derivation, on the heading the defect was found in.""" + # Assembled, for the reason given in the arm below: a literal corpus file name + # in a string makes the membership classifier read this file as driving that + # one. The anchor on the right keeps its literal form, because the dot is gone + # from it and so it is no longer that file's name. + heading = "14. " + "test" + "_harness" + "_deps" + ".py: the harness must self-test" + expect.text(_github_anchor(heading), + "14-test_harness_depspy-the-harness-must-self-test", + "the dot and the colon go, the underscores stay") + + +def test_an_anchor_that_strips_the_underscores_is_caught(expect): + """The exact shape that shipped, and a control beside it. Without the control a + sweep that matches nothing reports the same clean answer.""" + # THE FILENAME IS ASSEMBLED, NOT WRITTEN. A literal `test_harness_deps.py` in + # this string makes `_mentioned_files` in that very file read this one as a + # driver of it, and the membership classifier then calls this file + # cluster-bound. That is CONTEXT.md's rule 3 -- "a word that merely looks like + # a filename" -- and it cost two red arms before the cause was found. The + # heading here is sample markdown, not a reference. + _f = "test" + "_harness" + "_deps" + ".py" + good = (f"## 14. {_f}: the harness\n" + "- [14. x](#14-test_harness_depspy-the-harness)\n") + bad = (f"## 14. {_f}: the harness\n" + "- [14. x](#14-testharnessdepspy-the-harness)\n") + expect.text(", ".join(_unresolved_links(good)) or "none", "none", + "control: an anchor that keeps the underscores resolves") + expect.text(", ".join(_unresolved_links(bad)), + "14-testharnessdepspy-the-harness", + "and one that strips them is named rather than passed over") + + +def test_every_in_document_link_in_this_directory_reaches_a_heading(expect): + """The real sweep, over every document beside this file. + + With a coverage premise: a sweep over an empty file list reports nothing + broken, which is what a correct set of documents reports too. + """ + docs = sorted(HERE.glob("*.md")) + expect.at_least(len(docs), 3, "premise: the sweep found this directory's documents") + total_links = 0 + for doc in docs: + text = doc.read_text(encoding="utf-8") + total_links += len(_LINK.findall(text)) + expect.text(", ".join(_unresolved_links(text)) or "none", "none", + f"every in-document link in {doc.name} reaches a heading") + expect.at_least(total_links, 15, + "premise: and it parsed links rather than finding none") diff --git a/test/pytest/test_guards_pinned.py b/test/pytest/test_guards_pinned.py index e69655d6..4840670f 100644 --- a/test/pytest/test_guards_pinned.py +++ b/test/pytest/test_guards_pinned.py @@ -292,3 +292,47 @@ def test_empty_directions(expect): expect.ordering_observable([], [], "no rows either way") '''), "ordering_observable names its own both-empty refusal", "both directions are empty") + +# ---- plan_marker's refusal must stay ahead of the arms it protects -------------- +# +# MOVED HERE FROM test/selftest/370 (#432). That part read `pgc_vacuity.py`'s TEXT from +# the shell harness, which is the coupling CONTEXT.md's independence rule forbids: a +# shell arm asserting a text pin cannot prove a python arm is caught, which is why #927 +# deleted the part whose subject was a python module's source rather than repairing it. +# Python reading its OWN module is not a cross-harness reference -- it is one harness +# checking itself -- so the property keeps its home and loses the boundary crossing. +# +# IT IS A SOURCE CHECK BECAUSE THE ORDERING IS NOT OBSERVABLE TODAY, and that is worth +# stating rather than implying. Measured: with the refusal moved to the very END of +# `plan_marker`, `plan_marker([], absent=True)` STILL refuses. `plan_marker` has no early +# return -- the passing case falls off the end -- so the refusal fires wherever it sits, +# and no input distinguishes the orders. +# +# 370's stated reason was that "an empty plan leaves found=False and the absent arm +# returns a pass first". The absent arm does not return; it falls through. So the defect +# that comment describes cannot arise in this shape, and the behavioural arm above +# (`test_plan_marker_refuses_an_absence_claim_over_an_empty_plan`) does not redden when +# the order changes -- I checked, expecting it to, and it did not. +# +# The arm is therefore PROSPECTIVE: it is insurance against a refactor that adds an early +# return, after which the order would decide whether an empty plan passes. That is a real +# risk and cheap to pin, but it is a claim about source order and it says so. + + +def test_the_empty_plan_refusal_precedes_the_arms_it_protects(expect): + """The refusal must sit before the arms, so a future early return cannot skip it.""" + import inspect + + # THE CLASS COMES OFF THE FIXTURE, so this file still imports nothing: the + # `expect` object IS an instance of the layer being read. + body = inspect.getsource(type(expect).plan_marker) + lines = body.split("\n") + i_refusal = next((n for n, l in enumerate(lines) if "if not nodes:" in l), None) + i_absent = next((n for n, l in enumerate(lines) if "if absent and found:" in l), None) + i_present = next((n for n, l in enumerate(lines) + if "if not absent and not found:" in l), None) + expect.text(f"{i_refusal is not None} {i_absent is not None} {i_present is not None}", + "True True True", + "premise: all three were found, so the ordering can mean something") + expect.num(int(i_refusal < i_absent and i_refusal < i_present), 1, + "the empty-plan refusal precedes both arms it protects") diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index 110aa9b8..2efed35d 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -871,3 +871,143 @@ def test_the_job_installs_no_database_driver(expect): if __name__ == "__main__": sys.exit(_main(sys.argv[1:])) + +# ---- the harness-independence inventory, as a mechanism (#432) ------------------ +# +# CONTEXT.md: the two harnesses are parallel in functionality and independent in +# implementation, and they must not call, import or reference each other. Its inventory +# of what still does was PROSE -- falsifiable by hand, but nothing reddened when a new +# reference appeared. #923 nearly landed a fourth coupled file, and the review that +# caught it was a person reading, not an arm. +# +# So the inventory is declared here and asserted in BOTH directions. A new file that +# reaches into the shell harness reddens this; a file that stops reaching and is not +# removed from the declaration reddens it too, which is what stops the list rotting +# into a permanent exemption. +# +# A FILE-LEVEL GUARD, which is what CONTEXT.md's rule asks for -- "count files, not +# lines" -- and also the most this can honestly be. Within a flagged file it cannot tell +# a path joined onto the REAL tree from the same name joined onto a `tmp_path`: both are +# the string "lib.sh" and only the dataflow says which. test_build_refusal.py contains +# both, and CONTEXT.md already records that the fake ones are rule 2 rather than +# references. So the assertion is over the SET OF FILES, and the per-file entry carries +# the mechanism a reader needs to check it by hand. +# +# THE COUNTING RULE IS CONTEXT.md'S, and the three exclusions are the ones it names, +# in the order it says they get confused: prose is not a reference, a file the test +# BUILDS ITSELF under tmp_path is not a reference even with the same name, and a word +# that merely looks like a filename is not one. Docstrings are dropped by AST position +# rather than by pattern, which is the only way to tell the first from a string the +# code actually passes to bash. + +# THE DESCRIPTIONS NAME NO FILE, and that is not squeamishness: the first version +# spelled the helper library's path in them, and the detector flagged THIS file for its +# own inventory -- four files where the tree has three. The mechanism is what the entry +# has to carry; the filenames are in CONTEXT.md, which is prose and may name anything. +SHELL_REFERENCES = { + "pgc_cluster.py": + "sources the shell harness's helper library to call its build-and-install " + "function, so the build refusal has one implementation rather than two", + "test_build_refusal.py": + "sources that library for the pure-shell stamp and freshness helpers, and " + "for the one permitted cross-implementation arm; see its module docstring", + "test_suite_accounting.py": + "reads the matrix runner's text and executes the real runner", + "test_mutation_ledger.py": + "executes the matrix runner with its list flag to get the registered suite " + "list, which is the same mechanism the entry above uses", +} + +_SHELL_NAMES = re.compile( + r"(^|/)(lib\.sh|run_all_versions\.sh|portlib\.sh|harness_selftest\.sh)" + r"|(^|/)test/[A-Za-z0-9_]+\.sh" + r"|(^|/)selftest/") + + +def _docstring_nodes(tree): + """Every Constant that is a docstring, by position rather than by content.""" + out = set() + for node in ast.walk(tree): + if isinstance(node, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, + ast.ClassDef)): + body = getattr(node, "body", None) + if body and isinstance(body[0], ast.Expr) \ + and isinstance(body[0].value, ast.Constant) \ + and isinstance(body[0].value.value, str): + out.add(id(body[0].value)) + return out + + +def _shell_reference_sites(source): + """-> [(lineno, text)] for executable strings naming a shell file in this tree. + + An f-string's PIECES are Constants of their own, so counting both the JoinedStr and + its parts reported one reference as two. Measured: the fixture below asserted 1 and + the first version answered 2. + """ + tree = ast.parse(source) + skip = _docstring_nodes(tree) + for node in ast.walk(tree): + if isinstance(node, ast.JoinedStr): + for piece in node.values: + if isinstance(piece, ast.Constant): + skip.add(id(piece)) + hits = [] + for node in ast.walk(tree): + if isinstance(node, ast.Constant) and isinstance(node.value, str): + if id(node) in skip: + continue + if _SHELL_NAMES.search(node.value): + hits.append((node.lineno, node.value[:60])) + elif isinstance(node, ast.JoinedStr): + txt = "".join(v.value for v in node.values + if isinstance(v, ast.Constant) and isinstance(v.value, str)) + if _SHELL_NAMES.search(txt): + hits.append((node.lineno, txt[:60])) + return sorted(set(hits)) + + +# THE FIXTURES ARE ASSEMBLED, never written out, and the first version of this arm got +# that wrong: spelling the shell file's name in its own test data made THIS FILE a +# referencing file, and the inventory arm below reported four files where the tree has +# three. A scan flagging its own fixtures is the third time that shape has cost me a +# measurement today, so the name is built from fragments no single string contains. +_LIB = "test/" + "lib" + ".sh" + + +def test_the_shell_reference_detector_sees_code_and_not_prose(expect): + """The premise, and the distinction the whole arm below rests on.""" + expect.num(len(_shell_reference_sites( + 'def f():\n """This drives ' + _LIB + ' and says so."""\n return 1\n')), 0, + "a docstring naming a shell file is prose, not a reference") + expect.num(len(_shell_reference_sites( + 'import subprocess\nsubprocess.run(["bash", "-c", ". ' + _LIB + '"])\n')), 1, + "a string the code passes to bash is a reference") + expect.num(len(_shell_reference_sites( + 'srcdir = "/x"\nscript = f\'. "{srcdir}/' + _LIB + '" || exit 1\'\n')), 1, + "and so is one built with an f-string, counted ONCE rather than per piece") + expect.num(len(_shell_reference_sites('sharedir = "/usr/share"\n')), 0, + "a word that merely looks like a path is not a reference") + + +def test_the_harness_independence_inventory_is_exactly_what_the_corpus_does(expect): + """CONTEXT.md's inventory, asserted in both directions. + + A file that starts reaching into the shell harness reddens this. A file that stops + and is left in the declaration reddens it too -- otherwise the list becomes a + permanent exemption that outlives the coupling it was written for, which is how + every hand-maintained exempt list in this tree has gone wrong. + """ + here = pathlib.Path(__file__).parent + files = sorted(here.glob("*.py")) + expect.at_least(len(files), 10, "premise: the scan has a corpus to read") + found = {} + for f in files: + sites = _shell_reference_sites(f.read_text(encoding="utf-8")) + if sites: + found[f.name] = sites + expect.text(repr(sorted(found)), repr(sorted(SHELL_REFERENCES)), + "the files that reach into the shell harness are exactly the declared ones") + for name in sorted(found): + expect.at_least(len(SHELL_REFERENCES.get(name, "")), 20, + f"{name}'s entry says by what mechanism it reaches across") diff --git a/test/selftest/350-the-pytest-corpus-must-be.sh b/test/selftest/350-the-pytest-corpus-must-be.sh index 88ce1a0a..c178ba56 100644 --- a/test/selftest/350-the-pytest-corpus-must-be.sh +++ b/test/selftest/350-the-pytest-corpus-must-be.sh @@ -1,398 +1,59 @@ -# ---- the pytest corpus must be documented, or the documentation is decoration - +# ---- the gate must run the pytest corpus's guards ---------------------------- +# +# WHAT THIS PART IS NOW (#432). It used to hold three groups of arms over +# test/pytest/: that every file and test is named in TESTS.md, that every +# contents-list link resolves, and that VACUITY_MODES.md's section 1a counts itself +# correctly. Fifty checks, and the subject of forty-six of them lived on the other +# side of the harness boundary. +# +# THE TWO HARNESSES ARE PARALLEL IN FUNCTIONALITY AND MUST NOT REFERENCE EACH OTHER +# (jd, 2026-09-10). Every one of those arms either globbed test/pytest/*.py and +# parsed Python out of it, or swept the markdown in that directory. The pytest +# corpus asserts all three properties natively in test_docs_cover_the_corpus.py, +# where the subject is, and the shell copy could only ever agree with it. +# +# WHAT MOVED RATHER THAN BEING DELETED. Two of the three rules were implemented on +# both sides and SELF-TESTED only here -- the side that cannot run the corpus it +# counts. Deleting those fixtures would have left the python implementation with no +# fixtures at all, so they went with them: +# +# the mode-counting rule's edges an id of fewer than three words, an id named +# twice, stopping at the next heading, section +# 3's back-references, and the row reader taking +# the value cell rather than a digit in its label +# the contents-list anchor rule GitHub's derivation, the broken link that +# shipped, and a control beside it +# +# AND THE REASON THE DOCUMENT GAVE FOR KEEPING THE SHELL COPY WAS STALE. TESTS.md +# section 6 said "Nothing runs pytest. Not run_all_versions.sh, not any workflow +# under .github/", and concluded that the .sh half was therefore the enforcement. +# CI has a pytest-guards job. That argument was stale in three places at once -- +# there, in selftest/360 and in selftest/380 -- each claiming the behavioural half +# could not run in CI. Nothing was wrong; the reason was, and a stale reason for +# keeping coverage in the wrong place is harder to find than a missing check, +# because nothing reddens. +# +# WHAT REMAINS HERE IS THE CI WORKFLOW, which belongs to neither harness. A shell +# part may read it for the same reason it may read the Makefile: ci.yml is not a +# test suite, and no pytest arm can assert that the gate runs pytest without +# assuming the very thing in question. +# +# THE MEMBERSHIP DECISION AND THE CONFTEST IMPORT ARE STILL NOT CHECKED HERE, and +# that was already jd's rule rather than an omission. An earlier version of this +# part read test/pytest/conftest.py and INVOKED test_harness_deps.py with +# `python3 ... --disagree`, which is the coupling rather than a second measurement: +# a shell arm driving the python decider agrees with it by construction and can +# never report it wrong. Both properties are asserted in the corpus: # -# WHY THIS EXISTS. `test/pytest/TESTS.md` says its job is "what each test asserts, -# and why it exists". It went stale inside a single rework: the corpus grew from -# 25 tests in three files to 54 in five, and the two new files -- 29 tests, every -# one of them added by the rework that answered a review -- were named nowhere in -# it. The header still read "Twenty-five tests in three files". -# -# That is worse than an undocumented directory. A reader who opens a file whose -# stated purpose is completeness does not then go and count the tests, so a -# partial index reads as a total one. The header was not merely out of date, it -# was a false statement of coverage, which is the same defect class the vacuity -# layer in that corpus exists to refuse one level down. -# -# WHY HERE AND NOT IN THE CORPUS ITSELF. Nothing runs pytest. `SUITES` in -# run_all_versions.sh does not list it and no workflow in .github/ invokes it -# (README.md in that directory records the decision and its price). A guard -# written as a pytest test would therefore never run in the gate, and a guard -# that does not run is a comment. `harness_selftest` IS registered, so this is -# the only place the rule can actually bite. The corpus carries a twin of this -# check for anyone running it by hand; this copy is the one with teeth. -# -# WHAT IS POLICED. Three properties, each mechanical: -# * every test_*.py file in the corpus is named in TESTS.md -# * every `def test_` in those files is named in TESTS.md -# * the totals TESTS.md states are the totals on disk -# The third is what the stale header got wrong, and neither of the first two -# would have caught it: a file can name every test and still miscount them. - -_dcv_dir="$PGC_TESTDIR/pytest" -_dcv_doc="$_dcv_dir/TESTS.md" - -check "premise: the pytest corpus is where this part thinks it is" \ - "$([ -d "$_dcv_dir" ] && echo yes || echo no)" "yes" - -check "premise: the corpus carries the documentation this part polices" \ - "$([ -f "$_dcv_doc" ] && echo yes || echo no)" "yes" - -# The sweep, as a function over a DIRECTORY and a DOC, so the arms below can run -# the identical logic over a fixture. A guard that can only be pointed at the -# real tree is proved by nothing: it passes today and there is no way to see it -# fire. selftest 320 makes the same move with the runner's classifier. -# -# Prints the offenders, capped, in the "[]" / "[n:...]" shape the other parts use -# so a failure names what is wrong rather than only that something is. -_dcv_missing() { # _dcv_missing DIR DOC -> "[]" or "[n: a b c]" - local dir="$1" doc="$2" f base name n=0 bad="" doctext - doctext="$(cat "$doc" 2>/dev/null)" - for f in "$dir"/test_*.py; do - # Without nullglob an unmatched glob stays literal, so a corpus with no - # test files would iterate once over a path that does not exist. The - # file test guards that; the premise below asserts the sweep saw files. - [ -f "$f" ] || continue - base="${f##*/}" - case "$doctext" in - *"$base"*) ;; - *) n=$((n + 1)); [ "$n" -le 6 ] && bad="$bad $base" ;; - esac - while IFS= read -r name; do - [ -n "$name" ] || continue - case "$doctext" in - *"$name"*) ;; - *) n=$((n + 1)); [ "$n" -le 6 ] && bad="$bad $name" ;; - esac - done < <(grep -oE '^def (test_[A-Za-z0-9_]+)' "$f" | sed 's/^def //') - done - [ "$n" -eq 0 ] && { printf '[]'; return; } - printf '[%d:%s]' "$n" "$bad" -} - -_dcv_count() { # _dcv_count DIR -> " " - local dir="$1" f t=0 c=0 - for f in "$dir"/test_*.py; do - [ -f "$f" ] || continue - c=$((c + 1)) - t=$((t + $(grep -cE '^def test_' "$f"))) - done - printf '%d %d' "$t" "$c" -} - -# A sweep that found nothing reports "nothing missing" and is indistinguishable -# from a sweep that works. Assert it saw the corpus before believing its verdict. -_dcv_seen="$(_dcv_count "$_dcv_dir")" -check "premise: the sweep found the corpus rather than an empty glob" \ - "$([ "${_dcv_seen%% *}" -ge 20 ] && [ "${_dcv_seen##* }" -ge 3 ] && echo enough || echo "$_dcv_seen")" \ - "enough" - -check "every test file and every test in the corpus is named in TESTS.md" \ - "$(_dcv_missing "$_dcv_dir" "$_dcv_doc")" "[]" - -# TESTS.md STATES NO TOTALS, AND THAT IS THE POINT (#908). -# -# It used to, and this part compared the stated pair against the corpus, which is -# what made it a claim rather than decoration. The problem was never the check: it -# was that the number was WRITTEN rather than DERIVED, and its correct value is a -# function of the MERGE rather than of either branch. It collided on essentially -# every rebase touching the corpus -- ten times in one day, both sides wrong every -# time, so there was no side to pick. -# -# REMOVING IT COSTS NOTHING, provably. The two sweeps are strictly stronger than -# any count: the arm above requires every test on disk to be NAMED here, and the -# reverse arm below requires every name here to EXIST on disk. Two subsets in -# opposite directions is set equality, so any count over the document equals the -# count over the corpus. A stated total was a derived value maintained by hand. -_dcv_stated="$(grep -oE '^\*\*[0-9]+ tests in [0-9]+ files\.\*\*' "$_dcv_doc" \ - | head -1 | grep -oE '[0-9]+' | tr '\n' ' ' | sed 's/ $//')" - -check "TESTS.md states no totals line for a merge to get wrong" \ - "$([ -z "$_dcv_stated" ] && echo none || echo "$_dcv_stated")" "none" - -# PREMISE: the reader must still be able to FIND a totals line, or the arm above -# passes because the parser is broken rather than because the line is gone -- -# which is the shape this whole part exists to refuse. -# $_dcv_fix is not created until the failure-proof section below, and this file -# runs under `set -u`, so this fixture makes its own path rather than borrowing -# one that does not exist yet. -_dcv_ht="$PGC_WORKDIR/doccov-hastotals"; mkdir -p "$_dcv_ht" -printf '**7 tests in 3 files.** and prose\n' > "$_dcv_ht/HASTOTALS.md" -check "premise: the reader still finds a totals line when one is there" \ - "$(grep -oE '^\*\*[0-9]+ tests in [0-9]+ files\.\*\*' "$_dcv_ht/HASTOTALS.md" \ - | head -1 | grep -oE '[0-9]+' | tr '\n' ' ' | sed 's/ $//')" "7 3" - -# And the counts still REACH a reader -- they move to this run's output, where -# they are computed from the corpus and cannot go stale. -echo " CORPUS: $_dcv_seen (test functions, files)" - -# ---- and the guard must be able to FAIL -------------------------------------- -# -# Everything above passes on a healthy tree, which is exactly what a guard that -# does nothing also does. These arms run the same two functions over fixtures -# built to be wrong, so a future edit that neuters the sweep reddens here even -# while the real corpus stays clean. - -_dcv_fix="$PGC_WORKDIR/doccov"; rm -rf "$_dcv_fix"; mkdir -p "$_dcv_fix" -printf 'def test_alpha(expect):\n pass\ndef test_beta(expect):\n pass\n' \ - > "$_dcv_fix/test_one.py" - -# Documented completely: file named, both tests named, totals stated. -printf '**2 tests in 1 files.**\ntest_one.py: test_alpha and test_beta\n' \ - > "$_dcv_fix/GOOD.md" -check "control: a fully documented corpus reports nothing missing" \ - "$(_dcv_missing "$_dcv_fix" "$_dcv_fix/GOOD.md")" "[]" - -# One test left out. This is the exact shape that shipped. -printf '**2 tests in 1 files.**\ntest_one.py: test_alpha\n' > "$_dcv_fix/PARTIAL.md" -check "an undocumented test is named rather than passed over" \ - "$(_dcv_missing "$_dcv_fix" "$_dcv_fix/PARTIAL.md")" "[1: test_beta]" - -# A whole file left out, which is how 29 tests went missing at once. -printf '**2 tests in 1 files.**\nnothing about the corpus at all\n' > "$_dcv_fix/NONE.md" -check "an undocumented file is caught along with the tests inside it" \ - "$(_dcv_missing "$_dcv_fix" "$_dcv_fix/NONE.md")" "[3: test_one.py test_alpha test_beta]" - -# The count arm, proved separately: a doc can name every test and still state a -# wrong total, which is precisely what the stale header did. -check "the sweep counts the fixture's tests and files" \ - "$(_dcv_count "$_dcv_fix")" "2 1" - -# Kept although the real document no longer states a total: it proves the -# comparison still works, so "no totals line" above is a fact about the document -# rather than about a broken reader. -check "a stated total that disagrees with disk is visible" \ - "$([ "$(grep -oE '^\*\*[0-9]+ tests in [0-9]+ files\.\*\*' "$_dcv_fix/GOOD.md" \ - | grep -oE '[0-9]+' | tr '\n' ' ' | sed 's/ $//')" = "$(_dcv_count "$_dcv_fix")" ] \ - && echo agrees || echo differs)" "agrees" - - -# ---- and the sweep must go the OTHER way too (#908) ------------------------- -# -# `_dcv_missing` above computes tests on disk the document fails to name. NOTHING -# computed the reverse, so a test DELETED or RENAMED while its entry survived was -# held by the totals line and by nothing else -- and that line is a merge target -# whose correct value is a function of the merge, so it is the half most likely to -# be removed. Removing it while this direction was uncovered would have retired a -# check silently, which is the move this part exists to prevent. -# -# IT FOUND TWO ON THE SHIPPED CORPUS. Section 3 named -# test_layer_rejects_an_absence_assertion_over_an_empty_plan and a control beside -# it, and neither had ever been written; the real work lives in -# test_guards_pinned.py under a different name and is documented correctly in its -# own section. Two rows claimed coverage under names nobody had written. -# -# A BACKTICKED NAME, not any occurrence. This document discusses fixtures and dead -# names in prose, and a bare-word sweep would report those as missing. Backticks -# are how it already marks a real identifier, so backticking a name IS the claim -# that it exists -- which is why the paragraph describing this defect writes the -# dead names without them. - -_dcv_absent() { # _dcv_absent DIR DOC -> "[]" or "[n: a b c]" - local dir="$1" doc="$2" name n=0 bad="" - local ondisk - # Every function and every file the corpus actually has, one per line. - ondisk="$( { grep -hoE '^def (test_[A-Za-z0-9_]+)' "$dir"/test_*.py 2>/dev/null \ - | sed 's/^def //' - for f in "$dir"/test_*.py; do [ -f "$f" ] && printf '%s\n' "${f##*/}"; done - } | sort -u )" - while IFS= read -r name; do - [ -n "$name" ] || continue - # grep -cxF on a here-string, NOT `printf ... | grep -qxF`. grep -q exits - # the moment it matches, the printf takes EPIPE, and under this suite's - # `set -o pipefail` the pipeline reports failure though the name WAS - # present -- so a name in the corpus is reported absent. Selftest 080 - # states this rule and demonstrates it; its sweep is deliberately - # non-recursive and never looked in here. - # - # It is not latent. It reddened #923's `suites (PG 17)` on a name that - # exists, while PG 18 passed, and reproduces at this corpus size only - # under load: 170 names, 400 trials on a busy machine, 6 false absences - # piped and 0 on a here-string. An independent run of the same shape in - # isolation gave 10 in 40, so the rate is load- and size-dependent rather - # than fixed -- the two measurements bracket it. - [ "$(grep -cxF "$name" <<<"$ondisk" || true)" -ne 0 ] && continue - n=$((n + 1)); [ "$n" -le 6 ] && bad="$bad $name" - done < <(grep -oE '`test_[A-Za-z0-9_]*(\.py)?`' "$doc" 2>/dev/null \ - | tr -d '`' | sort -u) - [ "$n" -eq 0 ] && { printf '[]'; return; } - printf '[%d:%s]' "$n" "$bad" -} - -check "premise: the reverse sweep reads backticked names at all" \ - "$([ "$(grep -coE '`test_[A-Za-z0-9_]*(\.py)?`' "$_dcv_doc")" -ge 20 ] \ - && echo enough || echo too-few)" "enough" - -check "every test the document names exists in the corpus" \ - "$(_dcv_absent "$_dcv_dir" "$_dcv_doc")" "[]" - -# ---- and it must be able to FAIL, on a fixture rather than on the tree ------- - -printf 'def test_alpha(expect):\n pass\n' > "$_dcv_fix/test_one.py" - -printf '`test_one.py`: `test_alpha` and `test_beta`\n' > "$_dcv_fix/GHOST.md" -check "a documented test that does not exist is named, not passed over" \ - "$(_dcv_absent "$_dcv_fix" "$_dcv_fix/GHOST.md")" "[1: test_beta]" - -printf '`test_one.py`: `test_alpha`\n' > "$_dcv_fix/REAL.md" -check "control: a document naming only what exists is clean" \ - "$(_dcv_absent "$_dcv_fix" "$_dcv_fix/REAL.md")" "[]" - -printf '`test_one.py` and `test_gone.py`: `test_alpha`\n' > "$_dcv_fix/GONEFILE.md" -check "a documented file that does not exist is named" \ - "$(_dcv_absent "$_dcv_fix" "$_dcv_fix/GONEFILE.md")" "[1: test_gone.py]" - -# A name in prose without backticks is not a claim, so it must NOT be reported -- -# otherwise the document could never describe a test it removed. -printf 'the old test_beta was removed; `test_alpha` remains\n' > "$_dcv_fix/PROSE.md" -check "an unbackticked name in prose is not treated as a claim" \ - "$(_dcv_absent "$_dcv_fix" "$_dcv_fix/PROSE.md")" "[]" - -unset -f _dcv_absent - -# ---- and the names must be UNIQUE, or the equality argument does not hold ---- -# -# The two sweeps give equality of the two NAME SETS. That is not equality of -# DEFINITION COUNTS, which is what a total counts: two files defining one name -# leave both arms green while the counts differ (@jdatcmd, against #919). It -# cannot happen today, so the conclusion was true in fact and not by -# construction -- the difference between an argument and a guard. -# -# It closes something real beyond the argument: `_dcv_missing` asks whether a -# name appears in the document AT ALL, so a test defined TWICE and documented -# ONCE reads as fully covered while pytest runs both. - -_dcv_dupes() { # _dcv_dupes DIR -> "" or the repeated names - grep -hoE '^def (test_[A-Za-z0-9_]+)' "$1"/test_*.py 2>/dev/null \ - | sed 's/^def //' | sort | uniq -d -} - -check "no test name is defined twice in the corpus" \ - "$(_dcv_dupes "$_dcv_dir" | tr '\n' ' ' | sed 's/ $//')" "" - -# And it must be able to FAIL, on a fixture rather than on the tree. -printf 'def test_shared_shape(expect):\n pass\n' > "$_dcv_fix/test_a.py" -printf 'def test_shared_shape(expect):\n pass\n' > "$_dcv_fix/test_b.py" -check "a name defined in two files is named, not passed over" \ - "$(_dcv_dupes "$_dcv_fix" | tr '\n' ' ' | sed 's/ $//')" "test_shared_shape" -rm -f "$_dcv_fix/test_a.py" "$_dcv_fix/test_b.py" - -check "control: distinct names in the same corpus report no duplicate" \ - "$(_dcv_dupes "$_dcv_dir" | tr '\n' ' ' | sed 's/ $//')" "" - -unset -f _dcv_dupes - - -# ---- and every table-of-contents link must RESOLVE --------------------------- -# -# WHY THIS EXISTS. TESTS.md's contents list gained an entry whose anchor stripped -# the underscores out of the file name -- "#14-testharnessdepspy-..." against a -# heading GitHub renders as "#14-test_harness_depspy-..." -- so the link silently -# went nowhere. Eleven entries above it keep the underscores, so the document -# already stated the convention and the new entry simply disagreed with it. -# -# NEITHER EXISTING ARM COULD SEE IT. Both sweep for NAMES; an anchor is not a -# name, and a broken link is still a string containing the file name it points at. -# A reader finds out by clicking. -# -# THE RULE IS GITHUB'S, and it is mechanical: lowercase the heading text, drop -# every character that is not a letter, digit, space, hyphen or underscore, then -# turn spaces into hyphens. So the dot in ".py" and the colon after it disappear -# and the underscores stay. Over the three documents in this directory the sweep -# reports nothing, and over the document as it shipped it reported exactly the one -# entry -- which is the whole false-positive budget, measured rather than assumed. -# -# ONE DIRECTION ON PURPOSE: every link must reach a heading. The reverse, every -# heading must be linked, is a different property, and "## Contents" is itself a -# heading that no entry links to -- so the reverse needs an exemption list, which -# is the hand-maintained value this file keeps removing. - -_toc_anchor() { # _toc_anchor TEXT -> the anchor GitHub derives from it - printf '%s' "$1" | tr 'A-Z' 'a-z' | sed -e 's/[^a-z0-9 _-]//g' -e 's/ /-/g' -} - -_toc_unresolved() { # _toc_unresolved DOC -> "[]" or "[n: a b c]" - local doc="$1" anchor n=0 bad="" anchors - # Every anchor the document's own headings produce, one per line. - # Through _toc_anchor, so GitHub's rule has ONE definition here. The stream - # form was a second copy of the same sed program, and two copies drift. - anchors="$(while IFS= read -r _h; do [ -n "$_h" ] && _toc_anchor "$_h" && echo; done \ - < <(grep -E '^#{2,} ' "$doc" 2>/dev/null | sed -e 's/^#* //'))" - while IFS= read -r anchor; do - [ -n "$anchor" ] || continue - # grep -cxF on a here-string, for the reason given in _dcv_absent above. - [ "$(grep -cxF "$anchor" <<<"$anchors" || true)" -ne 0 ] && continue - n=$((n + 1)); [ "$n" -le 6 ] && bad="$bad $anchor" - done < <(grep -oE '\]\(#[A-Za-z0-9_-]+\)' "$doc" 2>/dev/null \ - | sed -e 's/^](#//' -e 's/)$//' | sort -u) - [ "$n" -eq 0 ] && { printf '[]'; return; } - printf '[%d:%s]' "$n" "$bad" -} - -# PREMISE: the sweep found links at all. A document whose links it cannot parse -# reports "nothing broken", which is what a correct document reports too. -check "premise: the sweep reads the contents list's links" \ - "$([ "$(grep -coE '\]\(#[A-Za-z0-9_-]+\)' "$_dcv_doc")" -ge 15 ] \ - && echo enough || echo too-few)" "enough" - -# AND THE SWEEP MUST HAVE SWEPT. Without nullglob an unmatched glob stays literal, -# the [ -f ] skips it, and the loop below runs NO checks while the part still -# reports every check it did run as passing. A clean sweep needs a coverage premise. -_toc_n=0 -for _toc_f in "$_dcv_dir"/*.md; do - [ -f "$_toc_f" ] || continue - _toc_n=$((_toc_n + 1)) - check "every in-document link in ${_toc_f##*/} reaches a heading" \ - "$(_toc_unresolved "$_toc_f")" "[]" -done -check "premise: the link sweep saw the directory's documents" \ - "$([ "$_toc_n" -ge 3 ] && echo enough || echo "$_toc_n")" "enough" -unset _toc_f _toc_n - -# ---- and it must be able to FAIL, on a fixture rather than on the tree ------- - -printf '## 14. test_harness_deps.py: the harness\n- [14. x](#14-test_harness_depspy-the-harness)\n' \ - > "$_dcv_fix/ANCHOR_GOOD.md" -check "control: an anchor that keeps the underscores resolves" \ - "$(_toc_unresolved "$_dcv_fix/ANCHOR_GOOD.md")" "[]" - -# The exact shape that shipped: the underscores stripped out of the file name. -printf '## 14. test_harness_deps.py: the harness\n- [14. x](#14-testharnessdepspy-the-harness)\n' \ - > "$_dcv_fix/ANCHOR_BAD.md" -check "an anchor that strips the underscores is named, not passed over" \ - "$(_toc_unresolved "$_dcv_fix/ANCHOR_BAD.md")" \ - "[1: 14-testharnessdepspy-the-harness]" - -# And the derivation itself, on the heading this defect was found in: the dot and -# the colon go, the underscores stay. -check "the anchor rule drops punctuation and keeps underscores" \ - "$(_toc_anchor '14. test_harness_deps.py: the harness must self-test')" \ - "14-test_harness_depspy-the-harness-must-self-test" - -unset -f _toc_anchor _toc_unresolved - - -# ---- the harness must self-test without a database, and the gate must run it -- -# -# THE MEMBERSHIP DECISION AND THE CONFTEST IMPORT ARE NOT CHECKED HERE, and that is -# jd's rule rather than an omission: the shell harness and the pytest corpus are -# PARALLEL IN FUNCTIONALITY and must not call, import or reference each other. An -# earlier version of this part read test/pytest/conftest.py and INVOKED -# test_harness_deps.py with `python3 ... --disagree`, which is the coupling, not a -# second measurement: a shell arm driving the python decider agrees with it by -# construction and can never report it wrong. -# -# Both properties are asserted in the corpus, where they are native: # conftest imports no driver at module scope # test_harness_deps.py::test_conftest_imports_no_database_driver_at_module_scope # the declaration is exactly the database-free half, both directions # test_harness_deps.py::test_the_declaration_is_exactly_the_database_free_half # the partition accounts for every file, and the three report cases # test_harness_deps.py and test_harness_deps_classifier.py -# -# What stays here is the CI WORKFLOW, which belongs to neither harness. _hd_ci="$PGC_SRCDIR/.github/workflows/ci.yml" - check "premise: the CI workflow is where this part thinks it is" \ "$([ -f "$_hd_ci" ] && echo yes || echo no)" "yes" @@ -410,198 +71,12 @@ check "and it derives the file list rather than repeating it" \ check "and derives the pins from requirements-test.txt" \ "$([ "$(grep -c 'requirements-test.txt' "$_hd_ci")" -ge 1 ] && echo yes || echo no)" "yes" -# ---- and the DECLARATION must be decided, not declaimed ---------------------- -# -unset _hd_ci - -unset _dcv_dir _dcv_doc _dcv_seen _dcv_stated _dcv_fix _dcv_ht -unset -f _dcv_missing _dcv_count - -# ---- the mode inventory must count itself, and the count must be checkable ---- -# -# WHY THIS EXISTS. `test/pytest/VACUITY_MODES.md` and `test/pytest/README.md` -# stated different numbers for how many vacuity modes the layer refuses -- 27 and -# 23 -- and a reviewer could check NEITHER, because the document offered no rule -# for what counts as a mode. That is the defect this whole directory exists to -# refuse, committed by the document that describes the refusal: a number nobody -# can recompute is an assertion, not a measurement. -# -# Section 1a now states the rule: a mode is a backticked kebab-case identifier of -# three or more words. These arms hold the document to its own rule. -# -# WHAT IS POLICED. Four properties: -# * section 1a's "refused today" total is the count of ids in section 2 -# * its "not refused" total is the count in section 3 -# * its document total is the sum of the two -# * README.md quotes the same refused number, so the two cannot drift again -# The enumeration run's own 79 is history, not a property of the tree: nothing -# here pretends to check it. What IS checked is that the gap the document admits -# to equals the run total minus what was actually written down. - -_mi_doc="$PGC_TESTDIR/pytest/VACUITY_MODES.md" -_mi_readme="$PGC_TESTDIR/pytest/README.md" - -check "premise: the mode inventory is where this part thinks it is" \ - "$([ -f "$_mi_doc" ] && echo yes || echo no)" "yes" - -# Distinct ids matching section 1a's rule, within one "## " section. -# A function over FILE and PREFIX so the fixture arms below run the same logic. -_mi_ids() { # _mi_ids FILE PREFIX -> count - local f="$1" pre="$2" - awk -v p="^## $pre" ' - $0 ~ p { inside = 1; next } - /^## / { inside = 0 } - inside { print } - ' "$f" 2>/dev/null \ - | grep -oE '`[a-z0-9]+(-[a-z0-9]+){2,}`' \ - | sort -u | wc -l | tr -d ' ' -} - -# A stated row from the section 1a table. The label is a substring of the cell, -# not the whole of it, so this matches the cell rather than anchoring to its start. -# -# Take the VALUE cell, not the first number on the line. The labels themselves -# contain digits -- "named in section 2, refused today" -- so the obvious -# `grep -oE '[0-9]+' | head -1` returns the 2 from "section 2" and never the -# total. It did, and it read 2 and 3 for totals of 21 and 51. The fixture below -# could not see it because there the label digit and the value were both 2, so -# there is now an arm whose whole job is to tell those two readings apart. -_mi_row() { # _mi_row FILE LABEL -> the number, or "" if the row is absent - local f="$1" label="$2" - grep -E "^\|[^|]*${label}[^|]*\|" "$f" 2>/dev/null | head -1 \ - | awk -F'|' 'NF > 2 { v = $(NF - 1); gsub(/[^0-9]/, "", v); print v }' -} - -# Section 3 keeps a back-reference to every mode that moved into section 2 -# ("`X` is now closed"), so a mode can be named in both. Section 2 wins, and -# section 3's total is what it names MINUS what section 2 claims. Counting the -# back-references as unrefused puts one mode in two states: measured at 25 + 50 -# against 72 named, which is three ids counted twice. -_mi_idlist() { # _mi_idlist FILE PREFIX -> the ids themselves, one per line, sorted - local f="$1" pre="$2" - awk -v p="^## $pre" ' - $0 ~ p { inside = 1; next } - /^## / { inside = 0 } - inside { print } - ' "$f" 2>/dev/null \ - | grep -oE '`[a-z0-9]+(-[a-z0-9]+){2,}`' \ - | sort -u -} - -_mi_ref="$(_mi_ids "$_mi_doc" '2\.')" -_mi_not="$(comm -23 <(_mi_idlist "$_mi_doc" '3\.') <(_mi_idlist "$_mi_doc" '2\.') \ - | grep -c . || true)" - -# The same trap as the sweep above: a counter that finds nothing agrees with a -# document that claims nothing, and both look like success. -check "premise: the counting rule finds modes at all" \ - "$([ "$_mi_ref" -ge 15 ] && [ "$_mi_not" -ge 30 ] && echo enough || echo "$_mi_ref/$_mi_not")" \ - "enough" - -check "section 1a's refused total is the count of ids in section 2" \ - "$(_mi_row "$_mi_doc" 'refused today')" "$_mi_ref" - -check "section 1a's not-refused total is the count of ids in section 3" \ - "$(_mi_row "$_mi_doc" 'not refused')" "$_mi_not" - -check "section 1a's document total is the sum of its two sections" \ - "$(_mi_row "$_mi_doc" 'named in this document')" "$((_mi_ref + _mi_not))" - -check "the admitted gap is the run total minus what is written down" \ - "$(_mi_row "$_mi_doc" 'named nowhere here')" \ - "$(( $(_mi_row "$_mi_doc" 'produced by the enumeration run') \ - - $(_mi_row "$_mi_doc" 'named in this document') ))" - -# README.md is the file that drifted. It must quote the inventory's number rather -# than carry one of its own. -check "README.md quotes the number of modes the inventory names as refused" \ - "$(grep -cE "$_mi_ref refused" "$_mi_readme")" "1" - -# ---- and these arms must be able to FAIL ------------------------------------- - -_mi_fix="$PGC_WORKDIR/modecount"; rm -rf "$_mi_fix"; mkdir -p "$_mi_fix" -{ - printf '## 2. refused\n' - printf 'text `alpha-beta-gamma` and `delta-epsilon-zeta` here\n' - printf '## 3. not refused\n' - printf '`eta-theta-iota`\n' - printf '## 4. table\n' - printf '| named in section 2, refused today | 2 |\n' - printf '| named in section 3, not refused | 1 |\n' - printf '| **named in this document** | **3** |\n' -} > "$_mi_fix/GOOD.md" - -check "the counter counts a fixture's section 2" "$(_mi_ids "$_mi_fix/GOOD.md" '2\.')" "2" -check "the counter counts a fixture's section 3" "$(_mi_ids "$_mi_fix/GOOD.md" '3\.')" "1" - -# Two-word and one-word ids are not modes under section 1a's rule, and a counter -# that took them would inflate every total in the document. -printf '## 2. refused\n`one-two` and `single` and `alpha-beta-gamma`\n' > "$_mi_fix/SHORT.md" -check "an id of fewer than three words is not counted as a mode" \ - "$(_mi_ids "$_mi_fix/SHORT.md" '2\.')" "1" - -# An id named twice is one mode. Section 2 names several ids in more than one row. -printf '## 2. refused\n`alpha-beta-gamma` again `alpha-beta-gamma`\n' > "$_mi_fix/DUP.md" -check "an id named twice counts once" "$(_mi_ids "$_mi_fix/DUP.md" '2\.')" "1" - -# The section boundary must hold: ids after the next heading belong to it. -printf '## 2. refused\n`alpha-beta-gamma`\n## 3. not\n`delta-epsilon-zeta`\n' > "$_mi_fix/BOUND.md" -check "the counter stops at the next heading" "$(_mi_ids "$_mi_fix/BOUND.md" '2\.')" "1" +# AND THE JOB MUST RUN WITHOUT THE DRIVER, which is what makes it able to run at +# all on a machine with no database. Without this the job could quietly install +# psycopg and the database-free claim would stop being tested while still being +# made. Added here because this part's subject is now the workflow, and this is a +# property of the workflow rather than of either harness. +check "and the job asserts the driver is absent rather than assuming it" \ + "$([ "$(grep -c 'pip show psycopg' "$_hd_ci")" -ge 1 ] && echo yes || echo no)" "yes" -# A wrong stated total is visible. This is the shape that shipped in two files. -printf '## 2. refused\n`alpha-beta-gamma`\n## 4. t\n| named in section 2, refused today | 9 |\n' \ - > "$_mi_fix/WRONG.md" -check "a stated total that disagrees with the ids is visible" \ - "$([ "$(_mi_row "$_mi_fix/WRONG.md" 'refused today')" = "$(_mi_ids "$_mi_fix/WRONG.md" '2\.')" ] \ - && echo agrees || echo differs)" "differs" - -check "and the same comparison agrees on the fixture that is right" \ - "$([ "$(_mi_row "$_mi_fix/GOOD.md" 'refused today')" = "$(_mi_ids "$_mi_fix/GOOD.md" '2\.')" ] \ - && echo agrees || echo differs)" "agrees" - -# A missing row must not read as a passing comparison. -printf '## 2. refused\n`alpha-beta-gamma`\n' > "$_mi_fix/NOROW.md" -# The label contains a digit and the value is a different digit, so a reader that -# takes the first number on the line and one that takes the value cell give -# different answers. This is the arm that would have caught the helper's own bug. -printf '## 4. t\n| named in section 2, refused today | 7 |\n' > "$_mi_fix/LABELDIGIT.md" -check "the row's value is read, not a digit inside its label" \ - "$(_mi_row "$_mi_fix/LABELDIGIT.md" 'refused today')" "7" - -check "an absent total is empty rather than a number that happens to match" \ - "$([ -z "$(_mi_row "$_mi_fix/NOROW.md" 'refused today')" ] && echo absent || echo present)" \ - "absent" - -# The table is not the only place a total lives. Three sentences outside it still -# asserted the run's 23 after the table said 21: section 2's opening, the closing -# paragraph, and TESTS.md. Gating the table alone just moves the drift into prose. -# -# The run's own 23 appears once on purpose, as history, and is not gated. What is -# gated is every sentence stating what the layer refuses TODAY. -_mi_prose() { # _mi_prose FILE REGEX -> the captured number, or "" - local f="$1" re="$2" - grep -oE "$re" "$f" 2>/dev/null | head -1 | grep -oE '[0-9]+' | head -1 -} - -_mi_tests="$PGC_TESTDIR/pytest/TESTS.md" - -check "section 2's opening states the counted number of refused modes" \ - "$(_mi_prose "$_mi_doc" '[0-9]+ of the 79, counted')" "$_mi_ref" - -check "the closing paragraph states the counted number too" \ - "$(_mi_prose "$_mi_doc" 'known to refuse [0-9]+ demonstrated modes')" "$_mi_ref" - -check "TESTS.md states the counted number as well" \ - "$(_mi_prose "$_mi_tests" 'This layer refuses [0-9]+')" "$_mi_ref" - -# And the prose reader must be able to fail, on a fixture rather than on the tree. -printf 'the layer is known to refuse 99 demonstrated modes here\n' > "$_mi_fix/PROSE.md" -check "a prose total that disagrees with the ids is visible" \ - "$(_mi_prose "$_mi_fix/PROSE.md" 'known to refuse [0-9]+ demonstrated modes')" "99" - -check "an absent prose total is empty rather than a stray number" \ - "$([ -z "$(_mi_prose "$_mi_fix/GOOD.md" 'known to refuse [0-9]+ demonstrated modes')" ] \ - && echo absent || echo present)" "absent" - -unset _mi_doc _mi_readme _mi_ref _mi_not _mi_fix _mi_tests -unset -f _mi_ids _mi_row _mi_prose +unset _hd_ci diff --git a/test/selftest/360-an-unrunnable-pytest-test-must.sh b/test/selftest/360-an-unrunnable-pytest-test-must.sh index 81effe10..831d491c 100644 --- a/test/selftest/360-an-unrunnable-pytest-test-must.sh +++ b/test/selftest/360-an-unrunnable-pytest-test-must.sh @@ -1,30 +1,47 @@ -# ---- an unrunnable pytest test must not leave the run green ------------------ +# ---- the two harnesses must agree about the third state ---------------------- # -# WHY THIS EXISTS. `expect.cannot_run(REASON, detail)` is the pytest corpus's -# third state, the counterpart of `check_unrunnable` here. It wrote -# `self.unrunnable` and NOTHING READ IT, so a test that declared itself -# unrunnable reported `1 passed` and exit 0. Measured, before the fix. +# WHY THIS EXISTS. `expect.cannot_run(REASON, detail)` in the pytest corpus is the +# counterpart of `check_unrunnable` here, and the two sides share three things that +# are WRITTEN DOWN TWICE, once in each language: the INCOMPLETE exit code, the +# closed list of reasons, and the one-line shape an unrunnable check prints. A +# value duplicated across a language boundary drifts, and the drift is invisible -- +# the corpus would exit 67, a runner would compare against something else, and an +# INCOMPLETE run would read as a failure or as a pass depending on which way it +# moved. # -# That is the write-only-flag shape selftest 320 already polices one level up, -# where the runner's INCOMPLETE branch set a variable the verdict never read. It -# mattered more here: a bare `@pytest.mark.skip` FAILS the pytest run, so the -# layer refused the cheap dishonest escape and permitted the expensive-looking -# one. An escape hatch that costs nothing is the default. +# WHAT THIS PART DOES **NOT** DO ANY MORE (#432). It used to also pin the SHAPE of +# `pgc_vacuity.py`: that the unrunnable field is written, that something reads it, +# that the read reaches `session.exitstatus`, and that the override is conditional. +# Eleven arms, every one a text pin on the other harness's source. Those are gone. +# A shell arm asserting a grep matches cannot prove a python arm is CAUGHT -- #927's +# precedent -- and the behaviour is pinned where it can actually be observed, by four +# arms in `test/pytest/test_layer.py` that run pytest inside pytest and assert on the +# inner run's exit status. # -# WHY THE CHECKS ARE STATIC. The behaviour itself is pinned in the corpus, by -# four arms in test/pytest/test_layer.py that run pytest inside pytest and assert -# on the inner run's exit status. Those need pytest, psycopg and a virtualenv; -# CI installs none of them, and `pgc_skip` treats a missing dependency as a -# failure rather than a skip. So the behavioural half lives where it can run and -# this half asserts the STRUCTURE that behaviour rests on, which is greppable -# from a checkout with nothing installed. Same division as selftest 320's last -# two arms, which grep the runner for the call and for the absence of the flag. +# THE OLD REASON FOR KEEPING THEM WAS TRUE AND IS NOT ANY MORE. It said those arms +# "need pytest, psycopg and a virtualenv; CI installs none of them". CI now has a +# `pytest-guards` job that installs pytest pinned from `requirements-test.txt`, +# asserts psycopg is absent, and runs the database-free file list -- which includes +# `test_layer.py`. So the behavioural half runs in CI and this half is not its last +# line of defence. # -# THE NUMBER IS THE POINT OF THE FIRST ARM. 67 now lives in two files. A number -# duplicated across a language boundary is a number that drifts, and the drift -# is invisible: the corpus would exit 67, the runner would compare against -# something else, and an INCOMPLETE run would read as a failure or as a pass -# depending on which way it moved. +# MEASURED BEFORE DELETING ANYTHING, because "the other side covers it" is a claim. +# With `session.exitstatus = EXIT_INCOMPLETE` made unreachable in `pgc_vacuity.py` +# -- the defect exactly as it shipped -- those four arms go from `4 passed` to +# `2 failed, 2 passed`. Two, not four, and that is correct rather than partial: the +# other two assert exit 1 for a run with a real failure and exit 0 for a run with +# nothing unrunnable, and neither of those outcomes moves. The file was restored and +# compared byte-for-byte afterwards. +# +# WHAT IS LEFT IS THE ONE PERMITTED CROSS-REFERENCE, in CONTEXT.md's sense: a +# property that IS the relationship between the two harnesses and so cannot be +# expressed from one side. Each of the three agreements below is PARSED OUT OF BOTH +# FILES rather than written here. A check that restates the value tests this file +# against itself: both copies could drift together and it would still pass. +# +# AND THE LIST AND THE SHAPE ARE NEW. The part checked the exit code only, while +# two more duplications sat beside it unchecked -- including the reason list, whose +# whole purpose is to be closed on both sides. _ts_lib="$PGC_TESTDIR/lib.sh" _ts_vac="$PGC_TESTDIR/pytest/pgc_vacuity.py" @@ -35,9 +52,8 @@ check "premise: the harness library is where this part thinks it is" \ check "premise: the pytest layer is where this part thinks it is" \ "$([ -f "$_ts_vac" ] && echo yes || echo no)" "yes" -# Parsed out of each file rather than written here. A check that restates the -# number tests this file against itself: both copies could drift together and it -# would still pass. +# ---- agreement 1: the INCOMPLETE exit code ---------------------------------- + _ts_sh_code="$(sed -n 's/^PGC_EXIT_INCOMPLETE=\([0-9]\{1,\}\).*/\1/p' "$_ts_lib" | head -1)" _ts_py_code="$(sed -n 's/^EXIT_INCOMPLETE[[:space:]]*=[[:space:]]*\([0-9]\{1,\}\).*/\1/p' "$_ts_vac" | head -1)" @@ -50,102 +66,101 @@ check "premise: the pytest layer states one too" \ check "the two harnesses agree on the INCOMPLETE exit code" \ "$_ts_py_code" "$_ts_sh_code" -# ---- the field must be READ, which is the defect this part is named after --- +# ---- agreement 2: the closed list of reasons -------------------------------- # -# Counted as two populations rather than asserted as a boolean, so the failure -# says which side is missing. -_ts_writes="$(grep -c 'self\.unrunnable[[:space:]]*=' "$_ts_vac")" -_ts_reads="$(grep -c 'rec\.unrunnable' "$_ts_vac")" +# Sorted, because the two files are free to list them in different orders and an +# order difference is not a drift. Compared as one string so the failure prints +# both lists side by side and names which token moved. -check "the layer still writes the unrunnable state" \ - "$([ "$_ts_writes" -ge 1 ] && echo yes || echo "$_ts_writes")" "yes" +_ts_sh_reasons="$(sed -n 's/^PGC_UNRUN_REASONS="\([^"]*\)".*/\1/p' "$_ts_lib" \ + | head -1 | tr ' ' '\n' | grep -E '^[A-Z_]+$' | sort | tr '\n' ' ')" +_ts_py_reasons="$(sed -n '/^UNRUNNABLE_REASONS = (/,/^)/p' "$_ts_vac" \ + | grep -oE '"[A-Z_]+"' | tr -d '"' | sort | tr '\n' ' ')" -check "and something READS it, rather than only writing it" \ - "$([ "$_ts_reads" -ge 1 ] && echo yes || echo "$_ts_reads")" "yes" +check "premise: lib.sh states a closed list of unrunnable reasons" \ + "$([ -n "$_ts_sh_reasons" ] && echo yes || echo no)" "yes" -# ---- and the read has to reach the run's exit status ------------------------ -# -# Reading the field into a list nothing acts on would satisfy the arm above and -# leave the defect exactly where it was. -# -# TWO CORRECTIONS THIS ARM ALREADY NEEDED, both found by running it against a -# LATER branch rather than by rereading it. -# -# `=` MATCHES INSIDE `==`. The first pattern was `exitstatus[[:space:]]*=`, which -# counted the comparison `if ... session.exitstatus == 0:` as an assignment. A -# read counted as a write, in the arm whose entire subject is the difference -# between the two. `[^=]` after the `=` is what separates them. +check "premise: the pytest layer states a closed list too" \ + "$([ -n "$_ts_py_reasons" ] && echo yes || echo no)" "yes" + +check "the two harnesses agree on the closed list of unrunnable reasons" \ + "$_ts_py_reasons" "$_ts_sh_reasons" + +# ---- agreement 3: the line an unrunnable check prints ----------------------- # -# AND THE COUNT IS A FLOOR, NOT AN EQUALITY. Written as "exactly 1", it went red -# on the next branch in this stack, which adds a second escalation for a -# different condition -- a legitimate addition reported as a defect. The property -# is "the read reaches the exit status", so one site satisfies it and two do not -# make it less true. An equality here is a guard that reddens on growth, and a -# guard that reddens on growth gets switched off. Measured: 1 site on this -# branch, 2 on audit/432-pytest-oracles, 0 on the pre-fix layer -- so the floor -# still fails exactly where it must. -_ts_assigns="$(grep -c 'session\.exitstatus[[:space:]]*=[^=]' "$_ts_vac")" -check "the layer ends a session by setting its exit status" \ - "$([ "$_ts_assigns" -ge 1 ] && echo yes || echo "$_ts_assigns")" "yes" - -# Failure dominates, the same rule lib.sh keeps: a run with a failure AND an -# unrunnable test is a failure. So the override must be conditional on a -# currently-clean run. Unconditional, it would MASK failures as INCOMPLETE. -check "and only ever moves a run off zero, so a failure still dominates" \ - "$(grep -c 'exitstatus == 0' "$_ts_vac")" "1" - -# The state says why. A third state that does not name its reason is a skip. -check "the layer prints the unrunnable reason in lib.sh's shape" \ - "$(grep -c 'UNRUN.*{reason}' "$_ts_vac")" "1" - -# ---- and these greps must be able to FAIL ----------------------------------- +# Normalised, not compared raw: one side interpolates `$name` and the other +# `{nodeid}`, which is a difference between two languages rather than between two +# behaviours. Every interpolation becomes `%`, so what is compared is the literal +# text around them -- the part a log reader and `pgc_record` actually see. + +_ts_shape_of() { # _ts_shape_of LINE -> the literal text with interpolations as % + printf '%s\n' "$1" \ + | grep -oE 'UNRUN [^"]*' \ + | sed -e 's/\$[A-Za-z_][A-Za-z_0-9]*/%/g' -e 's/{[A-Za-z_][A-Za-z_0-9]*}/%/g' \ + | head -1 +} + +# THE LINE HAS TO BE THE CODE, NOT THE PROSE ABOUT IT. The first version matched +# `UNRUN ` anywhere and took `pgc_vacuity.py`'s DOCSTRING, which spells the shape +# out as `UNRUN : : ` for a reader and then keeps talking -- +# so the arm compared a sentence against a format string and failed. Requiring a +# quote immediately before the marker selects the quoted string in both languages +# and excludes the prose, which opens with a backtick. The premise arms below could +# not have caught this: the sentence is not empty. +_ts_sh_shape="$(_ts_shape_of "$(grep -m1 '"UNRUN ' "$_ts_lib")")" +_ts_py_shape="$(_ts_shape_of "$(grep -m1 '"UNRUN ' "$_ts_vac")")" + +check "premise: lib.sh prints a line for an unrunnable check" \ + "$([ -n "$_ts_sh_shape" ] && echo yes || echo no)" "yes" + +check "premise: the pytest layer prints one too" \ + "$([ -n "$_ts_py_shape" ] && echo yes || echo no)" "yes" + +check "the two harnesses print an unrunnable check in the same shape" \ + "$_ts_py_shape" "$_ts_sh_shape" + +# ---- and each comparison must be able to FAIL ------------------------------- # -# Every arm above passes on a healthy tree, which a grep that matches nothing -# also does -- against an EMPTY file, `grep -c` returns 0 and every `-ge 1` arm -# would read "no" while every `-c ... "1"` arm would read 0. Those would be -# visible. The dangerous case is the opposite: a pattern that is subtly wrong -# still matching. So the fixtures below are the real code with ONE property -# removed, and the arms assert the greps notice. +# Three agreements that pass on a healthy tree, which three comparisons of an +# empty string against an empty string also do. The premises above refuse the +# empty case; these fixtures are the drifted case, which is the one that matters, +# because a reader cannot tell an agreement from a pair of blanks. _ts_fix="$PGC_WORKDIR/thirdstate"; rm -rf "$_ts_fix"; mkdir -p "$_ts_fix" -# The defect as it shipped: the field is written and never read. -{ - printf 'EXIT_INCOMPLETE = 67\n' - printf 'class Expect:\n' - printf ' def cannot_run(self, reason, detail=""):\n' - printf ' self.unrunnable = (reason, detail)\n' -} > "$_ts_fix/writeonly.py" - -check "a write-only unrunnable field is caught" \ - "$(grep -c 'rec\.unrunnable' "$_ts_fix/writeonly.py")" "0" - -check "premise: and that same fixture does show the write, so the arm is not blind" \ - "$(grep -c 'self\.unrunnable[[:space:]]*=' "$_ts_fix/writeonly.py")" "1" - -# The exit code drifted. Both files parse; the values differ. printf 'EXIT_INCOMPLETE = 66\n' > "$_ts_fix/drifted.py" _ts_drift="$(sed -n 's/^EXIT_INCOMPLETE[[:space:]]*=[[:space:]]*\([0-9]\{1,\}\).*/\1/p' "$_ts_fix/drifted.py" | head -1)" check "a drifted exit code is visible rather than absorbed" \ "$([ "$_ts_drift" = "$_ts_sh_code" ] && echo agrees || echo "differs:$_ts_drift/$_ts_sh_code")" \ "differs:66/67" -# An unconditional override, which would mask a failing run as INCOMPLETE. -printf ' session.exitstatus = EXIT_INCOMPLETE\n' > "$_ts_fix/unconditional.py" -check "an unconditional exit override is caught by the dominance arm" \ - "$(grep -c 'exitstatus == 0' "$_ts_fix/unconditional.py")" "0" - -# A file holding ONLY the comparison. Under the first pattern this counted as an -# assignment, which is the false positive that shipped; under the corrected one it -# is zero. Without this arm the correction above is itself unproved. -printf ' if exitstatus == 0 and session.exitstatus == 0:\n' > "$_ts_fix/compare.py" -check "a comparison on the exit status is not counted as an assignment" \ - "$(grep -c 'session\.exitstatus[[:space:]]*=[^=]' "$_ts_fix/compare.py")" "0" - -check "premise: while a real assignment on the same line shape IS counted" \ - "$(grep -c 'session\.exitstatus[[:space:]]*=[^=]' "$_ts_fix/unconditional.py")" "1" - -check "premise: while the real layer satisfies that same arm" \ - "$(grep -c 'exitstatus == 0' "$_ts_vac")" "1" - -unset _ts_lib _ts_vac _ts_sh_code _ts_py_code _ts_writes _ts_reads _ts_fix _ts_drift _ts_assigns +{ + printf 'UNRUNNABLE_REASONS = (\n' + printf ' "MISSING_DEPENDENCY",\n' + printf ' "UNSUPPORTED_MAJOR",\n' + printf ')\n' +} > "$_ts_fix/shortlist.py" +_ts_short="$(sed -n '/^UNRUNNABLE_REASONS = (/,/^)/p' "$_ts_fix/shortlist.py" \ + | grep -oE '"[A-Z_]+"' | tr -d '"' | sort | tr '\n' ' ')" +check "a reason dropped from one side only is visible" \ + "$([ "$_ts_short" = "$_ts_sh_reasons" ] && echo agrees || echo differs)" "differs" + +check "premise: and that fixture is a real list rather than an empty parse" \ + "$(printf '%s' "$_ts_short" | wc -w)" "2" + +# Two spaces after UNRUN, as the real shape has: the drift under test is the lost +# colons, not a lost space. The first fixture wrote ONE space, so the parse found +# nothing and the comparison was empty-against-real -- which "differs", for the +# wrong reason. Its own premise arm caught that, which is what the premise is for. +printf ' terminalreporter.write_line(f"UNRUN {nodeid} {reason} {detail}")\n' \ + > "$_ts_fix/drifted_shape.py" +_ts_shape_drift="$(_ts_shape_of "$(grep -m1 '"UNRUN ' "$_ts_fix/drifted_shape.py")")" +check "a drifted print shape is visible" \ + "$([ "$_ts_shape_drift" = "$_ts_sh_shape" ] && echo agrees || echo differs)" "differs" + +check "premise: and that fixture parses to a shape rather than to nothing" \ + "$([ -n "$_ts_shape_drift" ] && echo yes || echo no)" "yes" + +unset _ts_lib _ts_vac _ts_sh_code _ts_py_code _ts_sh_reasons _ts_py_reasons \ + _ts_sh_shape _ts_py_shape _ts_fix _ts_drift _ts_short _ts_shape_drift +unset -f _ts_shape_of diff --git a/test/selftest/370-the-plan-marker-guard-must.sh b/test/selftest/370-the-plan-marker-guard-must.sh deleted file mode 100644 index 072dd8b4..00000000 --- a/test/selftest/370-the-plan-marker-guard-must.sh +++ /dev/null @@ -1,90 +0,0 @@ -# ---- plan_marker must have all three of its guards --------------------------- -# -# WHY THIS EXISTS. @jdatcmd's #897 review named `plan_marker` as the guard to fix -# first: "both of its arms can be deleted independently with the suite green. -# Under one of those mutations the premise can never fail, so the provider-trap -# test would silently be about an ordinary plan." -# -# He was right, and it is the worst place in the layer for it to be true. -# plan_marker is the faithful port of pgc_is_columnar_scan; test_connection.py -# calls it three times, once as the PREMISE that the vectorized aggregate -# engaged. A premise that cannot fail turns its test into a test about an -# ordinary plan, and the test stays green while it happens. -# -# A THIRD HOLE SAT UNDER BOTH ARMS. An absence claim is satisfied by nothing -# being there at all: `plan_marker([], key, absent=True)` gave `1 passed`, exit 0, -# because a plan that never arrived looks exactly like a plan that legitimately -# lacks the node. That is now a refusal. -# -# WHY THE CHECKS ARE STATIC, same division as selftest 360. The behaviour is -# pinned by five arms in test/pytest/test_guards_pinned.py that run pytest inside -# pytest; those need pytest, psycopg and a virtualenv, and CI installs none of -# them. This half asserts the STRUCTURE they rest on, greppable from a checkout -# with nothing installed. - -_pm_vac="$PGC_TESTDIR/pytest/pgc_vacuity.py" - -check "premise: the pytest layer is where this part thinks it is" \ - "$([ -f "$_pm_vac" ] && echo yes || echo no)" "yes" - -# The function body, cut out once so every arm below reads the same text. A -# grep over the WHOLE file would match these shapes wherever they occur and -# report a guard present that lives in another method. -_pm_body="$(awk '/^ def plan_marker\(/{f=1} f&&/^ def /&&!/plan_marker/{exit} f' "$_pm_vac")" - -check "premise: plan_marker's body was actually cut out of the file" \ - "$([ "$(printf '%s\n' "$_pm_body" | wc -l)" -ge 20 ] && echo yes || echo "too short")" "yes" - -# The present arm: asked "does this plan carry the marker", a plan that does not -# must fail. -check "plan_marker keeps the arm that fails when the key is absent" \ - "$(printf '%s\n' "$_pm_body" | grep -c 'if not absent and not found:')" "1" - -# The absent arm: `absent=True` is how a test pins that a plan is NOT a scan. -check "plan_marker keeps the arm that fails when the key is present" \ - "$(printf '%s\n' "$_pm_body" | grep -c 'if absent and found:')" "1" - -# And the refusal under both of them. -check "plan_marker refuses a plan with no nodes at all" \ - "$(printf '%s\n' "$_pm_body" | grep -c 'if not nodes:')" "1" - -check "and that refusal is a VacuityError, not an ordinary assertion" \ - "$(printf '%s\n' "$_pm_body" | grep -A2 'if not nodes:' | grep -c 'raise VacuityError')" "1" - -# The refusal has to come BEFORE the walk that sets `found`, or it is dead code: -# an empty plan leaves found=False and the absent arm returns a pass first. -_pm_ln_empty="$(printf '%s\n' "$_pm_body" | grep -n 'if not nodes:' | cut -d: -f1)" -_pm_ln_absent="$(printf '%s\n' "$_pm_body" | grep -n 'if absent and found:' | cut -d: -f1)" -check "premise: both line numbers were found, so the ordering arm can mean something" \ - "$([ -n "$_pm_ln_empty" ] && [ -n "$_pm_ln_absent" ] && echo yes || echo no)" "yes" - -check "the empty-plan refusal precedes the arm it protects" \ - "$([ "$_pm_ln_empty" -lt "$_pm_ln_absent" ] && echo before || echo "AFTER, so it is dead code")" \ - "before" - -# ---- and these greps must be able to FAIL ----------------------------------- -# -# Every arm above passes on a healthy tree, which a grep that matches nothing -# also does. Each fixture is the real shape with ONE property removed. - -_pm_fix="$PGC_WORKDIR/planmarker"; rm -rf "$_pm_fix"; mkdir -p "$_pm_fix" - -printf ' if False and not absent and not found:\n' > "$_pm_fix/present.py" -check "a neutered present arm is caught" \ - "$(grep -c 'if not absent and not found:' "$_pm_fix/present.py")" "0" - -printf ' if False and absent and found:\n' > "$_pm_fix/absent.py" -check "a neutered absent arm is caught" \ - "$(grep -c 'if absent and found:' "$_pm_fix/absent.py")" "0" - -printf ' if False and not nodes:\n' > "$_pm_fix/empty.py" -check "a neutered empty-plan refusal is caught" \ - "$(grep -c 'if not nodes:' "$_pm_fix/empty.py")" "0" - -# The mirror of the three above: the same greps on the REAL body return 1, so a -# zero is a missing guard rather than a broken pattern. -check "premise: while the real body satisfies all three, so the greps work" \ - "$(printf '%s\n' "$_pm_body" | grep -cE 'if not absent and not found:|if absent and found:|if not nodes:')" \ - "3" - -unset _pm_vac _pm_body _pm_fix _pm_ln_empty _pm_ln_absent diff --git a/test/selftest/380-the-pytest-cluster-helpers.sh b/test/selftest/380-the-pytest-cluster-helpers.sh index 9eeec064..e1a31665 100644 --- a/test/selftest/380-the-pytest-cluster-helpers.sh +++ b/test/selftest/380-the-pytest-cluster-helpers.sh @@ -15,19 +15,33 @@ # the directory -- and conftest.py cannot clean up after it, because # `cluster, root = make_cluster(...)` never completes when the call raises. # -# WHY STATIC, same division as selftests 360 and 370: the behavioural arms live -# in test/pytest/test_build_refusal.py, which needs pytest, psycopg and a -# virtualenv that CI does not install. This half pins the structure they rest on. +# WHAT THIS PART COVERS, AND WHAT MOVED (#432). Finding 1 is here: its subject is +# test/pgc_fingerprint.py, which lib.sh runs with the system interpreter and which +# is NOT part of the pytest harness, so checking it is this part's own business. +# +# FINDING 2 HAS MOVED. Its subject is test/pytest/pgc_cluster.py, and the arms that +# read it as text are now source checks in test/pytest/test_build_refusal.py, beside +# the behavioural arm that provokes a real failed setup and asserts no directory is +# left. Python reading its own module is not a cross-harness reference; a shell part +# grepping it is the one CONTEXT.md refuses. +# +# AND THE OLD REASON FOR KEEPING THEM HERE IS STALE. It said those arms "need pytest, +# psycopg and a virtualenv that CI does not install". CI has a `pytest-guards` job +# that installs pytest pinned from requirements-test.txt, asserts psycopg is absent, +# and runs the database-free file list, which contains test_build_refusal.py. +# +# MEASURED BEFORE MOVING ANYTHING. Five mutations of pgc_cluster.py, each asserted to +# have applied and each leaving the file well-formed: BaseException narrowed to +# Exception, cluster.stop() removed, the bare re-raise turned into pass, a private +# hashlib.md5 added, and shutil.rmtree(root) removed. Every one reddens on the python +# side -- three of them only the source arm, two of them the behavioural arm as well. +# Restored byte-for-byte after each. # # THE DERIVATION IS THE POINT OF THE FIRST GROUP. Hard-coding objstore/ would fix # today and fail the next time a module is added, so the arms below require the # GLOB rather than the name -- and require that the name does NOT appear, which # is the only way to tell a derivation from a list that happens to be complete. -_pc_cl="$PGC_TESTDIR/pytest/pgc_cluster.py" - -check "premise: the pytest cluster helper is where this part thinks it is" \ - "$([ -f "$_pc_cl" ] && echo yes || echo no)" "yes" # THE FINGERPRINT MOVED (#907). It was an independent Python implementation in # pgc_cluster.py and an independent shell one in lib.sh; the pair produced four @@ -57,10 +71,6 @@ check "and no longer mixes in the bare filename" \ "$(grep -c 'h.update(path.name.encode())' "$_pc_fp")" "0" # AND NEITHER CALLER MAY KEEP A PRIVATE COPY. This is the arm that would catch -# #907 recurring: the whole point is one implementation, so a second one -# reappearing in either caller is the defect, not a detail. -check "the pytest helper keeps no private fingerprint implementation" \ - "$(grep -cE 'hashlib\.md5|glob\("\*/Makefile"\)' "$_pc_cl")" "0" check "and the shell keeps none either" \ "$(grep -cE 'md5sum < |xargs -0 cat' "$PGC_TESTDIR/lib.sh")" "0" @@ -71,40 +81,11 @@ check "and the shell keeps none either" \ check "the module imports nothing from the pytest tree" \ "$(grep -cE '^(import|from) +(pgc_|conftest|pytest|psycopg)' "$_pc_fp")" "0" -# ---- and the lifecycle ------------------------------------------------------ -# -# Cut make_cluster's body out before grepping: these shapes occur elsewhere in -# the file, and an arm that greps the whole file reports a guard present that -# lives in another function. -_pc_body="$(awk '/^def make_cluster\(/{f=1} f&&/^def /&&!/make_cluster/{exit} f' "$_pc_cl")" - -check "premise: make_cluster's body was actually cut out of the file" \ - "$([ "$(printf '%s\n' "$_pc_body" | wc -l)" -ge 15 ] && echo yes || echo "too short")" "yes" - -check "make_cluster removes its tree when setup raises" \ - "$(printf '%s\n' "$_pc_body" | grep -c 'shutil.rmtree(root')" "1" - -# BaseException, not Exception: a KeyboardInterrupt during initdb leaks a datadir -# and a possibly-running postmaster exactly like an error does. -check "and it catches BaseException, so an interrupt cleans up too" \ - "$(printf '%s\n' "$_pc_body" | grep -c 'except BaseException:')" "1" - -check "and it stops a partially started cluster before removing the tree" \ - "$(printf '%s\n' "$_pc_body" | grep -c 'cluster.stop()')" "1" - -# Re-raising is what keeps the failure visible. Swallowing it would turn a failed -# setup into a silent None. -check "and the original error is re-raised rather than swallowed" \ - "$(printf '%s\n' "$_pc_body" | grep -c '^ raise$')" "1" # ---- and these greps must be able to FAIL ----------------------------------- _pc_fix="$PGC_WORKDIR/clusterhelpers"; rm -rf "$_pc_fix"; mkdir -p "$_pc_fix" -printf 'def make_cluster(a, b):\n root = mkdtemp()\n return cluster, root\n' \ - > "$_pc_fix/noguard.py" -check "a make_cluster with no cleanup is caught" \ - "$(grep -c 'shutil.rmtree(root' "$_pc_fix/noguard.py")" "0" printf ' dirs = [root / "src"]\n' > "$_pc_fix/srconly.py" check "a fingerprint that reads src only is caught" \ @@ -131,7 +112,5 @@ check "and a hard-coded module list is caught by the name arm" \ check "premise: while the real module satisfies the derivation arm" \ "$(grep -cE '_is_plain_file\(d / "Makefile"\)' "$_pc_fp")" "1" -check "premise: and the real helper still carries its cleanup" \ - "$(grep -cE 'shutil.rmtree\(root' "$_pc_cl")" "1" -unset _pc_cl _pc_fp _pc_body _pc_fix +unset _pc_fp _pc_fix diff --git a/test/selftest/parts.manifest b/test/selftest/parts.manifest index 1a5f64d9..05fa1d47 100644 --- a/test/selftest/parts.manifest +++ b/test/selftest/parts.manifest @@ -34,7 +34,6 @@ 340-the-binary-must-be-built-from.sh 350-the-pytest-corpus-must-be.sh 360-an-unrunnable-pytest-test-must.sh -370-the-plan-marker-guard-must.sh 380-the-pytest-cluster-helpers.sh 390-a-registered-suite-must-account.sh 400-a-check-result-must-be-machine.sh