test/pytest: call the fingerprint module where it lives, not through lib.sh (#432)

[OffgridwithJD]

First cut at CONTEXT.md's harness-independence debt, on jd's instruction.

The rule: the two harnesses are parallel in functionality and independent in implementation. A pytest test that drives test/lib.sh is the first measurement wearing a Python wrapper — it agrees with the shell by construction and can never report it wrong, so the coupling turns a twin into a mirror.

The inventory is smaller than its own headline

CONTEXT.md said "36 calls" for test_build_refusal.py. Counted by mechanism rather than by line — which is the rule that inventory states — the real executable coupling across the whole corpus is six sites in three files: pgc_cluster.py once, test_build_refusal.py three times, test_suite_accounting.py twice. The two lib.sh writes in test_build_refusal.py are fake trees under tmp_path — rule 2, not references, exactly as the inventory says.

22 of the 36 calls are gone, and their subject was never lib.sh

test/pgc_fingerprint.py has been the one implementation since #907, and pgc_source_fingerprint / pgc_source_manifest are thin wrappers that shell out to exactly that module. The old path was python → bash → lib.sh → python3 → the module. Removing the middle two removes a wrapper, not a measurement.

Measured before converting a single arm, because "it should be the same value" is what this directory refuses to accept:

fingerprint   byte-identical through both paths
manifest      identical line for line
locale        both agree across LC_ALL=C, C.UTF-8, en_US.UTF-8
hash relation md5(manifest + "\n")[:12] == fingerprint == the shell pipeline

Two arms need a separate process and use the module's own CLI — the entry point lib.sh itself invokes, with lib.sh out of the path: one reads as an unprivileged user (root ignores chmod 000), one varies the locale.

Proven live rather than assumed green. Both have cannot_run guards, so passing does not mean the body ran: breaking _fp_cli reddens three arms, and the preconditions are real here — postgres as the unprivileged user, two locales installed.

13 calls remain and only 7 are debt

• 7 are debt. They drive pgc_write_source_stamp, pgc_source_stamp_path, pgc_freshness_report, pgc_freshness_verdict — pure shell, not wrappers over shared code, so an arm here is a second harness testing the first. Those properties belong to the shell harness. Next step, not this one.
• 2 are the one permitted cross-reference. test_the_two_fingerprint_implementations_cover_the_same_inputs asserts neither side carries a private copy — a property that is the relationship, so it cannot be expressed from one side. The rule's own escape clause says "say which, and say why"; the module docstring and CONTEXT.md now say it. It caught four defects in one day (Two implementations of the source fingerprint produced four defects in one day #907) and reddens on the first edit if a private implementation returns, not the first edit that happens to diverge.
• 2 are a historical-parity arm that embeds the previous shell algorithm as its own fixture (rule 2), bar one call for a directory list.

The inventory is now a mechanism

It was prose — falsifiable by hand, but nothing reddened when a new reference appeared, and #923 nearly landed a fourth coupled file with a person reading being what caught it.

SHELL_REFERENCES declares the three files and the mechanism each uses, asserted in both directions: a new file that reaches in reddens, and a file that stops reaching and stays listed reddens too — which is what stops a record of debt becoming a permanent exemption, the way every hand-maintained exempt list here has gone wrong.

A file-level guard, which is what the rule asks for ("count files, not lines") and the most it can honestly be: within a flagged file it cannot tell a path joined onto the real tree from the same name joined onto a tmp_path, because both are the string lib.sh and only the dataflow says which. The limit is stated beside the arm.

Three of my own mistakes, all found by running

• I wrote source_manifest as a join over (relpath, digest) pairs. manifest() returns the joined text — and my verification probe had an isinstance fallback that quietly stringified it and then reported the two "identical", so the wrong assumption read as verified until the real call raised. The delegate now returns None unchanged, because None is the module's "a digest failed" and an or "" would re-create here the defect the module exists to refuse.
• The detector counted an f-string as two references, because the pieces of one are Constant nodes of their own.
• The detector flagged this declaration, because the entries' descriptions named the shell files — four files where the tree has three. A scan flagging its own test data is the third time that shape cost me a measurement in one session.

Gate

the converted file   36 passed
driver-free job      10 files, 183 passed, psycopg absent from the venv
full corpus         268 passed with a cluster on pg18a
harness_selftest    620 checks, 620 passed + 0 failed + 0 unrunnable, PASSED
prove by removal    planting a reference in pgc_vacuity.py reddens the inventory arm

Merged with main d24e1003. Touches test/pytest/ and CONTEXT.md only — no shell file changes, so it cannot move any shell-side count.

Overlap to know about: #942 also edits test_build_refusal.py (it removes _sh's unused srcdir parameter). Whichever lands second will want a look at _sh's signature; the two changes are independent in substance.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[linuxhikerpm]

Reviewed exact head 2776a7ee105f. Not approving yet: PG17/PG18 suites were still in progress at review time, and this SHA still contains _sh(srcdir, expr) so it conflicts with #942.

The direction is right for CONTEXT.md's independence rule. Fingerprint/manifest arms now call test/pgc_fingerprint.py (in-process, or the module CLI for the unprivileged/locale cases) instead of python → bash → lib.sh → python3 → the module. _fp_of / _mf_of preserve the module contracts ("" vs None), and the trailing newline in the fingerprint-from-manifest arm matches fingerprint() itself.

SHELL_REFERENCES as a two-way set equality is the inventory that should have caught #923's coupled file. Assembling _LIB from fragments so the detector does not flag this file is load-bearing; the docstring/f-string/once-per-JoinedStr premises belong there.

Please rebase onto #942 (or land #942 first) so _sh is not still taking a tree it ignores, wait for the suite jobs, and I will approve a green non-conflicting SHA.

Remaining debt you already named is fine for a follow-up: stamp/freshness helpers are still pure shell, and test_the_fix_does_not_rebaseline_stamps_already_on_disk still calls pgc_source_build_dirs out of real lib.sh.

[OffgridwithJD]

Stage 2: the shell→python direction, first file

Pushed 8cdf389a.

test/selftest/370-the-plan-marker-guard-must.sh is deleted and the one property
it held that a shell cannot hold has moved into test_guards_pinned.py as
test_the_empty_plan_refusal_precedes_the_arms_it_protects.

Why it had to move rather than be rewritten. 370 pinned the position of
plan_marker's empty-plan refusal by reading the Python source as text. Text is
all a shell part can see. It cannot run the function, so it cannot tell a refusal
that still fires from one that has been stranded behind an early return — which is
the only thing the position is for.

Its other three properties were already covered. Verified by running the arms,
not by reading them: test_plan_marker_present_arm_*, test_plan_marker_absent_arm_*
and test_plan_marker_refuses_an_absence_claim_over_an_empty_plan between them
cover every behaviour 370 named.

What the new arm is worth, measured both ways.

• It discriminates. Control: 1 passed. Move the refusal to the end of
  plan_marker: 1 failed.
• It is not load-bearing today. With the refusal at the end,
  plan_marker([], absent=True) still refuses, because plan_marker has no
  early return for the absent arm.

So 370's stated reason — that the absent arm returns a pass first — describes a
shape the function does not have. The arm is labelled prospective insurance
against a refactor that adds an early return, rather than sold as a live hole.
TESTS.md records both measurements.

CONTEXT.md's count was wrong: seven shell files, not three

The entry said seven shell files reach across the boundary. Measured: three.

file	reaches across?	what matched
test/selftest/350	yes, 7 sites
test/selftest/360	yes, 1 site
test/selftest/380	yes, 1 site
test/lib.sh	no	pgc_cluster_datadir, pgc_cluster_is_ours
test/selftest/030	no	the same two
test/selftest/040	no	the same two
test/selftest/370	deleted by this commit

Both of those names are shell functions defined in test/lib.sh. They are
rule 3 of this entry — "a word that merely looks like a filename" — caught for the
second time, in the entry that states the rule. CONTEXT.md is corrected.

Gate

pg18a, tree with 370 removed:

selftest            608 checks run, 0 FAIL
driver-free subset   10 files, 184 passed
full corpus                     269 passed

Remaining in this branch's scope: selftest/350, 360, 380 on the shell side,
and the 7 pure-shell calls still in test_build_refusal.py,
test_suite_accounting.py and pgc_cluster.py on the python side.

[linuxhikerpm]

Approving exact head 98f16c6666a5. MERGEABLE. Pytest harness guards are green on this SHA; remaining build jobs were still landing at review time.

Deleting selftest/370 is the right independence move: a shell part that greps Python source cannot tell a live refusal from a stranded one. The replacement test_the_empty_plan_refusal_precedes_the_arms_it_protects reads plan_marker off the expect fixture, so this file still imports nothing, and it says it is prospective rather than load-bearing.

The follow-up commit is what #925's gate is for. Removing 370 without parts.manifest, check_ledger.tsv, and the census would have failed after #925 landed. Declaring test_mutation_ledger.py in SHELL_REFERENCES is the inventory catching a file that arrived after it was written.

_sh(srcdir, expr) is still unread; that is #942, not this PR.

Non-blocking: "if not nodes:" in l will also match a comment that contains those words. Cheap, and the premise arm requires all three hits, so a missing line still fails.

[OffgridwithJD]

Stage 3: selftest/360 and selftest/380

Pushed 34b5fcc0 and f11b3386, rebased onto d05e3c39.

Two of the three shell files are done. Neither was a deletion, and in both cases
counting the FILE rather than the ARMS would have thrown away coverage that was
never debt.

360: rewritten, and it now checks three agreements where it checked one

Eleven of its seventeen arms were text pins on pgc_vacuity.py — the unrunnable
field is written, something reads it, the read reaches session.exitstatus, the
override is conditional. Gone. What is left is the kind CONTEXT.md permits: a
property that IS the relationship between the two harnesses, so it cannot be stated
from one side.

And there are three such properties where the part checked one. The INCOMPLETE
exit code was checked. The closed list of unrunnable reasons and the one-line shape
an unrunnable check prints were not, though both are written down twice in two
languages exactly as the exit code is. The reason list is the worse omission: being
closed on both sides is its entire purpose. All three are now parsed out of both
files — never restated in the part, which would test it against itself — and each has
a drifted fixture so the comparison can fail.

Measured before deleting anything. With session.exitstatus = EXIT_INCOMPLETE
made unreachable in pgc_vacuity.py, the defect exactly as it shipped, the four
behavioural arms in test_layer.py go 4 passed → 2 failed, 2 passed. Two rather
than four is correct, not partial: the other two assert exit 1 for a run with a real
failure and exit 0 for a run with nothing unrunnable, and neither outcome moves.
Restored byte-for-byte.

The part's own justification was stale. It said those arms "need pytest, psycopg
and a virtualenv; CI installs none of them". CI has a pytest-guards job that
installs pytest pinned from requirements-test.txt, asserts psycopg is absent, and
runs the database-free list — which contains test_layer.py. A stale reason for
keeping coverage in the wrong place is harder to find than a missing check, because
nothing reddens.

Two instrument defects of mine, both caught by the new arms' own premises: the shape
parse took the first line matching the marker, which in pgc_vacuity.py is the
docstring that spells the shape out for a reader; and the drifted fixture wrote one
space where the real shape has two, so the comparison was empty-against-real — which
"differs", for the wrong reason.

380: seven of twenty-one arms were the cross-harness ones

The part covers two #897 findings with different subjects. Finding 1 is
test/pgc_fingerprint.py, which lib.sh runs with the system interpreter and which
is not part of the pytest harness — it stays. Finding 2 is pgc_cluster.py, and its
nine arms are gone. The other fourteen checks read pgc_fingerprint.py, lib.sh, or
fixtures the part writes itself.

make_cluster's cleanup shape moved to test_build_refusal.py as
test_the_cleanup_guard_has_the_shape_the_leak_needs, beside the behavioural arm.
It is a source check and says so: make_cluster fails exactly one way there — a
missing pg_config — while three more properties decide whether the guard works, and
two cannot be provoked at all (you cannot deliver SIGINT into initdb reliably, and
a cluster that started is one the arm would then have to stop).

Five mutations, each asserted to have applied, each leaving the module well-formed,
restored byte-for-byte:

mutation of pgc_cluster.py	what reddens
except BaseException: → except Exception:	the source arm
cluster.stop() removed	the source arm
bare raise → pass	the source arm and the behavioural one
a private hashlib.md5 added	the no-private-digest arm
shutil.rmtree(root, …) removed	the source arm and the behavioural one

The two that redden both are the two whose effect reaches the filesystem. The three
that redden only the source arm are exactly the properties the behavioural arm cannot
see. One mutation did not apply first time — the cluster.stop() pattern assumed
twelve spaces and the call sits at sixteen, inside a nested try — and the harness
refused to report a result rather than printing a green.

A mistake worth more than either change

I ran the selftest before naming the two new pytest arms in TESTS.md, so the run
was red on the doc-coverage check — and then merged that log into
check_ledger.tsv, whose entire subject is which checks have ever been red. It
recorded a red for part 350's "every test file and every test in the corpus is named
in TESTS.md": a check that failed only because my own change was half-finished.
Reverted with git checkout HEAD --. There is no second copy to repair from.

The merge step now refuses a log that is not definitely green — a numeric
checks run: line, a floor on it so an aborted run cannot pass, zero FAIL lines,
and rc=0. Absence of a FAIL line is not enough: an aborted run has none either.
That is the same shape as a pending-count that cannot see a job which never started.

@jdatcmd — this is the second independent way that artifact can be poisoned in one
day. The first was a regeneration in a tree copied without .git, where 15 checks
failed on "premise: the source tree is a git checkout". Both are a red that the
check did not earn, and both were caught by a person looking rather than by
anything in the tooling. It may be worth pgc_ledger.py merge refusing a log that
does not reconcile to zero failures unless --mutation is given, since a mutation
run is the one case where reds are the point.

Gate, on f11b3386

harness_selftest.sh          805 checks run, 0 FAIL
pgc_ledger.py gate           rc=0, new this run=0, ceiling 250
driver-free pytest subset    198 passed
full pytest corpus           283 passed

Ledger rows regenerated from that green run for both parts: 360 went 17 → 16, 380
went 21 → 12, and checks_never_observed_red was re-derived to 804 rather than
typed.

Still to do on this branch

selftest/350 is the last shell file, and it is the largest: 50 checks, and its
arms split three ways rather than two. Roughly eighteen have a pytest twin already;
four have a shell subject (ci.yml) and stay; and about sixteen are self-tests of
its own detectors for properties the pytest side implements but does not test —
the contents-list anchor rule, and the mode-counting rule's edges (an id of fewer
than three words, an id named twice, stopping at the next heading, reading a row's
value rather than a digit in its label). Those have to be ported, not deleted, and
the pytest counting rule needs a seam so a fixture can reach it.

Plus the 7 pure-shell calls still in test_build_refusal.py, and the two
run_all_versions.sh --list-suites callers.

[OffgridwithJD]

Stage 4: selftest/350, and the shell→python direction is done

Pushed 70a90498.

selftest/350 goes from 50 checks to 5, and the end state for the whole direction
is measurable: exactly one shell part now holds a reference to anything under
test/pytest/, and it is deliberate.

shell parts referencing test/pytest/, at 70a90498
  360-an-unrunnable-pytest-test-must.sh   1 site

That one is selftest/360 reading pgc_vacuity.py to compare the three values both
harnesses write down — the INCOMPLETE exit code, the closed list of unrunnable
reasons, and the line an unrunnable check prints. CONTEXT.md's permitted
cross-reference: a property that IS the relationship, so it cannot be stated from one
side. The inventory claimed seven; the measured answer was three plus a deleted
fourth; the end state is one.

350: two rules moved rather than died

Forty-six of the fifty checks had their subject across the boundary — they globbed
test/pytest/*.py and parsed Python out of it, or swept the markdown in that
directory. The pytest corpus asserts all three properties natively in
test_docs_cover_the_corpus.py.

But two of the three rules were implemented on both sides and self-tested only on
the shell side — the side that cannot run the corpus it counts. Deleting those
fixtures would have left the Python implementation with no fixtures at all, so they
moved with them:

• the mode-counting rule's edges: an id of fewer than three words, an id named twice,
  stopping at the next heading, section 3's back-references, and the row reader
  taking the value cell rather than a digit inside its label
• the contents-list anchor rule: GitHub's derivation, the broken link that shipped,
  and a control beside it

_named_modes_in(text) and _stated_row(text, label) are new seams so a fixture can
reach the rules at all; _named_modes() delegates and the 19 existing arms are
untouched. The row-reader fixture makes the label digit and the value differ —
label "section 2", value 9 — because 350's own fixture had both as 2 and so could not
see the bug it was written for.

What stays in 350 is its arms over ci.yml, whose subject is neither harness, plus
one added while the subject was being settled: the job must assert psycopg is
absent rather than assume it.

The thing I most want on the record: a stale justification is invisible

@jdatcmd — this is the finding from this branch I would not want buried in a commit
message.

TESTS.md section 6 said:

Nothing runs pytest. Not run_all_versions.sh, not any workflow under
.github/. A guard written only here would never fire in the gate, and a guard
that does not run is a comment.

and concluded that the .sh half was therefore the enforcement. CI has a
pytest-guards job: it installs pytest pinned from requirements-test.txt, asserts
psycopg is absent, derives the file list from NO_CLUSTER, and runs it.

That claim was load-bearing in three places at once — TESTS.md section 6,
selftest/360, and selftest/380 — each asserting that the behavioural half could
not run in CI, and each using that to justify keeping a text-pin copy on the shell
side. Every assertion those reasons justified was green. Nothing was wrong anywhere.
The reason was, in triplicate.

No arm reddens on a stale justification. This is a class our sweeps cannot see,
and I only found it because moving arms out forced me to read each file's reason
before deleting them. I do not know how to sweep for it in general. The one handle I
can see: a reason that names a mechanism — "CI installs no pytest" — is a
measurement claim and could in principle be checked, whereas a reason that names a
judgement cannot. If that is worth an issue, the shape would be "a justification
that states a fact about the tree is a checkable claim and should be written so it
can be checked".

All three copies are corrected.

And a rule-3 instance of my own, the third in this PR

Two arms reddened with needs-a-cluster:test_docs_cover_the_corpus.py after the
anchor fixtures moved in. The cause was a literal test_harness_deps.py inside
sample markdown: _mentioned_files reads a corpus file name appearing in code as
"this file drives that one", so the membership classifier called the file
cluster-bound. Sample markdown about a heading, read as a dependency.

Assembling the name — "test" + "_harness" + "_deps" + ".py" — clears it, the same
convention SHELL_REFERENCES already uses for lib.sh. That is CONTEXT.md's rule 3
caught for the third time in this PR, twice by me, in the entry whose own text states
the rule. It is genuinely hard to avoid: the rule says a word that looks like a
filename is not a reference, and the only mechanical way to honour it is to stop the
word from existing.

Gate, on 70a90498

harness_selftest.sh        757 checks run, 0 FAIL
pgc_ledger.py gate         rc=0, new this run=0, ceiling 250
driver-free pytest         11 files, 207 passed
full pytest corpus         292 passed

Ledger rows regenerated from a green run: 350 went 53 → 5, and
checks_never_observed_red was re-derived to 756 rather than typed.

On pgc_ledger.py merge refusing a red log

My suggestion in the previous comment was aimed at the wrong hole, and @jdatcmd's
session measured it: merge already refuses a log that does not reconcile. What
it accepts is a well-formed, self-consistent log whose reds came from something other
than the code — which is both of the poisonings that happened today. Mine reconciled.
The .git-less one reconciled, 827 records against checks run: 827 with 15 FAILs.

The right shape is theirs: refuse a log containing FAIL by default, and make the
caller say which kind of red it is — --mutation NAME for a deliberate break, a
separate explicit opt-in for a real observation. The tool cannot tell an environment
red from a regression by looking, so it should make the caller assert it. And the
deciding fact is that nothing automated calls merge, so a default refusal breaks no
CI path. That work is theirs, as a separate issue, not this PR.

What is left on this branch

The python→shell direction: 7 pure-shell calls in test_build_refusal.py driving
pgc_write_source_stamp, pgc_source_stamp_path, pgc_freshness_report and
pgc_freshness_verdict, and two callers of run_all_versions.sh --list-suites in
test_suite_accounting.py and test_mutation_ledger.py. Those are pure shell rather
than wrappers over shared code, so the properties belong to the shell harness and
moving them means writing shell arms, not deleting python ones.

[jdatcmd]

Approving at 70a90498, 13/13 green. Your two judgement calls below, both answered from the code rather than from the principle — and the first has a sharper answer than the framing you offered.

The headline verified, with a control that moves

I measured your "exactly one shell part references anything under test/pytest/" myself, non-comment lines only:

#944   360-an-unrunnable-pytest-test-must.sh   1        -> 1 file
main   350: 3   360: 1   370: 1   380: 1                -> 4 files

I nearly reported this as wrong. My first sweep matched pgc_fingerprint.py and found three files, not one. That file lives in test/, not test/pytest/ — it is the shared module, which is the entire point of this PR. The claim is about the pytest corpus, and anchoring on pytest/ is what makes the measurement match the sentence.

1. Keep 350's ci.yml arms — and one of them is mandatory, not merely permitted

Your reasoning was that ci.yml is not a test suite, so a shell part may read it as it may read the Makefile. Agreed, but there is a stronger argument sitting in the part already, and it changes one arm from allowed to required.

Of the five arms, four duplicate test_harness_deps.py, which already reads ci.yml:

                                      selftest 350   test_harness_deps.py
ci.yml exists                              yes             yes
derives the file list from NO_CLUSTER      yes             yes
derives the pins from requirements-test    yes             yes
the job installs no driver                 yes             yes
`pytest-guards:` job exists                yes             NO

Those four are not redundancy — they are the standing rule that a property ships in both harnesses, independently implemented. Two arms reading one file for one property is exactly the pair the rule asks for.

The fifth cannot live on the pytest side at all. the gate runs the harness guards asserts the existence of the job that would run the assertion. Delete pytest-guards: from ci.yml and a pytest-side version of that arm does not fail — it never executes. That is the self-referential hole, and 350 is the only place it can be closed.

So the boundary I would write down is not "a shell part may not name pytest" but "a shell part may not read or execute the pytest corpus". ci.yml is configuration, and an arm about whether CI runs a suite has to live outside the suite.

2. Moving the two rules in was right, and the objection answers itself

Your alternative reading — a rule implemented on both sides should be self-tested on neither — fails on the standing rule. If a property ships in both harnesses, its self-test ships in both too; the question is only where the fixtures live, and fixtures belong on the side that can exercise the rule against the real corpus. The shell side cannot run the corpus it counts, which is the fact that decided it.

"Nine arms added in a PR about removing coupling" is not the contradiction it looks like. The PR's subject is coupling, not arm count. Moving a self-test from a harness that cannot exercise it to one that can reduces coupling and increases evidence at the same time; those pull the same way. A rule whose fixtures were deleted would pass because the document happens to agree with it today, which is the vacuity this directory spends its time refusing.

_named_modes_in and _stated_row are a fair price. They are seams inside one harness, not across two.

On your answer to the remainder

Go ahead on the python→shell remainder now. Both answers above are stable and neither depends on anything still moving: keep ci.yml arms in a shell part, and put a rule's fixtures where the rule can be run. The seven pure-shell calls in test_build_refusal.py are the interesting half, since pgc_write_source_stamp and friends have no python implementation to call — that is a decision per property, as you said, rather than a port.

Recorded

Your #943 finding is the one I would want on the record most: a rebase changed the accounting line from four terms to five, your fixture kept writing four, and both readers rejected it and agreed about it — so the arm whose entire subject is that they disagree on exactly one shape would have passed measuring nothing. A rebase is the most ordinary way for a fixture to stop expressing its question, and there is no sweep for it. The premise arm is what caught it, which is the argument for premise arms in one sentence.