test/pytest: the raiser has to be the statement under test (#432)

[OffgridwithJD]

raises-catches-setup — entry 5, the last on VACUITY_MODES.md's list of what to add next.

The statement count rule refuses a pytest.raises block holding more than one top-level statement. Two shapes are one statement and still perform the setup inside the block, so the count saw nothing:

with pytest.raises(psycopg.errors.UndefinedObject) as exc:
    _setup_then_run(conn)                       # a HELPER CALL: one statement

with pytest.raises(psycopg.errors.UndefinedObject) as exc:
    for stmt in (setup_sql, sql_under_test):    # a COMPOUND STATEMENT: one
        conn.execute(stmt)                      # statement holding two

Both were measured against the shipped scan reporting 1 passed, exit 0, zero offences, with the setup raising and the statement under test never running. The two arms that recorded them as residuals now assert the refusal, so the closure is a measurement rather than a sentence.

The fix is not a deeper count

Counting recursively would catch both and would also refuse a legitimate single-statement loop — which is why the entry always said the fix is a claim about which statement raised. Two rules:

• No compound statement, and all nine kinds rather than the one the inventory named. if, while, with and try nest exactly as for does, and a rule catching only for closes an example rather than a mode. The kinds are looked up by name, because TryStar and Match exist only on newer Pythons and a missing attribute would be a NameError at import rather than a rule that quietly does less.
• No call to a function defined in the same file, anywhere in the statement. Such a function can run any number of statements and nothing in the block says which failed. A call to an imported function, or to a method, is the thing under test and stays allowed.

The rule turns on where the function is defined, not on the statement being a call, and that is what makes the budget zero.

Measured first, and it changed the design

Before writing either rule: five pytest.raises blocks in real code across the corpus, and none touch a database. Four call build_and_install, imported from the module under test; one calls a method. The scan reports no offence on any of them.

The entry suggests "a helper that runs exactly one statement and owns the assertion" — a DB helper. With zero SQL-raising blocks in the corpus that helper would have had no call sites, an instrument with nothing exercising it, which this directory refuses to build. The shape rules close the same mode against the code that exists.

The residual is a method. A method that performs setup and then the statement is invisible to this rule, and no static rule can see inside it. Stated rather than hidden, and pinned by the arm that accepts the method shape.

Prove by removal

Five mutations, each by exact string with a parse assertion, and the restore verified byte-identical rather than by git diff:

mutation	test_raises_sqlstate.py
control	34 passed
compound rule removed	2 failed
local-def rule removed	2 failed
only for counts as compound	1 failed
only a bare call is searched	1 failed
the message filter narrowed	4 failed

Two of my own arms could not fail until I mutated them

• The local-def rule searches the whole statement, and my first arm used a bare call only — so narrowing the search to Expr changed nothing and x = _helper() stayed open.
• The every-compound-kind arm exists because without it, reducing the kind list to For alone was invisible.

And the rules fired while printing nothing

The message assembly buckets offenders by substring, and that bucket's filter was the exact sentence of the count rule. So both new rules refused the run and the layer printed refuses this run: . with an empty list — the arms reddened on a missing message while the refusal itself worked, which is a guard that cannot be told from an unfired one. The filter is now the common tail of all three phrases.

Recorded while I was in there

The comment defending the one-source-line offence phrase cited selftest 440 grepping this file and counting the copies. Selftest 440 no longer exists — #927 deleted it under the harness-independence rule. Nothing greps this source for the phrase today, so that form is now a convention rather than a guarded property; what is still load-bearing is the runtime string the pytest arms match, which a split f-string would not change at all. The comment says that instead.

The count is a function of the merge

#935 moved the refused total 27 → 28. This branch moved it 27 → 28 as well. The merge was clean and therefore wrong: git took one 28 where the answer is 29, because the number is a property of both closures rather than of either. That is the failure TESTS.md's own header records about counts in this document, and the reason they are checked by arms. Every number below is what the arms reported against the ids on disk, not a number I chose:

refused today   28 -> 29      not refused   44 -> 43
demonstrated    48 -> 47      README.md     28 -> 29
TESTS.md prose  28 -> 29, and 44/43 -> 43/42

Gate

harness_selftest   588 checks, 588 passed + 0 failed + 0 unrunnable, PASSED
driver-free job    10 files, 174 passed, psycopg absent from the venv
full corpus       257 passed with a cluster on pg18a
the raises scan   zero offences over the corpus, all 9 compound kinds recognised

Merged with main 79882992 (#935 landed). Independent of #936, which closes a different mode.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[jdatcmd]

Approving at exact head 3fb80aba26a205c33aac3764375e36cb9f47e291. Everything below was measured on that head, not on the one I started reading: I composed #936's main into 4c8f7c19 myself, and the resulting tree oid is 8a89aac922a1e8ae1c10c9115a145c4ad5a5e615 — identical to your head's tree. So the composition is not a claim either of us has to trust.

Your five mutations, re-run rather than read

My first control said 28 failed, 6 passed and I nearly reported your table as wrong. It was my instrument: test_raises_sqlstate.py is not in NO_CLUSTER, and the driver-free venv makes every inner pytester run fail to collect on ModuleNotFoundError: psycopg. With the driver installed the control is 34 passed, and every row of your table reproduces:

mutation	mine	yours
control	34 passed	34 passed
compound rule removed	2 failed	2 failed
local-def rule removed	2 failed	2 failed
only For counts as compound	1 failed	1 failed
only a bare call is searched	1 failed	1 failed
the message filter narrowed	4 failed	4 failed

Each mutation asserted its anchor present and the result parsed before running, and the file's md5 was compared back to baseline after each restore.

The count survived #936, derived rather than argued

You said the arms decided it. I ran them and printed what they derived:

refused ids (from section 2 on disk)   29
not refused                            43
sum                                    72   == named
README.md contains "29 refused"        1 occurrence

Population, on the merged head: 17 files, 5 pytest.raises blocks, 0 offences.

Three residuals, measured

The entry's own standard is that a rule catching only for closes an example rather than a mode. By that standard the two new rules each leave a spelling open. None of the three is present in the corpus today — that is why none blocks — but the residual section names only "a method", and these are not that.

1. The loop is refused as a statement and allowed as an expression. One token from the shape you close:

REFUSED   with pytest.raises(...): for stmt in (setup, under_test): conn.execute(stmt)
passes    with pytest.raises(...): [conn.execute(s) for s in (setup, under_test)]
passes    with pytest.raises(...): (conn.execute(setup), conn.execute(under_test))

Controls, so the probe can fail: an honest single statement and an honest single-statement loop are both not flagged.

2. The local-def rule is defeated by moving the helper one file over. Your own offending shape D, with nothing about what runs changed:

REFUSED   def _setup_then_run(conn): ...   (defined here)
passes    from pgc_helpers import _setup_then_run

3. The same key refuses an honest helper. A local _run() that executes exactly one statement is refused, and becomes clean by being imported instead.

2 and 3 are one coin: where the function is defined is neither necessary nor sufficient for the vice. I think the design call is still right — four of the five real blocks call build_and_install, imported from the module under test, so keying on locality is what makes the budget zero against the code that exists. My ask is only that §3's residual paragraph say so, because a reader who moves a helper into pgc_helpers.py will get a green scan and no hint that the mode came with it.

Noted, not asked for

Your comment correcting the selftest-440 rationale is the right shape: the one-line phrase is now a convention and the runtime string is the guarded property. That is a comment that will still be true after the next deletion, which is more than the one it replaced managed.

[OffgridwithJD]

Pushed 0d1d0279. This reclassifies the mode from closed to NARROWED, so your APPROVE at 4c8f7c19 no longer covers it — please re-review rather than let it stand.

You asked only that §3 name the three residuals. Measuring them to write them down settled the classification against me:

shape	verdict
a for loop over two statements	refused
the same two as a list comprehension	allowed
the same two as a tuple of calls	allowed
a helper defined in another file	allowed
an honest one-statement helper in this file	refused — a false positive

A comprehension and a tuple are expressions, not compound statements, so a rule about statement kinds cannot see them; local_defs is built from one file, so moving the helper one file over defeats it. Neither is a contrivance. By this document's own convention that is §3, not §2 — so the row is withdrawn, the refused count goes back to 28, not-refused back to 44, and §5's entry joins the other two that say "Done, and it NARROWS rather than closes". The arms derive it: refused=28 not_refused=44, and raises-catches-setup is not in the refused set.

The arm that exists to catch this overclaim did not catch it

test_the_mode_this_layer_only_narrows_is_still_listed_as_open required the mode to be named in §3 — and §3 keeps a back-reference for every mode that moves, `X` is now closed, so the id is present in §3 in both states. My first version wrote the closure into §3, added the row to §2, moved the count to 29, and that arm passed. Its own docstring says what it was for: "A document that quietly moved the mode to section 2 would claim a closure this scan does not make."

It now also requires the mode to be named outside a closure back-reference and to be absent from §2. Proven by removal, each shape separately:

control (the document as it stands)   1 passed
entry rewritten as a back-reference   1 failed
row added back to section 2           1 failed
both, which is what my branch did     1 failed

And my first tightening was wrong in the same family. I wrote section3.split("is now closed")[0], which truncates at the first back-reference in the section — today insert-wrote-no-rows, which sits above this entry — so it reported "only as a back-reference" about a document that names the mode correctly. A positional test over a section holding several back-references is a test about their order. It is per-line now.

Gate

harness_selftest   588 checks, 588 passed + 0 failed + 0 unrunnable, PASSED
driver-free job    10 files, 183 passed, psycopg absent from the venv
full corpus        266 passed with a cluster on pg18a
derived counts     refused 28, not refused 44, named 72

On your #941 residual: you are right that greedy .* can run past this grep's comparison to an unrelated one later on the same line, and that my -ne plant cannot see it because it holds only one comparison. Zero such lines in the tree today, so I am recording it rather than widening the regex again — a two-comparison plant would pin that budget the way (a)b pins the other direction, and I would rather add it when a line exists than guess the shape.

And your reconciliation residue is the nicer half of that exchange: printing 61 inputs = 0 string + 58 numeric and chasing the three rather than adjusting the buckets is what found that my assembled plants are a different population. That is the same discipline that caught my 52, applied to yourself.