From 938ac957422341b18002f91dae0d89d8964f7114 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Thu, 10 Sep 2026 21:50:51 +0000 Subject: [PATCH 1/3] test/pytest: the raiser has to be the statement under test (#432) `raises-catches-setup`, entry 5 and the last on VACUITY_MODES.md's list of what to add next. The statement COUNT rule refused a `pytest.raises` block holding more than one top-level statement. Two shapes are ONE statement and still perform the setup inside the block, so the count saw nothing: with pytest.raises(psycopg.errors.UndefinedObject) as exc: _setup_then_run(conn) # a HELPER CALL: one statement with pytest.raises(psycopg.errors.UndefinedObject) as exc: for stmt in (setup_sql, sql_under_test): # a COMPOUND STATEMENT: one conn.execute(stmt) # statement holding two Both were measured against the shipped scan reporting `1 passed`, exit 0, zero offences, with the setup raising and the statement under test never running. The two arms that recorded them as residuals now assert the refusal, so the closure is a measurement rather than a sentence. THE FIX IS NOT A DEEPER COUNT, for the reason the entry always gave: counting recursively would also refuse a legitimate single-statement loop. It is a claim about WHICH statement raised, in two rules. NO COMPOUND STATEMENT, and all nine kinds rather than the one the inventory named. The document named the `for` spelling; `if`, `while`, `with` and `try` nest identically, and a rule catching only `for` would close an example rather than a mode. The kinds are looked up by name instead of written out, because `TryStar` and `Match` exist only on newer Pythons and a missing attribute would be a NameError at import rather than a rule that quietly does less. NO CALL TO A FUNCTION DEFINED IN THE SAME FILE, anywhere in the statement. Such a function can run any number of statements and nothing in the block says which failed. A call to an IMPORTED function, or to a METHOD, is the thing under test and stays allowed. THE RULE TURNS ON WHERE THE FUNCTION IS DEFINED, NOT ON THE STATEMENT BEING A CALL, and that is what makes the budget zero. Measured over the corpus before writing either rule: five `pytest.raises` blocks in real code, NONE touching a database -- four call `build_and_install`, imported from the module under test, and one calls a method. The scan reports no offence on any of them. MEASURED FIRST, AND IT CHANGED THE DESIGN. The entry suggests "a helper that runs exactly one statement and owns the assertion", which would be a DB helper. There are zero SQL-raising `pytest.raises` blocks in the corpus, so that helper would have had no call sites -- an instrument with nothing exercising it, which is the thing this directory refuses to build. The shape rules close the same mode against the code that exists. THE RESIDUAL IS A METHOD. A method that performs setup and then the statement is invisible to this rule, and no static rule can see inside it. Stated rather than hidden, and pinned by the arm that accepts the method shape. Prove by removal, five mutations, each by exact string with a parse assertion, and the restore verified byte-identical rather than by `git diff`: control 34 passed compound rule removed 2 failed local-def rule removed 2 failed only `for` counts as compound 1 failed only a bare call is searched 1 failed the message filter narrowed 4 failed TWO OF MY OWN ARMS COULD NOT FAIL UNTIL I MUTATED THEM. The local-def rule searches the WHOLE statement, and my first arm used a bare call only -- so narrowing the search to `Expr` changed nothing and `x = _helper()` stayed open. And the every-compound-kind arm exists because without it, reducing the kind list to `For` alone was invisible. AND THE RULES FIRED WHILE PRINTING NOTHING. The message assembly buckets offenders by substring, and the bucket's filter was the exact sentence of the COUNT rule -- so both new rules refused the run and the layer said `refuses this run: .` with an empty list. The arms reddened on a missing message while the refusal itself worked, which is a guard that cannot be told from an unfired one. The filter is now the common tail of all three phrases. RECORDED WHILE I WAS IN THERE: the comment defending the one-source-line offence phrase cited selftest 440 grepping this file and counting the copies. **Selftest 440 no longer exists** -- #927 deleted it under the harness-independence rule. Nothing greps this source for the phrase today, so that form is now a convention rather than a guarded property; what is still load-bearing is the RUNTIME string the pytest arms match, which a split f-string would not change at all. The comment says that instead. Gate: harness_selftest 588 checks, 588 passed + 0 failed + 0 unrunnable, PASSED driver-free job 9 files, 161 passed, psycopg absent from the venv full corpus 241 passed with a cluster on pg18a Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- test/pytest/README.md | 2 +- test/pytest/TESTS.md | 51 ++++++-- test/pytest/VACUITY_MODES.md | 47 ++++---- test/pytest/pgc_vacuity.py | 93 +++++++++++++-- test/pytest/test_raises_sqlstate.py | 175 ++++++++++++++++++++-------- 5 files changed, 274 insertions(+), 94 deletions(-) diff --git a/test/pytest/README.md b/test/pytest/README.md index 05db5b6d..27657f52 100644 --- a/test/pytest/README.md +++ b/test/pytest/README.md @@ -5,7 +5,7 @@ This is the issue #432 pilot. It runs beside `test/*.sh`, and replaces nothing. - `TESTS.md` in this directory documents every test and every assertion helper. - `VACUITY_MODES.md` is the inventory of ways a pytest harness can report a false pass: 79 modes produced by the enumeration, 72 of them named in that - file, 73 demonstrated by a run, and 27 refused by this layer today. + file, 73 demonstrated by a run, and 28 refused by this layer today. VACUITY_MODES.md section 1a gives the counting rule and reconciles the run's totals against what is actually written down. - `design/ISSUE_432_PYTEST_HARNESS.md` holds the design and the measurements diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index aeee6636..c9b40041 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1375,8 +1375,8 @@ over tests nothing ran. ## 18. What this corpus does NOT yet refuse `VACUITY_MODES.md` is the inventory: 79 ways a pytest harness can report a pass while -asserting nothing, 73 of them demonstrated by an actual run. **This layer refuses 27 -of them.** The other 45, of which 44 were demonstrated, are listed there with the +asserting nothing, 73 of them demonstrated by an actual run. **This layer refuses 28 +of them.** The other 44, of which 43 were demonstrated, are listed there with the refusal design each would need and the order worth building them in. Read it before adding a test. Two gaps are most likely to affect a new test now. @@ -1470,10 +1470,9 @@ performs the setup inside the block: - **a compound statement.** A `for` over the setup and the statement under test is one statement holding two; an `if`, a `with` or a `try` nests the same way. -Measured against the shipped scan, both report `1 passed`, exit 0, and **zero -offences**. `test_a_helper_hiding_the_setup_is_not_refused` and -`test_a_compound_statement_hiding_the_setup_is_not_refused` assert exactly that, so -the residual is a measurement rather than a sentence. Counting statements +Both were measured reporting `1 passed`, exit 0 and **zero offences**, with the setup +raising and the statement under test never running. **Both are now refused**, and the +arms that recorded them as residuals assert the refusal instead. Counting statements recursively would catch both and would also refuse a legitimate single-statement loop; what would close the mode is a claim about WHICH statement raised, and `VACUITY_MODES.md` section 5 carries it as the next entry. @@ -1541,10 +1540,46 @@ one condition faithfully, and require the copy to go blind. | `test_sqlstate_refuses_an_empty_set_of_codes` | and an empty tuple is satisfied by nothing, so the hatch is not the hole | | `test_the_raises_scan_leaves_the_unrunnable_state_alone` | a documented hatch the corpus never exercises: `cannot_run` still prints `UNRUN`, counts it, and exits 67 with this scan loaded | | `test_the_raises_scan_does_not_touch_a_recorder_made_in_the_body` | a test that fetches `expect` itself still satisfies the layer, because this scan runs at collection time | -| `test_a_helper_hiding_the_setup_is_not_refused` | **residual 1 of 2, pinned.** One statement, a narrow class, a pinned SQLSTATE, and the setup inside the helper still raised: `1 passed`, no offence | -| `test_a_compound_statement_hiding_the_setup_is_not_refused` | **residual 2 of 2, pinned.** A `for` holding the setup and the statement under test is one top-level statement: `1 passed`, no offence | +| `test_a_helper_hiding_the_setup_is_refused` | a call to a function **defined in the same file** cannot say which statement raised | +| `test_a_compound_statement_hiding_the_setup_is_refused` | a `for` holding the setup and the statement under test is one top-level statement, and refused | +| `test_a_helper_hidden_in_an_assignment_is_refused_too` | the rule looks anywhere in the statement: `x = _helper()` hides the setup as well as a bare call | +| `test_every_compound_statement_is_refused_not_only_a_loop` | `if`, `while`, `with` and `try` nest the same way, so all nine compound kinds are refused | +| `test_a_raises_block_calling_an_imported_function_is_accepted` | the budget: four of the five blocks in this corpus call an imported function | +| `test_a_raises_block_calling_a_method_is_accepted` | the fifth block's shape, accepted, with the residual it leaves stated | | `test_a_conftest_cannot_switch_the_broad_family_list_off` | the rule's own family list is not writable from the corpus it polices | | `test_a_bare_sqlstate_expression_does_not_pin_anything` | `exc.value.sqlstate` as a statement of its own asserts nothing, so mentioning the field is not pinning it | + +### How `raises-catches-setup` closed + +The count rule refuses a block holding more than one top-level statement. Two shapes +are **one** statement and still hide the setup inside the block, so the count saw +nothing: + +```python +with pytest.raises(psycopg.errors.UndefinedObject) as exc: + _setup_then_run(conn) # a helper call: one statement + +with pytest.raises(psycopg.errors.UndefinedObject) as exc: + for stmt in (setup_sql, sql_under_test): # a compound: one statement + conn.execute(stmt) # holding two +``` + +The fix is **not** a recursive count — that would also refuse a legitimate +single-statement loop. It is a claim about which statement raised, in two rules: + +- **No compound statement.** All nine kinds Python has, looked up by name rather than + written out so a missing `TryStar` or `Match` is not a NameError at import. +- **No call to a function defined in the same file**, anywhere in the statement — a + helper hides as well in `x = _helper()` as in a bare call. A call to an **imported** + function or to a **method** is the thing under test and stays allowed. + +**The rule turns on where the function is defined, not on the statement being a call**, +and that is what makes the budget zero. Measured over the corpus: five +`pytest.raises` blocks, four calling `build_and_install` (imported) and one calling a +method, and the scan reports **no offence** on any of them. + +**The residual is a method.** A method that performs setup and then the statement is +invisible to this rule, and no static rule can see inside it. | `test_a_sqlstate_assigned_and_never_read_does_not_pin_anything` | the same hole one step on: bound to a name nothing uses | | `test_one_hop_through_a_local_name_is_an_honest_pin` | the cost side — `code = exc.value.sqlstate` then `expect.text(code, ...)` stays collectable | | `test_the_keyword_form_is_checked_by_both_rules` | `pytest.raises(expected_exception=...)` is not an exemption from either rule | diff --git a/test/pytest/VACUITY_MODES.md b/test/pytest/VACUITY_MODES.md index fbc98802..51dc0a0f 100644 --- a/test/pytest/VACUITY_MODES.md +++ b/test/pytest/VACUITY_MODES.md @@ -47,8 +47,8 @@ recollection of the run: | | modes | | --- | ---: | -| named in section 2, refused today | 27 | -| named in section 3, not refused | 45 | +| named in section 2, refused today | 28 | +| named in section 3, not refused | 44 | | **named in this document** | **72** | | produced by the enumeration run | 79 | | **named nowhere here** | **7** | @@ -65,7 +65,7 @@ an id can be read, argued with and turned into a test, and a number cannot. ## 2. What the layer refuses today -27 of the 79, counted by section 1a's rule. Each is enforced by a mechanism, not a convention, and each has a red +28 of the 79, counted by section 1a's rule. Each is enforced by a mechanism, not a convention, and each has a red test in `test_layer.py` that fails without it. | mechanism | modes it closes | @@ -87,6 +87,7 @@ test in `test_layer.py` that fails without it. | an empty parameter set fails the run, with its own message | `empty-parametrize-is-a-silent-skip` | | a skip during fixture setup fails the run | `session-fixture-skip-greens-the-whole-suite` | | a broad `pytest.raises` must pin a SQLSTATE, found by AST | `raises-too-broad` | +| a `pytest.raises` block may not hold a compound statement or call a function this file defines | `raises-catches-setup` | | every comparison refuses a value carrying the `QUERY_ERROR` prefix, on either side | `error-swallowed-to-empty` | Three of those were added after checking this layer against the inventory rather @@ -120,7 +121,7 @@ guard whose subject is false greens has no business emitting a false red. ## 3. What it does not refuse -55 modes by the run's count, **45 of them named below**, **49 demonstrated by a run**. 51 have a refusal already designed. +55 modes by the run's count, **44 of them named below**, **48 demonstrated by a run**. 51 have a refusal already designed. Grouped by what a reader needs to decide about them. ### 3.1 The run can lose tests and still exit 0 @@ -185,33 +186,33 @@ Still open in this family: `DatabaseError`, `Exception` or `BaseException` does not collect unless the block binds the exception and the body pins its SQLSTATE. See section 2. -- `raises-catches-setup` — **still open, and the statement rule only narrows it.** - The scan refuses a `pytest.raises` block holding more than one TOP-LEVEL - statement, so the spelling where the setup sits on the line above the statement - under test is gone. Two shapes walk straight past a count of top-level - statements, and each is one statement that performs the setup inside the block: +**`raises-catches-setup` is now closed.** The statement COUNT rule refused a block + holding more than one top-level statement, and two shapes are ONE statement that still + performs the setup inside the block: with pytest.raises(psycopg.errors.UndefinedObject) as exc: _setup_then_run(conn) # a HELPER CALL: one statement - expect.sqlstate(exc.value, "42704", "the ALTER was refused") with pytest.raises(psycopg.errors.UndefinedObject) as exc: for stmt in (setup_sql, sql_under_test): # a COMPOUND STATEMENT: one conn.execute(stmt) # statement holding two - expect.sqlstate(exc.value, "42704", "the ALTER was refused") - Measured against the shipped scan: both report `1 passed`, exit 0, **zero + Both were measured against the shipped scan reporting `1 passed`, exit 0, **zero offences**, with the setup raising and the statement under test never running. - An `if`, a `with` or a `try` nests the same way. Counting statements RECURSIVELY - would catch these and would also refuse a legitimate single-statement loop, so - the fix is not a deeper count — it is a claim about WHICH statement raised: a - position, or a helper that runs exactly one statement and owns the assertion. - - `test_a_helper_hiding_the_setup_is_not_refused` and - `test_a_compound_statement_hiding_the_setup_is_not_refused` in - `test_raises_sqlstate.py` assert the scan reports nothing on these two shapes, so - the gap is a measurement rather than a sentence, and `test_raises_sqlstate.py` requires both - arms plus this entry to still exist. + + THE FIX IS NOT A DEEPER COUNT, for the reason this entry always gave: counting + recursively would also refuse a legitimate single-statement loop. It is a claim about + WHICH statement raised, in two rules — no compound statement (all nine kinds, looked + up by name so a missing `TryStar` or `Match` cannot silently narrow the rule), and no + call to a function DEFINED IN THE SAME FILE, anywhere in the statement. A call to an + IMPORTED function or to a METHOD is the thing under test and stays allowed, which is + what makes the budget zero: measured over the corpus, five `pytest.raises` blocks — + four calling an imported function, one calling a method — and no offence on any. + + THE RESIDUAL IS A METHOD. A method that performs setup and then the statement is + invisible to this rule, and no static rule can see inside it. The two arms that used + to assert these shapes were NOT refused now assert that they are, so the closure is a + measurement rather than a sentence. See TESTS.md section 20. - `same-broken-helper-both-sides`, `truthy-error-string`, `assert-not-unset-error`, `zero-on-both-arms`, `tuple-assert-always-true`, `approx-of-nothing` - `same-broken-helper-both-sides`, `truthy-error-string`, `assert-not-unset-error`, @@ -332,6 +333,6 @@ have tried to defeat them did not run. Every design states its own residual, and those residuals are the authors' own, unchallenged. So treat §2 as measured, §3 as measured, and §5 as a plan that has not yet met an -adversary. The layer is known to refuse 27 demonstrated modes -- the ids named in section 2, +adversary. The layer is known to refuse 28 demonstrated modes -- the ids named in section 2, not the run's larger total, for the reason section 1a gives. It is not known to be undefeatable on any of them. diff --git a/test/pytest/pgc_vacuity.py b/test/pytest/pgc_vacuity.py index 02dd9836..cbff15ad 100644 --- a/test/pytest/pgc_vacuity.py +++ b/test/pytest/pgc_vacuity.py @@ -1115,6 +1115,20 @@ def _sqlstate_pinned_names(fn): return pinned +# Every statement that can HOLD other statements. Built by lookup rather than +# written out, because `TryStar` and `Match` exist only on newer Pythons and a +# missing name would be a NameError at import rather than a rule that quietly does +# less. The inventory named only the `for` spelling; a rule catching only that one +# would leave three spellings of the same shape, which is closing an example rather +# than a mode. +_COMPOUND_STATEMENTS = tuple( + c for c in (getattr(ast, n, None) for n in ( + "For", "AsyncFor", "While", "If", "With", "AsyncWith", "Try", "TryStar", + "Match", + )) if c is not None +) + + def _raises_sites(path): """Every `with pytest.raises(...)` in one file, and what is wrong with it. @@ -1200,6 +1214,14 @@ def _raises_sites(path): return [] out = [] name = pathlib.Path(path).name + # EVERY FUNCTION THIS FILE DEFINES, nested ones included. A `pytest.raises` block + # whose one statement calls one of these is the helper shape: the helper can run + # any number of statements and nothing in the block says which of them failed. A + # call to an IMPORTED function, or a method, is the thing under test -- which is + # the shape all five blocks in this corpus use, so the rule turns on where the + # function is DEFINED rather than on the statement being a call. + local_defs = {n.name for n in ast.walk(tree) + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))} for fn in [n for n in ast.walk(tree) if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))]: pinned = _sqlstate_pinned_names(fn) @@ -1240,13 +1262,18 @@ def _raises_sites(path): broad = [c for c in _raises_class_names(expected) if c in broad_families] if broad and (bound is None or bound not in pinned): - # THE OFFENCE PHRASE STAYS ON ONE SOURCE LINE. The message - # assembly below filters for this exact substring, and selftest - # 440 counts both copies to catch them drifting apart. Split as - # `"... names no " f"SQLSTATE"` it reads identically at runtime - # and the arm counts one where it wants two -- a guard that can - # no longer see its own drift. Measured: that split is what - # reddened 440 the first time this scan ran under it. + # THE OFFENCE PHRASE STAYS ON ONE SOURCE LINE, and the reason + # has CHANGED. It was selftest 440, which grepped this source for + # the phrase and counted the copies, so a split into + # `"... names no " f"SQLSTATE"` read identically at runtime while + # the count saw one where it wanted two. **Selftest 440 no longer + # exists** -- #927 deleted it under the harness-independence rule, + # because a shell part asserting a text pin cannot prove a python + # arm is caught. Nothing greps this source for the phrase today, so + # the one-line form is now a convention rather than a guarded + # property. What IS still load-bearing is the RUNTIME string: the + # arms in test_raises_sqlstate.py match it against stderr, and a + # split f-string would not change that at all. where = f"{name}:{call.lineno}" out.append( f"{where} pytest.raises({broad[0]}) names no SQLSTATE" @@ -1260,6 +1287,42 @@ def _raises_sites(path): f"{held} {len(node.body)} statements, " f"so which one raised is not pinned" ) + # AND ONE STATEMENT IS NOT ENOUGH, which is the half `raises-catches-setup` + # stayed open on. Two shapes are one top-level statement and still hide the + # setup inside the block, so the count rule above saw nothing: + # + # with pytest.raises(...): _setup_then_run(conn) # a helper call + # with pytest.raises(...): # a compound + # for stmt in (setup, under_test): run(stmt) + # + # Both were measured reporting `1 passed`, exit 0, zero offences, with the + # setup raising and the statement under test never running. The fix is not a + # RECURSIVE count -- that would also refuse a legitimate single-statement + # loop -- it is a claim about which statement raised. + elif sites: + only = node.body[0] + kind = type(only).__name__ + if isinstance(only, _COMPOUND_STATEMENTS): + # One source line for the phrase, as above. + out.append( + f"{name}:{node.lineno} the pytest.raises block holds a {kind}, " + f"so which statement inside it raised is not pinned" + ) + else: + # ANYWHERE IN THE STATEMENT, not only as the whole of it: a helper + # hides just as well in `x = _helper()` or `assert _helper()` as it + # does in a bare call. + called = sorted({ + n.func.id for n in ast.walk(only) + if isinstance(n, ast.Call) and isinstance(n.func, ast.Name) + and n.func.id in local_defs + }) + if called: + out.append( + f"{name}:{node.lineno} the pytest.raises block calls " + f"{called[0]}(), defined in this file, so which statement " + f"raised is not pinned" + ) return out @@ -1316,8 +1379,13 @@ def pytest_collection_modifyitems(config, items): excepts = [o for o in offenders if "catches Exception broadly" in o] ordered = [o for o in offenders if "feeds an ordered claim" in o] raises_broad = [o for o in offenders if "names no SQLSTATE" in o] - raises_setup = [o for o in offenders - if "which one raised is not pinned" in o] + # THE FILTER IS THE COMMON TAIL OF ALL THREE PHRASES. It was the exact + # sentence of the statement-COUNT rule, so the two rules added for the helper + # and compound shapes refused the run and then printed NOTHING -- the layer + # said "refuses this run: ." and the arms could not tell a fired rule from an + # unfired one. Measured: both new arms reddened on a missing message while the + # refusal itself was working. + raises_setup = [o for o in offenders if "raised is not pinned" in o] parts = [] if skips: parts.append( @@ -1350,9 +1418,10 @@ def pytest_collection_modifyitems(config, items): ) if raises_setup: parts.append( - "a pytest.raises block holding more than one statement cannot say " - "which statement raised, so a failure in the SETUP passes for a " - "failure in the statement under test: " + "a pytest.raises block must say WHICH statement raised, or a " + "failure in the SETUP passes for a failure in the statement under " + "test -- more than one statement, a compound statement holding " + "several, or a call to a helper defined in the same file all hide it: " + "; ".join(raises_setup) + " -- move the setup above the block, leaving the statement under " "test alone inside it" diff --git a/test/pytest/test_raises_sqlstate.py b/test/pytest/test_raises_sqlstate.py index 5b5a76bf..55d98a25 100644 --- a/test/pytest/test_raises_sqlstate.py +++ b/test/pytest/test_raises_sqlstate.py @@ -26,8 +26,8 @@ nothing pinned cannot be collected. Property 2 only MITIGATES `raises-catches-setup`, because it counts TOP-LEVEL statements: a call to a helper that performs the setup is one statement, and so is a `for` holding the setup and -the statement under test. `test_a_helper_hiding_the_setup_is_not_refused` and -`test_a_compound_statement_hiding_the_setup_is_not_refused` assert this scan +the statement under test. `test_a_helper_hiding_the_setup_is_refused` and +`test_a_compound_statement_hiding_the_setup_is_refused` assert this scan reports NOTHING on those two shapes, which is why `raises-catches-setup` stays in `VACUITY_MODES.md` section 3.4 rather than moving to section 2. @@ -408,20 +408,19 @@ def test_pins_its_error(expect): # below are one top-level statement that performs the setup inside the block. # --------------------------------------------------------------------------- -def test_a_helper_hiding_the_setup_is_not_refused(pytester, expect): - """Residual shape 1 of 2: a CALL TO A HELPER that performs the setup. +def test_a_helper_hiding_the_setup_is_refused(pytester, expect): + """Shape 1 of the two `raises-catches-setup` residuals: a CALL TO A HELPER. - One statement, a narrow class, a pinned SQLSTATE, and the setup still raised. - The block holds one top-level statement, so the statement rule is satisfied; - the class names a single SQLSTATE and the test pins it, so the SQLSTATE rule is - satisfied; and the error came from inside the helper rather than from the - statement the test is about. Against this guard: `1 passed`, exit 0, and the - scan reported no offence. + One top-level statement, a narrow class, a pinned SQLSTATE -- and the setup + raised while the statement the test names never ran. The statement COUNT cannot + see this, because a helper call is one statement. - What would close it is a claim about WHICH statement raised -- a position, or a - helper that runs exactly one statement and owns the assertion -- and this scan - makes no such claim. So `raises-catches-setup` stays in `VACUITY_MODES.md` - section 3.4. + THE RULE IS NOT A DEEPER COUNT. Counting recursively would also refuse a + legitimate single-statement loop. The claim is about which statement raised, so + the block may not call a function DEFINED IN THE SAME FILE: such a function can + run any number of statements, and nothing in the block says which of them failed. + A call to an imported function or to a method is the thing under test and stays + allowed -- that is the shape all five `pytest.raises` blocks in this corpus use. """ result = _inner(pytester, ''' import psycopg @@ -437,54 +436,130 @@ def test_the_alter_is_refused(expect): _do_the_thing(True) expect.sqlstate(exc.value, "42704", "the statement under test") ''') - expect.outcomes(result, "the setup hidden in a helper is NOT refused -- the " - "first residual this guard leaves open", - passed=1, failed=0) - # `passed=1` IS THE ZERO-OFFENCE ASSERTION, and it is the whole of it. An - # offence makes the layer raise pytest.UsageError, which collects nothing and - # exits 4, so `passed=1` is reachable only when the scan reported nothing at - # all. A `result.stderr.no_fnmatch_line("*names no SQLSTATE*")` after this line - # was tried and removed: `expect.outcomes` fails first in the only case where an - # offence could appear, so the extra line can never execute and reads as a check - # while being none. Measured -- under a mutation that flags every block, both - # residual arms redden on `outcomes` and the stderr lines are never reached. - - -def test_a_compound_statement_hiding_the_setup_is_not_refused(pytester, expect): - """Residual shape 2 of 2: a COMPOUND STATEMENT holding the setup and the - statement under test. - - A `for` over two statements is ONE top-level statement, so `len(node.body) == 1` - and the rule is satisfied. The loop raises on its FIRST iteration -- the setup -- - and the statement the test names never runs. Measured here as `1 passed`, exit 0, - with the scan reporting no offence, which is the same failure - `test_setup_inside_a_raises_block_is_refused` catches when the two statements sit - side by side instead. - - An `if`, a `with`, or a `try` nests the same way. Counting statements RECURSIVELY - would catch this and would also refuse a legitimate single-statement loop, so the - fix is not a deeper count; it is a claim about which statement raised. + expect.run_failed(result, "a helper call inside the block must not be collectable") + result.stderr.fnmatch_lines(["*calls _do_the_thing(), defined in this file*"]) + + +def test_a_compound_statement_hiding_the_setup_is_refused(pytester, expect): + """Shape 2 of the two: a COMPOUND STATEMENT holding setup and the statement. + + A `for` over two statements is ONE top-level statement, so the count rule was + satisfied while the loop raised on its first iteration -- the setup -- and the + statement the test names never ran. """ result = _inner(pytester, ''' import psycopg import pytest - def _run(stmt): - if stmt.startswith("SET"): - raise psycopg.errors.UndefinedObject("no such GUC") - raise psycopg.errors.UndefinedFunction("the statement under test") - def test_the_alter_is_refused(expect): with pytest.raises(psycopg.errors.UndefinedObject) as exc: for stmt in ("SET pgcolumnar.no_such_guc = 1", "ALTER TABLE t SET ACCESS METHOD no_such_am"): - _run(stmt) + raise psycopg.errors.UndefinedObject("no such object: " + stmt) expect.sqlstate(exc.value, "42704", "the statement under test") ''') - expect.outcomes(result, "a compound statement hiding the setup is NOT refused " - "-- the second residual this guard leaves open", + expect.run_failed(result, "a loop inside the block must not be collectable") + result.stderr.fnmatch_lines(["*holds a For, so which statement inside it raised*"]) + + +def test_a_helper_hidden_in_an_assignment_is_refused_too(pytester, expect): + """The rule looks ANYWHERE in the statement, not only at the whole of it. + + `x = _helper()` hides the setup exactly as well as a bare `_helper()` does, and a + rule that matched only `Expr` would leave the assignment spelling open. Without + this arm, narrowing the search to a bare call changes nothing and the narrowing + is invisible -- which is how the `for` spelling came to be the only one the + inventory named. + """ + result = _inner(pytester, ''' + import psycopg + import pytest + + def _setup_then_run(): + raise psycopg.errors.UndefinedObject("SET pgcolumnar.no_such_guc") + + def test_the_alter_is_refused(expect): + with pytest.raises(psycopg.errors.UndefinedObject) as exc: + outcome = _setup_then_run() + expect.sqlstate(exc.value, "42704", "the statement under test") + ''') + expect.run_failed(result, "a helper hidden in an assignment must not be collectable") + result.stderr.fnmatch_lines(["*calls _setup_then_run(), defined in this file*"]) + + +def test_every_compound_statement_is_refused_not_only_a_loop(pytester, expect): + """`if`, `while`, `with` and `try` nest exactly as `for` does. + + The inventory named the `for` spelling. A rule that caught only that one would + leave three spellings of the same shape, which is the difference between closing + a mode and closing an example of it. + """ + bodies = { + "If": 'if True:\n raise psycopg.errors.UndefinedObject("x")', + "While": 'while True:\n raise psycopg.errors.UndefinedObject("x")', + "With": 'with open("/dev/null"):\n raise psycopg.errors.UndefinedObject("x")', + "Try": 'try:\n raise psycopg.errors.UndefinedObject("x")\n finally:\n pass', + } + for kind, body in bodies.items(): + result = _inner(pytester, f''' + import psycopg + import pytest + + def test_the_alter_is_refused(expect): + with pytest.raises(psycopg.errors.UndefinedObject) as exc: + {body} + expect.sqlstate(exc.value, "42704", "the statement under test") + ''') + expect.run_failed(result, f"a {kind} inside the block must not be collectable") + result.stderr.fnmatch_lines([f"*holds a {kind}, so which statement inside it raised*"]) + + +def test_a_raises_block_calling_an_imported_function_is_accepted(pytester, expect): + """The false-positive budget, and it is the shape the corpus actually uses. + + Four of the five `pytest.raises` blocks in `test/pytest/` call + `build_and_install(...)`, imported from the module under test. Refusing a call + because it is a call would refuse every one of them, so the rule turns on where + the function is DEFINED rather than on the statement being a call. + """ + result = _inner(pytester, ''' + import psycopg + import pytest + from json import loads + + def test_the_thing_under_test_raises(expect): + with pytest.raises(ValueError) as exc: + loads("{not json") + expect.at_least(len(str(exc.value)), 1, "the imported call raised") + ''') + expect.outcomes(result, "a call to an IMPORTED function is the thing under test", + passed=1, failed=0) + + +def test_a_raises_block_calling_a_method_is_accepted(pytester, expect): + """The fifth block's shape: a method call on an object. + + A method cannot be matched against this file's function definitions, and the + object it belongs to is usually the thing under test. This is a stated residual + rather than an oversight: a method that performs setup and then the statement is + invisible to this rule, and no static rule can see inside it. + """ + result = _inner(pytester, ''' + import psycopg + import pytest + + class _Conn: + def execute(self, sql): + raise psycopg.errors.UndefinedObject("no such object: " + sql) + + def test_the_alter_is_refused(expect): + conn = _Conn() + with pytest.raises(psycopg.errors.UndefinedObject) as exc: + conn.execute("ALTER TABLE t SET ACCESS METHOD no_such_am") + expect.sqlstate(exc.value, "42704", "the ALTER was refused") + ''') + expect.outcomes(result, "a method call is accepted, and why is recorded", passed=1, failed=0) - # Zero offences, by the argument given in the arm above. # --------------------------------------------------------------------------- From 4c8f7c19f95fe63dcf0484387e3c5ce8b1f2c3d1 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Thu, 10 Sep 2026 21:54:03 +0000 Subject: [PATCH 2/3] test/pytest: the refused count is a function of the merge, not of either branch #935 closed `insert-wrote-no-rows` and moved the refused total 27 -> 28. This branch closes `raises-catches-setup` and moved it 27 -> 28 as well. The merge was CLEAN and therefore wrong: git took one 28 where the answer is 29, because the number is a property of both closures rather than of either. That is the exact failure TESTS.md's own header records about counts in this document -- "a claim whose correct value is a function of the MERGE rather than of either branch, so it collided on essentially every rebase" -- and the reason the counts are checked by arms instead of trusted. The arms named every number: refused today 28 -> 29 not refused 44 -> 43 demonstrated 48 -> 47 (section 3's sentence) TESTS.md prose 28 -> 29, and 44/43 -> 43/42 README.md 28 -> 29 I did not decide any of them; each one is what `test_the_prose_totals_match_the_counted_modes` and its siblings reported against the ids on disk. Gate on the merged tree: harness_selftest 588 checks, 588 passed + 0 failed + 0 unrunnable, PASSED driver-free job 10 files, 174 passed, psycopg absent from the venv full corpus 257 passed with a cluster on pg18a the raises scan zero offences over the corpus, all 9 compound kinds recognised Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- test/pytest/README.md | 2 +- test/pytest/TESTS.md | 4 ++-- test/pytest/VACUITY_MODES.md | 10 +++++----- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/test/pytest/README.md b/test/pytest/README.md index 27657f52..2e38cf91 100644 --- a/test/pytest/README.md +++ b/test/pytest/README.md @@ -5,7 +5,7 @@ This is the issue #432 pilot. It runs beside `test/*.sh`, and replaces nothing. - `TESTS.md` in this directory documents every test and every assertion helper. - `VACUITY_MODES.md` is the inventory of ways a pytest harness can report a false pass: 79 modes produced by the enumeration, 72 of them named in that - file, 73 demonstrated by a run, and 28 refused by this layer today. + file, 73 demonstrated by a run, and 29 refused by this layer today. VACUITY_MODES.md section 1a gives the counting rule and reconciles the run's totals against what is actually written down. - `design/ISSUE_432_PYTEST_HARNESS.md` holds the design and the measurements diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index c20f5e87..270d9743 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1378,8 +1378,8 @@ over tests nothing ran. ## 18. What this corpus does NOT yet refuse `VACUITY_MODES.md` is the inventory: 79 ways a pytest harness can report a pass while -asserting nothing, 73 of them demonstrated by an actual run. **This layer refuses 28 -of them.** The other 44, of which 43 were demonstrated, are listed there with the +asserting nothing, 73 of them demonstrated by an actual run. **This layer refuses 29 +of them.** The other 43, of which 42 were demonstrated, are listed there with the refusal design each would need and the order worth building them in. Read it before adding a test. One gap is most likely to affect a new test now. diff --git a/test/pytest/VACUITY_MODES.md b/test/pytest/VACUITY_MODES.md index de455ce3..6d42f1b7 100644 --- a/test/pytest/VACUITY_MODES.md +++ b/test/pytest/VACUITY_MODES.md @@ -47,8 +47,8 @@ recollection of the run: | | modes | | --- | ---: | -| named in section 2, refused today | 28 | -| named in section 3, not refused | 44 | +| named in section 2, refused today | 29 | +| named in section 3, not refused | 43 | | **named in this document** | **72** | | produced by the enumeration run | 79 | | **named nowhere here** | **7** | @@ -65,7 +65,7 @@ an id can be read, argued with and turned into a test, and a number cannot. ## 2. What the layer refuses today -28 of the 79, counted by section 1a's rule. Each is enforced by a mechanism, not a convention, and each has a red +29 of the 79, counted by section 1a's rule. Each is enforced by a mechanism, not a convention, and each has a red test in `test_layer.py` that fails without it. | mechanism | modes it closes | @@ -122,7 +122,7 @@ guard whose subject is false greens has no business emitting a false red. ## 3. What it does not refuse -55 modes by the run's count, **44 of them named below**, **48 demonstrated by a run**. 51 have a refusal already designed. +55 modes by the run's count, **43 of them named below**, **47 demonstrated by a run**. 51 have a refusal already designed. Grouped by what a reader needs to decide about them. ### 3.1 The run can lose tests and still exit 0 @@ -354,6 +354,6 @@ have tried to defeat them did not run. Every design states its own residual, and those residuals are the authors' own, unchallenged. So treat §2 as measured, §3 as measured, and §5 as a plan that has not yet met an -adversary. The layer is known to refuse 28 demonstrated modes -- the ids named in section 2, +adversary. The layer is known to refuse 29 demonstrated modes -- the ids named in section 2, not the run's larger total, for the reason section 1a gives. It is not known to be undefeatable on any of them. From 0d1d0279ecbc5fa12827080389053976e0a0ed10 Mon Sep 17 00:00:00 2001 From: OffgridwithJD Date: Thu, 10 Sep 2026 23:05:24 +0000 Subject: [PATCH 3/3] test/pytest: this NARROWS raises-catches-setup, and the count should not have moved I claimed a closure I can defeat three ways, and @jdatcmd's review named the residuals. Measuring them settles the classification against me: a `for` loop over two statements REFUSED the same two as a list comprehension allowed the same two as a tuple of calls allowed a helper defined in ANOTHER file allowed an honest one-statement helper in THIS file REFUSED (a false positive) A comprehension and a tuple are EXPRESSIONS, not compound statements, so a rule about statement kinds cannot see them; `local_defs` is built from one file, so moving the helper one file over defeats it. Neither is a contrivance -- both are ordinary Python. So by this document's own convention the mode stays in section 3, the section-2 row is withdrawn, and the refused count goes back to 28 with not-refused back to 44. The arms derive it: refused=28, not_refused=44, and `raises-catches-setup` is not in the refused set. Section 5's entry now says "Done, and it NARROWS rather than closes", beside the two other entries that say the same thing. THE ARM THAT EXISTS TO CATCH THIS OVERCLAIM DID NOT CATCH IT, and that is the part worth keeping. `test_the_mode_this_layer_only_narrows_is_still_listed_as_open` required the mode to be named in section 3 -- and section 3 keeps a back-reference for every mode that MOVES, "`X` is now closed", so the id is present in section 3 in BOTH states. My first version wrote the closure into section 3, added the row to section 2, moved the count to 29, and the arm passed. Its own docstring says what it was for: "A document that quietly moved the mode to section 2 would claim a closure this scan does not make." It now also requires the mode to be named OUTSIDE a closure back-reference, and to be absent from section 2. Proven by removal, each shape of the overclaim separately: control (the document as it stands) 1 passed entry rewritten as a back-reference 1 failed row added back to section 2 1 failed both, which is what my branch did 1 failed AND MY FIRST TIGHTENING WAS WRONG IN THE SAME FAMILY. I wrote `section3.split("is now closed")[0]`, which truncates at the FIRST back-reference in the section -- today `insert-wrote-no-rows`, which sits above this entry -- so it reported "only as a back-reference" about a document that names the mode correctly. A positional test over a section holding several back-references is a test about their order. It is per-line now. Gate: harness_selftest 588 checks, 588 passed + 0 failed + 0 unrunnable, PASSED driver-free job 10 files, 183 passed, psycopg absent from the venv full corpus 266 passed with a cluster on pg18a derived counts refused 28, not refused 44, named 72 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a --- test/pytest/README.md | 2 +- test/pytest/TESTS.md | 38 ++++++++++++++--- test/pytest/VACUITY_MODES.md | 65 ++++++++++++++++++----------- test/pytest/test_raises_sqlstate.py | 27 ++++++++++++ 4 files changed, 101 insertions(+), 31 deletions(-) diff --git a/test/pytest/README.md b/test/pytest/README.md index 2e38cf91..27657f52 100644 --- a/test/pytest/README.md +++ b/test/pytest/README.md @@ -5,7 +5,7 @@ This is the issue #432 pilot. It runs beside `test/*.sh`, and replaces nothing. - `TESTS.md` in this directory documents every test and every assertion helper. - `VACUITY_MODES.md` is the inventory of ways a pytest harness can report a false pass: 79 modes produced by the enumeration, 72 of them named in that - file, 73 demonstrated by a run, and 29 refused by this layer today. + file, 73 demonstrated by a run, and 28 refused by this layer today. VACUITY_MODES.md section 1a gives the counting rule and reconciles the run's totals against what is actually written down. - `design/ISSUE_432_PYTEST_HARNESS.md` holds the design and the measurements diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 91243e02..55dd7aeb 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -1418,8 +1418,8 @@ over tests nothing ran. ## 18. What this corpus does NOT yet refuse `VACUITY_MODES.md` is the inventory: 79 ways a pytest harness can report a pass while -asserting nothing, 73 of them demonstrated by an actual run. **This layer refuses 29 -of them.** The other 43, of which 42 were demonstrated, are listed there with the +asserting nothing, 73 of them demonstrated by an actual run. **This layer refuses 28 +of them.** The other 44, of which 43 were demonstrated, are listed there with the refusal design each would need and the order worth building them in. Read it before adding a test. One gap is most likely to affect a new test now. @@ -1596,7 +1596,7 @@ one condition faithfully, and require the copy to go blind. | `test_a_conftest_cannot_switch_the_broad_family_list_off` | the rule's own family list is not writable from the corpus it polices | | `test_a_bare_sqlstate_expression_does_not_pin_anything` | `exc.value.sqlstate` as a statement of its own asserts nothing, so mentioning the field is not pinning it | -### How `raises-catches-setup` closed +### How `raises-catches-setup` narrowed, and what is left The count rule refuses a block holding more than one top-level statement. Two shapes are **one** statement and still hide the setup inside the block, so the count saw @@ -1625,8 +1625,36 @@ and that is what makes the budget zero. Measured over the corpus: five `pytest.raises` blocks, four calling `build_and_install` (imported) and one calling a method, and the scan reports **no offence** on any of them. -**The residual is a method.** A method that performs setup and then the statement is -invisible to this rule, and no static rule can see inside it. +### What is still reachable, measured + +The mode stays in `VACUITY_MODES.md` section 3, and the refused count did not move, +because two ordinary spellings still reach it: + +| shape | verdict | +| --- | --- | +| a `for` loop over two statements | refused | +| the same two as a **list comprehension** | allowed | +| the same two as a **tuple of calls** | allowed | +| a helper defined in **another file** | allowed | +| an honest one-statement helper defined in **this** file | refused — a false positive | + +A comprehension and a tuple are **expressions**, not compound statements, so a rule +about statement kinds cannot see them. And `local_defs` is built from one file, so +moving the helper one file over defeats it. Neither is a contrivance; both are ordinary +Python. The last row is the rule's cost rather than a gap — an honest single-statement +local helper is refused, and the author must inline it. + +A method that performs setup and then the statement is invisible for the same reason, +and no static rule can see inside it. + +**The arm that should have caught the overclaim did not.** +`test_the_mode_this_layer_only_narrows_is_still_listed_as_open` required the mode to be +named in section 3 — and section 3 keeps a back-reference for every mode that *moves* +("`X` is now closed"), so the id is present in section 3 whichever state the document +claims. A first version of this work wrote the closure into section 3, added the row to +section 2, moved the count to 29, and that arm passed. It now also requires the mode to +be named outside a closure back-reference and to be absent from section 2; all three +shapes of the overclaim redden it. Residuals named by @jdatcmd on review. | `test_a_sqlstate_assigned_and_never_read_does_not_pin_anything` | the same hole one step on: bound to a name nothing uses | | `test_one_hop_through_a_local_name_is_an_honest_pin` | the cost side — `code = exc.value.sqlstate` then `expect.text(code, ...)` stays collectable | | `test_the_keyword_form_is_checked_by_both_rules` | `pytest.raises(expected_exception=...)` is not an exemption from either rule | diff --git a/test/pytest/VACUITY_MODES.md b/test/pytest/VACUITY_MODES.md index a2bdf335..eff7ef88 100644 --- a/test/pytest/VACUITY_MODES.md +++ b/test/pytest/VACUITY_MODES.md @@ -47,8 +47,8 @@ recollection of the run: | | modes | | --- | ---: | -| named in section 2, refused today | 29 | -| named in section 3, not refused | 43 | +| named in section 2, refused today | 28 | +| named in section 3, not refused | 44 | | **named in this document** | **72** | | produced by the enumeration run | 79 | | **named nowhere here** | **7** | @@ -65,7 +65,7 @@ an id can be read, argued with and turned into a test, and a number cannot. ## 2. What the layer refuses today -29 of the 79, counted by section 1a's rule. Each is enforced by a mechanism, not a convention, and each has a red +28 of the 79, counted by section 1a's rule. Each is enforced by a mechanism, not a convention, and each has a red test in `test_layer.py` that fails without it. | mechanism | modes it closes | @@ -87,7 +87,6 @@ test in `test_layer.py` that fails without it. | an empty parameter set fails the run, with its own message | `empty-parametrize-is-a-silent-skip` | | a skip during fixture setup fails the run | `session-fixture-skip-greens-the-whole-suite` | | a broad `pytest.raises` must pin a SQLSTATE, found by AST | `raises-too-broad` | -| a `pytest.raises` block may not hold a compound statement or call a function this file defines | `raises-catches-setup` | | every comparison refuses a value carrying the `QUERY_ERROR` prefix, on either side | `error-swallowed-to-empty` | | a write whose command tag reports 0 rows fails the test unless the zero is named | `insert-wrote-no-rows` | @@ -122,7 +121,7 @@ guard whose subject is false greens has no business emitting a false red. ## 3. What it does not refuse -55 modes by the run's count, **43 of them named below**, **47 demonstrated by a run**. 51 have a refusal already designed. +55 modes by the run's count, **44 of them named below**, **48 demonstrated by a run**. 51 have a refusal already designed. Grouped by what a reader needs to decide about them. ### 3.1 The run can lose tests and still exit 0 @@ -187,9 +186,9 @@ Still open in this family: `DatabaseError`, `Exception` or `BaseException` does not collect unless the block binds the exception and the body pins its SQLSTATE. See section 2. -**`raises-catches-setup` is now closed.** The statement COUNT rule refused a block - holding more than one top-level statement, and two shapes are ONE statement that still - performs the setup inside the block: +- `raises-catches-setup` — **narrowed again, and still not closed.** The statement COUNT + rule refused a block holding more than one top-level statement, and two shapes are ONE + statement that still performs the setup inside the block: with pytest.raises(psycopg.errors.UndefinedObject) as exc: _setup_then_run(conn) # a HELPER CALL: one statement @@ -199,21 +198,36 @@ binds the exception and the body pins its SQLSTATE. See section 2. conn.execute(stmt) # statement holding two Both were measured against the shipped scan reporting `1 passed`, exit 0, **zero - offences**, with the setup raising and the statement under test never running. + offences**, with the setup raising and the statement under test never running. Both are + refused now, by two rules: no compound statement (all nine kinds Python has, looked up + by name so a missing `TryStar` or `Match` cannot silently narrow the rule), and no call + to a function DEFINED IN THE SAME FILE, anywhere in the statement. + + WHAT REMAINS, measured rather than reasoned, which is why this entry stays in section 3 + and the refused count did not move: + + a `for` loop over two statements REFUSED + the same two statements as a list comprehension allowed + the same two as a tuple of calls allowed + a helper defined in ANOTHER file allowed + an honest one-statement helper defined in THIS file REFUSED (a false positive) + + A comprehension and a tuple are EXPRESSIONS rather than compound statements, so a rule + about statement kinds cannot see them; and `local_defs` is built from one file, so + moving the helper one file over defeats it. Both are ordinary Python, not contrivances. + The last row is the rule's cost rather than a gap: an honest single-statement local + helper is refused, and the author must inline it. THE FIX IS NOT A DEEPER COUNT, for the reason this entry always gave: counting - recursively would also refuse a legitimate single-statement loop. It is a claim about - WHICH statement raised, in two rules — no compound statement (all nine kinds, looked - up by name so a missing `TryStar` or `Match` cannot silently narrow the rule), and no - call to a function DEFINED IN THE SAME FILE, anywhere in the statement. A call to an - IMPORTED function or to a METHOD is the thing under test and stays allowed, which is - what makes the budget zero: measured over the corpus, five `pytest.raises` blocks — - four calling an imported function, one calling a method — and no offence on any. - - THE RESIDUAL IS A METHOD. A method that performs setup and then the statement is - invisible to this rule, and no static rule can see inside it. The two arms that used - to assert these shapes were NOT refused now assert that they are, so the closure is a - measurement rather than a sentence. See TESTS.md section 20. + recursively would also refuse a legitimate single-statement loop. What would close it + is a claim about which statement raised that does not depend on the SHAPE of the + statement -- a helper that runs exactly one statement and owns the assertion. Measured, + the corpus has no SQL-raising `pytest.raises` block at all, so that helper would have + no call sites today and would be an instrument with nothing exercising it. + + The two arms that used to assert these shapes were NOT refused now assert that they + are, so the narrowing is a measurement rather than a sentence. Residuals named by + @jdatcmd on review. See TESTS.md section 20. - `same-broken-helper-both-sides`, `truthy-error-string`, `assert-not-unset-error`, `zero-on-both-arms`, `tuple-assert-always-true`, `approx-of-nothing` - `same-broken-helper-both-sides`, `truthy-error-string`, `assert-not-unset-error`, @@ -362,9 +376,10 @@ Each entry names the red test to write first. It closes `raises-too-broad` and narrows `raises-catches-setup`, which stays open in 3.4 with the two shapes it cannot see named there. What would close the sibling is the next entry: -5. `test_layer_requires_the_raiser_to_be_the_statement_under_test` — closes - `raises-catches-setup`. It needs a claim about WHICH statement raised, not a - deeper statement count; 3.4 says why a recursive count is the wrong fix. +5. ~~`test_layer_requires_the_raiser_to_be_the_statement_under_test` — closes + `raises-catches-setup`.~~ **Done, and it NARROWS rather than closes** — 3.4 lists what + remains, measured: a comprehension or a tuple instead of a `for`, and a helper defined + in another file. Both are ordinary Python. The refused count therefore did not move. The three that turned a whole run green rather than one test are done. What remains is per-assertion work, so the ordering matters less: take the sentinel first, since @@ -377,6 +392,6 @@ have tried to defeat them did not run. Every design states its own residual, and those residuals are the authors' own, unchallenged. So treat §2 as measured, §3 as measured, and §5 as a plan that has not yet met an -adversary. The layer is known to refuse 29 demonstrated modes -- the ids named in section 2, +adversary. The layer is known to refuse 28 demonstrated modes -- the ids named in section 2, not the run's larger total, for the reason section 1a gives. It is not known to be undefeatable on any of them. diff --git a/test/pytest/test_raises_sqlstate.py b/test/pytest/test_raises_sqlstate.py index 55d98a25..102b7510 100644 --- a/test/pytest/test_raises_sqlstate.py +++ b/test/pytest/test_raises_sqlstate.py @@ -826,6 +826,33 @@ def test_the_mode_this_layer_only_narrows_is_still_listed_as_open(expect): "the document still names the mode this layer only narrows") expect.text("open" if "raises-catches-setup" in section3 else "moved", "open", "and names it in the section for what is NOT refused") + # APPEARING IN SECTION 3 IS NOT THE SAME AS BEING OPEN, and this arm passed while + # the document said the opposite of what it exists to assert. Section 3 keeps a + # back-reference for every mode that MOVES -- "`X` is now closed" -- so the id is + # present in section 3 whichever state it is in. A branch that wrote + # "`raises-catches-setup` is now closed." into section 3 and added the row to + # section 2 satisfied the line above and moved the refused count to 29. + # + # The arm already split on "is now closed" for the OTHER mode, two lines below, and + # did not apply the same care to its own subject. Two further conditions, because + # either alone can be satisfied by the wrong document: the id must appear in section + # 3 OUTSIDE any closure back-reference, and it must not appear in section 2 at all. + # Found by taking the closure claim seriously enough to measure the residuals, which + # showed the mode is reachable by a comprehension and by a helper one file over. + # PER LINE, NOT BY SPLITTING, and my first version of this condition got that wrong. + # `split("is now closed")` truncates at the FIRST back-reference in section 3, which + # today belongs to `insert-wrote-no-rows` and sits ABOVE this mode's entry -- so the + # check reported "only as a back-reference" about a document that names the mode + # correctly. A positional test on a section that holds several back-references is a + # test about their order. + named_open = [ln for ln in section3.splitlines() + if "raises-catches-setup" in ln and "is now closed" not in ln] + expect.at_least(len(named_open), 1, + "and names it OUTSIDE a closure back-reference, which reads the same way") + section2 = doc[re.search(r"^## 2\. ", doc, re.M).end():start.start()] + expect.text("absent" if "raises-catches-setup" not in section2 else "claimed refused", + "absent", + "and does not also claim it refused in the section for what IS refused") expect.text("absent" if "raises-too-broad" not in section3.split("is now closed")[0] else "present", "absent", "premise: and the mode this layer DOES close is not loose in section 3")