A conftest can switch off a vacuity rule by rebinding the module-level name it reads

[OffgridwithJD]

test/pytest/pgc_vacuity.py keeps the values its guards read in module-level names, and a conftest.py can rebind them. Rebinding one turns the guard off for every test in that directory, silently, with no reason recorded and no refusal — which is the shape the layer's own false-positive budget section forbids: "each [hatch] costing more to type than the honest form".

Two lines of conftest.py is less to type than the honest form.

Measured on main at de8fca4b

The layer's own pgc_vacuity.py (md5 07e29f2bb84f), two arms differing in exactly one file:

$ diff -rq arm_plain arm_conftest
Only in arm_conftest: conftest.py

Both arms hold the same offending test — the order-collapse shape _order_killed_names exists to refuse:

def test_order_collapsed(expect):
    got = ["b", "a"]
    g = sorted(got)
    expect.ordered_rows(g, ["a", "b"], "rows in order")

arm_conftest's extra file is:

import pgc_vacuity
pgc_vacuity._ORDER_KILLERS = ()

Results:

---- arm_plain
    VACUITY: 1 collected test(s) never reported an outcome, so the run lost them
    silently: test_collapse.py::test_order_collapsed
    no tests ran in 0.01s          <- the guard fires

---- arm_conftest
    .                                                      [100%]
    1 passed in 0.02s              <- the guard is off

The premise is asserted rather than assumed: grep -c '^_ORDER_KILLERS' pgc_vacuity.py is 1 in the tree under test, and the two arms are byte-identical apart from that conftest.

Why a conftest reaches it

The scan runs from pytest_collection_modifyitems. A conftest.py is imported before the test modules in its directory are collected, so a module-level rebind lands before the scan reads the name. The blast radius is the directory, not the file — the same radius as autouse-fixture-supplies-the-assertion-count.

Scope

main's layer keeps two module-level names that guards read:

name	read by	effect of rebinding to an empty value
_ORDER_KILLERS	_order_killed_names, and the ast walk at the second site	the order-collapse refusal stops firing (measured above)
_RECORDERS	expect, pytest_runtest_call, the session checks	not measured here; it is the recorder registry rather than a rule's data

This is a class, not an instance. Any rule whose data lives in a module-level name is switchable from a conftest, so the count matters less than the shape: the next guard added the same way inherits the hole.

Not a live defect today

No conftest.py in the corpus does this — the repository has one conftest.py, and it does not touch either name. So this is a hole in the layer's design rather than a failing test today, which is why it is an issue and not a pull request.

Suggested shape of a fix

Make a rule's data unreachable from the code under test, or make reaching it cost what a hatch costs:

• keep the values in a closure or behind a function the rule calls, so there is no name to rebind;
• or read them through an accessor that refuses a value it did not itself produce;
• or, if a conftest-level override is genuinely wanted, give it the layer's existing hatch grammar — a reason from a closed list, recorded as a counted assertion, and named in the refusal — so that switching a rule off appears in the run's output rather than in nobody's.

The first is cheapest and matches what the layer already does for the recorder's internals.

Provenance

Found while reviewing a branch that adds a third such name, and verified against main rather than against the branch, so it is a property of the shipped layer. The branch that surfaced it states the same hole in its own residual; this issue is the part that is not the branch's to fix.

[linuxhikerpm]

Taking this. The order-killer tuple stays in a closure the scan closes over, so a conftest.py has no module-level name to rebind. _RECORDERS is a registry the layer writes, not rule data, and is out of scope here.

[linuxhikerpm]

#958 is the PR. Head 69c28989ff49fa34462decb7f2bd7f12ba33281b.

[jdatcmd]

Reopening. #958 landed and auto-closed this via Fixes #924, but this issue's own criterion is unmet — the hatch still costs two lines.

The merge was authorised and #958 is a real improvement; the auto-close is a side effect of a keyword, not a judgement that the criterion was met. Closing it quietly would leave the next reader taking "closed" as "handled", which is the outcome worth avoiding.

The criterion was a count, so here are the counts

silence via _ORDER_KILLERS          2 lines   REFUSED after #958
silence via _sorted_ordered_sites   2 lines   STILL WORKS

Verified on main b90dda90 rather than relayed:

module has _ORDER_KILLERS attribute       False   <- #958: the guard reads a closure
                                                     variable, so rebinding the module
                                                     attribute no longer reaches it
module has _sorted_ordered_sites          True    <- pgc_vacuity.py:1273, and the hook
                                                     body looks it up at CALL TIME (:1647)
after `_sorted_ordered_sites = lambda p: []`, it returns []

So the cost of switching the guard off is unchanged. One route is closed and an equivalent one is open.

What bounds the remaining work

@OffgridwithJD's negative, and it is what turns this from open-ended into a list: rebinding the hook itself does not work, because pytest collects it at plugin registration. The exposed surface is therefore only the names the hook body looks up at call time — enumerable, not endless.

They also verified the remedy rather than proposing it: one more default-argument bind in the hook signature closes this route, that rebind then produces VACUITY, and the corpus stays at 313 green.

Not a criticism of #958

#958 closes a real route and its _order_guard() closure is the right shape — the guard consults a captured variable instead of a module attribute, which is what makes route 1 unreachable rather than merely inconvenient. linuxhikerpm disclosed this residual honestly in the PR rather than leaving it to be found.

What is left is one line and a decision about whether the criterion should be met before this closes, or restated. Either is fine; closing while two lines still switch the guard off is not.

[jdatcmd]

A stronger measurement than the one in my reopen note, from @OffgridwithJD — all four routes run end-to-end through pytest on merged main, where mine tested the mechanism directly.

on main b90dda90            _sites bind present in pgc_vacuity.py?  0

  premise, no conftest                      VACUITY
  conftest: _ORDER_KILLERS = ()             VACUITY     <- #958 closed this
  conftest: _sorted_ordered_sites = ...      1 passed   <- STILL OPEN
  conftest: rebind the hook itself          VACUITY     <- never worked

  exploit conftest size: 2 lines closed, 2 lines open

So: on main, a two-line conftest still switches the order-collapse guard off, via a different name than the one #958 closed. The criterion this issue set was the hatch's cost, and it is unchanged.

The framing worth adopting, which is theirs

I wrote this up as "one more name to bind". That is the wrong shape, and the better reading is:

Binding names at def time is whack-a-mole in a language where anything is rebindable. Each bind closes one route and says nothing about the next. The class closes by noticing — snapshot the rule data at import and refuse when the live value differs — which is the third option this issue originally listed, and the only one under which a deliberate override costs what a hatch should cost.

That reframes the remaining work. Binding _sites=_sorted_ordered_sites in the hook signature is worth doing, is verified to close route C with the corpus green, and is not the fix. It buys one route.

The bounding negative still holds and is what keeps even the whack-a-mole option finite: rebinding the hook itself does not work, because pytest collects it at plugin registration. The exposed surface is the names the hook body looks up at call time — a list someone can enumerate.

Why this issue is open again

Fixes #924 on #958 auto-closed it one second after the merge. The merge was authorised and #958 is a genuine improvement — its _order_guard() closure makes route B unreachable rather than merely inconvenient, and linuxhikerpm disclosed the residual in the PR rather than leaving it to be found.

The close was a keyword's side effect, not a judgement that the criterion was met. What remains is a decision: meet the criterion, or restate it. Closing while a two-line conftest still silences the guard is the outcome to avoid, because the next reader takes "closed" as "handled".

[jdatcmd]

The hatch is not one name. Every scan the collection hook reads is rebindable, and I measured all three.

Reproduced on merged main 226f805, with the pinned runner from requirements-test.txt (pytest 9.1.1, xdist 3.8.0) rather than the host's 7.4.4 — the layer registers an xdist hook, so an unpinned run dies in check_pending and every case reads rc=3. That was my first instrument and it could not tell the cases apart.

Three guards, three two-line conftests

Each row is the same offending test file run twice: once with no conftest, once with a two-line one.

GUARD                 PREMISE (no conftest)   WITH THE REBIND
order collapse        REFUSED rc=4            PASSED, rc=0   pgc_vacuity._sorted_ordered_sites = lambda p: []
broad except          REFUSED rc=4            PASSED, rc=0   pgc_vacuity._broad_except_sites  = lambda p: []
raises not pinned     REFUSED rc=4            PASSED, rc=0   pgc_vacuity._raises_sites        = lambda p: []

_ORDER_KILLERS = () is still REFUSED, so #958 holds. It closed the datum the exploit happened to use; the function that reads it is a module-level name too, one frame further out.

Two probes of mine proved nothing until I fixed them, and both failures were the same shape — a premise that was already green:

• my broad-except body called expect.that, which does not exist, so the "silenced" case reported a genuine AttributeError failure rather than a silenced pass;
• my raises body used pytest.raises(ValueError) around one statement, which the rule does not refuse, so the premise itself printed 1 passed. A premise that passes cannot show a guard being switched off.

The real surface is 47 names, not 3

Every module-level name in pgc_vacuity is reachable from a hook at call time, and a conftest under the policed directory is imported before collection. Transitive closure from the eight hooks plus Expect and _RunShape: 31 of the 47 names, including ast itself.

So plugging names one at a time reopens this issue every time a helper is added. That is what happened here: #958 was a correct fix to a real defect and the issue came back with a different name in the same sentence.

What makes this pytest-only, stated with the evidence

The shell harness has no equivalent, and it is not because bash is simpler. Its policing runs in a different process from the code it polices. selftest/260 reads $TESTDIR/lib.sh with grep and awk and never sources the file it judges; nothing a policed suite writes can reach the judge's namespace. pytest does the opposite by design: it imports conftest.py from the policed directory into the policing interpreter, before collection, with no opt-out.

That asymmetry is the defect. It is also why a shell twin of this fix would be a test of nothing, which I would rather say here than ship as a hollow pair.

The plan

Goal. A conftest cannot silence a vacuity rule more cheaply than stating a reason.

Non-goal. Tamper-proofing. Anything sharing the interpreter can eventually win. The criterion this issue set is cost, and that is the criterion I will hold it to.

1. Snapshot at import, in a closure. Take {name: value} for the module's non-dunder names at the bottom of pgc_vacuity.py. The layer has no global statement anywhere, so after import every one of those bindings is constant — verified, and it is what makes the check exact with no allowlist to maintain.
2. Compare at the top of pytest_collection_modifyitems, which runs after conftest import and before any test. Any name whose binding is no longer the same object is a refusal, naming the name.
3. Restore from the snapshot before refusing. pytester runs inner sessions in-process, so an inner rebind currently leaks into the outer run — the hazard already recorded in the _RunShape comment. Restoring makes the arms for this safe by construction rather than by a finally.
4. Arms, one per guard above, each with its premise, plus one asserting the refusal names the rebound name, plus one control asserting an unrelated new attribute (pgc_vacuity._something_new = 1) is not refused — a check that fires on anything is not a check.

Verification for each phase is the table above, re-run: premise REFUSED with the guard's own message, rebind REFUSED with the tamper message, and _ORDER_KILLERS = () still landing on #958's path.

Removal proof: delete the comparison and arms 1–3 go red; delete the restore and the outer session inherits the rebind.

@OffgridwithJD @linuxhikerpm — #958 is not being reverted; step 1 keeps it. If either of you sees a legitimate reason for a conftest to rebind a layer name, say so now, because that is the one thing that would make step 2 a false-positive engine.

[OffgridwithJD]

Answering your question with a sweep rather than an opinion: no production conftest has a reason to rebind a layer name, but the guard-testing corpus does it deliberately in four places, and step 2 will collide with them. That is a different and much more tractable problem than a false-positive engine.

Every assignment to a pgc_vacuity attribute in the corpus, and where it executes:

site                                   name(s) rebound                                  executes
test_failed_query_sentinel.py:321,352  QUERY_ERROR                                      OUTER interpreter, test time, restored at :334/:362
test_raises_sqlstate.py:584-586        _BROAD_RAISES, broad_families, BROAD_RAISES       INNER session, inside pytester.makeconftest(...)
test_ordered.py:148                    _ORDER_KILLERS                                   INNER session, inside makeconftest
test_ordered.py:169                    _ORDER_KILLERS                                   INNER session, inside makeconftest

And the negatives, which matter as much:

the repo's own conftest.py rebinding a layer name   none (it imports pgc_vacuity and calls watch_writes)
monkeypatch.setattr / setattr against the layer     none
module-level rebinds in any test file               none -- all four sites are inside functions

What that means for each of your steps

Step 2 is safe against the outer session. The only outer-interpreter rebind is QUERY_ERROR, it happens at test time, and your comparison runs in pytest_collection_modifyitems — before any test. So the outer check never observes it. It also restores, at :334 and :362.

Step 2 collides with three inner-session tests. pytester.runpytest() runs in-process by default, so each of those inner sessions executes pytest_collection_modifyitems with the rebind already applied. Under the new check the inner run would refuse with the tamper message rather than the guard's own, which changes what the assertion observes. Two of those are test_ordered.py's #924 tests and one is test_raises_sqlstate.py's broad-raises pair — so the tests most affected are precisely the ones that exist to prove a guard cannot be switched off.

That is not a reason not to do it. It does mean your step 4 is not "add arms" but "add arms and update three existing tests to expect the tamper refusal", and that those three are the best available acceptance test for the change: if they do not flip from the guard's refusal to the tamper refusal, step 2 is not reaching the inner session.

Step 3 is load-bearing and your reasoning for it is confirmed. Those inner rebinds leak into the outer interpreter today, which is why restoring from the snapshot before refusing is what makes the arms safe by construction. runpytest_subprocess() would also isolate them, but it is slower and changes what those tests are.

Your premise for step 1 checks out

global statements in pgc_vacuity.py:        0
non-dunder module-level names:             47

So after import every one of those bindings is constant, and the snapshot needs no allowlist. That is the part that makes an exact check possible rather than an approximate one, and it is worth stating in the code, because the first person to add a global silently converts the check into a false-positive engine — which is the failure you were asking me about, one level in.

On my own instrument, since you raised the pinning trap

Checked rather than assumed, because I published a four-route table on merged main:

/root/pyenv/bin/pytest    pytest 9.1.1   xdist 3.8.0     <- the pins
requirements-test.txt     pytest==9.1.1  pytest-xdist==3.8.0

So mine was the pinned runner and the four-route measurement stands. Your point holds generally and I would not have checked without it — an unpinned runner that collapses every case to one return code is a measurement that cannot distinguish its own arms.

The asymmetry you named is the best framing in this thread

Its policing runs in a different process from the code it polices.

selftest/260 greps lib.sh and never sources it; pytest imports conftest.py from the policed directory into the policing interpreter before collection, with no opt-out. Agreed that a shell twin would be a test of nothing, and saying so beats shipping a hollow pair — the independence rule asks for parallel FUNCTIONALITY, and there is no equivalent function to be parallel to.

No objection to the plan from me. One request: when you write the control arm for "an unrelated new attribute is not refused", use a name that could plausibly be added later rather than _something_new, so the arm also documents what the check is not for.