test/pytest: a guard in a teardown cannot fail the test it guards (#432)

[OffgridwithJD]

The same AssertionError, raised from two places:

raised from	the run reports
a pytest_runtest_call wrapper	1 failed
a fixture teardown	1 passed, 1 error

By the time a teardown runs, pytest has already recorded the call phase as passed. A refusal there arrives as a separate ERROR on the same node-id and the test's own outcome stays passed. Anything counting passes — --pgc-expect-tests, a CI summary, a human reading "N passed" — sees a pass.

Nothing was broken

This layer's vacuity guard is already in the call-phase wrapper, which is why a test that concludes nothing is failed rather than passed-with-an-error. What was missing is that nothing said so.

Moving it into the expect fixture's teardown is a plausible-looking refactor — the fixture is right there, it already has a teardown, and a guard reads as cleanup — and it would have turned every vacuous test in the corpus from failed into passed with an error beside it. The corpus would still have exited non-zero, so it is not silent; but the per-test verdict, which is what --pgc-expect-tests and every summary read, would have said pass.

Two arms, and the second is the control

Without the control the first arm passes whatever phase the guard is in: "a vacuous test fails" is equally true of a correctly placed guard and of no guard at all standing next to an unrelated failure. So the second arm asserts that the teardown shape really does report a pass, which states the mode as a measurement rather than as a warning.

Proved by moving it: with the call-phase raise disabled and the same refusal placed after the expect fixture's yield, the first arm reddens. Restored, 11 passed.

Narrowed, not closed

VACUITY_MODES.md 3.7 now says which half is which.

• Closed: this layer's own guard placement, pinned with a control.
• Still open: the general shape. A guard anyone adds later in a teardown still cannot fail its test, and nothing refuses that.

I would rather leave that written down than claim the family. The inventory's counts are therefore unchanged — the mode stays in section 3, which is why none of the totals move in this diff.

Evidence

pytest corpus       164 passed
harness_selftest    538 checks, 538 passed + 0 failed + 0 unrunnable, rc 0
the new arms        2, and the first reddens when the guard moves to the teardown

Relationship to my other open PRs

Independent, off main at f0f1f40, like #926, #927 and #930 — none blocks another. This one touches test_runshape.py, TESTS.md section 10, VACUITY_MODES.md 3.7 and CHANGELOG.md.

It will conflict with #927 and #930 in VACUITY_MODES.md and CHANGELOG.md — the append-at-the-top kind, not a code conflict. Those two also move a mode from section 3 to section 2 and so edit the count rows; this one does not, because 3.7 is narrowed rather than closed. Whichever lands last, I will resolve and re-run both harnesses rather than adjust numbers by hand.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[jdatcmd]

Approving at 3d87b4c0, 12/12 SUCCESS. I verified the claim, the removal proof, and the control separately, because the control is what makes the first arm worth anything and you say so yourself.

The claim reproduces, independently of your fixtures. Two files of my own, the same AssertionError from two places:

call phase  ->  FAILED test_call.py::test_x       ->  1 failed
teardown    ->  ERROR  test_teardown.py::test_x   ->  1 passed, 1 error

And the thing that matters, what a pass-counter reads off the teardown run:

1 passed
1 error

So --pgc-expect-tests and every summary would see a pass. The table in your body is exact.

The removal proof holds. I moved the guard the way the refactor would — deleted the VacuityError from the pytest_runtest_call wrapper and raised the identical error after the expect fixture's yield:

FAILED test_runshape.py::test_the_vacuity_guard_fails_the_test_rather_than_erroring_beside_it
1 failed, 10 passed

Exactly the arm that claims the guard fails the test, and nothing else. Baseline 11 passed; pgc_vacuity.py restored byte-identical.

The control is a real assertion, not a shape. expect.outcomes(result, ..., passed=1, failed=0, errors=1) reddens on each field independently — I mutated each of the three in turn and got 1 failed every time. So it pins the outcome triple rather than merely observing that something happened. It also cannot be reddened by any change to this layer, which is correct for what it is: it pins an assumption about pytest, and if pytest ever stopped reporting a teardown failure this way, the placement argument would need revisiting and this is the arm that would say so.

It still composes. #927 landed on main while I was reviewing this, so I merged your branch onto the new main locally: clean, 4 files changed, 121 insertions(+), 2 deletions(-), no conflict.

On the scoping, which is the part I would have argued about if you had not. "Nothing was broken — what was missing is that nothing said so" is the honest framing, and narrowing 3.7 rather than claiming the family is the right call. A guard anyone adds later in a teardown still cannot fail its test, the totals correctly do not move, and the mode stays in section 3. That is a measurement with its limits stated, which is worth more than a closed row that is not really closed.

Nothing blocking, nothing I could break.