test/pytest: a guard in a teardown cannot fail the test it guards (#432)

CHANGELOG.md

@@ -160,6 +160,30 @@ true until the next version shipped.
 
 ### Fixed
 
+- The vacuity guard's PLACEMENT is now a checked property, because a guard in a
+  teardown cannot fail the test it guards (#432).
+
+  Measured, the same `AssertionError` raised from two places: from a
+  `pytest_runtest_call` wrapper the run reports `1 failed`; from a fixture teardown it
+  reports `1 passed, 1 error`. pytest has already recorded the call phase as passed, so
+  a teardown refusal arrives as a separate error on the same node-id and the test's own
+  outcome stays `passed`. Anything counting passes -- `--pgc-expect-tests`, a CI
+  summary, a human reading "N passed" -- sees a pass.
+
+  This layer's guard was already in the call-phase wrapper, so nothing was broken. What
+  was missing is that nothing said so: moving it into the `expect` fixture's teardown
+  was a plausible-looking refactor that would have turned every vacuous test from
+  `failed` into `passed` with an error beside it. Two arms in `test_runshape.py` pin the
+  placement, and the second is the control -- without it the first passes whatever phase
+  the guard is in, because "a vacuous test fails" is equally true of a correct guard and
+  of no guard at all next to an unrelated failure. Proved by moving the guard into the
+  teardown: the first arm reddens.
+
+  `guard-as-teardown-fixture-still-reports-passed` is NARROWED rather than closed, and
+  `VACUITY_MODES.md` 3.7 says which half. The half closed is this layer's own guard
+  placement. The half still open is the general shape: a guard anyone adds later in a
+  teardown still cannot fail its test, and nothing refuses that.
+
 - `ALTER TABLE ... RENAME COLUMN` now carries the new name into
   `pgcolumnar.projection_declaration`, for the named relation and for every
   inheritance descendant, including a `PARTITION OF` child (#888).

test/pytest/TESTS.md

@@ -935,6 +935,36 @@ The empty-parametrize refusal carries its own message rather than folding into t
 bare-skip refusal. When a corpus glob matches nothing, the cause the reader needs to
 see is the corpus, not the marker.
 
+### Where a guard runs decides what the run reports
+
+A guard implemented as a **fixture teardown** cannot fail the test it guards. pytest has
+already recorded the call phase as passed, so the refusal arrives as a separate `ERROR`
+on the same node-id and the test's own outcome stays `passed`. Anything counting
+passes — `--pgc-expect-tests`, a CI summary, a human reading "N passed" — sees a pass.
+
+Measured, the same `AssertionError` raised from each place:
+
+| raised from | the run reports |
+| --- | --- |
+| a `pytest_runtest_call` wrapper | `1 failed` |
+| a fixture teardown | `1 passed, 1 error` |
+
+This layer's vacuity guard is in a `pytest_runtest_call` wrapper, which is why a test
+that concludes nothing is *failed* rather than passed-with-an-error. Two arms keep that
+from being an accident, and the second is the control: without it the first passes
+whatever phase the guard is in, because "a vacuous test fails" is equally true of a
+correctly-placed guard and of no guard at all beside an unrelated failure.
+
+| test | what it asserts | how it fails |
+| --- | --- | --- |
+| `test_the_vacuity_guard_fails_the_test_rather_than_erroring_beside_it` | a test concluding nothing is `failed`, with no error | moving the guard into the `expect` fixture's teardown reddens it |
+| `test_a_guard_in_a_teardown_would_report_a_pass_which_is_why_it_is_not_there` | a teardown refusal leaves the test reported `passed` with an error beside it | the mode stated as a measurement rather than a warning |
+
+This closes the part of `guard-as-teardown-fixture-still-reports-passed`
+(`VACUITY_MODES.md` 3.7) that is about **this layer's own guard placement**. It does not
+close the family: a guard anyone adds later in a teardown is still a guard that cannot
+fail its test, and nothing refuses that shape in general.
+
 ## 11. test_zonemap_boundaries.py: exact boundaries
 
 ### `test_exact_zonemap_boundaries`

test/pytest/VACUITY_MODES.md

@@ -215,8 +215,14 @@ the sentinels that close the other half, empty compared with empty.
 
 ### 3.7 The guard itself goes quiet
 
-- `guard-as-teardown-fixture-still-reports-passed` — a guard implemented as a
-  teardown fixture leaves the test reporting PASSED.
+- `guard-as-teardown-fixture-still-reports-passed` — **narrowed, not closed.** Measured:
+  the same `AssertionError` raised from a `pytest_runtest_call` wrapper gives `1 failed`,
+  and raised from a fixture teardown gives `1 passed, 1 error` — the test's own outcome
+  stays `passed`, so anything counting passes sees a pass. This layer's vacuity guard is
+  in the call-phase wrapper, and two arms in `test_runshape.py` now pin that placement
+  with a control, so a refactor into a teardown reddens rather than going quiet. What is
+  NOT closed is the general shape: a guard anyone adds later in a teardown still cannot
+  fail its test, and nothing refuses that. See TESTS.md section 10.
 - `session-accounting-guard`, `session-exit-rewrite-masks-a-real-failure`,
   `description-guard-reopens-psycopg-raise`,
   `mitigations-measured-and-the-one-that-does-not-work`

test/pytest/test_runshape.py

@@ -193,3 +193,62 @@ def test_keep_f(expect): expect.num(1, 1, "f")
                                 "--max-worker-restart=0")
     expect.run_failed(result, "a lost test is still caught when others were deselected")
     result.stdout.fnmatch_lines(["*never reported*"])
+
+
+# ---- WHERE a guard runs decides what the run reports -------------------------
+#
+# `guard-as-teardown-fixture-still-reports-passed` (VACUITY_MODES.md 3.7). A guard
+# implemented as a fixture teardown cannot fail the test it guards: pytest has
+# already recorded the call phase as passed, so the refusal arrives as a separate
+# ERROR on the same node-id and the test's own outcome stays `passed`. Anything
+# counting passes -- `--pgc-expect-tests`, a CI summary, a human reading "N passed"
+# -- sees a pass.
+#
+# This layer's own vacuity guard is in a `pytest_runtest_call` wrapper, which is the
+# right place, and these two arms are why that stops being an accident. Measured,
+# the same AssertionError raised from each place:
+#
+#     from a pytest_runtest_call wrapper   1 failed
+#     from a fixture teardown              1 passed, 1 error
+#
+# The second arm is the control. Without it the first passes whatever phase the
+# guard is in, because "a vacuous test fails" is true of a correctly-placed guard
+# and of no guard at all plus an unrelated failure.
+
+
+def test_the_vacuity_guard_fails_the_test_rather_than_erroring_beside_it(pytester, expect):
+    """A test that concludes nothing must be FAILED, not passed-with-an-error."""
+    pytester.makepyfile(
+        """
+        def test_asserts_nothing(expect):
+            pass
+        """
+    )
+    result = pytester.runpytest("-p", "pgc_vacuity")
+    expect.outcomes(result, "a test that concludes nothing is failed, not errored",
+                    passed=0, failed=1, errors=0)
+
+
+def test_a_guard_in_a_teardown_would_report_a_pass_which_is_why_it_is_not_there(pytester, expect):
+    """The control, and the mode stated as a measurement rather than a warning.
+
+    The body asserts something, so the vacuity guard is satisfied; the refusal comes
+    from the teardown. pytest reports the test PASSED and adds an error, which is the
+    shape that makes a guard in a teardown unable to fail what it guards.
+    """
+    pytester.makepyfile(
+        """
+        import pytest
+
+        @pytest.fixture
+        def guard_in_teardown():
+            yield
+            raise AssertionError("a guard placed in a teardown instead")
+
+        def test_body(guard_in_teardown, expect):
+            expect.num(1, 1, "the body itself is fine")
+        """
+    )
+    result = pytester.runpytest("-p", "pgc_vacuity")
+    expect.outcomes(result, "a teardown refusal leaves the test reported as passed",
+                    passed=1, failed=0, errors=1)