diff --git a/CHANGELOG.md b/CHANGELOG.md index ee62d1c1..d074c249 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -160,6 +160,30 @@ true until the next version shipped. ### Fixed +- The vacuity guard's PLACEMENT is now a checked property, because a guard in a + teardown cannot fail the test it guards (#432). + + Measured, the same `AssertionError` raised from two places: from a + `pytest_runtest_call` wrapper the run reports `1 failed`; from a fixture teardown it + reports `1 passed, 1 error`. pytest has already recorded the call phase as passed, so + a teardown refusal arrives as a separate error on the same node-id and the test's own + outcome stays `passed`. Anything counting passes -- `--pgc-expect-tests`, a CI + summary, a human reading "N passed" -- sees a pass. + + This layer's guard was already in the call-phase wrapper, so nothing was broken. What + was missing is that nothing said so: moving it into the `expect` fixture's teardown + was a plausible-looking refactor that would have turned every vacuous test from + `failed` into `passed` with an error beside it. Two arms in `test_runshape.py` pin the + placement, and the second is the control -- without it the first passes whatever phase + the guard is in, because "a vacuous test fails" is equally true of a correct guard and + of no guard at all next to an unrelated failure. Proved by moving the guard into the + teardown: the first arm reddens. + + `guard-as-teardown-fixture-still-reports-passed` is NARROWED rather than closed, and + `VACUITY_MODES.md` 3.7 says which half. The half closed is this layer's own guard + placement. The half still open is the general shape: a guard anyone adds later in a + teardown still cannot fail its test, and nothing refuses that. + - `ALTER TABLE ... RENAME COLUMN` now carries the new name into `pgcolumnar.projection_declaration`, for the named relation and for every inheritance descendant, including a `PARTITION OF` child (#888). diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 2f591ddd..0a1884a3 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -935,6 +935,36 @@ The empty-parametrize refusal carries its own message rather than folding into t bare-skip refusal. When a corpus glob matches nothing, the cause the reader needs to see is the corpus, not the marker. +### Where a guard runs decides what the run reports + +A guard implemented as a **fixture teardown** cannot fail the test it guards. pytest has +already recorded the call phase as passed, so the refusal arrives as a separate `ERROR` +on the same node-id and the test's own outcome stays `passed`. Anything counting +passes — `--pgc-expect-tests`, a CI summary, a human reading "N passed" — sees a pass. + +Measured, the same `AssertionError` raised from each place: + +| raised from | the run reports | +| --- | --- | +| a `pytest_runtest_call` wrapper | `1 failed` | +| a fixture teardown | `1 passed, 1 error` | + +This layer's vacuity guard is in a `pytest_runtest_call` wrapper, which is why a test +that concludes nothing is *failed* rather than passed-with-an-error. Two arms keep that +from being an accident, and the second is the control: without it the first passes +whatever phase the guard is in, because "a vacuous test fails" is equally true of a +correctly-placed guard and of no guard at all beside an unrelated failure. + +| test | what it asserts | how it fails | +| --- | --- | --- | +| `test_the_vacuity_guard_fails_the_test_rather_than_erroring_beside_it` | a test concluding nothing is `failed`, with no error | moving the guard into the `expect` fixture's teardown reddens it | +| `test_a_guard_in_a_teardown_would_report_a_pass_which_is_why_it_is_not_there` | a teardown refusal leaves the test reported `passed` with an error beside it | the mode stated as a measurement rather than a warning | + +This closes the part of `guard-as-teardown-fixture-still-reports-passed` +(`VACUITY_MODES.md` 3.7) that is about **this layer's own guard placement**. It does not +close the family: a guard anyone adds later in a teardown is still a guard that cannot +fail its test, and nothing refuses that shape in general. + ## 11. test_zonemap_boundaries.py: exact boundaries ### `test_exact_zonemap_boundaries` diff --git a/test/pytest/VACUITY_MODES.md b/test/pytest/VACUITY_MODES.md index 8ffb92ac..c73a094f 100644 --- a/test/pytest/VACUITY_MODES.md +++ b/test/pytest/VACUITY_MODES.md @@ -215,8 +215,14 @@ the sentinels that close the other half, empty compared with empty. ### 3.7 The guard itself goes quiet -- `guard-as-teardown-fixture-still-reports-passed` — a guard implemented as a - teardown fixture leaves the test reporting PASSED. +- `guard-as-teardown-fixture-still-reports-passed` — **narrowed, not closed.** Measured: + the same `AssertionError` raised from a `pytest_runtest_call` wrapper gives `1 failed`, + and raised from a fixture teardown gives `1 passed, 1 error` — the test's own outcome + stays `passed`, so anything counting passes sees a pass. This layer's vacuity guard is + in the call-phase wrapper, and two arms in `test_runshape.py` now pin that placement + with a control, so a refactor into a teardown reddens rather than going quiet. What is + NOT closed is the general shape: a guard anyone adds later in a teardown still cannot + fail its test, and nothing refuses that. See TESTS.md section 10. - `session-accounting-guard`, `session-exit-rewrite-masks-a-real-failure`, `description-guard-reopens-psycopg-raise`, `mitigations-measured-and-the-one-that-does-not-work` diff --git a/test/pytest/test_runshape.py b/test/pytest/test_runshape.py index 2249da2d..cc2604d0 100644 --- a/test/pytest/test_runshape.py +++ b/test/pytest/test_runshape.py @@ -193,3 +193,62 @@ def test_keep_f(expect): expect.num(1, 1, "f") "--max-worker-restart=0") expect.run_failed(result, "a lost test is still caught when others were deselected") result.stdout.fnmatch_lines(["*never reported*"]) + + +# ---- WHERE a guard runs decides what the run reports ------------------------- +# +# `guard-as-teardown-fixture-still-reports-passed` (VACUITY_MODES.md 3.7). A guard +# implemented as a fixture teardown cannot fail the test it guards: pytest has +# already recorded the call phase as passed, so the refusal arrives as a separate +# ERROR on the same node-id and the test's own outcome stays `passed`. Anything +# counting passes -- `--pgc-expect-tests`, a CI summary, a human reading "N passed" +# -- sees a pass. +# +# This layer's own vacuity guard is in a `pytest_runtest_call` wrapper, which is the +# right place, and these two arms are why that stops being an accident. Measured, +# the same AssertionError raised from each place: +# +# from a pytest_runtest_call wrapper 1 failed +# from a fixture teardown 1 passed, 1 error +# +# The second arm is the control. Without it the first passes whatever phase the +# guard is in, because "a vacuous test fails" is true of a correctly-placed guard +# and of no guard at all plus an unrelated failure. + + +def test_the_vacuity_guard_fails_the_test_rather_than_erroring_beside_it(pytester, expect): + """A test that concludes nothing must be FAILED, not passed-with-an-error.""" + pytester.makepyfile( + """ + def test_asserts_nothing(expect): + pass + """ + ) + result = pytester.runpytest("-p", "pgc_vacuity") + expect.outcomes(result, "a test that concludes nothing is failed, not errored", + passed=0, failed=1, errors=0) + + +def test_a_guard_in_a_teardown_would_report_a_pass_which_is_why_it_is_not_there(pytester, expect): + """The control, and the mode stated as a measurement rather than a warning. + + The body asserts something, so the vacuity guard is satisfied; the refusal comes + from the teardown. pytest reports the test PASSED and adds an error, which is the + shape that makes a guard in a teardown unable to fail what it guards. + """ + pytester.makepyfile( + """ + import pytest + + @pytest.fixture + def guard_in_teardown(): + yield + raise AssertionError("a guard placed in a teardown instead") + + def test_body(guard_in_teardown, expect): + expect.num(1, 1, "the body itself is fine") + """ + ) + result = pytester.runpytest("-p", "pgc_vacuity") + expect.outcomes(result, "a teardown refusal leaves the test reported as passed", + passed=1, failed=0, errors=1)