test/pytest: a loud refusal is not a silent loss (#991)

[OffgridwithJD]

The layer refused a run and then contradicted itself. One bare @pytest.mark.skip:

ERROR: the pgColumnar vacuity layer refuses this run: a bare skip is refused ...
VACUITY: 1 collected test(s) never reported an outcome, so the run lost them
         silently: tmp/test_offender.py::test_one_offending_arm

The run did not lose it silently. It refused it loudly, one line above. A reader who typed one bare skip got the correct diagnosis and then a second finding telling them a test vanished without saying so — and the natural response is to go hunting a lost test that was never lost.

collected - reported is the right set difference and the wrong meaning. A silent loss is when nobody said anything; here the layer itself is what stopped the run. The guard whose entire subject is a silent loss was firing on the one event that is its opposite.

The fix is one assignment, because #963 built the chokepoint

Every refusal in the layer routes through _collection_usage_error, so recording it there covers all five call sites:

    # RECORDED BEFORE EITHER BRANCH, so the reconciliation cannot call this a silent loss
    config._pgc_vacuity_refused = msg

and the reconciliation skips only that problem:

    refused = getattr(session.config, "_pgc_vacuity_refused", None)
    ...
    if missing and not refused:

The setup-skip problem still prints. A fixture removing every test that depends on it is not something a refusal accounts for, and the two are independent findings.

This PR had to wait for #963 rather than being written on the base it was filed against: on main before that, the UsageError was raised inline at each site and there was no single place to record it. Writing it earlier would have meant touching five raise sites or inventing the helper #963 was already adding.

Serial was the only path left, which is why the change is this small

main   serial   refusal + the silent-loss line
main   xdist    no refusal at all                 (that was #963)
#963   serial   refusal + the silent-loss line    <- what this closes
#963   xdist    refusal, no line

#963 clears items[:] in the worker, so the controller's collected set is already empty under -n and the contradiction cannot arise there.

Red to green, driven on both trees

the new arms against main's pgc_vacuity.py   1 failed
    FAILED ...test_a_collection_refusal_is_not_also_reported_as_a_silent_loss[serial]
    the [xdist] row PASSED on main, which confirms #963 closed that half rather than
    my having guessed it
the same arms with the fix                   3 passed, 11 checks
whole corpus on the branch                   365 passed, 923 checks

The second arm is the control, and the first is worth nothing without it

Dropping a problem from a reconciliation is one edit away from dropping the guard. So test_a_genuine_silent_loss_is_still_reported asserts that an item collected and never reported, with no refusal anywhere, is still named and still reddens the run:

@pytest.hookimpl(trylast=True)
def pytest_collection_modifyitems(config, items):
    del items[-1]

Removed after the layer's own hook recorded it, and without deselecting — which is the shape of a crashed xdist worker. pytest_deselected is the hook that tells a deliberate subset from a loss, and nothing calls it there. The arm checks the sentence, the name of the vanished test, and rc=1.

Two of this tree's guards caught my omissions, and it was one root cause

test_docs_cover_the_corpus refused the new tests until TESTS.md named them. test_harness_deps then reported the no-driver batch as rc=1 — which looked like a second, unrelated failure and is not: the docs guard is inside that batch. I checked that my own tests pass with psycopg shimmed out before concluding it, rather than assuming the obvious culprit.

Limits

• No ledger change, measured: harness_selftest is 907 checks on this tree and on main. No shell check was added, removed or renamed.
• Only the missing-outcome problem is suppressed, and only on a refusal. Any other run, and any setup skip, reports exactly as before.
• The flag lives on config, not the session. pytest_sessionfinish receives session and reaches session.config; a worker's config is a different object from the controller's, which is why the worker branch sets it too even though the worker's own sessionfinish returns early.

Part of #991. The issue also notes session.exitstatus = 1 being discarded on this path — UsageError decides the code after sessionfinish — which is the right outcome reached by accident rather than by the guard deciding not to lower it. That is untouched here and #991 stays open for it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[linuxhikerpm]

The hatch is real and I drove it at 578932a80702b61ddfa3a49ff2ddb058b2312cc9.

main pgc_vacuity + these tests   serial: lost them silently got 1 want 0
this head                        7 passed, including both modes and the genuine-loss control

The serial row is the one #963 left, and the control still names test_vanishes_without_a_word when nothing refused. I would approve the code on that.

Blocking: docs_style.sh fails on this CHANGELOG. Three em dashes, got [3] want [0]. docs_style is a registered suite, so the PG17/PG18 jobs will red for a reason unrelated to the refusal. The three sites are the new #991 bullet (the "so — and" / "prints — a" / "worker — pytest" sentences). -- is the spelling the rest of Unreleased uses.

Do not merge from this review.

[linuxhikerpm]

Re-reviewed at e11e1fc59e80cddefba65a7160bc98ec295bc6a3. CHANGELOG em dashes are gone (em 0 en 0), which is the docs_style.sh hatch that failed suites on the previous head.

Drove the two new tests:

this head                                      3 passed, 11 checks
main pgc_vacuity + these tests                 serial: lost them silently got 1 want 0
                                               (xdist still 0, as #963 already emptied collected)

The control (test_a_genuine_silent_loss_is_still_reported) stayed green on both trees. Do not merge from this side.

[jdatcmd]

Approving. The diagnosis is right, the fix is at the one place that covers every surface, and the removal proof lands.

Verified rather than read

One assignment really does cover every call site. config._pgc_vacuity_refused = msg sits at line 1507, inside _collection_usage_error (1484), and there are exactly five call sites — 1535, 1543, 2108, 2154, 2218. So the claim in the comment holds by construction rather than by inspection of each caller.

Worth noting the composition: two of those five are surfaces that only started going through this function today. #963 routed the module-binding refusal; #986's conflict resolution routed the Expect-method refusal this afternoon. Before that merge, a method-stub refusal would have set no flag and would still have been reported as a silent loss. This PR lands on the tree where "every refusal comes through here" became true, and it would have been a partial fix a day ago.

Removal proof:

revert `if missing and not refused:` to `if missing:`
  -> test_a_collection_refusal_is_not_also_reported_as_a_silent_loss[serial]   FAILED

The full database-free batch on the branch: 267 passed, 637 checks, 0 fail.

One note, and your docstring already contains the answer

The mutation reddens [serial] and not [xdist]. Your own table says why:

#963   serial   refusal + the silent-loss line   <- what this closes
#963   xdist    refusal, no line

Clearing items[:] in the worker had already emptied the controller's collected set, so the xdist row had no line to suppress before this PR. That makes the [xdist] arm a regression guard for #963's property, not a second measurement of this fix — it passes on the mutant and on the repair alike.

That is the right arm to have and I would not remove it. What I would add is one sentence saying so, because the docstring states the four-row history and leaves the reader to infer that only one of the two parametrized arms can fail for this change. Two green arms read as two confirmations; here they are one confirmation and one guard. Given how much of tonight went into arms that could not fail, naming the one that cannot fail for this reason seems worth the line.

The diagnosis is the part worth keeping

collected - reported is the right set difference and the wrong MEANING: a silent loss is when nobody said anything, and here the layer itself is what stopped the run.

That is the whole bug in one sentence, and it is a shape that will recur: a set difference that is arithmetically correct and semantically wrong because the reason for the difference is not in the sets. The guard whose subject is silence fired on the loudest event the layer produces.

Verdict

Approved. Correct fix, correct place, proof both ways, and the limitation of the xdist arm is disclosed rather than papered over.