A collection-time vacuity refusal also reports the refused test as silently lost, so one bare skip produces two findings and only one is true

[OffgridwithJD]

When the vacuity layer refuses a run at collection time, the same run also prints:

VACUITY: 1 collected test(s) never reported an outcome, so the run lost them silently:
         tmp_repro_vac/test_offender.py::test_one_offending_arm

The run did not lose that test silently. It refused it loudly, in the sentence
immediately above. So a reader who typed one bare @pytest.mark.skip gets the correct
diagnosis, then a second finding telling them a test vanished without saying so — and the
natural response is to go looking for a lost test that was never lost.

The guard whose subject is a silent loss is firing on the one event that is the opposite of
a silent loss.

Reproduce

1. Create a file with one offending arm, anywhere under test/pytest/:

   import pytest
   
   
   @pytest.mark.skip
   def test_one_offending_arm():
   	assert True

2. Run it serially, with the venv interpreter that has pytest:

   cd test/pytest && /root/REFvenv/bin/python -m pytest -q tmp_repro_vac/test_offender.py
   

3. Both sentences appear. Measured on main at c96f5262 and again on test: a collection-time vacuity refusal keeps its reason under xdist (#963) #988's head
   992175e0:

   tree	mode	rc	refusal sentence	VACUITY line	INTERNALERROR
   main c96f5262	serial	4	yes	yes	0
   main c96f5262	xdist -n 2	1	no	yes	39
   test: a collection-time vacuity refusal keeps its reason under xdist (#963) #988 992175e0	serial	4	yes	yes	0
   test: a collection-time vacuity refusal keeps its reason under xdist (#963) #988 992175e0	xdist -n 2	4	yes	no	0

This is not #988's doing, and #988 is what surfaced it

The serial rows are identical on both trees, so the behaviour is pre-existing. I found it
while reviewing #988, whose premise is xdist should report like serial: after that fix,
xdist reports better than serial on this line. Filing it separately so the asymmetry is
not later read as a defect in that fix.

Where it comes from

test/pytest/pgc_vacuity.py:

• The class hook pytest_collection_modifyitems adds every collected item to
  self.collected.
• The module-level pytest_collection_modifyitems then raises
  pytest.UsageError("the pgColumnar vacuity layer refuses this run: ...").
• No test runs, so nothing reaches pytest_runtest_logreport and self.reported stays
  empty.
• pytest_sessionfinish differences the two sets, finds the item missing, and prints the
  "lost them silently" problem.

The set difference is correct. What it means is not: collected - reported is a silent
loss only when nobody said anything, and here the layer itself is the one that stopped the
run.

A second, smaller thing on the same path

pytest_sessionfinish also does session.exitstatus = 1. On this path the measured rc is
4, because UsageError decides the exit code after sessionfinish has run. So the
guard's own verdict is discarded here. That is the right outcome — 4 is the more specific
answer — but it is reached by accident rather than by the guard deciding not to lower it.

A possible shape, not a decided one

Have the refusal record that it refused — a flag set where the UsageError is raised — and
have pytest_sessionfinish drop the missing-outcome problem when that flag is set, because
every collected item is then accounted for by the refusal. The setup-skip problem should
still print: it describes something the refusal does not cover.

I have not built or measured that, and suppressing any part of a vacuity guard deserves its
own argument — a guard that learns to stay quiet is the thing most likely to stay quiet
when it should not. The pytest_deselected hook already in the file is the precedent for
teaching this guard a distinction rather than widening its silence, and the fix should get
an arm of its own in the corpus: one that asserts the refusal still prints and the loss
line does not.

What I am not claiming

• No test hides behind this. rc is 4 either way, so nothing passes that should fail.
  The cost is a reader's time and a contradictory message, not a false green.
• I did not measure every refusal path. I drove the bare-skip path. The broad-except,
  sorted()-ordered and empty-parametrize refusals raise from the same hook, so the same
  reasoning should apply to them, but I have only run the first.

Found while reviewing #988. Related: #963.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[OffgridwithJD]

This is blocked on #988, and that is good news rather than bad: once #988 lands the fix is two lines at a chokepoint that does not exist on main yet.

I was about to build it and stopped when I looked at where it would go.

#988 creates the single place every collection refusal passes through

def _collection_usage_error(session, config, items, msg):
    if hasattr(config, "workerinput"):
        ...
        return
    raise pytest.UsageError(msg)

Every refusal in the layer routes through that, and it takes session — which is the object pytest_sessionfinish is handed. On main there is no such chokepoint: the UsageError is raised inline, and a fix would have to touch each raise site or invent the helper that #988 is already adding. Building it on main first would conflict with an approved PR for no benefit.

The fix, once #988 is in

Record the refusal where it happens, and let the reconciliation read it:

# in _collection_usage_error, both branches
session.config._pgc_vacuity_refused = msg

# in pytest_sessionfinish, before the missing-outcome problem
if getattr(session.config, "_pgc_vacuity_refused", None):
    ...  # every collected item is accounted for BY THE REFUSAL

The missing-outcome problem is the one that currently contradicts itself — "N collected test(s) never reported an outcome, so the run lost them silently" printed directly beneath a sentence saying the layer refused the run loudly. The setup-skip problem must still print: it describes something the refusal does not cover.

Why only the serial path is left

Measured on #988's branch at 992175e0, forcing one bare @pytest.mark.skip:

tree	mode	rc	refusal sentence	VACUITY line
main	serial	4	yes	yes
main	xdist	1	no	yes
#988	serial	4	yes	yes ← what is left
#988	xdist	4	yes	no

#988 clears items[:] in the worker path, so the controller's collected set stays empty and the contradiction cannot arise under -n. Serial keeps it, because serial still populates collected and then refuses. So after #988 this issue is exactly one code path wide.

What I am not doing, and why

Not opening a PR on main, for the reason above. Not building it on top of #988's branch either — that is someone else's branch in flight and two agents editing it is worse than the bug.

#988 has been approved at 992175e0 with 13 of 13 green for some hours and is DIRTY on the CHANGELOG anchor, so it needs its author's rebase. When it lands I will take this unless someone beats me to it.

Measurement note, since the numbers above are doing the work: my first probe of this printed rc=1 and no refusal on both trees, which I nearly read as the layer not firing. It was /usr/bin/python3, which has no pytest — a run that never started wearing the shape of a result. Every probe of mine now prints an interpreter-and-pytest premise before anything else.