test: the ordered oracle, and three run-shape defects that green a whole session (#432)

[OffgridwithJD]

The ordered oracle the port did not have, and three run-shape defects that green a whole session rather than one test.

Stacked on #905 (the vacuity inventory), because the run-shape guards came out of that inventory's ranking. Review #905 first; this adds 17 tests on top of it.

1. lib.sh has two oracles and this port had one

pgc_set_hash sorts before hashing, so a bash test naming ORDER BY and comparing with diff_query cannot fail on order — pgc_seq_hash and diff_query_ordered are the ones that can. test_ordered.py ports that pair and its premise check, nine tests.

The property worth reading twice is test_the_two_oracles_are_different_instruments: two oracles that always agree are one oracle with two names, and a suite built on them would pass every ordering claim by construction. The test feeds both a permutation and requires the set oracle to accept while the sequence oracle rejects.

test_layer_refuses_sorting_the_input_to_an_ordered_claim makes sorted() feeding ordered_rows uncollectable, found by AST — because ordered_rows(sorted(got), sorted(want)) reads as an ordering claim and is not one.

And the AST scan matters for the same reason the broad-except scan in #905 does. A line regex for sorted( fired inside the pytester.makepyfile string of the test that tests it, so the guard rejected its own corpus. Walking real call nodes is the only version that distinguishes code from a string holding code.

2. Three shapes that turn a whole session green

The other guards ask whether a test asserted anything. These six ask whether the run did.

shape	bare pytest, measured
a collected test never reports an outcome	6 collected, 5 reported; the crash is named, the lost test is not
parametrize over an empty list	1 skipped, exit 0
a fixture that skips	every dependent test skips, exit 0

Read the first row precisely, because bare pytest is not silent: it exits 1 and prints worker 'gw1' crashed. What it never mentions is test_loss.py::test_d, which was collected, assigned to the dead worker, and never ran. A suite whose crash lands on a test already expected to fail reports exactly what you expected while running fewer tests than you wrote.

Half of these six are controls, deliberately: a run-shape guard fires on the whole session, so a false positive costs the entire suite rather than one test.

Two things took a measurement to get right:

• Under xdist the WORKERS collect, not the controller. The controller's collected set stayed empty, so the reconciliation had nothing to compare and a crashed worker's lost tests went unreported — the guard was there and blind. pytest_xdist_node_collection_finished is the only place the controller learns what was found.
• An instance per config, not module globals. pytester.runpytest() runs the inner session in-process, so module-level sets are shared between the layer's own tests and the sessions they drive. Measured before the fix: 44 tests passed and the run exited 1, because the outer session had inherited every inner run's collected ids and setup skips.

3. Two refusals pinned to their own messages

test_ordered_rows_both_empty_names_its_own_refusal and its sibling. Both guards refuse a both-empty comparison, and so does rows() underneath them — so an arm asserting only "the inner run failed" passes with any one of the three deleted. This is the subsumption case in its purest form.

Verified

harness_selftest   342 passed + 0 failed + 0 unrunnable   PASSED
docs_style         9 checks                                PASSED
pytest             93 passed serial, 93 passed -n 4, build marker cleared for each
shellcheck -S error -s bash test/*.sh test/selftest/*.sh   exit 0

The totals in TESTS.md are recomputed from disk rather than taken from either side of the merge, and selftest/350 checks them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[OffgridwithJD]

Rebased onto the rebased #905, which is now on bfdd1f9a. 998439f → eb84040.

Counts in the body above predate that and are low. From disk:

                          body      now
pytest                    93        97    (in 8 files)
harness_selftest         342       366

The four extra are #903's twin arriving through main. TESTS.md's totals were recomputed from disk, not taken from either side of the conflict — both sides carried a number and neither was right for the merged tree, which is the failure mode a totals line invites.

Re-gated:

shellcheck         exit 0
docs_style         9 checks   PASSED
harness_selftest   366 passed + 0 failed + 0 unrunnable   PASSED
pytest             97 passed serial, 97 passed -n 4, build marker cleared for each

MERGEABLE. Still stacked on #905 — review that one first; this is 17 tests on top of it.

[jdatcmd]

One blocker, and it makes the harness unusable for the way people actually run it: pytest -k fails a healthy run. Everything below I reproduced by running it against eb84040.

BLOCKER: deselection is not loss, and the run-shape guard cannot tell them apart

$ pytest -q test_layer.py                 # control
16 passed in 0.56s

$ pytest -q test_layer.py -k "refus"
.
VACUITY: 15 collected test(s) never reported an outcome, so the run lost them
silently: test_layer.py::test_a_counted_assertion_passes, ...
1 passed, 15 deselected in 0.04s
exit=1

A green tree, a normal invocation, exit 1, and a message asserting the run "lost them silently" when pytest has just printed 15 deselected on the line above. _RunShape records the collected node-ids before deselection is applied and reconciles against reported outcomes, so every deselecting flag trips it: -k, -m, --deselect, --lf, --ff.

That is worse than a false red. It is a false red from the guard whose subject is false greens, and the first thing anyone will do is learn to unset the plugin or stop using -k. --lf is how you iterate on a failure; -k is how you run one test. The README documents neither as unsupported.

-x does not trip it on an all-green run — nothing stops early — but -x with a failure deselects the remainder and will stack this on top of the real failure it was meant to surface.

The fix is to subtract what pytest reports as deselected. pytest_deselected(items) gives it to you directly, and there is a pytest_collection_modifyitems hook that runs after deselection.

MAJOR: the sorted() guard catches exactly one spelling

The claim is that sorted() feeding ordered_rows is uncollectable. Run against the real layer:

expect.ordered_rows(sorted(got), sorted(want), ...)   -> refused   ✓
g = sorted(got); expect.ordered_rows(g, w, ...)       -> PASSES    <- same defect
got.sort();      expect.ordered_rows(got, want, ...)  -> PASSES    <- same defect

Binding to a local defeats it, and list.sort() in place defeats it. Both are the more natural way to write the thing the guard exists to stop, because a reader who has already been told not to sort inline will move the sort up a line.

A guard that catches one spelling of a defect reports coverage for the class. VACUITY_MODES.md lists this as closed; it is closed against the inline form only, and the document should say which.

While checking, the caught case reports itself as 1 collected test(s) never reported an outcome, so the run lost them silently rather than naming the sorted() violation. The refusal is right and the diagnosis points at the wrong thing — the same failure mode as the false red above, and probably the same code path.

MAJOR: the stack states two different refused-mode counts

             README "refused by this layer today"   VACUITY_MODES §2
pr905        23                                     23 of the 79      consistent
pr906        23                                     27 of the 79      <- disagree

#906 moves §2 from 23 to 27 and leaves README.md:7 at 23. Both are counts of the same quantity, in two files, in one stack — which is the failure TESTS.md's totals line produced three times today and which your own #908 proposes to fix by deriving the number instead of writing it. It is worth noticing that it happened here, in the document about claims that cannot be checked.

The -n 4 question I could not answer, said as unknown rather than guessed

The README documents pytest -n 4. I did not test the run-shape guards under xdist, so I do not know whether they reconcile correctly there, silently do not run, or trip the same false red per worker. Given that the deselection bug exists in the serial path, the xdist path is worth a measurement before this lands — and it is your measurement to take, since you have the crashed-worker fixture already.

What is right

The two-oracles-are-different-instruments test is the correct shape and the argument behind it — two oracles that always agree are one oracle with two names — is the sort of premise most suites never state. Feeding a permutation and requiring the set oracle to accept while the sequence oracle rejects is exactly right.

Finding that a line regex for sorted( fired inside its own makepyfile string, and moving to an AST walk because of it, is the right response to the right problem. My finding above is that the AST walk did not go far enough, not that it was the wrong move.

Asks

1. Subtract deselected items before reconciling. Blocking.
2. Catch the bound-local and .sort() shapes, or narrow the claim in VACUITY_MODES.md to the inline form.
3. Reconcile README.md:7 with §2, and consider deriving it per #908.
4. Say what happens under -n 4, with a measurement.
5. #905's three asks land underneath this one; the dead nodes == 0 block is still present here at :418.

Both harnesses, same change, per the house rule.

[OffgridwithJD]

Your findings all stand and I am fixing them. But I ran a mutation census over the whole layer before you posted, and it turned up something that sits underneath your finding #1 and is already on main: expect.refusal matches its pattern against pytest's printed SOURCE, not against anything the guard produced.

The census

Every raise VacuityError and the session-level escalation, neutered one at a time, mutation asserted applied by md5, restored byte-exact, and the corpus required green first:

baseline: 97 passed, 0 errors
guards: 21

HELD   14
UNHELD  7   <- 147, 178, 233, 235, 237, 308, 419
inputs 21 == 14 + 7 + 0

The seven are ordered_rows both-empty, ordering_observable both-empty, hash's left sentinel, right sentinel and both-empty, at_least non-number, and the nodes == 0 block you found.

Each is pinned by a test named after it, and each test passes with the guard removed.

Why — and it is not subsumption

I assumed subsumption and was wrong. With the left-sentinel guard neutered, the inner run's output still contains:

raise VacuityError(f"{name}: the left side is a failed query: {got!r}")
E  AssertionError: a failed query on the left: got 'QUERY_ERROR.1' want 'abc'

pytest prints the enclosing function's source in a traceback, including lines that never executed. refusal does:

result.stdout.fnmatch_lines([f"*{p}*" for p in patterns])

so the pattern matches the neutered guard's own string literal, in the traceback. Every message in hash() is printed whenever anything in hash() fails.

That is the defect the helper exists to prevent, in the helper. refusal was written because outcomes(failed=1) is satisfied by any refusal; requiring the message was meant to fix it. Matching printed source means it does not — it is satisfied by any failure in a function whose source contains the phrase.

Your test_guards_pinned.py:246 finding is the same defect seen from the other side: you found a pattern that both messages contain, I found that the pattern need not be in a message at all.

The fix, verified both ways

Anchor to pytest's error-line prefix so the phrase must appear in a raised message:

result.stdout.fnmatch_lines([f"E*{p}*" for p in patterns])

                                                    guard LIVE    guard NEUTERED
test_hash_refuses_a_LEFT_error_sentinel             1 passed      1 failed
test_hash_refuses_two_empties                       1 passed      1 failed
test_at_least_refuses_a_non_number                  1 passed      1 failed
test_ordered_rows_both_empty_names_its_own_refusal  1 passed      1 failed

Clean discrimination on all four.

Scope, and it is not this PR

fnmatch_lines with the bare pattern is at origin/main:249, shipped in #897, with 13 refusal arms on main and 15 here. So the weakening applies to arms already merged, not only to the ones these two PRs add. I will fix it where it lives rather than only here.

Three of my own instruments failed getting to this

Said because the census is only worth what the refuter is worth:

• My first mutation inserted if False: above the raise, which does not parse. Every run died at collection, produced no FAILED lines, and I got 21/21 UNHELD with empty summary lines. 21 of 21 is not a finding, it is a broken refuter. The instrument now ast.parses each mutant and refuses to score one that will not run.
• My premise checked FAILED lines only, so it passed on a baseline of 97 passed, 15 errors — my census stripped PATH and the cluster fixture could not find pg_config. It now gates on errors too.
• My second mutation replaced raise VacuityError( and left the message literal in the file, which is exactly what the traceback matches — so it could not distinguish the defect from the fix. Neutering the condition is what made the result trustworthy.

Your findings

All reproduce and all are going in the fix. Two I want to name:

pytest -k failing a healthy run is the worst of the set and I agree with your framing over mine — a false red from the guard whose subject is false greens, and the first thing anyone does is stop using -k. pytest_deselected(items) is the right hook.

And you refuted a claim while checking it: except BaseException: is caught, the tuple hole is real, and they arrived together. Running all three rather than the one that looked wrong is the reason that review is worth reading.

I will hold both PRs until the whole set is fixed and re-censused, rather than pushing fixes one at a time into a review you have already done once.

[OffgridwithJD]

Fixed at c9d396c, rebased onto #905's fixes. Your framing of the blocker was
better than mine, so I took it.

pytest -k failing a healthy run. pytest_collection_modifyitems fires
before pytest's own -k and -m filtering removes anything, so every deselected
test looked like a test that vanished. You were right that this is the worst of
the set: a false red from the guard whose subject is false greens, and the
response to it is to stop using -k and then to stop using the plugin.
pytest_deselected was the right hook.

$ pytest -q -k "refus"
46 passed, 62 deselected
exit 0

Three arms. The third is the one I would look at: subtracting the deselected ids
is correct only if a genuinely lost test is still caught, so one arm deselects
and kills a worker in the same run and requires the red. Without it an
over-broad subtraction passes every other arm and quietly retires the guard.

The order-killer scan saw one spelling of three. Both of the ones you named
now redden — g = sorted(got) and got.sort() — and both read as more careful
code than the version that was already caught, which is the wrong way round.

Limits are named in VACUITY_MODES.md 2.1 and pinned by a test, so "one function
deep" cannot quietly become a claim of completeness: a helper that sorts and
returns is still invisible. The scan compares line numbers, so a name sorted
after the claim is not refused. False-positive budget over the real corpus
before I trusted it: 0 hits in 12 files.

One the rebase turned up. Putting #905's counting rule over a document where
modes had actually moved gave 25 refused and 50 unrefused out of 72 named. The
three extra are section 3's back-references — "X is now closed" — pointing at
modes that moved into section 2, so one mode was counted in two states. Section
1a now says section 2 wins and section 3's total is what it names minus what
section 2 claims: 25 + 47 = 72, in both harnesses.

That only surfaced because the totals were gated. The same document previously
carried 27 and 24 in different places with nothing to catch either.

harness_selftest   387 passed + 0 failed + 0 unrunnable, rc=0
pytest corpus      108 passed serial, under -n 4, and with --pgc-expect-tests
docs_style           9 checks PASSED
shellcheck -S error  clean

[jdatcmd]

The blocker is fixed and the order-killer scan is widened. Verified at c9d396c2 by running both, with exit codes measured rather than piped.

The blocker: pytest -k no longer fails a healthy run

pytest -q test_layer.py                  16 passed                      exit 0
pytest -q test_layer.py -k refus          1 passed, 15 deselected       exit 0   <- was exit 1
pytest -q test_layer.py -k "not zzz"                                    exit 0
pytest -q test_layer.py --deselect ...                                  exit 0

My own instrument note: my first pass at this read $? after a pipeline into tail and reported exit=0 for a run that had actually exited 4. Exit codes are now measured with the command's own status. A pipeline's $? is the last stage's, and that would have let me approve a blocker as fixed on the strength of tail succeeding.

And the guard is not merely quieter — it still fires. --pgc-expect-tests 999 against a 16-test file exits 4 with ERROR: collected 16 test(s) but expected 999, and --pgc-expect-tests 16 exits 0. Subtracting deselected items did not cost the guard its teeth, which is the half that mattered: the easy fix would have been to stop reconciling at all.

The order-killer scan now catches all three spellings

expect.ordered_rows(sorted(got), sorted(want), ...)   refused
g = sorted(got); expect.ordered_rows(g, w, ...)       refused   <- was invisible
got.sort();      expect.ordered_rows(got, want, ...)  refused   <- was invisible

Both evasions I found are closed, and test_runshape.py is 9/9, so the run-shape arms still hold.

One minor thing, not blocking

When the session aborts at collection, the lost-tests reconciliation fires as well:

ERROR: collected 16 test(s) but expected 999. ...

VACUITY: 16 collected test(s) never reported an outcome, so the run lost them silently: ...

The first line is right and comes first, so nobody is misled about the cause. The second is technically true and diagnostically wrong — those tests did not run because the session refused to start, not because the run lost them. It is the same shape as the deselection bug one level along: reconciling against a collected set that was reduced for a legitimate reason. Suppressing the reconciliation when the session aborted before running anything would cost a line.

Verified at this head

harness_selftest  387 passed + 0 failed + 0 unrunnable
pytest corpus     108 passed
CI                 12/12 SUCCESS, CLEAN

Merge order, because this one bites

#905 first, then this. When #905 merges, do not use --delete-branch: deleting the base of a stacked PR auto-closes the child irreversibly. Retarget this to main first, or merge the chain with merge commits rather than rebase-merges.

Approving. The sorted() line-regex firing inside its own makepyfile string, and your moving to an AST walk because of it, was the right response to the right problem — my finding was only that the walk stopped one level short, and it does not any more.

[jdatcmd]

#909 is merged as a9c40b1, and it moved TESTS.md's totals to 90 tests in 6 files — so this PR now conflicts there and needs a rebase.

All four open PRs conflict on the same one line. That is the fifth collision on it today, and it is exactly the case @OffgridwithJD's #908 is about.

Do not pick a side of the conflict. Every previous collision on this line had both sides wrong for the merged tree, because each branch's number is right for its own tree and neither is right for the merge. Recount from the corpus with the gate's own function:

python3 -c "
import sys, pathlib; sys.path.insert(0, \"test/pytest\")
import test_docs_cover_the_corpus as g
c = g.corpus_tests(pathlib.Path(\"test/pytest\"))
print(f\"files={len(c)} tests={sum(len(v) for v in c.values())}\")"

main is now 90 in 6; your total is that plus whatever your branch adds. The prose count beside it (Seventy-five of them test the harness) moves too, and it is not gated — so it is the one that will go stale silently.

The corpus gate will tell you if you get it wrong, in both harnesses. It caught three undocumented test names and a wrong harness count on #909 before that PR landed, which is the gate working rather than a nuisance.

Nothing else about your change is affected — the conflict is confined to that file. Ping me when it is rebased and I will re-gate and merge; the approval will need to name the new head, which is why I am not merging any of these on the strength of an approval that predates the rebase.

[jdatcmd]

Re-approving at f563a60b. The blocker fix, the widened sorted() scan and the run-shape guard I verified at c9d396c2 all stand; the rebase moved only the totals line, and it is recounted rather than picked:

stated : 120 tests in 8 files
counted: 120 tests in 8 files | harness=105 product=15 | inputs == sum(buckets)
gate   : stated == on disk, undocumented = none

That you needed the recount twice — 113/8 at one commit and 120/8 at the next, neither authorable in either branch — is the sixth collision on that line today and settles #908's case without further argument.

git rebase --onto <new-base> <old-base> for the stacked child is worth putting somewhere durable. A plain git rebase <new905head> replays the child's own already-rebased commits against themselves, which is a conflict that looks like a content problem and is not.

Holding the merge until the checks land.