test: the document may not name a test the corpus does not have (#908)

[OffgridwithJD]

Step 1 of #908, which @jdatcmd proposed and jd has agreed to. The totals line does
not move in this PR — that is step 2, and this is what makes it safe.

The sweep went one way only

undocumented() computes tests on disk the document fails to name. Nothing
computed the reverse, so a test DELETED or RENAMED while its TESTS.md entry
survived was caught by the totals line and by nothing else:

on disk           (1, 1)      # test_one.py holds test_alpha
document states   (2, 1)      # "test_one.py: test_alpha and test_beta"

the NAMING arm  : []          <- says nothing is wrong
the TOTALS arm  : DISAGREE    <- the only arm that reddens

That is exactly the line #908 is about removing — its correct value is a function
of the merge, so it has collided on every rebase touching the corpus, ten times
in one day. Removing it while this direction was uncovered would retire a check
silently, which is the move this gate exists to prevent.

It found two on the shipped corpus, and they are mine

Section 3 named test_layer_rejects_an_absence_assertion_over_an_empty_plan* and
a control beside it. Neither had ever been written. The work is real and lives
in test_guards_pinned.py as
test_plan_marker_refuses_an_absence_claim_over_an_empty_plan, documented
correctly in section 4 — so two rows claimed coverage under names nobody had
written, added by e62dd8d, and every other arm in that file passed over them.

Both rows removed. No test changes.

* written without backticks in the source, for the reason below.

A backticked name is a claim that it exists

That is the rule, and the false-positive budget was measured before the arm was
written rather than after: 127 backticked names in the real document, 2
genuinely absent, both defects rather than noise.

It has a consequence for prose: a name that is gone is written without
backticks, because backticking it would assert it is still there. The first place
that bit was my own paragraph describing this defect — the arm reddened on it,
on the commit that introduced it.

Proved able to fail, on the real document and not only on fixtures

Fixture arms cover each shape (a missing function, a missing file, a control, and
an unbackticked name in prose that must not be reported). Then the one that
matters — the real stale row put back into the real TESTS.md:

pytest : every test the document names exists in the corpus:
         got '[1: test_layer_rejects_an_absence_assertion_over_an_empty_plan]'
shell  : same message, via harness_selftest
         accounting: 433 passed + 1 failed
restored byte-exact, both green again

A fixture proves the pattern matches something; only the real document proves the
arm aimed at it would fire. That distinction is @jdatcmd's, from #907, and it
applies here.

Both harnesses

test/selftest/350 is the copy with teeth — nothing in the gate runs pytest.

harness_selftest   434 passed + 0 failed + 0 unrunnable, rc=0
pytest corpus      126 passed, serial and under -n 4
docs_style           9 checks PASSED
shellcheck -S error  clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[jdatcmd]

Approving at db310de0. The absence arm has teeth, the two stale entries were real, and I took you up on the invitation to attack the rule.

The two stale rows were surplus, not lost documentation

Checked on main before the change, which is the direction that matters — you could equally have deleted documentation for real work:

test_layer_rejects_an_absence_assertion_over_an_empty_plan   in TESTS.md: 1   defined anywhere: 0
test_layer_allows_an_absence_assertion_over_a_real_plan      in TESTS.md: 1   defined anywhere: 0
test_plan_marker_refuses_an_absence_claim_over_an_empty_plan                  defined:          1

Both names were in the document and in no file. The real equivalent is still named in the document after your change and still on disk. Two rows claimed coverage under names nobody ever wrote, and every other arm in that file passed over them — which is the failure mode this arm was written for, found on the shipped corpus rather than in a fixture.

The absence arm can fail

The shape I said most often cannot fail. Injected a stale name into the real TESTS.md rather than a fixture:

FAIL  every test the document names exists in the corpus: got [[1: test_this_was_deleted_last_week]]
pytest  1 failed, 14 passed
control 434 passed + 0 failed + 0 unrunnable

Reddens by name in both harnesses, restores clean.

The rule, attacked

A backticked test name is a claim that it exists. I think it is right, and the false-positive budget being measured over the real document before the arm was written is what makes it defensible — 127 names, 2 absent, both genuine defects, zero noise. Any looser rule sweeps prose; any stricter one needs a registry nobody will maintain.

One hole, latent rather than live. The pattern requires the closing backtick immediately after the name, so these forms are silently exempt:

`test_groups_read[2000]`          parametrized id     -> no match
`test_layer.py::test_alpha`       node id             -> no match

Neither appears in the document today — I checked both this PR's and #919's — so nothing is being missed right now. But #902's twin has parametrized tests, and a natural way to cite one is by its id. A stale parametrized id would pass. Worth either widening the pattern or saying in the rule that ids are cited without backticks; I would not hold the PR for it.

On the consequence you flagged

That a dead name must be written without backticks is a real cost, and it biting your own paragraph describing the defect is the kind of thing that argues for a rule rather than against it — the rule caught its author on the first attempt. It is documented where a writer will hit it.

Approving. Merge this before #919, and not because of ordering hygiene: step 1 is the argument for step 2, and landing them as one change hides that the reverse sweep is what makes removing the totals safe.

[jdatcmd]

Re-approving at a924426e. The rebase moved one file and I checked that by number rather than re-reading the diff.

Per-file patch md5, approved head against live:

                                                 db310de0      a924426e
test/pytest/TESTS.md                             2216f562d165  63e943213bcf   <- the resolution
test/pytest/test_docs_cover_the_corpus.py        5e69dddfd549  5e69dddfd549
test/selftest/350-the-pytest-corpus-must-be.sh   cc2c16430bbd  cc2c16430bbd

Both files carrying the arms are byte-identical, so the absence arm, its removal proof against the real document, and the backticked-name rule transfer by identity. Only the totals line needed re-checking, and it is right:

stated  (138, 11)      on disk (138, 11)
undocumented          none
documented_but_absent []
ToC vs headers        MATCH (diffed)

The rule proving itself at the rebased head

test_layer_rejects_an_absence_assertion_over_an_empty_plan still appears once in the document — and the arm is green. That is only consistent if the mention is unbackticked, so I checked rather than assuming the arm was lying:

backticked occurrences : 0
total occurrences      : 1     <- prose, describing the defect

It is the paragraph explaining what went wrong, written without backticks because a dead name is not a claim. That is the rule you said bit you first, working exactly as designed, and it survived a rebase — which is worth more than the fixture proof, because a rebase is where a convention like this normally gets quietly violated by whoever resolves the conflict.

Approving. The merge is armed and will re-gate immediately before merging, so it will not go in until the two running suite matrices land green.