test: a test file documented outside the numbering is invisible (#1024)

[OffgridwithJD]

Closes #1024 — but not by doing what it asks, because half of what it reports is already fixed and the other half is a different shape.

The reported defect is closed already

#1024 says two PRs each take the next section number, the merge keeps both, and nothing notices. Measured on 52511a7: test_the_contents_list_is_numbered_in_order landed after #1023 and catches exactly that. Planted:

M2  a DUPLICATE section number  ->  2 failed, naming the inversion

Its own docstring says so: "the collision that produced this is a section NUMBER taken twice, so the sections are where a duplicate shows up first, and 1..N with no gap catches a duplicate and an omission in one rule." The issue was filed against 9663f9bf, before it.

So this does not drop the numbers

That was the issue's preferred option, and it would remove the subject of a guard that works. It also costs more than the issue estimated. "One sed over the file plus the anchors in the index" misses the prose:

'section N' references in TESTS.md      30
  naming VACUITY_MODES.md's sections    27   unaffected
  naming THIS document's sections        3   would break

Those three need individual judgement — they have to be classified, not counted, and my first pass at classifying them was wrong in both directions before I read the enclosing section of each.

What is still open, and I shipped it

A test file whose section is written as an unnumbered ### instead of a numbered ## is invisible to every arm:

not in the numbering	so 1..N with no gap never sees it
not in the contents	so the link arms never see it
still NAMED	so the coverage arm is satisfied

test_iceberg_fdw.py went in that way in #1057 and sat undetected until this arm was written. Planted as a mutation against main, nothing catches it — 36 passed.

It is also the cheaper failure. The collision needs two PRs in flight; this needs one person writing a heading at the wrong level.

test files on disk             33
with a NUMBERED ## section     32
with an unnumbered ### section  1   <- test_iceberg_fdw.py, the defect

The rule was already true everywhere else, which is why it can be asserted rather than declared as a goal.

The change

test_iceberg_fdw.py becomes section 37 where it already sits in the body; test_hilbert_cluster.py moves to 38. Body order and numbering must agree — the #1023 arm checks inversions as well as gaps, so inserting anywhere else reddens it.

The new arm asserts in both directions, so a section cannot outlive the file it documents.

Removal proof

CONTROL                                     37 passed
M1  a section demoted to an unnumbered ###   1 failed   <- the new arm, alone
M2  a DUPLICATE section number               2 failed   <- #1023's arm, still working
TESTS.md restored to 70a78a69

M2 is in the proof deliberately. A change that found an existing guard working must not weaken it, and the only way to say that is to re-run its mutation afterwards.

guard leg  341 passed, 870 checks, 0 fail

guard_tests 340 → 341 by collection.

One note on process

I twice lost uncommitted work in this change to git checkout -- <path> inside a mutation script's restore — the file goes back to HEAD, not to the pre-mutation state. Both times the work was re-done from the scripts that produced it. It is the third instance of that trap for me this week and the fix is the same each time: commit before mutating.

🤖 Generated with Claude Code

https://claude.ai/code/session_012RSw4qMHS7ByE7PY8Ns4cs

[jdatcmd]

Approving at f80bb8a, 14/14. The arm is load-bearing and I verified it by mutation rather than by reading.

The arm catches the defect, in every construction I tried

CONTROL                                            37 passed
M1  numbered section demoted to unnumbered ###      3 failed
M1' demoted AND its contents entry removed          2 failed

Both times test_every_test_file_has_a_NUMBERED_section_of_its_own is among them, which is the claim.

I did not reproduce your "1 failed, the new arm alone", and I think that is my mutation rather than your result. Demoting a section that is already numbered and already in the contents also breaks the link arm and the numbering arm, because it orphans a contents entry and leaves a gap. Removing the contents entry too takes it to 2, still with a gap. Getting to 1 needs the contents renumbered so the document is internally consistent — which is exactly what an unnumbered arrival looks like, and what test_iceberg_fdw.py actually was in #1057.

Worth stating because the distinction is the finding: a section that was never numbered is invisible to all three arms; a section demoted from numbered is caught by all three. Your defect was the first kind, which is why nothing caught it for a whole PR cycle. The regression direction was already covered.

What makes the arm sound rather than merely present

• expect.at_least(len(files), 20) — a vacuity premise, so an emptied glob cannot pass it. That is the failure mode this repo keeps paying for, and it is guarded here rather than assumed.
• Both directions: a file with no numbered section, and a numbered section naming a file that no longer exists. A one-way list rots into a permanent exemption.
• The docstring records the population measured before the arm was written — 33 files, 32 numbered, one not, and that one was the live defect. "The rule was already true everywhere else, which is why it can be asserted rather than declared as a goal" is the right justification and it is rare to see it written down.

On not doing what the issue asked

Declining to drop the numbers is the right call, and the reason is better than the issue's own: dropping them removes the subject of a guard that works. Your classification of the three section N prose references — 30 total, 27 naming VACUITY_MODES.md, 3 naming this document — is the measurement the issue's "cheapest first" ordering rested on and did not have. That a 60-character window got it wrong in both directions is the part I would keep.

[jdatcmd]

Re-approving at 5933b21. Your resolution is better than mine — objstore takes 38 where it already sits and hilbert_cluster moves to 39, which is less edit surface for the same property.

Verified on your head rather than carried over from f80bb8a:

unnumbered ### test_*.py sections   0
test_docs_cover_the_corpus.py      37 passed, 95 checks, 0 fail
guard leg                         341 passed, 870 checks, 0 fail (tracked 341)

On the M1 difference: your reproduction is the correct one and mine was the wrong model. An arriving section is invisible to the other two arms; a demoted one is not — demoting leaves an orphaned contents entry and a numbering gap, which is why I saw three arms fire. #1057 and #1064 were both arrivals, which is exactly why neither was caught for a full cycle. Glad that is in the commit message rather than in a channel.

And the thing this PR demonstrated about itself: the defect it fixes was re-introduced by the very next PR to touch the document, by the same author, one cycle after measuring the shape. That is the argument for the arm existing, made better than any rationale could.