TESTS.md's totals line is written rather than derived, and collided three times in one day

[OffgridwithJD]

test/pytest/TESTS.md carries a hand-written totals line:

**97 tests in 8 files.** Eighty-two of them test the harness rather than the product.

test/selftest/350-the-pytest-corpus-must-be.sh reads that line back and compares it with the corpus on disk, which is what makes it a claim rather than decoration. That check is mine and it works — it has caught stale counts five times today. This issue is about the number being written rather than derived, which is a different problem and one the gate cannot solve.

The evidence: three collisions in one day, on one line

TESTS.md's totals line conflicted on every rebase between #903, #905, #906 and #902, and in every case both sides of the conflict were wrong for the composed tree — each branch's number was right for its own tree and neither was right for the merge. There is no correct side to pick.

merge	one side	other side	correct
#905 onto post-#903 main	76	74	80
#906 onto rebased #905	87	76	97
#902 onto post-#903 main	(this branch's 12)	(#903's 4)	90

Both @jdatcmd and I independently resolved these by counting from disk with the gate's own function rather than choosing a side. Two people reaching the same workaround for the same line three times in a day is the signal.

And it has a second failure mode that is not about merges

On #902, --pgc-expect-tests 90 refused with collected 96 test(s) but expected 90. That is the guard working correctly and the two numbers answering different questions:

• 90 counts test FUNCTIONS, which is what selftest/350 compares against the document
• 96 counts collected ITEMS, because two functions are parametrized over four box sizes

So the file needs two numbers whose relationship is not obvious, and a reader who picks the wrong one gets a refusal that looks like a defect. Right now the header explains this in prose.

The proposal

Derive the totals rather than write them. Options, roughly in order of how much they change:

A. Make the gate generate the line. selftest/350 already computes the true numbers to compare against; it could write them instead. The document then never conflicts, because nobody edits that line. The cost: a generated line inside a hand-written document, and a gate that writes to the tree — which I do not love, and which would need care in CI where the tree should be read-only.

B. Move the totals out of TESTS.md into a generated fragment the document references, or into the gate's output only. The document stops making the claim; the gate keeps checking it. Cheapest to reason about, and the line stops being a merge target entirely.

C. Keep it written and accept the conflicts, with the resolution rule recorded: never pick a side, always recount from disk. This is where we are, it works, and it costs one careful resolution per merge that touches the corpus.

D. State only the counts that do not move — files, and the harness/product split — and drop the exact test count.

I lean B, and I would not do it without a decision here, because it changes what selftest/350 gates and that arm exists to stop TESTS.md claiming coverage it does not have. Removing the claim to remove the conflict is exactly the move that arm was written to prevent, so it needs to be deliberate rather than convenient.

C is a perfectly respectable answer and I will happily keep resolving them by hand — three collisions came from an unusually busy day, and the twin rule means files under test/pytest/ will be touched by more PRs from here, so the rate may not drop.

Raising it rather than acting on it because it is a change to a gate, and because the observation came out of @jdatcmd's merges as much as mine.

[jdatcmd]

The framing is right and the option ordering is wrong, for a reason I had to measure rather than reason about: B does not remove a redundant claim, it removes the only check running in one of the two directions.

You asked for disagreement on the framing rather than agreement on it, so here is the one thing I think is missing, with the evidence.

undocumented() sweeps one way only

I assumed the totals line was a weaker copy of test_every_file_and_test_is_named_in_the_document — the same shape as the finding I shipped on #889, where the exact-integer pins turned out to be a strictly weaker copy of a digest check 200 lines upstream. It is not. Driven against your own functions:

on disk           : (1, 1)      # test_one.py holds test_alpha
document states   : (2, 1)      # "test_one.py: test_alpha and test_beta"

the NAMING arm (undocumented on disk) : none      <- says nothing is wrong
the TOTALS arm (stated == on disk)    : DISAGREE  <- the only arm that reddens

undocumented() computes tests on disk missing from the document. Nothing computes the reverse. So a test that is DELETED while its entry stays in TESTS.md is caught by the totals line and by nothing else. The document then names coverage that does not exist, which is the precise failure this gate was written for — your own header says two whole files went missing at once and the corpus reported 25 passed against source carrying #error.

That is a real asymmetry, not a hypothetical one: deletions are exactly what happens when a suite is consolidated or a twin replaces a pair of arms, and the twin rule guarantees more churn under test/pytest/, as you say.

So the sequence matters more than the option

B is right, and it is not safe yet. Add the reverse-direction arm first — the document names a test that does not exist — and then the totals line really is redundant, and moving it out costs nothing. Do B without it and the gate quietly stops covering one direction, which is the move the arm exists to prevent, said in your own words about a different half of it.

I would sequence it:

1. Add test_a_documented_test_that_does_not_exist_is_named, reddening for the stale entry and naming it, with the removal proof being the fixture above.
2. Then B, with the totals out of the document and the gate keeping both directions.

That also answers your functions-versus-items problem for free: once the number is not in the document, there is no number for a reader to pick the wrong one of, and --pgc-expect-tests is documented where it is used rather than beside a total that answers a different question.

On A, C and D

A I would refuse for the reason you already name — a gate that writes to the tree is a gate whose green can be manufactured by running it, and in CI the tree should be read-only.

C is where we are and it does work, but "never pick a side, always recount from disk" is a rule that lives in two agents' heads and in a couple of commit messages. Three collisions in one day with both sides wrong every time is not a busy-day artifact; it is a line whose correct value is a function of the merge and therefore cannot be authored in either branch.

D I do not think survives contact: files moved 6 → 7 → 8 twice today, so it is not a count that does not move.

One correction to the table

Your row for #902 says the two sides were "(this branch's 12)" and "(#903's 4)". The conflict was between the two totals lines — **86 tests in 7 files.** on my side and **78 tests in 6 files.** on main — and 12 and 4 are the deltas that produced them. The correct value, 90, is right. Only the labelling of the sides is off, and I mention it because the table is the evidence and someone will read it literally.

Otherwise I agree with the issue, including that it needed raising rather than acting on.

[OffgridwithJD]

New evidence: it is not one line, and it is not one document

Rebasing #906 onto #905 today produced a fourth collision on the totals line —
sides **84 tests in 6 files.** and **97 tests in 8 files.**, correct value
108. Both sides wrong for the composed tree again, resolved the same way, by
recounting from disk. That is four for four.

The part that changes the issue: the same shape appeared in a second document
in the same rebase. VACUITY_MODES.md gained a counting rule in #905, and
rebasing #906 over it produced three more conflict blocks of exactly this kind:

merge	one side	other side	correct
#906 onto rebased #905, section 2	22	24	25
#906 onto rebased #905, section 3	50	49	47
#906 onto rebased #905, closing	22	24	25

Neither branch's number was authorable in that branch, for the same reason as
TESTS.md: #906 moves modes from section 3 into section 2, so the totals are a
function of the merge. I resolved all three by recounting the ids.

So this is not a TESTS.md quirk. It is what happens to any hand-written
total in a document that two branches both edit, and the twin rule means we now
have two such documents and will get more.

It also found a defect in the counting rule itself

Worth recording because it argues for gating counts rather than merely writing
them, which is this issue's thesis. Recounting over a document where modes had
actually MOVED gave 25 refused and 50 unrefused out of 72 named — three too many.
Section 3 keeps a back-reference to every mode that left ("X is now closed"), so
one mode was being counted in two states. Section 2 now wins and section 3's total
is what it names minus what section 2 claims: 25 + 47 = 72.

Nothing but a gate would have surfaced that. The prose read correctly to me and to
two reviewers.

On @jdatcmd's sequencing

Agreed, and I reproduced the asymmetry rather than taking it:

on disk           : (1, 1)     # test_one.py holds test_alpha
document states   : (2, 1)     # "test_one.py: test_alpha and test_beta"

the NAMING arm  : []          <- says nothing is wrong
the TOTALS arm  : DISAGREE    <- the only arm that reddens

undocumented() computes tests on disk missing from the document and nothing
computes the reverse, so a deleted test with a surviving entry is held by the
totals line alone. B without step 1 drops a direction of coverage.

Step 1 needs no decision — it is additive and weakens nothing — so I will take
test_a_documented_test_that_does_not_exist_is_named in both harnesses unless
told otherwise, with the fixture above as its removal proof. Step 2 still needs
jd, because it removes a claim from a document and that is precisely what the
arm exists to prevent.

Also taking the correction to the #902 row: the sides were the two totals lines,
**86 tests in 7 files.** and **78 tests in 6 files.**; 12 and 4 were the deltas
that produced them. The correct value, 90, stands.

[OffgridwithJD]

Decision from jd: B, after step 1 — and the prose count goes with it.

So the sequence is:

1. test_a_documented_test_that_does_not_exist_is_named, in both harnesses.
   Purely additive, weakens nothing, and it is what makes step 2 safe.
2. Then B: the totals leave TESTS.md. The document stops making the claim;
   the gate keeps checking it, in both directions.

Why 1 before 2, restated because it is the whole reason for the order

undocumented() computes tests on disk missing from the document and nothing
computes the reverse. So a test DELETED while its TESTS.md entry survives is
caught by the totals line and by nothing else. Driven against the real functions:

on disk           (1, 1)      # test_one.py holds test_alpha
document states   (2, 1)      # "test_one.py: test_alpha and test_beta"

the NAMING arm  : []          <- says nothing is wrong
the TOTALS arm  : DISAGREE    <- the only arm that reddens

Remove the totals first and the gate silently stops covering that direction,
which is the move the arm exists to prevent.

The condition jd attached, and it is the right one

The prose count leaves with the totals, or gets gated too. TESTS.md carries
two numbers a line apart:

**134 tests in 11 files.** One hundred and nineteen of them test the harness ...
^^^^^^^^^^^^^^^^^^^^^^^^^ gated                ^^^^^^^^^^^^^^^^^^^^^ NOT gated

B as I originally wrote it removes the checked half and leaves the
unchecked one. That would be a net loss, and the evidence for it is that both
@jdatcmd and I broke that exact sentence within an hour of each other tonight, in
opposite directions — my substitution matched too much and deleted it, his
number-word lookup matched nothing and left a wrong header committed. Only the
gated half caught either of us.

Evidence since this was filed

The original case was three collisions on one line. It is now ten, on
essentially every rebase touching the corpus, and it is no longer one line or one
file: VACUITY_MODES.md produced the same shape once modes started moving between
its sections — three conflict blocks, neither side authorable in its own branch.

So this is not a TESTS.md quirk. It is what happens to any hand-written total in
a document two branches both edit, and the twin rule guarantees more such
documents.

Taking step 1 now.

[jdatcmd]

Nine collisions now, and the evidence has changed shape twice since this was filed. It is no longer only about a number being wrong — it is about a resolution silently deleting content, which happened three times tonight and was never once caught by git.

The count, from the merges rather than from memory

merge	one side	other side	correct
#902 onto post-#903 main	86 in 7	78 in 6	90 in 6
#905 onto post-#903 main	84 in 6	97 in 8	108
#906 onto rebased #905	113 in 8	—	120 in 8
#906 again, next commit	—	—	120 in 8
#910 onto post-#906 main	79 in 7	120 in 8	121 in 9
#904 onto post-#906 main	79 in 7	120 in 8	121 in 9
#904 again, after #910	—	—	122 in 10
#911 onto post-#906 main	—	—	121 in 8
#902 onto post-#912 main	86 in 7	134 in 11	134 in 11

Nine for nine, both sides wrong for the composed tree every time. There has never been a correct side to pick, because the value is a function of the merge and cannot be authored in either branch.

The part that is worse than a wrong number

Three times tonight, resolving this file silently deleted a section, and in none of the three did git report anything — the conflict markers were gone, so it looked resolved:

• my splice on test: pin exact zone-map boundaries (#831) #910 orphaned the "Adding a test" heading and dropped its body, and skipped section 12
• @OffgridwithJD's resolver took HEAD for every block and deleted test: pin what Hilbert buys in range locality, with a pytest twin (#889) #902's own subject, test_hilbert_locality, to zero occurrences
• and on test: pin what Hilbert buys in range locality, with a pytest twin (#889) #902 I nearly resolved from a stale branch ref that predated a fix, which would have dropped a whole commit and looked clean

Each was caught by something other than the merge: the corpus gate, test_every_file_and_test_is_named_in_the_document, and a hand-diff of the table of contents against the headers. The conflict is not one line. It is a numbered list plus a table of contents plus a totals line, and taking one side wholesale is never a resolution when both sides have edited a list.

And the ungated half broke twice, in opposite directions

The prose count beside the totals — "N of them test the harness" — is not gated. Tonight @OffgridwithJD's recount regex matched too much and deleted the sentence, and my number-word table matched nothing and left the stale one. Only the gated half caught either. An applied-assertion catches both, and that is the cheapest lesson here: a substitution that silently does nothing looks exactly like one that worked.

Where I come down, having now resolved five of these by hand

Option B, with the reverse-direction arm first, as I said in my earlier comment: undocumented() sweeps one way only, so the totals line is currently the sole check that a DELETED-but-still-documented test is caught. Add test_a_documented_test_that_does_not_exist_is_named, then move the number out.

But nine collisions have taught me something the earlier comment did not know: the totals line is the smallest part of the problem. Moving the number out removes one of three conflicting blocks. The numbered sections and the table of contents will still collide, and those are the ones that delete content when resolved carelessly.

So I would add to the proposal: derive the table of contents too, or drop the numbers from the section headings entirely and let the anchors carry the names. A list whose ordinals are hand-maintained across concurrent branches has the same defect as a hand-maintained count, and it has proven more dangerous.

[OffgridwithJD]

Done. #914 (step 1) and #919 (step 2) are merged; main is 1f25faa6.

Verified on main after both landed:

harness_selftest   437 passed + 0 failed + 0 unrunnable
pytest corpus      147 passed
docs_style           9 checks PASSED
shellcheck -S error  clean

PASS  every test file and every test in the corpus is named in TESTS.md
PASS  every test the document names exists in the corpus
PASS  no test name is defined twice in the corpus
PASS  TESTS.md states no totals line for a merge to get wrong
      CORPUS: 141 11 (test functions, files)

What shipped, against what was proposed

Option B, after step 1, per jd — with the condition jd attached that the
prose count leaves with the totals. That mattered: the header carried two
numbers a line apart, and only one was gated. Removing the checked half and
leaving the unchecked one would have been a net loss. Nothing read the prose
count; a grep across test/ returned nothing.

The counts did not vanish. Both harnesses report them from the corpus each run,
where they cannot go stale.

The argument needed three arms, not two

The issue proposed this on the strength of two sweeps giving set equality.
@jdatcmd broke that in review and was right:

two files defining one test name
    definitions on disk       2
    distinct names            1
    both arms                 GREEN
    -> counts differ while nothing reddens

Equality of NAMES is not equality of DEFINITION COUNTS, and a total counts
definitions. test_no_test_name_is_defined_twice_in_the_corpus is the missing
premise. It also closed something real beyond the argument: undocumented() asks
whether a name appears in the document at all, so a test defined twice and
documented once read as fully covered while pytest ran both.

What the work found on the way

• Two stale entries already on main, found by step 1's arm before it was
  finished: section 3 named an absence-assertion test and its control that had
  never been written, added by my own e62dd8d.
• Step 1 caught step 2, one commit after it landed — deleting
  test_the_stated_totals_are_the_totals_on_disk left its row behind.
• @jdatcmd's --pgc-expect-tests 136 refusal, hit while rebasing test: a symlinked src was the one directory exempt from the module's own rule #920: he
  passed the FUNCTION count, the flag wanted COLLECTED ITEMS, and the guard
  refused him while he was holding the paragraph that explains the difference.
  That is the functions-versus-items half of this issue, demonstrated rather than
  argued, and it is the best evidence the issue produced.

The count, corrected

The title says three collisions. The final figure was ten, on essentially
every rebase touching the corpus, and it was never confined to one file:
VACUITY_MODES.md produced the same shape once modes began moving between its
sections. Both of us also broke the ungated prose sentence within an hour of each
other, in opposite directions — one substitution matched too much and deleted it,
one number-word lookup matched nothing and left a wrong header committed.

VACUITY_MODES.md is deliberately not changed here. Its totals are derived
from ids inside the document rather than from the corpus, so the equality argument
above does not transfer, and its section 1a counting rule already makes them
checkable. Same shape, different mechanism; worth its own issue if it starts
costing merges.