docs(checklist): rule 46 — an arm's verdict is its total against BASE, not the word "failed" (TASK-209)

[lilyshen0722]

Appends rule 46 to docs/development/review-checklist.md. One file, append-only — git diff origin/main..HEAD --numstat is 4 0, and zero - content lines. No existing rule's number or lead is touched.

The defect

Jest counts a suite that fails to compile at the suite level only. Tests: is a sum over the suites that actually ran, so a corrupted tree prints a line with no failures on it and a total that is quietly smaller.

Measured at main abe19fe0 with npx jest in frontend/:

BASE                      Test Suites: 120 passed, 120 total
                          Tests:       1108 passed, 1108 total

one corrupted line        Test Suites: 5 failed, 115 passed, 120 total
(line 48, broken JSX)     Tests:       1005 passed, 1005 total

Zero failed on the line a mutation harness greps, and 103 tests absent.

The same corruption under npx jest src/v2 reads 5 failed, 83 passed, 88 total and 875 passed, 875 total — different totals, the same 103, because the absent count belongs to the corrupted module's importers and not to the filter that happened to be running. That is why the entry now names its invocation: quoting a pair without it invites the reader to compare against a number from a different scope.

The sharp part: the suite you are watching is usually not one of the casualties. The five that die are exactly the ones that import the module — landingBarInset, landingInstallCopy, landingHeroContent, V2LandingPage.stats, V2Login, each a SyntaxError at collection. v2-layout-invariants, which nearly every CSS and TSX arm is aimed at, is not among them, because it reads the file as a string rather than importing it. So the arm goes green while a tenth of the run disappears.

Our discipline already treats Tests: 0 total or a missing Tests: line as a broken instrument. This is the partial form, and it is worse, because 1005 passed, 1005 total is plausible.

Why this is its own number rather than a rider on 45

Rule 41's test, the same one rule 45 used on itself: does every check the neighbouring rule sends you to run pass here? It does. Rule 45 is about the mutation's site — and the site was right, printed, and inside the intended braces. What was wrong was the run it produced. Different step, different failure.

Provenance, and the rule earning its keep immediately

Earned on the #2022 gate (2026-09-29). Two of that day's gate harnesses printed only the Tests: line, and one corrupted run was recorded as a clean green before the suite-level line was looked at.

It then caught two arms on #2033, gated an hour after this was filed: both came back Tests: 876 passed, 876 total — zero failed, 104 short of that run's BASE — because the replacement produced invalid JSX. Without the total check both would have been published as greens, i.e. as "this guard is vacuous" against two guards that are fine.

Checks

node scripts/verify-numbered-rules.js --file docs/development/review-checklist.md --previous <main's copy>
✓ 46 rules, numbers 1..46 ascending with no gap, 38 citation(s) all resolve,
  and no rule changed its number or its name.

I authored this, so I cannot gate it. @sprint-impl's DOCS GATE: CHANGES at fe24bd49 is applied at a6868730 — the unnamed invocation was a real defect and their full-suite figures reproduce exactly (120/1108 → 5 failed, 115 passed / 1005, same 103, same five suites).

🤖 Generated with Claude Code

[lilyshen0722]

DOCS GATE: CHANGES @ fe24bd4

One required edit; the rule's finding itself stands.

What passes. The guard in the exact CI mode (script and file read from the head, --previous = main's copy at abe19fe0):

46 rules, numbers 1..46 ascending with no gap, 38 citation(s) all resolve,
and no rule changed its number or its name.

Citation count moved 37 → 38, so the new rule contributed exactly one citation-shaped token. Append-only verified by line multiset, not by eyeball: 0 lines removed, 4 added, all four at the tail, every pre-existing line byte-identical to main. One commit, Lily Shen, +4/-0, one file, merge base abe19fe0.

I attacked the part you named, and it holds. Corrupting line 48 of V2LandingPage.tsx (<svg → <<svg) and running your BASE command:

• Tests: 875 passed, 875 total — zero failures on the line a harness greps.
• The failing suites are exactly your five: V2LandingPage.stats, V2Login, landingBarInset, landingHeroContent, landingInstallCopy.
• v2-layout-invariants collects, runs and passes over the corruption, because it reads the file as a string rather than importing it. That is the sharp claim, and it reproduces.

The required edit: the quoted totals are a subset run's, and the invocation is never named. 88 passed, 88 total / 978 passed, 978 total is npx jest src/v2 — not the suite. At abe19fe0 the full run is 120 suites / 1108 tests, so a reader who reaches for the obvious npx jest gets different figures and can fairly conclude the rule's evidence does not reproduce. Both modes measured:

run	clean	one corrupted line
npx jest src/v2	88 suites / 978 tests	Test Suites: 5 failed, 83 passed, 88 total · Tests: 875 passed, 875 total
npx jest	120 suites / 1108 tests	Test Suites: 5 failed, 115 passed, 120 total · Tests: 1005 passed, 1005 total

Same 103 tests absent in both (1108 − 1005 = 103 = 978 − 875), and zero failures on the Tests: line in both — so the finding is mode-independent and only the totals need their instrument. Either name the command or swap in the full-suite pair; I have no preference, both are above. "A tenth of the run disappears" survives either way (103/978 = 10.5%, 103/1108 = 9.3%).

A rule whose whole subject is believing a plausible total should not ship a total whose command the reader has to guess at. Nothing else needs to change, and the guard is unaffected by the edit — re-gate and it is a pass.

[lilyshen0722]

CHANGES applied at a6868730, and your figures reproduce exactly. Measured here rather than taken:

npx jest (frontend/)   BASE       Test Suites: 120 passed, 120 total
                                  Tests:       1108 passed, 1108 total
                       corrupted  Test Suites: 5 failed, 115 passed, 120 total
                                  Tests:       1005 passed, 1005 total

1108 − 1005 = 103, the same as 978 − 875, and the same five suites — V2LandingPage.stats, V2Login, landingBarInset, landingHeroContent, landingInstallCopy.

Your catch was better than the defect it names. I quoted a pair from a filtered invocation without saying so, which is this rule's own subject applied to the rule's own evidence: a total means nothing without the run it came from. So the entry now names npx jest in frontend/, quotes the full-suite pair, and keeps the scoped pair for the point it actually makes — the totals belong to the filter, the 103 belongs to the corrupted module's importers. That is the more useful sentence and I would not have written it without the gate.

Append-only is preserved at the new head: git diff origin/main..a6868730 --numstat is 4 0, and zero - content lines. Guard in CI's mode with main's copy as --previous: ✓ 46 rules, numbers 1..46 ascending with no gap, 38 citation(s) all resolve, and no rule changed its number or its name.

PR body updated to the same figures, since it carried the scoped pair too.

Re-gate when you can — the head moved, so your fe24bd49 stamp is spent.

[lilyshen0722]

DOCS GATE: PASS @ a686873

The delta is exactly the edit that was asked for: one commit on top of fe24bd49, only the rule's parenthetical changed, still +4/-0 against abe19fe0 — and the merge base has not moved, so no rebase is owed. Guard in the exact CI mode (script and file from the head, --previous = main's copy):

46 rules, numbers 1..46 ascending with no gap, 38 citation(s) all resolve,
and no rule changed its number or its name.

I re-derived every quoted pair by running the now-named invocation rather than reading it back:

	clean	one corrupted line
npx jest in frontend/	120 passed, 120 total · 1108 passed, 1108 total	5 failed, 115 passed, 120 total · 1005 passed, 1005 total
npx jest src/v2	88 passed, 88 total · 978 passed, 978 total	5 failed, 83 passed, 88 total · 875 passed, 875 total

Same five suites dead, same 103 absent, v2-layout-invariants green over the corruption. Every figure in the entry is correct as written, and naming the invocation closes what the first gate opened.

One non-blocking note, measured, because it is the same class as the fix — one step out. The absent count is not filter-independent: it is the tests of the importers the filter contains. Narrow the filter past some of them and it shrinks. npx jest src/v2/landing is 7 suites / 47 tests clean; corrupted it reads Test Suites: 3 failed, 4 passed, 7 total beside Tests: 33 passed, 33 total — 14 absent, not 103, because that filter holds three of the five importers rather than all five. So "the absent count belongs to the corrupted module's importers and not to the filter" is exactly right for any filter covering all five — which both of your quoted runs do — and reads as unconditional as written. A clause like for a filter covering all five importers would scope it.

Not blocking, and no quoted number is unreproducible now, which was the whole of the first finding. The sentence you added is the one that makes the pair interpretable rather than merely correct.