test: the corpus document states no totals (#908)

test/pytest/TESTS.md

@@ -4,9 +4,31 @@ Reference for anyone reading, running, or adding to `test/pytest/`. The design a
 the decisions behind the harness are in `design/ISSUE_432_PYTEST_HARNESS.md`. This
 file covers the tests themselves.
 
-**135 tests in 11 files.** One hundred and twenty of them test the harness rather than the
-product, and they come first, because a harness that can report a false green makes
-every other result in this directory worthless.
+This file names every test in the corpus and says what each one asserts. **It
+states no totals**, and that is deliberate (#908).
+
+A count here was a claim whose correct value is a function of the MERGE rather
+than of either branch, so it collided on essentially every rebase that touched
+the corpus -- ten times in one day, and both sides wrong every time.
+
+It was also redundant, and the argument needs THREE arms rather than the two it
+was first written with. `test_every_file_and_test_is_named_in_the_document`
+requires every test on disk to be named here;
+`test_a_documented_test_that_does_not_exist_is_named` requires every name here to
+exist on disk; and `test_no_test_name_is_defined_twice_in_the_corpus` requires
+those names to be UNIQUE. The first two give equality of the two NAME SETS, which
+is not equality of DEFINITION COUNTS -- two files defining one name leave both
+arms green while the counts differ (@jdatcmd). With uniqueness as well, a count
+over this document is a count over the corpus, and a number added nothing except
+a thing to get wrong.
+
+The harness prints the counts on every run, where they cannot go stale. Note that
+a count of test FUNCTIONS is not the count of collected ITEMS -- parametrized
+tests expand -- so `--pgc-expect-tests` takes the run's own collected count and is
+documented in README.md beside the invocation that uses it.
+
+The harness tests come first, because a harness that can report a false green
+makes every other result in this directory worthless.
 
 That ratio is not an accident of taste. Two of those files exist because a reviewer
 neutered the guards one at a time and found most of them deletable with the suite
@@ -130,8 +152,6 @@ one of those eight measurements exited 0.
 | `test_an_unrunnable_test_names_its_reason_and_its_detail` | the `UNRUN` line carries reason and detail | nothing was printed at all |
 | `test_a_real_failure_outranks_an_unrunnable_test` | a run with both exits 1, not 67 | — |
 | `test_a_run_with_nothing_unrunnable_still_exits_zero` | **control**: a green run is untouched | — |
-| `test_layer_rejects_an_absence_assertion_over_an_empty_plan` | an absence claim over `[]` is refused | it passes: nothing is there to find |
-| `test_layer_allows_an_absence_assertion_over_a_real_plan` | **control**: `absent=True` still works on a plan that arrived | — |
 | `test_layer_rejects_psycopgs_no_count_sentinel` | `rowcount` of `-1` is refused | `-1` and `1` are both numbers, so `num` compares them happily |
 | `test_layer_rejects_a_broad_except_in_a_test_file` | a broad `except` is uncollectable | it was forbidden in a COMMENT, which enforces nothing |
 
@@ -642,6 +662,28 @@ observed variants are closed", not "the function is now infallible".
 
 ## 6. test_docs_cover_the_corpus.py: this document, checked
 
+**THE SWEEP GOES BOTH WAYS NOW (#908).** `undocumented()` computes tests on disk
+the document fails to name; `documented_but_absent()` computes the reverse. Only
+the second catches a test that is DELETED or RENAMED while its entry survives —
+until it existed, that case was held by the totals line alone, and the totals line
+is a merge target whose correct value is a function of the merge. Removing it
+while this direction was uncovered would have retired a check silently.
+
+**A BACKTICKED TEST NAME IS A CLAIM THAT IT EXISTS.** That is the rule the arm
+enforces, and it has a consequence for prose: a name that is gone is written
+WITHOUT backticks, because backticking it would assert it is still there. This
+paragraph is the first place that bit — the arm reddened on my own description of
+the defect.
+
+It found two on the corpus that shipped. The rows named
+test_layer_rejects_an_absence_assertion_over_an_empty_plan and a control beside
+it, in section 3, and neither had ever been written. The work is real and lives in
+`test_guards_pinned.py` as
+`test_plan_marker_refuses_an_absence_claim_over_an_empty_plan`, documented
+correctly in section 4 — so two rows claimed coverage under names never written,
+and every other arm here passed over them.
+
+
 The file you are reading is checked mechanically, because it went stale inside a
 single rework and nothing noticed. The corpus grew from 25 tests in three files to
 54 in five; the two new files, 29 tests, were named nowhere here, and the header
@@ -668,7 +710,13 @@ many times.
 | --- | --- |
 | `test_the_sweep_finds_the_corpus_rather_than_an_empty_glob` | **premise**: the sweep saw files and tests, so "nothing missing" means something |
 | `test_every_file_and_test_is_named_in_the_document` | every file and test is named here, and a failure says WHICH |
-| `test_the_stated_totals_are_the_totals_on_disk` | the bold totals line matches the corpus |
+| `test_a_documented_test_that_does_not_exist_is_named` | the reverse sweep: the document may not claim a test the corpus lacks |
+| `test_a_document_naming_a_test_that_was_deleted_is_caught` | **removal proof**: the shape the real defect had, on a fixture |
+| `test_a_documented_file_that_does_not_exist_is_caught` | a whole file can go the same way, which is how a rename shows up |
+| `test_the_document_states_no_totals_for_a_merge_to_get_wrong` | the totals line must not come back; its absence is a decision, not an accident |
+| `test_no_test_name_is_defined_twice_in_the_corpus` | the premise the set-equality argument needs: names must be unique |
+| `test_a_name_defined_in_two_files_is_caught` | **removal proof**: the shape that defeats the argument, on a fixture |
+| `test_the_corpus_counts_are_reported_rather_than_written` | the counts move to the run's output, where they cannot go stale |
 | `test_a_fully_documented_corpus_reports_nothing_missing` | **control**: no false positive on a complete document |
 | `test_an_undocumented_test_is_named_rather_than_passed_over` | the exact shape that shipped: file named, one test inside it not |
 | `test_the_mode_inventory_states_its_own_totals_correctly` | the totals in VACUITY_MODES.md section 1a are the modes on disk |

test/pytest/test_docs_cover_the_corpus.py

@@ -56,10 +56,15 @@ def undocumented(directory, doc_path):
     return sorted(missing)
 
 
+def stated_totals_in(text):
+    """-> (tests, files) the TEXT claims, or None if it states none."""
+    m = TOTALS.search(text)
+    return (int(m.group(1)), int(m.group(2))) if m else None
+
+
 def stated_totals(doc_path):
     """-> (tests, files) the document claims, or None if it states none."""
-    m = TOTALS.search(pathlib.Path(doc_path).read_text())
-    return (int(m.group(1)), int(m.group(2))) if m else None
+    return stated_totals_in(pathlib.Path(doc_path).read_text())
 
 
 def _fixture(tmp_path, doc_body):
@@ -95,21 +100,184 @@ def test_every_file_and_test_is_named_in_the_document(expect):
                 "every test file and every test in the corpus is named in TESTS.md")
 
 
-def test_the_stated_totals_are_the_totals_on_disk(expect):
-    """Neither arm above would catch a wrong count: a document can name every test
-    and still miscount them, which is exactly what the stale header did."""
-    stated = stated_totals(DOC)
-    expect.text(repr(stated is not None), "True",
-                "TESTS.md states its totals in a form that can be read back")
+def documented_but_absent(directory, doc):
+    """Names the DOCUMENT claims that the corpus does not have.
+
+    THE SWEEP ABOVE GOES ONE WAY ONLY. `undocumented()` computes tests on disk
+    that the document fails to name, and nothing computed the reverse. So a test
+    DELETED or RENAMED while its entry survived was caught by the totals line and
+    by nothing else -- and the totals line is a merge target whose correct value
+    is a function of the merge, so it is the half most likely to be removed
+    (#908). Removing it while this direction was uncovered would have retired a
+    check silently, which is the move this file exists to prevent.
+
+    Driven against the real functions, on the corpus that shipped:
+
+        on disk           (1, 1)      # test_one.py holds test_alpha
+        document states   (2, 1)      # "test_one.py: test_alpha and test_beta"
+
+        the NAMING arm  : []          <- says nothing is wrong
+        the TOTALS arm  : DISAGREE    <- the only arm that reddens
+
+    A BACKTICKED NAME, not any occurrence. The document discusses fixtures and
+    hypothetical tests in prose, and a bare-word sweep would report those as
+    missing. Backticks are how this document already marks a real identifier, and
+    the false-positive budget over the corpus was measured before this was
+    written rather than after: 127 backticked names, 2 of which were genuinely
+    absent, and both were real defects rather than noise.
+    """
+    found = corpus_tests(directory)
+    on_disk_fns = {n for names in found.values() for n in names}
+    on_disk_files = set(found)
+    named = set(re.findall(r"`(test_[A-Za-z0-9_]*(?:\.py)?)`", doc.read_text()))
+    # A SORTED LIST, like undocumented() beside it. It returned a preformatted
+    # "[n: a b c]" string first, copying the bash twin's shape rather than its
+    # Python neighbour's, and the two-return-types-one-concept split immediately
+    # cost something real: a check written against it read `x in ("[]", "")`,
+    # which is False for an empty LIST, and reported a defect as unreproducible.
+    return sorted({n for n in named if n.endswith(".py")} - on_disk_files) \
+         + sorted({n for n in named if not n.endswith(".py")} - on_disk_fns)
+
+
+def test_a_documented_test_that_does_not_exist_is_named(expect):
+    """The document must not claim a test the corpus does not have.
+
+    It did. `test_layer_rejects_an_absence_assertion_over_an_empty_plan` and its
+    control `..._allows_an_absence_assertion_over_a_real_plan` were named in the
+    test_layer.py section and existed nowhere: the work is real but lives in
+    test_guards_pinned.py as `test_plan_marker_refuses_an_absence_claim_over_an_empty_plan`,
+    and is documented correctly there. Two rows claimed coverage under names that
+    had never been written, and every other arm in this file passed over them --
+    which is the point.
+    """
+    expect.text(", ".join(documented_but_absent(HERE, DOC)) or "none", "none",
+                "every test the document names exists in the corpus")
+
+
+def test_a_document_naming_a_test_that_was_deleted_is_caught(tmp_path, expect):
+    """The removal proof, on a fixture: the shape the real defect had.
+
+    Without this the arm above passes on a healthy tree, which is exactly what an
+    arm that computes nothing also does.
+    """
+    (tmp_path / "test_one.py").write_text("def test_alpha(expect):\n    pass\n")
+    doc = tmp_path / "DOC.md"
+    doc.write_text("**1 tests in 1 files.**\n`test_one.py`: `test_alpha` and `test_beta`\n")
+    expect.text(", ".join(documented_but_absent(tmp_path, doc)), "test_beta",
+                "a documented test that does not exist is named, not passed over")
+    doc.write_text("**1 tests in 1 files.**\n`test_one.py`: `test_alpha`\n")
+    expect.text(", ".join(documented_but_absent(tmp_path, doc)) or "none", "none",
+                "control: a document naming only what exists is clean")
+
+
+def test_a_documented_file_that_does_not_exist_is_caught(tmp_path, expect):
+    """A whole file can go the same way, and it is how a rename usually shows up."""
+    (tmp_path / "test_one.py").write_text("def test_alpha(expect):\n    pass\n")
+    doc = tmp_path / "DOC.md"
+    doc.write_text("`test_one.py` and `test_gone.py`: `test_alpha`\n")
+    expect.text(", ".join(documented_but_absent(tmp_path, doc)), "test_gone.py",
+                "a documented file that does not exist is named")
+
+
+def test_no_test_name_is_defined_twice_in_the_corpus(expect):
+    """The premise the set-equality argument needs, and it was missing (@jdatcmd).
+
+    #919 argues that removing the totals line costs nothing because the two
+    sweeps give set EQUALITY between the document and the corpus. That is true of
+    NAMES and it is not true of DEFINITION COUNTS, which is what a total counts:
+
+        two files defining test_shared_shape
+            definitions on disk           2
+            distinct names                1
+            undocumented()                clean
+            documented_but_absent()       clean
+            -> BOTH ARMS GREEN, and the counts differ
+
+    Measured, not argued. It cannot happen today -- 139 definitions against 139
+    distinct names, zero duplicates -- so the conclusion was true in fact but not
+    by construction, which is the difference between an argument and a guard.
+
+    IT CLOSES SOMETHING REAL BEYOND THE ARGUMENT. `undocumented()` asks whether a
+    name appears in the document at all, so a test defined TWICE and documented
+    ONCE reads as fully covered. The second definition is invisible to every arm
+    here, and pytest runs both.
+    """
     found = corpus_tests(HERE)
-    expect.text(repr(stated), repr((sum(len(v) for v in found.values()), len(found))),
-                "and the totals it states are the totals on disk")
+    names = [n for tests in found.values() for n in tests]
+    expect.at_least(len(names), 20, "premise: the corpus was found")
+    dupes = sorted({n for n in names if names.count(n) > 1})
+    expect.text(", ".join(dupes) or "none", "none",
+                "no test name is defined twice in the corpus")
+    expect.num(len(names), len(set(names)),
+               "so definitions and distinct names are the same count")
+
+
+def test_a_name_defined_in_two_files_is_caught(tmp_path, expect):
+    """The removal proof, and the exact shape that defeats the equality argument."""
+    (tmp_path / "test_a.py").write_text("def test_shared_shape(expect):\n    pass\n")
+    (tmp_path / "test_b.py").write_text("def test_shared_shape(expect):\n    pass\n")
+    found = corpus_tests(tmp_path)
+    names = [n for tests in found.values() for n in tests]
+    expect.num(len(names), 2, "premise: both definitions were seen")
+    expect.num(len(set(names)), 1, "and they share one name")
+    expect.text(", ".join(sorted({n for n in names if names.count(n) > 1})),
+                "test_shared_shape",
+                "a name defined in two files is named, not passed over")
+
+
+def test_the_document_states_no_totals_for_a_merge_to_get_wrong(expect):
+    """TESTS.md must NOT carry a totals line (#908, step 2).
+
+    It used to, and `selftest/350` compared it against the corpus, which is what
+    made it a claim rather than decoration. The problem was never the check: it
+    was that the number was WRITTEN rather than DERIVED, and its correct value is
+    a function of the MERGE rather than of either branch. It collided on
+    essentially every rebase touching the corpus -- ten times in one day, both
+    sides wrong every time, so there was no side to pick.
+
+    REMOVING IT COSTS NOTHING, and that is provable rather than hopeful. The two
+    sweeps together are strictly stronger than any count:
+
+        test_every_file_and_test_is_named_in_the_document
+            every test on disk is named here          (disk  subset of  document)
+        test_a_documented_test_that_does_not_exist_is_named
+            every name here exists on disk            (document  subset of  disk)
+
+    Two subsets in opposite directions is set EQUALITY, so the documented set and
+    the corpus are the same set, and any count over one equals the count over the
+    other. A stated total was a derived value written by hand.
+
+    WHY THIS IS AN ARM AND NOT JUST A DELETION. Nothing stops the next person
+    adding the sentence back -- it reads like an improvement. This arm is what
+    makes its absence a decision rather than an accident, and `stated_totals`
+    stays for it: the reader still has to work, or "no totals line" would be
+    indistinguishable from "cannot find one".
+    """
+    expect.text(repr(stated_totals(DOC)), "None",
+                "TESTS.md states no totals line for a merge to get wrong")
 
+    # And the reader that reports it must still be able to FIND one, or the arm
+    # above passes because the parser is broken rather than because the line is
+    # gone -- the exact shape this corpus exists to refuse.
+    expect.text(repr(stated_totals_in("**7 tests in 3 files.** and prose")),
+                "(7, 3)",
+                "premise: the reader still finds a totals line when one is there")
+
+
+def test_the_corpus_counts_are_reported_rather_than_written(expect):
+    """The counts do not vanish; they move to where they cannot go stale.
+
+    A number nobody maintains is better than a wrong one, but a number nobody can
+    SEE is worse than both. The harness prints them every run, computed from the
+    corpus, so a reader gets the same information without the document asserting
+    anything.
+    """
+    found = corpus_tests(HERE)
+    total = sum(len(v) for v in found.values())
+    expect.at_least(total, 20, "premise: the corpus was found, so a count means something")
+    expect.num(len(found), len({f for f in found}), "each file counted once")
+    print(f"\nCORPUS: {total} test functions in {len(found)} files")
 
-# ---------------------------------------------------------------------------
-# And the guard must be able to FAIL. Everything above passes on a healthy tree,
-# which is exactly what a guard that does nothing also does.
-# ---------------------------------------------------------------------------
 
 def test_a_fully_documented_corpus_reports_nothing_missing(tmp_path, expect):
     """Control. A guard with a bad false-positive rate gets switched off, and then