test: a test file documented outside the numbering is invisible (#1024)

CHANGELOG.md

@@ -53,6 +53,18 @@ true until the next version shipped.
   Arms in `test/objstore_userinfo.sh` beside the existing ones, with a clean-endpoint
   control in the same run, and in `test/pytest/test_objstore_endpoint_userinfo.py`
   independently.
+- A test file documented as an unnumbered `###` section was invisible to every arm
+  that checks `TESTS.md` (#1024).
+
+  Not in the numbering, so `1..N with no gap` never saw it. Not in the contents, so the
+  link arms never saw it. Still NAMED in the document, so the coverage arm was
+  satisfied. `test_iceberg_fdw.py` shipped that way in #1057 and sat undetected.
+
+  #1024's own report -- two PRs each taking the next section number -- is no longer
+  open: `test_the_contents_list_is_numbered_in_order` landed after #1023 and catches a
+  duplicate and an inversion in one rule. Planted, it reddens. This change keeps that
+  guard and closes the remaining shape, which is cheaper to hit: the collision needs two
+  PRs in flight, a heading at the wrong level needs one person.
 
 - A bash suite that unrolls a family as literals graded MISSING against the port that
   parametrises it (#1045 class 3).

test/pytest/TESTS.md

@@ -82,7 +82,9 @@ behaviour, the source of that number is named.
 - [34. test_docs_stripe_floor.py: the stripe floor is below a vector](#34-test_docs_stripe_floorpy-the-stripe-floor-is-below-a-vector)
 - [35. test_projection_privilege.py: the projection read helpers are a privilege boundary](#35-test_projection_privilegepy-the-projection-read-helpers-are-a-privilege-boundary)
 - [36. test_compare_to_bash.py: the parity tool reads the NAME](#36-test_compare_to_bashpy-the-parity-tool-reads-the-name)
-- [37. test_hilbert_cluster.py: the Hilbert clustering SQL surface](#37-test_hilbert_clusterpy-the-hilbert-clustering-sql-surface)
+- [37. test_iceberg_fdw.py: the Iceberg FDW's pruning surface](#37-test_iceberg_fdwpy-the-iceberg-fdws-pruning-surface)
+- [38. test_objstore_endpoint_userinfo.py: userinfo in an object-store endpoint](#38-test_objstore_endpoint_userinfopy-userinfo-in-an-object-store-endpoint)
+- [39. test_hilbert_cluster.py: the Hilbert clustering SQL surface](#39-test_hilbert_clusterpy-the-hilbert-clustering-sql-surface)
 
 ## 1. How to read a test in here
 
@@ -1066,6 +1068,7 @@ many times.
 | `test_an_anchor_that_strips_the_underscores_is_caught` | the exact broken link that shipped, with a control |
 | `test_every_in_document_link_in_this_directory_reaches_a_heading` | every contents-list link resolves, with a coverage premise |
 | `test_the_contents_list_is_numbered_in_order` | the contents list and the sections both count 1..N with no gap or inversion — the link arms above ask only whether a link RESOLVES, and a shuffled list resolves perfectly |
+| `test_every_test_file_has_a_NUMBERED_section_of_its_own` | a section written as an unnumbered `###` is invisible to every other arm: not in the numbering, not in the contents, and the file is still NAMED so the coverage arm is satisfied — `test_iceberg_fdw.py` shipped that way in #1057 |
 | `test_a_shuffled_contents_list_is_caught_on_a_fixture` | **removal proof**: the `29, 31, 30` shape that shipped, with a clean control and an omitted entry named apart from an inversion |
 | `test_the_next_steps_list_is_anchored_to_the_inventory` | every section 5 entry names a mode id, so the entry can be checked at all |
 | `test_no_open_next_step_names_work_the_document_calls_done` | an un-struck entry whose id reached section 2 is stale work to do |
@@ -3875,8 +3878,9 @@ the tool grades THIS tree.
 | `test_every_pair_in_the_tree_is_declared` | the declaration is asserted BOTH ways, so a new pair cannot be silently ungraded |
 | `test_the_ported_suites_in_this_tree_are_graded_one_for_one` | the standing arm: every pair in the tree, graded |
 
+## 37. test_iceberg_fdw.py: the Iceberg FDW's pruning surface
 
-### `test_objstore_endpoint_userinfo.py` -- userinfo in an object-store endpoint (#995)
+## 38. test_objstore_endpoint_userinfo.py: userinfo in an object-store endpoint
 
 Not a port and not a pair: `objstore_endpoint_userinfo.sh` does not exist. These assert
 the same properties as `test/objstore_userinfo.sh`'s endpoint arms, independently,
@@ -3889,8 +3893,6 @@ through the python harness.
 | `test_a_clean_endpoint_is_not_refused_as_userinfo` | the control -- a clean endpoint gets past the guard and fails for another reason |
 | `test_an_at_sign_in_the_object_key_is_not_userinfo` | the other direction: `@` is legal in a key and is untouched |
 
-### `test_iceberg_fdw.py` -- the Iceberg FDW's pruning surface (#388, #432)
-
 Ports `test/iceberg_fdw.sh`. 74 of its 76 check names, one for one; the two it cannot
 carry are `pgc_skip`'s refusal names, which are structural and declared in
 `INCOMPLETE` with their reason.
@@ -3925,7 +3927,7 @@ carry are `pgc_skip`'s refusal names, which are structural and declared in
 | `test_a_plan_with_no_pruning_marker_is_not_read_as_zero` | a plan that never mentions `Files Pruned` is not read as 0; needs no server |
 
 
-## 37. test_hilbert_cluster.py: the Hilbert clustering SQL surface
+## 39. test_hilbert_cluster.py: the Hilbert clustering SQL surface
 
 The port of `test/hilbert_cluster.sh` (#432, #889's SQL half). The bash suite pins the SQL
 surface of `pgcolumnar.cluster_hilbert` and `recluster_hilbert`, the recorded

test/pytest/expected_tests.txt

@@ -129,7 +129,10 @@
 # right, the additive constraint run against three REAL pairs, the three refusals with
 # a control, and the name-argument restriction.
 # Re-derived by collection: `340 tests collected`.
-guard_tests 340
+# 340 -> 341: a test file documented as an unnumbered `###` is invisible to every
+# other arm -- outside the numbering, outside the contents, and still NAMED, so the
+# coverage arm passes (#1024). Re-derived by collection: `341 tests collected`.
+guard_tests 341
 
 # The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran
 # in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and

test/pytest/test_docs_cover_the_corpus.py

@@ -714,6 +714,42 @@ def _unresolved_links(text):
     return sorted({a for a in _LINK.findall(text) if a not in have})
 
 
+def test_every_test_file_has_a_NUMBERED_section_of_its_own(expect):
+    """A SECTION AT THE WRONG LEVEL IS INVISIBLE TO EVERY OTHER ARM (#1024).
+
+    The arm above catches a section NUMBER taken twice, which is the collision #1024
+    describes, and it does catch it -- planted, two arms redden. What nothing caught is
+    a test file whose section was written as an unnumbered `###` instead of a numbered
+    `##`: it is not in the numbering, so `1..N with no gap` never sees it; it is not in
+    the contents, so the link arms never see it; and the file IS named in the document,
+    so the coverage arm is satisfied.
+
+    That is not hypothetical. `test_iceberg_fdw.py` shipped that way in #1057 and sat
+    undetected until this arm was written -- one person writing a heading at the wrong
+    level, where the collision needs two PRs in flight.
+
+    MEASURED BEFORE WRITING IT: 33 test files, 32 with a numbered section, one without,
+    and that one was the defect. The rule was already true everywhere else, which is
+    why it can be asserted rather than declared as a goal.
+    """
+    text = (HERE / "TESTS.md").read_text(encoding="utf-8")
+    files = sorted(p.name for p in HERE.glob("test_*.py"))
+    numbered = set(re.findall(r"^## \d+\. (test_\w+\.py)", text, re.M))
+
+    expect.at_least(len(files), 20,
+                    "premise: the corpus was found, so the comparison is not vacuous")
+    missing = [f for f in files if f not in numbered]
+    expect.text(", ".join(missing) or "none", "none",
+                "every test file has a NUMBERED top-level section, so none is "
+                "documented outside the numbering the arms above check")
+
+    # AND THE OTHER DIRECTION, so a section cannot outlive the file it documents --
+    # the same both-ways shape the declaration arms use.
+    gone = [n for n in sorted(numbered) if n not in files]
+    expect.text(", ".join(gone) or "none", "none",
+                "and every numbered section names a file that exists")
+
+
 def test_the_anchor_rule_drops_punctuation_and_keeps_underscores(expect):
     """The derivation, on the heading the defect was found in."""
     # Assembled, for the reason given in the arm below: a literal corpus file name