From bc5ac29c88accf04a3838c4823272b51e56f9fe2 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sun, 13 Sep 2026 08:17:33 -0600 Subject: [PATCH 1/7] test: the parity tool's name rule is wrong for four helpers (#432, #1036) #1036 replaced "the first quoted argument" with "the last argument". That is true of 14 of `Expect`'s 18 helpers, but it is a property of most of them rather than of the class, and the last argument is a real string in each of the other four, so a wrong name looked exactly like a right one: refusal(result, name, *patterns) the last argument is a PATTERN cannot_run(reason, detail="") records name=reason, argument 0 plan_marker(plan, key, name=None) the last argument is a plan KEY plan_node(plan, ..., name=None) the last argument describes the NODE `refusal` is the worst: the real name goes MISSING and a fragment of an error message arrives as an EXTRA, two false entries from one call. Measured over every pair in the tree, with the table as the only variable: 68 extras, now 67. Two were false, both on hilbert_locality, and one TRUE extra appeared in their place, UNMET_PRECONDITION, the reason code cannot_run really records. No pair's verdict moved, because rc is driven by MISSING and extras never moved it. That is why nothing caught this. The positions live in a `_NAME_ARG` table, because the tool is deliberately standalone (ast, re, sys) and importing Expect to ask would pull in pytest. A hand-written derived value goes stale, so it is pinned: a drift guard reads the real signatures out of pgc_vacuity.py, recomputes every entry, and fails with the helper named. That guard first passed over a missing `refusal` entry, because `name` IS its last DECLARED parameter -- `*patterns` is not in `args` -- so it now requires the absence of a vararg before treating "last" as the name's own index. Found by mutation, not by reading. Removal proof, each mutation asserted to apply by md5: delete the _NAME_ARG table entirely all four new arms drop the refusal entry its arm, and the drift guard plan_marker None -> -1 its arm, and the drift guard cannot_run 0 -> -1 its arm a wrong entry for at_least the drift guard, whole-tree arm add a helper whose name is not last the drift guard, naming it Guard job green at 303 collected, re-derived by collection and not by adding five: 303 passed, 752 checks, 752 pass + 0 fail + 0 unrun. No bash twin: compare_to_bash.py is pytest-harness plumbing, and the two harnesses parallel the product's functionality rather than each other's tools. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- CHANGELOG.md | 32 +++++++ test/pytest/TESTS.md | 60 +++++++++++- test/pytest/compare_to_bash.py | 37 ++++++- test/pytest/expected_tests.txt | 5 +- test/pytest/test_compare_to_bash.py | 144 ++++++++++++++++++++++++++++ 5 files changed, 274 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d6f5579e..e84f0ca2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1111,6 +1111,38 @@ true until the next version shipped. ### Fixed +- `compare_to_bash.py` read the wrong argument for the four helpers whose name is not + last (#432, #1036). + + The previous fix replaced "the first quoted argument" with "the last argument". That + is true of 14 of `Expect`'s 18 helpers, but it is a property of most of them rather + than of the class, and the last argument is a real string in each of the other four -- + so a wrong name looked exactly like a right one: + + | call | last argument | the name it records | + | --- | --- | --- | + | `refusal(result, name, *patterns)` | a message PATTERN | `name`, argument 1 | + | `cannot_run(reason, detail="")` | the DETAIL of one run | `reason`, argument 0 | + | `plan_marker(plan, key, name=None)` | a plan KEY | the `name=` keyword only | + | `plan_node(plan, ..., name=None)` | a field of the NODE | the `name=` keyword only | + + `refusal` is the worst: the real name goes MISSING and a fragment of an error message + arrives as an EXTRA, which is two false entries from one call. + + Measured over every pair in the tree, with the table as the only variable: **68 extras, + now 67.** Two were false -- a `plan_marker` key and a `cannot_run` detail, both on + `hilbert_locality` -- and one TRUE extra appeared in their place, `UNMET_PRECONDITION`, + the reason code `cannot_run` really records. No pair's verdict moved, because `rc` is + driven by MISSING and extras never moved it. That is why nothing caught this: the tool + reported a plausible list, and only the list was evidence either way. + + The positions live in a `_NAME_ARG` table, because the tool is deliberately standalone + (`ast`, `re`, `sys`) and importing `Expect` to ask would pull in pytest. A + hand-written derived value goes stale, so it is pinned: a drift guard reads the real + signatures out of `pgc_vacuity.py`, recomputes every entry, and fails with the helper + named. That guard first passed over a missing `refusal` entry -- `name` IS its last + DECLARED parameter, since `*patterns` is not -- and now accounts for the vararg. + - A BOGUS-verdict ledger record is refused by naming the verdict, not by field count (#1013). - The star-schema join how-to names clustering on the join key (#752). diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 2c88f5c1..c6d917cb 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -3388,7 +3388,8 @@ the tool was reporting the wrong string. | shape | read as | | --- | --- | -| the last string argument | the name | +| the last argument | the name, for the 14 helpers that put it there | +| the last argument of `refusal`, `cannot_run`, `plan_marker`, `plan_node` | NOT the name -- see below | | an f-string | a `{}` template, matched against bash interpolations reduced the same way | | `"a" if cond else "b"` | both arms | | `@pytest.mark.parametrize("func,name", ROWS)` | the `name` column, resolved through module constants | @@ -3398,6 +3399,53 @@ the tool was reporting the wrong string. template reducer missed every one of them, because its pattern required `[A-Za-z_]` after the dollar. +### The name is not always the last argument (#1036) + +The fix above replaced "the first quoted argument" with "the last argument", and that is +true of 14 of `Expect`'s 18 helpers. It is not a property of the helpers, only of most of +them, and the four exceptions were then read wrong in silence -- the last argument is a +real string in each case, so a wrong name looks exactly like a right one. + +| call | what the last argument is | the name it records | +| --- | --- | --- | +| `refusal(result, name, *patterns)` | a message PATTERN | `name`, argument 1 | +| `cannot_run(reason, detail="")` | the DETAIL of one run | `reason`, argument 0 | +| `plan_marker(plan, key, name=None)` | a plan KEY | the `name=` keyword only | +| `plan_node(plan, ..., name=None)` | a field of the NODE | the `name=` keyword only | + +`refusal` is the worst of the four: the name goes MISSING and a fragment of an error +message arrives as an EXTRA, so one call produces two false entries -- the same defect the +section above closes, one helper along. + +`plan_marker` and `plan_node` contribute NOTHING when called without `name=`. The key is +not the name even then, only a fragment of one (`plan_marker` records +`name or f"plan carries {key!r}"`), and reporting no name states MISSING rather than +inventing one. + +**Measured over the tree**, with the table as the only variable: + +| pair | extras before | after | +| --- | --- | --- | +| hilbert_locality | 3 | 2 | +| every other pair | unchanged | unchanged | +| **total** | **68** | **67** | + +Two false extras went (`Columnar Projected Columns`, a `plan_marker` key; and `the two +partitions are not different ({})`, a `cannot_run` detail) and one TRUE extra appeared: +`UNMET_PRECONDITION`, the reason code `cannot_run` actually records, which the bash suite +has no check for. No pair's verdict moved, because `rc` is driven by MISSING and extras +never moved it -- which is why nothing caught this. + +`refusal` moved no pair either: it is used only by `test_raises_sqlstate.py` and +`test_guards_pinned.py`, neither of which has a bash twin. Its arm drives the real +extractor rather than a pair. + +**The table is a hand-written derived value, so it is pinned.** The tool is deliberately +standalone (`ast`, `re`, `sys`) and cannot import `Expect` to ask where each name sits. +`test_the_tools_table_agrees_with_the_signatures_it_describes` reads the real signatures +out of `pgc_vacuity.py`, recomputes every entry, and fails with the helper named when the +two disagree. + ### Removal proof | mutation | red | @@ -3406,6 +3454,11 @@ the dollar. | drop the conditional-name case | its own arm, and the whole-tree arm | | drop parametrize resolution | its own arm, and the whole-tree arm | | read the name column by position instead of by its declared name | its own arm, and the whole-tree arm | +| delete the `_NAME_ARG` table entirely | all four #1036 arms | +| drop the `refusal` entry | its own arm, and the drift guard | +| `plan_marker` `None` -> `-1`, taking the key | its own arm, and the drift guard | +| `cannot_run` `0` -> `-1`, taking the detail | its own arm | +| a wrong entry for a helper no arm covers (`at_least`) | the drift guard, and the whole-tree arm | `test_the_ported_suites_in_this_tree_are_graded_one_for_one` catches all four. It is the arm that matters: a guard over invented sources proves the extractor reads python, not that @@ -3422,4 +3475,9 @@ the tool grades THIS tree. | `test_a_parametrized_name_is_resolved_from_the_decorator` | the idiom a repeated bash property should be ported to, with a no-`name` decorator as the control | | `test_the_parametrize_reader_takes_the_column_called_name` | the declared column, not position | | `test_the_two_harnesses_interpolations_land_on_one_template` | bash and python spell interpolation differently and must meet | +| `test_refusal_names_its_second_argument_not_its_last_pattern` | the name is in the middle; the last argument is a pattern | +| `test_refusal_with_no_pattern_is_not_the_arm_that_proves_it` | the control: that shape reads the same under either rule, so it proves nothing alone | +| `test_cannot_run_names_its_reason_not_its_detail` | the only helper whose name is argument zero | +| `test_a_helper_whose_name_is_optional_takes_it_only_from_the_keyword` | `plan_marker` and `plan_node` carry no name positionally; absent beats a key | +| `test_the_tools_table_agrees_with_the_signatures_it_describes` | the drift guard: every entry re-derived from the real signatures | | `test_the_ported_suites_in_this_tree_are_graded_one_for_one` | the standing arm: every pair in the tree, graded | diff --git a/test/pytest/compare_to_bash.py b/test/pytest/compare_to_bash.py index f7f03178..69683c56 100755 --- a/test/pytest/compare_to_bash.py +++ b/test/pytest/compare_to_bash.py @@ -116,12 +116,40 @@ def test_a_role_with_only_schema_usage_is_refused(..., func, name): return out +# WHERE THE NAME SITS, for the helpers where it is not the last argument (#1036). +# +# The rule for most of `Expect` is "the name is the last argument", and for 14 of its 18 +# helpers that is true. It is not a property of the helpers, though, only of most of them, +# and the four below were read wrong in silence: the last argument is a real string in each +# case, so a wrong name looked exactly like a right one. +# +# refusal(result, name, *patterns) the last argument is a PATTERN +# cannot_run(reason, detail="") records `name=reason`, the FIRST argument +# plan_marker(plan, key, name=None) the last argument is a plan KEY +# plan_node(plan, ..., name=None) the last argument describes the NODE +# +# A value here is the index of the call argument carrying the name; `None` means no +# positional argument carries it and only a `name=` keyword can. `-1`, the default for +# every helper not listed, means the last one. +# +# This is a hand-written derived value, so it is pinned: the drift guard in +# `test_compare_to_bash.py` re-derives every entry from the real signatures in +# `pgc_vacuity.py` and fails with the helper named when the two disagree. Add a helper +# whose name is not last and that arm goes red before this table is wrong in the field. +_NAME_ARG = { + "refusal": 1, + "cannot_run": 0, + "plan_marker": None, + "plan_node": None, +} + + def _py_names(src): """Every assertion name in the port, by parsing rather than matching. The name is the LAST argument of an `expect.(...)` call, or the value of a `name=` keyword, read through `_as_names` so a conditional carries both of - its arms. + its arms. Four helpers put it somewhere else and are read through `_NAME_ARG`. """ tree = ast.parse(src) out = _parametrized_names(tree) @@ -137,7 +165,12 @@ def _py_names(src): out.extend(_as_names(kw.value)) if not is_expect or not node.args: continue - out.extend(_as_names(node.args[-1])) + idx = _NAME_ARG.get(func.attr, -1) + if idx is None: + continue + if idx != -1 and len(node.args) <= idx: + continue + out.extend(_as_names(node.args[idx])) return out diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 39e5213b..d4a8a93e 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -51,7 +51,10 @@ # 290 -> 298 when test_compare_to_bash.py landed: eight arms over the parity tool's own # extractors. Re-derived by collection on the merged tree, per the recipe above: # `298 tests collected`. -guard_tests 298 +# 298 -> 303 when the same file gained five arms for the helpers whose name is not the +# last argument (#1036), four of them behavioural and one a drift guard over the table. +# Re-derived by collection, not by adding five: `303 tests collected`. +guard_tests 303 # The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran # in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 25ee5abd..8111b1b3 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -160,6 +160,150 @@ def test_the_two_harnesses_interpolations_land_on_one_template(expect): "a bare variable reference too") +def test_refusal_names_its_second_argument_not_its_last_pattern(expect): + """`refusal(result, name, *patterns)` puts the name in the MIDDLE. + + The last argument is a pattern -- a fragment of the message the refusal must carry -- + so the last-argument rule read a substring of an error message as the property's name. + The real name went MISSING and the pattern arrived as an EXTRA: two false entries from + one call, which is the same defect this file exists to close, one helper along. + """ + src = ('def t(expect):\n' + ' expect.refusal(result, "a role with no privilege is refused",\n' + ' "permission denied", "for table")\n') + got = _names(src) + expect.text(", ".join(sorted(got)), "a role with no privilege is refused", + "the name is read and neither pattern is") + expect.num(len(got), 1, "one call contributes exactly one name") + + +def test_refusal_with_no_pattern_is_not_the_arm_that_proves_it(expect): + """THE CONTROL that keeps the arm above honest. + + `expect.refusal(result, NAME)` has the name last, so it is read correctly by the rule + this change replaces AND by the rule that replaces it. An arm built only on that shape + would pass against the defect, which is how the shape got missed in the first place. + """ + src = 'def t(expect):\n expect.refusal(result, "the write is refused")\n' + expect.text(", ".join(_names(src)), "the write is refused", + "the no-pattern shape reads the same either way, so it proves nothing alone") + + +def test_cannot_run_names_its_reason_not_its_detail(expect): + """`cannot_run(reason, detail="")` records `name=reason`: the FIRST argument. + + It is the only helper whose name is argument zero, and the detail beside it is prose + about one run -- "the two partitions are not different ({})" -- which can never match + a bash check name. Reading it produced an extra that no bash suite could ever satisfy. + """ + src = ('def t(expect):\n' + ' expect.cannot_run("MISSING_DEPENDENCY",\n' + ' "the two partitions are not different")\n') + got = _names(src) + expect.text(", ".join(got), "MISSING_DEPENDENCY", + "the reason CODE is the name, and the detail is not a name at all") + expect.num(len(got), 1, "the detail contributes nothing") + + +def test_a_helper_whose_name_is_optional_takes_it_only_from_the_keyword(expect): + """`plan_marker` and `plan_node` carry no name positionally. Better absent than wrong. + + `plan_marker(plan, key, name=None)` records `name or f"plan carries {key!r}"`, so the + KEY is not the name even when no name is given -- it is a fragment of one. The + last-argument rule emitted the bare key as a name the bash suite does not have, and + `Columnar Projected Columns` duly appeared as an extra on a pair that is complete. + + With no name= the call contributes NOTHING, which reports MISSING rather than inventing + a name: the rule this file already applies to a name it cannot read. + """ + named = _names('def t(expect):\n' + ' expect.plan_marker(plan, "Columnar Projected Columns",\n' + ' name="the plan projects two columns")\n') + expect.text(", ".join(named), "the plan projects two columns", + "the name= keyword is the name, and the key is not also collected") + expect.num(len(named), 1, "one call, one name -- the key is not a second entry") + + bare = _names('def t(expect):\n' + ' expect.plan_marker(plan, "Columnar Projected Columns")\n') + expect.num(len(bare), 0, "with no name= the key is still not a name") + + node = _names('def t(expect):\n' + ' expect.plan_node(plan, provider="columnar",\n' + ' name="the scan is columnar")\n') + expect.text(", ".join(node), "the scan is columnar", + "plan_node reads its name= and not the arguments describing the node") + + +def test_the_tools_table_agrees_with_the_signatures_it_describes(expect): + """THE DRIFT GUARD, and the reason the table is allowed to be a hand-written map. + + `compare_to_bash.py` is deliberately standalone -- `ast`, `re`, `sys` -- so it cannot + import `Expect` to ask where each name sits, and a hand-written table is a derived + value that goes stale the day somebody adds a helper. This arm is what stops that: it + reads the REAL signatures out of `pgc_vacuity.py` and recomputes, for every public + helper, which call argument carries the name the helper records. + + It is not a copy of the table. The table says where to look; this derives where to look + from the source of truth and compares. A helper added with its name anywhere but last, + or a signature reordered, fails here with the helper named. + """ + from compare_to_bash import _NAME_ARG + + # The ONE thing a signature cannot state: which parameter becomes the record's name. + # `cannot_run` records `name=reason`; every other helper calls its parameter `name`. + # Pinned below against the body, so this line cannot quietly become wrong either. + records_name_as = {"cannot_run": "reason"} + + src = (HERE / "pgc_vacuity.py").read_text() + tree = ast.parse(src) + klass = [n for n in ast.walk(tree) + if isinstance(n, ast.ClassDef) and n.name == "Expect"] + expect.num(len(klass), 1, "premise: exactly one Expect class to read") + + helpers = [f for f in klass[0].body + if isinstance(f, ast.FunctionDef) and not f.name.startswith("_")] + # `records` and `count` take no arguments and record no name. + helpers = [f for f in helpers if [a.arg for a in f.args.args if a.arg != "self"]] + expect.at_least(len(helpers), 15, + "premise: the Expect helpers were found, not an empty list") + + disagree, checked = [], 0 + for f in helpers: + params = [a.arg for a in f.args.args if a.arg != "self"] + ndef = len(f.args.defaults) + required = params[:len(params) - ndef] if ndef else params + param = records_name_as.get(f.name, "name") + # Optional => no positional carries it; only a `name=` keyword can. + want = required.index(param) if param in required else None + got = _NAME_ARG.get(f.name, -1) + # -1 is "the last positional". That equals the name's own index only when the + # name really is last AT THE CALL SITE, and a `*args` AFTER it means it is not: + # `refusal(result, name, *patterns)` declares `name` last and is still called + # with patterns beyond it. Without that clause this arm accepted a missing + # `refusal` entry, which is the very shape it is here to catch. + if (got == -1 and want is not None and want == len(required) - 1 + and f.args.vararg is None): + got = want + checked += 1 + if got != want: + disagree.append(f"{f.name}: table says {got!r}, signature says {want!r}") + + expect.num(checked, len(helpers), "inputs == sum(buckets): every helper was compared") + expect.text("; ".join(disagree) or "none", "none", + "every entry in the table matches the signature it describes") + + # The one hand-written semantic claim above, pinned against the body it describes: + # read cannot_run's own `_record(...)` call and check which parameter it names. + fn = [f for f in helpers if f.name == "cannot_run"] + expect.num(len(fn), 1, "premise: cannot_run is among the helpers read") + recorded = [kw.value.id for call in ast.walk(fn[0]) + if isinstance(call, ast.Call) + and isinstance(call.func, ast.Attribute) and call.func.attr == "_record" + for kw in call.keywords + if kw.arg == "name" and isinstance(kw.value, ast.Name)] + expect.text(", ".join(recorded), "reason", + "cannot_run really does record its reason as the name") + def test_the_ported_suites_in_this_tree_are_graded_one_for_one(expect): """THE STANDING ARM, and the reason this file is not only about fixtures. From be9b9005a5110b569e681aae245d3520f993b961 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sun, 13 Sep 2026 08:34:35 -0600 Subject: [PATCH 2/7] test: close the second coincidence, and stop calling an extra TRUE (#432, #1036) Both found by @OffgridwithJD reviewing the first commit, and both re-measured here rather than taken on the review's word. 1. A PUBLISHED SENTENCE WAS FALSE. CHANGELOG.md and TESTS.md said UNMET_PRECONDITION was a TRUE extra "which the bash suite has no check for". The bash suite HAS four checks for it, at hilbert_locality.sh:574 and the three after it: check_unrunnable "box $box: groups read, Z-order" UNMET_PRECONDITION ... The tool cannot see them. Its bash extractor reads `check(_num|_ratio|_text|_timing)?`, and `check_unrunnable` matches no branch. Widening that regex by that one alternative and changing nothing else takes hilbert_locality from rc=0 missing=0 to rc=1 missing=2 -- "box $box: groups read, Hilbert" and "box $box: groups read, Z-order" -- every other pair unchanged. So the port emits ONE record where bash emits four per box, and two have no counterpart in the port. Eight bash check helpers are invisible to that regex (89 invocations across test/*.sh, counted as invocations with definitions excluded). Filed separately: widening it reddens a pair and is a port's worth of work, not a tool fix. The defensible sentence is the narrow one, and both files now say it. 2. A SECOND COINCIDENCE, INSIDE THE CLAUSE THAT FIXED THE FIRST. `-1` is a claim about the CALL SITE; the drift guard reads the SIGNATURE. They agree only while no optional parameter sits after the name, because an optional one can still be passed positionally: expect.rows(got, want, "THE NAME", "the reason") -> read 'the reason' expect.plan_marker(plan, "key", "THE NAME") -> read nothing at all Both legal, both read wrong, every guard green. The second is worse: a DROPPED name reports the bash property MISSING, and MISSING drives rc. Latent rather than live -- no call site in the tree passes a trailing optional positionally, verified over all 1025 -- but #1037 makes allow_empty a reason STRING, which is exactly that argument. Closed in the SIGNATURES rather than patched in the reader: rows, row_set, plan_marker and plan_node take everything after the name as keyword-only, so a wrong call is a TypeError instead of a misread name: Expect.rows() takes 4 positional arguments but 5 were given No call site changed; all four already used keywords, verified over all 1025 before the change. Removal proof, each mutation asserted to apply by md5 and the tree restored: revert row_set's * names row_set revert rows' * names rows revert plan_marker's * names plan_marker revert plan_node's * names plan_node Guard job green at 304 collected, re-derived by collection: 304 passed, 756 checks, 756 pass + 0 fail + 0 unrun. docs_style.sh 14 checks PASSED. Counts are now labelled with the tree they were taken on, because they move against their own branch head. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- CHANGELOG.md | 30 +++++++++++--- test/pytest/TESTS.md | 62 ++++++++++++++++++++++++++--- test/pytest/expected_tests.txt | 4 +- test/pytest/pgc_vacuity.py | 8 ++-- test/pytest/test_compare_to_bash.py | 60 ++++++++++++++++++++++++++++ 5 files changed, 148 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e84f0ca2..b0287fe7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1129,12 +1129,19 @@ true until the next version shipped. `refusal` is the worst: the real name goes MISSING and a fragment of an error message arrives as an EXTRA, which is two false entries from one call. - Measured over every pair in the tree, with the table as the only variable: **68 extras, - now 67.** Two were false -- a `plan_marker` key and a `cannot_run` detail, both on - `hilbert_locality` -- and one TRUE extra appeared in their place, `UNMET_PRECONDITION`, - the reason code `cannot_run` really records. No pair's verdict moved, because `rc` is - driven by MISSING and extras never moved it. That is why nothing caught this: the tool - reported a plausible list, and only the list was evidence either way. + Measured over every pair in the tree at `73e8e3d`, with the table as the only variable: + **68 extras, now 67.** Two were false -- a `plan_marker` key and a `cannot_run` detail, + both on `hilbert_locality` -- and `UNMET_PRECONDITION` appeared in their place, the + reason code `cannot_run` really records. No pair's verdict moved, because `rc` is driven + by MISSING and extras never moved it. That is why nothing caught this: the tool reported + a plausible list, and only the list was evidence either way. + + `UNMET_PRECONDITION` is reported as an extra only because the tool cannot see the bash + side of it. `hilbert_locality.sh:574` and three lines after it check that property with + `check_unrunnable`, which the bash extractor's `check(_num|_ratio|_text|_timing)?` does + not match. Widening it by that one alternative and nothing else takes that pair from + `rc=0 missing=0` to `rc=1 missing=2`, every other pair unchanged. Filed separately: it + is a port's worth of work, not a tool fix, and this change is only what made it visible. The positions live in a `_NAME_ARG` table, because the tool is deliberately standalone (`ast`, `re`, `sys`) and importing `Expect` to ask would pull in pytest. A @@ -1143,6 +1150,17 @@ true until the next version shipped. named. That guard first passed over a missing `refusal` entry -- `name` IS its last DECLARED parameter, since `*patterns` is not -- and now accounts for the vararg. + A second coincidence sat inside the clause that fixed the first, found by + @OffgridwithJD in review. `-1` is a claim about the CALL SITE while the guard reads the + SIGNATURE, and they agree only while no optional parameter sits after the name: + `expect.rows(got, want, "THE NAME", "the reason")` read `the reason`, and + `expect.plan_marker(plan, "key", "THE NAME")` read nothing at all, dropping a name + silently. Latent rather than live -- no call site passes a trailing optional + positionally -- but #1037 makes `allow_empty` a reason string, which is exactly that + argument. Closed in the signatures rather than in the reader: `rows`, `row_set`, + `plan_marker` and `plan_node` take everything after the name as keyword-only, so the + wrong call is now a `TypeError`. No call site changed; all four already used keywords. + - A BOGUS-verdict ledger record is refused by naming the verdict, not by field count (#1013). - The star-schema join how-to names clustering on the join key (#752). diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index c6d917cb..60e158f5 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -3422,7 +3422,8 @@ not the name even then, only a fragment of one (`plan_marker` records `name or f"plan carries {key!r}"`), and reporting no name states MISSING rather than inventing one. -**Measured over the tree**, with the table as the only variable: +**Measured over the tree** at `73e8e3d`, with the table as the only variable (the count +is labelled with the tree because it moves as pairs are added): | pair | extras before | after | | --- | --- | --- | @@ -3431,15 +3432,63 @@ inventing one. | **total** | **68** | **67** | Two false extras went (`Columnar Projected Columns`, a `plan_marker` key; and `the two -partitions are not different ({})`, a `cannot_run` detail) and one TRUE extra appeared: -`UNMET_PRECONDITION`, the reason code `cannot_run` actually records, which the bash suite -has no check for. No pair's verdict moved, because `rc` is driven by MISSING and extras -never moved it -- which is why nothing caught this. +partitions are not different ({})`, a `cannot_run` detail) and one appeared in their place: +`UNMET_PRECONDITION`, the reason code `cannot_run` actually records. No pair's verdict +moved, because `rc` is driven by MISSING and extras never moved it -- which is why nothing +caught this. + +**`UNMET_PRECONDITION` is an extra only because the tool cannot see the bash side of it**, +and saying otherwise would be the same mistake one level down. `hilbert_locality.sh:574` +and three lines after it DO check that property: + + check_unrunnable "box $box: groups read, Z-order" UNMET_PRECONDITION ... + +The bash extractor reads `check(_num|_ratio|_text|_timing)?`, and `check_unrunnable` +matches no branch of it. Widening that regex by that one alternative and changing nothing +else takes `hilbert_locality` from `rc=0 missing=0` to **`rc=1 missing=2`** -- `box $box: +groups read, Hilbert` and `box $box: groups read, Z-order` -- with every other pair +unchanged. The port emits ONE record named `UNMET_PRECONDITION` where bash emits four per +box, and two of them have no counterpart in the port at all. + +That gap is NOT caused by the change above; the change is what made it visible, and it is +filed separately rather than widened here, because widening the regex reddens a pair and +is a port's worth of work rather than a tool fix. Eight more bash check helpers are +invisible to the same regex (`check_skip`, `check_structure`, `check_reconstruct`, +`check_split_happened`, `check_ratio_needs_quiet_machine`, `check_float`, +`check_stack_depth`, and `check_unrunnable` itself). `refusal` moved no pair either: it is used only by `test_raises_sqlstate.py` and `test_guards_pinned.py`, neither of which has a bash twin. Its arm drives the real extractor rather than a pair. +### A second coincidence, inside the clause that fixed the first + +`-1` is a claim about the CALL SITE. The drift guard reads the SIGNATURE. They agree only +while no OPTIONAL parameter sits after the name, because an optional one can still be +passed positionally: + +| written | read as | +| --- | --- | +| `expect.rows(got, want, "THE NAME", "the reason")` | `the reason` | +| `expect.plan_marker(plan, "key", "THE NAME")` | nothing at all | + +Both were legal, both read wrong, and every guard here stayed green. The second is worse: +a DROPPED name reports the bash property MISSING, and MISSING is what drives `rc`. + +Latent rather than live -- no call site in the tree passes a trailing optional +positionally -- but #1037 makes `allow_empty` a reason STRING, which is exactly the +argument somebody writes positionally next to a name. + +**Closed in the signatures rather than patched in the reader.** `rows`, `row_set`, +`plan_marker` and `plan_node` now take everything after the name as keyword-only, so the +wrong call is a `TypeError` instead of a silently misread name: + + Expect.rows() takes 4 positional arguments but 5 were given + +`test_no_later_argument_can_overtake_the_name` holds it, and it is a signature fact, which +is what this guard is already good at reading. `cannot_run` needs no change: its name is +argument 0 and nothing after it can overtake it. + **The table is a hand-written derived value, so it is pinned.** The tool is deliberately standalone (`ast`, `re`, `sys`) and cannot import `Expect` to ask where each name sits. `test_the_tools_table_agrees_with_the_signatures_it_describes` reads the real signatures @@ -3459,6 +3508,8 @@ two disagree. | `plan_marker` `None` -> `-1`, taking the key | its own arm, and the drift guard | | `cannot_run` `0` -> `-1`, taking the detail | its own arm | | a wrong entry for a helper no arm covers (`at_least`) | the drift guard, and the whole-tree arm | +| add a helper to `Expect` whose name is not last | the drift guard, naming it | +| revert any one of the four `*` keyword-only markers | `test_no_later_argument_can_overtake_the_name`, naming the helper | `test_the_ported_suites_in_this_tree_are_graded_one_for_one` catches all four. It is the arm that matters: a guard over invented sources proves the extractor reads python, not that @@ -3480,4 +3531,5 @@ the tool grades THIS tree. | `test_cannot_run_names_its_reason_not_its_detail` | the only helper whose name is argument zero | | `test_a_helper_whose_name_is_optional_takes_it_only_from_the_keyword` | `plan_marker` and `plan_node` carry no name positionally; absent beats a key | | `test_the_tools_table_agrees_with_the_signatures_it_describes` | the drift guard: every entry re-derived from the real signatures | +| `test_no_later_argument_can_overtake_the_name` | nothing after the name may be passed positionally, so `-1` is true of every CALL and not just every signature | | `test_the_ported_suites_in_this_tree_are_graded_one_for_one` | the standing arm: every pair in the tree, graded | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index d4a8a93e..83b263d4 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -54,7 +54,9 @@ # 298 -> 303 when the same file gained five arms for the helpers whose name is not the # last argument (#1036), four of them behavioural and one a drift guard over the table. # Re-derived by collection, not by adding five: `303 tests collected`. -guard_tests 303 +# 303 -> 304 when review found a second coincidence: `-1` is a claim about the CALL SITE +# and the drift guard reads the SIGNATURE. Re-derived by collection: `304 tests collected`. +guard_tests 304 # The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran # in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and diff --git a/test/pytest/pgc_vacuity.py b/test/pytest/pgc_vacuity.py index 063b0d40..2727b777 100644 --- a/test/pytest/pgc_vacuity.py +++ b/test/pytest/pgc_vacuity.py @@ -438,7 +438,7 @@ def num(self, got, want, name): if got != want: raise AssertionError(f"{name}: got {got!r} want {want!r}") - def row_set(self, got, want, name, allow_empty=None): + def row_set(self, got, want, name, *, allow_empty=None): """Compare two result sets as SETS, order deliberately ignored. The counterpart to ordered_rows, and the port of pgc_set_hash. It exists so @@ -645,7 +645,7 @@ def wrote(self, cur, want, name): # -- row sets ---------------------------------------------------------- @_resolving - def rows(self, got, want, name, allow_empty=None): + def rows(self, got, want, name, *, allow_empty=None): """Compare two result sets. Refuses two empty sides unless declared. Both sides empty is issue #418: it passes while asserting nothing, because @@ -760,7 +760,7 @@ def sqlstate(self, exc, want, name): # -- plans ------------------------------------------------------------- @_resolving - def plan_node(self, plan, node_type=None, provider=None, name=None): + def plan_node(self, plan, *, node_type=None, provider=None, name=None): """Assert a node exists, by EXACT equality on a typed EXPLAIN JSON field. `EXPLAIN (FORMAT JSON)` arrives from psycopg as parsed Python, so there is @@ -904,7 +904,7 @@ def run_failed(self, result, name): ) @_resolving - def plan_marker(self, plan, key, name=None, absent=False): + def plan_marker(self, plan, key, *, name=None, absent=False): """Assert a plan node carries (or does not carry) a Columnar property KEY. This is the faithful port of `pgc_is_columnar_scan` (`lib.sh`), which greps diff --git a/test/pytest/test_compare_to_bash.py b/test/pytest/test_compare_to_bash.py index 8111b1b3..dd8fd653 100644 --- a/test/pytest/test_compare_to_bash.py +++ b/test/pytest/test_compare_to_bash.py @@ -304,6 +304,66 @@ def test_the_tools_table_agrees_with_the_signatures_it_describes(expect): expect.text(", ".join(recorded), "reason", "cannot_run really does record its reason as the name") +def test_no_later_argument_can_overtake_the_name(expect): + """`-1` is a claim about the CALL SITE, and the arm above only reads the SIGNATURE. + + Found by @OffgridwithJD reviewing the change this file documents, inside the very + clause that fixed the vararg coincidence. The guard asks "which parameter carries the + name", which is a fact about the declaration. `-1` says "the last argument", which is + a fact about the call. They agree only while no OPTIONAL parameter sits after the + name, because an optional one may still be passed POSITIONALLY: + + expect.rows(got, want, "THE NAME", "the reason") -> read 'the reason' + expect.plan_marker(plan, "key", "THE NAME") -> read nothing at all + + Both were legal, both read wrong, and every guard in this file stayed green. The + second is the worse one: a DROPPED name reports the bash property MISSING, and + MISSING is what drives `rc`. + + Latent rather than live -- no call site in the tree passes a trailing optional + positionally -- but #1037 makes `allow_empty` a reason STRING, which is exactly the + argument somebody writes positionally next to a name. + + So the property is closed in the SIGNATURES rather than patched in the reader: every + parameter after the name is keyword-only, and this arm holds that. A wrong call is + then a `TypeError`, not a silently misread name. + """ + src = (HERE / "pgc_vacuity.py").read_text() + klass = [n for n in ast.walk(ast.parse(src)) + if isinstance(n, ast.ClassDef) and n.name == "Expect"] + expect.num(len(klass), 1, "premise: exactly one Expect class to read") + helpers = [f for f in klass[0].body + if isinstance(f, ast.FunctionDef) and not f.name.startswith("_") + and [a.arg for a in f.args.args if a.arg != "self"]] + expect.at_least(len(helpers), 15, "premise: the helpers were found, not an empty list") + + from compare_to_bash import _NAME_ARG + + overtakable, checked = [], 0 + for f in helpers: + params = [a.arg for a in f.args.args if a.arg != "self"] + checked += 1 + if "name" not in params: + # Carried only as a keyword, or named something else (`cannot_run`, whose + # name is argument 0 and cannot be overtaken by anything after it). + continue + if _NAME_ARG.get(f.name, -1) is None: + # The table says NO positional argument carries the name, so the reader + # skips the call entirely. If `name` can still be written positionally the + # name is DROPPED, which reports the bash property MISSING and moves `rc`. + overtakable.append(f"{f.name}: the table reads no positional name, yet name " + f"can be passed positionally") + continue + after = params[params.index("name") + 1:] + if after: + overtakable.append(f"{f.name}: {', '.join(after)} can be passed positionally " + f"after name") + + expect.num(checked, len(helpers), "inputs == sum(buckets): every helper was examined") + expect.text("; ".join(overtakable) or "none", "none", + "no positional argument can be written after the name and be read as it") + + def test_the_ported_suites_in_this_tree_are_graded_one_for_one(expect): """THE STANDING ARM, and the reason this file is not only about fixtures. From 3d35e0d7b16f037c480acb2b2874943c42a200d5 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sun, 13 Sep 2026 08:35:38 -0600 Subject: [PATCH 3/7] docs: name the extractor gap issue (#1040) The two places that said "filed separately" now cite the issue, so a reader does not have to search for it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- CHANGELOG.md | 4 ++-- test/pytest/TESTS.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b0287fe7..11fc79d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1140,8 +1140,8 @@ true until the next version shipped. side of it. `hilbert_locality.sh:574` and three lines after it check that property with `check_unrunnable`, which the bash extractor's `check(_num|_ratio|_text|_timing)?` does not match. Widening it by that one alternative and nothing else takes that pair from - `rc=0 missing=0` to `rc=1 missing=2`, every other pair unchanged. Filed separately: it - is a port's worth of work, not a tool fix, and this change is only what made it visible. + `rc=0 missing=0` to `rc=1 missing=2`, every other pair unchanged. Filed as #1040: it is a + port's worth of work, not a tool fix, and this change is only what made it visible. The positions live in a `_NAME_ARG` table, because the tool is deliberately standalone (`ast`, `re`, `sys`) and importing `Expect` to ask would pull in pytest. A diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 60e158f5..81c5e684 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -3451,8 +3451,8 @@ unchanged. The port emits ONE record named `UNMET_PRECONDITION` where bash emits box, and two of them have no counterpart in the port at all. That gap is NOT caused by the change above; the change is what made it visible, and it is -filed separately rather than widened here, because widening the regex reddens a pair and -is a port's worth of work rather than a tool fix. Eight more bash check helpers are +filed as #1040 rather than widened here, because widening the regex reddens a pair and is +a port's worth of work rather than a tool fix. Eight more bash check helpers are invisible to the same regex (`check_skip`, `check_structure`, `check_reconstruct`, `check_split_happened`, `check_ratio_needs_quiet_machine`, `check_float`, `check_stack_depth`, and `check_unrunnable` itself). From eea489486428c51e1af3f89fc3197db288865ea7 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sun, 13 Sep 2026 08:42:44 -0600 Subject: [PATCH 4/7] docs: state the extractor gap as a structure, not a sweep total (#1040) TESTS.md said "eight more bash check helpers are invisible", listing a mix of helpers lib.sh defines with helpers an individual suite defines, and including check_stack_depth, which is not one of these helpers at all. @OffgridwithJD caught it. Derived from test/lib.sh instead of swept for: DEFINED IN lib.sh 8 READ by the tool 5 check, check_num, check_text, check_ratio, check_timing INVISIBLE 3 check_unrunnable, check_skip, check_ratio_needs_quiet_machine Four more are defined locally by a single suite and are invisible to the same regex: check_structure, check_reconstruct, check_split_happened (parallel_copy.sh) and check_float (parquet_export_stats.sh). NO INVOCATION TOTAL IS QUOTED, and that is the point rather than an omission. Four sweeps across two agents produced 89, 64, 54 and 50 for the same quantity; the differences are all method -- definition lines, whole-line comments, command position, line continuations. The STRUCTURE is stable under every method and the totals are not, so the structure is what is published. A census four runs cannot reproduce is not evidence yet. #1040 is edited to match, and records the correction rather than hiding it. It also now records a trap in the other direction: THIRTEEN suites define their own check(), so a lib.sh-derived helper list must say so rather than imply that `check` is one function. docs_style.sh 14 checks PASSED. Guard job unchanged at 304 collected, 756 checks, 756 pass + 0 fail + 0 unrun. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- test/pytest/TESTS.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 81c5e684..0b602b41 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -3452,10 +3452,19 @@ box, and two of them have no counterpart in the port at all. That gap is NOT caused by the change above; the change is what made it visible, and it is filed as #1040 rather than widened here, because widening the regex reddens a pair and is -a port's worth of work rather than a tool fix. Eight more bash check helpers are -invisible to the same regex (`check_skip`, `check_structure`, `check_reconstruct`, -`check_split_happened`, `check_ratio_needs_quiet_machine`, `check_float`, -`check_stack_depth`, and `check_unrunnable` itself). +a port's worth of work rather than a tool fix. + +Derived from `test/lib.sh` rather than swept for, because three different sweeps gave +three different totals: **`lib.sh` defines 8 check helpers, the tool reads 5, and 3 are +invisible** -- `check_unrunnable`, `check_skip`, `check_ratio_needs_quiet_machine`. +Individual suites define four more of their own (`check_structure`, +`check_reconstruct`, `check_split_happened` in `parallel_copy.sh`, `check_float` in +`parquet_export_stats.sh`), invisible to the same regex. + +No invocation TOTAL is quoted here on purpose. Counting them is method-sensitive -- a +plain `grep -c`, a command-position match, and a comment-stripped pass disagree, and +between two agents four sweeps produced 89, 64, 54 and 50. The structure above is stable +under every method; the totals are not, so #1040 carries the method rather than a number. `refusal` moved no pair either: it is used only by `test_raises_sqlstate.py` and `test_guards_pinned.py`, neither of which has a bash twin. Its arm drives the real From ff27a549256baf3008b6677d946240ec991cda9b Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sun, 13 Sep 2026 08:48:02 -0600 Subject: [PATCH 5/7] docs: the census reconciles to 50; "method-sensitive" was my own false claim (#1040) @OffgridwithJD refuted the paragraph I added in eea4894, and they were right. I had written that the invisible-invocation total was method-sensitive and declined to quote one, on the grounds that four sweeps across two agents read 89, 64, 54 and 50. That statement was FALSE. The quantity was never unstable. Three of the four sweeps were buggy, each with a findable cause, and two independent methods now agree helper for helper: check_unrunnable 25 check_skip 23 check_ratio_needs_quiet_machine 2 -- 50 Both causes are recorded because a later re-derivation meets them: 1. A `\bNAME\s` sweep counts each helper's OWN DEFINITION LINE, because the definition carries a trailing usage comment repeating the name: lib.sh:1231 check_unrunnable() {# check_unrunnable NAME REASON_CODE DETAIL lib.sh:1407 check_skip() {# check_skip NAME DISPLAY [REASON] Two further matches were ordinary prose (lib.sh:1338, planner_choice_quality.sh:149). That is 89 with definitions in and 54 with comments in. 2. A command-position match misses a call after `&&`: hilbert_curve.sh:321 [ -n "$_a" ] && check_unrunnable "$_a" "$2" "$3" Anchoring on `^` alone gives 24 for that helper rather than 25. The recipe that reproduces 50 is in both TESTS.md and #1040: strip trailing comments as well as whole-line ones, exclude definitions, and accept a call after `;`, `&&` or `||`. DECLINING TO QUOTE A NUMBER LOOKED LIKE RIGOUR AND WAS THE LAZIER CLAIM. It published a false statement about the tree in order to avoid finding out why three runs disagreed. Two instruments disagreeing is evidence about the frame, not a range to average, and "not reproducible" is averaging with extra steps. The structural half (lib.sh defines 8, the tool reads 5, 3 invisible, plus 4 suite-local) needed no correction and is unchanged. docs_style.sh 14 checks PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- test/pytest/TESTS.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 0b602b41..0376add5 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -3461,10 +3461,25 @@ Individual suites define four more of their own (`check_structure`, `check_reconstruct`, `check_split_happened` in `parallel_copy.sh`, `check_float` in `parquet_export_stats.sh`), invisible to the same regex. -No invocation TOTAL is quoted here on purpose. Counting them is method-sensitive -- a -plain `grep -c`, a command-position match, and a comment-stripped pass disagree, and -between two agents four sweeps produced 89, 64, 54 and 50. The structure above is stable -under every method; the totals are not, so #1040 carries the method rather than a number. +**50 invisible invocations**, reconciled between two agents and two independent methods, +which agree helper for helper: `check_unrunnable` 25, `check_skip` 23, +`check_ratio_needs_quiet_machine` 2. + +Getting there took four sweeps that read 89, 64, 54 and 50, and the three wrong ones were +not method-sensitivity -- they were two defects, both worth knowing because any later +re-derivation meets them: + +- **A `\bNAME\s` sweep counts each helper's own definition line.** `lib.sh:1231` is + `check_unrunnable() {# check_unrunnable NAME REASON_CODE DETAIL` -- the trailing + USAGE COMMENT repeats the name followed by a space, so the definition matches as though + it were a call. Same shape at `lib.sh:1407`. Two more matches were ordinary prose. That + is 89 (definitions included) and 54 (comments included). +- **A command-position match misses an invocation after `&&`.** `hilbert_curve.sh:321` is + `[ -n "$_a" ] && check_unrunnable "$_a" "$2" "$3"`. Anchoring on `^` alone gives 24 for + that helper rather than 25. + +Strip trailing comments as well as whole-line ones, exclude definitions, and accept a call +after `;`, `&&` or `||`, and the number is reproducible. `refusal` moved no pair either: it is used only by `test_raises_sqlstate.py` and `test_guards_pinned.py`, neither of which has a bash twin. Its arm drives the real From 6dbbbfdc6724ebdda8d5852edde6d7d49b831910 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sun, 13 Sep 2026 08:52:05 -0600 Subject: [PATCH 6/7] docs: a number without its population is not a number (#1040) The reconciled 50 was published without saying what it counts, and I found the next trap immediately by walking the history with a recursive glob: 56. test/*.sh 265 files, what the tool grades 25 + 23 + 2 = 50 test/**/*.sh 312 files, adds the harness selftests 29 + 25 + 2 = 56 The extra 6 are all in test/selftest/, which compare_to_bash.py never reads: it grades a test/.sh against a test/pytest/test_.py, and nothing else. Neither number is wrong. A number without its population is. This is the THIRD time in one day that this quantity moved for a reason that was not the tree -- definitions counted as calls, comments counted as calls, and now a population that was never stated. TESTS.md and #1040 now name it in the same breath as the number. #1040 also gains the history, same method and same population at three revisions: v1.0-alpha3 2026-09-02 0 + 0 + 2 = 2 0cbf574 extractor 2026-09-08 21 + 0 + 2 = 23 main 2026-09-13 25 + 23 + 2 = 50 Which says two things the issue did not. The regex NEVER covered these helpers rather than drifting out of date: 0cbf574 is the commit that introduced it, and check_unrunnable already had 21 call sites that day. And check_skip went 0 -> 23 in five days, so the blind spot widens faster than the suite count. The 2026-09-02 row independently corroborates this file's own 1.0-alpha3 entry, which says check_unrunnable had no production call site and the INCOMPLETE state was unreachable. True when written; 25 call sites reach it now, none visible to the tool. That entry is left as it stands, per the scope note in docs_style.sh. docs_style.sh 14 checks PASSED. Guard job unchanged: 304 collected, 756 checks, 756 pass + 0 fail + 0 unrun. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- test/pytest/TESTS.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 0376add5..0a05140f 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -3461,10 +3461,15 @@ Individual suites define four more of their own (`check_structure`, `check_reconstruct`, `check_split_happened` in `parallel_copy.sh`, `check_float` in `parquet_export_stats.sh`), invisible to the same regex. -**50 invisible invocations**, reconciled between two agents and two independent methods, -which agree helper for helper: `check_unrunnable` 25, `check_skip` 23, +**50 invisible invocations over `test/*.sh`**, reconciled between two agents and two +independent methods, which agree helper for helper: `check_unrunnable` 25, `check_skip` 23, `check_ratio_needs_quiet_machine` 2. +**The population is half the number.** `test/*.sh` is the 265 top-level suites, which are +the only files the tool grades. Globbing `test/**/*.sh` instead adds the harness selftests +and gives **56**, the extra 6 all in `test/selftest/`, which `compare_to_bash.py` never +reads. Neither number is wrong; a number without its population is. + Getting there took four sweeps that read 89, 64, 54 and 50, and the three wrong ones were not method-sensitivity -- they were two defects, both worth knowing because any later re-derivation meets them: From 69b71e69cfb8b7c2a23b3a69dcd4ff7f1e66190f Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sun, 13 Sep 2026 09:03:26 -0600 Subject: [PATCH 7/7] docs: "no arm covers" listed two arms that cover it (#1036) A removal-proof row said "a wrong entry for a helper no arm covers (at_least)" and then named the two arms that redden. The claim meant no BEHAVIOURAL arm, which is the whole reason that mutation exists, and the row now says so. Re-measured rather than reasoned: with `"at_least": 0` added to the table, the drift guard and the whole-tree arm fail and the four behavioural arms stay green -- 2 failed, 12 passed. Found by auditing my own diff for absence-shaped claims, prompted by @OffgridwithJD correcting one of their own on #1039. Their rule is the reusable part and it is better than the correction: **"nothing checks this" is a MEASUREMENT claim that costs one grep, and it is most tempting exactly when the finding beside it is solid.** A positive result feels complete and an unmeasured sentence rides in behind it. The other three absence claims in this branch's diff were checked and stand: extras never move `rc` (measured across all seven pairs), and the tool cannot see `check_unrunnable` (measured by widening the regex). docs_style.sh 14 checks PASSED. Guard job 310 collected, 774 checks, 774 pass + 0 fail + 0 unrun. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK --- test/pytest/TESTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 9afe98db..c0fd5ade 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -3650,7 +3650,7 @@ two disagree. | drop the `refusal` entry | its own arm, and the drift guard | | `plan_marker` `None` -> `-1`, taking the key | its own arm, and the drift guard | | `cannot_run` `0` -> `-1`, taking the detail | its own arm | -| a wrong entry for a helper no arm covers (`at_least`) | the drift guard, and the whole-tree arm | +| a wrong entry for a helper no BEHAVIOURAL arm covers (`at_least`) | the drift guard, and the whole-tree arm -- the four behavioural arms stay green, which is the point of it | | add a helper to `Expect` whose name is not last | the drift guard, naming it | | revert any one of the four `*` keyword-only markers | `test_no_later_argument_can_overtake_the_name`, naming the helper |