test: port hilbert_cluster.sh to pytest, graded one-for-one (#432)

CHANGELOG.md

@@ -18,6 +18,77 @@ true until the next version shipped.
 
 ### Added
 
+- `test/hilbert_cluster.sh` has a pytest twin: the Hilbert clustering SQL surface, graded
+  one-for-one (#432).
+
+  45 collected tests, 184 checks, across the bash suite's eight arms: the surface and its
+  refusals by SQLSTATE, "it only reorders", the recorded kind, the self-gate in every
+  direction, the single-column identity, the vacuum_sorted ruling, the daemon, and the
+  enumerations. `compare_to_bash.py` reports **0 MISSING** both as the tool ships today
+  (124 bash checks) and under the widened extractor in #1044 (133). The difference is that
+  suite's nine `check_unrunnable` sites, all of which are twinned, so this is the first
+  exercise of the widening on a suite that uses those helpers correctly rather than on the
+  one where they were broken.
+
+  TWO PLACES THE PORT ASSERTS SOMETHING THE ORIGINAL GETS FOR FREE, and they are one class:
+  wherever a port replaces a structural guarantee with a procedural one, it owes an arm the
+  original does not need.
+
+  The bash suite gives the daemon's naptime and thresholds to the server through
+  `PGC_EXTRA_CONF`, so they sit in `postgresql.conf` before the postmaster starts and the
+  suite cannot run without them. `pgc_cluster.py` has no such hook, so the port sets them
+  with `ALTER SYSTEM` and a reload, which is available because all three are `PGC_SIGHUP`
+  and which can silently fail to take effect. It would fail silently in the worst way: at
+  the default naptime the daemon still acts, the poll still sees the tail fold, and every
+  arm still passes while the values the fixture claims to have set were never in force. So
+  the three are read back from the server. The same for
+  `max_parallel_workers_per_gather = 0`, which the fixture set and nothing read until the
+  parity tool reported the bash suite's own premise as missing.
+
+  A RELOAD IS NOT A READ, and this one is inherited by any later port that changes
+  postmaster-level state. `ALTER SYSTEM SET pgcolumnar.autovacuum = on`, then
+  `pg_reload_conf()`, then `SHOW` on the same connection returned `off`: a reload signals
+  the postmaster and a backend already open absorbs it at its next command boundary, and
+  this module runs every statement through one connection by design. The bash suite never
+  meets it because every `q` is a fresh `psql`, so a new session here is the port of what
+  the original gets for free rather than a workaround.
+
+  Every conditional `check_unrunnable` in the bash suite is its own test here.
+  `expect.cannot_run()` records under the reason code, so two refusals in one test would
+  collapse onto a single `UNMET_PRECONDITION` record and neither could be told from the
+  other.
+
+  THE DAEMON ARM RECORDS HOW LONG IT WAITED, not only that it succeeded. A fixture one
+  poll from its window and one fourteen from it produce identical greens, so the count is
+  the only thing that distinguishes them and drift toward the bound is otherwise
+  invisible. Measured on all five assert builds, each run on its own:
+
+      PG15  PG16  PG17  PG18  PG19
+         1     1     1     1     1     poll(s) of 15, at a 2s naptime
+
+  READ THAT FOR WHAT IT DOES NOT SAY. One poll on every major means the loop NEVER
+  WAITED: the condition was true on the first check each time, so nothing in those runs
+  exercised the waiting at all, and a loop that always succeeds on poll one is
+  indistinguishable from no loop. The distribution is a SINGLE POINT, taken on an idle
+  container, and the case the bound will actually meet is a shared and loaded CI runner --
+  this repository has that divergence recorded elsewhere as 0 in 400 idle runs against 6
+  in 400 under load. So: **one poll on each of five majors on an idle container; the bound
+  of 5 is four above the only value ever observed; no loaded measurement exists.** A reader
+  who meets a red at six is the first person to see the loop do its job, rather than
+  someone looking at a regression.
+
+  The count is printed by the fixture, and pytest captures fixture stdout on a PASSING
+  run, so `-s` is what surfaces it; on a failing run the arm's own message carries the
+  number, which is where it is needed.
+
+  This is the one part of the change with a single source of evidence: @jdatcmd reviewed
+  the rest but has no PostgreSQL on their host, said so rather than offering a reading of
+  the loop, and the five-major measurement above is the only one that exists.
+
+  What the port does NOT buy is stated in the file: over one column the Hilbert index and
+  the Morton index are both the identity, so the single-column arm is green on a relabelled
+  Z-order implementation by construction. Four arms refuse one, and no others.
+
 - An unrunnable record names the check it stands in for, so a refused check keeps
   one ledger key instead of two (#1040).
 

test/pytest/TESTS.md

@@ -82,6 +82,7 @@ behaviour, the source of that number is named.
 - [34. test_docs_stripe_floor.py: the stripe floor is below a vector](#34-test_docs_stripe_floorpy-the-stripe-floor-is-below-a-vector)
 - [35. test_projection_privilege.py: the projection read helpers are a privilege boundary](#35-test_projection_privilegepy-the-projection-read-helpers-are-a-privilege-boundary)
 - [36. test_compare_to_bash.py: the parity tool reads the NAME](#36-test_compare_to_bashpy-the-parity-tool-reads-the-name)
+- [37. test_hilbert_cluster.py: the Hilbert clustering SQL surface](#37-test_hilbert_clusterpy-the-hilbert-clustering-sql-surface)
 
 ## 1. How to read a test in here
 
@@ -3740,3 +3741,98 @@ the tool grades THIS tree.
 | `test_a_longer_helper_name_is_not_shadowed_by_a_shorter_one` | `check_ratio` must not eat `check_ratio_needs_quiet_machine` |
 | `test_the_suite_local_helpers_are_known_and_excluded` | the four suite-local helpers, and that none of their suites is graded |
 | `test_the_ported_suites_in_this_tree_are_graded_one_for_one` | the standing arm: every pair in the tree, graded |
+
+## 37. test_hilbert_cluster.py: the Hilbert clustering SQL surface
+
+The port of `test/hilbert_cluster.sh` (#432, #889's SQL half). The bash suite pins the SQL
+surface of `pgcolumnar.cluster_hilbert` and `recluster_hilbert`, the recorded
+`sorted_kind`, the self-gate and the daemon. `test/hilbert_curve.sh` pins the CURVE itself
+in C and neither file re-tests the other; this port keeps that division.
+
+**45 collected tests, 184 checks, over the bash suite's eight arms.** Graded one-for-one:
+`compare_to_bash.py` reports **0 MISSING** both as the tool ships (124 bash checks) and
+with #1044's widened extractor (133), the difference being the suite's nine
+`check_unrunnable` sites, all of which are twinned.
+
+### Where the port asserts something the original gets for free
+
+Two of them, and both are the same class: **wherever a port replaces a STRUCTURAL
+guarantee with a PROCEDURAL one, it owes an arm the original does not need** (@jdatcmd).
+
+The bash suite hands the daemon's naptime and thresholds to the server through
+`PGC_EXTRA_CONF`, so they are in `postgresql.conf` before the postmaster starts and the
+suite cannot run without them. `pgc_cluster.py` has no such hook, so the port sets them
+with `ALTER SYSTEM` and a reload -- available because all three are `PGC_SIGHUP`. That can
+silently not take effect, and then every S7 arm still passes: at default naptime the daemon
+still acts and the poll still sees the tail fold. So the three values are read back from
+the server. The arm failed on its first run with `got '2s/0.2/0.05'` -- `SHOW` returns the
+unit -- which is the cheapest demonstration that it reads the server rather than restating
+the `ALTER SYSTEM` above it.
+
+The same for `max_parallel_workers_per_gather = 0`: the fixture SET it and nothing read it
+back until the parity tool reported the bash suite's premise as MISSING.
+
+### A reload is not a read
+
+`ALTER SYSTEM SET pgcolumnar.autovacuum = on`, `pg_reload_conf()`, then `SHOW` on the same
+connection returned **`off`**. A reload signals the postmaster and a backend already open
+absorbs it at its next command boundary; this module runs every statement through ONE
+connection, by design. The bash suite never meets it because every `q` is a fresh `psql`.
+`_show_fresh()` opens a new session for post-reload reads -- the port of what the original
+gets for free, not a workaround, and inherited by any later port that changes
+postmaster-level state.
+
+### Three refusals that the port must express differently
+
+`expect.cannot_run()` records under the REASON code, so two refusals in one test collapse
+onto a single `UNMET_PRECONDITION` record. Every conditional `check_unrunnable` in the bash
+suite is therefore its own test here: S3's two curve comparisons, S4's three
+layout-after-a-gated-call arms, and S7's two daemon-layout arms.
+
+### What buys the curve, and what does not
+
+Four arms refuse a relabelled Z-order implementation, and no others: S3's two
+`three different physical orders` arms, S4(d)'s `is NOT the ZORDER rewrite`, and S7's
+`is NOT the zorder layout`. **S5 is green on a relabelled implementation by construction** --
+over one column the Hilbert index and the Morton index are both the identity -- so it buys
+the surface and the recorded kind and must never be read as evidence of Hilbertness.
+
+### Every arm
+
+| test | what it holds |
+| --- | --- |
+| `test_the_two_digests_are_the_instruments_this_file_thinks_they_are` | the ordered oracle is order-sensitive and the set oracle is not; invisible to the parity tool because the bash names live in `lib.sh` |
+| `test_parallelism_is_off_so_a_scan_order_is_a_fact_about_the_layout` | a digest is about the layout and not about scheduling |
+| `test_the_refusal_fixture_and_its_roles_are_this_runs` | the fixture holds rows, the role owns nothing, can open a session, and holds EXECUTE and schema USAGE -- so a 42501 can only be the owner check |
+| `test_each_new_verb_refuses_what_its_sibling_refuses` | four inputs x two verb pairs, each new verb's SQLSTATE compared against the established verb's on the identical input |
+| `test_the_probe_can_report_both_success_and_an_unreachable_session` | the probe can say noerror, and can say it never reached the server |
+| `test_the_reorder_fixture_is_measurable_before_anything_moves` | 20 groups, the set matches the heap mirror, the order digest is stable, the plan is the columnar scan |
+| `test_cluster_hilbert_reorders_without_changing_the_row_set` | same rows, different order -- neither half stands alone |
+| `test_physlayout_is_blind_to_an_eager_rewrite` | the instrument's limit, pinned on plain `cluster()` so it cannot be perturbed by #889 |
+| `test_the_three_fixtures_and_the_owner_role_are_measurable` | three fixtures from one generator, and the catalog really is closed to the owner |
+| `test_each_verb_moved_its_own_table_from_its_own_baseline` | without this, "different from each other" is satisfied by insert order |
+| `test_the_kind_is_recorded_and_the_owner_can_read_it` | both routes: the superuser catalog and the owner's reporter |
+| `test_the_owner_alone_can_tell_the_three_kinds_apart` | asserted as a NAMED SET, so no NULL can stand in for one |
+| `test_the_three_kinds_stand_for_three_physical_orders` | the eager verb's curve defence, UNRUN unless both legs moved |
+| `test_the_gate_is_closed_on_an_already_hilbert_table` | (a) 0 groups, kind untouched |
+| `test_the_gated_call_left_the_layout_byte_identical` | UNRUN unless the return really was 0 |
+| `test_the_same_call_does_work_once_a_tail_is_appended` | the positive control: the fifth direction, and what the daemon depends on |
+| `test_the_gate_closes_again_on_the_refolded_table` | (a3) |
+| `test_the_refolded_layout_is_byte_identical_again` | (a3), gated the same way |
+| `test_the_gate_opens_for_a_zorder_table_over_the_same_columns` | (b), with the label asserted WITH the bytes |
+| `test_plain_recluster_is_a_noop_on_a_hilbert_table` | (c) THE RULING: the curve is sticky |
+| `test_the_noop_recluster_left_the_layout_byte_identical` | (c), gated |
+| `test_a_different_key_rewrites_in_both_directions` | (d) the gate must DISCRIMINATE |
+| `test_the_online_hilbert_rewrite_is_not_the_zorder_rewrite` | the online verb's only curve defence, gated on both rewrites |
+| `test_over_one_column_both_curves_are_the_identity` | S5: surface and identity, never the curve |
+| `test_vacuum_sorted_leaves_a_hilbert_table_alone` | S6, the reading of the ruling this suite pins |
+| `test_vacuum_sorted_still_works_on_a_table_with_no_recorded_kind` | the removal proof: it must not no-op on everything |
+| `test_the_daemon_fixtures_are_built_with_the_daemon_off` | the launcher is up, the daemon is off, and the thresholds took |
+| `test_the_two_hand_driven_references_are_built_and_folded` | both references folded, and av_hi does not yet match |
+| `test_the_two_references_are_two_different_layouts` | UNRUN unless both folded |
+| `test_the_daemon_reclustered_the_decayed_hilbert_table` | THE RULING through the daemon, with its own log line naming the dispatch |
+| `test_the_layout_the_daemon_produced_is_a_hilbert_layout` | gated on av_hi not already matching |
+| `test_the_daemons_layout_is_not_the_zorder_layout` | the daemon's curve defence, gated the same way |
+| `test_the_install_script_and_the_catalog_agree_on_the_symbol_set` | S8, symbols resolved from the AS clause and never derived |
+| `test_each_new_verb_is_installed_and_its_symbol_declared` | installed once, C, and declared |
+| `test_each_new_verb_has_its_siblings_signature` | args, VARIADIC element and return type, compared against the sibling rather than retyped |

test/pytest/expected_tests.txt

@@ -113,4 +113,9 @@ guard_tests 316
 # is not IN it.
 #
 # Re-derived by collection on the merged tree: `219 tests collected`.
-cluster_tests 219
+# 219 -> 264 when test_hilbert_cluster.py landed: the port of the Hilbert clustering
+# SQL surface (#432, #889's SQL half), 45 collected tests across the bash suite's eight
+# arms. FORTY-FIVE COLLECTED, NOT FORTY-FIVE FUNCTIONS -- S1's refusal matrix is one
+# function parametrized over four inputs and two verb pairs. Re-derived by collection on
+# the merged tree, per the recipe above: `264 tests collected`.
+cluster_tests 264

test/pytest/test_compare_to_bash.py

@@ -475,6 +475,19 @@ def test_the_ported_suites_in_this_tree_are_graded_one_for_one(expect):
 
     Only the pairs that reach zero today are listed. A pair with a real gap is not pinned to
     its gap: that would turn the gap into the expected state.
+
+    THE LIST IS HAND-WRITTEN FOR THAT REASON AND NOTHING ENFORCES IT, which is a
+    different thing from the reason being wrong. The comment below already says a new
+    port belongs here; no arm reddens when one does not arrive. Today the list happens
+    to equal the pairs that exist, so nothing has ever been silently ungraded -- but an
+    eighth complete pair omitted would leave this arm passing while it graded seven,
+    which is absent-from-the-list and no-gap-found producing the same green.
+
+    #1046 TRACKS MAKING THIS ASSERTION TWO-DIRECTIONAL, in the shape `SHELL_REFERENCES`
+    already uses: derive the pairs that EXIST and require the declared list to equal that
+    set. It is not a tidy-up -- it changes what happens to an INCOMPLETE port, which is
+    quietly absent today and would have to redden, so the issue records the design
+    question rather than settling it. Kept out of the PR that added the eighth pair.
     """
     from compare_to_bash import main
     import contextlib
@@ -484,9 +497,9 @@ def test_the_ported_suites_in_this_tree_are_graded_one_for_one(expect):
     # EVERY pair in the tree. When a new port lands it belongs here, and when one
     # cannot reach zero the reason belongs in its own file rather than in an omission
     # from this list.
-    complete = ["differential", "hilbert_locality", "native_ownership",
-                "native_projection", "projection_privilege", "stats_privilege",
-                "zonemap_boundaries"]
+    complete = ["differential", "hilbert_cluster", "hilbert_locality",
+                "native_ownership", "native_projection", "projection_privilege",
+                "stats_privilege", "zonemap_boundaries"]
     verdicts = {}
     for stem in complete:
         sh, py = root / "test" / f"{stem}.sh", HERE / f"test_{stem}.py"