diff --git a/CHANGELOG.md b/CHANGELOG.md index dfe8718f..2ef4489a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,59 @@ true until the next version shipped. ### Fixed +- Hilbert clustering shipped in alpha4 and the pages a user reads never mentioned + it (#1043 for the suite header). + + `pgcolumnar.cluster_hilbert` and `pgcolumnar.recluster_hilbert` were documented + in `docs/sql-reference.md` and NOWHERE ELSE. `features.md`, `how-to.md` and + `best-practices.md` all describe clustering as Z-order and name only `cluster` + and `recluster`: + + sql-reference.md cluster_hilbert 5 hits, recluster_hilbert 2 + features.md 0 + how-to.md 0 + best-practices.md 0 + user-guide.md 0 + + So a reader following the discovery path picked Z-order and never learned the + other option existed. It is the headline item of the release whose theme is + skipping and layout, with a measured 1.24x to 2.04x advantage on range filters. + + All three pages now carry the verbs, the measured advantage, and the rule for + choosing: Z-order for point lookups, Hilbert for ranges, measure when they look + close. `how-to.md` also states that the curve is sticky ON THE SAME KEY, because + that is the part which surprises people. + + THE QUALIFIER IS THE WHOLE CLAIM AND THE FIRST DRAFT DROPPED IT. `sql-reference.md` + already said "plain `cluster` and `recluster` ON THE SAME KEY maintain that curve"; + the paraphrase written for `how-to.md` said it unconditionally. + `cluster_inherited_curve` requires `sort_key_matches`, which compares the recorded + key position by position, so the column ORDER is part of it: + + cluster_hilbert('h','a','b') sorted_kind hilbert + recluster('h','a','b') sorted_kind hilbert + recluster('h','b','a') sorted_kind zorder + + The page now carries that sequence, because a reader copying the paragraph is + exactly who needs it. Caught in review by @OffgridwithJD; the function's own + header calls the condition the thing that keeps "sticky" from meaning + "unescapable". + + AND THE SUITE HEADER TOLD A READER TO UNDO THE FEATURE (#1043). + `test/hilbert_cluster.sh` said the suite is red on purpose, that neither verb + exists, and that registering it in the matrix is future work. All three were + true when written; `4b66555` carried out every instruction in them and left the + paragraphs in place. Verified against the tree: both verbs are defined in + `pgcolumnar--1.0-alpha4.sql`, the suite is registered, and it is green on all + five majors. + + A stale instruction is worse than a stale fact, because it tells the next person + to undo what was done. The one sentence that is still true is kept, and it is the + trap the paragraphs existed to name: a shim renaming the Z-order verbs passes 162 + of 181 arms, and S5 is green on it BY CONSTRUCTION because over one column both + curves are the identity. + + No check name moves: the sorted name list hashes `72eb1c4e4f42` before and after. - `META.json` named an install script the distribution does not contain, and nothing had ever read it. diff --git a/docs/best-practices.md b/docs/best-practices.md index 07a9505f..f3cb9109 100644 --- a/docs/best-practices.md +++ b/docs/best-practices.md @@ -100,6 +100,13 @@ A runtime filter then cannot drop groups. - `pgcolumnar.cluster(table)` does the eager, one-shot reorg under `AccessExclusiveLock`. Use it for a first sort or a full rewrite in a maintenance window, not on a live hot table. +- `pgcolumnar.cluster_hilbert(table)` and `pgcolumnar.recluster_hilbert(table)` are + the same two verbs on the Hilbert curve. **Choose the curve by the predicate.** + Z-order is fine for point lookups. Hilbert has no jumps at a bit boundary, so it + keeps neighbouring keys together and a range filter reads fewer chunk groups. + Measured on 200,000 rows over two columns, Hilbert read 1.24x to 2.04x fewer + groups. The gap narrows as the query box grows, so measure your own corpus when + the two look close. - `pgcolumnar.sort_status(table)` reports how much of the table is in order. Measure before and after a sort rather than guess. @@ -118,7 +125,8 @@ still reads and filters those rows. Match the verb to the damage: | `pgcolumnar.compact_rewrite(table, min_deleted_fraction, max_groups)` | rewrites partially deleted groups to drop dead rows | `ShareUpdateExclusiveLock` | | `pgcolumnar.recluster(table)` | restores sort order online | `ShareUpdateExclusiveLock` | | `pgcolumnar.truncate(table)` | returns reclaimed end blocks to the OS | `ShareUpdateExclusiveLock` plus a brief conditional `AccessExclusiveLock` | -| `pgcolumnar.cluster(table)` / `vacuum_sorted(table)` | eager full reorg | `AccessExclusiveLock` | +| `pgcolumnar.recluster_hilbert(table)` | restores Hilbert order online | `ShareUpdateExclusiveLock` | +| `pgcolumnar.cluster(table)` / `cluster_hilbert(table)` / `vacuum_sorted(table)` | eager full reorg | `AccessExclusiveLock` | **Let the daemon carry the routine load.** Set `pgcolumnar.autovacuum = on`; it is off by default. A background worker then calls only the two online verbs, diff --git a/docs/features.md b/docs/features.md index a77d6af3..cbcaa840 100644 --- a/docs/features.md +++ b/docs/features.md @@ -166,6 +166,12 @@ coverage. (Morton) curve on several numeric columns at once. Point and range filters on more than one clustered column then skip more groups. It holds `AccessExclusiveLock`, like core `CLUSTER`, so it is an eager bulk operation. +- `pgcolumnar.cluster_hilbert(table, col [, col ...])` does the same on the + Hilbert curve instead. The Hilbert curve has no jumps at a bit boundary, so keys + close in the data stay close in storage. **Prefer it when the clustered columns + carry range filters**. Measured on 200,000 rows over two columns, it read 1.24x + to 2.04x fewer chunk groups than Z-order. The advantage is largest on the most + selective queries. Everything else matches `cluster`. ### Online reclaim and clustering @@ -179,6 +185,9 @@ and writes continue. `max_groups` bounds how many one call rewrites. - `pgcolumnar.recluster(table, col [, col ...])` re-establishes the same Z-order as `cluster` online. It is a fast no-op when the recorded key is intact. +- `pgcolumnar.recluster_hilbert(table, col [, col ...])` is the online counterpart + to `cluster_hilbert`, and the way to move a Z-ordered table onto the Hilbert + curve. - `pgcolumnar.expire(table)` drops row groups whose rows have all passed the retention declared with `set_options(ttl_column, ttl_interval)`. A group with one live row is kept whole, so nothing inside the retention is dropped. It diff --git a/docs/how-to.md b/docs/how-to.md index ebbcd5a4..03b17313 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -95,12 +95,40 @@ SELECT pgcolumnar.vacuum_sorted('events'); -- Z-order (Morton) clustering over several columns at once SELECT pgcolumnar.cluster('events', 'customer_id', 'amount'); + +-- the same, on the Hilbert curve +SELECT pgcolumnar.cluster_hilbert('events', 'customer_id', 'ts'); ``` `vacuum_sorted` sorts ascending and tightens the first column most. `cluster` uses a Z-order curve, so filters on more than one of its columns all skip more groups. +**Which curve.** `cluster_hilbert` lays the rows on the Hilbert curve instead. +That curve has no jumps at a bit boundary, so keys close in the data stay close +in storage. Range filters on the clustered columns then read fewer chunk groups. +Measured on 200,000 rows over two columns, Hilbert read 1.24x to 2.04x fewer +groups than Z-order. The advantage is largest on the most selective queries. +Take Z-order for point lookups and Hilbert for ranges, and measure your own +corpus if the two are close. + +**The curve is sticky, on the same key.** The table records which curve it was laid +on, and `pgcolumnar.sort_status` reports it as `sorted_kind`. Once a table is on the +Hilbert curve, plain `cluster` and `recluster` **over the same key** maintain that +curve rather than converting it back. + +**A different key reverts to Z-order**, and the column order is part of the key: + +```sql +SELECT pgcolumnar.cluster_hilbert('h', 'a', 'b'); -- sorted_kind: hilbert +SELECT pgcolumnar.recluster('h', 'a', 'b'); -- sorted_kind: hilbert +SELECT pgcolumnar.recluster('h', 'b', 'a'); -- sorted_kind: zorder +``` + +That is what keeps "sticky" from meaning "unescapable". To move a Z-ordered table +onto Hilbert, or to keep Hilbert while changing the key, name the verb: +`pgcolumnar.recluster_hilbert`. + `cluster` and `recluster` take at most eight key columns, and each must be a boolean, an integer, a floating-point, a `date` or a timestamp. They do not take `numeric` or `text`. Sort on a `numeric` or `text` column with diff --git a/test/hilbert_cluster.sh b/test/hilbert_cluster.sh index 6f0a7aa1..b4531684 100755 --- a/test/hilbert_cluster.sh +++ b/test/hilbert_cluster.sh @@ -129,24 +129,27 @@ # identity -- so S5 buys the surface and the recorded identity, never the curve, # and must not be counted as evidence of Hilbertness. # -# THIS SUITE IS RED ON PURPOSE UNTIL #889's SQL HALF LANDS. Neither -# cluster_hilbert nor recluster_hilbert exists yet, and the shape of the red is -# worth stating exactly, because "every arm that names one fails with 42883" is -# what a reader would otherwise assume and it is not what happens. Measured on -# PG17.10, 2026-09-09: 98 passed + 74 failed + 9 unrunnable = 181, and 27 of the -# 74 reds carry 42883. Every one of those 27 names a missing verb. THE OTHER 47 -# ARE DOWNSTREAM, and they accuse code that ships today: a fixture that could -# not be clustered makes vacuum_sorted, recluster and the daemon print exactly -# what a real defect in them would print. Read the 42883 arms first, and treat -# every other red as fixture drift until the SQL half lands. +# THE THREE PARAGRAPHS THAT STOOD HERE WERE CARRIED OUT AND LEFT BEHIND (#1043). +# They said this suite was red on purpose, that neither verb existed, and that it +# was deliberately absent from the matrix. All three were true when written. The +# commit they named, `4b66555`, did every one of the things they instructed, and +# the paragraphs stayed: # -# IT IS DELIBERATELY NOT REGISTERED in test/run_all_versions.sh: a red suite must -# not enter the matrix. THAT HAS A PRICE, and it is stated here so it is not -# rediscovered: harness_selftest.sh sweeps test/*.sh and asserts every suite is -# registered, so this file makes harness_selftest fail its registration arm -# (measured: 260 passed + 1 failed with the file present, 261 + 0 without it). -# REGISTERING THIS SUITE IN run_all_versions.sh IS PART OF THE PR THAT LANDS -# #889's SQL HALF, in the same commit that turns the suite green. +# cluster_hilbert, recluster_hilbert defined in pgcolumnar--1.0-alpha4.sql +# registered test/run_all_versions.sh +# green in the matrix, on all five majors +# +# So a reader was told the suite is red on purpose, and it is green; that the +# verbs do not exist, and they ship; and that registering it is future work, and +# it is registered. A stale instruction is worse than a stale fact, because it +# tells the next person to undo what was done. +# +# WHAT SURVIVES IS THE ONE SENTENCE THAT IS STILL TRUE, and it is the trap the +# original paragraphs existed to name: a shim that renames the Z-order verbs and +# writes sorted_kind='hilbert' reddens only four arms and passes 162 of 181. S5 +# is green on such a shim BY CONSTRUCTION, because over one column both curves +# are the identity. S5 therefore buys the surface and the recorded identity, and +# never the curve. Do not count it as evidence of Hilbertness. # # Usage: test/hilbert_cluster.sh [PG_CONFIG] # Written fresh for pgColumnar.