feat: Hilbert curve clustering, and open the 1.0-alpha4 cycle (#889)

CHANGELOG.md

@@ -2,20 +2,72 @@
 
 All notable changes to pgColumnar are recorded here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). pgColumnar is
-pre-release; the version marker is `1.0-alpha3`, recorded in `VERSION`. New tables
+pre-release; the version marker is `1.0-alpha4`, recorded in `VERSION`. New tables
 are written in the native on-disk format, PGCN v1. For the forward-looking plan see
 [design/ROADMAP.md](design/ROADMAP.md); for full history see the git log.
 
-The extension's `default_version` is `1.0-alpha3`, which is tagged as
-`v1.0-alpha3` and is the latest published pre-release. Upgrade scripts from
+The extension's `default_version` is `1.0-alpha4`, which is in development and not
+yet tagged; `v1.0-alpha3` is the latest published pre-release. Upgrade scripts from
 every previously shipped version ship with it (`1.0-dev`, which the v1.0-alpha tag
-installed, `1.0-alpha`, and `1.0-alpha2`), so a single
-`ALTER EXTENSION pgcolumnar UPDATE` reaches `1.0-alpha3` from any of them. Older
+installed, `1.0-alpha`, `1.0-alpha2`, and `1.0-alpha3`), so a single
+`ALTER EXTENSION pgcolumnar UPDATE` reaches `1.0-alpha4` from any of them. Older
 notes in this file describe `default_version` as pinned at an earlier version, each
 true until the next version shipped.
 
 ## [Unreleased]
 
+### Added
+
+- Hilbert clustering: `pgcolumnar.cluster_hilbert` and
+  `pgcolumnar.recluster_hilbert` (#889).
+
+  **Two verbs rather than a parameter on the existing two.** PostgreSQL refuses
+  to extend `cluster(regclass, VARIADIC name[])` in either direction: a defaulted
+  parameter cannot precede a `VARIADIC` one, and an array-plus-kind overload
+  makes the documented `cluster('t','a','b')` call ambiguous. Both were measured
+  on 18.4. The new verbs match their siblings element for element in argument
+  types, variadic element type, return type and volatility, so a caller switches
+  between them by name alone.
+
+  **What the curve buys.** Z-order jumps a long way in key space at a bit
+  boundary; a Hilbert curve does not. Keys that are close in the data therefore
+  stay closer in storage, the min/max zone maps over the clustered columns are
+  tighter, and a range filter reads fewer chunk groups. The key is the same width
+  and sorts through the same `bytea` comparator, so nothing downstream of the
+  sort knows which curve produced it.
+
+  **The curve is sticky.** `sorted_kind` is the table's declared intent, not a
+  property of each call:
+
+  - plain `recluster` on a Hilbert table over the same key is a no-op returning
+    0, not a silent conversion back to Z-order;
+  - `recluster_hilbert` on a Z-ordered table over the same columns rewrites it;
+  - `vacuum_sorted` leaves a Hilbert table alone rather than sorting it
+    lexicographically and relabelling it;
+  - the maintenance daemon dispatches on the recorded kind, so a Hilbert table
+    is re-clustered with Hilbert instead of being converted on a timer;
+  - naming the other verb, or reclustering on a different key, is how a table
+    changes curve.
+
+  Held by `test/hilbert_cluster.sh` (181 arms) over the SQL surface, the recorded
+  kind, both self-gates and the daemon, and by `test/hilbert_curve.sh` (184 arms)
+  over the encoder itself.
+
+- `test/projection_rewrite.sh`, 84 checks. Nothing in the tree asserted that a
+  projection answers after a rewrite, which is why this was silent.
+
+  Every arm compares a `pgc_set_hash` of `read_projection` against the base table
+  rather than checking that the call did not raise, so a projection re-recorded
+  EMPTY fails -- which matters because the correct end state after a bare
+  `TRUNCATE` is an empty projection that answers. Every arm also asserts what its
+  operation DID (`REWROTE`, `NOOP` or `FAILED`) and reports its properties as
+  `UNMET_PRECONDITION` rather than as passes when it did not: an operation that
+  failed or no-opped leaves the storage id unchanged and `read_projection`
+  answering, which is indistinguishable from a path that handles projections
+  correctly. Three arms carry `pgcolumnar.vacuum`, `vacuum_sorted` and `cluster`,
+  which already re-record for themselves, so a future fix moved into the table-AM
+  callback reddens here instead of double-recording.
+
 ### Fixed
 
 - `ALTER TABLE ... RENAME COLUMN` now carries the new name into
@@ -139,23 +191,6 @@ true until the next version shipped.
   now re-records. It names the two cases that remain: a declaration that no longer
   resolves, and the implicit base projection, which is not readable by name at all.
 
-### Added
-
-- `test/projection_rewrite.sh`, 84 checks. Nothing in the tree asserted that a
-  projection answers after a rewrite, which is why this was silent.
-
-  Every arm compares a `pgc_set_hash` of `read_projection` against the base table
-  rather than checking that the call did not raise, so a projection re-recorded
-  EMPTY fails -- which matters because the correct end state after a bare
-  `TRUNCATE` is an empty projection that answers. Every arm also asserts what its
-  operation DID (`REWROTE`, `NOOP` or `FAILED`) and reports its properties as
-  `UNMET_PRECONDITION` rather than as passes when it did not: an operation that
-  failed or no-opped leaves the storage id unchanged and `read_projection`
-  answering, which is indistinguishable from a path that handles projections
-  correctly. Three arms carry `pgcolumnar.vacuum`, `vacuum_sorted` and `cluster`,
-  which already re-record for themselves, so a future fix moved into the table-AM
-  callback reddens here instead of double-recording.
-
 ## [1.0-alpha3] - 2026-09-02
 
 ### Added

Makefile

@@ -17,6 +17,7 @@ OBJS = \
 	src/columnar_customscan.o \
 	src/columnar_vector.o \
 	src/columnar_vacuum.o \
+	src/columnar_curve.o \
 	src/columnar_unique.o \
 	src/columnar_row_lock.o \
 	src/columnar_arrow.o \
@@ -39,7 +40,7 @@ OBJS = \
 	src/columnar_autovacuum.o
 
 EXTENSION = pgcolumnar
-DATA = pgcolumnar--1.0-alpha3.sql pgcolumnar--1.0-dev--1.0-alpha.sql pgcolumnar--1.0-alpha--1.0-alpha2.sql pgcolumnar--1.0-alpha2--1.0-alpha3.sql
+DATA = pgcolumnar--1.0-alpha4.sql pgcolumnar--1.0-dev--1.0-alpha.sql pgcolumnar--1.0-alpha--1.0-alpha2.sql pgcolumnar--1.0-alpha2--1.0-alpha3.sql pgcolumnar--1.0-alpha3--1.0-alpha4.sql
 PGFILEDESC = "pgColumnar - column-oriented table access method"
 
 # make installcheck. Not the project's gate -- that is test/run_all_versions.sh,

PROVENANCE.md

@@ -587,3 +587,42 @@ oracle, so none changes query results.
   (`hits.tsv.gz`) is downloaded for local measurement and is not redistributed;
   its own licensing is unestablished and it must not be added to the tree without
   one.
+
+- 2026-09-09. Hilbert curve clustering (#889) introduced a source category this
+  document had no precedent for, so the determination is recorded here rather
+  than left in a code comment.
+
+  `src/columnar_curve.c`'s `cluster_hilbert_transpose` is a transcription of
+  `AxestoTranspose` from J. Skilling, "Programming the Hilbert curve", AIP
+  Conference Proceedings 707 (2004), whose published listing carries an explicit
+  public-domain notice. Transcribed with the bit count fixed at 64 and the
+  coordinate type fixed at `uint64`; the structure of the algorithm is
+  unchanged. The provenance was checked against the published listing by
+  OffgridwithJD during the #899 review; I have not obtained the paper myself and
+  am recording their verification rather than a second one.
+
+  Why this is not the rule at the top of this file being bent. "Build only from
+  the specification and the public PostgreSQL API" exists to keep another
+  COLUMNAR ENGINE's source out of this tree, which is a competitive and
+  copyleft-contamination concern. A published, public-domain algorithm from the
+  academic literature is neither. The same reasoning already covers the codecs:
+  pglz, lz4 and zstd are used through their public APIs, and the min/max skip
+  list is long-standing prior art recorded as such above.
+
+  What was NOT done, stated so the boundary stays where it is. No other
+  implementation of a Hilbert curve was read or consulted -- not a library, not
+  another database, not published source beyond the paper's own listing. The
+  correctness evidence is property-based rather than comparative: 184 arms in
+  `test/hilbert_curve.sh` establish that the index set is exactly the contiguous
+  range, that consecutive indices are unit-adjacent, and that every dyadic
+  sub-cube occupies a contiguous run, with a serpentine and a Z-order encoder
+  carried as deliberately wrong controls because neither property alone
+  separates a Hilbert curve from those two.
+
+  One consequence worth recording for a future reader. Hilbert curves are not
+  unique above two dimensions, and this construction differs from the
+  Butz/Hamilton curve for three or more clustering columns; both are valid. So
+  the key bytes are an ON-DISK FORMAT COMMITMENT rather than an implementation
+  detail, disagreement with another library's Hilbert index is not evidence of a
+  defect, and the 96 golden byte vectors in that suite are what pin which curve
+  this is.

README.md

@@ -24,7 +24,7 @@ append-mostly data.
 pgColumnar builds from one source tree on PostgreSQL 15 through 18, with 19
 validated against 19beta2, and is
 licensed under the [MIT License](LICENSE). It is [pre-release](docs/limitations.md#release-status); the version marker
-is `1.0-alpha3`, recorded in `VERSION`. That version is in development and not
+is `1.0-alpha4`, recorded in `VERSION`. That version is in development and not
 tagged; the latest published pre-release is `v1.0-alpha2`. A table `USING pgcolumnar` is stored in the
 native on-disk format, PGCN v1.
 

VERSION

@@ -1 +1 @@
-1.0-alpha3
+1.0-alpha4

docs/limitations.md

@@ -2,7 +2,7 @@
 
 ## Release status
 
-pgColumnar is pre-release. The version marker is `1.0-alpha3`, recorded in `VERSION`,
+pgColumnar is pre-release. The version marker is `1.0-alpha4`, recorded in `VERSION`,
 and it is tagged `v1.0-alpha3`.
 
 On PGXN the same release is `1.0.0-alpha.3`. The two differ because PGXN requires a

docs/roadmap.md

@@ -12,7 +12,7 @@ Issues are the authority on anything being worked now.
 ## Status
 
 pgColumnar is [pre-release](limitations.md#release-status). The version marker is
-`1.0-alpha3`, recorded in `VERSION`. That version is in development and not tagged; the
+`1.0-alpha4`, recorded in `VERSION`. That version is in development and not tagged; the
 latest published pre-release is `v1.0-alpha2`. A table `USING pgcolumnar` is stored in the
 native on-disk format, PGCN v1.
 

docs/sql-reference.md

@@ -142,6 +142,25 @@ to reorder a live table without an exclusive lock.
 SELECT pgcolumnar.cluster('events', 'customer_id', 'ts');
 ```
 
+### pgcolumnar.cluster_hilbert(tablename regclass, VARIADIC columns name[])
+
+The same reorganisation on the Hilbert curve instead of the Z-order one. Take it
+when the clustered columns carry range filters. The Hilbert index has no jumps at
+a bit boundary. Keys that are close in the data therefore stay close in storage,
+and a range filter reads fewer chunk groups. Everything else matches `cluster`:
+the same arguments, the same refusals, the same `AccessExclusiveLock`.
+
+```sql
+SELECT pgcolumnar.cluster_hilbert('events', 'customer_id', 'ts');
+```
+
+**The curve is sticky.** The table records which curve it was laid on, and
+`pgcolumnar.sort_status` reports it as `sorted_kind`. Once a table is on the
+Hilbert curve, plain `cluster` and `recluster` on the same key maintain that
+curve rather than converting it back. `vacuum_sorted` leaves the table alone,
+and the maintenance daemon re-clusters it with Hilbert. To switch curves, name
+the other verb, or recluster on a different key.
+
 ### pgcolumnar.recluster(tablename regclass, VARIADIC columns name[]) returns bigint
 
 The online counterpart to `cluster`. Re-establishes the same Z-order clustering
@@ -159,6 +178,20 @@ without rewriting anything when the recorded key matches, the kind is Z-order,
 and the existing sorted run already covers every row group. This is what lets the
 maintenance daemon call it on a schedule without churning storage.
 
+On a table laid on the Hilbert curve over the key you name, `recluster`
+maintains that curve. It does not convert the table to Z-order.
+
+### pgcolumnar.recluster_hilbert(tablename regclass, VARIADIC columns name[]) returns bigint
+
+The online counterpart to `cluster_hilbert`, and the way to move a Z-ordered
+table onto the Hilbert curve. Same arguments, same lock and same return value as
+`recluster`; it re-establishes Hilbert clustering rather than Z-order, and
+records the curve it applied.
+
+```sql
+SELECT pgcolumnar.recluster_hilbert('events', 'customer_id', 'ts');
+```
+
 ### pgcolumnar.compact(tablename regclass) returns bigint
 
 Retires row groups that are fully deleted, dropping their metadata so scans skip

pgcolumnar--1.0-alpha3--1.0-alpha4.sql

@@ -0,0 +1,38 @@
+/*
+ * pgcolumnar--1.0-alpha3--1.0-alpha4.sql
+ *
+ * Upgrade from 1.0-alpha3 to 1.0-alpha4.
+ *
+ * 1.0-alpha3 is a PUBLISHED pre-release (tag v1.0-alpha3, 2026-09-03), so
+ * pgcolumnar--1.0-alpha2--1.0-alpha3.sql is a shipped artifact and must not
+ * change. Adding these functions there would leave two databases both reporting
+ * 1.0-alpha3 with different function sets and no upgrade path between them --
+ * exactly what extension versioning exists to prevent.
+ */
+
+
+-- The two Hilbert clustering verbs (#889). New functions, so plain CREATE: an
+-- alpha2 install has neither name. They mirror cluster() and recluster()
+-- element for element in argument types, variadic element type, return type and
+-- volatility, because the two pairs are one surface and a caller switches
+-- between them by name alone.
+
+CREATE FUNCTION pgcolumnar.cluster_hilbert(
+	tablename regclass,
+	VARIADIC columns name[])
+	RETURNS void
+	LANGUAGE C
+	AS 'MODULE_PATHNAME', 'pgcolumnar_cluster_hilbert';
+
+COMMENT ON FUNCTION pgcolumnar.cluster_hilbert(regclass, name[])
+	IS 'eager reorg on the Hilbert curve: as cluster(), but the rows are ordered by the Hilbert index over the given columns, which keeps neighbouring keys neighbouring in storage more tightly than Z-order does. Holds AccessExclusiveLock like CLUSTER/VACUUM FULL; the online counterpart is recluster_hilbert() (#889)';
+
+CREATE FUNCTION pgcolumnar.recluster_hilbert(
+	tablename regclass,
+	VARIADIC columns name[])
+	RETURNS bigint
+	LANGUAGE C
+	AS 'MODULE_PATHNAME', 'pgcolumnar_recluster_hilbert';
+
+COMMENT ON FUNCTION pgcolumnar.recluster_hilbert(regclass, name[])
+	IS 'lazy online reclustering on the Hilbert curve: as recluster(), but re-establishes Hilbert clustering over the given columns under ShareUpdateExclusiveLock (concurrent reads and writes). The curve is sticky -- plain recluster() maintains a Hilbert table rather than converting it, and naming this verb is how a Z-ordered table is switched (#889)';