feat: Hilbert curve clustering, and open the 1.0-alpha4 cycle (#889)

[jdatcmd]

Closes #889. Alpha4's first feature, and it opens the 1.0-alpha4 cycle.

pgcolumnar.cluster_hilbert(regclass, VARIADIC name[])    -> void
pgcolumnar.recluster_hilbert(regclass, VARIADIC name[])  -> bigint

Five commits, red-green-refactor-red-green. Each test commit precedes the code it holds, and the
red run is in each message.

The construction

Skilling's AxesToTranspose feeding the packer that already shipped, unchanged. Skilling's
observation is that the Hilbert index is the MSB-round-first interleave of the transposed
coordinates — bit for bit the loop that was already in cluster_zorder_key. So:

Z-order:  ordinals -> pack
Hilbert:  ordinals -> transpose -> pack

Same width (8*ncols), same bytea, same memcmp-equals-curve-order invariant, so the sort, the
augmented TupleDesc, byteaLt and the writer are untouched by the choice of curve. The curve is
threaded as one parameter through the key builder, the self-gate's strcmp and the recorded
extent, so those three cannot disagree.

Two suites, and why the property arms are not enough on their own

test/hilbert_curve.sh, 184 arms, pure C against stub headers — no cluster. It carries two
deliberately wrong encoders as controls, because unit-step adjacency alone certifies a
serpentine and dyadic contiguity alone certifies Z-order, the curve that already shipped. The
battery is non-vacuous only as the pair, and each control is pinned to an exact count:

serpentine: C1 violations=0 C2 violations=0 C3 violations=621870 in 25 of 25 nested cases
Z-order:    C1 violations=0 C2 violations=2123558 in 32 of 32 multi-column cases, C3 violations=0

test/hilbert_cluster.sh, 181 arms, the SQL surface, the gate, vacuum_sorted and the daemon.

96 golden byte vectors are the only arms that pin which Hilbert curve this is. Skilling's
differs from Butz/Hamilton for ncols >= 3 and both are valid, so no property arm can separate
them and the bytes are an on-disk format commitment. They were derived from the published algorithm
and reproduced byte-for-byte by two reviewers before src/columnar_curve.c existed, which is
the only reason a golden table is worth anything.

Removal proof

Five mutants of the curve, each asserted applied (the patch raises if its anchor is missing, and a
mutation whose md5 matches the original is refused as a no-op), each restored, file verified back
at bf5f7e07105f:

mutant	passed	failed
unmutated	184	0
M1 transpose is identity — i.e. Z-order	133	51
M2 no Gray encode	133	51
M3 no final t fold	145	39
M4 invert/exchange swapped	122	62
M5 pack LSB-first in byte	69	115

M1 is the one that matters: neutering the transpose turns this back into the curve that already
ships, and 51 arms notice.

The owner ruling: the curve is sticky

sorted_kind is the table's declared intent. Plain recluster() on a Hilbert table with a matching
key is a no-op, not a conversion; switching curves requires naming the other verb;
vacuum_sorted skips a Hilbert table rather than re-sorting it lexicographically; the daemon
dispatches on the recorded kind.

Stated rather than hidden: the daemon's dispatch is not load-bearing against this suite.
Reverting it to the hard-coded recluster() leaves all 181 arms green, because the sticky plain
recluster() already preserves the curve. It is kept because the ruling is about the daemon and
should be readable at the call rather than inferred from a rule two modules away — but no arm
guards it, and one that did would have to read the curve out of the daemon's log line, which the
log now carries.

Why this opens 1.0-alpha4

The first implementation put both functions in pgcolumnar--1.0-alpha2--1.0-alpha3.sql.
v1.0-alpha3 is a published pre-release (2026-09-03), so that file is a shipped artifact:
editing it leaves a database that upgraded from the release and a fresh install both reporting
1.0-alpha3 with different function sets and no path between them.

VERSION and pgcolumnar.control          -> 1.0-alpha4
pgcolumnar--1.0-alpha3.sql              -> pgcolumnar--1.0-alpha4.sql
pgcolumnar--1.0-alpha3--1.0-alpha4.sql     new, carrying the two verbs
pgcolumnar--1.0-alpha2--1.0-alpha3.sql     reverted, byte-identical to the branch point

And the newly-previous version gained coverage it never had.
test/native_upgrade_converge.sh staged frozen base scripts for 1.0-alpha and 1.0-alpha2 only,
so a database at the version people actually have had no arm. The shipped alpha3 base script is
now frozen into test/fixtures/ taken from the tag (diff against git show v1.0-alpha3: is
empty, and it contains neither new verb). 8 checks -> 11.

My first edit there added it to the staging loop alone and the count stayed at 8 — a silent no-op,
caught by watching the number rather than the exit status. The arms come from a second loop.

Removal proof: renaming cluster_hilbert in the new upgrade script reddens 9 of 11 arms
including all three 1.0-alpha3 ones, with
function pgcolumnar.cluster_hilbert(regclass, name[]) does not exist.

Verification, on the rebased tree

COPT=-Werror, PG 18.4, build OK with 0 warnings, 0 FAIL lines:

hilbert_cluster  181   projection_rewrite  84   native_upgrade_converge  11
hilbert_curve    184   projections         73   harness_selftest        261
native_cluster    11   recluster_gate      13   entry_point_privilege    31
native_recluster  12   recluster_extent    17   autovacuum               11
sorted_pathkeys  113   vacuum_sorted_gate  34   docs_style           PASSED

projection_rewrite and projections are #892's, run here because this rebase composes two
changes to the maintenance paths and a PR describes its branch, not the composition.

What this does NOT deliver

No locality measurement. The release plan asserts Hilbert "gives better locality" and cites no
measurement of ours; nothing here supplies one, and #889 records why the obvious fixture cannot:
on a dense power-of-two-aligned grid the two curves produce the identical row-to-group
partition and the measured skip ratio is 1.0000. A pilot needs a sparse, skewed, non-dyadic
fixture and must assert the two curves partition differently before any ratio means anything. If it
cannot separate them, "not measurable on our fixtures" is the result to report rather than a
threshold to lower.

The ordinal layer also caps the benefit and is deliberately untouched: matched int4 x int4 gives
mean contiguous run 7.14 vs 12.67, but bool x int4 gives 8.07 vs 8.10 — no win at all.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK

[OffgridwithJD]

Approving at 4b665553. 12/12, and I read the breakdown rather than the colour: {"SUCCESS":12}, nothing neutral or skipped inside a zero-failure count.

I verified the load-bearing claims myself rather than reading the report, because this PR opens a release cycle and adds an external source category — the two places where "it passes" is the weakest evidence.

What I ran

The battery discriminates. This is the question that matters for a second curve: does it distinguish Hilbert from the Z-order that already ships, or would it approve whatever it is handed? I applied M1 myself — neuter cluster_hilbert_transpose to the identity, which turns the curve back into Z-order — and asserted the mutation applied rather than assuming:

baseline    184 passed +  0 failed
M1          133 passed + 51 failed
restored    byte-identical, mutation gone, tree clean

51 arms notice. That reproduces your number exactly and it is the strongest thing in the PR.

The four "the curve is sticky" claims in docs/sql-reference.md all hold, with the two controls that stop them being vacuous:

claim	measured
plain cluster() on a Hilbert table maintains Hilbert	hilbert → hilbert
plain recluster() likewise	hilbert
cluster_hilbert on a Z-ordered table converts it	zorder → hilbert
vacuum_sorted leaves a Hilbert table alone	hilbert
control: a Z-order table still reports zorder	zorder — so sorted_kind is not a constant
control: the curves lay rows differently	hilbert 15035,9021,12028,6014,18042 vs zorder 9021,15035,6014,18042,12028

That last one is the one I would keep. Same rows, different order, so the Hilbert path is not silently falling back.

The upgrade path. test/native_upgrade_converge.sh is modified in the same PR that adds a version, which is exactly where a vacuous convergence check would hide. It is right: 1.0-alpha3 is added to both the fixture list and the upgrade loop. The new fixture shows +0/-0, which is a rename, and its blob sha at your head is identical to pgcolumnar--1.0-alpha3.sql at the released v1.0-alpha3 tag (c7269cf1e242…). So the arm upgrades from what actually shipped, not from a working copy.

The provenance claim is accurate. columnar_curve.c says Skilling's AxestoTranspose is public domain. I checked the source rather than taking it: the listing carries an explicit PUBLIC-DOMAIN SOFTWARE notice. No licensing problem.

One request, procedural rather than blocking

PROVENANCE.md's implementer rules say to build only from the spec and the public PostgreSQL API, and its Log is where provenance decisions get recorded. This PR introduces a source category the Log has no precedent for — a published algorithm listing, transcribed with parameters fixed and the structure unchanged, by the file's own description.

The licensing is fine and I am not asking you to rewrite the function. I am asking for a Log entry, because the value of that document is that a future reader can tell why this was acceptable without re-deriving it. Right now that reasoning lives only in a source comment, and the file that exists to hold it is silent.

One question

av_maintain_one now reads sorted_kind by joining maintenance_due against sort_status in one query. sort_status is RETURNS record, not SETOF, so the join cannot go empty — I checked, since an empty right side would have made SPI_processed == 1 fail and silently skipped compaction too, not just the curve dispatch.

What remains is a coupling: the daemon's compaction now depends on sort_status succeeding, where before it depended only on maintenance_due. If there is a state where sort_status raises and maintenance_due would not, that table stops being maintained at all, quietly. Is there an arm for that, or is it unreachable for a relation the daemon iterates?

One observation that is NOT yours

The 1.0-alpha and 1.0-alpha2 fixtures do not match their released tags — v1.0-alpha shipped no pgcolumnar--1.0-alpha.sql at all, only pgcolumnar--1.0-dev.sql, and the 1.0-alpha2 fixture differs in both sha and size from the tag's. Both are byte-identical between main and your head, so this predates you and is not this PR's to answer. Say the word and I will file it; the convergence arms for those two versions may be upgrading from a reconstructed base rather than a released one.

On the daemon fix

a Hilbert table was reclustered by the Z-order verb and came back relabelled 'zorder' -- the daemon converting a layout the user chose, on a timer, with nothing in the log to say so

That is the best find in the PR and it is the kind that only shows up when you compose two features. Writing the dispatch out explicitly even though recluster() already preserves the curve is the right call for the reason you give: the ruling should be readable at the call it governs, not two modules away.