Skip to content

Hilbert curve clustering: a second key kind for cluster and recluster (1.0-alpha4) #889

Description

@jdatcmd

Alpha4's first item, from design/RELEASE_PLAN_1.0.md:96. Target 2026-09-15.

Why it is this release or 2.0. Clustering by a new key kind is user-visible surface, and the
plan's governing rule is that beta 1 takes no new surface. There is no later alpha to catch it.

Everything below marked verified was read or run against main 9628414 today. The surface
probes were run on PostgreSQL 18.4. Nothing here restates the release plan as fact.

What already exists, and exactly where Hilbert would attach

Verified. Z-order clustering ships and the whole mechanism is four steps in
src/columnar_vacuum.c:

  1. cluster_type_ordinal(value, typid) maps one column value to an order-preserving uint64.
  2. cluster_zorder_key() (:1046) interleaves the per-column ordinals MSB-first into an
    ncols * 8-byte bytea, so that memcmp order over the bytea equals Z-order.
  3. The rewrite sorts on that bytea as an appended __zorder attribute (:687, :714).
  4. record_sorted_extent(..., "zorder") (:772, :1564) records the key kind in the storage
    catalog (Anum_native_storage_sorted_kind, src/columnar_metadata.c:80).

So the swap point is step 2 alone. A Hilbert build takes the same ord[] array and produces
a different bytea whose memcmp order is Hilbert order. Steps 1, 3 and 4 are untouched apart
from the recorded kind string. The release plan's "the same machinery" is accurate, and this is
the line that makes it so.

Verified — the supported key types are fixed by step 1 and Hilbert inherits them exactly:
bool, int2, int4, int8, date, timestamp, timestamptz, float4, float8
(cluster_type_supported, src/columnar_vacuum.c:1020). Anything else already raises 0A000 with a hint
naming Z-order, and that hint's wording will need to stop being Z-order-specific.

The surface question is not a matter of taste, and two of the three options are already ruled out by measurement

pgcolumnar.cluster and pgcolumnar.recluster both take (tablename regclass, VARIADIC columns name[])
(pgcolumnar--1.0-alpha3.sql:939, :976). Neither takes a key kind. Verified: that signature
cannot be extended in either direction.
Run on 18.4:

CREATE FUNCTION probe_a(t regclass, kind text DEFAULT 'z', VARIADIC cols name[]) ...
ERROR:  input parameters after one with a default value must also have defaults

CREATE FUNCTION probe_b(t regclass, VARIADIC cols name[], kind text DEFAULT 'z') ...
ERROR:  VARIADIC parameter must be the last input parameter

Verified: adding an array-plus-kind overload beside the variadic one breaks the existing call
style.
With both pc(regclass, VARIADIC name[]) and pc(regclass, name[], text DEFAULT ...)
defined, the call every current user writes stops resolving:

SELECT pc('probe_t','a','b');
ERROR:  function pc(unknown, unknown, unknown) is not unique
HINT:  Could not choose a best candidate function.

The array forms still resolve; the bare-name form does not. So overloading is not the
backward-compatible option it looks like.

That leaves three real choices, and this issue should not pick one silently:

option cost
separate pgcolumnar.cluster_hilbert / recluster_hilbert doubles the clustering surface; no ambiguity, no break
a GUC, e.g. pgcolumnar.cluster_kind no signature change, but makes the key kind ambient state for something we record per storage as sorted_kind
replace the variadic signature with (regclass, name[], text DEFAULT 'zorder') one clean surface; breaks every existing cluster('t','a','b') call, in an alpha

My recommendation is the first, on the grounds that the key kind is a property of the rewrite
being asked for and not of the session, but the owner should rule. Nothing else in this issue
depends on which is chosen
, so the implementation can start on the key function while the
surface is decided.

Build it test-first

Write each arm, run it, and confirm it fails for the intended reason before writing the
production code.

Arm 1 — correctness before locality. A Hilbert-clustered table returns exactly the rows a
heap oracle returns, full-table and under range predicates on each key column. This is the arm
that must exist before any locality claim, because a key function that is merely a permutation
of the rows will pass every locality measurement and lose data. Red first with 42883
(function does not exist), asserted as SQLSTATE, not as error text.

Arm 2 — the property the feature is for, measured as work and not as intent. Hilbert exists
because it gives better locality than Z-order on the same machinery. The instrument must be
chunk groups touched by a range query, read from the skipping counters, not wall clock and
not a timing ratio — this box is not idle and a millisecond count here would measure the load,
not the layout. Compare Hilbert against Z-order on the same fixture, same key columns, same row
count.

Arm 3 — the negative control that makes arm 2 mean something. On a one-column key, Hilbert
and Z-order are the same curve and must touch the same number of groups. An arm 2 that reports
an improvement there is measuring something other than the curve.

Arm 4 — the kind is recorded and is not confusable with Z-order.
src/columnar_vacuum.c:646 reuses a recorded sorted run only when
strcmp(skind, "zorder") == 0. A Hilbert run must record 'hilbert' and must not satisfy
that test, or a Hilbert-ordered extent will be reused as if it were Z-ordered. Assert both
directions: the kind is 'hilbert' in pgcolumnar.stats, and a Hilbert extent does not take
the Z-order reuse path.

The removal proof for all of it. Delete the Hilbert key construction and keep the
Z-order one; arms 1, 2 and 4 must go red. If arm 2 stays green with the curve gone, it was
measuring the sort and not the curve, which is the failure mode this project keeps rediscovering.

State of knowledge

  • Verified: the Z-order mechanism, the swap point, the supported key types, the recorded
    kind, the two signature refusals, and the overload ambiguity.
  • Assumed: that a Hilbert index over ncols 64-bit ordinals is practical at this key width.
    A d-dimensional Hilbert index over 64-bit-per-dimension coordinates is a well-understood
    construction, but the ordinals here are full-width uint64 and the output must stay a
    memcmp-ordered bytea. Confirm the bit width before writing the loop.
  • Unknown: how much better the locality actually is on our fixtures. The release plan says
    "better locality" and cites no measurement of ours. Arm 2 is what turns that into a number,
    and if the number is not an improvement, that is a result and this item should be reported
    rather than shipped quietly.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions