From fced490880058793106433609e8fd334abd09003 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Tue, 8 Sep 2026 18:00:15 -0600 Subject: [PATCH 1/6] test: the Hilbert curve battery, red before the curve exists (#889) A standalone C battery over the key construction #889 will add. It needs no cluster: it writes C, compiles it, runs it, and turns the program's output into check() arms. It is deliberately NOT registered in run_all_versions.sh, because a red suite in the matrix is everyone's problem; it joins the matrix in the commit that adds src/columnar_curve.c. Red now, and red for the right reason: 174 battery arms report "the code under test is absent: src/columnar_curve.c src/columnar_curve.h". The 10 INSTRUMENT arms pass, because they need nothing from src/ -- they are what proves the battery can tell curves apart before there is a curve to test: serpentine: C1 violations=0 C2 violations=0 C3 violations=621870 in 25 of 25 Z-order: C1 violations=0 C2 violations=2123558 in 32 of 32, C3 violations=0 That pair is the point. Unit-step adjacency alone certifies a serpentine, which is not a Hilbert curve. Dyadic contiguity alone certifies Z-order, which is the curve that already ships. The battery is non-vacuous only as C2 AND C3, and each control is pinned to an exact count rather than a bound, so a checker stuck at pass and a checker stuck at fail both redden the suite. The first version of this file was worse than it looked, which is why the arms were audited before being trusted. Three adversarial passes compiled thirteen wrong encoders against it, and TWO OF THEM PASSED 138 of 138 -- one differing from the correct key on 95.3% of random 8-column inputs. The causes were specific and are fixed: - C4b drew key bytes from the low byte of a 32-bit LCG, whose period there is 512. It claimed 24,000 round trips per ncols and performed 8 to 64 distinct keys, the same 8 keys 375 times each at ncols=8. Now splitmix64, and the premise arm asserts the DISTINCT keys the program counted rather than the iteration count -- a coverage number the harness could not manufacture. - Four arms never called the code under test at all, and stayed green with both product functions destroyed. Each is now composed with the encoder rather than deleted. - C1 had no negative control, though the header claimed every property had one, so nothing in the suite could make the permutation checker report a violation. A truncating control now does. The 96 golden byte vectors are the only arms that pin WHICH Hilbert curve this is -- Skilling's differs from Butz/Hamilton for ncols >= 3 and both are valid, so the property arms cannot distinguish them and the bytes are a format commitment. They were derived from the published AxestoTranspose, independently reproduced byte-for-byte by two auditors from their own transcriptions, and the small cases hand-checked. The code under test does not exist, so they could not have been generated from it. What is NOT claimed: that the battery catches every wrong encoder. It now catches the three blind spots that were measured, and the mutant matrix moved from 138/0 to 181/3 on the worst of them. That number is the reviser's and I have not reproduced it -- it needs a correct encoder to run against, which arrives with the implementation, and I will check it then rather than assert it now. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/hilbert_curve.sh | 1899 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1899 insertions(+) create mode 100755 test/hilbert_curve.sh diff --git a/test/hilbert_curve.sh b/test/hilbert_curve.sh new file mode 100755 index 00000000..00a87873 --- /dev/null +++ b/test/hilbert_curve.sh @@ -0,0 +1,1899 @@ +#!/usr/bin/env bash +# +# pgColumnar Hilbert clustering curve battery (issue #889). +# +# WHAT THIS SUITE IS FOR +# +# #889 replaces the Z-order clustering key with a Hilbert one. A space-filling +# curve is exactly the kind of code that is easy to get almost right: a wrong +# curve still returns a number for every point, still sorts, still clusters +# something, and still makes a benchmark look better than no clustering at all. +# Nothing downstream of it can tell. So the curve is pinned here, directly, by +# its mathematical properties and by frozen bytes, rather than inferred from a +# query plan or a page count. +# +# THE DEFECT THIS WOULD HAVE CAUGHT +# +# An encoder that truncates, or that mixes up the bit order, is injective on the +# subset anyone happens to sample. It passes a "no collisions" test. It fails +# arm C1 here, which asserts the encoded indices are EXACTLY the contiguous +# range [0, 2^(ncols*b)) -- every value hit once, none left over. +# +# A serpentine (boustrophedon) scan is unit-step adjacent everywhere and is not +# a Hilbert curve: its dyadic sub-cubes are scattered, so a range query over a +# sub-cube reads runs from all over the file. It passes an adjacency test alone. +# +# Z-order -- what ships today -- has perfect dyadic locality and jumps a long +# way at every quadrant boundary. It passes a contiguity test alone. +# +# So neither property alone certifies a Hilbert curve, and the battery is only +# non-vacuous as C2 AND C3 together. Both wrong encoders are compiled and run in +# arm C3 as named controls, and the suite asserts their verdicts explicitly, in +# both directions. If either control ever behaves the way the real encoder +# should, the instrument is broken and this suite says so instead of passing. +# +# A third control encoder, TRUNCATE, exists for C1, which had none: the +# serpentine and Z-order are both permutations, so neither can make the +# permutation checker report a violation, and until TRUNCATE was added nothing +# in this file proved C1 could go red at all. +# +# THE CONTROL ARMS ARE CALIBRATION, NOT COVERAGE. controls.c does not link +# src/columnar_curve.c, by design, and with that file deleted the control arms +# still print PASS. Their names are prefixed INSTRUMENT and the run prints the +# instrument and battery counts separately, so nobody counts them as evidence +# about the encoder. +# +# WHY THIS IS A STANDALONE C BATTERY +# +# The curve is a pure function of ncols uint64 words. It needs no cluster, no +# catalog and no rows, and running it through SQL would put an entire storage +# engine between the assertion and the thing asserted. So the suite writes C to +# a temp directory, compiles it, runs it, and turns the program's output into +# check() arms. test/harness_selftest.sh and test/docs_style.sh are the +# precedent for a suite that skips pgc_setup; test/build_san.sh is the precedent +# for a suite that compiles C. +# +# THE INTERFACE UNDER TEST +# +# void cluster_hilbert_transpose(uint64 *X, int ncols); +# void cluster_pack_interleave(const uint64 *ord, int ncols, +# unsigned char *out); +# +# The Hilbert key is: ordinals -> transpose -> pack. The Z-order key is: +# ordinals -> pack. Output is exactly 8*ncols bytes, and arm C5 measures that +# rather than trusting it. +# +# cluster_pack_interleave is the interleave loop that lives inside +# cluster_zorder_key in src/columnar_vacuum.c today, moved out unchanged. No +# line number: the four that used to be here rot the moment anything above them +# moves, and the frozen Z-order table is what actually pins the bytes. +# src/columnar_curve.c and src/columnar_curve.h do not exist yet; until they do, +# every arm that needs them is RED, and that is the intended state. +# +# THE ARMS +# +# C1 Permutation. For ncols 1..8 and every b with ncols*b <= 20, the b-bit +# coordinates go in the TOP b bits of each uint64. Every one of the +# 2^(ncols*b) points is encoded and the set of indices must be exactly +# [0, 2^(ncols*b)). 52 cases, 6,344,330 points. +# C2 Unit-step adjacency. The packed keys are sorted with real memcmp and the +# sorted walk is compared against the POINT COORDINATES, which the sort +# cannot manufacture: consecutive points differ in one coordinate by one. +# C3 Dyadic self-similarity, plus the two named negative controls above. +# C4 Production-width bridge. (a) The top ncols*b bits of the key for a +# full-width ordinal vector equal the exhaustively verified index of the +# sub-cube that vector sits in. (b) An inverse transcribed from Skilling's +# published TransposetoAxes round-trips random full-width keys, and +# consecutive keys decode to unit-adjacent points. A round trip against an +# inverse derived from the forward code proves nothing, so a deliberately +# mutated inverse is compiled beside it and the arm must go red under it. +# C5 Golden byte vectors, and the pack bit order and key length. +# C6 ncols == 1 is the identity, pinned absolutely and relatively. +# C7 The Z-order refactor is byte-for-byte inert. +# +# WHERE EVERY EXPECTED VALUE CAME FROM +# +# No expected value in this file was produced by the code under test. The code +# under test did not exist when they were written. +# +# 0. WHAT THE GOLDEN VECTORS DISCRIMINATE. "zero", "max" and "topbit0" are +# structural invariants -- zero survives any permutation of the output bits, +# max is all-ones so any permutation is identical, and topbit0's single set +# bit lands in output bit 0 under both interleaved and column-major packing. +# Measured against a column-major pack, 27 of the C7 arms and 13 of the C5 +# arms still passed. They are kept for the zero and all-ones edges and are +# not counted as discrimination. "unit", "tie", "mixed", "asym" and "lowbit" +# are what carry it; "asym" gives every column a different dense value, and +# "lowbit" puts one set bit in the LAST column's LOWEST position, which is +# the bit a pack that truncates or reverses column order loses first. +# +# 1. BY HAND, from the two loops: +# - Every all-zero vector encodes to 8*ncols zero bytes. Skilling's first +# pass leaves an all-zero X untouched (both branches XOR zero), the Gray +# encode XORs zeros, and the accumulated mask t is zero. So the key is +# zero. +# - At ncols == 1 the transpose is the IDENTITY, so the key is the ordinal +# big-endian: 0 -> 00.., 1 -> ..01, 2^63 -> 80.., UINT64_MAX -> ff.. +# Pass one XORs, for each set bit from 63 down to 1, the bits below it; +# pass two accumulates that same mask from the result and XORs it back, +# and the two cancel. Worked by hand at three bits: 5 -> 6 -> 7 -> 5, +# 3 -> 2 -> 3, 6 -> 5 -> 6. Arm C6 is what actually pins it. +# - The pack bit order. The loop emits ord[0].bit63, ord[1].bit63, ..., +# ord[n-1].bit63, ord[0].bit62, ... MSB first. So with ncols == 2, +# pack(2^63, 0) puts a 1 in output bit 0 and pack(0, 2^63) puts it in +# output bit 1: the keys must start 0x80 and 0x40. +# - Every key is 8*ncols bytes, because the loop writes 64*ncols bits. +# +# 2. FROM SKILLING, NOT FROM US, for the remaining Hilbert keys. Generated on +# 2026-09-08 from an independent transcription of AxestoTranspose at b = 64 +# -- John Skilling, "Programming the Hilbert curve", AIP Conf. Proc. 707, +# 381-387 (2004) -- run once, off-tree, and pasted in as literals. That +# transcription was checked before its output was trusted: it round-tripped +# 160,000 random full-width transposes against a separate transcription of +# the published TransposetoAxes with 0 mismatches, and it passed C1, C2 and +# C3 over all 52 cases with 0 violations. +# +# This arm is the only one that pins WHICH Hilbert curve was chosen. +# Skilling's curve differs from Butz/Hamilton for ncols >= 3 and both are +# valid Hilbert curves, so the property arms cannot tell them apart. A +# transcription bug here would be frozen into the goldens -- which is why C1 +# to C4 exist and do not read this table. +# +# 3. FROM THE CODE BEING REPLACED, for arm C7. The Z-order keys were produced +# by transcribing the interleave loop out of cluster_zorder_key in +# src/columnar_vacuum.c verbatim and running it, on 2026-09-08, at commit +# e84a5e5, BEFORE any refactor. They are a frozen record. If both sides of C7 +# ever call the same new function the arm is a tautology; it compares new +# code against these bytes and must keep doing so. +# +# The control program's own ctl_pack is held to this SAME table, so "the +# control packs the way the product packs" is an arm rather than a comment. +# It was a comment, and nothing could falsify it: a Z-order over reversed +# columns -- not what ships -- left all seven control arms green. +# +# WHAT IS NOT CLAIMED +# +# C1, C2 and C3 are exhaustive only up to ncols*b <= 20. Above that the evidence +# is C4's bridge to production width and C5's frozen bytes, not exhaustion. +# Nothing here measures clustering QUALITY: a correct curve that nothing calls +# would pass every arm in this file. +# +# C1 to C3 read only the TOP ncols*b bits of the key -- at most 20 of them. The +# low bits are the file's blind spot and C4b is what covers them, so C4b's +# sample size is load-bearing rather than incidental. It was not, once: the keys +# came from the low byte of a 32-bit LCG whose period there is 512, giving 344 +# distinct keys where the premise arm certified 24,000, and a correct Skilling +# transpose carrying one extra bit-flip on a condition those keys never met was +# compiled against this suite and reported "138 passed + 0 failed". C4b now +# draws from splitmix64 and asserts the DISTINCT keys it round-tripped. A pack +# that ignores the low 32 bits of every ordinal is a cruder form of the same +# blind spot: it passes C1, C2, C3, C4a and both C5 extent arms, and is caught +# only by the goldens, C7, the C5 content arm, two of the C6 pins and C4b. +# +# C7 pins the PACKING of ordinals that are handed to it. cluster_zorder_key also +# derives those ordinals (cluster_type_ordinal) and maps NULL to 0, and nothing +# here calls cluster_zorder_key, so a refactor that moved the ordinal derivation +# or changed the NULL rule stays green. The arm is named for the pack, not for +# the key. +# +# THIS SUITE IS NOT REGISTERED in test/run_all_versions.sh, so no CI run +# dispatches it. That is deliberate while src/columnar_curve.c does not exist +# and every battery arm is RED by design; registering it belongs in the commit +# that adds the encoder, together with the CHANGELOG entry. Until then +# test/selftest/070 reports it as UNREGISTERED, which is the accurate state. +# +# Usage: test/hilbert_curve.sh [PG_CONFIG] +# The argument is accepted and ignored; this suite needs no cluster. +# Environment: +# CC the compiler to use; smoke-tested before it is believed. +# PGC_KEEP_WORKDIR non-empty keeps the temp directory holding the generated +# C, the build logs and the two programs' output, for +# anyone reproducing an arm by hand. Local to this suite. +# Written fresh for pgColumnar. + +set -uo pipefail + +# Sourcing must not fail quietly. Measured: a copy run from a directory with no +# lib.sh printed this suite's two measurement lines, then "check: command not +# found" 21 times, recorded 0 checks, printed no summary and exited 127. A gate +# reading the status is safe; a log-scraper reading "violations=0" is not. +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" || { + echo "cannot source lib.sh beside $0" >&2 + exit 1 +} + +# No cluster, so pgc_setup is skipped deliberately -- the shape wal_envelope.sh +# uses. lib.sh already zeroes the counters; they are restated so a reader can see +# this suite keeps them itself and so pgc_summary's reconciliation is meaningful. +PGC_CHECKS=0 +PGC_FAIL=0 +PGC_PASSED=0 +PGC_FAILED=0 +PGC_UNRUN=0 + +SRCDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +echo "== pgColumnar test: hilbert_curve.sh ==" +echo "-- no cluster is started; this battery is pure C" + +# ---- the compiler ---------------------------------------------------------- +# +# Absent compiler is the one genuinely UNRUNNABLE state here: the question +# cannot be asked at all. Absent src/columnar_curve.c is NOT that. The suite can +# ask and the answer is no, so those arms FAIL. +CC="${CC:-}" +if [ -z "$CC" ]; then + for _c in gcc cc clang; do + if command -v "$_c" >/dev/null 2>&1; then CC="$_c"; break; fi + done +fi + +WORK="$(mktemp -d /tmp/pgcolumnar-hilbert.XXXXXX)" +cleanup() { + if [ -n "${PGC_KEEP_WORKDIR:-}" ]; then + echo "-- workdir kept at $WORK (PGC_KEEP_WORKDIR)" + else + rm -rf "$WORK" + fi +} +trap cleanup EXIT + +# The eight golden ordinal vectors, per ncols. Defined identically in the C +# below, and named here because the arm lists carry one arm per vector. +PGC_VECNAMES="zero max topbit0 unit tie mixed asym lowbit" + +# Every arm this suite can report, by group, EXPANDED. The claim these lists +# exist to support is that the three states -- ran, could not compile, has no +# code under test -- report the SAME set of arms and a reader can diff two runs. +# +# That claim used to be false, and the lists were how it was false: they held +# fourteen GROUP labels against a live run's 131 battery arms, five of those +# labels never appeared verbatim in a live run at all, and four whole arm +# families ("C4a premise", both "C4b premise" arms, "C5 the pack wrote nothing +# past the key") were named in neither list, so an absent encoder left twelve +# questions not merely unanswered but unnamed -- exactly what check_unrunnable +# exists to prevent. Diffing a red run against a green one added 129 arms and +# removed 12. +# +# So the names are generated here by the same loops that assert below, and the +# diff of the two runs' arm names is empty. Keep it that way: an arm added to +# the assertions and not to this function reintroduces the defect. +CONTROL_ARMS="\ +INSTRUMENT C0 premise: the control battery encoded every case and every point +INSTRUMENT C0 premise: the control encoders were called once per point +INSTRUMENT the control's own pack reproduces the frozen Z-order bytes +INSTRUMENT C1 control: the serpentine is a permutation of the index range +INSTRUMENT C2 control: the serpentine PASSES unit-step adjacency +INSTRUMENT C3 control: the serpentine FAILS dyadic contiguity in every nested case +INSTRUMENT C1 control: Z-order is a permutation of the index range +INSTRUMENT C2 control: Z-order FAILS unit-step adjacency in every multi-column case +INSTRUMENT C3 control: Z-order PASSES dyadic contiguity +INSTRUMENT C1 control: a truncating encoder FAILS the permutation test in every case" + +battery_arm_names() { + local _n _v + + echo "C0 premise: the battery encoded every case and every point" + echo "C0 premise: the encoder was called once per point" + echo "C1 the Hilbert index set is exactly the contiguous range" + echo "C2 consecutive points in memcmp key order are unit-adjacent" + echo "C3 every dyadic sub-cube occupies a contiguous run of indices" + echo "C4a premise: the bridge covered every case and every point" + echo "C4a the full-width key prefix does not move when the low bits change" + echo "C4a the full-width key prefixes are exactly the index range" + echo "C4b premise: 3,000 DISTINCT full-width keys per ncols were round-tripped" + echo "C4b the published inverse round-trips full-width keys" + echo "C4b control: a mutated inverse breaks every round trip the clean one made" + echo "C4b premise: consecutive key pairs were formed" + echo "C4b consecutive keys decode to unit-adjacent points, re-encoded by the code under test" + for _n in 1 2 3 4 5 6 7 8; do + for _v in $PGC_VECNAMES; do + echo "C5 golden Hilbert keys: ncols=$_n $_v" + done + done + echo "C5 the pack bit order: pack(2^63, 0) starts 0x80" + echo "C5 the pack bit order: pack(0, 2^63) starts 0x40" + for _n in 1 2 3 4 5 6 7 8; do + echo "C5 the pack establishes exactly 8*ncols bytes: ncols=$_n" + echo "C5 the pack wrote nothing past the key: ncols=$_n" + echo "C5 the pack sets every bit of the key for an all-ones vector: ncols=$_n" + done + echo "C6 ncols == 1 is the big-endian ordinal: 0" + echo "C6 ncols == 1 is the big-endian ordinal: 1" + echo "C6 ncols == 1 is the big-endian ordinal: 2^63" + echo "C6 ncols == 1 is the big-endian ordinal: UINT64_MAX" + echo "C6 premise: the comparison loop ran 200,000 times" + echo "C6 ncols == 1 is the big-endian ordinal over 200,000 random values" + echo "C6 ncols == 1 agrees with Z-order over 200,000 random values" + for _n in 1 2 3 4 5 6 7 8; do + for _v in $PGC_VECNAMES; do + echo "C7 the Z-order pack is byte-for-byte what it was: ncols=$_n $_v" + done + done +} +BATTERY_ARMS="$(battery_arm_names)" + +# Report a whole group of arms in one state, so a run that could not build +# still lists what it did not measure. +arms_unrunnable() { # arms_unrunnable "LIST" REASON DETAIL + local _a + while IFS= read -r _a; do + [ -n "$_a" ] && check_unrunnable "$_a" "$2" "$3" + done <<< "$1" +} +arms_failed() { # arms_failed "LIST" DETAIL + local _a + while IFS= read -r _a; do + [ -n "$_a" ] && pgc_fail "$_a" "$2" + done <<< "$1" +} + +# A CC INHERITED FROM THE ENVIRONMENT IS NOT A COMPILER UNTIL IT COMPILES. +# +# The search above runs only when CC is empty, and PGXS and most CI images +# export CC -- so the likely value is one this suite never checked. Measured: +# `CC=/bin/false bash test/hilbert_curve.sh` reported "0 passed + 21 failed + +# 0 unrunnable", which is the toolchain being absent counted as the encoder +# being wrong, in the very state the comment above calls the one genuinely +# UNRUNNABLE one. So the resolved CC must build and run a two-line program +# before anything else is believed of it. +cc_usable=no +if [ -n "$CC" ]; then + printf 'int main(void){return 0;}\n' > "$WORK/smoke.c" + if "$CC" -o "$WORK/smoke" "$WORK/smoke.c" >"$WORK/smoke.log" 2>&1 && + "$WORK/smoke" >>"$WORK/smoke.log" 2>&1; then + cc_usable=yes + fi +fi +if [ "$cc_usable" != yes ]; then + if [ -z "$CC" ]; then + echo "-- no C compiler found (looked for gcc, cc, clang)" + _why="no C compiler" + else + echo "-- CC=$CC did not build and run a two-line program:" + sed 's/^/ /' "$WORK/smoke.log" 2>/dev/null + _why="CC=$CC cannot build C" + fi + arms_unrunnable "$CONTROL_ARMS" MISSING_DEPENDENCY "$_why" + arms_unrunnable "$BATTERY_ARMS" MISSING_DEPENDENCY "$_why" + pgc_summary +fi +echo "-- compiler: $CC ($("$CC" --version 2>/dev/null | head -1))" + +# ---- the frozen tables ----------------------------------------------------- +# +# Read by the shell and compared against what the C program printed. They are +# deliberately NOT compiled into the C program: a program that holds both the +# answer and the question can only report that it agrees with itself. + +# The six ordinal vectors, per ncols. Defined identically in the C below. +# zero every ordinal 0 +# max every ordinal UINT64_MAX +# topbit0 ordinal 0 is 2^63, the rest 0 +# unit every ordinal 1 +# tie every ordinal 5 except the last, which is 4 (tie-heavy, small) +# mixed 0x0123456789ABCDEF rotated left by 8*j bits for column j +# asym 0xF0F0F0F0F0F0F0F0 >> j -- dense, and a different value per column +# lowbit ordinal 0 everywhere except the LAST column, which is 1 +# PGC_VECNAMES is defined near the arm lists, because those name one arm per +# vector and are built before this point. + +# Provenance 2 above: Skilling's AxestoTranspose at b = 64, then the interleave +# loop. Generated off-tree on 2026-09-08; not produced by the code under test. +PGC_GOLDEN_HILBERT=" +n1 zero 0000000000000000 +n1 max ffffffffffffffff +n1 topbit0 8000000000000000 +n1 unit 0000000000000001 +n1 tie 0000000000000004 +n1 mixed 0123456789abcdef +n1 asym f0f0f0f0f0f0f0f0 +n1 lowbit 0000000000000001 +n2 zero 00000000000000000000000000000000 +n2 max aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +n2 topbit0 eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +n2 unit 00000000000000000000000000000002 +n2 tie 00000000000000000000000000000023 +n2 mixed 040c9c1c868ed6d684869e36ae0e54fc +n2 asym c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0 +n2 lowbit 00000000000000000000000000000003 +n3 zero 000000000000000000000000000000000000000000000000 +n3 max b6db6db6db6db6db6db6db6db6db6db6db6db6db6db6db6d +n3 topbit0 f12492492492492492492492492492492492492492492492 +n3 unit 000000000000000000000000000000000000000000000005 +n3 tie 000000000000000000000000000000000000000000000146 +n3 mixed 1dc46b182507638ea95e6175bececbaa0ab100a019770fc1 +n3 asym fad4f6f2d840e3f8e2209562209562209562209562209562 +n3 lowbit 000000000000000000000000000000000000000000000001 +n4 zero 0000000000000000000000000000000000000000000000000000000000000000 +n4 max aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +n4 topbit0 f844444444444444444444444444444444444444444444444444444444444444 +n4 unit 000000000000000000000000000000000000000000000000000000000000000a +n4 tie 0000000000000000000000000000000000000000000000000000000000000a0d +n4 mixed 0e04480636ced888a0688e20142a7e6c2aec8ac47c481e6684c6442052ccdca6 +n4 asym fcd23edc54743ed23a947c56943e9ab83a96bc16547c56943e9ab83a96bc1654 +n4 lowbit 0000000000000000000000000000000000000000000000000000000000000003 +n5 zero 00000000000000000000000000000000000000000000000000000000000000000000000000000000 +n5 max ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5ad6b5 +n5 topbit0 fc108421084210842108421084210842108421084210842108421084210842108421084210842108 +n5 unit 00000000000000000000000000000000000000000000000000000000000000000000000000000015 +n5 tie 0000000000000000000000000000000000000000000000000000000000000000000000000000541a +n5 mixed 0f0183c8df970eab0bf7d87586fddb2263e65e35d488645b43dd0e8c92b3c1e4652cd11ece26b677 +n5 asym fe0aab4507ecab681c836dab410fdd1f07ee0f83ff07c21c871eb26c0fdd2917692f5dff07c21c87 +n5 lowbit 00000000000000000000000000000000000000000000000000000000000000000000000000000007 +n6 zero 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +n6 max aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +n6 topbit0 fe0410410410410410410410410410410410410410410410410410410410410410410410410410410410410410410410 +n6 unit 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002a +n6 tie 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002a035 +n6 mixed 0b8feeda23e06c90e054366c32ef4cf08448a1f8de5255fac96f7492acee7cf85a5f54fa03c558b323c4d913d4d11546 +n6 asym ff02cac841fc1b6d7449c45003e0466a85ec0bec4859276ec04efa08ed7449c45003e0466a85ec0bec4859276ec04efa +n6 lowbit 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f +n7 zero 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +n7 max ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5ab56ad5 +n7 topbit0 ff01020408102040810204081020408102040810204081020408102040810204081020408102040810204081020408102040810204081020 +n7 unit 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000055 +n7 tie 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000154056 +n7 mixed 0bc7fdcdb2d089745031e34d314d2d40d88d4132f3109e562a066d7be4dcde2ad855fd67d3350272a645a26b7cc7be84bf6eb64745a4f467 +n7 asym ff80beab529757effdfd059485050fcf88edd80e1d9b6e3d83828b6937bfffc24aaa2972769ca508007fc6b55da8d8880fea14a92205027b +n7 lowbit 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001 +n8 zero 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +n8 max aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +n8 topbit0 ff804040404040404040404040404040404040404040404040404040404040404040404040404040404040404040404040404040404040404040404040404040 +n8 unit 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000aa +n8 tie 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000aa00ad +n8 mixed 0ae2feb8dae8024c920ecc1ea668dab6d2723046aeda3c1088f6b2c23894ba98ae2698b240deaa986072d688529628e60880c86ca0bc2ce422a03c40a83a5c94 +n8 asym ffc02fd4ca0a040010142aaababe8038269a56b2743ad81c16f6d6d8222a0a76a85c922ec6c4644c5696dadc02069aa4586c321ad6d432240406e222ded838e4 +n8 lowbit 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003 +" + +# Provenance 3 above: transcribed from the interleave loop inside +# cluster_zorder_key in src/columnar_vacuum.c and run at commit e84a5e5 on +# 2026-09-08, BEFORE the refactor. A frozen record. The control program's own +# ctl_pack is held to this table too, so the control cannot drift away from the +# packing it claims to be. +PGC_GOLDEN_ZORDER=" +n1 zero 0000000000000000 +n1 max ffffffffffffffff +n1 topbit0 8000000000000000 +n1 unit 0000000000000001 +n1 tie 0000000000000004 +n1 mixed 0123456789abcdef +n1 asym f0f0f0f0f0f0f0f0 +n1 lowbit 0000000000000001 +n2 zero 00000000000000000000000000000000 +n2 max ffffffffffffffffffffffffffffffff +n2 topbit0 80000000000000000000000000000000 +n2 unit 00000000000000000000000000000003 +n2 tie 00000000000000000000000000000032 +n2 mixed 0407181b3437686bc4c7d8dbf4f7a8ab +n2 asym bf40bf40bf40bf40bf40bf40bf40bf40 +n2 lowbit 00000000000000000000000000000001 +n3 zero 000000000000000000000000000000000000000000000000 +n3 max ffffffffffffffffffffffffffffffffffffffffffffffff +n3 topbit0 800000000000000000000000000000000000000000000000 +n3 unit 000000000000000000000000000000000000000000000007 +n3 tie 0000000000000000000000000000000000000000000001c6 +n3 mixed 0500570e80ef39039772872fe50e57ee8eefd90d9792892f +n3 asym 9bf6409bf6409bf6409bf6409bf6409bf6409bf6409bf640 +n3 lowbit 000000000000000000000000000000000000000000000001 +n4 zero 0000000000000000000000000000000000000000000000000000000000000000 +n4 max ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff +n4 topbit0 8000000000000000000000000000000000000000000000000000000000000000 +n4 unit 000000000000000000000000000000000000000000000000000000000000000f +n4 tie 0000000000000000000000000000000000000000000000000000000000000f0e +n4 mixed 0350035f16a016af3c503c5f79a079aff350f35fe6a0e6afcc50cc5f89a089af +n4 asym 8cef73108cef73108cef73108cef73108cef73108cef73108cef73108cef7310 +n4 lowbit 0000000000000000000000000000000000000000000000000000000000000001 +n5 zero 00000000000000000000000000000000000000000000000000000000000000000000000000000000 +n5 max ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff +n5 topbit0 80000000000000000000000000000000000000000000000000000000000000000000000000000000 +n5 unit 0000000000000000000000000000000000000000000000000000000000000000000000000000001f +n5 tie 00000000000000000000000000000000000000000000000000000000000000000000000000007c1e +n5 mixed 099400995f1b2a01b2bf3e5403e55f7cea07cebff1940f195fe32a0e32bfc6540c655f84ea084ebf +n5 asym 8639e79c618639e79c618639e79c618639e79c618639e79c618639e79c618639e79c618639e79c61 +n5 lowbit 00000000000000000000000000000000000000000000000000000000000000000000000000000001 +n6 zero 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +n6 max ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff +n6 topbit0 800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +n6 unit 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003f +n6 tie 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003f03e +n6 mixed 0cc5400cc57f1d9a801d9abf3f35403f357f7a6a807a6abff0c540f0c57fe19a80e19abfc33540c3357f866a80866abf +n6 asym 830e3c78f1c3870e3c78f1c3870e3c78f1c3870e3c78f1c3870e3c78f1c3870e3c78f1c3870e3c78f1c3870e3c78f1c3 +n6 lowbit 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001 +n7 zero 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +n7 max ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff +n7 topbit0 8000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +n7 unit 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f +n7 tie 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001fc07e +n7 mixed 0e655000e6557f1ecea801eceaff3d995003d9957f7932a807932afff065500f06557fe0cea80e0ceaffc399500c39957f8732a808732aff +n7 asym 8183878787878787878787878787878787878787878787878787878787878787878787878787878787878787878787878787878787878787 +n7 lowbit 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001 +n8 zero 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +n8 max ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff +n8 topbit0 80000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +n8 unit 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000ff +n8 tie 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000ff00fe +n8 mixed 0f3355000f3355ff1e66aa001e66aaff3ccc55003ccc55ff7899aa007899aafff0335500f03355ffe166aa00e166aaffc3cc5500c3cc55ff8799aa008799aaff +n8 asym 80c0e0f0783c1e0f87c3e1f0783c1e0f87c3e1f0783c1e0f87c3e1f0783c1e0f87c3e1f0783c1e0f87c3e1f0783c1e0f87c3e1f0783c1e0f87c3e1f0783c1e0f +n8 lowbit 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001 +" + +# A value out of the frozen tables. An absent key returns the empty string, and +# every comparison below goes through check_text, which refuses an empty side -- +# so a typo in a name FAILS rather than comparing "" with "" and passing (#418). +frozen() { # frozen TABLE NCOLS VECNAME + awk -v k1="n$2" -v k2="$3" '$1 == k1 && $2 == k2 { print $3 }' <<< "$1" +} + +# A value out of a C program's name=value output. Same reasoning: missing is +# empty, and empty fails. +val() { # val FILE NAME + awk -F= -v k="$2" '$1 == k { print $2 }' "$1" +} + +# ---- the C the suite compiles ---------------------------------------------- + +# Stub PostgreSQL headers, so src/columnar_curve.c compiles with no server tree. +# The .c and .h under test are COPIED into this directory, so their own +# #include "..." finds these stubs first and never reaches the real headers. +cat > "$WORK/postgres.h" <<'CEOF' +#ifndef PGC_STUB_POSTGRES_H +#define PGC_STUB_POSTGRES_H +#include +#include +#include +#include +#include +typedef int8_t int8; typedef uint8_t uint8; +typedef int16_t int16; typedef uint16_t uint16; +typedef int32_t int32; typedef uint32_t uint32; +typedef int64_t int64; typedef uint64_t uint64; +#define Assert(p) ((void) 0) +#define StaticAssertDecl(c, m) extern int pgc_stub_sa_dummy +#define palloc(sz) malloc(sz) +#define palloc0(sz) calloc(1, (sz)) +#define pfree(p) free(p) +#define pg_attribute_unused() +#endif +CEOF +for _h in c.h fmgr.h miscadmin.h postgres_ext.h utils/elog.h \ + columnar.h columnar_compat.h; do + mkdir -p "$WORK/$(dirname "$_h")" + echo '#include "postgres.h"' > "$WORK/$_h" +done + +# The property checkers. Shared by the control program and the battery, so both +# ask the SAME question of their encoders and a difference in verdict is a +# difference in the encoder. +cat > "$WORK/props.h" <<'CEOF' +#ifndef PGC_PROPS_H +#define PGC_PROPS_H +#include +#include +#include +#include + +typedef uint64_t pgc_u64; + +/* + * Counted at the point of each enc() call. + * + * The C0 premise arms used to report `points += (double) (1u << (ncols * b))`, + * which is arithmetic over the loop bounds and reports 6,344,330 whether or not + * a single point was encoded -- the house rule "measure the work, never the + * intent", broken in the arm whose whole job is to establish that the work + * happened. This counter is incremented where the encoder is actually called, + * and the premise arms assert it. + */ +static long pgc_enc_calls = 0; + +/* ---- the golden ordinal vectors, shared by both programs ------------------ + * + * Here rather than in the battery because the control program pins its own + * pack against the same frozen Z-order table, and two copies of these + * definitions would let that pin drift from the thing it pins. + * + * WHICH OF THESE DISCRIMINATE. "zero" is invariant under every permutation of + * the output bits, "max" is all-ones so any permutation is identical, and + * "topbit0" has one set bit that lands in output bit 0 under both interleaved + * and column-major packing. Measured: under a column-major pack 27 of the 48 + * C7 arms and 13 of the 48 C5 arms then in the file still passed. They are kept + * for the zero and all-ones edges, not counted as discrimination. "unit", + * "tie", "mixed", "asym" and "lowbit" are what carry it, and "lowbit" -- one + * set bit in the LAST column's LOWEST position -- is the one a pack that drops + * low bits or reverses column order cannot reproduce. + */ +#define PGC_NVEC 8 +static const char *const pgc_vecname[PGC_NVEC] = +{"zero", "max", "topbit0", "unit", "tie", "mixed", "asym", "lowbit"}; + +static void +pgc_makevec(int which, int ncols, pgc_u64 *X) +{ + int j; + + for (j = 0; j < ncols; j++) + { + switch (which) + { + case 0: + X[j] = 0; + break; + case 1: + X[j] = UINT64_MAX; + break; + case 2: + X[j] = (j == 0) ? ((pgc_u64) 1 << 63) : 0; + break; + case 3: + X[j] = 1; + break; + case 4: + X[j] = (j == ncols - 1) ? 4 : 5; + break; + case 5: + { + pgc_u64 s = UINT64_C(0x0123456789ABCDEF); + int r = (8 * j) & 63; + + X[j] = r ? ((s << r) | (s >> (64 - r))) : s; + break; + } + case 6: + X[j] = UINT64_C(0xF0F0F0F0F0F0F0F0) >> j; + break; + default: + X[j] = (j == ncols - 1) ? 1 : 0; + break; + } + } +} + +static void +pgc_hex(const unsigned char *b, int n, char *o) +{ + int i; + + for (i = 0; i < n; i++) + sprintf(o + 2 * i, "%02x", b[i]); + o[2 * n] = '\0'; +} + +/* The point whose index is p, as ncols b-bit coordinates, column 0 first. */ +static void +pgc_digits(unsigned p, int ncols, int b, unsigned *c) +{ + int j; + + for (j = ncols - 1; j >= 0; j--) + { + c[j] = p & ((1u << b) - 1); + p >>= b; + } +} + +/* + * The index a key carries: its top ncols*b bits, read MSB first. + * + * With the b-bit coordinates in the TOP b bits of each uint64, the pack loop + * emits ord[0].bit63, ord[1].bit63, ..., so the first ncols*b bits of the key + * are exactly the b-round curve index and everything below them is zero. + */ +static unsigned +pgc_key_index(const unsigned char *key, int ncols, int b) +{ + unsigned idx = 0; + int t, + nb = ncols * b; + + for (t = 0; t < nb; t++) + idx = (idx << 1) | ((key[t >> 3] >> (7 - (t & 7))) & 1); + return idx; +} + +/* The transpose form whose packing is this index, for an encoder that produces + * an index directly rather than a transpose. */ +static void +pgc_index_to_transpose(unsigned idx, int ncols, int b, pgc_u64 *X) +{ + int t, + nb = ncols * b; + + for (t = 0; t < ncols; t++) + X[t] = 0; + for (t = 0; t < nb; t++) + X[t % ncols] |= (pgc_u64) ((idx >> (nb - 1 - t)) & 1) << (63 - (t / ncols)); +} + +typedef void (*pgc_enc) (const unsigned *c, int ncols, int b, unsigned char *key); + +static int pgc_g_ncols; +static const unsigned char *pgc_g_keys; + +static int +pgc_cmpkey(const void *a, const void *b) +{ + unsigned pa = *(const unsigned *) a, + pb = *(const unsigned *) b; + + return memcmp(pgc_g_keys + (size_t) pa * pgc_g_ncols * 8, + pgc_g_keys + (size_t) pb * pgc_g_ncols * 8, + (size_t) pgc_g_ncols * 8); +} + +/* + * Run C1, C2 and C3 over the whole 2^(ncols*b) point space for one encoder. + * + * C1 counts, together: an index outside [0, 2^(ncols*b)), an index hit twice, + * and an index never hit. "No collisions" alone is vacuous -- a truncating + * encoder is injective on a subset -- so the unhit sweep is the half that makes + * this an assertion about the whole range. + * + * C2 sorts the PACKED KEYS with memcmp and then walks the sorted order looking + * at the POINT COORDINATES. The sort cannot manufacture coordinates, so this is + * not a claim the ordering makes about itself. + * + * C3 walks every dyadic level and requires each sub-cube's indices to be a + * contiguous run. C1 has already established the indices are distinct, so + * max - min + 1 == count is exactly contiguity. + */ +static void +pgc_props(pgc_enc enc, int ncols, int b, long *c1, long *c2, long *c3) +{ + unsigned N = 1u << (ncols * b), + p; + unsigned *idx = malloc(sizeof(unsigned) * (size_t) N); + unsigned *ord = malloc(sizeof(unsigned) * (size_t) N); + unsigned char *seen = calloc(N, 1); + unsigned char *keys = malloc((size_t) N * ncols * 8); + unsigned c[8], + c2v[8]; + int L; + + *c1 = *c2 = *c3 = 0; + if (!idx || !ord || !seen || !keys) + { + fprintf(stderr, "out of memory at ncols=%d b=%d\n", ncols, b); + exit(2); + } + + for (p = 0; p < N; p++) + { + pgc_digits(p, ncols, b, c); + enc(c, ncols, b, keys + (size_t) p * ncols * 8); + pgc_enc_calls++; + idx[p] = pgc_key_index(keys + (size_t) p * ncols * 8, ncols, b); + if (idx[p] >= N) + (*c1)++; + else + { + if (seen[idx[p]]) + (*c1)++; + seen[idx[p]] = 1; + } + ord[p] = p; + } + for (p = 0; p < N; p++) + if (!seen[p]) + (*c1)++; + + pgc_g_ncols = ncols; + pgc_g_keys = keys; + qsort(ord, N, sizeof(unsigned), pgc_cmpkey); + for (p = 1; p < N; p++) + { + int j, + ndiff = 0, + ok = 1; + + pgc_digits(ord[p - 1], ncols, b, c); + pgc_digits(ord[p], ncols, b, c2v); + for (j = 0; j < ncols; j++) + if (c[j] != c2v[j]) + { + unsigned d = c[j] > c2v[j] ? c[j] - c2v[j] : c2v[j] - c[j]; + + ndiff++; + if (d != 1) + ok = 0; + } + if (ndiff != 1 || !ok) + (*c2)++; + } + + for (L = 1; L <= b; L++) + { + unsigned ncube = 1u << (ncols * L), + k; + unsigned *mn = malloc(sizeof(unsigned) * (size_t) ncube); + unsigned *mx = malloc(sizeof(unsigned) * (size_t) ncube); + unsigned *ct = calloc(ncube, sizeof(unsigned)); + + for (k = 0; k < ncube; k++) + { + mn[k] = 0xffffffffu; + mx[k] = 0; + } + for (p = 0; p < N; p++) + { + unsigned id = 0; + int j; + + pgc_digits(p, ncols, b, c); + for (j = 0; j < ncols; j++) + id = (id << L) | (c[j] >> (b - L)); + if (idx[p] < mn[id]) + mn[id] = idx[p]; + if (idx[p] > mx[id]) + mx[id] = idx[p]; + ct[id]++; + } + for (k = 0; k < ncube; k++) + if (ct[k] && mx[k] - mn[k] + 1 != ct[k]) + (*c3)++; + free(mn); + free(mx); + free(ct); + } + + free(idx); + free(ord); + free(seen); + free(keys); +} +#endif +CEOF + +# ---- the controls ---------------------------------------------------------- +# +# This program needs nothing from src/. It exists so the battery's instrument is +# proved to DISCRIMINATE before the real encoder is written, and it keeps proving +# it afterwards. +cat > "$WORK/controls.c" <<'CEOF' +/* + * Two deliberately wrong encoders, measured by the same C1/C2/C3 checkers the + * battery uses. + * + * SERPENTINE must pass C2 and fail C3. Z-ORDER must fail C2 and pass C3. That + * pair is the argument: adjacency alone certifies a serpentine, contiguity alone + * certifies the Z-order that ships today, and only the conjunction is a Hilbert + * curve. Each property therefore has a control on both sides -- one encoder that + * must pass it and one that must fail it -- so a checker stuck at "pass" or + * stuck at "fail" reddens this file instead of passing the battery. + * + * C1 had no such pair, though the header claimed it did: the serpentine and + * Z-order are BOTH permutations, so nothing here could make the permutation + * checker report a violation, and C1 is the arm the file sells hardest. TRUNCATE + * is that missing side -- it drops the low bit of the index, so half the range + * is hit twice and half never, and C1 must go red for every case under it. + */ +#include "props.h" + +/* + * The interleave loop out of cluster_zorder_key in src/columnar_vacuum.c. This + * program deliberately does NOT link the code under test. + * + * "Transcribed from the shipped loop" was a comment, and no arm could falsify + * it: measured, a Z-order over REVERSED columns -- which is not what ships -- + * left all seven control arms green with identical violation counts. The + * citation also carried a hand-maintained line number that rots the moment + * anything above it moves. + * + * Both are fixed by an arm rather than by prose: main() below packs the eight + * golden vectors with THIS loop and prints them, and the shell holds the result + * to PGC_GOLDEN_ZORDER -- the frozen record of the shipped bytes that C7 holds + * the code under test to. A ctl_pack that is not the shipped packing now + * reddens this file. + */ +static void +ctl_pack(const pgc_u64 *ord, int ncols, unsigned char *out) +{ + int c, + r, + outbit = 0; + + memset(out, 0, (size_t) ncols * 8); + for (r = 63; r >= 0; r--) + { + for (c = 0; c < ncols; c++) + { + if ((ord[c] >> r) & 1) + out[outbit >> 3] |= (unsigned char) (0x80 >> (outbit & 7)); + outbit++; + } + } +} + +/* + * Boustrophedon scan. Column 0 is the outermost axis and the next axis reverses + * whenever the position along this one is odd, which is what makes every step a + * unit step: the scan turns around at the end of each row rather than jumping + * back to its start. + */ +static unsigned +ctl_serpentine_index(const unsigned *c, int ncols, int b) +{ + unsigned M = (1u << b) - 1, + idx = 0, + rev = 0, + v; + int i; + + for (i = 0; i < ncols; i++) + { + v = rev ? (M - c[i]) : c[i]; + idx = (idx << b) | v; + rev = v & 1; + } + return idx; +} + +static void +enc_serpentine(const unsigned *c, int ncols, int b, unsigned char *key) +{ + pgc_u64 X[8]; + + pgc_index_to_transpose(ctl_serpentine_index(c, ncols, b), ncols, b, X); + ctl_pack(X, ncols, key); +} + +/* + * C1's negative control: the serpentine index with its low bit dropped. Every + * even index is then hit twice and every odd one never, so the permutation + * checker must report violations in every one of the 52 cases. Without it + * nothing in this file ever made C1 go red, and "no collisions" is exactly the + * property a truncating encoder satisfies. + */ +static void +enc_truncate(const unsigned *c, int ncols, int b, unsigned char *key) +{ + pgc_u64 X[8]; + + pgc_index_to_transpose(ctl_serpentine_index(c, ncols, b) & ~1u, ncols, b, X); + ctl_pack(X, ncols, key); +} + +/* Z-order: ordinals straight into the pack, no transpose. The packing is pinned + * against the frozen Z-order record below rather than asserted in a comment. */ +static void +enc_zorder(const unsigned *c, int ncols, int b, unsigned char *key) +{ + pgc_u64 X[8]; + int j; + + for (j = 0; j < ncols; j++) + X[j] = (pgc_u64) c[j] << (64 - b); + ctl_pack(X, ncols, key); +} + +int +main(void) +{ + int ncols, + b; + long cases = 0, + serp_c1 = 0, + serp_c2 = 0, + serp_c3 = 0, + zord_c1 = 0, + zord_c2 = 0, + zord_c3 = 0; + long serp_c3_want = 0, + serp_c3_got = 0, + zord_c2_want = 0, + zord_c2_got = 0, + trunc_c1_want = 0, + trunc_c1_got = 0; + double points = 0; + + for (ncols = 1; ncols <= 8; ncols++) + { + for (b = 1; ncols * b <= 20; b++) + { + long a1, + a2, + a3, + b1, + b2, + b3, + t1, + t2, + t3; + + cases++; + points += (double) (1u << (ncols * b)); + pgc_props(enc_serpentine, ncols, b, &a1, &a2, &a3); + pgc_props(enc_zorder, ncols, b, &b1, &b2, &b3); + pgc_props(enc_truncate, ncols, b, &t1, &t2, &t3); + trunc_c1_want++; + if (t1 > 0) + trunc_c1_got++; + (void) t2; + (void) t3; + serp_c1 += a1; + serp_c2 += a2; + serp_c3 += a3; + zord_c1 += b1; + zord_c2 += b2; + zord_c3 += b3; + + /* + * A serpentine's sub-cubes can only be scattered where a proper + * sub-cube exists: ncols >= 2 and b >= 2. Below that the scan IS the + * curve and contiguity holds, so those cases are not evidence either + * way and are excluded from the "must fail" domain rather than + * absorbed into a total that would hide them. + */ + if (ncols >= 2 && b >= 2) + { + serp_c3_want++; + if (a3 > 0) + serp_c3_got++; + } + /* Z-order jumps at the first quadrant boundary, so one column is the + * only case where it cannot: at ncols == 1 it is the identity. */ + if (ncols >= 2) + { + zord_c2_want++; + if (b2 > 0) + zord_c2_got++; + } + } + } + + /* + * The control's pack must be the shipped packing, and this is the arm that + * says so: one line holding every golden Z-order vector, compared in the + * shell against the frozen table. + */ + { + int n, + k; + pgc_u64 V[8]; + unsigned char key[64]; + char h[200]; + + printf("ctl_zorder_all="); + for (n = 1; n <= 8; n++) + for (k = 0; k < PGC_NVEC; k++) + { + pgc_makevec(k, n, V); + ctl_pack(V, n, key); + pgc_hex(key, n * 8, h); + printf("%s", h); + } + printf("\n"); + } + + printf("ctl_cases=%ld\n", cases); + printf("ctl_points=%.0f\n", points); + printf("ctl_enc_calls=%ld\n", pgc_enc_calls); + printf("trunc_c1_cases=%ld\n", trunc_c1_want); + printf("trunc_c1_cases_that_failed=%ld\n", trunc_c1_got); + printf("serp_c1=%ld\n", serp_c1); + printf("serp_c2=%ld\n", serp_c2); + printf("serp_c3_violations=%ld\n", serp_c3); + printf("serp_c3_nested_cases=%ld\n", serp_c3_want); + printf("serp_c3_nested_cases_that_failed=%ld\n", serp_c3_got); + printf("zord_c1=%ld\n", zord_c1); + printf("zord_c2_violations=%ld\n", zord_c2); + printf("zord_c2_multicol_cases=%ld\n", zord_c2_want); + printf("zord_c2_multicol_cases_that_failed=%ld\n", zord_c2_got); + printf("zord_c3=%ld\n", zord_c3); + return 0; +} +CEOF + +CTLOUT="$WORK/controls.out" +CTLLOG="$WORK/controls.log" + +# Where the instrument arms begin. The seven control arms link nothing from +# src/: with src/columnar_curve.{c,h} deleted they print seven PASSes, which is +# correct for calibration and misleading to anyone counting greens. They are +# prefixed INSTRUMENT and counted separately below, so "PASSED" can never be +# reached by arms that never touched the code under test. +_pre_checks=$PGC_CHECKS +_pre_passed=$PGC_PASSED +_pre_failed=$PGC_FAILED +if "$CC" -O2 -Wall -I"$WORK" -o "$WORK/controls" "$WORK/controls.c" >"$CTLLOG" 2>&1 \ + && "$WORK/controls" > "$CTLOUT" 2>>"$CTLLOG"; then + echo "-- controls: built and ran" + echo "-- serpentine: C1 violations=$(val "$CTLOUT" serp_c1)" \ + "C2 violations=$(val "$CTLOUT" serp_c2)" \ + "C3 violations=$(val "$CTLOUT" serp_c3_violations)" \ + "in $(val "$CTLOUT" serp_c3_nested_cases_that_failed)" \ + "of $(val "$CTLOUT" serp_c3_nested_cases) nested cases" + echo "-- Z-order: C1 violations=$(val "$CTLOUT" zord_c1)" \ + "C2 violations=$(val "$CTLOUT" zord_c2_violations)" \ + "in $(val "$CTLOUT" zord_c2_multicol_cases_that_failed)" \ + "of $(val "$CTLOUT" zord_c2_multicol_cases) multi-column cases," \ + "C3 violations=$(val "$CTLOUT" zord_c3)" + + # The premise. 52 cases and 6,344,330 points is sum over ncols 1..8 and every + # b with ncols*b <= 20 of 2^(ncols*b), computed independently of this program. + # Pinned exactly rather than bounded: a checker that silently stopped early + # would otherwise still satisfy "more than a few". + check "INSTRUMENT C0 premise: the control battery encoded every case and every point" \ + "$(val "$CTLOUT" ctl_cases) cases, $(val "$CTLOUT" ctl_points) points" \ + "52 cases, 6344330 points" + # And the same number counted where the encoders are CALLED, not derived from + # the loop bounds: three encoders over 6,344,330 points each. + check_num "INSTRUMENT C0 premise: the control encoders were called once per point" \ + "$(val "$CTLOUT" ctl_enc_calls)" "19032990" + + # The control's pack is the shipped packing, asserted rather than asserted in + # a comment. Built from the same frozen table C7 holds the code under test to. + _zall="" + for _n in 1 2 3 4 5 6 7 8; do + for _v in $PGC_VECNAMES; do + _zall="$_zall$(frozen "$PGC_GOLDEN_ZORDER" "$_n" "$_v")" + done + done + check_text "INSTRUMENT the control's own pack reproduces the frozen Z-order bytes" \ + "$(val "$CTLOUT" ctl_zorder_all)" "$_zall" + + check_num "INSTRUMENT C1 control: the serpentine is a permutation of the index range" \ + "$(val "$CTLOUT" serp_c1)" "0" + check_num "INSTRUMENT C2 control: the serpentine PASSES unit-step adjacency" \ + "$(val "$CTLOUT" serp_c2)" "0" + check "INSTRUMENT C3 control: the serpentine FAILS dyadic contiguity in every nested case" \ + "$(val "$CTLOUT" serp_c3_nested_cases_that_failed) of $(val "$CTLOUT" serp_c3_nested_cases)" \ + "25 of 25" + + check_num "INSTRUMENT C1 control: Z-order is a permutation of the index range" \ + "$(val "$CTLOUT" zord_c1)" "0" + check "INSTRUMENT C2 control: Z-order FAILS unit-step adjacency in every multi-column case" \ + "$(val "$CTLOUT" zord_c2_multicol_cases_that_failed) of $(val "$CTLOUT" zord_c2_multicol_cases)" \ + "32 of 32" + check_num "INSTRUMENT C3 control: Z-order PASSES dyadic contiguity" \ + "$(val "$CTLOUT" zord_c3)" "0" + # C1's missing side. Both encoders above are permutations, so until this arm + # existed nothing here could make the permutation checker report a violation. + check "INSTRUMENT C1 control: a truncating encoder FAILS the permutation test in every case" \ + "$(val "$CTLOUT" trunc_c1_cases_that_failed) of $(val "$CTLOUT" trunc_c1_cases)" \ + "52 of 52" +else + echo "---- the control program would not build or run ----" + sed 's/^/ /' "$CTLLOG" + arms_failed "$CONTROL_ARMS" "the control program would not build or run" +fi +echo "-- instrument arms: $((PGC_CHECKS - _pre_checks)) reported" \ + "($((PGC_PASSED - _pre_passed)) passed, $((PGC_FAILED - _pre_failed)) failed);" \ + "none of them links src/columnar_curve.c" +_ctl_checks=$((PGC_CHECKS - _pre_checks)) +_ctl_passed=$((PGC_PASSED - _pre_passed)) + +# ---- the battery ----------------------------------------------------------- + +CURVE_C="$SRCDIR/src/columnar_curve.c" +CURVE_H="$SRCDIR/src/columnar_curve.h" +BATOUT="$WORK/battery.out" +BATLOG="$WORK/battery.log" + +cat > "$WORK/battery.c" <<'CEOF' +/* + * The battery proper. This is the only program here that links the code under + * test, so everything it reports is about src/columnar_curve.c. + * + * It prints name=value lines and asserts nothing. Every expected value lives in + * the shell, out of this program's reach, because a program holding both the + * question and the answer can only report that it agrees with itself. + */ +#include "postgres.h" +#include "columnar_curve.h" +#include "props.h" + +/* + * A compile-time pin on the interface #889 agreed. Built with + * -Werror=incompatible-pointer-types, so a changed signature is a build failure + * and this suite goes red for the right reason rather than adapting silently. + */ +static void (*const pin_transpose) (uint64 *, int) = cluster_hilbert_transpose; +static void (*const pin_pack) (const uint64 *, int, unsigned char *) = cluster_pack_interleave; + +/* Deterministic, so two runs of this suite compare the same points. */ +static unsigned rs = 987654321u; +static unsigned +rnd(void) +{ + rs = rs * 1103515245u + 12345u; + return rs >> 1; +} +static uint64 +rnd64(void) +{ + return ((uint64) rnd() << 40) ^ ((uint64) rnd() << 20) ^ (uint64) rnd(); +} + +/* + * splitmix64, for the C4b keys. + * + * They were built byte by byte from `rnd() & 0xff`. rs is a 32-bit LCG and + * rnd() returns rs >> 1, so bit j of rnd() is bit j+1 of rs and has period + * 2^(j+2): the low byte repeats every 512 draws. A key of 8*ncols bytes cut + * from a period-512 stream therefore takes 512/gcd(512, 8*ncols) values -- + * measured 64, 32, 64, 16, 64, 32, 64 and 8 distinct keys for ncols 1..8, 344 + * in all, while the premise arm certified 24,000. At ncols == 8 the same eight + * keys were round-tripped 375 times each. + * + * That is what let a WRONG encoder pass this file. A correct Skilling transpose + * plus "if (ncols == 8 and the low byte of the transposed X[3] is 0x5a) flip + * bit 0 of X[0]" was compiled against the suite as it stood and reported + * "138 passed + 0 failed". Nothing below the top ncols*b bits of the key was + * pinned by more than a handful of inputs. + * + * splitmix64 has full 64-bit period, and the arm below counts and asserts the + * DISTINCT keys it round-tripped rather than the number of iterations it ran. + */ +static uint64 sm = UINT64_C(0x243F6A8885A308D3); +static uint64 +sm64(void) +{ + uint64 z = (sm += UINT64_C(0x9E3779B97F4A7C15)); + + z = (z ^ (z >> 30)) * UINT64_C(0xBF58476D1CE4E5B9); + z = (z ^ (z >> 27)) * UINT64_C(0x94D049BB133111EB); + return z ^ (z >> 31); +} + +/* memcmp over a fixed-width key, for the distinct-key count. */ +static int pgc_klen; +static int +cmp_rawkey(const void *a, const void *b) +{ + return memcmp(a, b, (size_t) pgc_klen); +} + +/* The transpose a full-width key carries: the pack loop run backwards. */ +static void +unpack(const unsigned char *key, int n, uint64 *X) +{ + int t, + nb = n * 64; + + for (t = 0; t < n; t++) + X[t] = 0; + for (t = 0; t < nb; t++) + X[t % n] |= (uint64) ((key[t >> 3] >> (7 - (t & 7))) & 1) << (63 - (t / n)); +} + +/* + * TransposetoAxes, transcribed from Skilling's published listing at b = 64. + * Deliberately NOT derived from cluster_hilbert_transpose: an inverse written by + * inverting our own forward code makes the round trip a tautology. + * + * N is 2 << (b-1), which is 0 at b == 64, and the loop `Q = 2; Q != N; Q <<= 1` + * therefore stops when Q wraps past 2^63. That is the published code's own + * arithmetic, kept rather than rewritten. + * + * mutate drops one line -- the Gray-decode fold back into X[0]. That is the + * negative control for arm C4b: with it set, every round trip must break. + */ +static void +untranspose(uint64 *X, int n, int mutate) +{ + uint64 N = 0; + uint64 P, + Q, + t; + int i; + + t = X[n - 1] >> 1; + for (i = n - 1; i > 0; i--) + X[i] ^= X[i - 1]; + if (!mutate) + X[0] ^= t; /* THE MUTATION IS DROPPING THIS LINE */ + for (Q = 2; Q != N; Q <<= 1) + { + P = Q - 1; + for (i = n - 1; i >= 0; i--) + { + if (X[i] & Q) + X[0] ^= P; + else + { + t = (X[0] ^ X[i]) & P; + X[0] ^= t; + X[i] ^= t; + } + } + } +} + +static void +enc_hilbert(const unsigned *c, int ncols, int b, unsigned char *key) +{ + uint64 X[8]; + int j; + + for (j = 0; j < ncols; j++) + X[j] = (uint64) c[j] << (64 - b); + cluster_hilbert_transpose(X, ncols); + cluster_pack_interleave(X, ncols, key); +} + +int +main(void) +{ + int ncols, + b, + k, + j; + uint64 X[8], + Y[8]; + unsigned char key[64], + key2[64]; + char h[200]; + + /* ---- C1, C2, C3 ---------------------------------------------------- */ + { + long cases = 0, + c1 = 0, + c2 = 0, + c3 = 0; + double points = 0; + + for (ncols = 1; ncols <= 8; ncols++) + for (b = 1; ncols * b <= 20; b++) + { + long a1, + a2, + a3; + + cases++; + points += (double) (1u << (ncols * b)); + pgc_props(enc_hilbert, ncols, b, &a1, &a2, &a3); + c1 += a1; + c2 += a2; + c3 += a3; + } + printf("cases=%ld\n", cases); + printf("points=%.0f\n", points); + printf("enc_calls=%ld\n", pgc_enc_calls); + printf("c1=%ld\n", c1); + printf("c2=%ld\n", c2); + printf("c3=%ld\n", c3); + } + + /* ---- C4a: the full-width key prefix is the verified sub-cube index --- + * + * The point is put at the TOP of a sub-cube's coordinate range and then the + * low 64-b bits are filled with noise, so the ordinals are production width + * and land anywhere inside the sub-cube. The top ncols*b bits of the key must + * still be the index C1 to C3 verified exhaustively for that sub-cube. That + * is the bridge from an exhaustible grid to the width the product uses. + * + * THE MISMATCH COUNT ALONE IS A SELF-COMPARISON. kb and kf are both produced + * by the code under test, so an encoder whose output does not depend on its + * input satisfies it: measured, with both product functions replaced by a + * constant fill, c4a_mismatch was 0 and the arm passed. The word "verified" + * in its name was carried entirely by other arms. + * + * So the full-width prefixes are also run through C1's own permutation test: + * the multiset of prefixes over p in [0, N) must be exactly [0, N). A + * constant or collapsing encoder now fails this arm on its own. + */ + { + long mismatch = 0, + permbad = 0; + double points = 0; + long cases = 0; + + for (ncols = 2; ncols <= 8; ncols++) + for (b = 1; ncols * b <= 20; b++) + { + unsigned N = 1u << (ncols * b), + p; + unsigned char *seen = calloc(N, 1); + + if (!seen) + { + fprintf(stderr, "out of memory in C4a\n"); + exit(2); + } + cases++; + for (p = 0; p < N; p++) + { + unsigned c[8], + ixf; + uint64 Xb[8], + Xf[8]; + unsigned char kb[64], + kf[64]; + + pgc_digits(p, ncols, b, c); + for (j = 0; j < ncols; j++) + { + Xb[j] = (uint64) c[j] << (64 - b); + Xf[j] = Xb[j] | (rnd64() & ((((uint64) 1) << (64 - b)) - 1)); + } + cluster_hilbert_transpose(Xb, ncols); + cluster_pack_interleave(Xb, ncols, kb); + cluster_hilbert_transpose(Xf, ncols); + cluster_pack_interleave(Xf, ncols, kf); + points += 1; + ixf = pgc_key_index(kf, ncols, b); + if (pgc_key_index(kb, ncols, b) != ixf) + mismatch++; + if (ixf >= N) + permbad++; + else + { + if (seen[ixf]) + permbad++; + seen[ixf] = 1; + } + } + for (p = 0; p < N; p++) + if (!seen[p]) + permbad++; + free(seen); + } + printf("c4a_cases=%ld\n", cases); + printf("c4a_points=%.0f\n", points); + printf("c4a_mismatch=%ld\n", mismatch); + printf("c4a_perm_bad=%ld\n", permbad); + } + + /* ---- C4b: the published inverse, at production width ----------------- */ + { + long trips = 0, + tripbad = 0, + mutbad = 0, + adj = 0, + adjbad = 0, + adjreenc = 0, + distinct = 0; + int t; + static unsigned char ks[3000][64]; + + for (ncols = 1; ncols <= 8; ncols++) + { + for (t = 0; t < 3000; t++) + { + uint64 T[8], + A0[8], + A1[8], + B[8]; + unsigned char kk[64]; + int c, + carry, + ok0, + ok1; + + /* splitmix64, not the LCG's low byte -- see sm64() above. */ + for (c = 0; c < ncols * 8; c++) + key[c] = (unsigned char) (sm64() >> 56); + memcpy(ks[t], key, (size_t) ncols * 8); + + unpack(key, ncols, T); + memcpy(A0, T, sizeof(uint64) * ncols); + untranspose(A0, ncols, 0); + memcpy(B, A0, sizeof(uint64) * ncols); + cluster_hilbert_transpose(B, ncols); + cluster_pack_interleave(B, ncols, kk); + trips++; + ok0 = (memcmp(kk, key, (size_t) ncols * 8) == 0); + if (!ok0) + tripbad++; + + /* the same round trip through the mutated inverse */ + memcpy(A1, T, sizeof(uint64) * ncols); + untranspose(A1, ncols, 1); + memcpy(B, A1, sizeof(uint64) * ncols); + cluster_hilbert_transpose(B, ncols); + cluster_pack_interleave(B, ncols, kk); + if (memcmp(kk, key, (size_t) ncols * 8) != 0) + mutbad++; + + /* key + 1, as a big-endian integer, must decode to a neighbour */ + memcpy(key2, key, (size_t) ncols * 8); + carry = 1; + for (c = ncols * 8 - 1; c >= 0 && carry; c--) + { + if (key2[c] == 0xff) + key2[c] = 0; + else + { + key2[c]++; + carry = 0; + } + } + if (carry) + continue; /* wrapped past the last index; not a pair */ + unpack(key2, ncols, T); + memcpy(A1, T, sizeof(uint64) * ncols); + untranspose(A1, ncols, 0); + + /* + * ROUTE THE ADJACENCY THROUGH THE CODE UNDER TEST. + * + * A0 and A1 come only from unpack() and untranspose(), both + * defined in this file, so comparing them to each other is a + * self-test of the fixture: measured, this arm stayed green with + * BOTH product functions destroyed, while 89 other arms went red. + * src/columnar_curve.c could have been dropped from the link and + * the arm would still have passed. + * + * So both decoded points are re-encoded here with + * cluster_hilbert_transpose and cluster_pack_interleave, the two + * keys must come back byte-identical, and the adjacency is + * counted only over the pairs the encoder actually reproduced. A + * wrong encoder now either loses the re-encode count or loses the + * adjacency, and either way the arm goes red. + */ + memcpy(B, A1, sizeof(uint64) * ncols); + cluster_hilbert_transpose(B, ncols); + cluster_pack_interleave(B, ncols, kk); + ok1 = (memcmp(kk, key2, (size_t) ncols * 8) == 0); + + adj++; + if (!ok0 || !ok1) + continue; + adjreenc++; + { + int nd = 0, + ok = 1; + + for (c = 0; c < ncols; c++) + if (A0[c] != A1[c]) + { + uint64 d = A0[c] > A1[c] ? A0[c] - A1[c] : A1[c] - A0[c]; + + nd++; + if (d != 1) + ok = 0; + } + if (nd != 1 || !ok) + adjbad++; + } + } + + /* + * How many keys this ncols actually round-tripped, counted rather + * than assumed. The premise arm asserts THIS, not the loop count. + */ + pgc_klen = ncols * 8; + qsort(ks, 3000, sizeof(ks[0]), cmp_rawkey); + distinct++; + for (t = 1; t < 3000; t++) + if (memcmp(ks[t - 1], ks[t], (size_t) pgc_klen) != 0) + distinct++; + } + printf("c4b_roundtrips=%ld\n", trips); + printf("c4b_distinct_keys=%ld\n", distinct); + printf("c4b_roundtrip_bad=%ld\n", tripbad); + printf("c4b_mutated_bad=%ld\n", mutbad); + printf("c4b_adjacent=%ld\n", adj); + printf("c4b_adjacent_reencoded=%ld\n", adjreenc); + printf("c4b_adjacent_bad=%ld\n", adjbad); + } + + /* ---- C5: golden keys, bit order, and the byte count ------------------ */ + for (ncols = 1; ncols <= 8; ncols++) + for (k = 0; k < PGC_NVEC; k++) + { + pgc_makevec(k, ncols, X); + memcpy(Y, X, sizeof(uint64) * ncols); + cluster_hilbert_transpose(Y, ncols); + cluster_pack_interleave(Y, ncols, key); + pgc_hex(key, ncols * 8, h); + printf("golden_n%d_%s=%s\n", ncols, pgc_vecname[k], h); + } + + { + uint64 v[2]; + + v[0] = (uint64) 1 << 63; + v[1] = 0; + cluster_pack_interleave(v, 2, key); + pgc_hex(key, 16, h); + printf("packpin_hi=%s\n", h); + v[0] = 0; + v[1] = (uint64) 1 << 63; + cluster_pack_interleave(v, 2, key); + pgc_hex(key, 16, h); + printf("packpin_lo=%s\n", h); + } + + /* + * How far the key extends, measured rather than assumed. + * + * The same vector is packed into a buffer pre-filled with 0x00 and one + * pre-filled with 0xff. A byte the pack established holds the same value in + * both; a byte it never touched still holds its fill and differs. So the + * count of leading agreeing bytes is the extent of the key, and the tail + * must still hold its fill. + * + * WHAT THIS ARM DOES NOT SEE, AND WHAT ITS NAME MAY THEREFORE NOT SAY. The + * pack zeroes the whole 8*ncols region before it writes, and the memset + * alone makes both buffers agree across the region whatever the loop then + * does. Measured: a pack that is NOTHING BUT the memset -- it never reads + * ord[] -- passes all eight of these arms and all eight of the tail arms, + * and a pack whose loop runs only 32 of its 64 rounds passes them too, while + * moving the memset one byte short reddens six of the eight. The subject is + * the memset's extent, so the arm is named for the extent, and it is paired + * with the content arm below so that "nothing past the key" cannot be + * satisfied by "nothing anywhere". + */ + for (ncols = 1; ncols <= 8; ncols++) + { + unsigned char a[200], + z[200]; + int n, + intact = 1; + + memset(a, 0x00, sizeof(a)); + memset(z, 0xff, sizeof(z)); + pgc_makevec(5, ncols, X); + cluster_pack_interleave(X, ncols, a); + cluster_pack_interleave(X, ncols, z); + for (n = 0; n < (int) sizeof(a) && a[n] == z[n]; n++) + /* count */ ; + for (k = ncols * 8; k < (int) sizeof(a); k++) + if (a[k] != 0x00 || z[k] != 0xff) + intact = 0; + printf("packlen_n%d=%d\n", ncols, n); + printf("packtail_n%d=%s\n", ncols, intact ? "intact" : "clobbered"); + } + + /* + * The content arm the two above cannot be. An all-ones ordinal vector sets + * every bit the pack emits, so the key must be 8*ncols bytes of 0xff with + * nothing beyond. A loop that stops short leaves 0x00 INSIDE the region -- + * verified against a pack truncated to 32 rounds, which produces 4*ncols + * bytes of 0xff and then zeros -- and a pack that never reads ord[] leaves + * the whole region zero. + */ + for (ncols = 1; ncols <= 8; ncols++) + { + unsigned char a[200]; + int nff = 0, + intact = 1; + + memset(a, 0x00, sizeof(a)); + pgc_makevec(1, ncols, X); /* every ordinal UINT64_MAX */ + cluster_pack_interleave(X, ncols, a); + while (nff < (int) sizeof(a) && a[nff] == 0xff) + nff++; + for (k = ncols * 8; k < (int) sizeof(a); k++) + if (a[k] != 0x00) + intact = 0; + printf("packones_n%d=%d of %d ff, tail %s\n", + ncols, nff, ncols * 8, intact ? "intact" : "clobbered"); + } + + /* ---- C6: ncols == 1 --------------------------------------------------- */ + { + static const char *const nm[4] = {"zero", "one", "topbit", "max"}; + uint64 vals[4]; + long mism = 0, + absmism = 0, + n6 = 0; + int t; + + vals[0] = 0; + vals[1] = 1; + vals[2] = (uint64) 1 << 63; + vals[3] = UINT64_MAX; + for (k = 0; k < 4; k++) + { + X[0] = vals[k]; + cluster_hilbert_transpose(X, 1); + cluster_pack_interleave(X, 1, key); + pgc_hex(key, 8, h); + printf("ident_n1_%s=%s\n", nm[k], h); + } + + /* + * The relative half compares two calls into the code under test, so a + * packing bug that hits both identically survives it and a pack that + * writes nothing satisfies it with eight zero bytes on each side. It is + * kept -- it is the arm that says the Hilbert path and the Z-order path + * agree at one column -- but the same 200,000 values are ALSO compared + * against the big-endian bytes of x built right here, by shifting, with + * nothing from src/ in the path. That turns the four absolute hex pins + * into 200,004 of them. + */ + for (t = 0; t < 200000; t++) + { + uint64 x = rnd64(); + unsigned char kh[8], + kz[8], + be[8]; + int q; + + X[0] = x; + cluster_hilbert_transpose(X, 1); + cluster_pack_interleave(X, 1, kh); + Y[0] = x; + cluster_pack_interleave(Y, 1, kz); + for (q = 0; q < 8; q++) + be[q] = (unsigned char) (x >> (56 - 8 * q)); + n6++; + if (memcmp(kh, kz, 8) != 0) + mism++; + if (memcmp(kh, be, 8) != 0) + absmism++; + } + printf("c6_random=%ld\n", n6); + printf("c6_random_mismatch=%ld\n", mism); + printf("c6_random_absolute_mismatch=%ld\n", absmism); + } + + /* ---- C7: the Z-order key, which must not have moved ------------------ */ + for (ncols = 1; ncols <= 8; ncols++) + for (k = 0; k < PGC_NVEC; k++) + { + pgc_makevec(k, ncols, X); + cluster_pack_interleave(X, ncols, key); + pgc_hex(key, ncols * 8, h); + printf("zorder_n%d_%s=%s\n", ncols, pgc_vecname[k], h); + } + + (void) pin_transpose; + (void) pin_pack; + return 0; +} +CEOF + +bat_ready=no +if [ ! -f "$CURVE_C" ] || [ ! -f "$CURVE_H" ]; then + _missing="" + [ -f "$CURVE_C" ] || _missing="$_missing src/columnar_curve.c" + [ -f "$CURVE_H" ] || _missing="$_missing src/columnar_curve.h" + echo "-- the code under test is absent:$_missing" + arms_failed "$BATTERY_ARMS" "the code under test is absent:$_missing" +else + cp "$CURVE_C" "$CURVE_H" "$WORK/" + if "$CC" -O2 -Wall -Werror=incompatible-pointer-types -I"$WORK" \ + -o "$WORK/battery" "$WORK/battery.c" "$WORK/columnar_curve.c" \ + >"$BATLOG" 2>&1 && "$WORK/battery" > "$BATOUT" 2>>"$BATLOG"; then + bat_ready=yes + echo "-- battery: built and ran against $CURVE_C" + else + echo "---- the battery would not build or run against $CURVE_C ----" + sed 's/^/ /' "$BATLOG" | head -40 + arms_failed "$BATTERY_ARMS" "the battery would not build or run; see the log above" + fi +fi + +if [ "$bat_ready" = yes ]; then + echo "-- C1/C2/C3 violations: $(val "$BATOUT" c1) / $(val "$BATOUT" c2) / $(val "$BATOUT" c3)" + + # 52 cases and 6,344,330 points is sum over ncols 1..8 and every b with + # ncols*b <= 20 of 2^(ncols*b), computed independently of this program. Pinned + # exactly, not bounded: a loop that stopped early would still satisfy a bound. + check "C0 premise: the battery encoded every case and every point" \ + "$(val "$BATOUT" cases) cases, $(val "$BATOUT" points) points" \ + "52 cases, 6344330 points" + # And the same number counted where enc() is CALLED. The arm above is + # arithmetic over the loop bounds and reports 6,344,330 whether or not + # anything was encoded: it passed under every mutant tried, including one + # whose two product functions did nothing at all. + check_num "C0 premise: the encoder was called once per point" \ + "$(val "$BATOUT" enc_calls)" "6344330" + + check_num "C1 the Hilbert index set is exactly the contiguous range" \ + "$(val "$BATOUT" c1)" "0" + check_num "C2 consecutive points in memcmp key order are unit-adjacent" \ + "$(val "$BATOUT" c2)" "0" + check_num "C3 every dyadic sub-cube occupies a contiguous run of indices" \ + "$(val "$BATOUT" c3)" "0" + + # 32 cases and 4,247,180 points is the same sum over ncols 2..8. ncols == 1 is + # excluded because a one-column sub-cube is the whole coordinate, so the + # bridge has nothing to say there; C6 pins ncols == 1 instead. + check "C4a premise: the bridge covered every case and every point" \ + "$(val "$BATOUT" c4a_cases) cases, $(val "$BATOUT" c4a_points) points" \ + "32 cases, 4247180 points" + # Named for what it measures. It used to say "equals the VERIFIED sub-cube + # index", but both sides are produced by the code under test, so a constant + # encoder satisfied it and the word "verified" was carried by other arms. + check_num "C4a the full-width key prefix does not move when the low bits change" \ + "$(val "$BATOUT" c4a_mismatch)" "0" + # The arm above compares the encoder's own base prefix with its own + # noise-filled prefix, so a constant encoder satisfies it. This one holds the + # full-width prefixes to C1's permutation test and a constant encoder cannot. + check_num "C4a the full-width key prefixes are exactly the index range" \ + "$(val "$BATOUT" c4a_perm_bad)" "0" + + # DISTINCT keys, counted by the program, not iterations counted by the loop. + # The old premise certified 24,000 round trips over a key stream whose real + # period gave between 8 and 64 distinct keys per ncols -- 344 in all -- and + # that gap is what let a wrong encoder pass this file 138/138. + check_num "C4b premise: 3,000 DISTINCT full-width keys per ncols were round-tripped" \ + "$(val "$BATOUT" c4b_distinct_keys)" "24000" + check_num "C4b the published inverse round-trips full-width keys" \ + "$(val "$BATOUT" c4b_roundtrip_bad)" "0" + # The control, as a JOINT condition. A round trip against an inverse derived + # from the forward code passes whatever either one does, so the arm above is + # evidence only if breaking the inverse alone breaks it. But "every round trip + # through the mutated inverse failed" is also satisfied when the FORWARD code + # is broken and every round trip fails for that reason -- measured, the arm + # passed with both product functions destroyed, which is precisely the state + # it exists to rule out. So the clean count is asserted in the same arm, and + # the control can no longer be satisfied by a run whose real round trip failed. + check "C4b control: a mutated inverse breaks every round trip the clean one made" \ + "clean $(val "$BATOUT" c4b_roundtrip_bad) bad, mutated $(val "$BATOUT" c4b_mutated_bad) of $(val "$BATOUT" c4b_roundtrips)" \ + "clean 0 bad, mutated 24000 of 24000" + check_num "C4b premise: consecutive key pairs were formed" \ + "$(val "$BATOUT" c4b_adjacent)" "24000" + # Both decoded points are re-encoded by the code under test and must give the + # two keys back; the adjacency is counted only over pairs that did. Without + # the re-encode this arm read only the suite's own inverse and stayed green + # with src/columnar_curve.c effectively deleted. + check "C4b consecutive keys decode to unit-adjacent points, re-encoded by the code under test" \ + "$(val "$BATOUT" c4b_adjacent_bad) bad over $(val "$BATOUT" c4b_adjacent_reencoded) re-encoded of $(val "$BATOUT" c4b_adjacent) pairs" \ + "0 bad over 24000 re-encoded of 24000 pairs" + + for _n in 1 2 3 4 5 6 7 8; do + for _v in $PGC_VECNAMES; do + check_text "C5 golden Hilbert keys: ncols=$_n $_v" \ + "$(val "$BATOUT" "golden_n${_n}_${_v}")" \ + "$(frozen "$PGC_GOLDEN_HILBERT" "$_n" "$_v")" + done + done + + check_text "C5 the pack bit order: pack(2^63, 0) starts 0x80" \ + "$(val "$BATOUT" packpin_hi)" "80000000000000000000000000000000" + check_text "C5 the pack bit order: pack(0, 2^63) starts 0x40" \ + "$(val "$BATOUT" packpin_lo)" "40000000000000000000000000000000" + + for _n in 1 2 3 4 5 6 7 8; do + # Named for what it measures. A pack that never reads its input passes + # both of these; the third is what reads the contents. + check_num "C5 the pack establishes exactly 8*ncols bytes: ncols=$_n" \ + "$(val "$BATOUT" "packlen_n${_n}")" "$((_n * 8))" + check_text "C5 the pack wrote nothing past the key: ncols=$_n" \ + "$(val "$BATOUT" "packtail_n${_n}")" "intact" + check_text "C5 the pack sets every bit of the key for an all-ones vector: ncols=$_n" \ + "$(val "$BATOUT" "packones_n${_n}")" \ + "$((_n * 8)) of $((_n * 8)) ff, tail intact" + done + + check_text "C6 ncols == 1 is the big-endian ordinal: 0" \ + "$(val "$BATOUT" ident_n1_zero)" "0000000000000000" + check_text "C6 ncols == 1 is the big-endian ordinal: 1" \ + "$(val "$BATOUT" ident_n1_one)" "0000000000000001" + check_text "C6 ncols == 1 is the big-endian ordinal: 2^63" \ + "$(val "$BATOUT" ident_n1_topbit)" "8000000000000000" + check_text "C6 ncols == 1 is the big-endian ordinal: UINT64_MAX" \ + "$(val "$BATOUT" ident_n1_max)" "ffffffffffffffff" + # Named for what it counts. Measured: wrapping the two encode calls in a + # condition that skips half of them leaves this arm GREEN at 200,000, because + # n6 is incremented by the loop and not by the encoder. It is a loop-ran + # premise and says so; the arm below is what noticed the 100,000 skipped + # encodes, at "got [100000] want [0]". + check_num "C6 premise: the comparison loop ran 200,000 times" \ + "$(val "$BATOUT" c6_random)" "200000" + # The absolute half: the key against the big-endian bytes of the input, + # built in the battery by shifting, with nothing from src/ on that side. + check_num "C6 ncols == 1 is the big-endian ordinal over 200,000 random values" \ + "$(val "$BATOUT" c6_random_absolute_mismatch)" "0" + # The relative half alone is vacuous: a packing bug that hits both paths + # identically survives it, and a pack that writes nothing satisfies it with + # eight zero bytes on each side. It says the two paths agree at one column; + # the arm above is what says WHICH value they agree on. + check_num "C6 ncols == 1 agrees with Z-order over 200,000 random values" \ + "$(val "$BATOUT" c6_random_mismatch)" "0" + + for _n in 1 2 3 4 5 6 7 8; do + for _v in $PGC_VECNAMES; do + check_text "C7 the Z-order pack is byte-for-byte what it was: ncols=$_n $_v" \ + "$(val "$BATOUT" "zorder_n${_n}_${_v}")" \ + "$(frozen "$PGC_GOLDEN_ZORDER" "$_n" "$_v")" + done + done +fi + +# The battery's own accounting, beside the instrument's. pgc_summary reports one +# total, and seven of its greens used to be arms that never link the code under +# test -- a reader counting greens counted those seven as coverage. Printed here +# so "PASSED" cannot be reached by calibration alone. +echo "-- arm split: $_ctl_checks instrument ($_ctl_passed passed)," \ + "$((PGC_CHECKS - _ctl_checks)) battery" \ + "($((PGC_PASSED - _ctl_passed)) passed)" + +pgc_summary From 0deef380724cb088bd0eabba45d347e936ef1cc5 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Tue, 8 Sep 2026 19:50:22 -0600 Subject: [PATCH 2/6] feat: the Hilbert curve key, and the battery goes green (#889) src/columnar_curve.{c,h}: two pure functions over uint64 ordinals, neither touching a PostgreSQL type beyond uint64, which is what lets the suite compile this file against four-line stub headers and exercise the curve with no server tree. cluster_hilbert_transpose -- Skilling 2004 AxestoTranspose, b fixed at 64 cluster_pack_interleave -- the interleave loop, moved from cluster_zorder_key Skilling's observation is that the Hilbert index IS the MSB-round-first interleave of the transposed coordinates, bit for bit the packing Z-order already used. So the curves differ by one in-place pass and share the packer, the width and everything downstream: Z-order: ordinals -> pack Hilbert: ordinals -> transpose -> pack test/hilbert_curve.sh: 184 passed, 0 failed, 0 unrunnable. It was 174 failed at eb45776, where the arms were committed before this code existed. REMOVAL PROOF, run here rather than inherited. Five mutants, each asserted to have applied (the patch raises if its anchor is missing, and a mutation whose md5 matches the original is refused as a no-op), each restored afterwards, with the file verified back at bf5f7e07105f at the end: unmutated bf5f7e07105f 184 passed 0 failed M1 transpose is identity 916c58ad8bd9 133 passed 51 failed M2 no Gray encode b9f72ca2c94d 133 passed 51 failed M3 no final t fold 9881723ebde0 145 passed 39 failed M4 invert/exchange swapped a1deb7f2cef1 122 passed 62 failed M5 pack LSB-first in byte 57dacde80b1e 69 passed 115 failed M1 is the one that matters: making the transpose a no-op turns this back into Z-order, the curve that already ships, and 51 arms notice. That is the evidence the battery distinguishes the new curve from the old one rather than merely approving of it. The 96 golden byte vectors passing is the strongest single signal, because they could not have been generated from this file. They were derived from Skilling's published algorithm and independently reproduced byte-for-byte by two reviewers BEFORE src/columnar_curve.c existed. They are also the only arms that pin WHICH Hilbert curve this is: Skilling's differs from Butz/Hamilton for ncols >= 3 and both are valid, so the property arms cannot tell them apart and the bytes are an on-disk format commitment. The suite is now registered in run_all_versions.sh, which the red commit deliberately did not do. harness_selftest: 261 passed, 0 failed, including "the suite list is sorted in C order" -- note C order, not the locale's: plain `sort` disagrees with the array and always has, so it is the wrong instrument for that question. Nothing calls the curve yet. Wiring it into the clustering paths, the gate and the daemon is the next commit, per the owner ruling recorded on #889. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- Makefile | 1 + src/columnar_curve.c | 127 +++++++++++++++++++++++++++++++++++++++ src/columnar_curve.h | 39 ++++++++++++ test/run_all_versions.sh | 1 + 4 files changed, 168 insertions(+) create mode 100644 src/columnar_curve.c create mode 100644 src/columnar_curve.h diff --git a/Makefile b/Makefile index c1cb7b81..4ddd3a58 100644 --- a/Makefile +++ b/Makefile @@ -17,6 +17,7 @@ OBJS = \ src/columnar_customscan.o \ src/columnar_vector.o \ src/columnar_vacuum.o \ + src/columnar_curve.o \ src/columnar_unique.o \ src/columnar_row_lock.o \ src/columnar_arrow.o \ diff --git a/src/columnar_curve.c b/src/columnar_curve.c new file mode 100644 index 00000000..8f945b8b --- /dev/null +++ b/src/columnar_curve.c @@ -0,0 +1,127 @@ +/*------------------------------------------------------------------------- + * + * columnar_curve.c + * Space-filling curve keys for clustering (#889). + * + * Both curves produce a fixed-width byte string whose memcmp order IS the + * curve's order, so the rewrite can sort on it with the ordinary bytea + * operator and nothing downstream needs to know which curve produced it. + * + * Held by test/hilbert_curve.sh, which compiles this file against stub headers + * and checks the properties that define a Hilbert curve rather than the + * arithmetic that happens to implement one: that the index set is exactly the + * contiguous range, that consecutive indices are unit-adjacent, and that every + * dyadic sub-cube occupies a contiguous run. The first two alone are not + * enough -- a boustrophedon order satisfies adjacency and is not a Hilbert + * curve, and Z-order satisfies contiguity and is the curve we already had -- + * so the suite carries both as deliberately wrong controls. + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "columnar_curve.h" + +/* + * cluster_hilbert_transpose + * Axes to Hilbert transpose, in place, at b = 64 bits per coordinate. + * + * Skilling, "Programming the Hilbert curve", AIP Conf. Proc. 707 (2004), + * AxestoTranspose, public domain. Transcribed with b fixed at 64 and the + * coordinate type fixed at uint64; the structure is unchanged. + * + * After this returns, X holds the transpose of the Hilbert index: the index's + * bits distributed across the ncols words so that taking bit r of every word, + * most significant round first, spells the index. That is exactly what + * cluster_pack_interleave then does, which is why the two curves share a + * packer and a width. + * + * ncols == 1 is the identity. The Q loop's two passes cancel, the Gray encode + * has no neighbour to fold in, and the final t is zero -- so a one-column + * Hilbert key is the plain big-endian ordinal, byte for byte what Z-order + * produces. The suite pins that both relatively and against fixed hex, because + * the relative half alone would survive a packing bug that hit both curves. + */ +void +cluster_hilbert_transpose(uint64 *X, int ncols) +{ + const uint64 M = ((uint64) 1) << 63; + uint64 P; + uint64 Q; + uint64 t; + int i; + + /* Inverse undo. */ + for (Q = M; Q > 1; Q >>= 1) + { + P = Q - 1; + for (i = 0; i < ncols; i++) + { + if (X[i] & Q) + X[0] ^= P; /* invert */ + else + { + /* exchange */ + t = (X[0] ^ X[i]) & P; + X[0] ^= t; + X[i] ^= t; + } + } + } + + /* Gray encode. */ + for (i = 1; i < ncols; i++) + X[i] ^= X[i - 1]; + + t = 0; + for (Q = M; Q > 1; Q >>= 1) + { + if (X[ncols - 1] & Q) + t ^= Q - 1; + } + for (i = 0; i < ncols; i++) + X[i] ^= t; +} + +/* + * cluster_pack_interleave + * Interleave ncols ordinals MSB-first into 8*ncols bytes. + * + * Moved verbatim from cluster_zorder_key in columnar_vacuum.c, where it has + * always been the second half of the Z-order key. The output bit stream is + * ord[0].bit63, ord[1].bit63, ..., ord[n-1].bit63, ord[0].bit62, ... packed + * MSB-first within each byte. + * + * Why memcmp order equals the curve's order, in two steps that are both + * arithmetic rather than assertion. Sixty-four rounds of ncols bits fill + * 8*ncols bytes exactly, with no padding, so the result is the fixed-width + * big-endian base-2^ncols expansion of a single integer. And memcmp over two + * equal-length big-endian expansions is numeric order, because the first + * differing byte sits at place value 256^k and everything after it sums to at + * most 256^k - 1. Every key in one rewrite has the same length, ncols being + * fixed for the whole run, so bytea's length tie-break can never fire. + * + * Establishes every byte it is given rather than OR-ing into whatever was + * there, so a caller cannot leave stale bits in the tail and a test cannot + * mistake a memset for a key. + */ +void +cluster_pack_interleave(const uint64 *ord, int ncols, unsigned char *out) +{ + int keybytes = ncols * 8; + int outbit = 0; + int c; + int r; + + memset(out, 0, keybytes); + + for (r = 63; r >= 0; r--) + { + for (c = 0; c < ncols; c++) + { + if ((ord[c] >> r) & 1) + out[outbit >> 3] |= (unsigned char) (0x80 >> (outbit & 7)); + outbit++; + } + } +} diff --git a/src/columnar_curve.h b/src/columnar_curve.h new file mode 100644 index 00000000..1b45c380 --- /dev/null +++ b/src/columnar_curve.h @@ -0,0 +1,39 @@ +/*------------------------------------------------------------------------- + * + * columnar_curve.h + * Space-filling curve keys for clustering (#889). + * + * Two pure functions over uint64 ordinals. Neither touches a PostgreSQL type + * beyond uint64, which is what lets test/hilbert_curve.sh compile this file + * against four-line stub headers and exercise the curve with no server tree. + * + * The clustering key is built in two steps: + * + * Z-order: ordinals -> cluster_pack_interleave + * Hilbert: ordinals -> cluster_hilbert_transpose -> cluster_pack_interleave + * + * Skilling's observation is that the Hilbert index IS the MSB-round-first + * interleave of the TRANSPOSED coordinates -- bit for bit the same packing + * Z-order already used. So the two curves differ by one in-place pass and + * share everything else, including the key width. + * + *------------------------------------------------------------------------- + */ +#ifndef COLUMNAR_CURVE_H +#define COLUMNAR_CURVE_H + +/* + * Transform ncols 64-bit coordinates in place, from axes to the Hilbert + * transpose (Skilling 2004, AxestoTranspose, at b = 64). + */ +extern void cluster_hilbert_transpose(uint64 *X, int ncols); + +/* + * Interleave ncols 64-bit ordinals MSB-first into out[0 .. 8*ncols-1], so that + * memcmp order over the result equals the curve's order. Establishes every one + * of those bytes; the caller need not clear them. + */ +extern void cluster_pack_interleave(const uint64 *ord, int ncols, + unsigned char *out); + +#endif /* COLUMNAR_CURVE_H */ diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index 603e1f9c..8ebcea94 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -92,6 +92,7 @@ SUITES=( groupagg_table_sizing hardening harness_selftest + hilbert_curve iceberg_catalog iceberg_data_files iceberg_deletes From 0ac8759e32ae311e922f65afd4cd6710a46f6f14 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Tue, 8 Sep 2026 19:54:02 -0600 Subject: [PATCH 3/6] refactor: the clustering key's two halves become shareable (#889) Inert by construction and by measurement. No behaviour changes here; this is the seam the Hilbert key needs. cluster_key_ordinals extracted -- the type gate, the NULL rule and cluster_type_ordinal now have one implementation rather than one per curve cluster_zorder_key keeps its name, its signature and its output, and now calls cluster_pack_interleave for the interleave The interleave moved to columnar_curve.c in 4557930 rather than being copied, so there is no second copy to drift. WHY THIS IS PROVED RATHER THAN ASSERTED. A refactor validated by "the tests still pass" is validated against itself when the tests call the refactored code on both sides. test/hilbert_curve.sh's 48 C7 arms hold cluster_pack_interleave against hex literals captured from this loop BEFORE it moved, so the comparison is new code against a frozen record. Those arms were committed at eb45776, when columnar_curve.c did not exist and could not have generated them. Measured with COPT=-Werror, PG18.4, build OK with 0 warnings: native_cluster 11 passed 0 failed sorted_pathkeys 113 passed 0 failed native_recluster 12 passed 0 failed sorted_mark_rename 46 passed 0 failed recluster_extent 17 passed 0 failed vacuum_sorted_gate 34 passed 0 failed recluster_gate 13 passed 0 failed hilbert_curve 184 passed 0 failed Every count identical to the baseline taken on this branch before any change. cluster_hilbert_key is deliberately NOT here. It was written, and removed again, because a static function with no caller is -Werror=unused-function and the build fails: src/columnar_vacuum.c:1104:1: error: 'cluster_hilbert_key' defined but not used [-Werror=unused-function] So the key arrives in the same commit as the entry points that call it, rather than sitting unreachable in the tree for a commit. That is a constraint worth stating: on this project a new static helper cannot land ahead of its caller. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- src/columnar_vacuum.c | 60 ++++++++++++++++++++++++++----------------- 1 file changed, 36 insertions(+), 24 deletions(-) diff --git a/src/columnar_vacuum.c b/src/columnar_vacuum.c index 5a1c96b6..0aa20336 100644 --- a/src/columnar_vacuum.c +++ b/src/columnar_vacuum.c @@ -21,6 +21,7 @@ */ #include "columnar.h" #include "columnar_metadata.h" +#include "columnar_curve.h" #include "columnar_storage.h" #include "columnar_write_state.h" #include "columnar_compat.h" @@ -1037,25 +1038,21 @@ cluster_type_supported(Oid typid) } /* - * Build the Z-order key for one row: interleave the ncols column ordinals - * MSB-first into an 8*ncols-byte string. Output bit stream is - * ord[0].bit63, ord[1].bit63, ..., ord[n-1].bit63, ord[0].bit62, ... packed - * MSB-first, so lexicographic (memcmp) order over the bytea equals Z-order. + * cluster_key_ordinals + * The ncols clustering columns of one row, as order-preserving uint64s. + * + * Shared by both curves so the type gate, the NULL rule and + * cluster_type_ordinal have exactly one implementation. NULL maps to 0, which + * sorts it first -- note that this is a statement about the ORDINAL and not + * about the key: under Hilbert the index is not monotone in any single + * coordinate, so a row NULL in one clustering column is not thereby first. */ -static bytea * -cluster_zorder_key(Datum *values, bool *isnull, AttrNumber *atts, int ncols, - TupleDesc tupdesc) +static uint64 * +cluster_key_ordinals(Datum *values, bool *isnull, AttrNumber *atts, int ncols, + TupleDesc tupdesc) { - int keybytes = ncols * 8; - bytea *result = (bytea *) palloc(VARHDRSZ + keybytes); - unsigned char *out = (unsigned char *) VARDATA(result); uint64 *ord = (uint64 *) palloc(ncols * sizeof(uint64)); int c; - int r; - int outbit = 0; - - SET_VARSIZE(result, VARHDRSZ + keybytes); - memset(out, 0, keybytes); for (c = 0; c < ncols; c++) { @@ -1065,21 +1062,36 @@ cluster_zorder_key(Datum *values, bool *isnull, AttrNumber *atts, int ncols, ord[c] = isnull[a - 1] ? 0 : cluster_type_ordinal(values[a - 1], att->atttypid); } + return ord; +} - for (r = 63; r >= 0; r--) - { - for (c = 0; c < ncols; c++) - { - if ((ord[c] >> r) & 1) - out[outbit >> 3] |= (unsigned char) (0x80 >> (outbit & 7)); - outbit++; - } - } +/* + * Build the Z-order key for one row: interleave the ncols column ordinals + * MSB-first into an 8*ncols-byte string. Output bit stream is + * ord[0].bit63, ord[1].bit63, ..., ord[n-1].bit63, ord[0].bit62, ... packed + * MSB-first, so lexicographic (memcmp) order over the bytea equals Z-order. + * + * The interleave now lives in columnar_curve.c, shared with the Hilbert key. + * The bytes it produces are frozen by test/hilbert_curve.sh's C7 arms, which + * hold it against hex captured from this loop BEFORE it moved -- so this + * refactor is checked against a record rather than against itself. + */ +static bytea * +cluster_zorder_key(Datum *values, bool *isnull, AttrNumber *atts, int ncols, + TupleDesc tupdesc) +{ + int keybytes = ncols * 8; + bytea *result = (bytea *) palloc(VARHDRSZ + keybytes); + uint64 *ord = cluster_key_ordinals(values, isnull, atts, ncols, tupdesc); + + SET_VARSIZE(result, VARHDRSZ + keybytes); + cluster_pack_interleave(ord, ncols, (unsigned char *) VARDATA(result)); pfree(ord); return result; } + /* * pgcolumnar_relation_storageid * SQL: columnar.get_storage_id(regclass) -> bigint. Reads the relation's From 1f52c90b1f864952ce116b9cd53e8437e8f9a526 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Tue, 8 Sep 2026 20:50:37 -0600 Subject: [PATCH 4/6] test: the SQL half's arms, red before the verbs exist (#889) The surface, the recorded kind, the self-gate, vacuum_sorted and the daemon. 181 checks: 98 passed, 74 failed, 9 unrunnable. Red, and red because pgcolumnar.cluster_hilbert and pgcolumnar.recluster_hilbert do not exist -- 27 of the reds carry 42883 directly, and the rest are fixtures that could not be built because of them. WHAT THE ADVERSARIAL PASS CAUGHT, which is the reason this file is 1061 lines and not 650. Three auditors built deliberately wrong implementations as SQL shims and ran the suite against them. The flagship gate arm was green on a stub: shim C: recluster_hilbert is `BEGIN RETURN 0; END`, touching nothing -> S4(a) all three arms PASS, including "and the physical layout is byte-identical, SO THE GATE FIRED rather than the count being 0 for some other reason" A stub returns 0 AND leaves the layout untouched, so physlayout was the same observation twice rather than corroboration. The header stated the rule correctly and the arm broke it. Fixed by making (a) carry its own positive control: decay the table just gated, require the SAME call to return >0 and move the layout, then require the gate to close again. That turns "nothing happened" into "this verb can do work here and chose not to", which is what the gate firing means. shim A: records 'hilbert', writes ZORDER bytes -> "(b) and the recorded kind became hilbert" PASS, beside two red siblings A bare string assertion with no layout beside it. The kind arms are compound now. THE 9 UNRUNNABLE ARE THE POINT, not a shortfall. Each is an arm that previously passed on nothing and now says why it cannot be evaluated: UNRUN (a) and the physical layout is byte-identical: UNMET_PRECONDITION: the call did not return 0 (returned []), so an unchanged layout would not be about the gate UNRUN and the layout the daemon produced IS a Hilbert layout: [...] av_hi already matched the reference [...] so the comparison cannot be a change That is #858's third state earning its keep in the first suite written after it landed. Failure still outranks unrunnable, so the suite reports FAILED. S7 drives the actual daemon rather than calling recluster() and reasoning about it, per the ruling on #889. It asserts the launcher is running, that the daemon was off beforehand and off again after, reads the daemon's own log line for attribution, and freezes both hand-driven references before the window so the comparison is a change rather than two untouched twins. NOT REGISTERED, deliberately -- a red suite must not enter the matrix. The cost, stated rather than discovered later: harness_selftest.sh fails its "every suite is registered" arm while this file exists unregistered. Measured both ways: 260 passed + 1 failed with it present, 261 + 0 with it absent. This file is the sole cause. It gets registered in the commit that lands the verbs. WHAT NO ARM HERE CAN DO: reach the C encoder. Nothing in this suite proves the bytes are Hilbert rather than merely "not the other two kinds" -- that is test/hilbert_curve.sh's 184 arms, and the 96 golden vectors in particular. These arms buy the surface, the gate, the kind and the daemon. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- test/hilbert_cluster.sh | 1061 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 1061 insertions(+) create mode 100755 test/hilbert_cluster.sh diff --git a/test/hilbert_cluster.sh b/test/hilbert_cluster.sh new file mode 100755 index 00000000..86c314a5 --- /dev/null +++ b/test/hilbert_cluster.sh @@ -0,0 +1,1061 @@ +#!/usr/bin/env bash +# +# pgColumnar Hilbert clustering: the SQL surface, the recorded kind, the +# self-gate and the daemon (issue #889, SQL half). +# +# WHAT THIS SUITE IS FOR +# +# test/hilbert_curve.sh pins the CURVE, in C, by its mathematical properties and +# by frozen bytes. Nothing in it goes through SQL, so nothing in it can tell +# whether the curve is ever REACHED from a user statement, whether the table +# remembers that it was laid on that curve, or whether the maintenance daemon +# preserves the choice. This file is that half and only that half; it does not +# re-test the encoder. +# +# THE SURFACE, AND WHY IT IS TWO NEW VERBS RATHER THAN A PARAMETER +# +# pgcolumnar.cluster_hilbert(regclass, VARIADIC name[]) +# pgcolumnar.recluster_hilbert(regclass, VARIADIC name[]) +# +# PostgreSQL refuses to extend the existing signature in either direction: a +# defaulted parameter cannot precede a VARIADIC one, and VARIADIC must be last. +# An array-plus-kind overload breaks the documented cluster('t','a','b') call +# style with "function ... is not unique". Both measured on 18.4. The surface is +# settled; these arms pin it rather than reopen it. +# +# THE OWNER RULING (2026-09-09, recorded on #889): THE CURVE IS STICKY +# +# sorted_kind is the table's DECLARED INTENT, not a property of each call: +# +# - the daemon dispatches on the recorded kind, so a Hilbert table stays +# Hilbert; +# - plain recluster() on a Hilbert table WITH A MATCHING KEY is a no-op, not +# a silent conversion to Z-order; +# - switching curves requires naming the other verb explicitly; +# - vacuum_sorted must not clobber a Hilbert table. Its gate +# (vacuum_sorted_gate_is_noop, src/columnar_vacuum.c) refuses anything that +# is not exactly 'lexicographic', so TODAY it would rewrite the table and +# relabel it. +# +# S6 IS THE ONE ARM WHOSE INTENDED VALUE IS A READING RATHER THAN A QUOTE. The +# ruling says vacuum_sorted "must not clobber"; it does not say in so many words +# whether the correct behaviour is a no-op or an honest relabel. This suite pins +# the no-op, because the parenthetical names "rewrite AND relabel" as the defect +# and a relabel alone is honest labelling. If the owner meant the other thing, +# S6 is the arm to change and it is deliberately isolated so that it can be. +# +# WHY EACH ARM IS SHAPED THE WAY IT IS +# +# - SQLSTATE, never message text. A grep for "permission denied" is also +# satisfied by a login FATAL, a missing function (42883), a bad argument +# (22023) or a transaction-block refusal (25001). Only aclcheck_error +# produces 42501, and a role that cannot open a session carries no SQLSTATE +# at all -- so the session premise is asserted first. +# - AND "no error" MUST MEAN THE SERVER ANSWERED. sqlstate() reports a state +# only when it finds one, so a probe that never reached the server used to +# read as success and satisfied its own removal proof. Every probe now +# carries a sentinel statement behind the one under test: no sentinel in the +# output is PROBE_UNREACHABLE, not noerror, and a NOLOGIN role proves the +# probe can still say so. +# - A CALL MADE WITH psql_run IS NOT AN ASSERTION. This suite runs under +# `set -uo pipefail` with no -e, so a verb that RAISES leaves no trace: the +# fixture is simply not rewritten, and "the verb left the table alone" is +# byte-identical to "the verb threw". A vacuum_sorted that raises on every +# Hilbert table scored as the pinned no-op and all of S6 went green +# (measured, 2026-09-09). So every maintenance verb here is called through +# hrun(), which asserts the SQLSTATE, or through q(), whose empty result on +# an error is rejected by check_num. +# - pgc_set_hash IS ORDER-BLIND BY DESIGN. A parity-only "it only reorders" +# arm therefore passes on a table nothing touched, so S2 carries the +# order-SENSITIVE half beside it and neither half stands alone. +# - AND AN "IT MOVED" ARM MUST NOT BE SATISFIED BY A MEASUREMENT NOBODY TOOK. +# pgc_seq_hash returns a unique QUERY_ERROR.$seq when its query cannot run +# and EMPTY when the relation has no rows; physlayout returns NO_LAYOUT when +# the relation has no stripes. Those sentinels stop two FAILED measurements +# comparing EQUAL (#418) -- and they are unequal to every baseline, so they +# satisfy an inequality arm outright. Thirteen arms were of that shape; all +# of them now go through changed()/differs(), which report UNMEASURED rather +# than "moved" when either side is a sentinel. +# - physlayout() IS BLIND TO AN EAGER REWRITE, AND THAT IS MEASURED HERE +# RATHER THAN ASSUMED. The eager verbs (cluster, cluster_hilbert, +# vacuum_sorted) swap the relfilenode and rewrite into a fresh file, so the +# (stripeid, fileoffset, rowcount) multiset comes back byte-identical: at +# 20 stripes and at 1, vacuum_sorted and cluster both left the digest +# unchanged while the row order moved (measured on PG17.10, 2026-09-08). +# So physlayout is the right corroboration for the ONLINE recluster gate, +# where retired groups and new file offsets do move it -- and it is the +# WRONG instrument for S2 and S6, which use the order digest and carry a +# control that pins this blindness on a verb that exists today. +# - A RETURN OF 0 IS NOT EVIDENCE THE GATE FIRED, AND NEITHER IS AN UNCHANGED +# LAYOUT BESIDE IT. Both are the same observation -- nothing happened -- and +# "the function does not exist" is the strongest instance of it: with the +# verb absent, "the physical layout is byte-identical, so the gate fired" +# printed PASS, and so did all three arms of (a) against a stub defined as +# `BEGIN RETURN 0; END` (both measured, 2026-09-09). A gate is proved by a +# POSITIVE CONTROL ON THE SAME TABLE: S4(a) appends a tail to the very table +# that was just gated, requires the SAME call to do work, and then requires +# the gate to close again. "This verb can rewrite this table and chose not +# to" is what "the gate fired" means; nothing weaker distinguishes a gate +# from a dead verb. +# - S7 DRIVES THE DAEMON. Calling recluster() by hand and reasoning about +# what the daemon would do answers the neighbouring question. The daemon +# hard-codes its recluster call at src/columnar_autovacuum.c:283-291, and +# that line is the subject of the ruling, so the daemon has to run. +# - AND THE DAEMON'S REFERENCE MUST BE FROZEN WHILE IT RUNS. "av_hi came to +# match the hand-driven hilbert twin" is also satisfied by the daemon +# reclustering the TWIN, which is what happened when a stubbed +# recluster_hilbert left the twin's tail unfolded and therefore due +# (measured). The twin's digest and appended count are captured before the +# daemon is enabled and asserted unchanged after, and the daemon's own LOG +# LINE names the dispatch, so the actor is read rather than inferred. +# +# WHAT DEFENDS THE CURVE, AND WHAT DOES NOT +# +# There is no SQL exposure of cluster_hilbert_transpose, so no arm here can +# compare a verb's output against the pinned encoder; that would need a +# test-only SQL helper in the install script, which is #889's own change to +# make. What this suite CAN do is refuse to accept a relabelled Z-order +# implementation, and four arms carry that and no others: +# +# S3 "three different physical orders (hilbert vs zorder)" -- the EAGER verb +# S4 "(d) ... is NOT the ZORDER rewrite over (b,a)" -- the ONLINE verb +# S7 "premise: the two references really are two different layouts" +# S7 "and it is NOT the zorder layout" -- the DAEMON +# +# Measured on PG17.10, 2026-09-09, against a shim whose two verbs delegate to +# the Z-order verbs and then write sorted_kind='hilbert': exactly those four +# reddened for the curve, and 162 of 181 arms passed. S5 in particular is fully +# green on such a shim BY CONSTRUCTION -- over one column both curves are the +# identity -- so S5 buys the surface and the recorded identity, never the curve, +# and must not be counted as evidence of Hilbertness. +# +# THIS SUITE IS RED ON PURPOSE UNTIL #889's SQL HALF LANDS. Neither +# cluster_hilbert nor recluster_hilbert exists yet, and the shape of the red is +# worth stating exactly, because "every arm that names one fails with 42883" is +# what a reader would otherwise assume and it is not what happens. Measured on +# PG17.10, 2026-09-09: 98 passed + 74 failed + 9 unrunnable = 181, and 27 of the +# 74 reds carry 42883. Every one of those 27 names a missing verb. THE OTHER 47 +# ARE DOWNSTREAM, and they accuse code that ships today: a fixture that could +# not be clustered makes vacuum_sorted, recluster and the daemon print exactly +# what a real defect in them would print. Read the 42883 arms first, and treat +# every other red as fixture drift until the SQL half lands. +# +# IT IS DELIBERATELY NOT REGISTERED in test/run_all_versions.sh: a red suite must +# not enter the matrix. THAT HAS A PRICE, and it is stated here so it is not +# rediscovered: harness_selftest.sh sweeps test/*.sh and asserts every suite is +# registered, so this file makes harness_selftest fail its registration arm +# (measured: 260 passed + 1 failed with the file present, 261 + 0 without it). +# REGISTERING THIS SUITE IN run_all_versions.sh IS PART OF THE PR THAT LANDS +# #889's SQL HALF, in the same commit that turns the suite green. +# +# Usage: test/hilbert_cluster.sh [PG_CONFIG] +# Written fresh for pgColumnar. + +set -uo pipefail + +# The daemon is left OFF here and enabled only inside S7, by ALTER SYSTEM. A +# daemon running for the whole suite would be free to recluster the gate +# fixtures between the "before" and "after" reads of S4, which reads as a gate +# that did not fire. +# +# All four pgcolumnar.autovacuum GUCs are PGC_SIGHUP (src/columnar_tableam.c: +# 3262, 3272, 3281, 3290), so none of them NEEDS to be here. They are set in the +# config file because this suite wants them stable for the whole run and +# identical for the launcher, its workers and every psql session -- one place to +# read them from, and no window in which a fixture is built under one threshold +# and measured under another. +# +# max_parallel_workers_per_gather=0 is not a tuning choice. Every scanorder() +# digest below is a claim about PHYSICAL layout, and a parallel plan interleaves +# workers' output independently of the layout, so a parallel scan would make the +# digest a claim about scheduling. +export PGC_EXTRA_CONF="pgcolumnar.autovacuum_naptime=2 +pgcolumnar.autovacuum_compact_threshold=0.2 +pgcolumnar.autovacuum_recluster_threshold=0.05 +max_parallel_workers_per_gather=0" + +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" + +# The row GROUP is the stripe, so stripe_row_limit is what decides how many +# groups a fixture has -- chunk_group_row_limit sizes the vector inside one. +# 20,000 rows at 1,000 to a stripe is 20 groups, which is what makes "the layout +# did not move" a statement about something rather than about a single group +# that could not move. Both values are at set_options' floor (stripe >= 1000, +# chunk_group >= 100); a value below it RAISES, and a suite that discards the +# error then measures a fixture built on the defaults. So the group count is +# ASSERTED by every fixture generator below rather than assumed from the call. +SR=1000 +CG=500 + +# ---- the instruments ------------------------------------------------------- + +# WHAT COUNTS AS A MEASUREMENT. The digest helpers deliberately return sentinels +# rather than the empty string, so two failed reads cannot compare equal and pass +# an equality arm (#418). The same sentinels satisfy an INEQUALITY arm outright: +# QUERY_ERROR.7 is not equal to any baseline, so "the layout moved" passes on a +# query that never ran. EMPTY is the same hazard from the other side -- a verb +# that destroys every row produces it, and "moved" is then true and useless. +# Nothing in this suite ever measures a legitimately empty relation. +pgc_measured() { # pgc_measured VALUE -> 0 when VALUE is a real measurement + case "$1" in + '' | QUERY_ERROR.* | EMPTY | NO_LAYOUT) return 1 ;; + esac + return 0 +} + +# changed BEFORE AFTER -> moved | unchanged | UNMEASURED[...] +# differs A B -> different | IDENTICAL | UNMEASURED[...] +# Every inequality arm in this file goes through one of these two. +changed() { + pgc_measured "$1" || { printf 'UNMEASURED[before=%s]\n' "$1"; return; } + pgc_measured "$2" || { printf 'UNMEASURED[after=%s]\n' "$2"; return; } + if [ "$1" != "$2" ]; then echo moved; else echo unchanged; fi +} +differs() { + pgc_measured "$1" || { printf 'UNMEASURED[a=%s]\n' "$1"; return; } + pgc_measured "$2" || { printf 'UNMEASURED[b=%s]\n' "$2"; return; } + if [ "$1" != "$2" ]; then echo different; else echo IDENTICAL; fi +} + +# A falsifiable PHYSICAL signal for "was this rewritten": the per-group +# (stripeid, fileoffset, rowcount) multiset from stats(). This is +# recluster_gate.sh's physlayout(), with one deliberate difference -- a relation +# with no stripes, or a query that could not run, yields NO_LAYOUT rather than +# the empty string, so it cannot read as a layout that moved. +# relfilenode does NOT move on any recluster path (pgcolumnar rewrites inside its +# own storage), so it could never falsify a rewrite. +physlayout() { + local d + d="$(q "SELECT md5(string_agg(stripeid::text||':'||fileoffset::text||':'||rowcount::text, ',' ORDER BY stripeid)) FROM pgcolumnar.stats('$1');")" + printf '%s\n' "${d:-NO_LAYOUT}" +} + +# The recorded kind, from the CATALOG. pgcolumnar.storage carries no GRANT and +# is superuser-only (measured: a plain owner gets "permission denied for table +# storage"). +skind() { q "SELECT coalesce(sorted_kind::text,'') FROM pgcolumnar.storage WHERE storage_id = pgcolumnar.get_storage_id('$1');"; } +# The same fact through the REPORTER, which is the only route a table's own +# owner has. Reading only the catalog tests the catalog and not the reporter -- +# and reading the reporter AS THE SUPERUSER tests neither, because q() is +# hard-wired to -U postgres. So S3 hands three fixtures to a plain role and +# reads them back as that role; qrole() is how. +skindst() { q "SELECT coalesce(sorted_kind::text,'') FROM pgcolumnar.sort_status('$1');"; } +skey() { q "SELECT coalesce(sort_key::text,'') FROM pgcolumnar.sort_status('$1');"; } +appended(){ q "SELECT appended_groups FROM pgcolumnar.sort_status('$1');"; } +groups() { q "SELECT total_groups FROM pgcolumnar.sort_status('$1');"; } + +# A scalar read AS A NAMED ROLE. q() is -U postgres and always will be; this is +# the only way an arm may claim to have measured what a table's OWNER can see. +qrole() { # qrole ROLE SQL + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U "$1" \ + -d "$PGC_DB" -At -c "$2" 2>/dev/null || true +} +skindst_as() { qrole "$1" "SELECT coalesce(sorted_kind::text,'') FROM pgcolumnar.sort_status('$2');"; } + +# The order the scan actually returns rows in. pgc_seq_hash keeps the query's own +# output order (ORDER BY row_number() OVER ()), so this can fail on a reordering; +# pgc_set_hash cannot, by design, which is why both appear in S2. +scanorder() { pgc_seq_hash "SELECT * FROM $1"; } +setof() { pgc_set_hash "SELECT * FROM $1"; } + +# The SQLSTATE a statement raises, as a value, for a named role. +# +# VERBOSITY is set with -v and NOT with -c '\set ...': psql takes a different +# code path for a -c argument beginning with a backslash, which is how a sibling +# suite ended up with deny arms that could never go green. The state is read out +# of the ERROR line rather than from a bare line of its own, because psql +# prefixes it. "noerror" rather than the empty string, so a statement that +# SUCCEEDED can never compare equal to one whose state could not be read. +# +# AND A SENTINEL STATEMENT BEHIND IT. Without one, "no ERROR line was found" is +# also what a probe that never reached the server produces, so `noerror` was +# satisfied by a psql that could not connect -- including the control that +# exists to prove the probe can report success. ON_ERROR_STOP=0 means the +# sentinel still runs after the statement under test raised (measured: the state +# and PGC_PROBE_OK both appear), so its ABSENCE means the session, not the +# statement, is what failed. +sqlstate() { # sqlstate ROLE SQL -> a five-character SQLSTATE, noerror, or PROBE_UNREACHABLE + local role="$1" sql="$2" out st + out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U "$role" \ + -d "$PGC_DB" -At -v VERBOSITY=sqlstate -v ON_ERROR_STOP=0 \ + -c "$sql" -c "SELECT 'PGC_PROBE_OK';" 2>&1)" + case "$out" in + *PGC_PROBE_OK*) ;; + *) echo PROBE_UNREACHABLE; return ;; + esac + st="$(printf '%s\n' "$out" | sed -n 's/^.*ERROR:[[:space:]]*\([0-9A-Z]\{5\}\).*$/\1/p' | head -1)" + if [ -n "$st" ]; then printf '%s\n' "$st"; else echo noerror; fi +} + +# Run a maintenance verb AS AN ASSERTION. psql_run's exit status is checked +# nowhere in this tree and this suite cannot afford that: "the verb left the +# table alone" and "the verb raised" produce identical fixtures. +hrun() { # hrun WHAT SQL + check_text "premise: $1 ran without raising" "$(sqlstate postgres "$2")" "noerror" +} + +# The premise behind every scanorder() comparison below. Without it those arms +# could pass by construction. +pgc_check_ordered_oracle + +# And the digests must be reading the columnar layout, not a parallel plan's +# interleaving of it. +check_text "premise: parallelism is off, so a scan order is a fact about the layout" \ + "$(q 'SHOW max_parallel_workers_per_gather;')" "0" + +# ============================================================================= +# S1 THE SURFACE AND ITS REFUSALS, BY SQLSTATE +# ============================================================================= +# +# Four refusals, for each of the two new verbs, each compared against the state +# the ESTABLISHED verb raises on the identical input and read from the same +# probe. Comparing against the sibling rather than only against a literal is +# what stops the two surfaces drifting apart: a change to cluster()'s refusal +# that is not made to cluster_hilbert()'s reddens here. + +psql_run "CREATE TABLE sur (c1 int, c2 int, c3 int, c4 int, c5 int, + c6 int, c7 int, c8 int, c9 int, txt text) USING pgcolumnar;" +psql_run "INSERT INTO sur SELECT g,g,g,g,g,g,g,g,g,'t'||g FROM generate_series(1,2000) g;" +check_num "premise: the refusal fixture holds rows, so a refusal is not an empty-table artifact" \ + "$(q 'SELECT count(*) FROM sur;')" "2000" + +psql_run "DROP ROLE IF EXISTS h_other;" +psql_run "CREATE ROLE h_other NOSUPERUSER LOGIN;" +psql_run "GRANT USAGE ON SCHEMA pgcolumnar TO h_other;" +# The DROP above is itself an unchecked psql_run, so a leftover h_other from a +# previous run -- one that owned objects, and so could not be dropped -- would +# be reused with whatever grants it had. lib.sh initdbs a fresh cluster per run, +# which is why this has never bitten; asserted rather than trusted. +check_num "premise: h_other owns nothing, so it is this run's role and not a leftover" \ + "$(q "SELECT count(*) FROM pg_class WHERE relowner = 'h_other'::regrole;")" "0" + +# A deny arm is evidence only if the call reached the code that denies it, and a +# login FATAL carries no SQLSTATE at all. The role must be able to open a +# session, or the 42501 arms measure connectivity. +check_text "premise: h_other can open a session, so its refusals are refusals and not login failures" \ + "$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U h_other -d "$PGC_DB" -At -c 'SELECT 1;' 2>&1 | head -1)" \ + "1" +check_text "premise: h_other does not own the table" \ + "$(q "SELECT pg_get_userbyid(relowner) = 'h_other' FROM pg_class WHERE oid = 'sur'::regclass;")" "f" + +# None of the four clustering verbs is REVOKEd from PUBLIC (unlike +# read_projection and friends), so EXECUTE is not the layer that refuses here. +# Asserted rather than assumed: if a future revision adds a REVOKE, the 42501 +# below stops attributing to the C owner check and this premise says so. +check_text "premise: h_other holds EXECUTE on all four verbs, so a 42501 below can only be the owner check" \ + "$(q "SELECT count(*) FILTER (WHERE has_function_privilege('h_other', p.oid, 'EXECUTE')) + || '/' || count(*) + FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace + WHERE n.nspname = 'pgcolumnar' + AND p.proname IN ('cluster','recluster','cluster_hilbert','recluster_hilbert');")" \ + "4/4" + +# The four inputs. Held in variables so the hilbert verb and its sibling are +# given byte-identical arguments; two hand-written call sites are how a "same +# input" comparison stops being one. +ARGS_OK="'sur','c1','c2'" +ARGS_TEXT="'sur','txt'" +ARGS_NINE="'sur','c1','c2','c3','c4','c5','c6','c7','c8','c9'" +# Zero columns is spelled with an explicit empty array. Bare +# pgcolumnar.cluster('sur') does not resolve at all -- measured 42883, not +# 22023 -- so it would test PostgreSQL's function resolution rather than the +# verb's own argument check, and the two new verbs would "agree" with the old +# ones about an error neither of them raised. +ARGS_ZERO="'sur', VARIADIC ARRAY[]::name[]" + +# case | role | args | expected SQLSTATE | what it is +# 42501 aclcheck_error(ACLCHECK_NOT_OWNER) +# 0A000 ERRCODE_FEATURE_NOT_SUPPORTED, an unsupported clustering type +# 22023 ERRCODE_INVALID_PARAMETER_VALUE, too many / no clustering columns +for _case in \ + "non-owner|h_other|$ARGS_OK|42501" \ + "a text clustering column|postgres|$ARGS_TEXT|0A000" \ + "nine clustering columns|postgres|$ARGS_NINE|22023" \ + "zero clustering columns|postgres|$ARGS_ZERO|22023" +do + _what="${_case%%|*}"; _rest="${_case#*|}" + _role="${_rest%%|*}"; _rest="${_rest#*|}" + _args="${_rest%%|*}"; _want="${_rest##*|}" + + for _pair in "cluster|cluster_hilbert" "recluster|recluster_hilbert"; do + _old="${_pair%%|*}"; _new="${_pair##*|}" + _sold="$(sqlstate "$_role" "SELECT pgcolumnar.$_old($_args);")" + _snew="$(sqlstate "$_role" "SELECT pgcolumnar.$_new($_args);")" + check_text "premise: $_old on $_what raises $_want (the probe reads the right state)" \ + "$_sold" "$_want" + check_text "$_new on $_what raises $_want" \ + "$_snew" "$_want" + check_text "and $_new's state on $_what is the one $_old raises on the identical input" \ + "$_snew" "$_sold" + done +done + +# The removal proof for the whole block, in both directions. The probe must be +# able to report success, or every arm above is satisfied by an instrument that +# always finds an error -- AND it must be able to report that it never reached +# the server, or "success" is what an unreachable probe says and the first +# control proves nothing. A NOLOGIN role is refused at connection time and +# carries no SQLSTATE, which is exactly the shape the sentinel exists to catch. +check_text "control: the probe reports noerror when the owner makes a call that works" \ + "$(sqlstate postgres "SELECT pgcolumnar.cluster('sur','c1','c2');")" "noerror" +psql_run "DROP ROLE IF EXISTS h_nologin;" +psql_run "CREATE ROLE h_nologin NOSUPERUSER NOLOGIN;" +check_text "control: and it reports PROBE_UNREACHABLE when it cannot open a session, so noerror means the server answered" \ + "$(sqlstate h_nologin "SELECT 1;")" "PROBE_UNREACHABLE" + +# ============================================================================= +# S2 IT ONLY REORDERS +# ============================================================================= +# +# Against a heap mirror. The set hash is order-blind ON PURPOSE, so it is the +# right instrument for "the rows are the same rows" and the WRONG one for "the +# rows moved". Both halves are here because a parity-only arm passes on a table +# that was never touched. + +psql_run "CREATE TABLE s2h (id int, x int, y int, pad text);" +psql_run "CREATE TABLE s2c (id int, x int, y int, pad text) USING pgcolumnar;" +psql_run "SELECT pgcolumnar.set_options('s2c', stripe_row_limit => 2048, chunk_group_row_limit => 1024);" +psql_run "INSERT INTO s2h SELECT g, ((g::bigint*7919)%200)::int, ((g::bigint*104729)%200)::int, 'p'||(g%97) + FROM generate_series(1,40960) g;" +psql_run "INSERT INTO s2c SELECT * FROM s2h;" + +S2_HEAP="$(pgc_set_hash 'SELECT id, x, y, pad FROM s2h')" +S2_SET_BEFORE="$(pgc_set_hash 'SELECT id, x, y, pad FROM s2c')" +S2_SEQ_BEFORE="$(scanorder s2c)" +S2_SEQ_AGAIN="$(scanorder s2c)" + +check_num "premise: set_options took, so s2c is 20 groups and not one default-sized group" \ + "$(groups s2c)" "20" +check_text "premise: before clustering, the columnar table already matches its heap mirror as a SET" \ + "$S2_SET_BEFORE" "$S2_HEAP" +check_text "premise: the order-sensitive digest is STABLE across two reads, so a later change is a change" \ + "$S2_SEQ_AGAIN" "$S2_SEQ_BEFORE" +check "premise: the plan being digested is the columnar custom scan, not a fallback" \ + "$(pgc_is_columnar_scan 'SELECT * FROM s2c')" "yes" + +hrun "cluster_hilbert('s2c','x','y')" "SELECT pgcolumnar.cluster_hilbert('s2c', 'x', 'y');" + +check_text "cluster_hilbert preserves the row SET exactly (unchanged from before)" \ + "$(pgc_set_hash 'SELECT id, x, y, pad FROM s2c')" "$S2_SET_BEFORE" +check_text "and the row set still equals the heap mirror's" \ + "$(pgc_set_hash 'SELECT id, x, y, pad FROM s2c')" "$S2_HEAP" +check "cluster_hilbert MOVED the rows: the order-sensitive digest changed" \ + "$(changed "$S2_SEQ_BEFORE" "$(scanorder s2c)")" "moved" +check_num "and no row was lost or gained" "$(q 'SELECT count(*) FROM s2c;')" "40960" + +# THE INSTRUMENT'S LIMIT, PINNED ON A VERB THAT ALREADY EXISTS. +# +# physlayout() is recluster_gate.sh's digest and it is the right corroboration +# for the ONLINE gate in S4. It cannot see an EAGER rewrite: the eager path +# writes a fresh file and reproduces the same (stripeid, fileoffset, rowcount) +# multiset, so the digest is byte-identical either side of a rewrite that +# reordered every row. Measured on plain cluster() below rather than asserted +# about cluster_hilbert(), so the control stands on code that ships today and +# cannot be perturbed by whatever #889 lands. +# +# WHAT THIS SECTION NO LONGER ASSERTS, AND WHY. It used to also require s2c's +# OWN layout digest to be unchanged across cluster_hilbert. That is not a +# property of the eager path; file_offset is the compressed size of the +# preceding groups, so reproducing the geometry is a data-dependent accident of +# this fixture. A correct Hilbert implementation that packs groups differently +# would redden it for no defect. The control below is the honest form of the +# same fact, because it is measured on a verb #889 cannot change. +psql_run "CREATE TABLE s2ctl (id int, x int, y int, pad text) USING pgcolumnar;" +psql_run "SELECT pgcolumnar.set_options('s2ctl', stripe_row_limit => 2048, chunk_group_row_limit => 1024);" +psql_run "INSERT INTO s2ctl SELECT * FROM s2h;" +S2CTL_PHYS="$(physlayout s2ctl)" +S2CTL_SEQ="$(scanorder s2ctl)" +hrun "cluster('s2ctl','x','y')" "SELECT pgcolumnar.cluster('s2ctl', 'x', 'y');" +check "control: an eager cluster() DOES reorder the rows" \ + "$(changed "$S2CTL_SEQ" "$(scanorder s2ctl)")" "moved" +check "control: and physlayout cannot see it, which is why the order digest is S2's instrument" \ + "$(changed "$S2CTL_PHYS" "$(physlayout s2ctl)")" "unchanged" + +# ============================================================================= +# S3 THE KIND IS RECORDED, AND IT IS DISTINGUISHABLE +# ============================================================================= +# +# Three tables from ONE fixture generator, so the only difference between them +# is the verb that rewrote them. Read BOTH ways: pgcolumnar.storage is the +# catalog and is superuser-only, pgcolumnar.sort_status is the reporter and is +# the only route a table's own owner has. Reading one tests one -- and reading +# both AS THE SUPERUSER tests one, which is why the three fixtures are handed to +# a plain role and the reporter is read back as that role. + +mk3() { + psql_run "CREATE TABLE $1 (id int, k int, j int) USING pgcolumnar;" + psql_run "SELECT pgcolumnar.set_options('$1', stripe_row_limit => $SR, chunk_group_row_limit => $CG);" + psql_run "INSERT INTO $1 SELECT g, (g*7919)%5000, g%13 FROM generate_series(1,5000) g;" + check_num "premise: set_options took on $1, so it is 5 groups and not one default-sized group" \ + "$(groups "$1")" "5" +} + +psql_run "DROP ROLE IF EXISTS h_owner;" +psql_run "CREATE ROLE h_owner NOSUPERUSER LOGIN;" +psql_run "GRANT USAGE ON SCHEMA pgcolumnar TO h_owner;" +check_text "premise: h_owner can open a session, so a read as h_owner is a read" \ + "$(qrole h_owner 'SELECT 1;')" "1" +check_text "premise: h_owner is NOT a superuser, or reading 'as the owner' reads as the superuser again" \ + "$(q "SELECT rolsuper::text FROM pg_roles WHERE rolname = 'h_owner';")" "false" +check_text "premise: and the catalog really is closed to it, so sort_status is the only route it has" \ + "$(sqlstate h_owner 'SELECT count(*) FROM pgcolumnar.storage;')" "42501" + +mk3 s3lex; S3LEX_SEQ0="$(scanorder s3lex)"; hrun "vacuum_sorted('s3lex','k')" "SELECT pgcolumnar.vacuum_sorted('s3lex', 'k');" +mk3 s3zo; S3ZO_SEQ0="$(scanorder s3zo)"; hrun "cluster('s3zo','k','j')" "SELECT pgcolumnar.cluster('s3zo', 'k', 'j');" +mk3 s3hi; S3HI_SEQ0="$(scanorder s3hi)"; hrun "cluster_hilbert('s3hi','k','j')" "SELECT pgcolumnar.cluster_hilbert('s3hi', 'k', 'j');" + +# THE PREMISE THE PHYSICAL-ORDER ARMS BELOW STAND ON, AND THE GATE ON THEM. +# "s3hi's order differs from s3zo's" is also true of an s3hi nobody rewrote: it +# is still in insert order, which differs from both of the tables that WERE +# rewritten -- both arms printed PASS, green, on a tree with no cluster_hilbert +# in it (measured). Each leg is anchored against its OWN pre-clustering +# baseline, and the comparison is recorded UNRUN unless both of its legs moved. +S3LEX_MOVED="$(changed "$S3LEX_SEQ0" "$(scanorder s3lex)")" +S3ZO_MOVED="$(changed "$S3ZO_SEQ0" "$(scanorder s3zo)")" +S3HI_MOVED="$(changed "$S3HI_SEQ0" "$(scanorder s3hi)")" +check "premise: vacuum_sorted moved s3lex from its own baseline" "$S3LEX_MOVED" "moved" +check "premise: cluster moved s3zo from its own baseline" "$S3ZO_MOVED" "moved" +check "premise: cluster_hilbert moved s3hi from its own baseline" "$S3HI_MOVED" "moved" +check "premise: the plan being digested here is the columnar custom scan too" \ + "$(pgc_is_columnar_scan 'SELECT * FROM s3hi')" "yes" + +psql_run "ALTER TABLE s3hi OWNER TO h_owner;" +psql_run "ALTER TABLE s3lex OWNER TO h_owner;" +psql_run "ALTER TABLE s3zo OWNER TO h_owner;" +check_text "premise: the three fixtures are now owned by h_owner, so the reads below are the owner's" \ + "$(q "SELECT count(*) FROM pg_class WHERE relowner = 'h_owner'::regrole AND relname IN ('s3hi','s3lex','s3zo');")" "3" + +check_text "cluster_hilbert records the kind in pgcolumnar.storage" "$(skind s3hi)" "hilbert" +check_text "and pgcolumnar.sort_status reports that same kind to the table's own (non-superuser) owner" \ + "$(skindst_as h_owner s3hi)" "hilbert" +check_text "and it records the key it applied" "$(skey s3hi)" "{k,j}" + +# The discrimination this buys. Three tables built from one fixture must be told +# apart by sort_status ALONE, without reading the superuser-only catalog. +# +# ASSERTED AS A NAMED SET, NOT PAIRWISE. Three pairwise `!=` arms are each +# satisfied by a NULL on one side -- all three passed, green, on a tree where +# cluster_hilbert did not exist and s3hi's kind was (measured). They can +# say which pair collapsed; they cannot say the three kinds are the three kinds. +# This one can, and it is strictly stronger than all three of them together. +check_text "the owner alone can tell the three apart: the kinds ARE hilbert, lexicographic and zorder, with no NULL standing in for one" \ + "$(printf '%s\n' "$(skindst_as h_owner s3hi)" "$(skindst_as h_owner s3lex)" "$(skindst_as h_owner s3zo)" | sort | tr '\n' '|')" \ + "hilbert|lexicographic|zorder|" + +# And the three layouts really are three layouts, not one relabelled three +# times. Without this, "distinguishable" could be true of a catalog column that +# nothing physical stands behind -- which is exactly what a relabelled Z-order +# implementation is. THIS IS ONE OF THE THREE ARMS IN THE FILE THAT REFUSE ONE. +if [ "$S3HI_MOVED" = "moved" ] && [ "$S3ZO_MOVED" = "moved" ]; then + check "the three kinds stand for three different physical orders (hilbert vs zorder)" \ + "$(differs "$(scanorder s3hi)" "$(scanorder s3zo)")" "different" +else + check_unrunnable "the three kinds stand for three different physical orders (hilbert vs zorder)" \ + UNMET_PRECONDITION \ + "a leg was never rewritten (s3hi=[$S3HI_MOVED], s3zo=[$S3ZO_MOVED]), so 'a different order' would only be insert order" +fi +if [ "$S3HI_MOVED" = "moved" ] && [ "$S3LEX_MOVED" = "moved" ]; then + check "the three kinds stand for three different physical orders (hilbert vs lexicographic)" \ + "$(differs "$(scanorder s3hi)" "$(scanorder s3lex)")" "different" +else + check_unrunnable "the three kinds stand for three different physical orders (hilbert vs lexicographic)" \ + UNMET_PRECONDITION \ + "a leg was never rewritten (s3hi=[$S3HI_MOVED], s3lex=[$S3LEX_MOVED]), so 'a different order' would only be insert order" +fi + +# ============================================================================= +# S4 THE SELF-GATE, IN EVERY DIRECTION, CORROBORATED BY A POSITIVE CONTROL +# ============================================================================= +# +# There are FIVE directions, not four, and the fifth is the one the daemon +# depends on: same kind, same key, WITH AN APPENDED TAIL must NOT gate. A gate +# keyed on kind and key alone satisfies every other arm here -- (a) 0, (b) >0, +# (c) 0, (d) >0 -- and breaks the daemon outright. Measured against a shim whose +# gate ignored the tail: NO ARM IN S4 COULD SEE IT, and the only arm that +# reddened for it was a FIXTURE PREMISE in S7 ("the hand-driven hilbert +# reference folded its tail back in: got [5] want [0]"). A defect in the gate +# must redden a gate arm, not read as a bad fixture. +# So (a) does not stop at "it returned 0": it appends a tail to the very table +# it just gated, requires the SAME call to do work, and then requires the gate +# to close again. +# +# physlayout is carried beside every return value, but only where it can +# corroborate: an unchanged layout beside a 0 return is the same observation +# twice, so on the no-op side it is asserted only AFTER the return value has +# been confirmed to be the number 0, and recorded UNRUN otherwise. + +mk4() { + psql_run "CREATE TABLE $1 (a int, b int, pad text) USING pgcolumnar;" + psql_run "SELECT pgcolumnar.set_options('$1', stripe_row_limit => $SR, chunk_group_row_limit => $CG);" + psql_run "INSERT INTO $1 SELECT (g*2654435761)::bigint % 1000, g, md5(g::text) + FROM generate_series(1,20000) g;" + check_num "premise: set_options took on $1, so it is 20 groups and 'nothing changed' cannot be true because there was nothing there" \ + "$(groups "$1")" "20" +} +decay4() { # the 25% appended tail every decay fixture gets + psql_run "INSERT INTO $1 SELECT (g*2654435761)::bigint % 1000, g, md5(g::text) + FROM generate_series(1,5000) g;" +} + +# ---- (a) recluster_hilbert on an already-hilbert table, same key ------------ +mk4 s4a +hrun "cluster_hilbert('s4a','a','b')" "SELECT pgcolumnar.cluster_hilbert('s4a','a','b');" +check_num "premise: the eager rewrite left no appended tail on s4a" "$(appended s4a)" "0" +S4A_PHYS="$(physlayout s4a)" +S4A_RET="$(q "SELECT pgcolumnar.recluster_hilbert('s4a','a','b');")" +check_num "(a) recluster_hilbert on an already-hilbert table with the same key reclusters 0 groups" \ + "$S4A_RET" "0" +if [ "$S4A_RET" = "0" ]; then + check_text "(a) and the physical layout is byte-identical" "$(physlayout s4a)" "$S4A_PHYS" +else + check_unrunnable "(a) and the physical layout is byte-identical" UNMET_PRECONDITION \ + "the call did not return 0 (returned [$S4A_RET]), so an unchanged layout would not be about the gate" +fi +check_text "(a) and the kind is untouched" "$(skind s4a)" "hilbert" + +# THE POSITIVE CONTROL, ON THE SAME TABLE AND THROUGH THE SAME CALL. Without it, +# "it returned 0 and nothing moved" is equally true of a verb that does nothing +# at all -- a stub defined as `BEGIN RETURN 0; END` passed all three arms above. +# It is also the fifth direction: a Hilbert table with an appended tail is +# exactly what the daemon hands this verb, and a gate that skips it is a gate +# that disables the daemon. +decay4 s4a +S4A_SET="$(setof s4a)" +S4A_PHYS2="$(physlayout s4a)" +check "premise: s4a now carries an appended tail for the gate to let through" \ + "$([ "$(appended s4a)" -gt 0 ] 2>/dev/null && echo decayed || echo clean)" "decayed" +S4A_RET2="$(q "SELECT pgcolumnar.recluster_hilbert('s4a','a','b');")" +check "(a2) THE SAME CALL ON THE SAME TABLE does work once a tail is appended, so the 0 above was a gate and not a dead verb (>0)" \ + "$([ "$S4A_RET2" -gt 0 ] 2>/dev/null && echo yes || echo no)" "yes" +check "(a2) and that rewrite moved the layout" \ + "$(changed "$S4A_PHYS2" "$(physlayout s4a)")" "moved" +check_num "(a2) and it folded the appended tail back in" "$(appended s4a)" "0" +check_text "(a2) and it is still a hilbert table on the same key" "$(skind s4a)/$(skey s4a)" "hilbert/{a,b}" +check_num "(a2) and no row was lost" "$(q 'SELECT count(*) FROM s4a;')" "25000" +check_text "(a2) and the rows are the same rows, only reordered" "$(setof s4a)" "$S4A_SET" + +S4A_PHYS3="$(physlayout s4a)" +S4A_RET3="$(q "SELECT pgcolumnar.recluster_hilbert('s4a','a','b');")" +check_num "(a3) and the gate closes again on the refolded table" "$S4A_RET3" "0" +if [ "$S4A_RET3" = "0" ]; then + check_text "(a3) and the layout is byte-identical again" "$(physlayout s4a)" "$S4A_PHYS3" +else + check_unrunnable "(a3) and the layout is byte-identical again" UNMET_PRECONDITION \ + "the call did not return 0 (returned [$S4A_RET3]), so an unchanged layout would not be about the gate" +fi + +# ---- (b) recluster_hilbert on a 'zorder' table over the same columns -------- +mk4 s4b +hrun "cluster('s4b','a','b')" "SELECT pgcolumnar.cluster('s4b','a','b');" +check_text "premise: s4b is a zorder table over exactly (a,b)" "$(skind s4b)/$(skey s4b)" "zorder/{a,b}" +S4B_PHYS="$(physlayout s4b)" +S4B_SET="$(setof s4b)" +check "(b) recluster_hilbert on a zorder table over the same columns does real work (>0)" \ + "$([ "$(q "SELECT pgcolumnar.recluster_hilbert('s4b','a','b');")" -gt 0 ] 2>/dev/null && echo yes || echo no)" "yes" +check "(b) and the physical layout moved" \ + "$(changed "$S4B_PHYS" "$(physlayout s4b)")" "moved" +# THE LABEL IS ASSERTED WITH THE BYTES. On its own this arm was green in a run +# where its own two siblings were red -- a shim that wrote 'hilbert' into the +# catalog and rewrote nothing at all satisfied it (measured). +check_text "(b) and the recorded kind became hilbert IN THE SAME CALL THAT MOVED THE LAYOUT" \ + "$(skind s4b)/$(changed "$S4B_PHYS" "$(physlayout s4b)")" "hilbert/moved" +check_num "(b) and no row was lost" "$(q 'SELECT count(*) FROM s4b;')" "20000" +check_text "(b) and the rows are the same rows, only reordered" "$(setof s4b)" "$S4B_SET" + +# ---- (c) plain recluster on a HILBERT table with a matching key ------------- +# THE RULING. The curve is sticky, so this is a no-op and NOT a conversion back +# to Z-order. Today's gate compares strcmp(skind, "zorder") == 0 +# (src/columnar_vacuum.c:646), so it falls through and rewrites. +mk4 s4c +hrun "cluster_hilbert('s4c','a','b')" "SELECT pgcolumnar.cluster_hilbert('s4c','a','b');" +S4C_PHYS="$(physlayout s4c)" +S4C_RET="$(q "SELECT pgcolumnar.recluster('s4c','a','b');")" +check_num "(c) plain recluster on a hilbert table with a matching key reclusters 0 groups" \ + "$S4C_RET" "0" +if [ "$S4C_RET" = "0" ]; then + check_text "(c) and the physical layout is byte-identical: it did not convert the table" \ + "$(physlayout s4c)" "$S4C_PHYS" +else + check_unrunnable "(c) and the physical layout is byte-identical: it did not convert the table" \ + UNMET_PRECONDITION \ + "the call did not return 0 (returned [$S4C_RET]), so an unchanged layout would not be about the gate" +fi +check_text "(c) and the table is still hilbert, not relabelled zorder" "$(skind s4c)" "hilbert" +check_num "(c) and no row was lost" "$(q 'SELECT count(*) FROM s4c;')" "20000" + +# ---- (d) a DIFFERENT key rewrites in both directions ------------------------ +# The removal proof for (a) and (c): a gate that never looks at anything +# satisfies both. It must still DISCRIMINATE. +# +# The two twins are built and laid on the curve FIRST, and asserted identical, +# so that the last arm in this block is a comparison of two curves over one key +# on one dataset rather than a comparison of two histories. THAT ARM IS THE +# ONLINE VERB'S ONLY CURVE DEFENCE IN THIS FILE. +mk4 s4d1 +hrun "cluster_hilbert('s4d1','a','b')" "SELECT pgcolumnar.cluster_hilbert('s4d1','a','b');" +mk4 s4d2 +hrun "cluster_hilbert('s4d2','a','b')" "SELECT pgcolumnar.cluster_hilbert('s4d2','a','b');" +S4D1_PHYS="$(physlayout s4d1)" +S4D2_PHYS="$(physlayout s4d2)" +check_text "premise: the two (d) twins are byte-identical before either is reclustered on the new key" \ + "$(scanorder s4d1)" "$(scanorder s4d2)" + +S4D1_RET="$(q "SELECT pgcolumnar.recluster_hilbert('s4d1','b','a');")" +S4D1_MOVED="$([ "$S4D1_RET" -gt 0 ] 2>/dev/null && echo yes || echo no)" +check "(d) recluster_hilbert over DIFFERENT columns still rewrites (>0)" "$S4D1_MOVED" "yes" +check "(d) and that rewrite moved the layout" \ + "$(changed "$S4D1_PHYS" "$(physlayout s4d1)")" "moved" +check_text "(d) and it is still a hilbert table, now on the new key" "$(skind s4d1)/$(skey s4d1)" "hilbert/{b,a}" +check_num "(d) and no row was lost" "$(q 'SELECT count(*) FROM s4d1;')" "20000" + +S4D2_RET="$(q "SELECT pgcolumnar.recluster('s4d2','b','a');")" +S4D2_MOVED="$([ "$S4D2_RET" -gt 0 ] 2>/dev/null && echo yes || echo no)" +check "(d) plain recluster over DIFFERENT columns rewrites a hilbert table (>0)" "$S4D2_MOVED" "yes" +check "(d) and that rewrite moved the layout" \ + "$(changed "$S4D2_PHYS" "$(physlayout s4d2)")" "moved" +check_text "(d) and naming the plain verb with a NEW key is the explicit switch back to zorder" \ + "$(skind s4d2)/$(skey s4d2)" "zorder/{b,a}" +check_num "(d) and no row was lost" "$(q 'SELECT count(*) FROM s4d2;')" "20000" + +# THE ONLINE VERB'S CURVE DEFENCE, and it is gated on both rewrites having +# happened. Untouched, s4d1 is still on the (a,b) layout while s4d2 is on the +# (b,a) one, so "they differ" is satisfied by recluster_hilbert doing nothing -- +# which is what it does today, and the arm printed PASS on it (measured). +if [ "$S4D1_MOVED" = "yes" ] && [ "$S4D2_MOVED" = "yes" ]; then + check "(d) and the HILBERT rewrite over (b,a) is NOT the ZORDER rewrite over (b,a) on the identical data" \ + "$(differs "$(scanorder s4d1)" "$(scanorder s4d2)")" "different" +else + check_unrunnable "(d) and the HILBERT rewrite over (b,a) is NOT the ZORDER rewrite over (b,a) on the identical data" \ + UNMET_PRECONDITION \ + "a rewrite did not happen (s4d1=[$S4D1_RET], s4d2=[$S4D2_RET]), so the two layouts are not the two curves" +fi + +# ============================================================================= +# S5 ncols == 1 IS THE IDENTITY, THROUGH SQL +# ============================================================================= +# +# Over one column the Hilbert index and the Morton index are both the identity, +# so the two verbs must produce the SAME physical order while recording +# DIFFERENT kinds. Each side is also compared against its OWN pre-cluster +# baseline: two calls that both no-opped would compare equal to each other and +# the arm would pass on nothing. +# +# THIS SECTION BUYS SURFACE AND IDENTITY, NEVER THE CURVE. Over one column a +# relabelled Z-order implementation is INDISTINGUISHABLE from a correct one -- +# all of S5 is green on one, by construction (measured). Do not read a green S5 +# as evidence of Hilbertness; S3, S4(d) and S7 carry that. + +mk5() { + psql_run "CREATE TABLE $1 (a int, b int, pad text) USING pgcolumnar;" + psql_run "SELECT pgcolumnar.set_options('$1', stripe_row_limit => $SR, chunk_group_row_limit => $CG);" + psql_run "INSERT INTO $1 SELECT (g*2654435761)::bigint % 1000, g, md5(g::text) + FROM generate_series(1,20000) g;" + check_num "premise: set_options took on $1, so it is 20 groups" "$(groups "$1")" "20" +} +mk5 s5hi +mk5 s5zo +S5HI_BEFORE="$(scanorder s5hi)" +S5ZO_BEFORE="$(scanorder s5zo)" +S5HI_SET="$(setof s5hi)" +S5ZO_SET="$(setof s5zo)" +check_text "premise: the two single-column fixtures start identical" "$S5HI_BEFORE" "$S5ZO_BEFORE" +check "premise: the plan being digested here is the columnar custom scan too" \ + "$(pgc_is_columnar_scan 'SELECT * FROM s5hi')" "yes" + +hrun "cluster_hilbert('s5hi','a')" "SELECT pgcolumnar.cluster_hilbert('s5hi','a');" +hrun "cluster('s5zo','a')" "SELECT pgcolumnar.cluster('s5zo','a');" + +check "premise: cluster_hilbert changed s5hi's order from its own baseline" \ + "$(changed "$S5HI_BEFORE" "$(scanorder s5hi)")" "moved" +check "premise: cluster changed s5zo's order from its own baseline" \ + "$(changed "$S5ZO_BEFORE" "$(scanorder s5zo)")" "moved" +# AND BOTH STILL HOLD THEIR ROWS. Two verbs that record their kind and then +# EMPTY the table satisfy both premises above -- the emptied digest differs from +# every baseline, so both read "moved", and then they compare equal to each +# other and the identity arm passes too. +check_num "premise: and s5hi still holds every row it started with" "$(q 'SELECT count(*) FROM s5hi;')" "20000" +check_num "premise: and s5zo still holds every row it started with" "$(q 'SELECT count(*) FROM s5zo;')" "20000" +check_text "premise: and s5hi's rows are the same rows, only reordered" "$(setof s5hi)" "$S5HI_SET" +check_text "premise: and s5zo's rows are the same rows, only reordered" "$(setof s5zo)" "$S5ZO_SET" + +check_text "over ONE column the two curves are the identity: the physical order is the same" \ + "$(scanorder s5hi)" "$(scanorder s5zo)" +# The bare inequality arm that used to sit here is gone: != 'zorder' +# passed it on a tree with no cluster_hilbert at all. The named pair below is +# strictly stronger and cannot be satisfied by a kind that was never written. +check_text "and each records its own verb's kind" "$(skind s5hi)/$(skind s5zo)" "hilbert/zorder" + +# ============================================================================= +# S6 vacuum_sorted MUST NOT CLOBBER A HILBERT TABLE +# ============================================================================= +# +# See the header: the pinned value here is a READING of the ruling, and this is +# the arm to change if the owner meant an honest relabel rather than a no-op. +# +# THE CALL IS ASSERTED, NOT MADE. A vacuum_sorted that RAISES on every Hilbert +# table leaves the order and the kind untouched, which is byte-identical to the +# no-op this section pins -- all of S6 was green on exactly that shim (measured, +# 2026-09-09). hrun's SQLSTATE arm is what tells the two apart. + +mk4 s6t +hrun "cluster_hilbert('s6t','a','b')" "SELECT pgcolumnar.cluster_hilbert('s6t','a','b');" +check_text "premise: s6t is a hilbert table before vacuum_sorted touches it" "$(skind s6t)" "hilbert" +check "premise: the plan being digested here is the columnar custom scan too" \ + "$(pgc_is_columnar_scan 'SELECT * FROM s6t')" "yes" +# THE ORDER DIGEST IS THE INSTRUMENT HERE, NOT physlayout. vacuum_sorted is an +# EAGER verb, and an eager rewrite reproduces the stripe geometry exactly, so +# the layout digest reads "unchanged" whether the table was clobbered or left +# alone -- a check that cannot fail. S2's control measures that directly. +S6_SEQ="$(scanorder s6t)" +hrun "vacuum_sorted('s6t','a') on a hilbert table" "SELECT pgcolumnar.vacuum_sorted('s6t','a');" +check_text "vacuum_sorted leaves a hilbert table's physical ORDER identical: it did not re-sort it" \ + "$(scanorder s6t)" "$S6_SEQ" +check_text "and leaves the recorded kind hilbert, not relabelled lexicographic" "$(skind s6t)" "hilbert" +check_text "and the reporter agrees with the catalog, so the two do not disagree about it" "$(skindst s6t)" "hilbert" +check_num "and no row was lost" "$(q 'SELECT count(*) FROM s6t;')" "20000" + +# The removal proof. Without this the arms above are satisfied by a +# vacuum_sorted that no-ops on EVERYTHING, which is a worse defect than the one +# they guard. +mk4 s6c +S6C_SEQ="$(scanorder s6c)" +hrun "vacuum_sorted('s6c','a') on a table with no recorded kind" "SELECT pgcolumnar.vacuum_sorted('s6c','a');" +check "control: vacuum_sorted still REORDERS a table with no recorded kind" \ + "$(changed "$S6C_SEQ" "$(scanorder s6c)")" "moved" +check_num "control: and the rows really are ascending on a afterwards" \ + "$(q "SELECT count(*) FROM (SELECT a < lag(a) OVER () AS d FROM s6c) z WHERE d;")" "0" +check_text "control: and it still records its own kind there" "$(skind s6c)" "lexicographic" + +# ============================================================================= +# S7 THE DAEMON PRESERVES THE CURVE +# ============================================================================= +# +# This is the arm the ruling exists for, so it DRIVES THE DAEMON. The daemon +# builds its own call at src/columnar_autovacuum.c:283-291 and hard-codes +# pgcolumnar.recluster; calling recluster() from the suite and reasoning about +# it would measure the neighbouring question. +# +# Three twins from one fixture, with identical appended decay: +# +# av_hi hilbert, left decayed -- the DAEMON's subject +# av_ref hilbert, reclustered BY HAND with recluster_hilbert -- what a +# Hilbert rewrite of this data produces +# av_zo zorder, reclustered BY HAND with recluster -- the control that +# makes "the layout is a Hilbert layout" a discrimination rather +# than a restatement of "something ran" +# +# The two hand-driven twins are reclustered BEFORE the daemon is enabled, so +# they have no appended tail left and the daemon has no reason to touch them. +# THAT IS A PREMISE, NOT A GUARANTEE: when recluster_hilbert was a stub, av_ref +# kept its tail, stayed due, and the daemon reclustered IT as well -- the two +# converged on the same Z-order layout and "av_hi matches the hilbert twin" +# passed on it (measured). So both references are FROZEN across the daemon +# window and asserted unchanged afterwards. +# +# AND THE DAEMON HAS TWO DISPATCHES. compact_rewrite would also fold the tail, +# and it records 'zorder' (src/columnar_vacuum.c:1576), so "the tail folded" +# alone cannot name which one ran. compact_rewrite_due is asserted false, and +# the daemon's own log line is read for the dispatch that did the work. + +check_num "premise: the maintenance launcher is running" \ + "$(q "SELECT count(*) FROM pg_stat_activity WHERE backend_type = 'pgcolumnar autovacuum launcher';")" "1" +check_text "premise: the daemon is OFF while the fixtures are built" "$(q "SHOW pgcolumnar.autovacuum;")" "off" + +mk7() { + psql_run "CREATE TABLE $1 (a int, b int, pad text) USING pgcolumnar;" + psql_run "SELECT pgcolumnar.set_options('$1', stripe_row_limit => $SR, chunk_group_row_limit => $CG);" + psql_run "INSERT INTO $1 SELECT (g*2654435761)::bigint % 1000, g, md5(g::text) + FROM generate_series(1,20000) g;" + check_num "premise: set_options took on $1, so it is 20 groups" "$(groups "$1")" "20" +} + +mk7 av_hi; hrun "cluster_hilbert('av_hi','a','b')" "SELECT pgcolumnar.cluster_hilbert('av_hi','a','b');" +mk7 av_ref; hrun "cluster_hilbert('av_ref','a','b')" "SELECT pgcolumnar.cluster_hilbert('av_ref','a','b');" +mk7 av_zo; hrun "cluster('av_zo','a','b')" "SELECT pgcolumnar.cluster('av_zo','a','b');" +decay4 av_hi; decay4 av_ref; decay4 av_zo + +check_text "premise: av_hi is a hilbert table with appended decay for the daemon to find" \ + "$(skind av_hi)/$([ "$(appended av_hi)" -gt 0 ] 2>/dev/null && echo decayed || echo clean)" \ + "hilbert/decayed" +check_text "premise: and the key the daemon will read off it is the one it was clustered by" \ + "$(skey av_hi)" "{a,b}" +# BOTH halves of the daemon's dispatch, so the recluster is its only reason to +# touch av_hi. Without the compaction half, a compact_rewrite that folded the +# tail and relabelled the table zorder would pass the arm below and redden THE +# RULING, and the reader would attribute both to the recluster path. +check_text "premise: the daemon agrees a recluster is due, and that a compaction is NOT" \ + "$(q "SELECT recluster_due::text || '/' || compact_rewrite_due::text FROM pgcolumnar.maintenance_due('av_hi');")" \ + "true/false" +check "premise: the plan being digested here is the columnar custom scan too" \ + "$(pgc_is_columnar_scan 'SELECT * FROM av_hi')" "yes" + +# The two references, driven by hand while the daemon is still off. +hrun "recluster_hilbert('av_ref','a','b')" "SELECT pgcolumnar.recluster_hilbert('av_ref','a','b');" +hrun "recluster('av_zo','a','b')" "SELECT pgcolumnar.recluster('av_zo','a','b');" +REF_FOLDED="$(appended av_ref)" +ZO_FOLDED="$(appended av_zo)" +check_num "premise: the hand-driven hilbert reference folded its tail back in" "$REF_FOLDED" "0" +check_num "premise: the hand-driven zorder control folded its tail back in" "$ZO_FOLDED" "0" +# Gated for the same reason S3's and S4(d)'s comparisons are: an av_ref that was +# never reclustered still carries its tail, so it differs from av_zo for a +# reason that has nothing to do with the curve, and the premise passes on it. +if [ "$REF_FOLDED" = "0" ] && [ "$ZO_FOLDED" = "0" ]; then + check "premise: the two references really are two different layouts, so the arm below discriminates" \ + "$(differs "$(scanorder av_ref)" "$(scanorder av_zo)")" "different" +else + check_unrunnable "premise: the two references really are two different layouts, so the arm below discriminates" \ + UNMET_PRECONDITION \ + "a hand-driven reference did not fold its tail (av_ref=[$REF_FOLDED], av_zo=[$ZO_FOLDED]), so the two layouts are not the two rewrites" +fi +# AND av_hi MUST NOT ALREADY EQUAL av_ref. Without this the "the daemon produced +# a Hilbert layout" arm below is satisfied by a daemon that did nothing at all: +# two twins nobody touched compare equal, and the arm passes on nothing. av_hi +# still carries its decayed tail here, so it must differ. THE ARM BELOW IS +# GATED ON THIS RESULT rather than merely preceded by it -- a premise that reds +# while the arm it guards still prints PASS leaves the flagship green on a tree +# with no Hilbert code in it, which is exactly what happened. +S7_PRE="$(differs "$(scanorder av_hi)" "$(scanorder av_ref)")" +check "premise: av_hi does NOT yet match the hilbert reference, so matching it later is a change" \ + "$S7_PRE" "different" + +# THE REFERENCES ARE FROZEN HERE. Everything below compares against these +# values, not against a fresh read, so a daemon that rewrites a reference during +# its window cannot make the comparison true by moving the target. +REF_SEQ="$(scanorder av_ref)" +REF_APP="$(appended av_ref)" +ZO_SEQ="$(scanorder av_zo)" +AV_HI_SET="$(setof av_hi)" + +psql_run "ALTER SYSTEM SET pgcolumnar.autovacuum = on;" +q "SELECT pg_reload_conf();" >/dev/null +check_text "the daemon is now ON" "$(q "SHOW pgcolumnar.autovacuum;")" "on" + +AV_AFTER="$(appended av_hi)" +for _ in $(seq 1 15); do + sleep 2 + AV_AFTER="$(appended av_hi)" + [ "${AV_AFTER:-1}" = "0" ] && break +done +check_num "the daemon reclustered av_hi (its appended tail folded to 0)" "$AV_AFTER" "0" + +# WHO DID IT, read from the daemon's own log rather than inferred from a +# counter. src/columnar_autovacuum.c:291 emits the recluster line at LOG and +# :275 the compact_rewrite line; both land in this suite's own server log. +AV_RECLOG="$(grep -c 'pgcolumnar autovacuum: recluster public\.av_hi ' "$PGC_LOGFILE")" +AV_CMPLOG="$(grep -c 'pgcolumnar autovacuum: compact_rewrite public\.av_hi' "$PGC_LOGFILE")" +AV_REFLOG="$(grep -c 'pgcolumnar autovacuum: \(recluster\|compact_rewrite\) public\.av_\(ref\|zo\)' "$PGC_LOGFILE")" +check "and the daemon's own log names the dispatch that did it: recluster on av_hi" \ + "$([ "$AV_RECLOG" -ge 1 ] 2>/dev/null && echo yes || echo no)" "yes" +check_num "and no compaction dispatch touched av_hi, so the recluster path is the only candidate" \ + "$AV_CMPLOG" "0" +check_num "and the daemon never touched either reference, so they are still the twins they were" \ + "$AV_REFLOG" "0" +check_text "and the hilbert reference is byte-for-byte what it was before the daemon ran" \ + "$(scanorder av_ref)" "$REF_SEQ" +check_num "and it still has no appended tail, so nothing rewrote it behind the comparison" \ + "$(appended av_ref)" "$REF_APP" + +check_text "THE RULING: the daemon left the table hilbert, it did not convert it to zorder" \ + "$(skind av_hi)" "hilbert" +check_text "and sort_status agrees with the catalog about it" "$(skindst av_hi)" "hilbert" + +if [ "$S7_PRE" = "different" ] && [ "$REF_FOLDED" = "0" ]; then + check_text "and the layout the daemon produced IS a Hilbert layout: identical to the hand-driven hilbert twin" \ + "$(scanorder av_hi)" "$REF_SEQ" + check "and it is NOT the zorder layout, which is what a hard-coded recluster would have left" \ + "$(differs "$(scanorder av_hi)" "$ZO_SEQ")" "different" +else + check_unrunnable "and the layout the daemon produced IS a Hilbert layout: identical to the hand-driven hilbert twin" \ + UNMET_PRECONDITION \ + "av_hi already matched the reference, or the reference was never rebuilt (pre=[$S7_PRE], av_ref tail=[$REF_FOLDED]), so the comparison cannot be a change" + check_unrunnable "and it is NOT the zorder layout, which is what a hard-coded recluster would have left" \ + UNMET_PRECONDITION \ + "av_hi already matched the reference, or the reference was never rebuilt (pre=[$S7_PRE], av_ref tail=[$REF_FOLDED]), so this discriminates nothing" +fi +check_num "and no row was lost in the process" "$(q 'SELECT count(*) FROM av_hi;')" "25000" +check_text "and the rows are the same rows, only reordered" "$(setof av_hi)" "$AV_HI_SET" + +psql_run "ALTER SYSTEM SET pgcolumnar.autovacuum = off;" +q "SELECT pg_reload_conf();" >/dev/null +sleep 1 +check_text "and the daemon is OFF again, so the suite ends on the invariant it opened with" \ + "$(q "SHOW pgcolumnar.autovacuum;")" "off" + +# ============================================================================= +# S8 THE ENUMERATIONS REACT +# ============================================================================= +# +# test/entry_point_privilege.sh enumerates C entry points from pg_proc.prosrc +# and cross-checks that set against the install script's +# AS 'MODULE_PATHNAME','' clauses. Two new functions must appear in +# BOTH, and the two sets must stay identical. +# +# THE PROJECT RULE, restated because it has been lost three times: RESOLVE THE +# C SYMBOL FROM THE AS CLAUSE, NEVER BY DERIVING pgcolumnar_. +# get_storage_id (-> pgcolumnar_relation_storageid) and columnar_handler +# (-> pgcolumnar_handler) both break that convention and have been dropped by +# three separate enumerations that derived the name. So nothing below asserts +# that cluster_hilbert's symbol IS 'pgcolumnar_cluster_hilbert'. It asserts that +# whatever prosrc names is also declared in the script -- which is true whatever +# the implementer calls it. + +_hc_root="$(dirname "${BASH_SOURCE[0]}")/.." +_hc_ver="$(sed -n "s/^default_version *= *'\(.*\)'.*/\1/p" "$_hc_root/pgcolumnar.control")" +SQLFILE="$_hc_root/pgcolumnar--$_hc_ver.sql" +check "premise: the install script derived from default_version exists" \ + "$([ -r "$SQLFILE" ] && echo yes || echo "missing: $SQLFILE")" "yes" + +# Comments blanked first: a doc comment naming a symbol is not a declaration, +# and counting one is how a census comes out wrong in the other direction. +# +# The symbol class is [A-Za-z0-9_] and the separator swallows a tab. Every one +# of the 42 MODULE_PATHNAME symbols in the current script is lowercase with no +# digit, so the narrower class was correct today and would have gone RED, loudly +# but at the wrong address, on the first symbol carrying a digit. +src_syms="$(sed 's,--.*,,' "$SQLFILE" \ + | tr '\n' ' ' \ + | grep -o "AS 'MODULE_PATHNAME'[,[:space:]]*'[A-Za-z0-9_]*'" \ + | grep -o "'[A-Za-z0-9_]*'$" | tr -d "'" | sort -u | tr '\n' ' ')" +cat_syms="$(q "SELECT string_agg(DISTINCT p.prosrc, ' ' ORDER BY p.prosrc) + FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace + WHERE n.nspname = 'pgcolumnar' + AND p.prolang = (SELECT oid FROM pg_language WHERE lanname = 'c');")" +cat_syms="$(printf '%s\n' $cat_syms | sort -u | tr '\n' ' ')" + +check "premise: the install script declares MODULE_PATHNAME symbols at all" \ + "$([ "$(printf '%s\n' $src_syms | grep -c .)" -ge 20 ] && echo yes || echo no)" "yes" +check "the catalog and the install script still agree on the symbol set" \ + "$(diff <(printf '%s\n' $src_syms) <(printf '%s\n' $cat_syms) >/dev/null && echo same || echo differs)" \ + "same" + +for fn in cluster_hilbert recluster_hilbert; do + check_num "pgcolumnar.$fn is installed, exactly once" \ + "$(q "SELECT count(*) FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace + WHERE n.nspname = 'pgcolumnar' AND p.proname = '$fn';")" "1" + + # From prosrc. The server resolved it; no naming convention is involved. + _sym="$(q "SELECT p.prosrc FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace + WHERE n.nspname = 'pgcolumnar' AND p.proname = '$fn' + AND p.prolang = (SELECT oid FROM pg_language WHERE lanname = 'c');")" + check "$fn is a C function and the catalog names its symbol" \ + "$([ -n "$_sym" ] && echo yes || echo "no symbol")" "yes" + check "and the symbol the CATALOG names for $fn is declared in the install script's AS clause" \ + "$(case " $src_syms " in *" ${_sym:-__none__} "*) echo declared ;; *) echo "MISSING: ${_sym:-}" ;; esac)" \ + "declared" +done + +# The surface itself, compared against the sibling verb rather than retyped: the +# argument types, the variadic element type and the return type must match the +# established verb the new one shadows. +sigof() { # sigof FN -> argtypes | variadic element type | return type + q "SELECT p.proargtypes::text || '|' || p.provariadic::text || '|' || p.prorettype::text + FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace + WHERE n.nspname = 'pgcolumnar' AND p.proname = '$1';" +} +check_text "cluster_hilbert has exactly cluster's signature (args, VARIADIC element, return type)" \ + "$(sigof cluster_hilbert)" "$(sigof cluster)" +check_text "recluster_hilbert has exactly recluster's signature (args, VARIADIC element, return type)" \ + "$(sigof recluster_hilbert)" "$(sigof recluster)" + +pgc_summary From 4b665553090be08e0c24ed891144739fb96a7584 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 9 Sep 2026 07:14:38 -0600 Subject: [PATCH 5/6] feat: the Hilbert clustering verbs, and open the 1.0-alpha4 cycle (#889) Turns test/hilbert_cluster.sh from 98/74/9 to 181 passed, 0 failed, 0 unrunnable, without touching an arm. `git diff` over test/ carries no change to either Hilbert suite (both byte-identical to 058b844) beyond the registration line and the version-cycle work described below. pgcolumnar.cluster_hilbert(regclass, VARIADIC name[]) -> void pgcolumnar.recluster_hilbert(regclass, VARIADIC name[]) -> bigint Separate verbs because PostgreSQL refuses to extend the existing signature in either direction and an array-plus-kind overload breaks the documented cluster('t','a','b') call style. Both measured on 18.4, recorded on #889. THE CURVE IS STICKY, per the owner ruling. sorted_kind is the table's declared intent, so plain recluster() on a Hilbert table with a matching key is a no-op rather than a conversion, switching curves requires naming the other verb, vacuum_sorted skips a Hilbert table rather than re-sorting it lexicographically, and the daemon dispatches on the recorded kind. The curve is threaded as one parameter through the key builder, the self-gate's strcmp and the recorded extent, so the three cannot disagree. The eager worker was renamed from _zorder to _curve: a function called _zorder that lays Hilbert is the same silent-mislabel trap the kind literals were. WHY THIS OPENS 1.0-alpha4 RATHER THAN ADDING TO ALPHA3. The first implementation put both functions in pgcolumnar--1.0-alpha2--1.0-alpha3.sql. v1.0-alpha3 is a PUBLISHED pre-release (2026-09-03), so that file is a shipped artifact: editing it leaves a database that upgraded from the release and a fresh install both reporting 1.0-alpha3 with different function sets and no path between them. That is the failure extension versioning exists to prevent. So: VERSION and pgcolumnar.control -> 1.0-alpha4 pgcolumnar--1.0-alpha3.sql -> pgcolumnar--1.0-alpha4.sql pgcolumnar--1.0-alpha3--1.0-alpha4.sql new, carrying the two verbs pgcolumnar--1.0-alpha2--1.0-alpha3.sql reverted, byte-identical to 058b844 The four documents that cite `VERSION` were updated with it, which docs_style checks by comparing the quoted string against the file rather than trusting them to be edited together. AND THE NEWLY-PREVIOUS VERSION GAINS COVERAGE. test/native_upgrade_converge.sh staged frozen base scripts for 1.0-alpha and 1.0-alpha2 only, so a database at the version people actually have had no arm. The shipped alpha3 base script is now frozen into test/fixtures, taken from the TAG rather than from this tree (diff against `git show v1.0-alpha3:` is empty, and it contains neither new verb), and both loops cover it. 8 checks -> 11. The first edit added it to the staging loop alone and the count stayed at 8 -- a silent no-op, caught by watching the number rather than the exit status. The arms come from a second loop. REMOVAL PROOF on the upgrade path: renaming cluster_hilbert in the upgrade script reddens 9 of 11 arms, including all three 1.0-alpha3 ones, with "function pgcolumnar.cluster_hilbert(regclass, name[]) does not exist". Restored and re-verified. COPT=-Werror, PG18.4, build OK with 0 warnings: hilbert_cluster 181 sorted_pathkeys 113 harness_selftest 261 hilbert_curve 184 sorted_mark_rename 46 entry_point_priv 31 native_cluster 11 vacuum_sorted_gate 34 autovacuum 11 native_recluster 12 native_upgrade_conv 11 docs_style PASSED recluster_gate 13 recluster_extent 17 all 0 failed, 0 unrunnable KNOWN AND MEASURED, not inferred: the daemon's curve dispatch is NOT load-bearing against this suite. Reverting it to the hard-coded recluster() leaves all 181 green, because the sticky plain recluster() already preserves the curve. It is kept because the ruling is about the daemon and should be readable at the call rather than inferred from a rule two modules away -- but no arm guards it, and an arm that did would have to read the curve out of the daemon's log line, which the log now carries. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- CHANGELOG.md | 79 +- Makefile | 2 +- README.md | 2 +- VERSION | 2 +- docs/limitations.md | 2 +- docs/roadmap.md | 2 +- docs/sql-reference.md | 33 + pgcolumnar--1.0-alpha3--1.0-alpha4.sql | 38 + pgcolumnar--1.0-alpha4.sql | 1913 +++++++++++++++++ pgcolumnar.control | 2 +- src/columnar_autovacuum.c | 44 +- src/columnar_curve.h | 16 + src/columnar_vacuum.c | 592 +++-- .../fixtures/pgcolumnar--1.0-alpha3.sql | 0 test/native_upgrade_converge.sh | 6 +- test/run_all_versions.sh | 1 + 16 files changed, 2517 insertions(+), 217 deletions(-) create mode 100644 pgcolumnar--1.0-alpha3--1.0-alpha4.sql create mode 100644 pgcolumnar--1.0-alpha4.sql rename pgcolumnar--1.0-alpha3.sql => test/fixtures/pgcolumnar--1.0-alpha3.sql (100%) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4aba729d..50adede9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,20 +2,72 @@ All notable changes to pgColumnar are recorded here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). pgColumnar is -pre-release; the version marker is `1.0-alpha3`, recorded in `VERSION`. New tables +pre-release; the version marker is `1.0-alpha4`, recorded in `VERSION`. New tables are written in the native on-disk format, PGCN v1. For the forward-looking plan see [design/ROADMAP.md](design/ROADMAP.md); for full history see the git log. -The extension's `default_version` is `1.0-alpha3`, which is tagged as -`v1.0-alpha3` and is the latest published pre-release. Upgrade scripts from +The extension's `default_version` is `1.0-alpha4`, which is in development and not +yet tagged; `v1.0-alpha3` is the latest published pre-release. Upgrade scripts from every previously shipped version ship with it (`1.0-dev`, which the v1.0-alpha tag -installed, `1.0-alpha`, and `1.0-alpha2`), so a single -`ALTER EXTENSION pgcolumnar UPDATE` reaches `1.0-alpha3` from any of them. Older +installed, `1.0-alpha`, `1.0-alpha2`, and `1.0-alpha3`), so a single +`ALTER EXTENSION pgcolumnar UPDATE` reaches `1.0-alpha4` from any of them. Older notes in this file describe `default_version` as pinned at an earlier version, each true until the next version shipped. ## [Unreleased] +### Added + +- Hilbert clustering: `pgcolumnar.cluster_hilbert` and + `pgcolumnar.recluster_hilbert` (#889). + + **Two verbs rather than a parameter on the existing two.** PostgreSQL refuses + to extend `cluster(regclass, VARIADIC name[])` in either direction: a defaulted + parameter cannot precede a `VARIADIC` one, and an array-plus-kind overload + makes the documented `cluster('t','a','b')` call ambiguous. Both were measured + on 18.4. The new verbs match their siblings element for element in argument + types, variadic element type, return type and volatility, so a caller switches + between them by name alone. + + **What the curve buys.** Z-order jumps a long way in key space at a bit + boundary; a Hilbert curve does not. Keys that are close in the data therefore + stay closer in storage, the min/max zone maps over the clustered columns are + tighter, and a range filter reads fewer chunk groups. The key is the same width + and sorts through the same `bytea` comparator, so nothing downstream of the + sort knows which curve produced it. + + **The curve is sticky.** `sorted_kind` is the table's declared intent, not a + property of each call: + + - plain `recluster` on a Hilbert table over the same key is a no-op returning + 0, not a silent conversion back to Z-order; + - `recluster_hilbert` on a Z-ordered table over the same columns rewrites it; + - `vacuum_sorted` leaves a Hilbert table alone rather than sorting it + lexicographically and relabelling it; + - the maintenance daemon dispatches on the recorded kind, so a Hilbert table + is re-clustered with Hilbert instead of being converted on a timer; + - naming the other verb, or reclustering on a different key, is how a table + changes curve. + + Held by `test/hilbert_cluster.sh` (181 arms) over the SQL surface, the recorded + kind, both self-gates and the daemon, and by `test/hilbert_curve.sh` (184 arms) + over the encoder itself. + +- `test/projection_rewrite.sh`, 84 checks. Nothing in the tree asserted that a + projection answers after a rewrite, which is why this was silent. + + Every arm compares a `pgc_set_hash` of `read_projection` against the base table + rather than checking that the call did not raise, so a projection re-recorded + EMPTY fails -- which matters because the correct end state after a bare + `TRUNCATE` is an empty projection that answers. Every arm also asserts what its + operation DID (`REWROTE`, `NOOP` or `FAILED`) and reports its properties as + `UNMET_PRECONDITION` rather than as passes when it did not: an operation that + failed or no-opped leaves the storage id unchanged and `read_projection` + answering, which is indistinguishable from a path that handles projections + correctly. Three arms carry `pgcolumnar.vacuum`, `vacuum_sorted` and `cluster`, + which already re-record for themselves, so a future fix moved into the table-AM + callback reddens here instead of double-recording. + ### Fixed - `ALTER TABLE ... RENAME COLUMN` now carries the new name into @@ -139,23 +191,6 @@ true until the next version shipped. now re-records. It names the two cases that remain: a declaration that no longer resolves, and the implicit base projection, which is not readable by name at all. -### Added - -- `test/projection_rewrite.sh`, 84 checks. Nothing in the tree asserted that a - projection answers after a rewrite, which is why this was silent. - - Every arm compares a `pgc_set_hash` of `read_projection` against the base table - rather than checking that the call did not raise, so a projection re-recorded - EMPTY fails -- which matters because the correct end state after a bare - `TRUNCATE` is an empty projection that answers. Every arm also asserts what its - operation DID (`REWROTE`, `NOOP` or `FAILED`) and reports its properties as - `UNMET_PRECONDITION` rather than as passes when it did not: an operation that - failed or no-opped leaves the storage id unchanged and `read_projection` - answering, which is indistinguishable from a path that handles projections - correctly. Three arms carry `pgcolumnar.vacuum`, `vacuum_sorted` and `cluster`, - which already re-record for themselves, so a future fix moved into the table-AM - callback reddens here instead of double-recording. - ## [1.0-alpha3] - 2026-09-02 ### Added diff --git a/Makefile b/Makefile index 4ddd3a58..40c8eb0d 100644 --- a/Makefile +++ b/Makefile @@ -40,7 +40,7 @@ OBJS = \ src/columnar_autovacuum.o EXTENSION = pgcolumnar -DATA = pgcolumnar--1.0-alpha3.sql pgcolumnar--1.0-dev--1.0-alpha.sql pgcolumnar--1.0-alpha--1.0-alpha2.sql pgcolumnar--1.0-alpha2--1.0-alpha3.sql +DATA = pgcolumnar--1.0-alpha4.sql pgcolumnar--1.0-dev--1.0-alpha.sql pgcolumnar--1.0-alpha--1.0-alpha2.sql pgcolumnar--1.0-alpha2--1.0-alpha3.sql pgcolumnar--1.0-alpha3--1.0-alpha4.sql PGFILEDESC = "pgColumnar - column-oriented table access method" # make installcheck. Not the project's gate -- that is test/run_all_versions.sh, diff --git a/README.md b/README.md index 516690c6..aae7fe53 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ append-mostly data. pgColumnar builds from one source tree on PostgreSQL 15 through 18, with 19 validated against 19beta2, and is licensed under the [MIT License](LICENSE). It is [pre-release](docs/limitations.md#release-status); the version marker -is `1.0-alpha3`, recorded in `VERSION`. That version is in development and not +is `1.0-alpha4`, recorded in `VERSION`. That version is in development and not tagged; the latest published pre-release is `v1.0-alpha2`. A table `USING pgcolumnar` is stored in the native on-disk format, PGCN v1. diff --git a/VERSION b/VERSION index ee3fd01a..a3c28a3a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.0-alpha3 +1.0-alpha4 diff --git a/docs/limitations.md b/docs/limitations.md index 72da7d5f..5f31c21e 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -2,7 +2,7 @@ ## Release status -pgColumnar is pre-release. The version marker is `1.0-alpha3`, recorded in `VERSION`, +pgColumnar is pre-release. The version marker is `1.0-alpha4`, recorded in `VERSION`, and it is tagged `v1.0-alpha3`. On PGXN the same release is `1.0.0-alpha.3`. The two differ because PGXN requires a diff --git a/docs/roadmap.md b/docs/roadmap.md index 7abf863f..d866f2ef 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -12,7 +12,7 @@ Issues are the authority on anything being worked now. ## Status pgColumnar is [pre-release](limitations.md#release-status). The version marker is -`1.0-alpha3`, recorded in `VERSION`. That version is in development and not tagged; the +`1.0-alpha4`, recorded in `VERSION`. That version is in development and not tagged; the latest published pre-release is `v1.0-alpha2`. A table `USING pgcolumnar` is stored in the native on-disk format, PGCN v1. diff --git a/docs/sql-reference.md b/docs/sql-reference.md index e4378afd..62be4943 100644 --- a/docs/sql-reference.md +++ b/docs/sql-reference.md @@ -142,6 +142,25 @@ to reorder a live table without an exclusive lock. SELECT pgcolumnar.cluster('events', 'customer_id', 'ts'); ``` +### pgcolumnar.cluster_hilbert(tablename regclass, VARIADIC columns name[]) + +The same reorganisation on the Hilbert curve instead of the Z-order one. Take it +when the clustered columns carry range filters. The Hilbert index has no jumps at +a bit boundary. Keys that are close in the data therefore stay close in storage, +and a range filter reads fewer chunk groups. Everything else matches `cluster`: +the same arguments, the same refusals, the same `AccessExclusiveLock`. + +```sql +SELECT pgcolumnar.cluster_hilbert('events', 'customer_id', 'ts'); +``` + +**The curve is sticky.** The table records which curve it was laid on, and +`pgcolumnar.sort_status` reports it as `sorted_kind`. Once a table is on the +Hilbert curve, plain `cluster` and `recluster` on the same key maintain that +curve rather than converting it back. `vacuum_sorted` leaves the table alone, +and the maintenance daemon re-clusters it with Hilbert. To switch curves, name +the other verb, or recluster on a different key. + ### pgcolumnar.recluster(tablename regclass, VARIADIC columns name[]) returns bigint The online counterpart to `cluster`. Re-establishes the same Z-order clustering @@ -159,6 +178,20 @@ without rewriting anything when the recorded key matches, the kind is Z-order, and the existing sorted run already covers every row group. This is what lets the maintenance daemon call it on a schedule without churning storage. +On a table laid on the Hilbert curve over the key you name, `recluster` +maintains that curve. It does not convert the table to Z-order. + +### pgcolumnar.recluster_hilbert(tablename regclass, VARIADIC columns name[]) returns bigint + +The online counterpart to `cluster_hilbert`, and the way to move a Z-ordered +table onto the Hilbert curve. Same arguments, same lock and same return value as +`recluster`; it re-establishes Hilbert clustering rather than Z-order, and +records the curve it applied. + +```sql +SELECT pgcolumnar.recluster_hilbert('events', 'customer_id', 'ts'); +``` + ### pgcolumnar.compact(tablename regclass) returns bigint Retires row groups that are fully deleted, dropping their metadata so scans skip diff --git a/pgcolumnar--1.0-alpha3--1.0-alpha4.sql b/pgcolumnar--1.0-alpha3--1.0-alpha4.sql new file mode 100644 index 00000000..66610b52 --- /dev/null +++ b/pgcolumnar--1.0-alpha3--1.0-alpha4.sql @@ -0,0 +1,38 @@ +/* + * pgcolumnar--1.0-alpha3--1.0-alpha4.sql + * + * Upgrade from 1.0-alpha3 to 1.0-alpha4. + * + * 1.0-alpha3 is a PUBLISHED pre-release (tag v1.0-alpha3, 2026-09-03), so + * pgcolumnar--1.0-alpha2--1.0-alpha3.sql is a shipped artifact and must not + * change. Adding these functions there would leave two databases both reporting + * 1.0-alpha3 with different function sets and no upgrade path between them -- + * exactly what extension versioning exists to prevent. + */ + + +-- The two Hilbert clustering verbs (#889). New functions, so plain CREATE: an +-- alpha2 install has neither name. They mirror cluster() and recluster() +-- element for element in argument types, variadic element type, return type and +-- volatility, because the two pairs are one surface and a caller switches +-- between them by name alone. + +CREATE FUNCTION pgcolumnar.cluster_hilbert( + tablename regclass, + VARIADIC columns name[]) + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_cluster_hilbert'; + +COMMENT ON FUNCTION pgcolumnar.cluster_hilbert(regclass, name[]) + IS 'eager reorg on the Hilbert curve: as cluster(), but the rows are ordered by the Hilbert index over the given columns, which keeps neighbouring keys neighbouring in storage more tightly than Z-order does. Holds AccessExclusiveLock like CLUSTER/VACUUM FULL; the online counterpart is recluster_hilbert() (#889)'; + +CREATE FUNCTION pgcolumnar.recluster_hilbert( + tablename regclass, + VARIADIC columns name[]) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_recluster_hilbert'; + +COMMENT ON FUNCTION pgcolumnar.recluster_hilbert(regclass, name[]) + IS 'lazy online reclustering on the Hilbert curve: as recluster(), but re-establishes Hilbert clustering over the given columns under ShareUpdateExclusiveLock (concurrent reads and writes). The curve is sticky -- plain recluster() maintains a Hilbert table rather than converting it, and naming this verb is how a Z-ordered table is switched (#889)'; diff --git a/pgcolumnar--1.0-alpha4.sql b/pgcolumnar--1.0-alpha4.sql new file mode 100644 index 00000000..fd5ae29f --- /dev/null +++ b/pgcolumnar--1.0-alpha4.sql @@ -0,0 +1,1913 @@ +/* pgColumnar 1.0 - native (PGCN v1) metadata catalog and access method + * registration. + * + * The catalog matches section 11 of + * design/NATIVE_FORMAT_AND_INTERFACE_SPEC.md. Column order and index + * definitions are part of the on-disk format. + * + * The catalog holds the native storage, row_group, column_chunk, zone_map, + * and bloom tables, the shared delete_vector and options tables, the storageid_seq + * sequence, the columnar_handler function, and the columnar access method. + */ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION pgcolumnar" to load this file. \quit + +/* --------------------------------------------------------------------------- + * Sequences (spec 7.6) + * ------------------------------------------------------------------------- */ + +CREATE SEQUENCE pgcolumnar.storageid_seq + MINVALUE 10000000000 + NO CYCLE; + +/* --------------------------------------------------------------------------- + * pgcolumnar.delete_vector (spec 7.5) + * + * Tracks deleted rows for updates and deletes without rewriting stripes. One + * row per chunk group, keyed by group_number; a set bit in "bitmap" marks a + * deleted row (bit i is the group's i-th row, row_group.first_row_number + i, + * LSB-first in byte i/8). deleted_count is the number of set bits. + * ------------------------------------------------------------------------- */ + +CREATE TABLE pgcolumnar.delete_vector ( + storage_id bigint NOT NULL, + group_number bigint NOT NULL, + bitmap bytea, + deleted_count integer NOT NULL +); + +CREATE UNIQUE INDEX delete_vector_pkey + ON pgcolumnar.delete_vector USING btree (storage_id, group_number); + +/* --------------------------------------------------------------------------- + * pgcolumnar.options (spec 7.4) + * + * Per-table overrides of the instance-wide compression, compression level, + * chunk-group row limit, and stripe row limit. A NULL column means the table + * uses the instance default (the GUC) for that option. Keyed by regclass. + * ------------------------------------------------------------------------- */ + +CREATE TABLE pgcolumnar.options ( + regclass regclass NOT NULL, + chunk_group_row_limit integer, + stripe_row_limit integer, + compression_level integer, + compression name, + encode_effort name, + sort_by name[], -- declared physical sort key (#288) + -- Declared retention (#403 item 5a), read only by pgcolumnar.expire. + -- Nothing drops rows on its own: expire is called by name. + ttl_column name, + ttl_interval interval +); + +/* + * sort_by holds COLUMN NAMES, not attnums, on purpose. pgcolumnar.options is + * the one catalog carried through pg_dump (pg_extension_config_dump below); a + * plain (non-binary-upgrade) pg_dump does not re-emit dropped columns, so live + * attnums renumber densely on restore while names do not. The governing rule: + * store NAMES in the dumped catalog (regclass, sort_by); store ATTNUMS only in + * the storage_id-keyed catalogs that are NOT dumped (projection.sort_key, + * row_group.sort_key), which are regenerated on restore anyway. See the + * regclass rationale below. NULL means no declared sort key; the apply path + * (vacuum_sorted with no explicit columns) resolves the names to attnums each + * time and re-validates them, so a later DROP/RENAME of a named column is + * caught then rather than corrupting anything. + */ + +CREATE UNIQUE INDEX options_pkey + ON pgcolumnar.options USING btree (regclass); + +/* + * Carry the per-table options through pg_dump (#248). + * + * Rows in an extension's own tables are not dumped unless the extension says so. + * Without this, pg_dump emitted the table definition and its data but never the + * options row, so a restored columnar table silently reverted to default + * stripe/chunk limits, compression and encode_effort. Silent, because nothing + * fails: the data is all there and only the settings are gone. + * + * This table and ONLY this table. Every other pgcolumnar catalog table is keyed + * by storage_id, which is assigned when the relation is created, so a restore + * generates new ones -- dumping those rows would restore metadata pointing at + * storage that no longer exists, which is worse than losing it. options is keyed + * by regclass, a name that survives dump and restore, and it holds user intent + * rather than physical layout, which is the same reason it is the only one worth + * carrying. + * + * Projections are user intent too and are still lost across a dump, for the + * storage_id reason above; re-emitting pgcolumnar.add_projection() calls is a + * different mechanism and its own problem. + */ +SELECT pg_catalog.pg_extension_config_dump('pgcolumnar.options', ''); + +/* + * The declared intent behind each projection, as opposed to pgcolumnar.projection + * which records the materialized result (#266). + * + * pgcolumnar.projection is keyed by storage_id and stores attnums, so pg_dump + * cannot carry it: a restore assigns new storage ids, and rows pointing at + * storage that does not exist would be worse than losing them. This table is + * keyed by regclass and stores column NAMES, for the same reason + * pgcolumnar.options is keyed by regclass and the sort_by key stores names: a + * name survives a dump and a restore, and a restore renumbers an attnum. + * + * So a dump carries the declaration and not the data. After a restore the + * declarations are present and the projection storage is not, and + * pgcolumnar.rebuild_projections() materializes them. Readers never consult this + * table. They read pgcolumnar.projection, where a row appears only after its + * storage exists. + */ +CREATE TABLE pgcolumnar.projection_declaration ( + rel regclass NOT NULL, + name name NOT NULL, + columns text[] NOT NULL, + sort_key text[] NOT NULL +); + +CREATE UNIQUE INDEX projection_declaration_pkey + ON pgcolumnar.projection_declaration USING btree (rel, name); + +SELECT pg_catalog.pg_extension_config_dump('pgcolumnar.projection_declaration', ''); + +/* --------------------------------------------------------------------------- + * pgcolumnar.projection (gap 26) + * + * Multiple physical projections per table (C-Store). Each projection is a named, + * ordered subset of the table's columns stored as its own columnar storage + * (proj_storage_id) sorted on sort_key, sharing the row-number identity space. + * projection_id 0 is the implicit base projection (all columns, insert order); + * a table with no rows here has a single implicit base projection, so a table + * with no declared projections behaves as one with only its base. + * ------------------------------------------------------------------------- */ + +CREATE TABLE pgcolumnar.projection ( + storage_id bigint NOT NULL, -- the table's base storage id + projection_id integer NOT NULL, -- 0 = base, 1..N additional + name name NOT NULL, + proj_storage_id bigint NOT NULL, -- this projection's own storage id + sort_key smallint[] NOT NULL, -- attnums in sort order ({} = insert order) + columns smallint[] NOT NULL -- attnums stored (base = all live columns) +); + +CREATE UNIQUE INDEX projection_pkey + ON pgcolumnar.projection USING btree (storage_id, projection_id); + +CREATE UNIQUE INDEX projection_name_idx + ON pgcolumnar.projection USING btree (storage_id, name); + +CREATE UNIQUE INDEX projection_storage_idx + ON pgcolumnar.projection USING btree (proj_storage_id); + +/* --------------------------------------------------------------------------- + * Native format catalog (format PGCN v1). + * + * The native on-disk format (design/NATIVE_FORMAT_AND_INTERFACE_SPEC.md + * section 11). Dropped with the extension; per-table row cleanup is wired + * into ColumnarDeleteMetadata. + * ------------------------------------------------------------------------- */ + +CREATE TABLE pgcolumnar.storage ( + storage_id bigint NOT NULL, -- native relation storage id + relation_oid oid NOT NULL, + format_version integer NOT NULL, -- native format major version (1) + vector_length integer NOT NULL, -- values per vector (1024) + row_group_limit integer NOT NULL, -- max rows per row group + -- The row group number the last ordering rewrite ended at (#301). NULL means + -- the storage was never ordered. + -- + -- pgcolumnar.vacuum_sorted, pgcolumnar.cluster and pgcolumnar.recluster order + -- every live row, so every group up to and including this number is part of + -- one ordered run. Groups numbered above it were written later, in insert + -- order, and are the unsorted tail. That is what makes a sorted layout decay, + -- and pgcolumnar.sort_status reports the size of each part. + -- + -- It is a boundary rather than a count because the online maintenance paths + -- retire groups and write replacements with fresh, higher numbers. A count + -- would silently re-point at those replacements as the run shrank; a boundary + -- leaves them above the mark, where they belong. + -- + -- It lives here rather than in pgcolumnar.options because a storage row has + -- exactly the right lifetime. Any rewrite creates a new storage id, so an + -- unsorted vacuum leaves this NULL and correctly reports the table as + -- unsorted, with no invalidation step. A value in an options row, which is + -- keyed by relation, would outlive the layout it describes. + sorted_through bigint, + -- Lower end of the ordered run (#342). The run is [sorted_from, + -- sorted_through]; a bare upper bound cannot exclude a concurrently written + -- group whose id was drawn below the rewrite's own first id, which is how a + -- foreign group came to be counted as ordered. + sorted_from bigint, + -- What the ordered run is clustered BY and HOW (#415). sorted_by is the + -- clustering columns; sorted_kind is 'zorder' (recluster/cluster) or + -- 'lexicographic' (vacuum_sorted). Both NULL on an unordered storage, or on + -- one ordered before this column existed -- treated as "unknown key", which + -- the self-gating recluster never skips. They let recluster tell "already + -- clustered by these columns this way" from "clustered by something else", + -- so it returns without a full rewrite when nothing decayed. + sorted_by name[], + sorted_kind text +); +CREATE UNIQUE INDEX storage_pkey + ON pgcolumnar.storage USING btree (storage_id); + +CREATE TABLE pgcolumnar.row_group ( + storage_id bigint NOT NULL, + -- ONE-BASED. group_number is the stripe id reserved from the metapage when + -- the group began buffering, and PgColumnarInitMetapage starts + -- reservedStripeId at 1, so there is no group 0 on any storage. The same + -- numbering is used by column_chunk, zone_map, bloom and delete_vector. + -- + -- This said "0-based row group ordinal" until #817. Nothing read the comment + -- at runtime, but a reader did: the planner's zone-map sample walked + -- [0, ngroups), so it spent its first probe on a number that cannot exist and + -- never probed the highest group at all, which priced a predicate differently + -- according to where in the table its groups sat. + group_number bigint NOT NULL, + file_offset bigint NOT NULL, -- logical byte offset of the group + row_count bigint NOT NULL, + byte_length bigint NOT NULL, + first_row_number bigint NOT NULL, -- row number of the group's first row + sort_key smallint[] NOT NULL DEFAULT '{}' -- attnums the group is sorted on +); +CREATE UNIQUE INDEX row_group_pkey + ON pgcolumnar.row_group USING btree (storage_id, group_number); + +CREATE TABLE pgcolumnar.column_chunk ( + storage_id bigint NOT NULL, + group_number bigint NOT NULL, + column_index smallint NOT NULL, -- 0-based attribute position + value_count bigint NOT NULL, + encoding_descriptor bytea NOT NULL, -- the chosen cascade (Phase D4) + block_codec smallint NOT NULL, -- optional final block codec (0 = none) + page_offset bigint NOT NULL, -- logical byte offset of the chunk's page + page_length bigint NOT NULL +); +CREATE UNIQUE INDEX column_chunk_pkey + ON pgcolumnar.column_chunk USING btree (storage_id, group_number, column_index); + +CREATE TABLE pgcolumnar.zone_map ( + storage_id bigint NOT NULL, + group_number bigint NOT NULL, + column_index smallint NOT NULL, + vector_index integer NOT NULL, -- -1 for the whole-chunk aggregate + minimum bytea, -- encoded per the column type + maximum bytea, + sum numeric, -- NULL when the type has no sum + value_count bigint NOT NULL, + null_count bigint NOT NULL +); +CREATE UNIQUE INDEX zone_map_pkey + ON pgcolumnar.zone_map USING btree (storage_id, group_number, column_index, vector_index); + +-- Per-column-chunk bloom filter for equality skipping on hashable columns +-- (native spec 7.2). One row per (storage_id, group_number, column_index). +CREATE TABLE pgcolumnar.bloom ( + storage_id bigint NOT NULL, + group_number bigint NOT NULL, + column_index smallint NOT NULL, + filter bytea NOT NULL +); +CREATE UNIQUE INDEX bloom_pkey + ON pgcolumnar.bloom USING btree (storage_id, group_number, column_index); + +-- Loads pgcolumnar.parallel_copy has already performed, for its opt-in dedup +-- (#403 item 7). One row per (table, file fingerprint) that committed. +-- +-- The fingerprint is the SHA-256 of the loaded file's bytes, so a file that +-- changed at the same path is a different load. Path, size and mtime would all +-- call that the same file. +-- +-- The row is written AFTER the data commits, never before. A crash between the +-- two leaves data with no fingerprint, so a retry loads again, which is the +-- behaviour without this feature and is the safe direction. The reverse order +-- would leave a fingerprint with no data and refuse rows that were never stored. +CREATE TABLE pgcolumnar.load_fingerprint ( + relation_oid oid NOT NULL, + fingerprint bytea NOT NULL, -- SHA-256 of the file's bytes + rows bigint NOT NULL, + loaded_at timestamptz NOT NULL DEFAULT now() +); +-- NOT unique, deliberately. The lookup is an existence test, and a unique +-- index would turn the one case that can produce a second row into an ERROR +-- raised AFTER the data committed: two concurrent loads of the same file both +-- check before either records, both commit, and the loser's record insert would +-- fail, reporting failure for a load that succeeded. A duplicate record is +-- harmless; a false failure is not. +CREATE INDEX load_fingerprint_idx + ON pgcolumnar.load_fingerprint USING btree (relation_oid, fingerprint); + +/* --------------------------------------------------------------------------- + * pgcolumnar.free_space (Phase F physical reclaim) + * + * Freed logical byte ranges from retired row groups, available for reuse by a + * later stripe reservation once no snapshot can still read them. file_offset is + * page-aligned; freed_xid is the retiring transaction's id, and the range is + * reusable only once the oldest-xmin horizon has passed it. Reuse makes online + * compaction space-neutral instead of forever advancing the file highwater. + * ------------------------------------------------------------------------- */ + +CREATE TABLE pgcolumnar.free_space ( + storage_id bigint NOT NULL, + file_offset bigint NOT NULL, + byte_length bigint NOT NULL, + freed_xid bigint NOT NULL +); +CREATE UNIQUE INDEX free_space_pkey + ON pgcolumnar.free_space USING btree (storage_id, file_offset); +CREATE INDEX free_space_fit + ON pgcolumnar.free_space USING btree (storage_id, byte_length); + +/* --------------------------------------------------------------------------- + * Access method (spec 8.1) + * ------------------------------------------------------------------------- */ + +CREATE FUNCTION pgcolumnar.columnar_handler(internal) + RETURNS table_am_handler + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_handler'; + +CREATE ACCESS METHOD pgcolumnar + TYPE TABLE + HANDLER pgcolumnar.columnar_handler; + +COMMENT ON ACCESS METHOD pgcolumnar IS 'pgColumnar column-oriented storage'; + +/* --------------------------------------------------------------------------- + * Conversion between heap and columnar (spec 8.2) + * + * alter_table_set_access_method converts a table between heap and columnar by + * driving PostgreSQL's own ALTER TABLE ... SET ACCESS METHOD, which rewrites + * the table through the target access method (columnar's insert path when + * converting to columnar, its scan path when converting away). Row counts and + * values round-trip. "t" is a table name (optionally schema-qualified); + * "method" is "pgcolumnar" or "heap" (or any other table access method). + * ------------------------------------------------------------------------- */ + +CREATE FUNCTION pgcolumnar.alter_table_set_access_method(t text, method text) + RETURNS void + LANGUAGE plpgsql + AS $alter_table_set_access_method$ +DECLARE + rel regclass := t::regclass; + nsp text; + tbl text; + tmp text; +BEGIN + /* + * PostgreSQL 15 introduced ALTER TABLE ... SET ACCESS METHOD, which + * rewrites the table in place through the target access method and + * preserves the relation's identity and dependents. Use it when available. + */ + IF current_setting('server_version_num')::int >= 150000 THEN + EXECUTE format('ALTER TABLE %s SET ACCESS METHOD %I', rel::text, method); + RETURN; + END IF; + + /* + * PostgreSQL 13 and 14 have no ALTER TABLE ... SET ACCESS METHOD. Convert + * by building a sibling table that uses the target access method, copying + * every row through it, and swapping names. Column definitions, defaults, + * NOT NULL and CHECK constraints, and indexes are carried over + * (LIKE ... INCLUDING ALL). This does not preserve the original table's OID + * or objects that depend on it (views, foreign keys); on those majors that + * is a documented limitation of the conversion helper. + */ + SELECT n.nspname, c.relname INTO nsp, tbl + FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE c.oid = rel; + tmp := tbl || '_pgcolumnar_conv'; + + EXECUTE format('CREATE TABLE %I.%I (LIKE %I.%I INCLUDING ALL) USING %I', + nsp, tmp, nsp, tbl, method); + EXECUTE format('INSERT INTO %I.%I SELECT * FROM %I.%I', + nsp, tmp, nsp, tbl); + EXECUTE format('DROP TABLE %I.%I', nsp, tbl); + EXECUTE format('ALTER TABLE %I.%I RENAME TO %I', nsp, tmp, tbl); +END; +$alter_table_set_access_method$; + +COMMENT ON FUNCTION pgcolumnar.alter_table_set_access_method(text, text) + IS 'convert a table between heap and columnar storage'; + +/* --------------------------------------------------------------------------- + * Per-table option set and reset (spec 8.2) + * + * set_options stores per-table option overrides; a NULL argument + * leaves that option unchanged. reset_options clears an option + * back to the instance default when its boolean argument is true. Options take + * effect for writes that begin after they are set. + * ------------------------------------------------------------------------- */ + +CREATE FUNCTION pgcolumnar.set_options( + table_name regclass, + chunk_group_row_limit int DEFAULT NULL, + stripe_row_limit int DEFAULT NULL, + compression name DEFAULT NULL, + compression_level int DEFAULT NULL, + encode_effort name DEFAULT NULL, + sort_by name[] DEFAULT NULL, + ttl_column name DEFAULT NULL, + ttl_interval interval DEFAULT NULL) + RETURNS void + LANGUAGE plpgsql + AS $set_options$ +DECLARE + col name; +BEGIN + /* + * The options are per-relation and are read by the columnar writer, so a row + * recorded for a relation that is not columnar can never be used. Storing one + * is not merely useless: the drop hook that clears pgcolumnar.options fires + * only for columnar relations, so the row outlives the table and is left + * keyed to a dangling oid that a later relation reusing that oid inherits. + * Measured before this guard, on the same cluster: set_options on a heap + * table stored a row, DROP TABLE left it behind, and regclass then rendered + * as the bare oid; the identical sequence on a columnar table cleaned up. + * + * Rejecting is safe for the one workflow that could want the other order: + * ALTER TABLE ... SET ACCESS METHOD pgcolumnar keeps the relation's oid + * (measured), so options set after the conversion apply to the same relation + * a caller would have been trying to name before it. + * + * The ERRCODE is explicit. plpgsql's RAISE EXCEPTION defaults to P0001, and + * the C paths raise this same sentence with ERRCODE_WRONG_OBJECT_TYPE + * (42809). Without it the identical message carried two different SQLSTATEs + * depending on which path refused the caller, in a tree whose own privilege + * suites deliberately assert SQLSTATE rather than message text. + * + * relkind is part of the test, and it is what makes the guard match the + * cleanup rather than merely look strict. The drop hook returns before it + * examines the access method for anything that is not an ordinary table + * (columnar_tableam.c: `if (get_rel_relkind(objectId) != RELKIND_RELATION) + * return;`), so 'r' is exactly the set of relations whose options row can + * ever be cleaned up. From PG17 a PARTITIONED table may carry an access + * method, so `relam = pgcolumnar` alone admits a parent that has no storage, + * that the writer never writes, and whose row the hook will never clear. + * Measured on 17.6 with the amname-only test: accepted, one row recorded, + * and the row still there after DROP TABLE keyed to the dropped oid, while + * an ordinary columnar table in the same run cleaned up. PG16 and earlier + * cannot reach it -- they refuse `PARTITION BY ... USING pgcolumnar` + * outright, checked on 16.14 -- so this is PG17, 18 and 19. + */ + IF NOT EXISTS (SELECT 1 FROM pg_class c + JOIN pg_am a ON a.oid = c.relam + WHERE c.oid = table_name + AND a.amname = 'pgcolumnar' + AND c.relkind = 'r') THEN + RAISE EXCEPTION 'relation "%" is not a columnar table', table_name + USING ERRCODE = 'wrong_object_type', + HINT = 'Per-table options are read by the columnar writer and ' + 'apply only to an ordinary table using the pgcolumnar access ' + 'method. A partitioned table has no storage of its own: set the ' + 'options on each partition. Otherwise convert the table first ' + 'with ALTER TABLE ... SET ACCESS METHOD pgcolumnar, then set ' + 'the options.'; + END IF; + + IF encode_effort IS NOT NULL AND + encode_effort NOT IN ('full', 'fast') THEN + RAISE EXCEPTION 'unknown columnar encode_effort "%"', encode_effort + USING HINT = 'Valid values are "full" and "fast".'; + END IF; + + IF compression IS NOT NULL AND + compression NOT IN ('none', 'pglz', 'lz4', 'zstd') THEN + RAISE EXCEPTION 'unknown columnar compression "%"', compression; + END IF; + + /* + * Bound the integer limits to the same valid ranges as the instance-wide + * GUCs (pgcolumnar.chunk_group_row_limit, pgcolumnar.stripe_row_limit, + * pgcolumnar.compression_level). A per-table value outside these ranges is + * rejected here rather than stored: a limit of zero or below would produce + * a stripe whose recorded chunk_row_count is zero and make the row-number + * arithmetic (chunk id = offset / chunk_row_count) divide by zero on + * delete, update, and index fetch. + */ + IF chunk_group_row_limit IS NOT NULL AND chunk_group_row_limit < 100 THEN + RAISE EXCEPTION 'chunk_group_row_limit must be at least 100'; + END IF; + IF stripe_row_limit IS NOT NULL AND stripe_row_limit < 1000 THEN + RAISE EXCEPTION 'stripe_row_limit must be at least 1000'; + END IF; + IF compression_level IS NOT NULL AND + (compression_level < 1 OR compression_level > 22) THEN + RAISE EXCEPTION 'compression_level must be between 1 and 22'; + END IF; + + /* + * A negative retention puts the cutoff in the FUTURE, so expire finds + * `maximum < cutoff` true for groups that are entirely inside their + * retention and retires them. That drops live rows, which is the failure + * this option exists to prevent. Every other option here is range-checked + * and this one was not. + * + * Zero is refused too. It is not a data-loss shape -- the cutoff is now, so + * only groups already wholly in the past go -- but "expire everything older + * than nothing" has no reading a caller means on purpose, and accepting it + * silently makes a typo indistinguishable from an instruction. + * + * ERRCODE is explicit for the reason the relkind guard above gives: this + * tree's suites assert SQLSTATE rather than message text, and plpgsql would + * otherwise default to P0001. + */ + IF ttl_interval IS NOT NULL AND ttl_interval <= interval '0' THEN + RAISE EXCEPTION 'ttl_interval must be a positive interval, not %', ttl_interval + USING ERRCODE = 'invalid_parameter_value', + HINT = 'A negative retention puts the cutoff in the future, ' + 'so pgcolumnar.expire() would retire groups whose rows are ' + 'still within their retention.'; + END IF; + + /* + * sort_by declares the physical sort key applied by vacuum_sorted() with no + * explicit columns (#288). This is a cheap early check only: each named + * column must exist, not be dropped, and not be a VIRTUAL generated column + * (its value is not stored, so it cannot be sorted on). Orderability + * (a default btree ordering operator) is NOT checked here -- the C apply + * path is authoritative and re-resolves and re-validates the names every + * run, because a column can be dropped or altered after it is declared. + * attgenerated is '' or 's' before PG18; 'v' only exists from PG18, so the + * "<> 'v'" test is correct and inert on older majors. + */ + IF sort_by IS NOT NULL THEN + FOREACH col IN ARRAY sort_by LOOP + IF NOT EXISTS (SELECT 1 FROM pg_attribute a + WHERE a.attrelid = table_name + AND a.attname = col + AND a.attnum > 0 + AND NOT a.attisdropped + AND a.attgenerated <> 'v') THEN + RAISE EXCEPTION 'column "%" cannot be used in sort_by for table %', + col, table_name + USING HINT = 'The column must exist, must not be dropped, ' + 'and must not be a VIRTUAL generated column.'; + END IF; + END LOOP; + END IF; + + INSERT INTO pgcolumnar.options AS o + (regclass, chunk_group_row_limit, stripe_row_limit, + compression, compression_level, encode_effort, sort_by, + ttl_column, ttl_interval) + VALUES (table_name, chunk_group_row_limit, stripe_row_limit, + compression, compression_level, encode_effort, sort_by, + ttl_column, ttl_interval) + ON CONFLICT (regclass) DO UPDATE SET + chunk_group_row_limit = + COALESCE(EXCLUDED.chunk_group_row_limit, o.chunk_group_row_limit), + stripe_row_limit = + COALESCE(EXCLUDED.stripe_row_limit, o.stripe_row_limit), + compression = + COALESCE(EXCLUDED.compression, o.compression), + compression_level = + COALESCE(EXCLUDED.compression_level, o.compression_level), + encode_effort = + COALESCE(EXCLUDED.encode_effort, o.encode_effort), + sort_by = + COALESCE(EXCLUDED.sort_by, o.sort_by), + ttl_column = + COALESCE(EXCLUDED.ttl_column, o.ttl_column), + ttl_interval = + COALESCE(EXCLUDED.ttl_interval, o.ttl_interval); +END; +$set_options$; + +COMMENT ON FUNCTION pgcolumnar.set_options(regclass, int, int, name, int, name, name[], name, interval) + IS 'set per-table columnar options; NULL leaves a value unchanged. sort_by declares the physical sort key applied by vacuum_sorted() with no explicit columns (#288); it is NOT auto-maintained -- rows inserted after a sort append in insert order, so re-run vacuum_sorted() to re-establish it, like PostgreSQL CLUSTER'; + +CREATE FUNCTION pgcolumnar.reset_options( + table_name regclass, + chunk_group_row_limit bool DEFAULT false, + stripe_row_limit bool DEFAULT false, + compression bool DEFAULT false, + compression_level bool DEFAULT false, + encode_effort bool DEFAULT false, + sort_by bool DEFAULT false) + RETURNS void + LANGUAGE plpgsql + AS $reset_options$ +BEGIN + UPDATE pgcolumnar.options o SET + chunk_group_row_limit = CASE + WHEN reset_options.chunk_group_row_limit + THEN NULL ELSE o.chunk_group_row_limit END, + stripe_row_limit = CASE + WHEN reset_options.stripe_row_limit + THEN NULL ELSE o.stripe_row_limit END, + compression = CASE + WHEN reset_options.compression + THEN NULL ELSE o.compression END, + compression_level = CASE + WHEN reset_options.compression_level + THEN NULL ELSE o.compression_level END, + encode_effort = CASE + WHEN reset_options.encode_effort + THEN NULL ELSE o.encode_effort END, + sort_by = CASE + WHEN reset_options.sort_by + THEN NULL ELSE o.sort_by END + WHERE o.regclass = table_name; +END; +$reset_options$; + +COMMENT ON FUNCTION pgcolumnar.reset_options(regclass, bool, bool, bool, bool, bool, bool) + IS 'reset per-table columnar options to the instance defaults'; + +/* --------------------------------------------------------------------------- + * Storage-id lookup, statistics, and vacuum (spec 8.2) + * ------------------------------------------------------------------------- */ + +CREATE FUNCTION pgcolumnar.get_storage_id(rel regclass) + RETURNS bigint + LANGUAGE C STABLE STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_relation_storageid'; + +COMMENT ON FUNCTION pgcolumnar.get_storage_id(regclass) + IS 'storage id linking a columnar table to its metadata rows'; + +CREATE FUNCTION pgcolumnar.add_projection( + rel regclass, + name text, + columns text[], + sort_key text[] DEFAULT '{}') + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_add_projection'; + +COMMENT ON FUNCTION pgcolumnar.add_projection(regclass, text, text[], text[]) + IS 'declare a physical projection: a named column subset sorted on sort_key (gap 26)'; + +CREATE FUNCTION pgcolumnar.drop_projection(rel regclass, name text) + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_drop_projection'; + +COMMENT ON FUNCTION pgcolumnar.drop_projection(regclass, text) + IS 'drop a declared projection and free its storage (gap 26)'; + +/* + * Materialize every declaration that has no projection behind it (#266). + * + * The case this exists for is a logical restore. pg_dump carries + * pgcolumnar.projection_declaration and cannot carry the projection storage, so + * a restored table has the declarations and none of the projections. This builds + * them, and returns the number that it built. + * + * You can run it at any time. It does not act on a declaration that is already + * materialized, so a second run builds nothing. + */ +CREATE FUNCTION pgcolumnar.rebuild_projections(rel regclass DEFAULT NULL) + RETURNS integer + LANGUAGE plpgsql + AS $$ +DECLARE + d record; + rebuilt integer := 0; +BEGIN + /* + * Forget a declaration whose relation is gone (#304). The drop hook removes + * these, so a current build does not make them. A database created by a + * build that did not clean up on drop still holds them, and one such row + * used to abort this function for every other table in the database: the + * guard below resolves pd.rel, and resolving a dropped relation raises. + * Deleting them here makes an affected database repair itself. + */ + DELETE FROM pgcolumnar.projection_declaration pd + WHERE NOT EXISTS (SELECT 1 FROM pg_catalog.pg_class c WHERE c.oid = pd.rel); + + FOR d IN + SELECT pd.rel, pd.name, pd.columns, pd.sort_key + FROM pgcolumnar.projection_declaration pd + WHERE (rebuild_projections.rel IS NULL OR pd.rel = rebuild_projections.rel) + AND NOT EXISTS ( + SELECT 1 + FROM pgcolumnar.projection p + WHERE p.storage_id = pgcolumnar.get_storage_id(pd.rel) + AND p.name = pd.name + AND p.projection_id > 0) + ORDER BY pd.rel::text, pd.name + LOOP + PERFORM pgcolumnar.add_projection(d.rel, d.name::text, d.columns, d.sort_key); + rebuilt := rebuilt + 1; + END LOOP; + RETURN rebuilt; +END; +$$; + +COMMENT ON FUNCTION pgcolumnar.rebuild_projections(regclass) + IS 'materialize declared projections that have no storage, after a logical restore (#266)'; + +CREATE FUNCTION pgcolumnar.read_projection(rel regclass, name text) + RETURNS SETOF text + LANGUAGE C STABLE + AS 'MODULE_PATHNAME', 'pgcolumnar_read_projection'; + +COMMENT ON FUNCTION pgcolumnar.read_projection(regclass, text) + IS 'read a projection''s stored columns (live rows), joined by | -- verification/debug (gap 26)'; + +CREATE FUNCTION pgcolumnar.reconstruct_via_projection(rel regclass, name text) + RETURNS SETOF text + LANGUAGE C STABLE + AS 'MODULE_PATHNAME', 'pgcolumnar_reconstruct_via_projection'; + +COMMENT ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass, text) + IS 'read all live rows via a projection, reconstructing non-covered columns from the base by row number (gap 26)'; + +-- #562: EXECUTE is granted to PUBLIC by CREATE FUNCTION, so without these the +-- C check below is the only boundary. Two layers on purpose: the C check is +-- what makes the functions safe, and this is what keeps an unprivileged role +-- from reaching them at all. +REVOKE ALL ON FUNCTION pgcolumnar.read_projection(regclass, text) FROM PUBLIC; +REVOKE ALL ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass, text) FROM PUBLIC; + +CREATE FUNCTION pgcolumnar.require_caller_select(rel regclass) RETURNS void + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_require_caller_select'; + +REVOKE ALL ON FUNCTION pgcolumnar.require_caller_select(regclass) FROM PUBLIC; + +COMMENT ON FUNCTION pgcolumnar.require_caller_select(regclass) + IS 'raise unless the calling role may SELECT the relation; for SECURITY DEFINER callers (#560)'; + +-- SECURITY DEFINER, because this reads pgcolumnar's catalog tables and those +-- carry no GRANT, so a columnar table's own owner could not read the statistics +-- of the table they own. Granting SELECT on the catalog instead would publish +-- pgcolumnar.zone_map, which holds per-column minimum, maximum and sum for every +-- columnar table. That is actual column data and a far larger disclosure than +-- the usability defect it would fix. +-- +-- Definer rights mean nothing else will refuse anyone, so the privilege check is +-- explicit and is the first statement. search_path is pinned because a definer +-- function must not resolve names through a caller-controlled path. +CREATE FUNCTION pgcolumnar.stats( + rel regclass, + OUT stripeid bigint, + OUT fileoffset bigint, + OUT rowcount bigint, + OUT deletedrows bigint, + OUT chunkcount integer, + OUT datalength bigint) + RETURNS SETOF record + LANGUAGE plpgsql STABLE SECURITY DEFINER + SET search_path = pg_catalog, pg_temp + AS $stats$ +BEGIN + PERFORM pgcolumnar.require_caller_select(rel); + RETURN QUERY + -- Native (PGCN v1) tables report one row per row group from the native + -- catalog. + SELECT rg.group_number, + rg.file_offset, + rg.row_count, + COALESCE((SELECT sum(rm.deleted_count)::bigint + FROM pgcolumnar.delete_vector rm + WHERE rm.storage_id = rg.storage_id + AND rm.group_number = rg.group_number), 0::bigint), + (SELECT count(DISTINCT zm.vector_index)::int + FROM pgcolumnar.zone_map zm + WHERE zm.storage_id = rg.storage_id + AND zm.group_number = rg.group_number + AND zm.vector_index >= 0), + rg.byte_length + FROM pgcolumnar.row_group rg + WHERE rg.storage_id = pgcolumnar.get_storage_id(rel) + ORDER BY 1; +END; +$stats$; + +COMMENT ON FUNCTION pgcolumnar.stats(regclass) + IS 'per-row-group statistics for a columnar table'; + +/* + * How much of an ordered layout is still ordered (#301). + * + * pgcolumnar.vacuum_sorted and pgcolumnar.cluster order the whole relation once. + * They do not keep it ordered: rows inserted later append in insert order, so + * the ordered run stays at the front and an unsorted tail grows behind it. This + * reports the size of each part, so a DBA can decide when a re-sort is worth its + * cost instead of guessing. + * + * The ordered run is every row group numbered within the range the rewrite left + * in pgcolumnar.storage: from sorted_from to sorted_through inclusive. Groups + * above it were written later. Groups below it belong to a writer that started + * before the rewrite did and so were never ordered by it (#342); recording only + * an upper bound counted those as ordered. + * + * The row counts are stored rows. Rows deleted but not yet reclaimed are still + * stored, so they are still counted. pgcolumnar.stats reports the deleted count + * per group for callers that need to subtract it. + * + * Limits to read before acting on the numbers: + * + * 1. The online pgcolumnar.recluster sets the mark only for the part of its + * output it can prove is one contiguous ordered run. It reorders under a lock + * that permits concurrent inserts, and a boundary can only mean "everything + * at or below this is ordered" if no other session's group is numbered below + * it. With no concurrent writer it records the whole relation. With one, it + * records the run up to the point the other session interrupted, which can be + * a small part of what it ordered, and reports the rest as decay. It errs + * toward reporting too much decay, never too little. + * + * 2. The mark says where an ordered run ended, not that the rows in it are still + * in that order. Nothing in the design can move a stored row, so the run + * holds its order; but an UPDATE writes the new row version at the end, which + * counts as appended, and the old version stays in the run until it is + * reclaimed. + * + * A relation that was never ordered reports no sorted groups, because a rewrite + * always creates a new storage row and only an ordering rewrite sets the mark on + * it. A relation with nothing written reports zeros. + */ +CREATE FUNCTION pgcolumnar.sort_status( + rel regclass, + OUT sort_key name[], + OUT sorted_kind text, + OUT total_groups bigint, + OUT sorted_groups bigint, + OUT appended_groups bigint, + OUT sorted_rows bigint, + OUT appended_rows bigint) + RETURNS record + -- SECURITY DEFINER, like stats() (#560): the body reads pgcolumnar's internal + -- catalogs (storage, row_group, options), which carry no GRANT, so an + -- invoker-rights function false-denied a table's own owner on their own table + -- (#608). require_caller_select gates the REAL caller via GetOuterUserId(), so + -- definer rights do not widen who may read a table's sort status. search_path + -- is pinned as a definer function must. + LANGUAGE plpgsql STABLE SECURITY DEFINER + SET search_path = pg_catalog, pg_temp + AS $sort_status$ +BEGIN + PERFORM pgcolumnar.require_caller_select(rel); + WITH s AS ( + SELECT st.storage_id, st.sorted_through, st.sorted_from + FROM pgcolumnar.storage st + WHERE st.storage_id = pgcolumnar.get_storage_id(rel) + ), + g AS ( + -- A NULL mark means the storage was never ordered, so no group is in the + -- run. Comparing against NULL would make every count NULL instead. + -- + -- The run is a range, not everything below a boundary (#342). A group + -- numbered below sorted_from was not written by the rewrite that set the + -- mark: its stripe id was drawn before the rewrite's first, so it is a + -- concurrent writer's group and is not ordered. sorted_from is NULL only + -- for a mark written before this column existed, where the old + -- everything-below reading is kept. + SELECT rg.row_count, + (s.sorted_through IS NOT NULL + AND rg.group_number <= s.sorted_through + AND (s.sorted_from IS NULL + OR rg.group_number >= s.sorted_from)) AS in_run + FROM pgcolumnar.row_group rg + JOIN s ON rg.storage_id = s.storage_id + ) + -- sort_key reports the ACTUAL clustering recorded by the last recluster + -- (#415, storage.sorted_by), falling back to the declared options.sort_by + -- when nothing has been reclustered yet. Before #415 this read only the + -- declared key, so it was NULL on a table clustered but never declared. + SELECT COALESCE( + (SELECT st.sorted_by FROM pgcolumnar.storage st + WHERE st.storage_id = pgcolumnar.get_storage_id(rel)), + (SELECT o.sort_by FROM pgcolumnar.options o WHERE o.regclass = rel)), + -- HOW that key is applied (#761). sort_key names the columns and says + -- nothing about whether they are sorted or laid on a Z-order curve, + -- and a Z-order over two or more columns is not a sort on any one of + -- them. The catalog has carried this since #758; pgcolumnar.storage + -- has no GRANT and is superuser-only, so a table's own owner could + -- read it nowhere. NULL when the storage was never ordered, or was + -- ordered before the column existed. + (SELECT st.sorted_kind FROM pgcolumnar.storage st + WHERE st.storage_id = pgcolumnar.get_storage_id(rel)), + (SELECT count(*)::bigint FROM g), + (SELECT count(*)::bigint FROM g WHERE g.in_run), + (SELECT count(*)::bigint FROM g WHERE NOT g.in_run), + COALESCE((SELECT sum(g.row_count)::bigint FROM g WHERE g.in_run), 0::bigint), + COALESCE((SELECT sum(g.row_count)::bigint FROM g WHERE NOT g.in_run), 0::bigint) + INTO sort_key, sorted_kind, total_groups, sorted_groups, appended_groups, + sorted_rows, appended_rows; +END; +$sort_status$; + +COMMENT ON FUNCTION pgcolumnar.sort_status(regclass) + IS 'how much of an ordered columnar table is still in its ordered run, and by what kind of ordering (#301, #761)'; + +CREATE FUNCTION pgcolumnar.expire(tablename regclass) + RETURNS bigint + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_expire'; + +COMMENT ON FUNCTION pgcolumnar.expire(regclass) + IS 'drop row groups whose rows are all older than the retention declared by set_options(ttl_column, ttl_interval), without rewriting them (#403)'; + +CREATE FUNCTION pgcolumnar.vacuum(tablename regclass, stripe_count int DEFAULT 0) + RETURNS void + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_vacuum'; + +COMMENT ON FUNCTION pgcolumnar.vacuum(regclass, int) + IS 'compact a columnar table by combining stripes and reclaiming deleted rows'; + +CREATE FUNCTION pgcolumnar.vacuum_sorted( + tablename regclass, + VARIADIC sort_columns name[]) + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_vacuum_sorted'; + +COMMENT ON FUNCTION pgcolumnar.vacuum_sorted(regclass, name[]) + IS 'compact a columnar table, storing rows sorted ascending (NULLS LAST) on the given columns. With no columns, applies the table''s declared sort_by key from set_options (#288), like a bare CLUSTER re-applying a remembered index; errors if none is declared. Supports any btree-orderable column including text and numeric, unlike Z-order cluster(), which takes integer, date/time, boolean and floating-point columns only. One-shot: not auto-maintained.'; + +/* + * One-argument form: apply the declared sort_by key (#288). A VARIADIC function + * cannot be called cleanly with zero variadic arguments from an unknown literal + * (vacuum_sorted('t') would not resolve), so this explicit overload gives a + * clean bare-table call. It shares the C entry point, which uses PG_NARGS() to + * detect the missing column list and fall back to the persisted key. + */ +CREATE FUNCTION pgcolumnar.vacuum_sorted(tablename regclass) + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_vacuum_sorted'; + +COMMENT ON FUNCTION pgcolumnar.vacuum_sorted(regclass) + IS 'apply the table''s declared sort_by key from set_options (#288); errors if none is declared. Equivalent to a bare CLUSTER re-applying a remembered index.'; + +CREATE FUNCTION pgcolumnar.cluster( + tablename regclass, + VARIADIC columns name[]) + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_cluster'; + +COMMENT ON FUNCTION pgcolumnar.cluster(regclass, name[]) + IS 'eager reorg: rewrite a columnar table with rows ordered by the Z-order space-filling curve over the given columns. Holds AccessExclusiveLock like CLUSTER/VACUUM FULL; the online incremental path is Phase F3'; + +CREATE FUNCTION pgcolumnar.cluster_hilbert( + tablename regclass, + VARIADIC columns name[]) + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_cluster_hilbert'; + +COMMENT ON FUNCTION pgcolumnar.cluster_hilbert(regclass, name[]) + IS 'eager reorg on the Hilbert curve: as cluster(), but the rows are ordered by the Hilbert index over the given columns, which keeps neighbouring keys neighbouring in storage more tightly than Z-order does. Holds AccessExclusiveLock like CLUSTER/VACUUM FULL; the online counterpart is recluster_hilbert() (#889)'; + +CREATE FUNCTION pgcolumnar.compact(tablename regclass) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_compact'; + +COMMENT ON FUNCTION pgcolumnar.compact(regclass) + IS 'lazy online compaction: retire row groups that are fully deleted, dropping their metadata so scans skip them. Holds only ShareUpdateExclusiveLock (concurrent reads and writes). Returns the number of groups retired (Phase F3a)'; + +CREATE FUNCTION pgcolumnar.truncate(tablename regclass) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_truncate'; + +COMMENT ON FUNCTION pgcolumnar.truncate(regclass) + IS 'physical end-truncation: return trailing reclaimed blocks to the OS. Best-effort -- takes AccessExclusiveLock conditionally for the brief physical step and returns 0 without waiting if the table is busy. Only removes space freed before the oldest-xmin horizon. Gated by pgcolumnar.enable_end_truncation. Returns the number of blocks truncated (Phase F)'; + +CREATE FUNCTION pgcolumnar.compact_rewrite( + tablename regclass, + min_deleted_fraction float8 DEFAULT 0.2, + max_groups int DEFAULT 0) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_compact_rewrite'; + +COMMENT ON FUNCTION pgcolumnar.compact_rewrite(regclass, float8, int) + IS 'lazy online space reclaim: rewrite partially-deleted row groups (deleted fraction >= min_deleted_fraction) to drop their dead rows, under ShareUpdateExclusiveLock (concurrent reads and writes). Returns the number of groups rewritten (Phase F3b)'; + +CREATE FUNCTION pgcolumnar.recluster( + tablename regclass, + VARIADIC columns name[]) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_recluster'; + +COMMENT ON FUNCTION pgcolumnar.recluster(regclass, name[]) + IS 'lazy online reclustering: re-establish global Z-order clustering over the given columns under ShareUpdateExclusiveLock (concurrent reads and writes), unlike the eager cluster() which holds AccessExclusiveLock. Returns the number of groups reclustered (Phase F3c)'; + +CREATE FUNCTION pgcolumnar.recluster_hilbert( + tablename regclass, + VARIADIC columns name[]) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_recluster_hilbert'; + +COMMENT ON FUNCTION pgcolumnar.recluster_hilbert(regclass, name[]) + IS 'lazy online reclustering on the Hilbert curve: as recluster(), but re-establishes Hilbert clustering over the given columns under ShareUpdateExclusiveLock (concurrent reads and writes). The curve is sticky -- plain recluster() maintains a Hilbert table rather than converting it, and naming this verb is how a Z-ordered table is switched (#889)'; + +CREATE FUNCTION pgcolumnar.export_arrow(rel regclass, path text) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_export_arrow'; + +COMMENT ON FUNCTION pgcolumnar.export_arrow(regclass, text) + IS 'export a columnar table to an Arrow IPC stream file; returns rows written'; + +CREATE FUNCTION pgcolumnar.export_parquet(rel regclass, path text) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_export_parquet'; + +COMMENT ON FUNCTION pgcolumnar.export_parquet(regclass, text) + IS 'export a columnar table to a Parquet file; returns rows written'; + +CREATE FUNCTION pgcolumnar.parallel_export_parquet(target regclass, path text, + workers int DEFAULT NULL) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_parallel_export_parquet'; + +COMMENT ON FUNCTION pgcolumnar.parallel_export_parquet(regclass, text, int) + IS 'parallel Parquet export using read-only background workers into a directory readable by pgcolumnar.read_parquet: a single columnar table split by row-group ranges, or a partitioned columnar table one file per partition; returns rows written (#300)'; + +CREATE FUNCTION pgcolumnar.import_arrow(rel regclass, path text) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_import_arrow'; + +COMMENT ON FUNCTION pgcolumnar.import_arrow(regclass, text) + IS 'insert rows from an Arrow IPC stream file into a columnar table; returns rows inserted'; + +CREATE FUNCTION pgcolumnar.import_parquet(rel regclass, path text) + RETURNS bigint + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_import_parquet'; + +COMMENT ON FUNCTION pgcolumnar.import_parquet(regclass, text) + IS 'insert rows from a Parquet file, directory, or glob into a table; returns rows inserted (gap 27)'; + +CREATE FUNCTION pgcolumnar.parquet_schema(path text) + RETURNS TABLE(column_name text, data_type text, nullable boolean, field_id integer) + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_parquet_schema'; + +COMMENT ON FUNCTION pgcolumnar.parquet_schema(text) + IS 'report the leaf columns of a Parquet file and the PostgreSQL type each maps to; for a directory or glob, of its first file; field_id is the SchemaElement field id Iceberg projects by, NULL when the writer emitted none (Phase G scan core, #388)'; + +CREATE FUNCTION pgcolumnar.read_parquet(path text) + RETURNS SETOF record + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_read_parquet'; + +COMMENT ON FUNCTION pgcolumnar.read_parquet(text) + IS 'read a Parquet file, directory, or glob in place as a set of rows; requires a column definition list covering every leaf column, e.g. SELECT * FROM pgcolumnar.read_parquet(path) AS t(id int, name text) (Phase G)'; + +CREATE FUNCTION pgcolumnar.read_parquet(path text, field_ids integer[]) + RETURNS SETOF record + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_read_parquet'; + +COMMENT ON FUNCTION pgcolumnar.read_parquet(text, integer[]) + IS 'read a Parquet file by field id: output column i is bound to the file column whose Parquet field id equals field_ids[i], reading only those columns in that order, e.g. SELECT * FROM pgcolumnar.read_parquet(path, ARRAY[12,7]) AS t(c int, a int) (#388)'; + +CREATE FUNCTION pgcolumnar.read_avro_manifest(path text) + RETURNS TABLE(status integer, content integer, file_path text, + file_format text, record_count bigint, + file_size_in_bytes bigint, partition text, + sequence_number bigint) + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_read_avro_manifest'; + +COMMENT ON FUNCTION pgcolumnar.read_avro_manifest(text) + IS 'decode an Apache Iceberg Avro manifest file and report its data-file entries; the first step of Iceberg read support (#388)'; + +CREATE FUNCTION pgcolumnar.read_manifest_list(path text) + RETURNS TABLE(manifest_path text, manifest_length bigint, content integer, + partition_spec_id integer, added_files_count integer, + existing_files_count integer, deleted_files_count integer, + added_rows_count bigint, existing_rows_count bigint, + deleted_rows_count bigint, sequence_number bigint, + min_sequence_number bigint, added_snapshot_id bigint) + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_read_manifest_list'; + +COMMENT ON FUNCTION pgcolumnar.read_manifest_list(text) + IS 'decode an Apache Iceberg snapshot manifest-list Avro file and report the manifest files it points at (#388)'; + +CREATE FUNCTION pgcolumnar.iceberg_current_snapshot(metadata_path text) + RETURNS TABLE(snapshot_id bigint, parent_snapshot_id bigint, + sequence_number bigint, timestamp_ms bigint, operation text, + manifest_list text, schema_id integer) + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_current_snapshot'; + +COMMENT ON FUNCTION pgcolumnar.iceberg_current_snapshot(text) + IS 'read an Apache Iceberg table metadata.json and report its current snapshot (#388)'; + +CREATE FUNCTION pgcolumnar.iceberg_data_files(metadata_path text) + RETURNS TABLE(file_path text, file_format text, record_count bigint, + partition text) + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_data_files'; + +COMMENT ON FUNCTION pgcolumnar.iceberg_data_files(text) + IS 'list the live data files of an Apache Iceberg table current snapshot; refuses tables with delete files (#388)'; + +CREATE FUNCTION pgcolumnar.iceberg_scan(metadata_path text) + RETURNS SETOF record + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_scan'; + +COMMENT ON FUNCTION pgcolumnar.iceberg_scan(text) + IS 'read an Apache Iceberg table at its current snapshot; supply a column definition list, whose names resolve to the table schema field ids, e.g. SELECT * FROM pgcolumnar.iceberg_scan(path) AS t(id bigint, region text); applies position, equality, and deletion-vector deletes (#388)'; + +CREATE FUNCTION pgcolumnar.iceberg_rest_table_location(catalog_uri text, + namespace text, + table_name text) + RETURNS text + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_rest_table_location'; + +COMMENT ON FUNCTION pgcolumnar.iceberg_rest_table_location(text, text, text) + IS 'resolve the current metadata-location of a table named by an Iceberg REST catalog (catalog URI + namespace + table); the bearer token is read from the server environment variable PGCOLUMNAR_ICEBERG_REST_TOKEN, never a SQL argument (#388)'; + +CREATE FUNCTION pgcolumnar.iceberg_rest_scan(catalog_uri text, + namespace text, + table_name text) + RETURNS SETOF record + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_rest_scan'; + +COMMENT ON FUNCTION pgcolumnar.iceberg_rest_scan(text, text, text) + IS 'read a table named by an Iceberg REST catalog at its current snapshot; supply a column definition list, as for iceberg_scan; the metadata location is resolved through the catalog and read like any other Iceberg table (#388)'; + +CREATE FUNCTION pgcolumnar.iceberg_rest_namespaces(catalog_uri text) + RETURNS SETOF text + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_rest_namespaces'; + +COMMENT ON FUNCTION pgcolumnar.iceberg_rest_namespaces(text) + IS 'list the namespaces of an Iceberg REST catalog, one per row, multi-level namespaces dot-joined (#388)'; + +CREATE FUNCTION pgcolumnar.iceberg_rest_tables(catalog_uri text, namespace text) + RETURNS SETOF text + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_rest_tables'; + +COMMENT ON FUNCTION pgcolumnar.iceberg_rest_tables(text, text) + IS 'list the table names in a namespace of an Iceberg REST catalog, one per row (#388)'; + +/* --------------------------------------------------------------------------- + * Parquet foreign-data wrapper (Phase G) + * + * A foreign table over a Parquet file, a directory of *.parquet files, or a glob + * pattern, read as one relation; its column definitions are bound against every + * file by position, like read_parquet's column list. Usage: + * CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet; + * CREATE FOREIGN TABLE ft (id int, name text) SERVER pq + * OPTIONS (path '/data/f.parquet'); + * ------------------------------------------------------------------------- */ + +CREATE FUNCTION pgcolumnar.parquet_fdw_handler() + RETURNS fdw_handler + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_parquet_fdw_handler'; + +CREATE FUNCTION pgcolumnar.parquet_fdw_validator(text[], oid) + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_parquet_fdw_validator'; + +CREATE FOREIGN DATA WRAPPER pgcolumnar_parquet + HANDLER pgcolumnar.parquet_fdw_handler + VALIDATOR pgcolumnar.parquet_fdw_validator; + +COMMENT ON FOREIGN DATA WRAPPER pgcolumnar_parquet + IS 'read a Parquet file, directory, or glob as a foreign table; table option: path (Phase G)'; + +CREATE FUNCTION pgcolumnar.iceberg_fdw_handler() + RETURNS fdw_handler + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_fdw_handler'; + +CREATE FUNCTION pgcolumnar.iceberg_fdw_validator(text[], oid) + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_fdw_validator'; + +CREATE FOREIGN DATA WRAPPER pgcolumnar_iceberg + HANDLER pgcolumnar.iceberg_fdw_handler + VALIDATOR pgcolumnar.iceberg_fdw_validator; + +COMMENT ON FOREIGN DATA WRAPPER pgcolumnar_iceberg + IS 'read an Apache Iceberg table as a foreign table, pruning data files by a predicate on an identity-partition column; table option: metadata_path (#388)'; + +CREATE FUNCTION pgcolumnar.iceberg_catalog_fdw_validator(text[], oid) + RETURNS void + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_iceberg_catalog_validator'; + +-- Validator-only wrapper (no HANDLER): a REST catalog creates no foreign tables. +-- A SERVER under it holds catalog_uri; a USER MAPPING holds the per-role token. +CREATE FOREIGN DATA WRAPPER pgcolumnar_iceberg_catalog + VALIDATOR pgcolumnar.iceberg_catalog_fdw_validator; + +COMMENT ON FOREIGN DATA WRAPPER pgcolumnar_iceberg_catalog + IS 'name an Iceberg REST catalog: a SERVER holds catalog_uri, a USER MAPPING holds the bearer token; the iceberg_rest_* functions accept a server name in place of a catalog URI (#656)'; + +CREATE FUNCTION pgcolumnar.vm_selftest(rel regclass, blk int) + RETURNS boolean + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_vm_selftest'; + +COMMENT ON FUNCTION pgcolumnar.vm_selftest(regclass, int) + IS 'gap 28 phase-1 self-test: set a VM-fork all-visible bit and read it back'; + +CREATE FUNCTION pgcolumnar.vm_is_visible(rel regclass, blk int) + RETURNS boolean + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_vm_is_visible'; + +COMMENT ON FUNCTION pgcolumnar.vm_is_visible(regclass, int) + IS 'gap 28: is the synthetic block marked all-visible in the VM fork?'; + +-- These two are phase-1 self-test helpers, not user API. CREATE FUNCTION grants +-- EXECUTE to PUBLIC, which put a visibility-map write behind nothing but USAGE on +-- this schema -- and the documented maintenance API lives in the same schema, so +-- any deployment exposing that also exposed these (#558). The C code checks +-- ownership as well; this REVOKE is the second layer, not the only one. +REVOKE ALL ON FUNCTION pgcolumnar.vm_selftest(regclass, int) FROM PUBLIC; +REVOKE ALL ON FUNCTION pgcolumnar.vm_is_visible(regclass, int) FROM PUBLIC; + +CREATE FUNCTION pgcolumnar.vacuum_full( + schema name DEFAULT 'public', + sleep_time real DEFAULT 0.0, + stripe_count int DEFAULT 0) + RETURNS void + LANGUAGE plpgsql + AS $vacuum_full$ +DECLARE + r record; +BEGIN + FOR r IN + SELECT c.oid AS reloid + FROM pg_class c + JOIN pg_am a ON a.oid = c.relam + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE a.amname = 'pgcolumnar' + AND c.relkind = 'r' + AND n.nspname = vacuum_full.schema + LOOP + PERFORM pgcolumnar.vacuum(r.reloid::regclass, stripe_count); + IF sleep_time > 0 THEN + PERFORM pg_sleep(sleep_time); + END IF; + END LOOP; +END; +$vacuum_full$; + +COMMENT ON FUNCTION pgcolumnar.vacuum_full(name, real, int) + IS 'compact every columnar table in a schema'; + +-- --------------------------------------------------------------------------- +-- Parallel bulk ingest (#300). Phase 1: the file range splitter. Given a +-- server-side file and a worker count, return workers+1 ascending byte offsets +-- that partition the file into that many line-aligned ranges, so a parallel load +-- can hand range [off[i], off[i+1]) to worker i. The ranges are record-aligned +-- for COPY *text* format only (a raw newline always ends a text record); they are +-- NOT safe for CSV, whose quoted fields may contain literal newlines. `workers` is +-- capped internally so a huge value cannot allocate unbounded memory. +-- --------------------------------------------------------------------------- +CREATE FUNCTION pgcolumnar.file_split_offsets(path text, workers int) + RETURNS bigint[] + LANGUAGE C STRICT + AS 'MODULE_PATHNAME', 'pgcolumnar_file_split_offsets'; + +COMMENT ON FUNCTION pgcolumnar.file_split_offsets(text, int) + IS 'byte offsets that split a COPY text-format file into N record-aligned ranges (#300)'; + +-- Parallel bulk ingest: atomically load a server-side COPY text-format file into a +-- RANGE-partitioned columnar table across N background workers. Each worker loads a +-- DISTINCT set of partitions (distinct storage), the only shape pgColumnar allows a +-- parallel AND atomic bulk load: concurrent writers to one non-partitioned table +-- serialize on the per-storage write lock and, under two-phase commit, deadlock +-- (single-table parallel load is a planned columnar-core enhancement). Loaders +-- PREPARE; a coordinator background worker COMMIT PREPAREDs them all, or ROLLBACK +-- PREPAREDs on any failure. Returns rows loaded. The target is either a single +-- columnar table (workers write its one storage concurrently) or a RANGE-partitioned +-- table (each worker loads a distinct partition; requires a single-column +-- numeric/date-time key, no DEFAULT partition, and the file sorted ascending by that +-- key). COPY text format, and max_prepared_transactions >= workers. workers => NULL +-- derives a default from max_parallel_workers. +-- +-- Two behaviors to know: (1) the load commits in background workers, INDEPENDENTLY +-- of the calling transaction, so its rows survive a subsequent ROLLBACK of the +-- caller -- treat the call like a COMMIT. (2) Do not call it while the calling +-- transaction holds a lock on the target (e.g. after LOCK TABLE or a write to it): +-- the loaders would block on that lock and the wait is invisible to the deadlock +-- detector. See design/PARALLEL_COPY_PLAN.md. +CREATE FUNCTION pgcolumnar.parallel_copy(target regclass, filename text, + workers int DEFAULT NULL, + dedup boolean DEFAULT false) + RETURNS bigint + LANGUAGE C + AS 'MODULE_PATHNAME', 'pgcolumnar_parallel_copy'; + +COMMENT ON FUNCTION pgcolumnar.parallel_copy(regclass, text, int, boolean) + IS 'atomic parallel bulk load of a COPY text file into a columnar table using background workers: a single columnar table (any row order), or a RANGE-partitioned columnar table sorted by the partition key with one distinct partition set per worker (#300). With dedup, a file already loaded into this table is refused rather than loaded twice (#403)'; + +/* + * Per-column statistics without reading the whole table (#414). + * + * Core ANALYZE decodes essentially the entire table. It samples a fixed 30,000 + * rows, and on a table of any size those rows fall in every row group, so every + * group is decoded for every column. Measured on 3M rows x 20 columns, 1237 MB, + * serial: ANALYZE costs 6,302 ms against 7,680 ms to decode all nineteen text + * columns outright, while decoding just one column costs 268 ms. + * + * That cannot be recovered inside the table-AM callbacks, which is why this is a + * function. acquire_sample_rows copies whole tuples (ExecCopySlotHeapTuple), so + * the AM cannot decline to produce columns core is about to copy: ANALYZE of one + * named column costs 6,073 ms against 6,302 ms for all twenty, a 6% saving. Nor + * is there slack in which groups the sample touches -- at a tenth the + * chunk_group_row_limit the cost was unchanged, because a fixed-size sample + * touches proportionally more groups when they are smaller. + * + * Core ANALYZE remains the correctness path and is what autovacuum runs. This is + * an opt-in accelerator for wide tables and, like pgcolumnar.vacuum(), nothing + * schedules it: see #415. + * + * Collected so far, all of it exact rather than sampled, and all of it from ONE + * read of the column: null_frac, n_distinct, the most-common values with their + * frequencies, and a histogram of what remains once those are excluded. + * + * One read is the property that matters, not merely the source of each number. + * null_frac came from the zone maps until #485, which was cheaper and was wrong + * after a DELETE, because those counts describe what was written. Taking it from + * the same read as the rest is what makes every statistic here describe one + * population, which is the identity the planner's selectivity arithmetic needs. + * + * "Exact" is the whole difference and it is not a refinement of core's numbers. + * Core samples 30,000 rows, so a value held by one row in 500,000 is missed + * entirely and every range estimate above the sampled maximum collapses; a + * frequency is right to about three digits rather than exactly. Reading the + * column removes the sampling error rather than reducing it -- which is also why + * core's own significance filter for the most-common list does not apply here, + * as analyze_mcv_list() says itself at analyze.c:2995. + */ +CREATE FUNCTION pgcolumnar.analyze(rel regclass, columns text[] DEFAULT NULL) + RETURNS void + LANGUAGE plpgsql + AS $$ +DECLARE + sid bigint; + att record; + nullfrac double precision; + ndistinct bigint; + totalrows bigint; + ndstat double precision; + hist text; + mcvvals text; + mcvfreqs real[]; + orderable boolean; + nmcv integer; + nremaining bigint; + nullcount bigint; /* live rows with no value, from the same read */ + nonnull bigint; /* rows with a value, from the aggregation below */ + mcvrows bigint; /* of those, the rows the MCV list holds */ + nv bigint; /* the population the histogram is placed over */ + nfrac integer; + -- The per-column target, resolved inside the loop. attstattarget is NULL when + -- the column has never been given one, and core reads that as "use the global + -- default" (analyze.c:1065 with :1897). A zero means do not collect at all. + deftarget integer := current_setting('default_statistics_target')::integer; + nbuckets integer; + seen integer := 0; + disabled integer := 0; + unknown text; + schname text; + relnm text; +BEGIN + /* + * Writing statistics uses pg_restore_attribute_stats, which core added in + * 18. On 15 to 17 this would mean writing pg_statistic directly, and the + * risk there is in the values rather than the insert: stavalues is anyarray + * and must carry the column's element type, typmod and collation; staop must + * be the right operator for the stakind; stadistinct has a sign convention + * that is easy to invert. Each of those produces plausible wrong estimates + * rather than an error. Refuse clearly instead of failing obscurely inside + * the call below. + */ + IF current_setting('server_version_num')::int < 180000 THEN + RAISE EXCEPTION 'pgcolumnar.analyze() requires PostgreSQL 18 or later' + USING DETAIL = 'it writes statistics through pg_restore_attribute_stats, which older majors do not have', + HINT = 'use ANALYZE on this server'; + END IF; + + /* + * pg_restore_attribute_stats identifies the column by schema and relation + * NAME, not by regclass, and rejects a null schemaname. Resolve both from the + * oid once rather than per column. + */ + SELECT n.nspname, c.relname INTO schname, relnm + FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE c.oid = rel; + + SELECT s.storage_id INTO sid + FROM pgcolumnar.storage s + WHERE s.relation_oid = rel; + + IF sid IS NULL THEN + RAISE EXCEPTION 'pgcolumnar.analyze(): % has no columnar storage', rel::text + USING HINT = 'this function only applies to pgcolumnar tables that have been written to'; + END IF; + + /* + * A named column that does not exist is a caller error, not a no-op. Silently + * collecting nothing is the failure mode that looks exactly like success. + */ + IF columns IS NOT NULL THEN + SELECT c INTO unknown + FROM unnest(columns) AS c + WHERE NOT EXISTS ( + SELECT 1 FROM pg_attribute a + WHERE a.attrelid = rel AND a.attname = c + AND a.attnum > 0 AND NOT a.attisdropped) + LIMIT 1; + IF unknown IS NOT NULL THEN + RAISE EXCEPTION 'pgcolumnar.analyze(): column "%" does not exist in %', + unknown, rel::text; + END IF; + END IF; + + FOR att IN + SELECT a.attname, a.attnum, a.atttypid, a.attstattarget + FROM pg_attribute a + WHERE a.attrelid = rel AND a.attnum > 0 AND NOT a.attisdropped + AND (columns IS NULL OR a.attname = ANY (columns)) + ORDER BY a.attnum + LOOP + /* + * The per-column statistics target, which is core's rule and not the + * global setting: + * + * attstattarget = isnull ? -1 : DatumGetInt16(dat); analyze.c:1065 + * if (attstattarget == 0) return NULL; :1070 + * if (stats->attstattarget < 0) :1897 + * stats->attstattarget = default_statistics_target; + * + * Zero means the DBA turned this column off, and honouring it is not + * optional: writing statistics for such a column overrides an explicit + * instruction and hands the planner numbers somebody disabled. Reading + * the global default for every column, as this function did, ignored + * ALTER TABLE ... SET STATISTICS entirely. + */ + IF att.attstattarget = 0 THEN + disabled := disabled + 1; + CONTINUE; + END IF; + nbuckets := coalesce(att.attstattarget, deftarget); + /* + * Has this column been written yet? The zone maps answer that and + * nothing else here. + * + * They used to answer null_frac as well -- + * sum(null_count) / sum(value_count + null_count) -- and that was wrong + * after a DELETE. Those counts describe what was WRITTEN; deleting a row + * marks it dead without rewriting them, so the denominator keeps counting + * rows the table no longer holds. On 1,000 rows with 100 nulls, deleting + * the 301 rows holding one value leaves a true null_frac of 0.1431 and a + * zone-map null_frac of 0.1000, a 30% understatement that VACUUM does not + * heal. Worse than the size of the error: null_frac came from the zone + * maps while the most-common-value frequencies came from count(*), so the + * two were normalised against different populations and + * null_frac + sum(mcv_freqs) + rest = 1 -- the identity the planner's + * selectivity arithmetic rests on -- silently stopped holding. + * + * So the fraction is taken from the same read as everything else below, + * and the zone maps keep only the job they can still do exactly: telling + * us whether there are any row groups at all. + * + * column_index is the 0-based attribute position. attnum is stable + * across a dropped column, so attnum - 1 keeps pointing at the same + * column after a DROP COLUMN. + */ + PERFORM 1 + FROM pgcolumnar.zone_map z + WHERE z.storage_id = sid + AND z.column_index = att.attnum - 1 + AND z.vector_index = -1; + + CONTINUE WHEN NOT FOUND; /* no zone map rows: nothing exact to say */ + + /* + * n_distinct, the row count and the null count, by reading this column + * and nothing else. This is the whole point of the function: on the + * 3M x 20 fixture a projected single-column read costs 268 ms where + * core's whole-table sample costs 6,302 ms, because core's fixed + * 30,000-row sample lands in every row group and so decodes every column + * of the table. + * + * count(DISTINCT) ignores NULLs, which is what n_distinct means. The + * null count comes from the same scan so that it cannot disagree with the + * denominator the frequencies below are divided by. + */ + EXECUTE format('SELECT count(DISTINCT %I)::bigint, count(*)::bigint,' + ' count(*) FILTER (WHERE %I IS NULL)::bigint' + ' FROM %I.%I', + att.attname, att.attname, schname, relnm) + INTO ndistinct, totalrows, nullcount; + + nullfrac := CASE WHEN totalrows > 0 + THEN nullcount::double precision / totalrows::double precision + ELSE 0 END; + + /* + * Core's own convention, and the sign is load-bearing: positive is an + * absolute count, negative is the negated fraction of rows. analyze.c + * switches to the fraction once the distinct count passes 10% of the + * rows, on the grounds that such a column's cardinality tracks the table + * size rather than sitting at a fixed value. Mirror it rather than always + * writing the absolute count, or a column that is unique today reads as + * having a fixed cardinality once the table grows. + * + * Getting this backwards does not raise -- it produces plausible wrong + * estimates -- so it is asserted in test/analyze_function.sh against a + * fixture pinned to the absolute-count side of the rule. + */ + IF totalrows > 0 THEN + IF ndistinct::double precision > 0.1 * totalrows::double precision THEN + ndstat := -(ndistinct::double precision / totalrows::double precision); + ELSE + ndstat := ndistinct::double precision; + END IF; + ELSE + ndstat := 0; + END IF; + + /* + * Whether this type can be ordered at all. Hoisted out of the histogram + * test below because the most-common-value list needs the same answer: + * both order by the column, and a type with no btree opclass has no + * histogram in core either. + */ + orderable := EXISTS (SELECT 1 FROM pg_catalog.pg_type t + JOIN pg_catalog.pg_opclass oc ON oc.opcintype = t.oid + JOIN pg_catalog.pg_am am ON am.oid = oc.opcmethod + WHERE t.oid = att.atttypid AND am.amname = 'btree'); + + /* + * most_common_vals and most_common_freqs (#414 slice 3b). + * + * The selection rule is core's, and reading a complete column removes + * most of it. analyze_mcv_list() opens with + * + * if (samplerows == totalrows || totalrows <= 1.0) + * return num_mcv; -- analyze.c:2995 + * + * so the entire significance filter -- a continuity-corrected Wald + * interval over a hypergeometric variance -- is skipped when the whole + * table was read. That machinery exists to judge whether a SAMPLE + * frequency can be trusted; we do not sample, so the question does not + * arise and core's own answer is to keep the list. What remains: + * + * only values appearing more than once are eligible analyze.c:2549 + * the top default_statistics_target of those, by count analyze.c:2552 + * frequency = count / TOTAL rows, nulls included analyze.c:2720 + * + * That last one is the one that fails quietly. Dividing by the non-null + * count instead scales every frequency by 1/(1-null_frac): still ordered, + * still summing to less than one, still plausible, and wrong everywhere + * the column has nulls. test/analyze_function.sh pins it with a fixture + * that is one-tenth null, so the two denominators cannot agree. + * + * HAVING count(*) > 1 also reproduces core's unique-column case without a + * branch: when nothing repeats the aggregate is empty, array_agg returns + * NULL, and no MCV list is written -- which is what core does at + * analyze.c:2588 when nmultiple is zero. + * + * array_agg(...)::text rather than string_agg builds the array literal + * through the type's own output function, so quoting, embedded commas and + * braces are correct for text columns instead of being hand-assembled. + */ + mcvvals := NULL; + mcvfreqs := NULL; + nonnull := 0; + mcvrows := 0; + IF orderable THEN + /* + * The same aggregation, split into the full group and the most-common + * slice of it, so it can also report how many ROWS each covers. The + * histogram below is built over the non-null rows the MCV list does + * NOT hold, and it has to know how many those are to place a bound at + * a position rather than at a fraction. + * + * Both counts come from this one aggregation rather than from the zone + * maps or a second scan, so the population the histogram is placed + * over is by construction the population the MCV list was taken from. + */ + EXECUTE format( + 'WITH g AS MATERIALIZED (' + ' SELECT %I AS v, count(*)::bigint AS c' + ' FROM %I.%I WHERE %I IS NOT NULL GROUP BY 1),' + ' m AS MATERIALIZED (' + ' SELECT v, c FROM g WHERE c > 1 ORDER BY c DESC, v LIMIT %s)' + 'SELECT (SELECT array_agg(v ORDER BY c DESC, v)::text FROM m),' + ' (SELECT array_agg((c::double precision / %s::double precision)::real' + ' ORDER BY c DESC, v) FROM m),' + ' (SELECT coalesce(sum(c), 0)::bigint FROM g),' + ' (SELECT coalesce(sum(c), 0)::bigint FROM m)', + att.attname, schname, relnm, att.attname, nbuckets, totalrows) + INTO mcvvals, mcvfreqs, nonnull, mcvrows; + END IF; + + /* + * histogram_bounds, whose ends are exact because the read is complete + * (#414 slice 3). + * + * percentile_disc over an array of fractions returns ACTUAL column + * values, one per fraction, in a single ordered pass. Fraction 1.0 is + * therefore the true maximum and 0.0 the true minimum, which is the + * whole gain: core samples, so a value held by one row in 500,000 is + * missed and every range estimate above the sampled maximum collapses. + * percentile_cont would interpolate and invent values the column does + * not contain, which is wrong for a histogram of stored data and wrong + * for any non-numeric type. + * + * Only for types that can be ordered. A column with no btree ordering + * has no histogram in core either, and ORDER BY would simply fail. + * + * The most-common values are EXCLUDED, which core does at analyze.c:2744 + * and :2768-2799 by collapsing them out of the sorted array before + * building buckets. Keeping them in counts them twice in selectivity: + * eqsel takes the value's frequency from the MCV list, and the range + * estimators count it again inside whichever bucket holds it. Nothing + * raises -- the estimates are simply inflated for the values a skewed + * column repeats most, which is where estimates matter. + * + * The population and the bucket count therefore both shrink, and both + * have to. Core sizes the histogram from what is LEFT: + * + * num_hist = ndistinct - num_mcv; + * if (num_hist > num_bins) num_hist = num_bins + 1; + * if (num_hist >= 2) { ... } -- analyze.c:2744-2747 + * + * so it emits between 2 and num_bins+1 bounds and none at all below two. + * Asking percentile_disc for a fixed default_statistics_target+1 + * fractions regardless would repeat values once the remaining population + * is smaller than that -- a 150-distinct column with 100 most-common + * values has 50 left and would get 101 bounds, most of them duplicates. + * A histogram with repeated bounds describes buckets holding no rows, + * which is a shape core never emits. + */ + nmcv := coalesce(array_length(mcvfreqs, 1), 0); + nremaining := ndistinct - nmcv; + + nv := nonnull - mcvrows; + + hist := NULL; + IF att.attnum > 0 + AND orderable + AND nremaining >= 2 + AND nv > 1 + THEN + /* + * least(nbuckets, nremaining - 1) fractions, so the bound count is + * least(nbuckets + 1, nremaining): core's cap, reached from below. + */ + nfrac := least(nbuckets, nremaining - 1); + + /* + * A bound is a POSITION, not a quantile, and the difference is not + * academic. core's compute_scalar_stats places bound i at + * + * values[floor(i * (nvals - 1) / (num_hist - 1))] + * + * among the rows left after the most-common values are removed. + * percentile_disc resolves fraction p to index ceil(p * nv) - 1, which + * is a different index whenever frac(i*nv/nfrac) is small, and a + * different VALUE whenever that shift crosses a value boundary. On a + * column with many rows per distinct value the two agree and the + * distinction is invisible; on eleven distinct rows at a statistics + * target of 3 they disagree at the third bound, 8 against 7. + * + * So ask percentile_disc for the fractions that resolve to core's + * positions instead of for evenly spaced quantiles: + * + * p_i = (floor(i * (nv - 1) / nfrac) + 0.5) / nv + * + * The half is load-bearing rather than decorative. The exact boundary + * (T + 1)/nv is a double, and nv up to a few million leaves roughly + * 1e-9 of slack in p*nv; landing a hair above T+1 makes ceil() return + * T+2 and takes the NEXT value. Half a row of margin cannot be crossed + * by that error, and any p in (T/nv, (T+1)/nv] resolves to T. + * + * nv is the count from the aggregation above, not a derived figure: + * deriving it as totalrows minus a null_frac read off the zone maps + * would put a rounded float in a position index. + */ + + /* + * The exclusion is a literal list rather than a re-aggregation. The + * alternative -- recomputing the most-common set in a subquery -- is + * a third full pass over a column this function exists to read once, + * and it can disagree with the list actually written if the tie-break + * ever differs. format_type gives the element type without a typmod, + * which is what the array literal must be parsed against. + */ + EXECUTE format( + 'SELECT percentile_disc( + (SELECT array_agg(((floor(i::numeric * (%s - 1) / %s) + 0.5) + / %s)::double precision ORDER BY i) + FROM generate_series(0, %s) i)) + WITHIN GROUP (ORDER BY %I)::text + FROM %I.%I WHERE %I IS NOT NULL %s', + nv, nfrac, nv, nfrac, att.attname, schname, relnm, att.attname, + CASE WHEN mcvvals IS NULL THEN '' + ELSE format('AND %I <> ALL (%L::%s[])', att.attname, mcvvals, + format_type(att.atttypid, NULL)) + END) + INTO hist; + END IF; + + /* + * The casts are load-bearing. pg_restore_attribute_stats takes VARIADIC + * "any", so a mistyped argument is a WARNING and the value is dropped, + * not an error: attname must be text (attname is `name`) and null_frac + * must be real (the division yields double precision). Without these the + * call "succeeds" having stored nothing. + * + * histogram_bounds and most_common_vals are passed as text, which is what + * the function takes (attribute_stats.c:70,72): it parses each array + * literal against the column's own type. most_common_freqs is real[] + * (:71) -- a float8[] there is dropped with a WARNING, not an error. + * + * One call with typed NULLs rather than a branch per combination. A NULL + * argument is not written: each statistic is gated on PG_ARGISNULL + * (:162-163 for the MCV pair), so a typed NULL and an omitted argument + * mean the same thing. Four optional statistics would otherwise be + * sixteen call sites. The NULLs must still be TYPED -- an untyped NULL + * reaches VARIADIC "any" as `unknown` and is the mistyped-argument case + * these casts exist to avoid. + * + * most_common_vals and most_common_freqs are a pair: supplying one + * without the other is a WARNING and drops both (stats_check_arg_pair, + * :265). They are computed together above, so they are null together. + */ + PERFORM pg_catalog.pg_restore_attribute_stats( + 'schemaname', schname, + 'relname', relnm, + 'attname', att.attname::text, + 'inherited', false, + 'null_frac', nullfrac::real, + 'n_distinct', ndstat::real, + 'most_common_vals', mcvvals::text, + 'most_common_freqs', mcvfreqs::real[], + 'histogram_bounds', hist::text); + + seen := seen + 1; + END LOOP; + + /* + * Collecting nothing is an error only when nothing ASKED us not to. A column + * at SET STATISTICS 0 is an instruction, and core does not raise for + * `ANALYZE t (col)` when col is disabled -- it collects nothing and returns. + * Without the second term this guard turned that instruction into an error + * whose hint blamed missing row groups, which is a different fault entirely + * and would send somebody looking at the storage. + */ + IF seen = 0 AND disabled = 0 THEN + RAISE EXCEPTION 'pgcolumnar.analyze(): collected statistics for no columns of %', rel::text + USING HINT = 'the table may have no written row groups yet'; + END IF; +END; +$$; + +COMMENT ON FUNCTION pgcolumnar.analyze(regclass, text[]) + IS 'collect per-column statistics by reading one column at a time rather than sampling every column (#414); null_frac, n_distinct and the most-common frequencies all come from that read, so they describe one population (#485); core ANALYZE remains the correctness path and nothing schedules this, see #415'; + +-- pgcolumnar.maintenance_due (#415): report whether an online maintenance verb +-- is worth running on a columnar table, from table statistics alone. A pure +-- report -- it takes no lock and rewrites nothing. A cron job or an operator +-- consults it; a background worker, if one is ever built, is a thin consumer of +-- the same verdict (see #415). +-- +-- Thresholds are PARAMETERS, not GUCs: the pgcolumnar GUC prefix is reserved +-- (MarkGUCPrefixReserved), so an unregistered pgcolumnar.* GUC is rejected, and +-- a report is better configured at the call site than globally. The defaults are +-- measured on #415 -- compact_rewrite's overhead reaches ~10% of a query at a +-- deleted fraction of 0.2 (the knee), and clustering decay is already costly at +-- the smallest fraction measured, so recluster gates low at 0.05. +-- +-- recluster_due gates on the sort key EXISTING. sort_status() reports a +-- never-ordered table as entirely appended, because it has no sorted run; that +-- is not decay and there is no ordering to restore, so the sort-key guard +-- suppresses the recommendation there. +CREATE FUNCTION pgcolumnar.maintenance_due( + rel regclass, + compact_due_fraction float8 DEFAULT 0.2, + recluster_due_fraction float8 DEFAULT 0.05, + OUT total_rows bigint, + OUT deleted_rows bigint, + OUT deleted_fraction float8, + OUT sort_key name[], + OUT appended_groups bigint, + OUT appended_rows bigint, + OUT appended_fraction float8, + OUT compact_rewrite_due boolean, + OUT recluster_due boolean, + OUT recommendation text) + RETURNS record + -- SECURITY DEFINER, mirroring stats(): the report reads pgcolumnar's internal + -- catalogs through sort_status(), which ordinary roles cannot SELECT, so an + -- invoker-rights function false-denied every non-superuser caller -- the + -- cron/monitoring role this report is for. require_caller_select (inside + -- stats()) still gates the REAL caller via GetOuterUserId(), so definer rights + -- do not widen who may read a table's statistics. search_path is pinned as a + -- definer function must. + LANGUAGE plpgsql STABLE SECURITY DEFINER + SET search_path = pg_catalog, pg_temp + AS $maintenance_due$ +DECLARE + st_rows bigint; + st_del bigint; + ss record; +BEGIN + -- Validate both thresholds before reading anything (#860). Neither one was + -- checked, and this is the gate the autovacuum daemon consults BEFORE it ever + -- calls compact_rewrite, which does check its own. Four ways an unchecked + -- threshold goes wrong, none of which raises anything: + -- NaN -- `fraction >= NaN` is false in IEEE, so nothing is ever due and + -- the work is suppressed silently and permanently. + -- > 1 -- the same outcome for any fraction: never due. + -- < 0 -- `fraction >= -1` is true for EVERY table, so the daemon believes + -- compaction is always due and rewrites every columnar table on + -- every pass. This is the dangerous direction: not a suppressed + -- report but a permanent, self-renewing rewrite. + -- NULL -- the verdict is NULL, and the daemon reads a NULL verdict as + -- "not due" (SPI_getbinval isnull), so it is the NaN case again. + -- 0.0 and 1.0 are LEGAL and stay legal: 0.0 means "any decay at all is worth + -- acting on", 1.0 means "only a fully dead table". The bounds are inclusive, + -- matching pgcolumnar.compact_rewrite's own guard, and test/native_reclaim.sh + -- pins both endpoints as ACCEPTED so this guard cannot quietly become + -- over-broad, which is how a bounds check usually breaks. + -- + -- The explicit NaN test is redundant with `> 1.0` today, because PostgreSQL + -- float8 ordering is not IEEE ordering: it sorts NaN above every other value. + -- It is written out anyway so the intent survives an edit to the bounds. + IF compact_due_fraction IS NULL + OR compact_due_fraction = 'NaN'::float8 + OR compact_due_fraction < 0.0 + OR compact_due_fraction > 1.0 THEN + RAISE EXCEPTION 'compact_due_fraction must be a number between 0 and 1' + USING ERRCODE = 'invalid_parameter_value'; + END IF; + IF recluster_due_fraction IS NULL + OR recluster_due_fraction = 'NaN'::float8 + OR recluster_due_fraction < 0.0 + OR recluster_due_fraction > 1.0 THEN + RAISE EXCEPTION 'recluster_due_fraction must be a number between 0 and 1' + USING ERRCODE = 'invalid_parameter_value'; + END IF; + + -- stats() enforces require_caller_select(rel) before it returns a row, so a + -- caller without SELECT on rel is refused here rather than reported to. + SELECT COALESCE(sum(s.rowcount), 0), COALESCE(sum(s.deletedrows), 0) + INTO st_rows, st_del + FROM pgcolumnar.stats(rel) s; + + SELECT * INTO ss FROM pgcolumnar.sort_status(rel); + + total_rows := st_rows; + deleted_rows := st_del; + deleted_fraction := CASE WHEN st_rows > 0 + THEN st_del::float8 / st_rows ELSE 0 END; + + sort_key := ss.sort_key; + appended_groups := ss.appended_groups; + appended_rows := ss.appended_rows; + appended_fraction := CASE WHEN (ss.sorted_rows + ss.appended_rows) > 0 + THEN ss.appended_rows::float8 + / (ss.sorted_rows + ss.appended_rows) + ELSE 0 END; + + compact_rewrite_due := (deleted_fraction >= compact_due_fraction); + -- A sorted RUN must exist for recluster to mean anything. sort_status() + -- reports a never-ordered table as entirely appended (no run), and + -- vacuum_sorted() establishes a run without setting options.sort_by, so the + -- run -- sorted_groups > 0 -- is the signal, not the sort_by label (sort_key + -- is reported for information and may be NULL on an ordered table). + recluster_due := (ss.sorted_groups > 0 + AND ss.appended_groups > 0 + AND appended_fraction >= recluster_due_fraction); + + recommendation := NULLIF( + concat_ws(', ', + CASE WHEN compact_rewrite_due THEN 'compact_rewrite' END, + CASE WHEN recluster_due THEN 'recluster' END), + ''); + RETURN; +END; +$maintenance_due$; + +COMMENT ON FUNCTION pgcolumnar.maintenance_due(regclass, float8, float8) + IS 'report whether an online maintenance verb (compact_rewrite, recluster) is worth running, from table statistics alone; thresholds are parameters with defaults measured on #415, each required to be a number between 0 and 1 inclusive (#860); pure report, takes no lock and rewrites nothing (#415)'; diff --git a/pgcolumnar.control b/pgcolumnar.control index e6194a67..fb2c648a 100644 --- a/pgcolumnar.control +++ b/pgcolumnar.control @@ -4,7 +4,7 @@ # independent MIT implementation; the re-origination line builds from # design/NATIVE_FORMAT_AND_INTERFACE_SPEC.md. comment = 'Columnar storage for PostgreSQL (pgColumnar)' -default_version = '1.0-alpha3' +default_version = '1.0-alpha4' module_pathname = '$libdir/pgcolumnar' relocatable = false schema = pgcolumnar diff --git a/src/columnar_autovacuum.c b/src/columnar_autovacuum.c index 19471d58..20827767 100644 --- a/src/columnar_autovacuum.c +++ b/src/columnar_autovacuum.c @@ -55,6 +55,7 @@ #include "utils/snapmgr.h" #include "columnar.h" +#include "columnar_curve.h" /* GUCs (defined in columnar_tableam.c _PG_init, declared in columnar.h) */ extern bool pgcolumnar_autovacuum; @@ -234,6 +235,8 @@ av_maintain_one(const char *qualname) bool compactDue = false; bool reclusterDue = false; char *sortKey = NULL; + char *sortedKind = NULL; + const char *reclusterVerb; Oid argtypes[3] = {REGCLASSOID, FLOAT8OID, FLOAT8OID}; Datum argvals[3]; char q[512]; @@ -245,10 +248,19 @@ av_maintain_one(const char *qualname) if (SPI_connect() != SPI_OK_CONNECT) elog(ERROR, "pgcolumnar autovacuum: SPI_connect failed"); + /* + * sorted_kind comes from sort_status, alongside the verdict, because the + * daemon has to dispatch on THE CURVE THE TABLE IS ON (#889). Reading it + * here rather than adding an output column to maintenance_due keeps that + * function's signature -- and so the upgrade path -- unchanged; both are + * SECURITY DEFINER reports over the same storage row, and this worker is + * a superuser, so neither adds an access question. + */ if (SPI_execute_with_args( - "SELECT compact_rewrite_due, recluster_due, " - " array_to_string(sort_key, ',') " - "FROM pgcolumnar.maintenance_due($1, $2, $3)", + "SELECT m.compact_rewrite_due, m.recluster_due, " + " array_to_string(m.sort_key, ','), s.sorted_kind " + "FROM pgcolumnar.maintenance_due($1, $2, $3) m, " + " pgcolumnar.sort_status($1) s", 3, argtypes, argvals, NULL, true, 1) == SPI_OK_SELECT && SPI_processed == 1) { @@ -262,6 +274,9 @@ av_maintain_one(const char *qualname) d = SPI_getbinval(SPI_tuptable->vals[0], SPI_tuptable->tupdesc, 3, &isnull); if (!isnull) sortKey = pstrdup(TextDatumGetCString(d)); + d = SPI_getbinval(SPI_tuptable->vals[0], SPI_tuptable->tupdesc, 4, &isnull); + if (!isnull) + sortedKind = pstrdup(TextDatumGetCString(d)); } /* compact_rewrite: online space reclaim (SUEL) */ @@ -283,12 +298,31 @@ av_maintain_one(const char *qualname) */ if (reclusterDue && sortKey != NULL && sortKey[0] != '\0') { + /* + * DISPATCH ON THE RECORDED KIND (#889). This line hard-coded + * pgcolumnar.recluster, so a Hilbert table was reclustered by the + * Z-order verb and came back relabelled 'zorder' -- the daemon + * converting a layout the user chose, on a timer, with nothing in + * the log to say so. Plain recluster() now maintains the recorded + * curve on a matching key, so this is belt and braces; it is written + * out anyway because "the daemon preserves the curve" should be + * readable HERE, at the call the ruling is about, and not depend on + * a rule two modules away. + */ + reclusterVerb = + (sortedKind != NULL && + strcmp(sortedKind, COLUMNAR_CURVE_HILBERT) == 0) + ? "pgcolumnar.recluster_hilbert" : "pgcolumnar.recluster"; + snprintf(q, sizeof(q), - "SELECT pgcolumnar.recluster(%s, VARIADIC string_to_array(%s, ',')::name[])", + "SELECT %s(%s, VARIADIC string_to_array(%s, ',')::name[])", + reclusterVerb, quote_literal_cstr(qualname), quote_literal_cstr(sortKey)); (void) SPI_execute(q, false, 0); - elog(LOG, "pgcolumnar autovacuum: recluster %s by (%s)", qualname, sortKey); + elog(LOG, "pgcolumnar autovacuum: recluster %s by (%s) on the %s curve", + qualname, sortKey, + sortedKind != NULL ? sortedKind : COLUMNAR_CURVE_ZORDER); } SPI_finish(); diff --git a/src/columnar_curve.h b/src/columnar_curve.h index 1b45c380..90511317 100644 --- a/src/columnar_curve.h +++ b/src/columnar_curve.h @@ -22,6 +22,22 @@ #ifndef COLUMNAR_CURVE_H #define COLUMNAR_CURVE_H +/* + * The two curve names, as recorded in pgcolumnar.storage.sorted_kind and as + * compared by every self-gate. + * + * They are constants because the string is written at one end of the system and + * compared at the other: a rewrite records the kind, and a later call gates on + * it. A typo in either half is accepted silently -- the catalog takes any text, + * and strcmp against a misspelt literal simply never matches -- so the gate + * refuses forever, invisibly, and the only symptom is a verb that always does + * the full work. Spelling the name once makes that a compile error instead. + * + * 'lexicographic', the third kind, is vacuum_sorted's and is not a curve. + */ +#define COLUMNAR_CURVE_ZORDER "zorder" +#define COLUMNAR_CURVE_HILBERT "hilbert" + /* * Transform ncols 64-bit coordinates in place, from axes to the Hilbert * transpose (Skilling 2004, AxestoTranspose, at b = 64). diff --git a/src/columnar_vacuum.c b/src/columnar_vacuum.c index 0aa20336..57fd9130 100644 --- a/src/columnar_vacuum.c +++ b/src/columnar_vacuum.c @@ -63,10 +63,12 @@ PG_FUNCTION_INFO_V1(pgcolumnar_relation_storageid); PG_FUNCTION_INFO_V1(pgcolumnar_vacuum); PG_FUNCTION_INFO_V1(pgcolumnar_vacuum_sorted); PG_FUNCTION_INFO_V1(pgcolumnar_cluster); +PG_FUNCTION_INFO_V1(pgcolumnar_cluster_hilbert); PG_FUNCTION_INFO_V1(pgcolumnar_compact); PG_FUNCTION_INFO_V1(pgcolumnar_expire); PG_FUNCTION_INFO_V1(pgcolumnar_compact_rewrite); PG_FUNCTION_INFO_V1(pgcolumnar_recluster); +PG_FUNCTION_INFO_V1(pgcolumnar_recluster_hilbert); PG_FUNCTION_INFO_V1(pgcolumnar_truncate); PG_FUNCTION_INFO_V1(pgcolumnar_debug_advance_reserved_offset); PG_FUNCTION_INFO_V1(pgcolumnar_debug_set_metapage_version); @@ -203,14 +205,34 @@ PgColumnarRequireTableOwnerByOid(Oid relid) aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_TABLE, get_rel_name(relid)); } -/* Z-order helpers (defined later, used by the online recluster below) */ +/* Clustering helpers (defined later, used by the online recluster below) */ static bool cluster_type_supported(Oid typid); static bool group_has_live_null(Relation rel, NativeRowGroupMetadata *rg, AttrNumber attno); + +/* + * One row's clustering key, for one curve. Both curves produce the same width + * from the same ordinals and differ only by the transpose, so a rewrite picks + * the function once and the read loop below is identical for either (#889). + */ +typedef bytea *(*ClusterKeyFn) (Datum *values, bool *isnull, AttrNumber *atts, + int ncols, TupleDesc tupdesc); + static bytea *cluster_zorder_key(Datum *values, bool *isnull, AttrNumber *atts, int ncols, TupleDesc tupdesc); +static bytea *cluster_hilbert_key(Datum *values, bool *isnull, AttrNumber *atts, + int ncols, TupleDesc tupdesc); +static ClusterKeyFn cluster_key_fn(const char *curve); +static const char *cluster_inherited_curve(Relation rel, int ncols, + AttrNumber *atts); +static void pgcolumnar_compact_relation_curve(Relation rel, int ncols, + AttrNumber *atts, + const char *curve); /* Names an ordering rewrite records as its key (defined later, #415) */ static List *sort_key_names(TupleDesc tupdesc, AttrNumber *atts, int ncols); +static bool sort_key_matches(TupleDesc tupdesc, AttrNumber *atts, int ncols, + List *recorded); +static bool relation_is_hilbert(Relation rel); static bool vacuum_sorted_gate_is_noop(Relation rel, int ncols, AttrNumber *atts); @@ -553,22 +575,30 @@ record_online_sorted_extent(Relation rel, uint64 storageId, /* * pgcolumnar_recluster_online - * Re-establish global Z-order clustering over the relation's live rows + * Re-establish global clustering on `curve` over the relation's live rows * online (Phase F3c): read all live rows under a snapshot taken after - * advisory-locking every group, Morton-sort them, write them back as fresh - * groups with online index maintenance, and retire the old groups in the same - * transaction. Holds ShareUpdateExclusiveLock (the caller's), so reads never - * block; deletes to the reclustered groups serialize and retry via the F3b - * conflict protocol. Returns the number of groups retired. + * advisory-locking every group, sort them on the curve key, write them back + * as fresh groups with online index maintenance, and retire the old groups + * in the same transaction. Holds ShareUpdateExclusiveLock (the caller's), so + * reads never block; deletes to the reclustered groups serialize and retry + * via the F3b conflict protocol. Returns the number of groups retired. + * + * `curve` is one of the COLUMNAR_CURVE_* names and decides three things + * together, which is why it is one parameter rather than three: the key + * this rewrite sorts on, the kind its self-gate compares against, and the + * kind it records. Splitting them is how a table gets laid on one curve and + * labelled the other. */ static int64 -pgcolumnar_recluster_online(Relation rel, int ncols, AttrNumber *atts) +pgcolumnar_recluster_online(Relation rel, int ncols, AttrNumber *atts, + const char *curve) { + ClusterKeyFn keyfn = cluster_key_fn(curve); uint64 storageId = PgColumnarStorageId(rel); Oid relid = RelationGetRelid(rel); TupleDesc tupdesc = RelationGetDescr(rel); int natts = tupdesc->natts; - AttrNumber zAtt = (AttrNumber) (natts + 1); + AttrNumber keyAtt = (AttrNumber) (natts + 1); Snapshot listSnap; List *rgList; ListCell *lc; @@ -628,41 +658,30 @@ pgcolumnar_recluster_online(Relation rel, int ncols, AttrNumber *atts) qsort(oldGroups, nGroups, sizeof(RetiredGroup), retired_group_cmp); /* - * Self-gate (#415): if the whole live relation is already the Z-order run + * Self-gate (#415): if the whole live relation is already THIS curve's run * over exactly these columns -- nothing appended past the recorded run, same * kind, same key -- there is nothing to recluster. Return before locking and * rewriting every group, so a scheduler (or a user) can call recluster * speculatively without paying a full rewrite each time. Any mismatch - * (appended groups, a different key, a lexicographic or unknown run) falls - * through to the full reorg below, so re-clustering by a new key still works. + * (appended groups, a different key, a lexicographic or unknown run, or a + * run on the OTHER curve) falls through to the full reorg below, so + * re-clustering by a new key and switching curves both still work. + * + * The kind compared is `curve`, the curve this call will lay, and NOT the + * literal 'zorder' it was before #889. Comparing against a fixed literal + * gates a Hilbert recluster on a Z-order label, which reads as "already + * clustered" for a table that is not on this curve at all. */ { int64 sfrom, sthrough; List *skey; char *skind; - bool sameKey = false; + bool sameKey; PgColumnarGetSortedInfo(storageId, &sfrom, &sthrough, &skey, &skind); - if (skind != NULL && strcmp(skind, "zorder") == 0 && - list_length(skey) == ncols) - { - ListCell *klc; - - sameKey = true; - i = 0; - foreach(klc, skey) - { - const char *want = NameStr(TupleDescAttr(tupdesc, atts[i] - 1)->attname); - - if (strcmp((char *) lfirst(klc), want) != 0) - { - sameKey = false; - break; - } - i++; - } - } + sameKey = (skind != NULL && strcmp(skind, curve) == 0 && + sort_key_matches(tupdesc, atts, ncols, skey)); if (sameKey && sfrom >= 0 && sthrough >= 0 && (int64) oldGroups[0].groupNumber >= sfrom && (int64) oldGroups[nGroups - 1].groupNumber <= sthrough) @@ -685,7 +704,7 @@ pgcolumnar_recluster_online(Relation rel, int ncols, AttrNumber *atts) augdesc = CreateTemplateTupleDesc(natts + 1); for (i = 1; i <= natts; i++) TupleDescCopyEntry(augdesc, (AttrNumber) i, tupdesc, (AttrNumber) i); - TupleDescInitEntry(augdesc, zAtt, "__zorder", BYTEAOID, -1, 0); + TupleDescInitEntry(augdesc, keyAtt, "__curvekey", BYTEAOID, -1, 0); #if PG_VERSION_NUM >= 190000 /* PG19 requires a manually-built TupleDesc to be finalized before use, which * computes firstNonCachedOffsetAttr (asserted by the tuple routines) after the @@ -695,7 +714,7 @@ pgcolumnar_recluster_online(Relation rel, int ncols, AttrNumber *atts) tce = lookup_type_cache(BYTEAOID, TYPECACHE_LT_OPR); byteaLt = tce->lt_opr; - tsort = tuplesort_begin_heap(augdesc, 1, &zAtt, &byteaLt, &sortColl, + tsort = tuplesort_begin_heap(augdesc, 1, &keyAtt, &byteaLt, &sortColl, &nullsFirst, maintenance_work_mem, NULL, COLUMNAR_TUPLESORT_NONACCESS); @@ -707,14 +726,14 @@ pgcolumnar_recluster_online(Relation rel, int ncols, AttrNumber *atts) while (PgColumnarReadNextRow(readState, readSlot->tts_values, readSlot->tts_isnull, &rowNumber)) { - bytea *zkey; + bytea *ckey; CHECK_FOR_INTERRUPTS(); memcpy(putSlot->tts_values, readSlot->tts_values, natts * sizeof(Datum)); memcpy(putSlot->tts_isnull, readSlot->tts_isnull, natts * sizeof(bool)); - zkey = cluster_zorder_key(readSlot->tts_values, readSlot->tts_isnull, - atts, ncols, tupdesc); - putSlot->tts_values[natts] = PointerGetDatum(zkey); + ckey = keyfn(readSlot->tts_values, readSlot->tts_isnull, + atts, ncols, tupdesc); + putSlot->tts_values[natts] = PointerGetDatum(ckey); putSlot->tts_isnull[natts] = false; ExecStoreVirtualTuple(putSlot); tuplesort_puttupleslot(tsort, putSlot); @@ -770,7 +789,7 @@ pgcolumnar_recluster_online(Relation rel, int ncols, AttrNumber *atts) /* record how far the reordered run reaches (#311) and BY WHAT (#415) */ record_online_sorted_extent(rel, storageId, writeState, stripeMark, - sort_key_names(tupdesc, atts, ncols), "zorder"); + sort_key_names(tupdesc, atts, ncols), curve); PopActiveSnapshot(); UnregisterSnapshot(snap); @@ -781,17 +800,46 @@ pgcolumnar_recluster_online(Relation rel, int ncols, AttrNumber *atts) } /* - * pgcolumnar_recluster - * SQL: pgcolumnar.recluster(tablename regclass, VARIADIC columns name[]). - * The lazy online counterpart to cluster(): re-establish global Z-order - * clustering under ShareUpdateExclusiveLock (concurrent reads and writes), - * not the AccessExclusiveLock the eager cluster() reorg takes. Returns the - * number of groups reclustered. + * cluster_curve_label + * The curve's name as it appears in an error message. One place, so the + * two verbs cannot describe the same refusal differently. */ -Datum -pgcolumnar_recluster(PG_FUNCTION_ARGS) +static const char * +cluster_curve_label(const char *curve) { - Oid relid = PG_GETARG_OID(0); + return (strcmp(curve, COLUMNAR_CURVE_HILBERT) == 0) ? "Hilbert" : "Z-order"; +} + +/* + * cluster_verb_validate + * The argument checking every clustering entry point does, once (#889). + * + * cluster(), recluster(), cluster_hilbert() and recluster_hilbert() take + * the identical arguments and must refuse the identical inputs with the + * identical SQLSTATEs. That was ~90 lines duplicated per verb; at four + * verbs it is the shape in which two of them drift apart, and + * test/hilbert_cluster.sh S1 asserts each new verb's state against the + * state its sibling raises on the byte-identical input precisely because + * that drift is silent. + * + * On return the relation is open under `lockmode`, *ncolsp holds the column + * count and the returned array holds their attribute numbers. Every refusal + * below closes the relation first, and ownership is checked BEFORE + * table_open (#568) so an unprivileged caller never joins the lock queue. + * + * The SQLSTATEs, which are the contract: + * 22004 a null table name + * 22023 no columns, more than eight, or a null column name + * 42809 not a columnar table + * 42703 no such column + * 0A000 a column whose type has no order-preserving ordinal + * 42501 the caller does not own the table + */ +static AttrNumber * +cluster_verb_validate(FunctionCallInfo fcinfo, const char *curve, + LOCKMODE lockmode, Relation *relp, int *ncolsp) +{ + Oid relid; ArrayType *colArray; Datum *colDatums; bool *colNulls; @@ -799,7 +847,6 @@ pgcolumnar_recluster(PG_FUNCTION_ARGS) Relation rel; TupleDesc tupdesc; AttrNumber *atts; - int64 reclustered; int i; if (PG_ARGISNULL(0)) @@ -811,6 +858,7 @@ pgcolumnar_recluster(PG_FUNCTION_ARGS) (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("at least one clustering column is required"))); + relid = PG_GETARG_OID(0); colArray = PG_GETARG_ARRAYTYPE_P(1); deconstruct_array(colArray, NAMEOID, NAMEDATALEN, false, 'c', &colDatums, &colNulls, &ncols); @@ -821,16 +869,17 @@ pgcolumnar_recluster(PG_FUNCTION_ARGS) if (ncols > 8) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("Z-order clustering supports at most 8 columns"))); + errmsg("%s clustering supports at most 8 columns", + cluster_curve_label(curve)))); + /* Ownership before the lock (#568), as in pgcolumnar_vacuum. */ PgColumnarRequireTableOwnerByOid(relid); - /* the lazy lock: concurrent reads and writes during the recluster */ - rel = table_open(relid, ShareUpdateExclusiveLock); + rel = table_open(relid, lockmode); if (!PgColumnarIsColumnarRelation(relid)) { - table_close(rel, ShareUpdateExclusiveLock); + table_close(rel, lockmode); ereport(ERROR, (errcode(ERRCODE_WRONG_OBJECT_TYPE), errmsg("relation \"%s\" is not a columnar table", @@ -847,7 +896,7 @@ pgcolumnar_recluster(PG_FUNCTION_ARGS) if (colNulls[i]) { - table_close(rel, ShareUpdateExclusiveLock); + table_close(rel, lockmode); ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("clustering column name cannot be null"))); @@ -857,7 +906,7 @@ pgcolumnar_recluster(PG_FUNCTION_ARGS) if (attno == InvalidAttrNumber || attno <= 0 || TupleDescAttr(tupdesc, attno - 1)->attisdropped) { - table_close(rel, ShareUpdateExclusiveLock); + table_close(rel, lockmode); ereport(ERROR, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("column \"%s\" does not exist in table \"%s\"", @@ -866,22 +915,112 @@ pgcolumnar_recluster(PG_FUNCTION_ARGS) att = TupleDescAttr(tupdesc, attno - 1); if (!cluster_type_supported(att->atttypid)) { - table_close(rel, ShareUpdateExclusiveLock); + table_close(rel, lockmode); ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("column \"%s\" of type %s cannot be used as a clustering key", colname, format_type_be(att->atttypid)), - errhint("Z-order clustering supports integer, date/time, boolean, and floating-point columns. " + errhint("%s clustering supports integer, date/time, boolean, and floating-point columns. " "For a text or other btree-orderable key, use pgcolumnar.vacuum_sorted() " - "(lexicographic sort on the given columns), optionally declared via set_options(..., sort_by => ...)."))); + "(lexicographic sort on the given columns), optionally declared via set_options(..., sort_by => ...).", + cluster_curve_label(curve)))); } atts[i] = attno; } - reclustered = pgcolumnar_recluster_online(rel, ncols, atts); + *relp = rel; + *ncolsp = ncols; + return atts; +} + +/* + * recluster_verb + * The body behind pgcolumnar.recluster and pgcolumnar.recluster_hilbert. + * + * `curve` is NULL for the plain verb, which names no curve and therefore + * INHERITS the table's own (the ruling on #889: see + * cluster_inherited_curve). recluster_hilbert names its curve, so it never + * inherits -- naming the verb is how a curve is switched. + */ +static int64 +recluster_verb(FunctionCallInfo fcinfo, const char *curve) +{ + Relation rel; + int ncols; + AttrNumber *atts; + int64 reclustered; + + /* the lazy lock: concurrent reads and writes during the recluster */ + atts = cluster_verb_validate(fcinfo, + curve ? curve : COLUMNAR_CURVE_ZORDER, + ShareUpdateExclusiveLock, &rel, &ncols); + + if (curve == NULL) + curve = cluster_inherited_curve(rel, ncols, atts); + + reclustered = pgcolumnar_recluster_online(rel, ncols, atts, curve); table_close(rel, NoLock); - PG_RETURN_INT64(reclustered); + return reclustered; +} + +/* + * cluster_verb + * The body behind pgcolumnar.cluster and pgcolumnar.cluster_hilbert. Same + * curve rule as recluster_verb, under AccessExclusiveLock. + */ +static void +cluster_verb(FunctionCallInfo fcinfo, const char *curve) +{ + Relation rel; + int ncols; + AttrNumber *atts; + + atts = cluster_verb_validate(fcinfo, + curve ? curve : COLUMNAR_CURVE_ZORDER, + AccessExclusiveLock, &rel, &ncols); + + if (curve == NULL) + curve = cluster_inherited_curve(rel, ncols, atts); + + pgcolumnar_compact_relation_curve(rel, ncols, atts, curve); + + /* keep the lock until end of transaction */ + table_close(rel, NoLock); +} + +/* + * pgcolumnar_recluster + * SQL: pgcolumnar.recluster(tablename regclass, VARIADIC columns name[]). + * The lazy online counterpart to cluster(): re-establish global clustering + * under ShareUpdateExclusiveLock (concurrent reads and writes), not the + * AccessExclusiveLock the eager cluster() reorg takes. Returns the number + * of groups reclustered. + * + * Z-order, unless the table is already on another curve over exactly these + * columns, in which case that curve is maintained (#889). This verb names + * no curve, so it does not silently re-declare one. + */ +Datum +pgcolumnar_recluster(PG_FUNCTION_ARGS) +{ + PG_RETURN_INT64(recluster_verb(fcinfo, NULL)); +} + +/* + * pgcolumnar_recluster_hilbert + * SQL: pgcolumnar.recluster_hilbert(tablename regclass, + * VARIADIC columns name[]). recluster() on the Hilbert curve (#889). + * + * A separate verb rather than a parameter because PostgreSQL cannot extend + * the existing signature in either direction: a defaulted parameter cannot + * precede a VARIADIC one, and an array-plus-kind overload makes the + * documented recluster('t','a','b') call ambiguous. + */ +Datum +pgcolumnar_recluster_hilbert(PG_FUNCTION_ARGS) +{ + PG_RETURN_INT64(recluster_verb(fcinfo, COLUMNAR_CURVE_HILBERT)); } /* @@ -1091,6 +1230,49 @@ cluster_zorder_key(Datum *values, bool *isnull, AttrNumber *atts, int ncols, return result; } +/* + * Build the Hilbert key for one row (#889). + * + * The same ordinals, the same packer and the same 8*ncols width as the Z-order + * key; the whole difference between the two curves is the transpose in the + * middle. Skilling's observation is that the Hilbert index IS the MSB-first + * interleave of the transposed coordinates, so memcmp order over the result is + * Hilbert order for exactly the reason it is Z-order above, and every caller + * downstream -- the tuplesort, the bytea comparator, the writer -- is unchanged. + * + * The transpose is held by test/hilbert_curve.sh against the properties that + * define a Hilbert curve, and against frozen hex. + */ +static bytea * +cluster_hilbert_key(Datum *values, bool *isnull, AttrNumber *atts, int ncols, + TupleDesc tupdesc) +{ + int keybytes = ncols * 8; + bytea *result = (bytea *) palloc(VARHDRSZ + keybytes); + uint64 *ord = cluster_key_ordinals(values, isnull, atts, ncols, tupdesc); + + SET_VARSIZE(result, VARHDRSZ + keybytes); + cluster_hilbert_transpose(ord, ncols); + cluster_pack_interleave(ord, ncols, (unsigned char *) VARDATA(result)); + + pfree(ord); + return result; +} + +/* + * The key builder for a named curve, resolved ONCE per rewrite rather than per + * row. An unrecognised name is a programming error, not user input: every call + * site passes one of the two constants. + */ +static ClusterKeyFn +cluster_key_fn(const char *curve) +{ + if (strcmp(curve, COLUMNAR_CURVE_HILBERT) == 0) + return cluster_hilbert_key; + Assert(strcmp(curve, COLUMNAR_CURVE_ZORDER) == 0); + return cluster_zorder_key; +} + /* * pgcolumnar_relation_storageid @@ -1159,6 +1341,100 @@ sort_key_names(TupleDesc tupdesc, AttrNumber *atts, int ncols) return names; } +/* + * sort_key_matches + * Is the key RECORDED on this storage exactly atts[0 .. ncols-1], in this + * order? + * + * The inverse of sort_key_names, and the comparison all three gates make: + * the online recluster's, vacuum_sorted's, and the sticky-curve resolver's. + * Written once because three copies of a loop that must agree is how two of + * them drift. + * + * Order matters: (a,b) and (b,a) are different layouts under either curve, + * so this is a sequence comparison and not a set one. + */ +static bool +sort_key_matches(TupleDesc tupdesc, AttrNumber *atts, int ncols, List *recorded) +{ + ListCell *lc; + int i = 0; + + if (list_length(recorded) != ncols) + return false; + + foreach(lc, recorded) + { + const char *want = NameStr(TupleDescAttr(tupdesc, atts[i] - 1)->attname); + + if (strcmp((char *) lfirst(lc), want) != 0) + return false; + i++; + } + return true; +} + +/* + * relation_is_hilbert + * Is this relation's recorded layout the Hilbert curve, over ANY key? + * + * The key is not compared, on purpose. This answers "is this table + * clustered on a curve that a lexicographic rewrite would destroy", and + * that is true of every Hilbert key, not only the one the caller happened + * to name. + */ +static bool +relation_is_hilbert(Relation rel) +{ + int64 sfrom, + sthrough; + List *skey; + char *skind; + + PgColumnarGetSortedInfo(PgColumnarStorageId(rel), &sfrom, &sthrough, + &skey, &skind); + + return (skind != NULL && strcmp(skind, COLUMNAR_CURVE_HILBERT) == 0); +} + +/* + * cluster_inherited_curve + * The curve the PLAIN verbs -- cluster() and recluster() -- lay on this + * relation. + * + * THE CURVE IS STICKY (the owner's ruling on #889). sorted_kind is the + * table's DECLARED INTENT, not a property of each call. cluster() and + * recluster() name no curve, so on a table already laid on one over + * exactly the key they were handed they MAINTAIN it, rather than + * converting it to Z-order and relabelling it. That conversion is what the + * ruling forbids: it is silent, it is not what the caller asked for, and + * nothing in the call says it happened. + * + * A DIFFERENT key is the explicit re-declaration, so nothing is inherited + * there and the plain verbs' own curve applies. recluster('t','b','a') on a + * Hilbert table over (a,b) is therefore an honest switch back to Z-order, + * which is what keeps "sticky" from meaning "unescapable". + * + * Switching curves on the SAME key is done by naming the other verb. + */ +static const char * +cluster_inherited_curve(Relation rel, int ncols, AttrNumber *atts) +{ + int64 sfrom, + sthrough; + List *skey; + char *skind; + + PgColumnarGetSortedInfo(PgColumnarStorageId(rel), &sfrom, &sthrough, + &skey, &skind); + + if (skind != NULL && strcmp(skind, COLUMNAR_CURVE_HILBERT) == 0 && + sort_key_matches(RelationGetDescr(rel), atts, ncols, skey)) + return COLUMNAR_CURVE_HILBERT; + + return COLUMNAR_CURVE_ZORDER; +} + /* * record_sorted_extent * Mark where the ordered run this rewrite just wrote ends (issue #301), and @@ -1437,20 +1713,29 @@ pgcolumnar_compact_relation(Relation rel, int nsortkeys, AttrNumber *sortAtts) } /* - * pgcolumnar_compact_relation_zorder - * Rewrite every live row of a columnar relation ordered by the Z-order - * (Morton) code over atts[0..ncols-1] (Phase F2). Mirrors + * pgcolumnar_compact_relation_curve + * Rewrite every live row of a columnar relation ordered by `curve`'s + * space-filling code over atts[0..ncols-1] (Phase F2, #889). Mirrors * pgcolumnar_compact_relation, but sorts by a computed key carried as a * trailing bytea column of an augmented tuple, so the sort still spills to * disk through tuplesort. The relation is already open AccessExclusiveLock. + * + * `curve` picks the key builder AND is the kind recorded at the end, for + * the same reason it is one parameter in the online path: a table laid on + * one curve and labelled the other gates wrongly forever afterwards. It was + * named _zorder while Z-order was the only curve; the name went with the + * parameter, because a function called _zorder that lays Hilbert is the + * same trap in a different place. */ static void -pgcolumnar_compact_relation_zorder(Relation rel, int ncols, AttrNumber *atts) +pgcolumnar_compact_relation_curve(Relation rel, int ncols, AttrNumber *atts, + const char *curve) { + ClusterKeyFn keyfn = cluster_key_fn(curve); Oid relid = RelationGetRelid(rel); TupleDesc tupdesc = RelationGetDescr(rel); int natts = tupdesc->natts; - AttrNumber zAtt = (AttrNumber) (natts + 1); + AttrNumber keyAtt = (AttrNumber) (natts + 1); uint64 oldStorageId; Snapshot snapshot; PgColumnarReadState *readState; @@ -1481,11 +1766,11 @@ pgcolumnar_compact_relation_zorder(Relation rel, int ncols, AttrNumber *atts) snapshot = RegisterSnapshot(GetLatestSnapshot()); PushActiveSnapshot(snapshot); - /* augmented descriptor: the table's columns plus a trailing bytea Z-order key */ + /* augmented descriptor: the table's columns plus a trailing bytea curve key */ augdesc = CreateTemplateTupleDesc(natts + 1); for (i = 1; i <= natts; i++) TupleDescCopyEntry(augdesc, (AttrNumber) i, tupdesc, (AttrNumber) i); - TupleDescInitEntry(augdesc, zAtt, "__zorder", BYTEAOID, -1, 0); + TupleDescInitEntry(augdesc, keyAtt, "__curvekey", BYTEAOID, -1, 0); #if PG_VERSION_NUM >= 190000 /* PG19 requires a manually-built TupleDesc to be finalized before use, which * computes firstNonCachedOffsetAttr (asserted by the tuple routines) after the @@ -1495,7 +1780,7 @@ pgcolumnar_compact_relation_zorder(Relation rel, int ncols, AttrNumber *atts) tce = lookup_type_cache(BYTEAOID, TYPECACHE_LT_OPR); byteaLt = tce->lt_opr; - tsort = tuplesort_begin_heap(augdesc, 1, &zAtt, &byteaLt, &sortColl, + tsort = tuplesort_begin_heap(augdesc, 1, &keyAtt, &byteaLt, &sortColl, &nullsFirst, maintenance_work_mem, NULL, COLUMNAR_TUPLESORT_NONACCESS); @@ -1507,14 +1792,14 @@ pgcolumnar_compact_relation_zorder(Relation rel, int ncols, AttrNumber *atts) while (PgColumnarReadNextRow(readState, readSlot->tts_values, readSlot->tts_isnull, &rowNumber)) { - bytea *zkey; + bytea *ckey; CHECK_FOR_INTERRUPTS(); memcpy(putSlot->tts_values, readSlot->tts_values, natts * sizeof(Datum)); memcpy(putSlot->tts_isnull, readSlot->tts_isnull, natts * sizeof(bool)); - zkey = cluster_zorder_key(readSlot->tts_values, readSlot->tts_isnull, - atts, ncols, tupdesc); - putSlot->tts_values[natts] = PointerGetDatum(zkey); + ckey = keyfn(readSlot->tts_values, readSlot->tts_isnull, + atts, ncols, tupdesc); + putSlot->tts_values[natts] = PointerGetDatum(ckey); putSlot->tts_isnull[natts] = false; ExecStoreVirtualTuple(putSlot); tuplesort_puttupleslot(tsort, putSlot); @@ -1556,7 +1841,7 @@ pgcolumnar_compact_relation_zorder(Relation rel, int ncols, AttrNumber *atts) } } - /* write the live rows back in Z-order; the trailing key column is ignored */ + /* write the live rows back in curve order; the trailing key column is ignored */ writeState = PgColumnarGetWriteState(rel); while (tuplesort_gettupleslot(tsort, true, false, augSlot, NULL)) { @@ -1572,8 +1857,8 @@ pgcolumnar_compact_relation_zorder(Relation rel, int ncols, AttrNumber *atts) } PgColumnarFlushWriteStateForRelation(relid); - /* Z-order is an order, so the same extent applies (see record_sorted_extent). */ - record_sorted_extent(rel, sort_key_names(tupdesc, atts, ncols), "zorder"); + /* A curve is an order, so the same extent applies (see record_sorted_extent). */ + record_sorted_extent(rel, sort_key_names(tupdesc, atts, ncols), curve); tuplesort_end(tsort); ExecDropSingleTupleTableSlot(augSlot); @@ -1702,7 +1987,6 @@ vacuum_sorted_gate_is_noop(Relation rel, int ncols, AttrNumber *atts) Snapshot snap; List *rgList; ListCell *lc; - int i; bool noop = true; PgColumnarGetSortedInfo(storageId, &sfrom, &sthrough, &skey, &skind); @@ -1710,17 +1994,8 @@ vacuum_sorted_gate_is_noop(Relation rel, int ncols, AttrNumber *atts) /* 1 + 2: a lexicographic run over exactly this key */ if (skind == NULL || strcmp(skind, "lexicographic") != 0) return false; - if (list_length(skey) != ncols) + if (!sort_key_matches(tupdesc, atts, ncols, skey)) return false; - i = 0; - foreach(lc, skey) - { - const char *want = NameStr(TupleDescAttr(tupdesc, atts[i] - 1)->attname); - - if (strcmp((char *) lfirst(lc), want) != 0) - return false; - i++; - } if (sfrom < 0 || sthrough < 0) return false; @@ -1933,12 +2208,38 @@ pgcolumnar_vacuum_sorted(PG_FUNCTION_ARGS) sortAtts[i++] = attno; } + /* + * THE CURVE IS STICKY, so vacuum_sorted leaves a Hilbert table alone (the + * owner's ruling on #889). + * + * sorted_kind is the table's declared intent. A lexicographic rewrite would + * destroy a Hilbert layout AND relabel it 'lexicographic', so the table + * would afterwards claim an ordering nobody asked for and the recluster + * gates would agree with the label. That pair -- rewrite and relabel -- is + * what the ruling names as the defect. + * + * It is a NO-OP, not a refusal. vacuum_sorted is what the maintenance + * daemon and an operator's cron both call across a whole database; raising + * there turns one clustered table into a failing maintenance pass for + * everything behind it. Skipping is reported at DEBUG1, exactly as the #760 + * gate below reports its own skip. + * + * Z-order is deliberately NOT included. Re-sorting a Z-ordered table + * lexicographically is behaviour that shipped and is pinned by + * test/vacuum_sorted_gate.sh; the ruling is about Hilbert, and widening it + * to every curve is a separate decision with its own removal proof. + */ + if (relation_is_hilbert(rel)) + ereport(DEBUG1, + (errmsg("pgcolumnar: \"%s\" is clustered on the Hilbert curve, skipping the lexicographic rewrite", + RelationGetRelationName(rel)))); + /* * Self-gate (#760): skip the rewrite when the relation is already exactly * this lexicographic run with nothing appended and nothing to reclaim. See * vacuum_sorted_gate_is_noop for why the reclaim half is not optional. */ - if (vacuum_sorted_gate_is_noop(rel, ncols, sortAtts)) + else if (vacuum_sorted_gate_is_noop(rel, ncols, sortAtts)) ereport(DEBUG1, (errmsg("pgcolumnar: \"%s\" is already sorted on this key with nothing to reclaim, skipping rewrite", RelationGetRelationName(rel)))); @@ -1955,7 +2256,9 @@ pgcolumnar_vacuum_sorted(PG_FUNCTION_ARGS) * pgcolumnar_cluster * SQL: pgcolumnar.cluster(tablename regclass, VARIADIC columns name[]). * Physically reorders a columnar table by the Z-order (Morton) space-filling - * curve over the named columns (Phase F2, spec 9). Unlike vacuum_sorted's + * curve over the named columns (Phase F2, spec 9) -- or by the curve the + * table is already laid on over exactly those columns, which this verb + * maintains rather than re-declaring (#889). Unlike vacuum_sorted's * single lead-column sort, Z-order clustering tightens the min/max zone maps * of ALL clustered columns at once, so multi-column range and point * predicates skip far more vectors and chunks. Results are unchanged; this @@ -1972,100 +2275,27 @@ pgcolumnar_vacuum_sorted(PG_FUNCTION_ARGS) Datum pgcolumnar_cluster(PG_FUNCTION_ARGS) { - Oid relid = PG_GETARG_OID(0); - ArrayType *colArray; - Datum *colDatums; - bool *colNulls; - int ncols; - Relation rel; - TupleDesc tupdesc; - AttrNumber *atts; - int i; - - if (PG_ARGISNULL(0)) - ereport(ERROR, - (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), - errmsg("table name cannot be null"))); - if (PG_ARGISNULL(1)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("at least one clustering column is required"))); - - colArray = PG_GETARG_ARRAYTYPE_P(1); - deconstruct_array(colArray, NAMEOID, NAMEDATALEN, false, 'c', - &colDatums, &colNulls, &ncols); - if (ncols < 1) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("at least one clustering column is required"))); - if (ncols > 8) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("Z-order clustering supports at most 8 columns"))); - - /* Ownership before the AccessExclusiveLock (#568), as in pgcolumnar_vacuum. */ - PgColumnarRequireTableOwnerByOid(relid); - - rel = table_open(relid, AccessExclusiveLock); - - if (!PgColumnarIsColumnarRelation(relid)) - { - table_close(rel, AccessExclusiveLock); - ereport(ERROR, - (errcode(ERRCODE_WRONG_OBJECT_TYPE), - errmsg("relation \"%s\" is not a columnar table", - RelationGetRelationName(rel)))); - } - - tupdesc = RelationGetDescr(rel); - atts = palloc(ncols * sizeof(AttrNumber)); - - for (i = 0; i < ncols; i++) - { - char *colname; - AttrNumber attno; - Form_pg_attribute att; - - if (colNulls[i]) - { - table_close(rel, AccessExclusiveLock); - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("clustering column name cannot be null"))); - } - - colname = NameStr(*DatumGetName(colDatums[i])); - attno = get_attnum(relid, colname); - if (attno == InvalidAttrNumber || attno <= 0 || - TupleDescAttr(tupdesc, attno - 1)->attisdropped) - { - table_close(rel, AccessExclusiveLock); - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_COLUMN), - errmsg("column \"%s\" does not exist in table \"%s\"", - colname, RelationGetRelationName(rel)))); - } - - att = TupleDescAttr(tupdesc, attno - 1); - if (!cluster_type_supported(att->atttypid)) - { - table_close(rel, AccessExclusiveLock); - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("column \"%s\" of type %s cannot be used as a clustering key", - colname, format_type_be(att->atttypid)), - errhint("Z-order clustering supports integer, date/time, boolean, and floating-point columns. " - "For a text or other btree-orderable key, use pgcolumnar.vacuum_sorted() " - "(lexicographic sort on the given columns), optionally declared via set_options(..., sort_by => ...)."))); - } - atts[i] = attno; - } - - pgcolumnar_compact_relation_zorder(rel, ncols, atts); - - /* keep the lock until end of transaction */ - table_close(rel, NoLock); + cluster_verb(fcinfo, NULL); + PG_RETURN_VOID(); +} +/* + * pgcolumnar_cluster_hilbert + * SQL: pgcolumnar.cluster_hilbert(tablename regclass, + * VARIADIC columns name[]). cluster() on the Hilbert curve (#889). + * + * Hilbert keeps neighbouring keys neighbouring in storage more tightly than + * Morton order does -- Morton's jumps at a bit boundary are what a Hilbert + * curve has none of -- so the min/max zone maps over the clustered columns + * are tighter and a range predicate skips more vectors. Everything else, + * including the key width and the sort, is identical to cluster(). + * + * A separate verb rather than a parameter: see pgcolumnar_recluster_hilbert. + */ +Datum +pgcolumnar_cluster_hilbert(PG_FUNCTION_ARGS) +{ + cluster_verb(fcinfo, COLUMNAR_CURVE_HILBERT); PG_RETURN_VOID(); } diff --git a/pgcolumnar--1.0-alpha3.sql b/test/fixtures/pgcolumnar--1.0-alpha3.sql similarity index 100% rename from pgcolumnar--1.0-alpha3.sql rename to test/fixtures/pgcolumnar--1.0-alpha3.sql diff --git a/test/native_upgrade_converge.sh b/test/native_upgrade_converge.sh index 7b8d8720..6b5ab356 100755 --- a/test/native_upgrade_converge.sh +++ b/test/native_upgrade_converge.sh @@ -62,12 +62,12 @@ pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" EXTDIR="$("$PGC_PG_CONFIG" --sharedir)/extension" TARGET="$(sed -n "s/^default_version *= *'\\(.*\\)'.*/\\1/p" "$HERE/../pgcolumnar.control")" -check "control default_version is 1.0-alpha3" "$TARGET" "1.0-alpha3" +check "control default_version is 1.0-alpha4" "$TARGET" "1.0-alpha4" # Stage the frozen old base install scripts so an old-version extension can be # created. These are fixtures, not shipped; remove them at the end. STAGED=() -for v in 1.0-alpha 1.0-alpha2; do +for v in 1.0-alpha 1.0-alpha2 1.0-alpha3; do src="$HERE/fixtures/pgcolumnar--$v.sql" dst="$EXTDIR/pgcolumnar--$v.sql" if [ -f "$src" ] && [ ! -f "$dst" ]; then @@ -106,7 +106,7 @@ check "fresh $TARGET install has objects to compare" \ "$([ "$(wc -l <"$REF")" -gt 100 ] && echo yes || echo no)" "yes" # Each released starting point must upgrade to an identical catalog. -for from in 1.0-alpha 1.0-alpha2; do +for from in 1.0-alpha 1.0-alpha2 1.0-alpha3; do [ -f "$EXTDIR/pgcolumnar--$from.sql" ] || { check "fixture for $from present" "missing" "present"; continue; } db="conv_from_$(echo "$from" | tr '.-' '__')" P -d postgres -c "DROP DATABASE IF EXISTS $db;" >/dev/null diff --git a/test/run_all_versions.sh b/test/run_all_versions.sh index 8ebcea94..97e3ae5c 100755 --- a/test/run_all_versions.sh +++ b/test/run_all_versions.sh @@ -92,6 +92,7 @@ SUITES=( groupagg_table_sizing hardening harness_selftest + hilbert_cluster hilbert_curve iceberg_catalog iceberg_data_files From 8474bd63321f04f05fbd21764fd2de9a49b273c7 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 9 Sep 2026 08:55:00 -0600 Subject: [PATCH 6/6] docs: record the Hilbert provenance determination in the Log (#889) PROVENANCE.md's Log is where provenance decisions live, and #889 introduced a source category it had no precedent for. The licensing reasoning existed only in a comment in src/columnar_curve.c; the document that exists to hold it was silent. (OffgridwithJD, #899 review.) The entry records what was transcribed and from where, why the "build only from the specification and the public PostgreSQL API" rule is not being bent -- that rule exists to keep another COLUMNAR ENGINE's source out of this tree, which a published public-domain algorithm from the literature is not, and the same reasoning already covers pglz/lz4/zstd and the min/max skip list -- and what was deliberately NOT done: no other Hilbert implementation was read, and the correctness evidence is property-based rather than comparative. The provenance itself was checked against Skilling's published listing by OffgridwithJD during the #899 review. I have not obtained the paper and the entry says so, rather than presenting their verification as a second one. It also records the consequence a future reader needs: Hilbert curves are not unique above two dimensions, this construction differs from Butz/Hamilton for three or more columns, and both are valid -- so the key bytes are an on-disk format commitment, a disagreement with another library is not a defect, and the 96 golden vectors are what pin which curve this is. docs_style.sh: 9 checks, PASSED. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK --- PROVENANCE.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/PROVENANCE.md b/PROVENANCE.md index 1c8338ad..a496c389 100644 --- a/PROVENANCE.md +++ b/PROVENANCE.md @@ -587,3 +587,42 @@ oracle, so none changes query results. (`hits.tsv.gz`) is downloaded for local measurement and is not redistributed; its own licensing is unestablished and it must not be added to the tree without one. + +- 2026-09-09. Hilbert curve clustering (#889) introduced a source category this + document had no precedent for, so the determination is recorded here rather + than left in a code comment. + + `src/columnar_curve.c`'s `cluster_hilbert_transpose` is a transcription of + `AxestoTranspose` from J. Skilling, "Programming the Hilbert curve", AIP + Conference Proceedings 707 (2004), whose published listing carries an explicit + public-domain notice. Transcribed with the bit count fixed at 64 and the + coordinate type fixed at `uint64`; the structure of the algorithm is + unchanged. The provenance was checked against the published listing by + OffgridwithJD during the #899 review; I have not obtained the paper myself and + am recording their verification rather than a second one. + + Why this is not the rule at the top of this file being bent. "Build only from + the specification and the public PostgreSQL API" exists to keep another + COLUMNAR ENGINE's source out of this tree, which is a competitive and + copyleft-contamination concern. A published, public-domain algorithm from the + academic literature is neither. The same reasoning already covers the codecs: + pglz, lz4 and zstd are used through their public APIs, and the min/max skip + list is long-standing prior art recorded as such above. + + What was NOT done, stated so the boundary stays where it is. No other + implementation of a Hilbert curve was read or consulted -- not a library, not + another database, not published source beyond the paper's own listing. The + correctness evidence is property-based rather than comparative: 184 arms in + `test/hilbert_curve.sh` establish that the index set is exactly the contiguous + range, that consecutive indices are unit-adjacent, and that every dyadic + sub-cube occupies a contiguous run, with a serpentine and a Z-order encoder + carried as deliberately wrong controls because neither property alone + separates a Hilbert curve from those two. + + One consequence worth recording for a future reader. Hilbert curves are not + unique above two dimensions, and this construction differs from the + Butz/Hamilton curve for three or more clustering columns; both are valid. So + the key bytes are an ON-DISK FORMAT COMMITMENT rather than an implementation + detail, disagreement with another library's Hilbert index is not evidence of a + defect, and the 96 golden byte vectors in that suite are what pin which curve + this is.