diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d6d121e..f8dbe0aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,7 @@ installed, `1.0-alpha`, `1.0-alpha2`, and `1.0-alpha3`), so a single notes in this file describe `default_version` as pinned at an earlier version, each true until the next version shipped. -## [Unreleased] +## [1.0-alpha4] - 2026-09-17 ### Fixed @@ -208,6 +208,41 @@ true until the next version shipped. ### Added +- Hilbert clustering: `pgcolumnar.cluster_hilbert` and + `pgcolumnar.recluster_hilbert` (#889). + + **Two verbs rather than a parameter on the existing two.** PostgreSQL refuses + to extend `cluster(regclass, VARIADIC name[])` in either direction: a defaulted + parameter cannot precede a `VARIADIC` one, and an array-plus-kind overload + makes the documented `cluster('t','a','b')` call ambiguous. Both were measured + on 18.4. The new verbs match their siblings element for element in argument + types, variadic element type, return type and volatility, so a caller switches + between them by name alone. + + **What the curve buys.** Z-order jumps a long way in key space at a bit + boundary; a Hilbert curve does not. Keys that are close in the data therefore + stay closer in storage, the min/max zone maps over the clustered columns are + tighter, and a range filter reads fewer chunk groups. The key is the same width + and sorts through the same `bytea` comparator, so nothing downstream of the + sort knows which curve produced it. + + **The curve is sticky.** `sorted_kind` is the table's declared intent, not a + property of each call: + + - plain `recluster` on a Hilbert table over the same key is a no-op returning + 0, not a silent conversion back to Z-order; + - `recluster_hilbert` on a Z-ordered table over the same columns rewrites it; + - `vacuum_sorted` leaves a Hilbert table alone rather than sorting it + lexicographically and relabelling it; + - the maintenance daemon dispatches on the recorded kind, so a Hilbert table + is re-clustered with Hilbert instead of being converted on a timer; + - naming the other verb, or reclustering on a different key, is how a table + changes curve. + + Held by `test/hilbert_cluster.sh` (181 arms) over the SQL surface, the recorded + kind, both self-gates and the daemon, and by `test/hilbert_curve.sh` (184 arms) + over the encoder itself. + - Six more guards counted their callers instead of pinning their property (#1078's class). Two were blind to the defect they name. @@ -2275,41 +2310,6 @@ true until the next version shipped. Per-element evaluation is capped at 128 non-NULL entries. Larger lists retain the bounded two-key hull because exact vector refinement otherwise costs elements times rows. -- Hilbert clustering: `pgcolumnar.cluster_hilbert` and - `pgcolumnar.recluster_hilbert` (#889). - - **Two verbs rather than a parameter on the existing two.** PostgreSQL refuses - to extend `cluster(regclass, VARIADIC name[])` in either direction: a defaulted - parameter cannot precede a `VARIADIC` one, and an array-plus-kind overload - makes the documented `cluster('t','a','b')` call ambiguous. Both were measured - on 18.4. The new verbs match their siblings element for element in argument - types, variadic element type, return type and volatility, so a caller switches - between them by name alone. - - **What the curve buys.** Z-order jumps a long way in key space at a bit - boundary; a Hilbert curve does not. Keys that are close in the data therefore - stay closer in storage, the min/max zone maps over the clustered columns are - tighter, and a range filter reads fewer chunk groups. The key is the same width - and sorts through the same `bytea` comparator, so nothing downstream of the - sort knows which curve produced it. - - **The curve is sticky.** `sorted_kind` is the table's declared intent, not a - property of each call: - - - plain `recluster` on a Hilbert table over the same key is a no-op returning - 0, not a silent conversion back to Z-order; - - `recluster_hilbert` on a Z-ordered table over the same columns rewrites it; - - `vacuum_sorted` leaves a Hilbert table alone rather than sorting it - lexicographically and relabelling it; - - the maintenance daemon dispatches on the recorded kind, so a Hilbert table - is re-clustered with Hilbert instead of being converted on a timer; - - naming the other verb, or reclustering on a different key, is how a table - changes curve. - - Held by `test/hilbert_cluster.sh` (181 arms) over the SQL surface, the recorded - kind, both self-gates and the daemon, and by `test/hilbert_curve.sh` (184 arms) - over the encoder itself. - - `test/projection_rewrite.sh`, 84 checks. Nothing in the tree asserted that a projection answers after a rewrite, which is why this was silent. diff --git a/README.md b/README.md index aae7fe53..ffcb3f8d 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ pgColumnar builds from one source tree on PostgreSQL 15 through 18, with 19 validated against 19beta2, and is licensed under the [MIT License](LICENSE). It is [pre-release](docs/limitations.md#release-status); the version marker is `1.0-alpha4`, recorded in `VERSION`. That version is in development and not -tagged; the latest published pre-release is `v1.0-alpha2`. A table `USING pgcolumnar` is stored in the +tagged; the latest published pre-release is `v1.0-alpha3`. A table `USING pgcolumnar` is stored in the native on-disk format, PGCN v1. ## Documentation diff --git a/RELEASE_NOTES_1.0-alpha4.md b/RELEASE_NOTES_1.0-alpha4.md new file mode 100644 index 00000000..fbbed170 --- /dev/null +++ b/RELEASE_NOTES_1.0-alpha4.md @@ -0,0 +1,150 @@ +# pgColumnar 1.0-alpha4 release notes + +Release date: 2026-09-17 +Previous release: 1.0-alpha3 (2026-09-02) + +pgColumnar is a columnar table access method for PostgreSQL. This is the fourth +alpha. Its theme is layout and skipping. A table can now be laid out on the +Hilbert curve, which keeps neighbouring keys closer together than Z-order does. A +star-schema join now skips fact-table groups and rejects non-matching rows, and it +does so without being asked. The on-disk native format, PGCN v1, is unchanged. +Existing tables are read and written as before. + +This release requires one upgrade command. See "Upgrading" at the end. The upgrade +is the smallest of any release so far: it adds two functions and changes nothing +else. + +## Highlights + +- **Hilbert clustering**. `pgcolumnar.cluster_hilbert` and + `pgcolumnar.recluster_hilbert` lay a table out on the Hilbert curve. The curve + has no jumps at a bit boundary, so a range filter reads fewer chunk groups. + Measured on 200,000 rows over two columns, Hilbert read 1.24x to 2.04x fewer + groups than Z-order. +- **Star-schema joins skip and reject by default**. A serial inner Hash Join now + uses the build-side keys to skip fact-table chunk groups and to reject + non-matching rows. `pgcolumnar.enable_join_runtime_filter` is on. +- **An index-driven read of a wide table does less I/O**. Adjacent column reads in + one row group are coalesced into a single read. +- **A parallel index build now uses its workers**. Every participant claims + distinct row groups. Before this release one backend read the whole table while + the launched workers sat idle. +- **The planner stops preferring a fetching index scan that does far more work**. + A correlated range over tens of thousands of rows now takes the columnar scan. + +## Hilbert clustering + +`pgcolumnar.cluster_hilbert(table, VARIADIC columns)` rewrites a table in Hilbert +order. It holds `AccessExclusiveLock`, as `CLUSTER` and `VACUUM FULL` do. +`pgcolumnar.recluster_hilbert(table, VARIADIC columns)` does the same work online +under `ShareUpdateExclusiveLock`, so reads and writes continue. + +The curve is sticky. Plain `pgcolumnar.recluster` maintains a Hilbert table rather +than converting it back. Naming the Hilbert verb is how a Z-ordered table is +switched to the curve. + +Choose the curve by the predicate. Z-order is fine for point lookups. Hilbert wins +on range filters over several columns, and the gap narrows as the query box grows. +Measure your own corpus when the two look close. See +[docs/best-practices.md](docs/best-practices.md) for the guidance and +`pgcolumnar.sort_status` for how much of a table is currently in order. + +There are two verbs rather than a parameter on the existing two because PostgreSQL +cannot extend `cluster(regclass, VARIADIC name[])` in either direction. A defaulted +parameter cannot precede a `VARIADIC` one. + +## Join acceleration + +A serial inner Hash Join over a columnar fact table now builds a filter from the +join keys it has already hashed. The filter does two things. A key range skips +whole fact-table chunk groups. A Bloom filter rejects rows that cannot match. + +Three measured cases decided the default: + +| fact table | result | +| --- | --- | +| clustered on the join key | 19 of 20 chunk groups removed, 1 read | +| scattered | 0 groups removed, Bloom rejects over 15,000 of 19,800 non-matches | +| build side too large | the Bloom disables itself | + +The third case is why this is on for everyone. A filter that helps nothing turns +itself off. Group skip still needs the fact table clustered on the join key. Set +`pgcolumnar.enable_join_runtime_filter` to `off` to compare. + +An ungrouped vectorized aggregate also keeps running over a unique-key inner Hash +Join. A unique dimension acts as a filter of the fact table, so the fold does not +have to stop. Duplicate-key dimensions and LEFT joins stay on the core plan. + +## Reads and the planner + +- **Coalesced column reads**. An index-driven fetch of several columns in one row + group issues one read for adjacent chunks instead of one per column. +- **Parallel index build**. The table access method's parallel scan claims row + groups per participant from the shared counter, the way the custom scan already + did. A parallel `CREATE INDEX` now spreads across its workers. +- **Parallel scan cost**. The planner no longer divides a parallel scan's I/O by + the worker count. PostgreSQL divides CPU across workers and leaves the disk work + whole, and the columnar cost now matches. +- **Clustered index fetch cost**. The fetch penalty now charges a per-row term, + capped at half a chunk group. Without it a correlated range of 50,000 rows stayed + on a fetching index scan while doing far more work than a scan. + +## Correctness fixes + +- **A truncated column chunk is refused rather than read**. A chunk whose recorded + length exceeded 4 GB was cast to 32 bits on the index-fetch path. A fetch could + therefore read the wrong bytes and report them as data. Both cast sites now + raise `XX001`. +- **A coalesced fetch cannot read past its buffer**. The validity bitmap copy is + now bounded by the chunk length before it runs. +- **Projections survive DDL**. A rewrite re-records its projections. `ALTER TABLE + ... RENAME COLUMN` carries the new name into the projection. `ALTER TABLE ... + DROP COLUMN` is refused when a projection depends on the column. An in-place + `TRUNCATE` clears each projection's storage as well as the base. +- **The block codec frees its buffer**. Both paths abandoned it. +- **Object storage refuses URL userinfo** on `s3://` and `gs://`, as `http(s)://` + has since #706. + +## Known issues + +- **Setting a codec can make a table larger, on high-entropy text** (#1074). The + writer keeps FSST only when it beats the alternative by + `fsst_min_gain_percent`, and it measures both sides after the block codec has + run. Storing the FSST codes uncompressed is never compared. Measured on 200,000 + rows of random hex text, `zstd` wrote 1.777% more than `compression = none`. + `lz4` was unaffected. If a table stores long high-entropy text and size matters, + measure both settings. +- **The block codec compresses a region it then discards** (#1075). On + incompressible data this costs about 25% more write CPU. It does not affect what + is stored or read. + +## Upgrading + +Install this build, then run the following in every database that has the +extension: + +```sql +ALTER EXTENSION pgcolumnar UPDATE; +``` + +This is required. The upgrade creates `pgcolumnar.cluster_hilbert` and +`pgcolumnar.recluster_hilbert`. It changes nothing else. No table data is +converted, no existing function is replaced, no catalog column is added, and no +SQL you write changes. + +See [docs/installation.md](docs/installation.md) for the commands, including how +to list the databases that need the update. + +## Scope and limitations + +- This is an alpha. Interfaces may change before 1.0. +- Hilbert clustering orders whole row groups on rewrite. A table that is written + to after the rewrite drifts out of order until the next one. +- The join runtime filter applies to a serial inner Hash Join on a direct columnar + scan. It does not wrap LEFT, SEMI, ANTI, CROSS, parallel, or projection scans. +- On PGXN this release is `1.0.0-alpha.4`, while `CREATE EXTENSION` reports + `1.0-alpha4`. PGXN requires a semantic version, which needs three integer + components. The extension's own version has two. The two names refer to the same + release. + +The complete, itemized list of changes is in `CHANGELOG.md`. diff --git a/design/RELEASE_PLAN_1.0.md b/design/RELEASE_PLAN_1.0.md index 4020693a..577c45f5 100644 --- a/design/RELEASE_PLAN_1.0.md +++ b/design/RELEASE_PLAN_1.0.md @@ -162,27 +162,49 @@ Nothing further should be added. It is 11 days into a 14-day cycle. Three defects found by the investigation are tracked separately and do NOT close with it: #1074, #1075, #1076. -### 1.0-alpha5, target 2026-09-29. Theme: join acceleration +### 1.0-alpha5, target 2026-09-29. Theme: encoding (adaptive cascade selection) + +**BOTH OF THIS ALPHA'S PLANNED ITEMS SHIPPED EARLY, IN ALPHA4**, so the encoding +item has been moved up from alpha6 to fill it. Recorded 2026-09-17 while auditing +the documentation for the alpha4 tag. Verified rather than assumed: each commit +below is absent from `v1.0-alpha3` and present in `main`. + + 9f7dcd8 perf: prune scattered IN lists by element (#752) + ae623cb feat: add serial join runtime range filter (#752) + cbd0c2e feat: add serial join runtime Bloom filter (#752) + 60ddc10 feat: turn join runtime filter on by default (#752) + +The two items as they were planned: - **Per-element evaluation of a set predicate** (#752). Already specified, with a measured ceiling. On a clustered fact table it reaches 9 to 12 chunk groups of 27, against 25 today. On an unclustered one the ceiling is provably zero. The design, the cost bound, the buffer constraint and the negative control are all - recorded on the issue. + recorded on the issue. **Shipped in alpha4 as `9f7dcd8`.** - **Runtime filters from a join's build side**, with clustering on the join key as a stated precondition rather than an assumption. The ceiling is zero without it, which is measured, so the documentation half of this item is as important as the - code. + code. **Shipped in alpha4, and on by default.** The precondition is documented in + `docs/how-to.md` and `docs/features.md`. + +This is a scheduling fact rather than a problem. The series compressed on +2026-08-29 to reach beta sooner, and work moving forward is that decision working. -### 1.0-alpha6, target 2026-10-13. Theme: encoding and interoperability +**What this alpha now carries**, moved up from alpha6: - **Adaptive cascade encoding selection.** The primitives exist; the missing piece is a sampling selector that chooses per block. High value at low to medium effort, and it changes what the writer emits. + +No date moves. alpha6 keeps Parquet partition inference and stays the last alpha. + +### 1.0-alpha6, target 2026-10-13. Theme: Parquet partition inference + - **Parquet partition inference**, the one remaining item inside Parquet. -Two items rather than three, deliberately. The Arrow C Data Interface export was -cut from this alpha on 2026-08-29. alpha6 is now the last one, and the series +One item. The Arrow C Data Interface export was cut from this alpha on 2026-08-29, +and adaptive cascade encoding selection moved up to alpha5 on 2026-09-17 when +alpha4 absorbed alpha5's join work. alpha6 is still the last one, and the series should not lose an item it needs to a slip in October. ## What compressing the series costs @@ -264,3 +286,46 @@ It does not claim the dates will hold. It claims three things. The cadence is observed rather than invented. Every item named is traceable to an issue or to `design/ROADMAP.md`. And the beta 1 test can be failed, which matters, because a plan whose entry criteria cannot be failed is a wish. + +## Cutting a release + +**This section exists because it did not, and alpha4 reached tag day with no +release notes**. Every previous release has a `RELEASE_NOTES_*.md`, so the step +was known and simply never written down. A step that lives only in someone's +memory is a step that gets skipped under time pressure. + +Documentation, in this order, before the tag: + +1. **Write `RELEASE_NOTES_.md`.** Model it on the previous one. Build the + highlights from what a USER can see, which means the upgrade script, the GUC + defaults, and the `feat`/`fix` commits touching `src/`. Do not build them from + the changelog's headline count: alpha4's `[Unreleased]` ran to 4,229 lines and + almost all of it was test-harness work. +2. **Check the changelog's categories.** Alpha4's headline feature sat under + `### Fixed`. Keep a Changelog wants Added for new surface, and a reader looking + for what is new will not find it under Fixed. +3. **Update the two version claims.** They are different sentences in different + places and only one of them is gated: + + ", recorded in `VERSION`" gated by docs_style.sh + "the latest published pre-release is" agreement-checked only + + The second drifted a whole cycle across README.md, docs/roadmap.md and + docs/installation.md while the first stayed green. `docs_style.sh` now checks + that every document making the claim makes the SAME claim, which catches one + file drifting. It cannot catch all of them being stale together, because no + tracked file records the newest tag. That is this step's job. +4. **Re-theme the next alpha if its scope shipped early.** Alpha5's two items both + landed in alpha4, so the published roadmap promised work already delivered. +5. **Run the gates.** `test/docs_style.sh` and `python3 + test/plain_language_check.py docs/*.md README.md`. Release notes are outside + `docs_style.sh`'s scope by design, so run the language check on the new file by + hand. + +Then the tag itself: + +6. `## [Unreleased]` becomes `## [] - `. +7. Tag, then capture the fixture FROM THE TAG (#901): + `git show v:pgcolumnar--.sql > test/fixtures/pgcolumnar--.sql`. + Capturing it from the working tree at the next cycle-open is how the alpha2 + fixture came to differ from what alpha2 actually shipped. diff --git a/docs/installation.md b/docs/installation.md index 3732adab..c31ec6e4 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -150,14 +150,15 @@ while columnar tables exist, because they depend on the access method. The form that succeeds is `DROP EXTENSION pgcolumnar CASCADE`, and it drops every columnar table with it. -### Upgrading to 1.0-alpha3 - -`1.0-alpha3` is what this source tree installs. It is in development and not -tagged; the latest published pre-release is `v1.0-alpha2`. `ALTER EXTENSION pgcolumnar UPDATE` (step 3 -above) reaches it from either previously published version: `1.0-dev`, which the -`v1.0-alpha` tag installed, or `1.0-alpha`. PostgreSQL applies the shipped upgrade -scripts in sequence, so a `1.0-dev` install is carried `1.0-dev` to `1.0-alpha` to -`1.0-alpha2` by that one command. +### Upgrading to 1.0-alpha4 + +`1.0-alpha4`, recorded in `VERSION`, is what this source tree installs. It is in +development and not tagged; the latest published pre-release is `v1.0-alpha3`. +`ALTER EXTENSION pgcolumnar UPDATE` (step 3 above) reaches it from every +previously published version: `1.0-dev`, which the `v1.0-alpha` tag installed, +`1.0-alpha`, `1.0-alpha2`, and `1.0-alpha3`. PostgreSQL applies the shipped +upgrade scripts in sequence. One command therefore carries a `1.0-dev` install +through `1.0-alpha`, `1.0-alpha2` and `1.0-alpha3` to `1.0-alpha4`. The `1.0-alpha` cycle renamed the extension's C symbols into the `pgcolumnar` namespace, so that two extensions named `columnar` can be loaded without diff --git a/docs/roadmap.md b/docs/roadmap.md index d866f2ef..75ea14f9 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -13,7 +13,7 @@ Issues are the authority on anything being worked now. pgColumnar is [pre-release](limitations.md#release-status). The version marker is `1.0-alpha4`, recorded in `VERSION`. That version is in development and not tagged; the -latest published pre-release is `v1.0-alpha2`. A table `USING pgcolumnar` is stored in the +latest published pre-release is `v1.0-alpha3`. A table `USING pgcolumnar` is stored in the native on-disk format, PGCN v1. ## Releases to 1.0 @@ -26,12 +26,18 @@ documentation, and tests. A feature deferred past beta 1 waits for 2.0. | release | target | theme | | --- | --- | --- | -| `1.0-alpha3` | 2026-09-01 | feature complete: retention, load deduplication, sort reporting, more skipping | -| `1.0-alpha4` | 2026-09-15 | skipping and layout | -| `1.0-alpha5` | 2026-09-29 | join acceleration | -| `1.0-alpha6` | 2026-10-13 | encoding and Parquet partition inference | +| `1.0-alpha3` | 2026-09-02 | feature complete: retention, load deduplication, sort reporting, more skipping | +| `1.0-alpha4` | 2026-09-17 | layout, skipping, and join acceleration | +| `1.0-alpha5` | 2026-09-29 | encoding: adaptive cascade selection | +| `1.0-alpha6` | 2026-10-13 | Parquet partition inference | | `1.0-beta1` | 2026-10-27 | feature freeze | +Alpha4 carries join acceleration because that work finished inside its cycle +rather than waiting for the alpha5 it was planned for. The serial join runtime +filter ships on by default, and per-element pruning of a set predicate ships with +it. The two items alpha6 held are therefore split across alpha5 and alpha6, and no +date moves. + Dates are a cadence, not a commitment. They follow the 14 days observed between `v1.0-alpha` and `v1.0-alpha2`, and each is the date a tag is cut. diff --git a/test/docs_style.sh b/test/docs_style.sh index ac486d49..692e6712 100755 --- a/test/docs_style.sh +++ b/test/docs_style.sh @@ -236,6 +236,43 @@ done check "every document citing VERSION quotes the version VERSION holds" \ "$(printf '%s' "$_stale" | sed 's/^ //')" "" +# ---- the OTHER version claim, which the check above cannot see -------------- +# +# A second version sentence sits beside the first: "the latest published +# pre-release is `vX`". It drifted for a whole cycle in THREE documents while the +# check above stayed green, because that check only looks at files carrying the +# phrase "recorded in `VERSION`", and this is a different sentence: +# +# README.md v1.0-alpha2 while v1.0-alpha3 was tagged +# docs/roadmap.md v1.0-alpha2 +# docs/installation.md v1.0-alpha2 and it named the tree's version wrong too +# CHANGELOG.md v1.0-alpha3 the only one right +# +# `docs/installation.md` is the one that shows why the scope mattered. It never +# writes the phrase "recorded in `VERSION`", so it was outside the other check +# entirely, and it also carried a stale heading and a stale upgrade chain. +# +# THIS IS AN AGREEMENT CHECK, NOT A COMPARISON, and that is a real limit. No +# tracked file holds "the newest tag" the way VERSION holds the version, and +# reading `git tag` fails in a tree copied without `.git`, which this harness is +# run from. So it catches one document drifting away from the others, which is +# what happened. It CANNOT catch every document being stale together, and the +# release procedure carries that step instead. +_pubdocs="$(grep -rln 'latest published pre-release' \ + "$SRCDIR/CHANGELOG.md" "$SRCDIR/README.md" "$SRCDIR/docs" 2>/dev/null | sort)" +check "premise: at least one document names the latest published pre-release" \ + "$([ -n "$_pubdocs" ] && echo yes || echo no)" "yes" + +# shellcheck disable=SC2086 +_pubvers="$(grep -rhoE 'latest published pre-release is `v[^`]*`' $_pubdocs 2>/dev/null \ + | grep -oE '`v[^`]*`' | tr -d '`' | sort -u)" +check "premise: the claim was parsed, not merely present" \ + "$([ -n "$_pubvers" ] && echo yes || echo no)" "yes" + +check "every document names the same latest published pre-release" \ + "$([ "$(printf '%s\n' "$_pubvers" | grep -c .)" = 1 ] && echo "" \ + || printf '%s' "$_pubvers" | tr '\n' ' ' | sed 's/ $//')" "" + # ---- and META.json, which NOTHING read at all ------------------------------ # # `META.json` is the PGXN distribution metadata. It hardcodes the version TWICE