docs: alpha4 release notes, and the three version claims that drifted

CHANGELOG.md

@@ -14,7 +14,7 @@ installed, `1.0-alpha`, `1.0-alpha2`, and `1.0-alpha3`), so a single
 notes in this file describe `default_version` as pinned at an earlier version, each
 true until the next version shipped.
 
-## [Unreleased]
+## [1.0-alpha4] - 2026-09-17
 
 ### Fixed
 
@@ -208,6 +208,41 @@ true until the next version shipped.
 
 ### Added
 
+- Hilbert clustering: `pgcolumnar.cluster_hilbert` and
+  `pgcolumnar.recluster_hilbert` (#889).
+
+  **Two verbs rather than a parameter on the existing two.** PostgreSQL refuses
+  to extend `cluster(regclass, VARIADIC name[])` in either direction: a defaulted
+  parameter cannot precede a `VARIADIC` one, and an array-plus-kind overload
+  makes the documented `cluster('t','a','b')` call ambiguous. Both were measured
+  on 18.4. The new verbs match their siblings element for element in argument
+  types, variadic element type, return type and volatility, so a caller switches
+  between them by name alone.
+
+  **What the curve buys.** Z-order jumps a long way in key space at a bit
+  boundary; a Hilbert curve does not. Keys that are close in the data therefore
+  stay closer in storage, the min/max zone maps over the clustered columns are
+  tighter, and a range filter reads fewer chunk groups. The key is the same width
+  and sorts through the same `bytea` comparator, so nothing downstream of the
+  sort knows which curve produced it.
+
+  **The curve is sticky.** `sorted_kind` is the table's declared intent, not a
+  property of each call:
+
+  - plain `recluster` on a Hilbert table over the same key is a no-op returning
+    0, not a silent conversion back to Z-order;
+  - `recluster_hilbert` on a Z-ordered table over the same columns rewrites it;
+  - `vacuum_sorted` leaves a Hilbert table alone rather than sorting it
+    lexicographically and relabelling it;
+  - the maintenance daemon dispatches on the recorded kind, so a Hilbert table
+    is re-clustered with Hilbert instead of being converted on a timer;
+  - naming the other verb, or reclustering on a different key, is how a table
+    changes curve.
+
+  Held by `test/hilbert_cluster.sh` (181 arms) over the SQL surface, the recorded
+  kind, both self-gates and the daemon, and by `test/hilbert_curve.sh` (184 arms)
+  over the encoder itself.
+
 - Six more guards counted their callers instead of pinning their property (#1078's
   class). Two were blind to the defect they name.
 
@@ -2275,41 +2310,6 @@ true until the next version shipped.
   Per-element evaluation is capped at 128 non-NULL entries. Larger lists retain
   the bounded two-key hull because exact vector refinement otherwise costs
   elements times rows.
-- Hilbert clustering: `pgcolumnar.cluster_hilbert` and
-  `pgcolumnar.recluster_hilbert` (#889).
-
-  **Two verbs rather than a parameter on the existing two.** PostgreSQL refuses
-  to extend `cluster(regclass, VARIADIC name[])` in either direction: a defaulted
-  parameter cannot precede a `VARIADIC` one, and an array-plus-kind overload
-  makes the documented `cluster('t','a','b')` call ambiguous. Both were measured
-  on 18.4. The new verbs match their siblings element for element in argument
-  types, variadic element type, return type and volatility, so a caller switches
-  between them by name alone.
-
-  **What the curve buys.** Z-order jumps a long way in key space at a bit
-  boundary; a Hilbert curve does not. Keys that are close in the data therefore
-  stay closer in storage, the min/max zone maps over the clustered columns are
-  tighter, and a range filter reads fewer chunk groups. The key is the same width
-  and sorts through the same `bytea` comparator, so nothing downstream of the
-  sort knows which curve produced it.
-
-  **The curve is sticky.** `sorted_kind` is the table's declared intent, not a
-  property of each call:
-
-  - plain `recluster` on a Hilbert table over the same key is a no-op returning
-    0, not a silent conversion back to Z-order;
-  - `recluster_hilbert` on a Z-ordered table over the same columns rewrites it;
-  - `vacuum_sorted` leaves a Hilbert table alone rather than sorting it
-    lexicographically and relabelling it;
-  - the maintenance daemon dispatches on the recorded kind, so a Hilbert table
-    is re-clustered with Hilbert instead of being converted on a timer;
-  - naming the other verb, or reclustering on a different key, is how a table
-    changes curve.
-
-  Held by `test/hilbert_cluster.sh` (181 arms) over the SQL surface, the recorded
-  kind, both self-gates and the daemon, and by `test/hilbert_curve.sh` (184 arms)
-  over the encoder itself.
-
 - `test/projection_rewrite.sh`, 84 checks. Nothing in the tree asserted that a
   projection answers after a rewrite, which is why this was silent.
 

README.md

@@ -25,7 +25,7 @@ pgColumnar builds from one source tree on PostgreSQL 15 through 18, with 19
 validated against 19beta2, and is
 licensed under the [MIT License](LICENSE). It is [pre-release](docs/limitations.md#release-status); the version marker
 is `1.0-alpha4`, recorded in `VERSION`. That version is in development and not
-tagged; the latest published pre-release is `v1.0-alpha2`. A table `USING pgcolumnar` is stored in the
+tagged; the latest published pre-release is `v1.0-alpha3`. A table `USING pgcolumnar` is stored in the
 native on-disk format, PGCN v1.
 
 ## Documentation

RELEASE_NOTES_1.0-alpha4.md

@@ -0,0 +1,150 @@
+# pgColumnar 1.0-alpha4 release notes
+
+Release date: 2026-09-17
+Previous release: 1.0-alpha3 (2026-09-02)
+
+pgColumnar is a columnar table access method for PostgreSQL. This is the fourth
+alpha. Its theme is layout and skipping. A table can now be laid out on the
+Hilbert curve, which keeps neighbouring keys closer together than Z-order does. A
+star-schema join now skips fact-table groups and rejects non-matching rows, and it
+does so without being asked. The on-disk native format, PGCN v1, is unchanged.
+Existing tables are read and written as before.
+
+This release requires one upgrade command. See "Upgrading" at the end. The upgrade
+is the smallest of any release so far: it adds two functions and changes nothing
+else.
+
+## Highlights
+
+- **Hilbert clustering**. `pgcolumnar.cluster_hilbert` and
+  `pgcolumnar.recluster_hilbert` lay a table out on the Hilbert curve. The curve
+  has no jumps at a bit boundary, so a range filter reads fewer chunk groups.
+  Measured on 200,000 rows over two columns, Hilbert read 1.24x to 2.04x fewer
+  groups than Z-order.
+- **Star-schema joins skip and reject by default**. A serial inner Hash Join now
+  uses the build-side keys to skip fact-table chunk groups and to reject
+  non-matching rows. `pgcolumnar.enable_join_runtime_filter` is on.
+- **An index-driven read of a wide table does less I/O**. Adjacent column reads in
+  one row group are coalesced into a single read.
+- **A parallel index build now uses its workers**. Every participant claims
+  distinct row groups. Before this release one backend read the whole table while
+  the launched workers sat idle.
+- **The planner stops preferring a fetching index scan that does far more work**.
+  A correlated range over tens of thousands of rows now takes the columnar scan.
+
+## Hilbert clustering
+
+`pgcolumnar.cluster_hilbert(table, VARIADIC columns)` rewrites a table in Hilbert
+order. It holds `AccessExclusiveLock`, as `CLUSTER` and `VACUUM FULL` do.
+`pgcolumnar.recluster_hilbert(table, VARIADIC columns)` does the same work online
+under `ShareUpdateExclusiveLock`, so reads and writes continue.
+
+The curve is sticky. Plain `pgcolumnar.recluster` maintains a Hilbert table rather
+than converting it back. Naming the Hilbert verb is how a Z-ordered table is
+switched to the curve.
+
+Choose the curve by the predicate. Z-order is fine for point lookups. Hilbert wins
+on range filters over several columns, and the gap narrows as the query box grows.
+Measure your own corpus when the two look close. See
+[docs/best-practices.md](docs/best-practices.md) for the guidance and
+`pgcolumnar.sort_status` for how much of a table is currently in order.
+
+There are two verbs rather than a parameter on the existing two because PostgreSQL
+cannot extend `cluster(regclass, VARIADIC name[])` in either direction. A defaulted
+parameter cannot precede a `VARIADIC` one.
+
+## Join acceleration
+
+A serial inner Hash Join over a columnar fact table now builds a filter from the
+join keys it has already hashed. The filter does two things. A key range skips
+whole fact-table chunk groups. A Bloom filter rejects rows that cannot match.
+
+Three measured cases decided the default:
+
+| fact table | result |
+| --- | --- |
+| clustered on the join key | 19 of 20 chunk groups removed, 1 read |
+| scattered | 0 groups removed, Bloom rejects over 15,000 of 19,800 non-matches |
+| build side too large | the Bloom disables itself |
+
+The third case is why this is on for everyone. A filter that helps nothing turns
+itself off. Group skip still needs the fact table clustered on the join key. Set
+`pgcolumnar.enable_join_runtime_filter` to `off` to compare.
+
+An ungrouped vectorized aggregate also keeps running over a unique-key inner Hash
+Join. A unique dimension acts as a filter of the fact table, so the fold does not
+have to stop. Duplicate-key dimensions and LEFT joins stay on the core plan.
+
+## Reads and the planner
+
+- **Coalesced column reads**. An index-driven fetch of several columns in one row
+  group issues one read for adjacent chunks instead of one per column.
+- **Parallel index build**. The table access method's parallel scan claims row
+  groups per participant from the shared counter, the way the custom scan already
+  did. A parallel `CREATE INDEX` now spreads across its workers.
+- **Parallel scan cost**. The planner no longer divides a parallel scan's I/O by
+  the worker count. PostgreSQL divides CPU across workers and leaves the disk work
+  whole, and the columnar cost now matches.
+- **Clustered index fetch cost**. The fetch penalty now charges a per-row term,
+  capped at half a chunk group. Without it a correlated range of 50,000 rows stayed
+  on a fetching index scan while doing far more work than a scan.
+
+## Correctness fixes
+
+- **A truncated column chunk is refused rather than read**. A chunk whose recorded
+  length exceeded 4 GB was cast to 32 bits on the index-fetch path. A fetch could
+  therefore read the wrong bytes and report them as data. Both cast sites now
+  raise `XX001`.
+- **A coalesced fetch cannot read past its buffer**. The validity bitmap copy is
+  now bounded by the chunk length before it runs.
+- **Projections survive DDL**. A rewrite re-records its projections. `ALTER TABLE
+  ... RENAME COLUMN` carries the new name into the projection. `ALTER TABLE ...
+  DROP COLUMN` is refused when a projection depends on the column. An in-place
+  `TRUNCATE` clears each projection's storage as well as the base.
+- **The block codec frees its buffer**. Both paths abandoned it.
+- **Object storage refuses URL userinfo** on `s3://` and `gs://`, as `http(s)://`
+  has since #706.
+
+## Known issues
+
+- **Setting a codec can make a table larger, on high-entropy text** (#1074). The
+  writer keeps FSST only when it beats the alternative by
+  `fsst_min_gain_percent`, and it measures both sides after the block codec has
+  run. Storing the FSST codes uncompressed is never compared. Measured on 200,000
+  rows of random hex text, `zstd` wrote 1.777% more than `compression = none`.
+  `lz4` was unaffected. If a table stores long high-entropy text and size matters,
+  measure both settings.
+- **The block codec compresses a region it then discards** (#1075). On
+  incompressible data this costs about 25% more write CPU. It does not affect what
+  is stored or read.
+
+## Upgrading
+
+Install this build, then run the following in every database that has the
+extension:
+
+```sql
+ALTER EXTENSION pgcolumnar UPDATE;
+```
+
+This is required. The upgrade creates `pgcolumnar.cluster_hilbert` and
+`pgcolumnar.recluster_hilbert`. It changes nothing else. No table data is
+converted, no existing function is replaced, no catalog column is added, and no
+SQL you write changes.
+
+See [docs/installation.md](docs/installation.md) for the commands, including how
+to list the databases that need the update.
+
+## Scope and limitations
+
+- This is an alpha. Interfaces may change before 1.0.
+- Hilbert clustering orders whole row groups on rewrite. A table that is written
+  to after the rewrite drifts out of order until the next one.
+- The join runtime filter applies to a serial inner Hash Join on a direct columnar
+  scan. It does not wrap LEFT, SEMI, ANTI, CROSS, parallel, or projection scans.
+- On PGXN this release is `1.0.0-alpha.4`, while `CREATE EXTENSION` reports
+  `1.0-alpha4`. PGXN requires a semantic version, which needs three integer
+  components. The extension's own version has two. The two names refer to the same
+  release.
+
+The complete, itemized list of changes is in `CHANGELOG.md`.

design/RELEASE_PLAN_1.0.md

@@ -162,27 +162,49 @@ Nothing further should be added. It is 11 days into a 14-day cycle.
   Three defects found by the investigation are tracked separately and do NOT
   close with it: #1074, #1075, #1076.
 
-### 1.0-alpha5, target 2026-09-29. Theme: join acceleration
+### 1.0-alpha5, target 2026-09-29. Theme: encoding (adaptive cascade selection)
+
+**BOTH OF THIS ALPHA'S PLANNED ITEMS SHIPPED EARLY, IN ALPHA4**, so the encoding
+item has been moved up from alpha6 to fill it. Recorded 2026-09-17 while auditing
+the documentation for the alpha4 tag. Verified rather than assumed: each commit
+below is absent from `v1.0-alpha3` and present in `main`.
+
+    9f7dcd8  perf: prune scattered IN lists by element (#752)
+    ae623cb  feat: add serial join runtime range filter (#752)
+    cbd0c2e  feat: add serial join runtime Bloom filter (#752)
+    60ddc10  feat: turn join runtime filter on by default (#752)
+
+The two items as they were planned:
 
 - **Per-element evaluation of a set predicate** (#752). Already specified, with a
   measured ceiling. On a clustered fact table it reaches 9 to 12 chunk groups of
   27, against 25 today. On an unclustered one the ceiling is provably zero. The
   design, the cost bound, the buffer constraint and the negative control are all
-  recorded on the issue.
+  recorded on the issue. **Shipped in alpha4 as `9f7dcd8`.**
 - **Runtime filters from a join's build side**, with clustering on the join key as
   a stated precondition rather than an assumption. The ceiling is zero without it,
   which is measured, so the documentation half of this item is as important as the
-  code.
+  code. **Shipped in alpha4, and on by default.** The precondition is documented in
+  `docs/how-to.md` and `docs/features.md`.
+
+This is a scheduling fact rather than a problem. The series compressed on
+2026-08-29 to reach beta sooner, and work moving forward is that decision working.
 
-### 1.0-alpha6, target 2026-10-13. Theme: encoding and interoperability
+**What this alpha now carries**, moved up from alpha6:
 
 - **Adaptive cascade encoding selection.** The primitives exist; the missing piece
   is a sampling selector that chooses per block. High value at low to medium
   effort, and it changes what the writer emits.
+
+No date moves. alpha6 keeps Parquet partition inference and stays the last alpha.
+
+### 1.0-alpha6, target 2026-10-13. Theme: Parquet partition inference
+
 - **Parquet partition inference**, the one remaining item inside Parquet.
 
-Two items rather than three, deliberately. The Arrow C Data Interface export was
-cut from this alpha on 2026-08-29. alpha6 is now the last one, and the series
+One item. The Arrow C Data Interface export was cut from this alpha on 2026-08-29,
+and adaptive cascade encoding selection moved up to alpha5 on 2026-09-17 when
+alpha4 absorbed alpha5's join work. alpha6 is still the last one, and the series
 should not lose an item it needs to a slip in October.
 
 ## What compressing the series costs
@@ -264,3 +286,46 @@ It does not claim the dates will hold. It claims three things. The cadence is
 observed rather than invented. Every item named is traceable to an issue or to
 `design/ROADMAP.md`. And the beta 1 test can be failed, which matters, because a
 plan whose entry criteria cannot be failed is a wish.
+
+## Cutting a release
+
+**This section exists because it did not, and alpha4 reached tag day with no
+release notes**. Every previous release has a `RELEASE_NOTES_*.md`, so the step
+was known and simply never written down. A step that lives only in someone's
+memory is a step that gets skipped under time pressure.
+
+Documentation, in this order, before the tag:
+
+1. **Write `RELEASE_NOTES_<version>.md`.** Model it on the previous one. Build the
+   highlights from what a USER can see, which means the upgrade script, the GUC
+   defaults, and the `feat`/`fix` commits touching `src/`. Do not build them from
+   the changelog's headline count: alpha4's `[Unreleased]` ran to 4,229 lines and
+   almost all of it was test-harness work.
+2. **Check the changelog's categories.** Alpha4's headline feature sat under
+   `### Fixed`. Keep a Changelog wants Added for new surface, and a reader looking
+   for what is new will not find it under Fixed.
+3. **Update the two version claims.** They are different sentences in different
+   places and only one of them is gated:
+
+       "<version>, recorded in `VERSION`"        gated by docs_style.sh
+       "the latest published pre-release is"     agreement-checked only
+
+   The second drifted a whole cycle across README.md, docs/roadmap.md and
+   docs/installation.md while the first stayed green. `docs_style.sh` now checks
+   that every document making the claim makes the SAME claim, which catches one
+   file drifting. It cannot catch all of them being stale together, because no
+   tracked file records the newest tag. That is this step's job.
+4. **Re-theme the next alpha if its scope shipped early.** Alpha5's two items both
+   landed in alpha4, so the published roadmap promised work already delivered.
+5. **Run the gates.** `test/docs_style.sh` and `python3
+   test/plain_language_check.py docs/*.md README.md`. Release notes are outside
+   `docs_style.sh`'s scope by design, so run the language check on the new file by
+   hand.
+
+Then the tag itself:
+
+6. `## [Unreleased]` becomes `## [<version>] - <date>`.
+7. Tag, then capture the fixture FROM THE TAG (#901):
+   `git show v<version>:pgcolumnar--<version>.sql > test/fixtures/pgcolumnar--<version>.sql`.
+   Capturing it from the working tree at the next cycle-open is how the alpha2
+   fixture came to differ from what alpha2 actually shipped.

docs/installation.md

@@ -150,14 +150,15 @@ while columnar tables exist, because they depend on the access method. The
 form that succeeds is `DROP EXTENSION pgcolumnar CASCADE`, and it drops every
 columnar table with it.
 
-### Upgrading to 1.0-alpha3
-
-`1.0-alpha3` is what this source tree installs. It is in development and not
-tagged; the latest published pre-release is `v1.0-alpha2`. `ALTER EXTENSION pgcolumnar UPDATE` (step 3
-above) reaches it from either previously published version: `1.0-dev`, which the
-`v1.0-alpha` tag installed, or `1.0-alpha`. PostgreSQL applies the shipped upgrade
-scripts in sequence, so a `1.0-dev` install is carried `1.0-dev` to `1.0-alpha` to
-`1.0-alpha2` by that one command.
+### Upgrading to 1.0-alpha4
+
+`1.0-alpha4`, recorded in `VERSION`, is what this source tree installs. It is in
+development and not tagged; the latest published pre-release is `v1.0-alpha3`.
+`ALTER EXTENSION pgcolumnar UPDATE` (step 3 above) reaches it from every
+previously published version: `1.0-dev`, which the `v1.0-alpha` tag installed,
+`1.0-alpha`, `1.0-alpha2`, and `1.0-alpha3`. PostgreSQL applies the shipped
+upgrade scripts in sequence. One command therefore carries a `1.0-dev` install
+through `1.0-alpha`, `1.0-alpha2` and `1.0-alpha3` to `1.0-alpha4`.
 
 The `1.0-alpha` cycle renamed the extension's C symbols into the `pgcolumnar`
 namespace, so that two extensions named `columnar` can be loaded without