From 2ae19d6687d64964af14881c3f0c99bb8c710246 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sat, 12 Sep 2026 12:55:40 -0600 Subject: [PATCH 1/4] docs: name 1024 as the floor for stripe_row_limit (#1017) A vector is a fixed 1024 values (COLUMNAR_NATIVE_VECTOR_LENGTH), so a row group smaller than one never fills it, the chunk-shared FSST table is not built, and a text column is stored plain. Measured, 200,000 rows, compression=none, against 12,800,000 raw bytes, two identical passes: stripe_row_limit 1000 0 FSST tables 13,625,000 106.4% of raw stripe_row_limit 1200 166 of 167 6,998,031 54.7% of raw stripe_row_limit 2000 100 of 100 6,990,641 54.6% of raw THE ACCEPTED MINIMUM IS 1000, enforced in set_options, so the most aggressive legal setting is the one that pays this -- and administration.md tells a reader to LOWER the setting for point-lookup-heavy tables, which is the path in. The warning is in that block, not only in a reference table. Documentation only. Whether to raise the minimum, make FSST work below a vector, or warn at set_options stays open on #1017. THE GUARD WAS BORN GREEN TWICE AND THE MEASUREMENT IS WHY IT IS NOT NOW: blank-line block passes on main: configuration.md's GUC table has no blank lines, so stripe_row_limit's row shares a block with chunk_group_row_limit's "fixed 1024-value vectors" 3-line window passes on main: those rows are adjacent one line, both 0 on all three pages on main Requiring one line is also a claim about the PROSE -- the floor has to be stated in a sentence rather than inferred from two neighbouring tokens -- and it is why best-practices.md was reworded to name the setting and the number together. THE TWO HARNESSES DISAGREED AND THE SHELL ONE WAS WRONG. The awk arm used paragraph mode and passed on main for two pages; the python twin split on blank lines and did not. That is the argument for keeping both halves, paid back the day it was written. Removal proof: restore main's three pages and all three arms go red, both harnesses. the 14 database-free files 277 passed, 668 checks, 0 fail Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- CHANGELOG.md | 30 +++++++++++++ docs/administration.md | 6 +++ docs/best-practices.md | 5 ++- docs/configuration.md | 18 +++++++- test/docs_style.sh | 42 +++++++++++++++++ test/pytest/TESTS.md | 53 ++++++++++++++++++++++ test/pytest/test_docs_stripe_floor.py | 65 +++++++++++++++++++++++++++ test/pytest/test_harness_deps.py | 3 ++ 8 files changed, 220 insertions(+), 2 deletions(-) create mode 100644 test/pytest/test_docs_stripe_floor.py diff --git a/CHANGELOG.md b/CHANGELOG.md index d11ade72..66e2702a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,36 @@ true until the next version shipped. ### Added +- The docs name 1024 as the floor for `stripe_row_limit` (#1017). + + A vector is a fixed 1024 values, so a row group smaller than one never fills it and + the chunk-shared FSST symbol table is not built. Measured on 200,000 rows of a text + column, `compression = none`, against 12,800,000 raw bytes, two identical passes: + + stripe_row_limit 1000 0 FSST tables 13,625,000 106.4% of raw + stripe_row_limit 1200 166 of 167 6,998,031 54.7% of raw + stripe_row_limit 2000 100 of 100 6,990,641 54.6% of raw + + At 1000 the column costs more than storing the bytes uncompressed. THE ACCEPTED + MINIMUM IS 1000, enforced in `set_options`, so the most aggressive legal setting is + the one that pays this -- and `docs/administration.md` tells a reader to LOWER this + setting for point-lookup-heavy tables, which is the path in. The warning now sits in + that block rather than in a reference table. + + Documentation only. The minimum is unchanged: whether to raise it, make FSST work + below a vector, or warn at `set_options` is still open on #1017. + + THE GUARD WAS BORN GREEN TWICE BEFORE IT WORKED, and the measurement is why it does + now. A blank-line block reader passed on main, because `configuration.md`'s GUC table + has no blank lines and `stripe_row_limit`'s row shares a block with + `chunk_group_row_limit`'s "fixed 1024-value vectors". A three-line proximity window + passed for the same reason. One line naming both is 0 on all three pages on main, and + it is also a claim about the prose: the floor has to be stated in a sentence. + + The two harnesses disagreed while that was being found -- the awk arm used paragraph + mode and passed on main for two pages, the python twin split on blank lines and did + not -- which is the argument for keeping both halves, paid back the day it was written. + - A check record names the PostgreSQL major it was observed under (#1010). RESULTsuitepartnameverdictmajorreason diff --git a/docs/administration.md b/docs/administration.md index 0d195ac8..f2e73a2c 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -55,6 +55,12 @@ each fetch expensive. Lower this setting for a table that takes many point lookups. `pgcolumnar.chunk_group_row_limit` does not change this cost. Use `stripe_row_limit` for this, not `chunk_group_row_limit`. +Do not lower `pgcolumnar.stripe_row_limit` below **1024**. A vector is a fixed 1024 +values, so a smaller row group never fills one and FSST is not applied to text +columns. Measured at the +accepted minimum of 1000, a text column stored at 106.4% of its raw bytes against +54.7% at 1200 (#1017). Lower it to 1024 or above, not to the floor. + Measured on 500,000 rows of 1 KiB incompressible data, which is the shape where the effect is largest: diff --git a/docs/best-practices.md b/docs/best-practices.md index 54a5f4eb..6f937550 100644 --- a/docs/best-practices.md +++ b/docs/best-practices.md @@ -27,7 +27,10 @@ own. Small transactions produce small, poorly compressed row groups, and many of them to scan later. Prefer `COPY` or a multi-row `INSERT ... SELECT` over row-at-a-time inserts. Load in batches that fill a row group (`pgcolumnar.stripe_row_limit`, default 150000 rows), so each row group compresses -well. +well. Keep `pgcolumnar.stripe_row_limit` at 1024 or above if you lower it: a vector +is a fixed 1024 values, and a row group below one vector gets no FSST on its text +columns, which cost 106.4% of their raw bytes at the accepted minimum of 1000 +against 54.7% at 1200 (#1017). **Use `parallel_copy` for a large file.** `pgcolumnar.parallel_copy` splits a server-side file across workers and scales the load into one table. It runs a diff --git a/docs/configuration.md b/docs/configuration.md index 4afa88a9..c7dc8f3a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -19,7 +19,7 @@ pgColumnar has two kinds of settings: | Setting | Type | Default | Description | | --- | --- | --- | --- | -| `pgcolumnar.stripe_row_limit` | integer | `150000` | Maximum rows per row group. The row group is the unit of write and the granularity at which whole segments are appended. Range 1000 to INT_MAX. | +| `pgcolumnar.stripe_row_limit` | integer | `150000` | Maximum rows per row group. The row group is the unit of write and the granularity at which whole segments are appended. Range 1000 to INT_MAX, but see the note below: a value under 1024 costs text compression. | | `pgcolumnar.chunk_group_row_limit` | integer | `10000` | Maximum rows per chunk group. The chunk group is the band a scan skips as a unit when a filter cannot match its minimum and maximum. Within a chunk group each column is encoded in fixed 1024-value vectors. Range 100 to INT_MAX. | | `pgcolumnar.encoding_sample_rows` | integer | `2048` | The number of rows that the writer samples to select the value encoding of a vector. The writer estimates each candidate on a sample of windows. The windows contain consecutive values and have an equal distance between them. Thus the sample shows the global shape and also the local runs. The writer then applies only the two best candidates to the full vector. A value of `0` applies each candidate to each vector. This is the behaviour of earlier versions. The writer changes a value below 128 to `0`, because a smaller sample cannot put the candidates in order. This setting changes the write speed. It can also change the compression ratio. It does not change correctness. | @@ -156,6 +156,22 @@ SELECT pgcolumnar.set_options( | `table_name` | regclass | The columnar table to change. Anything that is not an ordinary table using the `pgcolumnar` access method is rejected, including a partitioned table. | | `chunk_group_row_limit` | integer | Per-table override of `pgcolumnar.chunk_group_row_limit`. | | `stripe_row_limit` | integer | Per-table override of `pgcolumnar.stripe_row_limit`. | + +**A `stripe_row_limit` below 1024 disables FSST on text columns.** A vector is a +fixed 1024 values, so a row group smaller than that never fills one, and the +chunk-shared FSST symbol table is not built. Measured on 200,000 rows of a text +column, `compression = none`, against 12,800,000 raw bytes: + +| `stripe_row_limit` | FSST tables | stored | of raw | +| --- | --- | --- | --- | +| 1000 | 0 | 13,625,000 | **106.4%** | +| 1200 | 166 of 167 | 6,998,031 | 54.7% | +| 2000 | 100 of 100 | 6,990,641 | 54.6% | + +At 1000 the column costs more than storing the bytes uncompressed. The accepted +minimum is 1000 and the vector is 1024, so **the most aggressive legal setting is +the one that pays this cost**. Use 1024 or more unless you have measured that you +want the opposite (#1017). | `compression` | name | One of `none`, `pglz`, `lz4`, `zstd`. | | `compression_level` | integer | Level for the `zstd` codec, 1 to 22. | | `encode_effort` | name | `full` (default) or `fast`. How much work the writer spends choosing an encoding. See below. | diff --git a/test/docs_style.sh b/test/docs_style.sh index f4fd677e..7d1369c0 100755 --- a/test/docs_style.sh +++ b/test/docs_style.sh @@ -148,6 +148,48 @@ PY check "best-practices names clustering on the join key" \ "$(_practices_jk)" "yes" +# ---- the stripe floor is below a vector, and the pages must say so (#1017) ---- +# +# A vector is a fixed 1024 values (COLUMNAR_NATIVE_VECTOR_LENGTH). A row group +# smaller than one never fills it and FSST is not applied to text columns. +# Measured, 200,000 rows, compression=none, against 12,800,000 raw bytes: +# +# stripe_row_limit 1000 0 FSST tables 13,625,000 106.4% of raw +# stripe_row_limit 1200 166 tables 6,998,031 54.7% of raw +# +# The ACCEPTED MINIMUM IS 1000, so the most aggressive legal setting is the one +# that pays this, and administration.md tells a reader to LOWER the setting for +# point lookups. The warning has to sit in the block that gives that advice, not +# in a reference table three pages away -- so these arms are scoped to the block +# and not to the page. An earlier version grepped whole pages and passed on main, +# which already says 1024 and FSST elsewhere. +# ONE LINE CARRYING BOTH, which is the claim stated in one sentence rather than two +# tokens that happen to be near each other. +# +# Two weaker versions were born green and the measurement is why they were replaced. +# awk paragraph mode (RS='') made configuration.md's whole GUC table one record, so +# `stripe_row_limit`'s row shared a record with `chunk_group_row_limit`'s "fixed +# 1024-value vectors" -- green on main. A three-line proximity window failed the same +# way, because those rows are adjacent. Measured on main: same-line is 0 for all +# three pages, which is the only one of the three that is actually red there. +# +# The pytest twin splits on blank lines in python and did NOT have the paragraph +# behaviour, so the two harnesses disagreed and the shell one was wrong. That is the +# argument for keeping both, paid back the day it was written. +_floor_line() { # _floor_line FILE -- yes if one line names the setting and the floor + if grep -qE 'stripe_row_limit.*1024|1024.*stripe_row_limit' "$1"; then + echo yes + else + echo no + fi +} +check "configuration.md states the 1024 floor where it documents the setting" \ + "$(_floor_line "$SRCDIR/docs/configuration.md")" "yes" +check "administration.md states it beside the advice to lower the setting" \ + "$(_floor_line "$SRCDIR/docs/administration.md")" "yes" +check "best-practices.md carries it with the load-sizing advice" \ + "$(_floor_line "$SRCDIR/docs/best-practices.md")" "yes" + # ---- a document that quotes the version must quote the current one ---------- # # Nothing reads the VERSION file mechanically: no Makefile rule, no CI step. Two diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 70d2643e..1d7356c7 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -75,6 +75,7 @@ behaviour, the source of that number is named. - [27. test_skip_loop_arms.py: a skipped arm records under its own name](#27-test_skip_loop_armspy-a-skipped-arm-records-under-its-own-name) - [28. test_docs_join_clustering.py: the runtime filter's layout precondition](#28-test_docs_join_clusteringpy-the-runtime-filters-layout-precondition) - [29. test_join_vector_agg.py: ungrouped fold over a unique-key join](#29-test_join_vector_aggpy-ungrouped-fold-over-a-unique-key-join) +- [30. test_docs_stripe_floor.py: the stripe floor is below a vector](#30-test_docs_stripe_floorpy-the-stripe-floor-is-below-a-vector) ## 1. How to read a test in here @@ -2797,3 +2798,55 @@ join. A non-equi join clause is the same kind of extra Join Filter. EXPLAIN has no vectorized agg node. The sum matches a heap twin. + + +## 30. test_docs_stripe_floor.py: the stripe floor is below a vector + +A vector is a fixed 1024 values (`COLUMNAR_NATIVE_VECTOR_LENGTH`). A row group +smaller than one never fills it, so the chunk-shared FSST symbol table is not built +and a text column is stored plain. + +Measured on 200,000 rows, one text column, `compression = none`, against 12,800,000 +raw bytes, two identical passes: + +| `stripe_row_limit` | FSST tables | stored | of raw | +| --- | --- | --- | --- | +| 1000 | 0 | 13,625,000 | **106.4%** | +| 1200 | 166 of 167 | 6,998,031 | 54.7% | +| 2000 | 100 of 100 | 6,990,641 | 54.6% | + +**The accepted minimum is 1000**, enforced in `set_options`, so the most aggressive +legal setting is the one that pays this — and at it the column costs more than +storing the bytes uncompressed. `docs/administration.md` tells a reader to *lower* +this setting for point-lookup-heavy tables, which is the path in, so the warning has +to sit in the block that gives the advice rather than in a reference table. + +### Why every arm asserts ONE LINE, not a block or a window + +Two weaker signals were tried and **both were born green on `main`**: + +| signal | why it passed on main | +| --- | --- | +| blank-line block | `configuration.md`'s GUC table has no blank lines, so `stripe_row_limit`'s row shares a block with `chunk_group_row_limit`'s "fixed 1024-value vectors" | +| three-line window | those same rows are adjacent | +| **one line naming both** | **0 on all three pages on `main`** | + +Requiring one line is also a claim about the prose: the floor has to be stated in a +sentence, not inferred from two tokens that happen to be neighbours. + +**The two harnesses disagreed, and the shell one was wrong.** The awk arm used +paragraph mode and passed on `main` for two pages; the python twin split on blank +lines and did not. That is the argument for keeping both halves, paid back the day +it was written. + +| test | what it pins | +| --- | --- | +| `test_configuration_states_the_floor_where_it_documents_the_setting` | the floor is on the setting's own line | +| `test_administration_states_it_beside_the_advice_to_lower_it` | it is on the page that tells readers to lower the setting, naming what is lost | +| `test_best_practices_carries_the_floor_with_the_load_sizing_advice` | the load-sizing guidance states it too | + +### Removal proof + +Restore `main`'s three pages and **all three** arms go red. The shell twin is three +arms in `docs_style.sh`; the two halves share no code, one matching in `grep` and the +other in Python. diff --git a/test/pytest/test_docs_stripe_floor.py b/test/pytest/test_docs_stripe_floor.py new file mode 100644 index 00000000..56cb2723 --- /dev/null +++ b/test/pytest/test_docs_stripe_floor.py @@ -0,0 +1,65 @@ +"""The docs must name 1024 as the floor for `stripe_row_limit` (#1017). + +A vector is a fixed 1024 values (`COLUMNAR_NATIVE_VECTOR_LENGTH`), so a row group +smaller than one vector never fills one and the chunk-shared FSST symbol table is +not built. Measured on 200,000 rows of a text column, `compression = none`: + + stripe_row_limit 1000 0 FSST tables 13,625,000 106.4% of raw + stripe_row_limit 1200 166 tables 6,998,031 54.7% of raw + +The accepted minimum is 1000, so the most aggressive legal setting is the one that +pays this. `docs/administration.md` tells a reader to LOWER this setting for +point-lookup-heavy tables, which is the path into it, so the warning has to live +beside that advice and not only in a reference table. + +Public seam: the three published pages. Read independently of docs_style.sh -- +this parses the pages itself rather than sharing a helper with the shell arm. +""" + +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +CONFIG = ROOT / "docs" / "configuration.md" +ADMIN = ROOT / "docs" / "administration.md" +PRACTICES = ROOT / "docs" / "best-practices.md" + + +def _states_the_floor(path): + """True when ONE LINE names the setting and the floor together. + + Not a block, and not a proximity window. Both were tried and both were born + green: `configuration.md`'s GUC table has no blank lines, so a paragraph reader + puts `stripe_row_limit`'s row and `chunk_group_row_limit`'s "fixed 1024-value + vectors" in one unit, and those rows are adjacent so a three-line window does + the same. Measured on `main`: block and window both say yes, one line says no + for all three pages. + + Requiring one line is also a claim about the PROSE -- the floor has to be stated + in a sentence, not inferred from two tokens that happen to be neighbours. + """ + for line in path.read_text(encoding="utf-8").splitlines(): + if "stripe_row_limit" in line and "1024" in line: + return True + return False + + +def test_configuration_states_the_floor_where_it_documents_the_setting(expect): + expect.num(int(CONFIG.is_file()), 1, "premise: configuration.md is in the tree") + expect.num(int(_states_the_floor(CONFIG)), 1, + "configuration.md states the 1024 floor on the setting's own line") + + +def test_administration_states_it_beside_the_advice_to_lower_it(expect): + """`administration.md` tells a reader to LOWER this setting for point lookups. + That is the path into the cliff, so the floor has to be on this page.""" + expect.num(int(ADMIN.is_file()), 1, "premise: administration.md is in the tree") + expect.num(int(_states_the_floor(ADMIN)), 1, + "administration.md states the floor beside the lowering advice") + low = ADMIN.read_text(encoding="utf-8").lower() + expect.num(int("fsst" in low), 1, "and names what lowering past it costs") + + +def test_best_practices_carries_the_floor_with_the_load_sizing_advice(expect): + expect.num(int(PRACTICES.is_file()), 1, "premise: best-practices.md is in the tree") + expect.num(int(_states_the_floor(PRACTICES)), 1, + "the load-sizing advice states the floor on the same line") diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index 7f24030b..37984435 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -105,6 +105,9 @@ # #752 docs. Reads docs/how-to.md and docs/best-practices.md. No cluster, # no driver: the public seam is the published page. "test_docs_join_clustering.py", + # #1017 docs. Reads configuration.md, administration.md and best-practices.md. + # No cluster, no driver: the public seam is the published page. + "test_docs_stripe_floor.py", ] From a8c1048c2854752eaa28d79946b19a5a52591229 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sat, 12 Sep 2026 13:06:47 -0600 Subject: [PATCH 2/4] docs: renumber the TESTS.md section to 31, since #1020 takes 30 #1020 rebased onto main and took section 30 for test_differential.py. This branch claimed the same number. Heading, contents entry and anchor moved together. FOURTH TIME TODAY on this one anchor: #1007 and #1012 both claimed 28, #1012 and #1020 both claimed 29, and now this and #1020 both claimed 30. Every change appends a section with the next sequential number, so any two open at once collide by construction. That is #996's shape in TESTS.md rather than in the CHANGELOG, and it is worth saying that renumbering by hand each time is the cost of not fixing it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- test/pytest/TESTS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 1d7356c7..3279dae4 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -75,7 +75,7 @@ behaviour, the source of that number is named. - [27. test_skip_loop_arms.py: a skipped arm records under its own name](#27-test_skip_loop_armspy-a-skipped-arm-records-under-its-own-name) - [28. test_docs_join_clustering.py: the runtime filter's layout precondition](#28-test_docs_join_clusteringpy-the-runtime-filters-layout-precondition) - [29. test_join_vector_agg.py: ungrouped fold over a unique-key join](#29-test_join_vector_aggpy-ungrouped-fold-over-a-unique-key-join) -- [30. test_docs_stripe_floor.py: the stripe floor is below a vector](#30-test_docs_stripe_floorpy-the-stripe-floor-is-below-a-vector) +- [31. test_docs_stripe_floor.py: the stripe floor is below a vector](#31-test_docs_stripe_floorpy-the-stripe-floor-is-below-a-vector) ## 1. How to read a test in here @@ -2800,7 +2800,7 @@ A non-equi join clause is the same kind of extra Join Filter. EXPLAIN has no vectorized agg node. The sum matches a heap twin. -## 30. test_docs_stripe_floor.py: the stripe floor is below a vector +## 31. test_docs_stripe_floor.py: the stripe floor is below a vector A vector is a fixed 1024 values (`COLUMNAR_NATIVE_VECTOR_LENGTH`). A row group smaller than one never fills it, so the chunk-shared FSST symbol table is not built From 5e84adac98006d8175c29d42a7615f992eabd802 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sat, 12 Sep 2026 14:02:57 -0600 Subject: [PATCH 3/4] docs: split two long sentences, and assert the SECTION not the page (#1017) Two findings from @OffgridwithJD's review, both reproduced here before acting. 1. THE PLAIN-LANGUAGE GATE REFUSED MY OWN PROSE. Sentence length in WORDS: best-practices.md 49 words configuration.md 35 words Both split. `plain_language_check.py` over docs/ and README.md: 15 ok, 0 fail. The reviewer went hunting for long LINES first; the longest lines are identical on both sides and exempt. The rule is words. 2. ONE LINE SAYS NOTHING ABOUT WHERE, and the arm's NAME claimed it did. Moving the floor line out of the advice block to the end of administration.md -- 402 lines away -- left this passing: PASS administration.md states it beside the advice to lower the setting Reproduced, then fixed by tightening rather than renaming. The arm now asserts the SECTION: the floor and the lowering advice must sit under one `## ` heading, both under `## Row-group sizing` today. A HEADING IS A DECLARED BOUNDARY, which is what the paragraph reader lacked -- blank lines are absent inside a markdown table and arbitrary in prose. The reviewer argued for renaming the four arms instead, on the ground that bounding reintroduces "what is a section". It does, but a `## ` heading answers it in a way a blank line cannot, and the property the name states is worth keeping. branch all three green line moved 402 lines away administration arm RED, other two green main's three pages all three RED THE REMOVAL PROOF BROKE AND PASSED SILENTLY. `git stash` on the three pages stopped reverting anything once the change was committed rather than staged, so "restore main's pages" restored the branch's own pages and all three arms passed. Checking them out from origin/main explicitly is what makes that row mean anything. A proof that stops proving reads exactly like a proof that succeeded. Also carried: the chunk-group clause is now measured rather than assumed -- chunk_group_row_limit at its floor of 100 stores 7,086,080 bytes, byte-identical to 1024 and 10000 (@OffgridwithJD). the 14 database-free files 280 passed, 687 checks, 0 fail Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- CHANGELOG.md | 10 ++++ docs/best-practices.md | 8 +-- docs/configuration.md | 2 +- test/docs_style.sh | 42 ++++++++++------ test/pytest/TESTS.md | 53 +++++++++++++------- test/pytest/test_docs_stripe_floor.py | 70 +++++++++++++++++++-------- 6 files changed, 127 insertions(+), 58 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a31004e..842c76d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -99,6 +99,10 @@ true until the next version shipped. Documentation only. The minimum is unchanged: whether to raise it, make FSST work below a vector, or warn at `set_options` is still open on #1017. + The chunk-group limit does not affect this, and that is now measured rather than + assumed: `chunk_group_row_limit` at its floor of 100 stores 7,086,080 bytes, the same + byte count as 1024 and 10000. Measured by @OffgridwithJD. + THE GUARD WAS BORN GREEN TWICE BEFORE IT WORKED, and the measurement is why it does now. A blank-line block reader passed on main, because `configuration.md`'s GUC table has no blank lines and `stripe_row_limit`'s row shares a block with @@ -110,6 +114,12 @@ true until the next version shipped. mode and passed on main for two pages, the python twin split on blank lines and did not -- which is the argument for keeping both halves, paid back the day it was written. + AND ONE LINE SAYS NOTHING ABOUT WHERE. Moving the line out of the advice block to the + end of administration.md, 402 lines away, left the arm passing while its name claimed + the floor was stated beside the advice. Reported by @OffgridwithJD. The arm now + asserts the SECTION: the floor and the lowering advice must sit under one `## ` + heading. A heading is a declared boundary, which is what the paragraph reader lacked. + - A check record names the PostgreSQL major it was observed under (#1010). RESULTsuitepartnameverdictmajorreason diff --git a/docs/best-practices.md b/docs/best-practices.md index 6f937550..f5e332a8 100644 --- a/docs/best-practices.md +++ b/docs/best-practices.md @@ -27,10 +27,10 @@ own. Small transactions produce small, poorly compressed row groups, and many of them to scan later. Prefer `COPY` or a multi-row `INSERT ... SELECT` over row-at-a-time inserts. Load in batches that fill a row group (`pgcolumnar.stripe_row_limit`, default 150000 rows), so each row group compresses -well. Keep `pgcolumnar.stripe_row_limit` at 1024 or above if you lower it: a vector -is a fixed 1024 values, and a row group below one vector gets no FSST on its text -columns, which cost 106.4% of their raw bytes at the accepted minimum of 1000 -against 54.7% at 1200 (#1017). +well. Keep `pgcolumnar.stripe_row_limit` at 1024 or above if you lower it. A vector +is a fixed 1024 values. A row group below one vector gets no FSST on its text +columns. Those columns cost 106.4% of their raw bytes at the accepted minimum of +1000, against 54.7% at 1200 (#1017). **Use `parallel_copy` for a large file.** `pgcolumnar.parallel_copy` splits a server-side file across workers and scales the load into one table. It runs a diff --git a/docs/configuration.md b/docs/configuration.md index c7dc8f3a..c0007f87 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -158,7 +158,7 @@ SELECT pgcolumnar.set_options( | `stripe_row_limit` | integer | Per-table override of `pgcolumnar.stripe_row_limit`. | **A `stripe_row_limit` below 1024 disables FSST on text columns.** A vector is a -fixed 1024 values, so a row group smaller than that never fills one, and the +fixed 1024 values. A row group smaller than that never fills one, so the chunk-shared FSST symbol table is not built. Measured on 200,000 rows of a text column, `compression = none`, against 12,800,000 raw bytes: diff --git a/test/docs_style.sh b/test/docs_style.sh index 7d1369c0..b664a5ac 100755 --- a/test/docs_style.sh +++ b/test/docs_style.sh @@ -163,19 +163,22 @@ check "best-practices names clustering on the join key" \ # in a reference table three pages away -- so these arms are scoped to the block # and not to the page. An earlier version grepped whole pages and passed on main, # which already says 1024 and FSST elsewhere. -# ONE LINE CARRYING BOTH, which is the claim stated in one sentence rather than two -# tokens that happen to be near each other. -# -# Two weaker versions were born green and the measurement is why they were replaced. -# awk paragraph mode (RS='') made configuration.md's whole GUC table one record, so -# `stripe_row_limit`'s row shared a record with `chunk_group_row_limit`'s "fixed -# 1024-value vectors" -- green on main. A three-line proximity window failed the same -# way, because those rows are adjacent. Measured on main: same-line is 0 for all -# three pages, which is the only one of the three that is actually red there. -# -# The pytest twin splits on blank lines in python and did NOT have the paragraph -# behaviour, so the two harnesses disagreed and the shell one was wrong. That is the -# argument for keeping both, paid back the day it was written. +# ONE LINE CARRYING BOTH, AND FOR administration.md THE RIGHT SECTION TOO. +# +# One line, because a blank-line block and a three-line window are both green on +# main: configuration.md's GUC table has no blank lines, so stripe_row_limit's row +# shares a block with chunk_group_row_limit's "fixed 1024-value vectors", and those +# rows are adjacent. One line naming both is 0 on all three pages on main, and it +# makes the prose state the floor in a sentence, which is what a warning needs. +# +# THE SECTION, because one line alone says nothing about WHERE. @OffgridwithJD moved +# the line out of the advice block to the end of administration.md, 402 lines away, +# and the page-wide arm still passed while its name claimed the floor was stated +# "beside the advice to lower the setting". Reproduced before changing anything. +# +# A `## ` heading is the boundary, not a blank line. That is what the paragraph +# reader got wrong: blank lines are absent inside a markdown table and arbitrary in +# prose, while a heading is declared. _floor_line() { # _floor_line FILE -- yes if one line names the setting and the floor if grep -qE 'stripe_row_limit.*1024|1024.*stripe_row_limit' "$1"; then echo yes @@ -183,10 +186,19 @@ _floor_line() { # _floor_line FILE -- yes if one line names the setting and the echo no fi } +_floor_same_section() { # _floor_same_section FILE ADVICE -- yes if both are under one `## ` + awk -v advice="$2" ' + /^## / { h = substr($0, 4) } + { + if (index(tolower($0), tolower(advice))) a[h] = 1 + if (/stripe_row_limit/ && /1024/) f[h] = 1 + } + END { for (k in a) if (k in f) { print "yes"; exit } print "no" }' "$1" +} check "configuration.md states the 1024 floor where it documents the setting" \ "$(_floor_line "$SRCDIR/docs/configuration.md")" "yes" -check "administration.md states it beside the advice to lower the setting" \ - "$(_floor_line "$SRCDIR/docs/administration.md")" "yes" +check "administration.md states it in the section that says to lower the setting" \ + "$(_floor_same_section "$SRCDIR/docs/administration.md" "Lower this setting")" "yes" check "best-practices.md carries it with the load-sizing advice" \ "$(_floor_line "$SRCDIR/docs/best-practices.md")" "yes" diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index fbe6a920..e21ff62e 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -2957,32 +2957,51 @@ storing the bytes uncompressed. `docs/administration.md` tells a reader to *lowe this setting for point-lookup-heavy tables, which is the path in, so the warning has to sit in the block that gives the advice rather than in a reference table. -### Why every arm asserts ONE LINE, not a block or a window +### Why one line, and why the section as well -Two weaker signals were tried and **both were born green on `main`**: +Three signals were tried. Two were born green on `main`: -| signal | why it passed on main | +| signal | on `main` | | --- | --- | -| blank-line block | `configuration.md`'s GUC table has no blank lines, so `stripe_row_limit`'s row shares a block with `chunk_group_row_limit`'s "fixed 1024-value vectors" | -| three-line window | those same rows are adjacent | -| **one line naming both** | **0 on all three pages on `main`** | +| blank-line block | **passes** — `configuration.md`'s GUC table has no blank lines, so `stripe_row_limit`'s row shares a block with `chunk_group_row_limit`'s "fixed 1024-value vectors" | +| three-line window | **passes** — those rows are adjacent | +| one line naming both | **0 on all three pages** | -Requiring one line is also a claim about the prose: the floor has to be stated in a -sentence, not inferred from two tokens that happen to be neighbours. +One line is also a claim about the prose: the floor has to be stated in a sentence, +not inferred from two neighbouring tokens. That is why `best-practices.md` names the +setting and the number together. -**The two harnesses disagreed, and the shell one was wrong.** The awk arm used -paragraph mode and passed on `main` for two pages; the python twin split on blank -lines and did not. That is the argument for keeping both halves, paid back the day -it was written. +**And one line alone says nothing about WHERE.** @OffgridwithJD moved the line out of +the advice block to the end of `administration.md` — **402 lines away** — and the arm +still passed while its name claimed the floor was stated "beside the advice to lower +the setting". Reproduced here before anything changed. + +So the `administration.md` arm asserts the **section**: the floor and the lowering +advice must sit under one `## ` heading, both under `## Row-group sizing` today. A +heading is a declared boundary, which is exactly what the paragraph reader lacked — +blank lines are absent inside a markdown table and arbitrary in prose. | test | what it pins | | --- | --- | | `test_configuration_states_the_floor_where_it_documents_the_setting` | the floor is on the setting's own line | -| `test_administration_states_it_beside_the_advice_to_lower_it` | it is on the page that tells readers to lower the setting, naming what is lost | +| `test_administration_states_it_in_the_section_that_says_to_lower_it` | it is in the **same section** as the advice that leads there | | `test_best_practices_carries_the_floor_with_the_load_sizing_advice` | the load-sizing guidance states it too | -### Removal proof +### Removal proof, three ways + +| mutation | result | +| --- | --- | +| `main`'s three pages | all three arms red | +| the floor line moved 402 lines from the advice | the administration arm red, the other two green | +| the branch as it stands | all three green | + +The second row is the one the page-wide version could not produce. + +**The proof itself broke once and said so.** `git stash` on the three pages stopped +reverting them the moment the change was committed rather than staged, so the +"restore main's pages" step was restoring the branch's own pages and every arm passed. +Checking the files out from `origin/main` explicitly is what makes the row mean +anything. -Restore `main`'s three pages and **all three** arms go red. The shell twin is three -arms in `docs_style.sh`; the two halves share no code, one matching in `grep` and the -other in Python. +The shell twin is three arms in `docs_style.sh`: `grep` for the two one-line pages and +an awk heading walker for `administration.md`. The two halves share no code. diff --git a/test/pytest/test_docs_stripe_floor.py b/test/pytest/test_docs_stripe_floor.py index 56cb2723..9e34c218 100644 --- a/test/pytest/test_docs_stripe_floor.py +++ b/test/pytest/test_docs_stripe_floor.py @@ -24,42 +24,70 @@ PRACTICES = ROOT / "docs" / "best-practices.md" -def _states_the_floor(path): - """True when ONE LINE names the setting and the floor together. - - Not a block, and not a proximity window. Both were tried and both were born - green: `configuration.md`'s GUC table has no blank lines, so a paragraph reader - puts `stripe_row_limit`'s row and `chunk_group_row_limit`'s "fixed 1024-value - vectors" in one unit, and those rows are adjacent so a three-line window does - the same. Measured on `main`: block and window both say yes, one line says no - for all three pages. - - Requiring one line is also a claim about the PROSE -- the floor has to be stated - in a sentence, not inferred from two tokens that happen to be neighbours. +def _floor_line_sections(path): + """The `##` headings under which a line names both the setting and the floor. + + ONE LINE, because a blank-line block and a three-line window are both green on + `main`: `configuration.md`'s GUC table has no blank lines, so + `stripe_row_limit`'s row shares a block with `chunk_group_row_limit`'s "fixed + 1024-value vectors", and those rows are adjacent. One line naming both is 0 on + all three pages there, and it constrains the prose to state the floor in a + sentence, which is what a warning needs. + + AND THE SECTION, because one line alone says nothing about WHERE. Reported by + @OffgridwithJD, who moved the line out of the advice block to the end of + `administration.md` -- 402 lines away -- and the arm still passed while claiming + the floor was stated "beside the advice to lower the setting". Reproduced here + before changing anything. + + A `##` heading is the boundary, not a blank line. That is what the paragraph + reader got wrong: blank lines are absent inside a markdown table and arbitrary + in prose, while a heading is declared. """ + out, heading = set(), None for line in path.read_text(encoding="utf-8").splitlines(): - if "stripe_row_limit" in line and "1024" in line: - return True - return False + if line.startswith("## "): + heading = line[3:].strip() + elif "stripe_row_limit" in line and "1024" in line: + out.add(heading) + return out + + +def _sections_containing(path, needle): + out, heading = set(), None + for line in path.read_text(encoding="utf-8").splitlines(): + if line.startswith("## "): + heading = line[3:].strip() + elif needle.lower() in line.lower(): + out.add(heading) + return out def test_configuration_states_the_floor_where_it_documents_the_setting(expect): expect.num(int(CONFIG.is_file()), 1, "premise: configuration.md is in the tree") - expect.num(int(_states_the_floor(CONFIG)), 1, + expect.num(int(len(_floor_line_sections(CONFIG)) > 0), 1, "configuration.md states the 1024 floor on the setting's own line") -def test_administration_states_it_beside_the_advice_to_lower_it(expect): +def test_administration_states_it_in_the_section_that_says_to_lower_it(expect): """`administration.md` tells a reader to LOWER this setting for point lookups. - That is the path into the cliff, so the floor has to be on this page.""" + + That is the path into the cliff, so the floor has to be in THAT section. The + arm asserts the section and not merely the page, because the page-wide version + passed with the two 402 lines apart. + """ expect.num(int(ADMIN.is_file()), 1, "premise: administration.md is in the tree") - expect.num(int(_states_the_floor(ADMIN)), 1, - "administration.md states the floor beside the lowering advice") + advice = _sections_containing(ADMIN, "lower this setting") + expect.num(int(len(advice) > 0), 1, + "premise: administration.md still tells a reader to lower the setting") + floor = _floor_line_sections(ADMIN) + expect.num(int(len(advice & floor) > 0), 1, + "and the 1024 floor is stated in that same section") low = ADMIN.read_text(encoding="utf-8").lower() expect.num(int("fsst" in low), 1, "and names what lowering past it costs") def test_best_practices_carries_the_floor_with_the_load_sizing_advice(expect): expect.num(int(PRACTICES.is_file()), 1, "premise: best-practices.md is in the tree") - expect.num(int(_states_the_floor(PRACTICES)), 1, + expect.num(int(len(_floor_line_sections(PRACTICES)) > 0), 1, "the load-sizing advice states the floor on the same line") From 5928438afe89ec2af1bf527b2db54b741cbe00e2 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sat, 12 Sep 2026 20:56:03 -0600 Subject: [PATCH 4/4] docs: the 1024 note split the set_options table in half (#1026) I put the note and its measurement table between `stripe_row_limit` and `compression` in the set_options argument table. A blank line ends a markdown table, so the six arguments after it -- compression, compression_level, encode_effort, sort_by, ttl_column, ttl_interval -- rendered as a headerless block or as literal pipes. docs_style.sh passed this page with 14 checks, and passes the fixed page with 14 checks, because all seven of its rules are about prose. That is what #1028 adds a structure rule for; @OffgridwithJD found this reviewing #1022. The note now follows the table it was splitting, still inside the section that documents the setting, and the `stripe_row_limit` row points at it the way the GUC table's row already did. Verified: #1028's headerless_tables() flags docs/configuration.md at line 175 on the previous commit and flags nothing on this one; main was clean, so this branch introduced it. Both #1017 guards still pass in both harnesses. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw --- docs/configuration.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index c0007f87..13a37874 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -155,7 +155,13 @@ SELECT pgcolumnar.set_options( | --- | --- | --- | | `table_name` | regclass | The columnar table to change. Anything that is not an ordinary table using the `pgcolumnar` access method is rejected, including a partitioned table. | | `chunk_group_row_limit` | integer | Per-table override of `pgcolumnar.chunk_group_row_limit`. | -| `stripe_row_limit` | integer | Per-table override of `pgcolumnar.stripe_row_limit`. | +| `stripe_row_limit` | integer | Per-table override of `pgcolumnar.stripe_row_limit`. See the note below: a value under 1024 costs text compression. | +| `compression` | name | One of `none`, `pglz`, `lz4`, `zstd`. | +| `compression_level` | integer | Level for the `zstd` codec, 1 to 22. | +| `encode_effort` | name | `full` (default) or `fast`. How much work the writer spends choosing an encoding. See below. | +| `sort_by` | name[] | Declared physical sort key (#288), applied by `pgcolumnar.vacuum_sorted(t)` with no columns. Column names, so it survives `pg_dump`/restore. Not auto-maintained; re-run after inserts. Cannot name a virtual generated column. Clear with `reset_options(t, sort_by => true)`. | +| `ttl_column` | name | The `timestamp` or `timestamptz` column a retention is measured on. Set it with `ttl_interval`; neither works alone. Nothing is deleted until you call `pgcolumnar.expire(t)` by name. | +| `ttl_interval` | interval | How long a row is kept, measured from `ttl_column`. `pgcolumnar.expire(t)` then drops row groups whose rows are all older than this. A group with one live row is kept whole. | **A `stripe_row_limit` below 1024 disables FSST on text columns.** A vector is a fixed 1024 values. A row group smaller than that never fills one, so the @@ -172,12 +178,6 @@ At 1000 the column costs more than storing the bytes uncompressed. The accepted minimum is 1000 and the vector is 1024, so **the most aggressive legal setting is the one that pays this cost**. Use 1024 or more unless you have measured that you want the opposite (#1017). -| `compression` | name | One of `none`, `pglz`, `lz4`, `zstd`. | -| `compression_level` | integer | Level for the `zstd` codec, 1 to 22. | -| `encode_effort` | name | `full` (default) or `fast`. How much work the writer spends choosing an encoding. See below. | -| `sort_by` | name[] | Declared physical sort key (#288), applied by `pgcolumnar.vacuum_sorted(t)` with no columns. Column names, so it survives `pg_dump`/restore. Not auto-maintained; re-run after inserts. Cannot name a virtual generated column. Clear with `reset_options(t, sort_by => true)`. | -| `ttl_column` | name | The `timestamp` or `timestamptz` column a retention is measured on. Set it with `ttl_interval`; neither works alone. Nothing is deleted until you call `pgcolumnar.expire(t)` by name. | -| `ttl_interval` | interval | How long a row is kept, measured from `ttl_column`. `pgcolumnar.expire(t)` then drops row groups whose rows are all older than this. A group with one live row is kept whole. | The function does not change an argument that keeps its default value of `NULL`. The function refuses a value that is outside the permitted range of a