From 73e011b201f55d70348f7a00330314e7651cd108 Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" <136637981+ChronicallyJD@users.noreply.github.com> Date: Wed, 23 Sep 2026 08:31:33 -0600 Subject: [PATCH 1/5] cost: the parallel covering clamp is reachable, and decides nothing (#1209) #1127 called the clamp unreachable "with projRun = serialRun * projScale". #1155 computed projRun independently, which is the falsifier #1127 named, and the comment has read UNPROVEN since. Measured, with a probe at the clamp site. IT IS REACHABLE. Both sides are linear in seq_page_cost because the CPU term is not, so three constants fitted from six points predict the crossing: pre = ioRun * projScale = spc * A A = 3.1 projRun = C + spc * B B = 3.0 C = 262.5 binding needs spc > C/(A-B) = 2625 Predicted before it was run. 2048 does not bind, missing by 0.9%; 4096 does, and the plan changes from Gather -> Parallel Custom Scan to a serial Custom Scan, which is the consequence #1127 wrote down. AT THE DEFAULT GUCS IT CANNOT BIND, for a constant rather than a property of the fixture: binding needs the projection to save more than 5.12*W + 82 bytes per row, because cpu_operator_cost * W/4 is 5.12 times seq_page_cost * W/8192. Measured storage runs 0.13x and 0.04x of W. A second fixture built to move that margin did not move it: 2 pages of difference and the same 2625 threshold in both, because the base compresses the same data almost as well as the projection does. THE CLAMP CHANGES NO PLAN. Removing it leaves every plan in projection_parallel.sh identical and the suite green -- the unclamped total is LARGER, so the serial covering path wins either way. It keeps cpuRunProj from going negative, which no plan exposes. So the two new arms are named for I/O amortisation rather than for the clamp. I wrote them as clamp arms first; the removal mutation did not redden them, which would have shipped a vacuous guard. What does redden the second arm is (ioRunProj + cpuRunProj) / divisor: got [gather+projection] want [projection-only] Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP --- CHANGELOG.md | 41 ++++++++++++++++++++++ src/columnar_customscan.c | 67 ++++++++++++++++++++++++++---------- test/check_ledger.tsv | 2 ++ test/check_ledger_budget.txt | 26 +++++++++++++- test/projection_parallel.sh | 57 ++++++++++++++++++++++++++++++ 5 files changed, 173 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5deb64c1..7212b096 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,47 @@ true until the next version shipped. ### Changed +- The parallel covering clamp's comment records what was measured rather than + "unproven" (#1209). It is reachable, the threshold was predicted before it was + observed, and the clamp itself decides nothing. + + Probed at the clamp site. Both sides are linear in `seq_page_cost`, because the + CPU term is not, so three constants fitted from six points predict the crossing: + + ``` + pre = ioRun * projScale = spc * A A = 3.1 + projRun = C + spc * B B = 3.0 C = 262.5 + binding needs spc > C/(A-B) = 2625 + ``` + + Then measured: 2048 does not bind, missing by 0.9%, and 4096 does. When it binds + the plan changes from `Gather -> Parallel Custom Scan` to a serial + `Custom Scan (PgColumnarScan)`, which is the consequence #1127 wrote down. + + At the default cost settings it cannot bind, and the reason is a constant rather + than a property of the fixture: per row, binding needs the projection to save + more than `5.12 * W + 82` bytes, where `W` is the analyzed width of the columns + read, because `cpu_operator_cost * W/4` is 5.12 times `seq_page_cost * W/8192`. + Measured storage runs at 0.13x and 0.04x of `W`. + + A second fixture built to move that margin did not move it. With a blob constant + across runs of 500 sort-key values, inserted scrambled so only the clustered + projection sees the runs, `basePagesRead - projPages` came out at 2 pages in both + fixtures and the threshold landed at 2625 twice. The base compresses the same + data almost as well as the projection does, so the margin is structural. + + **The clamp changes no plan**, which is worth stating because it reads as though + it should. Removing it leaves every plan in `projection_parallel.sh` identical + and the suite green: at the binding point the unclamped total is `startup + pre + + (projRun - pre)/divisor`, which is LARGER than `startup + projRun`, so the serial + covering path wins either way. The clamp keeps `cpuRunProj` from going negative, + an internal quantity no plan exposes. + + Two arms in `projection_parallel.sh` pin the page-cost ladder. They are named for + I/O amortisation rather than for the clamp, because the clamp-removal mutation + does not redden them and an arm named for it would have been vacuous. What does + redden the second is `(ioRunProj + cpuRunProj) / divisor`. + - `build_all_versions.sh` reads the matrix's major list from `run_all_versions.sh` instead of carrying its own copy (#1219). The two copies were identical, so the rot was latent rather than live -- but a major moving in one and not the other diff --git a/src/columnar_customscan.c b/src/columnar_customscan.c index 38f18b52..99356018 100644 --- a/src/columnar_customscan.c +++ b/src/columnar_customscan.c @@ -3364,30 +3364,59 @@ PgColumnarSetRelPathlist(PlannerInfo *root, RelOptInfo *rel, Index rti, * Clamp ioRunProj to projRun. #1127 wrote that this was * unreachable "with projRun = serialRun * projScale", and * named "computed independently (for example from the - * projection's own pages)" as what would make it live. - * THIS CHANGE IS THAT, so the premise no longer holds and - * the sentence is corrected rather than carried. + * projection's own pages)" as what would make it live. #1155 + * did exactly that, so the premise is gone. * - * IT IS NOT KNOWN TO BE REACHABLE EITHER, and that is a - * measurement rather than an argument. @OffgridwithJD probed - * it: reached three times in projection_parallel.sh and bound - * zero, with margins 24.4794 against 2122.2110 and 0.2473 - * against 163.8619; a fixture built to bind it reached once - * and still did not, 0.2504 against 148.2537. Binding needs + * IT IS REACHABLE, and #1209 measured where. Both sides are + * linear in seq_page_cost, because the CPU term is not: * - * 2*ioBase - serialRun > baseSurvival * seq_page_cost * projPages + * pre = ioRun * projScale = spc * A + * projRun = C + spc * B * - * in which sel cancels, and both attempts moved the margin the - * wrong way, by 87x and then 592x, because - * pgcolumnar_scan_io_run_cost prices only the columns read. + * with A = sel * basePagesRead, B = sel * projPages and + * C = cpuRun * projScale. On a 20,000-row fixture A = 3.1, + * B = 3.0, C = 262.5 reproduced six probe points to the + * decimal, so binding needs spc > C/(A-B) = 2625. Predicted + * before it was run, then 2048 missed by 0.9% and 4096 bound. * - * So: the old premise is FALSE, and reachability is UNPROVEN. - * The clamp stays because it is cheap and its absence would be - * a silently negative cpuRunProj. + * AT THE DEFAULT GUCS IT CANNOT BIND, and the reason is a + * constant rather than a property of that fixture. Per row, + * binding at seq_page_cost = 1 needs the projection to save + * more than 5.12 * W + 82 bytes, where W is the analyzed width + * of the columns read: cpu_operator_cost * W/4 is 5.12 times + * seq_page_cost * W/8192. The saving is bounded by what the + * base stores, measured at 0.13x and 0.04x of W on two + * fixtures. * - * When the clamp binds fully, cpuRunProj is zero and the - * partial covering path totals exactly like the serial - * covering path, so Gather loses. + * A SECOND FIXTURE BUILT TO MOVE THAT MARGIN DID NOT MOVE IT. + * A blob constant across runs of 500 sort-key values, inserted + * scrambled so only the clustered projection sees the runs, + * left basePagesRead - projPages at 2 pages -- the same as the + * original -- and the same threshold of 2625. The base + * compresses the same data almost as well as the projection + * does, so the margin is structural rather than incidental. + * + * WHEN IT BINDS, cpuRunProj is zero and the partial covering + * path totals exactly like the serial covering path, so Gather + * loses. Measured: the plan is + * "Gather -> Parallel Custom Scan" up to 2048 and a serial + * "Custom Scan (PgColumnarScan)" at 4096. + * + * THE CLAMP ITSELF CHANGES NO PLAN, which is worth stating + * because it reads as though it should. Removing it entirely + * leaves every plan in test/projection_parallel.sh identical + * and the suite green. At the binding point, with pre > projRun + * and divisor > 1: + * + * clamped startup + projRun + * unclamped startup + pre + (projRun - pre)/divisor + * + * and the unclamped total is LARGER, so the serial covering + * path wins either way. The clamp earns its place by keeping + * cpuRunProj from going negative -- an internal quantity no + * plan exposes -- and not by deciding anything. Do not write a + * test that claims to guard it through a plan shape: that arm + * passes with the clamp removed, which is how this was found. */ ioRunProj = ioRun * projScale; if (ioRunProj > projRun) diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 9342e33a..0c220527 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1492,6 +1492,8 @@ parallel_scan_cost parallel_scan_cost premise: the serial plan has no Gather 15; parallel_scan_cost parallel_scan_cost premise: the serial plan is a columnar scan 15;16;17;18;19 never - parallel_scan_cost parallel_scan_cost premise: the table holds every inserted row 15;16;17;18;19 never - parallel_scan_cost parallel_scan_cost the leader-participation branch changes the divisor 15;16;17;18;19 never - +projection_parallel projection_parallel a page cost that makes I/O dominate costs the parallel covering path 15;16;17;18;19 2026-09-23 the partial covering total amortises I/O as well as CPU: (ioRunProj + cpuRunProj) / divisor +projection_parallel projection_parallel premise: the covering plan is parallel at the default page cost 15;16;17;18;19 never - projection_parallel projection_parallel a covering projection can be a parallel scan 15;16;17;18;19 2026-09-18 - projection_parallel projection_parallel a parallel covering projection returns the covering rows once 15;16;17;18;19 never - projection_parallel projection_parallel premise: ANALYZE printed a rows= line per launched worker 15;16;17;18;19 never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 00ef9841..5abc9d96 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -430,4 +430,28 @@ suites_not_covered 249 # what these numbers are. # # suites_not_covered does NOT move: harness_selftest is already a covered suite. -checks_never_observed_red 1526 +# +# 1526 -> 1527 for #1209's two new projection_parallel checks. Counted on this +# tree, with the premise beside it because a pattern that matches nothing counts +# 0 and reads like a clean answer: +# +# awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l -> 1527 +# 1589 rows total, 62 not `never` +# +# TWO rows and the census moves by ONE. The arm that carries a last-red is +# +# a page cost that makes I/O dominate costs the parallel covering path +# +# reddened by amortising the I/O in the partial covering total, +# `(ioRunProj + cpuRunProj) / divisor`, which gives +# `got [gather+projection] want [projection-only]` at the top of the ladder. +# +# THE MUTATION I EXPECTED TO REDDEN IT DOES NOT, and that is why the arm is named +# for I/O amortisation rather than for the clamp. Removing +# `if (ioRunProj > projRun) ioRunProj = projRun;` changes no plan at all: every +# rung of the ladder reads the same and the suite passes. The unclamped total is +# LARGER than the clamped one, not smaller, so the serial covering path wins +# either way. An arm named for the clamp would have been vacuous, and the only +# reason it is not in this file is that the mutation was run before it was +# written up. +checks_never_observed_red 1527 diff --git a/test/projection_parallel.sh b/test/projection_parallel.sh index ee3ec6e8..cc80865a 100755 --- a/test/projection_parallel.sh +++ b/test/projection_parallel.sh @@ -163,6 +163,63 @@ check "a covering projection can be a parallel scan" \ check "a parallel covering projection returns the covering rows once" \ "$par_on_count" "$WANT" +# ---- an I/O-dominated covering scan is not quoted at 1/N of its cost (#1209) - +# +# Raising seq_page_cost until the base relation's I/O dominates its decode CPU +# must cost the PARALLEL covering path, because core leaves disk I/O whole and +# amortises only CPU. If the partial covering total divided both, an I/O-bound +# scan would be quoted at 1/N and Gather would keep winning however expensive +# the pages became. +# +# THIS IS NOT A GUARD ON THE CLAMP, and the difference is measured rather than +# assumed. Removing `if (ioRunProj > projRun) ioRunProj = projRun;` entirely +# changes NOTHING here: every rung of the ladder below reads the same, and the +# suite passes. The arithmetic says why. At the binding point, with pre = +# ioRun*projScale > projRun: +# +# clamped total = startup + projRun +# unclamped total = startup + pre + (projRun - pre)/divisor +# +# and since pre > projRun and divisor > 1, the unclamped total is LARGER. The +# serial covering path wins either way, so no plan moves. The clamp earns its +# place by keeping cpuRunProj from going negative, which is an internal quantity +# no plan exposes -- not by changing a decision. +# +# What DOES redden the second arm is amortising the I/O: rewriting the total as +# `(ioRunProj + cpuRunProj) / divisor` gives `got [gather+projection] want +# [projection-only]` at the top of the ladder. +# +# #1209 for the measurement behind this: the clamp IS reachable, at +# seq_page_cost > C/(A-B) = 2625 on a 20,000-row fixture, predicted from a fit +# of six points and then confirmed at 2048 (no, by 0.9%) and 4096 (yes). +SPC_HI=1000000 +explain_cov_spc() { # $1 = seq_page_cost + env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ + -d "$PGC_DB" -At \ + -c "SET min_parallel_table_scan_size = 0;" \ + -c "SET parallel_setup_cost = 0;" \ + -c "SET parallel_tuple_cost = 0;" \ + -c "SET max_parallel_workers_per_gather = 2;" \ + -c "SET pgcolumnar.enable_projection_scan = on;" \ + -c "SET seq_page_cost = $1;" \ + -c "EXPLAIN (COSTS OFF) $Q;" 2>/dev/null || true +} + +# The ladder is printed rather than summarised: a failure of the second arm has +# two possible causes -- the partial total amortising I/O, or a host where the +# covering projection is not smaller than the base's read columns so no page +# cost ever tips it -- and the rungs tell them apart. +echo "-- page-cost ladder, seq_page_cost against plan shape:" +for _spc in 1 1024 4096 $SPC_HI; do + printf ' seq_page_cost=%-8s %s\n' "$_spc" "$(shape "$(explain_cov_spc "$_spc")")" +done + +check "premise: the covering plan is parallel at the default page cost" \ + "$(shape "$(explain_cov_spc 1)")" "gather+projection" + +check "a page cost that makes I/O dominate costs the parallel covering path" \ + "$(shape "$(explain_cov_spc $SPC_HI)")" "projection-only" + check "premise: EXPLAIN ANALYZE launched two workers" \ "$(echo "$par_on_ana" | grep -oE 'Workers Launched: [0-9]+' | head -1 | grep -oE '[0-9]+')" "2" From 6af60701a3cf3ce1e5901ce8307fc355a1f7b087 Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" <136637981+ChronicallyJD@users.noreply.github.com> Date: Wed, 23 Sep 2026 08:38:50 -0600 Subject: [PATCH 2/5] comment: put the sign in, so the claim is checkable without a cluster (#1209) @jdatcmd's review: unclamped - clamped = (pre - projRun) * (1 - 1/divisor), strictly positive because pre > projRun IS the binding condition and divisor > 1 always. A reader can check that in ten seconds; the mutation needs a build and a cluster. And name why the old sentence misled. "When the clamp binds fully ... so Gather loses" is TRUE and reads as causal. Unclamped, Gather loses harder. A true sentence about a coincidence reads as a mechanism, and nothing in review catches that. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP --- src/columnar_customscan.c | 17 +++++++++++++++-- test/projection_parallel.sh | 6 ++++-- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/src/columnar_customscan.c b/src/columnar_customscan.c index 99356018..1f4fde35 100644 --- a/src/columnar_customscan.c +++ b/src/columnar_customscan.c @@ -3411,8 +3411,21 @@ PgColumnarSetRelPathlist(PlannerInfo *root, RelOptInfo *rel, Index rti, * clamped startup + projRun * unclamped startup + pre + (projRun - pre)/divisor * - * and the unclamped total is LARGER, so the serial covering - * path wins either way. The clamp earns its place by keeping + * and the difference has a sign a reader can check without a + * cluster: + * + * unclamped - clamped = (pre - projRun) * (1 - 1/divisor) + * + * pre > projRun IS the binding condition and divisor > 1 always, + * so that is strictly positive: removing the clamp makes the + * partial path DEARER. The serial covering path wins either way. + * + * The sentence above this one used to end "so Gather loses", + * which is TRUE and reads as causal. Unclamped, Gather loses + * harder. A true sentence about a coincidence reads as a + * mechanism, and nothing in review catches that. + * + * The clamp earns its place by keeping * cpuRunProj from going negative -- an internal quantity no * plan exposes -- and not by deciding anything. Do not write a * test that claims to guard it through a plan shape: that arm diff --git a/test/projection_parallel.sh b/test/projection_parallel.sh index cc80865a..fc9ded0a 100755 --- a/test/projection_parallel.sh +++ b/test/projection_parallel.sh @@ -180,8 +180,10 @@ check "a parallel covering projection returns the covering rows once" \ # clamped total = startup + projRun # unclamped total = startup + pre + (projRun - pre)/divisor # -# and since pre > projRun and divisor > 1, the unclamped total is LARGER. The -# serial covering path wins either way, so no plan moves. The clamp earns its +# and the difference has a sign: unclamped - clamped = (pre - projRun) * +# (1 - 1/divisor), which is strictly positive because pre > projRun IS the +# binding condition and divisor > 1 always. Removing the clamp makes the partial +# path DEARER, so the serial covering path wins either way and no plan moves. The clamp earns its # place by keeping cpuRunProj from going negative, which is an internal quantity # no plan exposes -- not by changing a decision. # From 15a804c4a34deaa5844874ba12008160525d6a39 Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" <136637981+ChronicallyJD@users.noreply.github.com> Date: Wed, 23 Sep 2026 08:53:29 -0600 Subject: [PATCH 3/5] test: port the page-cost ladder to the pytest twin (#1209) The shell suite gained two arms and test_projection_parallel.py did not, so test_compare_to_bash graded the pair INCOMPLETE and three CI jobs went red. The grader's number is an exit code, not a count: `verdicts[stem] = rc`, and rc=1 means "PORT IS INCOMPLETE". Run locally it names both: MISSING a page cost that makes I/O dominate costs the parallel covering path MISSING premise: the covering plan is parallel at the default page cost literal matches: 9 | missing: 2 | VERDICT: PORT IS INCOMPLETE Both are in the port now, with its own cursor, its own SETs and its own fixture. Nothing crosses the boundary but the assertion NAMES, which is what the grader compares. Parity rc=0, 11 literal matches, 0 missing, and the file passes on pg18a. The threshold is scale-invariant -- C and (A - B) are both proportional to the row count -- so the port's different N needs no different number, and its ladder reads the same as the shell one. THE TWO ARMS ARE A PAIR, recorded in both halves (@jdatcmd, review): the second cannot separate "I/O is left whole" from "there is no parallel covering path at all", since both read projection-only. The premise at seq_page_cost=1 excludes the second reading, so deleting it as redundant leaves a passing arm that proves nothing. TESTS.md carries the ladder and the sign. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP --- test/projection_parallel.sh | 6 +++ test/pytest/TESTS.md | 12 +++++- test/pytest/test_projection_parallel.py | 55 +++++++++++++++++++++++++ 3 files changed, 72 insertions(+), 1 deletion(-) diff --git a/test/projection_parallel.sh b/test/projection_parallel.sh index fc9ded0a..47eceee0 100755 --- a/test/projection_parallel.sh +++ b/test/projection_parallel.sh @@ -207,6 +207,12 @@ explain_cov_spc() { # $1 = seq_page_cost -c "EXPLAIN (COSTS OFF) $Q;" 2>/dev/null || true } +# THE TWO ARMS ARE A PAIR AND NEITHER HALF IS SOUND ALONE (@jdatcmd, review). +# The second cannot separate "I/O is left whole" from "there is no parallel +# covering path at all": both read projection-only. The premise at +# seq_page_cost=1 is what excludes the second reading, so deleting it as +# redundant leaves a passing arm that proves nothing. +# # The ladder is printed rather than summarised: a failure of the second arm has # two possible causes -- the partial total amortising I/O, or a host where the # covering projection is not smaller than the base's read columns so no page diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 8e58a7b1..c1bb4267 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -6018,9 +6018,19 @@ shell twin uses `cvppar` / `byik` / 32000 rows / `ik BETWEEN 40 AND 8039`; this file uses `pcvgath` / `onskey` / 50000 rows / `skey BETWEEN 200 AND 12299`. Assertion names match. +It also walks a page-cost ladder (#1209). Core leaves disk I/O whole and +amortises only CPU, so raising `seq_page_cost` until the base relation's I/O +dominates its decode CPU must cost the parallel covering path; if the partial +covering total divided both, an I/O-bound scan would be quoted at 1/N and Gather +would keep winning however expensive the pages became. It is not a guard on the +clamp in that block: `unclamped - clamped = (pre - projRun) * (1 - 1/divisor)` +is strictly positive, so removing the clamp makes the partial path dearer and no +plan moves. The threshold is scale-invariant, so the two fixtures' different row +counts need no different numbers. + | test | what it asserts | | --- | --- | -| `test_projection_parallel` | the table and covering projection exist; a serial covering query uses the projection; a parallel base scan is available when the projection is off; a covering projection can be a parallel scan; a parallel covering projection returns the covering rows once; EXPLAIN ANALYZE launched two workers and printed a rows= line for each; both launched workers produced rows | +| `test_projection_parallel` | the table and covering projection exist; a serial covering query uses the projection; a parallel base scan is available when the projection is off; a covering projection can be a parallel scan; a parallel covering projection returns the covering rows once; EXPLAIN ANALYZE launched two workers and printed a rows= line for each; both launched workers produced rows; the covering plan is parallel at the default page cost; a page cost that makes I/O dominate costs the parallel covering path | ## 78. test_ttl_expire.py: the one function that deletes rows, tested twice diff --git a/test/pytest/test_projection_parallel.py b/test/pytest/test_projection_parallel.py index 5e806274..f9a6ef19 100644 --- a/test/pytest/test_projection_parallel.py +++ b/test/pytest/test_projection_parallel.py @@ -92,6 +92,22 @@ def _plan(conn, sql, workers, projection_scan, analyze=False): return cur.fetchone()[0] +def _plan_at_page_cost(conn, sql, spc): + """The same plan with seq_page_cost raised until the base I/O dominates. + + Its own cursor and its own SETs. Nothing here reads the shell suite: the + assertion NAMES match so the two can be compared by name, which is the only + thing the two harnesses share. + """ + with conn.cursor() as cur: + _apply_parallel(cur) + cur.execute("SET max_parallel_workers_per_gather = 2") + cur.execute("SET pgcolumnar.enable_projection_scan = on") + cur.execute(f"SET seq_page_cost = {spc}") + cur.execute("EXPLAIN (FORMAT JSON, COSTS OFF) " + sql) + return cur.fetchone()[0] + + def _count(conn, sql, workers, projection_scan): with conn.cursor() as cur: _apply_parallel(cur) @@ -206,3 +222,42 @@ def test_projection_parallel(pgc_conn, expect): 2, "workers share the covering projection scan, it is not a single claimer", ) + + # ---- the page-cost ladder (#1209) ------------------------------------ + # + # Core leaves disk I/O whole and amortises only CPU, so raising + # seq_page_cost until the base relation's I/O dominates its decode CPU must + # cost the PARALLEL covering path. If the partial covering total divided + # both, an I/O-bound scan would be quoted at 1/N and Gather would keep + # winning however expensive the pages became. + # + # NOT A GUARD ON THE CLAMP, and the sign says why: + # + # unclamped - clamped = (pre - projRun) * (1 - 1/divisor) + # + # pre > projRun IS the binding condition and divisor > 1 always, so removing + # the clamp makes the partial path DEARER and no plan moves. Measured on the + # shell side too: the clamp-removal mutation reddens nothing. + # + # THE TWO ARMS ARE A PAIR AND NEITHER HALF IS SOUND ALONE (@jdatcmd, + # review). The second cannot separate "I/O is left whole" from "there is no + # parallel covering path at all": both read projection-only. The premise at + # seq_page_cost = 1 is what excludes the second reading, so deleting it as + # redundant leaves a passing arm that proves nothing. + # + # The threshold is scale-invariant -- both C and (A - B) are proportional to + # the row count -- so this fixture's different N needs no different number. + ladder = {spc: _shape(_plan_at_page_cost(pgc_conn, sql, spc)) + for spc in (1, 1024, 4096, 1000000)} + print("-- page-cost ladder:", ", ".join(f"{k}={v}" for k, v in ladder.items())) + + expect.text( + ladder[1], + "gather+projection", + "premise: the covering plan is parallel at the default page cost", + ) + expect.text( + ladder[1000000], + "projection-only", + "a page cost that makes I/O dominate costs the parallel covering path", + ) From 5d64231cf498cec514fe2cbe135c8e47e6b5dfc0 Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" <136637981+ChronicallyJD@users.noreply.github.com> Date: Wed, 23 Sep 2026 09:08:58 -0600 Subject: [PATCH 4/5] comment: the threshold is not scale-invariant, and the arm is safe by margin (#1209) @jdatcmd's review. I wrote that the threshold is scale-invariant because C and (A - B) are both proportional to N. The claim rested on two fixtures that agreed -- and both were 20,000 rows, so their agreement says the margin is insensitive to CONTENT and says nothing about the row count. The measurement that settles it was already in the output. The rung at seq_page_cost = 4096, on three fixtures: 20,000 rows, range 300 projection-only threshold 2625 32,000 rows, range 8,000 gather+projection threshold above 4096 50,000 rows, range 12,100 gather+projection threshold above 4096 It GROWS with N: cpuRun scales while basePagesRead - projPages came out at 2 pages regardless, so 2625 * 32/20 = 4200 and 2625 * 50/20 = 6560, both above 4096, which is what the rungs show. So the arms are safe by MARGIN -- 1000000 is roughly 150x the largest threshold observed -- not by invariance. No rung near a crossover is asserted, because that would rest on whatever ANALYZE sampled that day. This is the same defect the PR exists to correct, one level in: a comment claiming slightly more than it measured. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP --- CHANGELOG.md | 13 ++++++++++++- test/projection_parallel.sh | 19 +++++++++++++++++++ test/pytest/TESTS.md | 8 ++++++-- test/pytest/test_projection_parallel.py | 20 ++++++++++++++++++-- 4 files changed, 55 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7212b096..07d694a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -54,7 +54,18 @@ true until the next version shipped. covering path wins either way. The clamp keeps `cpuRunProj` from going negative, an internal quantity no plan exposes. - Two arms in `projection_parallel.sh` pin the page-cost ladder. They are named for + The threshold is **not** scale-invariant, which an earlier draft of this entry + claimed. That claim rested on two fixtures that agreed and were both 20,000 + rows, so their agreement said the margin is insensitive to content and said + nothing about the row count. Measured on three fixtures at + `seq_page_cost = 4096`: 20,000 rows binds, 32,000 and 50,000 do not. It grows + with N, because `cpuRun` scales while `basePagesRead - projPages` stayed at 2 + pages. The arms are safe by margin -- `1000000` is roughly 150x the largest + threshold observed -- rather than by invariance, and no rung near a crossover is + asserted. + + Two arms in `projection_parallel.sh` and its pytest twin pin the page-cost + ladder. They are named for I/O amortisation rather than for the clamp, because the clamp-removal mutation does not redden them and an arm named for it would have been vacuous. What does redden the second is `(ioRunProj + cpuRunProj) / divisor`. diff --git a/test/projection_parallel.sh b/test/projection_parallel.sh index 47eceee0..039453c4 100755 --- a/test/projection_parallel.sh +++ b/test/projection_parallel.sh @@ -207,6 +207,25 @@ explain_cov_spc() { # $1 = seq_page_cost -c "EXPLAIN (COSTS OFF) $Q;" 2>/dev/null || true } +# THE THRESHOLD IS NOT SCALE-INVARIANT, and an earlier draft of this +# comment said it was (@jdatcmd caught it). The claim rested on two fixtures +# that agreed -- and both were 20,000 rows, so their agreement says the +# difference is insensitive to CONTENT and says nothing about N. Measured on +# three fixtures, the rung at seq_page_cost = 4096: +# +# 20,000 rows, range 300 projection-only threshold 2625 +# 32,000 rows, range 8,000 gather+projection threshold above 4096 +# 50,000 rows, range 12,100 gather+projection threshold above 4096 +# +# It GROWS with the row count. C = cpuRun * projScale scales with N while +# basePagesRead - projPages came out at 2 pages regardless, so the quotient +# rises: 2625 * 32/20 = 4200 and 2625 * 50/20 = 6560, both above 4096, which +# is what the two rungs show. +# +# So THE ARM IS SAFE BY MARGIN, NOT BY INVARIANCE. 1000000 is roughly 150x +# the largest threshold observed. Pinning a rung near a crossover would rest +# on whatever ANALYZE sampled that day; this does not. +# # THE TWO ARMS ARE A PAIR AND NEITHER HALF IS SOUND ALONE (@jdatcmd, review). # The second cannot separate "I/O is left whole" from "there is no parallel # covering path at all": both read projection-only. The premise at diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index c1bb4267..dc382895 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -6025,8 +6025,12 @@ covering total divided both, an I/O-bound scan would be quoted at 1/N and Gather would keep winning however expensive the pages became. It is not a guard on the clamp in that block: `unclamped - clamped = (pre - projRun) * (1 - 1/divisor)` is strictly positive, so removing the clamp makes the partial path dearer and no -plan moves. The threshold is scale-invariant, so the two fixtures' different row -counts need no different numbers. +plan moves. The threshold is NOT scale-invariant: it grows with the row count, because +`cpuRun` scales with N while `basePagesRead - projPages` came out at 2 pages +regardless. Measured at `seq_page_cost = 4096`, 20,000 rows binds and both +32,000 and 50,000 do not. The arm is safe by margin, 1000000 being roughly 150x +the largest threshold observed, rather than by invariance, and no rung near a +crossover is asserted. | test | what it asserts | | --- | --- | diff --git a/test/pytest/test_projection_parallel.py b/test/pytest/test_projection_parallel.py index f9a6ef19..31440407 100644 --- a/test/pytest/test_projection_parallel.py +++ b/test/pytest/test_projection_parallel.py @@ -245,8 +245,24 @@ def test_projection_parallel(pgc_conn, expect): # seq_page_cost = 1 is what excludes the second reading, so deleting it as # redundant leaves a passing arm that proves nothing. # - # The threshold is scale-invariant -- both C and (A - B) are proportional to - # the row count -- so this fixture's different N needs no different number. + # THE THRESHOLD IS NOT SCALE-INVARIANT, and an earlier draft of this + # comment said it was (@jdatcmd caught it). The claim rested on two fixtures + # that agreed -- and both were 20,000 rows, so their agreement says the + # difference is insensitive to CONTENT and says nothing about N. Measured on + # three fixtures, the rung at seq_page_cost = 4096: + # + # 20,000 rows, range 300 projection-only threshold 2625 + # 32,000 rows, range 8,000 gather+projection threshold above 4096 + # 50,000 rows, range 12,100 gather+projection threshold above 4096 + # + # It GROWS with the row count. C = cpuRun * projScale scales with N while + # basePagesRead - projPages came out at 2 pages regardless, so the quotient + # rises: 2625 * 32/20 = 4200 and 2625 * 50/20 = 6560, both above 4096, which + # is what the two rungs show. + # + # So THE ARM IS SAFE BY MARGIN, NOT BY INVARIANCE. 1000000 is roughly 150x + # the largest threshold observed. Pinning a rung near a crossover would rest + # on whatever ANALYZE sampled that day; this does not. ladder = {spc: _shape(_plan_at_page_cost(pgc_conn, sql, spc)) for spc in (1, 1024, 4096, 1000000)} print("-- page-cost ladder:", ", ".join(f"{k}={v}" for k, v in ladder.items())) From 18de86cf7b87a8725d7089cde7a2e75a322cd9a9 Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" <136637981+ChronicallyJD@users.noreply.github.com> Date: Wed, 23 Sep 2026 09:09:35 -0600 Subject: [PATCH 5/5] docs: say why the twin reads JSON where the shell twin greps text (#1209) @jdatcmd's observation, and it was an accident of how each half was written rather than a decision: a second instrument is worth what its failures do not share. A renamed field, a malformed plan or a truncated pipe hits a JSON node walk and a text grep in different places. Two greps with the same blind spot are one run twice, and agreeing by name would not have told us. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP --- test/pytest/test_projection_parallel.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/test/pytest/test_projection_parallel.py b/test/pytest/test_projection_parallel.py index 31440407..22f04ebc 100644 --- a/test/pytest/test_projection_parallel.py +++ b/test/pytest/test_projection_parallel.py @@ -16,6 +16,13 @@ count. Independent of test/projection_parallel.sh: same public seam, own fixture, own observations. Assertion names match the shell suite so the two can be compared by name, not by importing each other. + +A SECOND INSTRUMENT IS WORTH WHAT ITS FAILURES DO NOT SHARE (@jdatcmd). +This file reads `EXPLAIN (FORMAT JSON, COSTS OFF)` and walks the node tree; +the shell suite greps the text plan for `Columnar Projection: byik`. A +renamed field, a malformed plan or a truncated pipe hits those two in +different places, which is the point of having both. Two greps with the same +blind spot are one run twice, and agreeing by name would not have told us. """