cost: the parallel covering clamp is reachable, and decides nothing (#1209)

[OffgridwithJD]

Closes #1209.

The comment on the parallel covering clamp has read UNPROVEN since #1155 falsified #1127's reason for calling it unreachable. This replaces that with what was measured, and the measurement turned up something the issue did not ask about: the clamp decides nothing.

It is reachable, and the threshold was predicted before it was observed

Probed with an elog at the clamp site itself, so nothing is inferred from a total. Both sides are linear in seq_page_cost, because the CPU term is not:

  pre = ioRun * projScale = spc * A          A = 3.1
  projRun                 = C + spc * B      B = 3.0   C = 262.5

Those three constants reproduce six probe points to the decimal, so binding needs spc > C/(A-B) = 2625. Predicted, then measured: 2048 does not bind, missing by 0.9% (6348.8 against 6406.5), and 4096 does (12697.6 against 12550.5).

When it binds, the plan changes, which is the consequence #1127 wrote down:

  up to 2048   Gather -> Parallel Custom Scan (PgColumnarScan) on cvppar
  at 4096      Custom Scan (PgColumnarScan) on cvppar   cost=0.00..12550.50

At the default settings it cannot bind, for a constant rather than a fixture

Per row, binding at seq_page_cost = 1 needs the projection to save more than 5.12 * W + 82 bytes, where W is the analyzed width of the columns read, because cpu_operator_cost * W/4 is 5.12 times seq_page_cost * W/8192. The saving is bounded by what the base stores for those columns. Measured storage: 50.0 and 17.2 bytes per row against W ~ 392, so 0.13x and 0.04x where 5.12x is needed.

A second fixture built to move that margin did not move it. With the blob constant across runs of 500 sort-key values and the rows inserted scrambled, so only the clustered projection sees the runs, basePagesRead - projPages came out at 2 pages in both fixtures and the threshold landed at 2625 twice. The base compresses the same data almost as well as the projection does. That is how the margin is known to be structural rather than incidental, and it is in the comment as prominently as the threshold.

The clamp changes no plan, and that is why these arms are not named for it

I wrote the arms as clamp arms and ran the removal mutation before writing them up. Removing if (ioRunProj > projRun) ioRunProj = projRun; entirely:

  control  rc=0  FAILs=0
  mutant   rc=0  FAILs=0     every rung of the ladder identical

The arithmetic says why. At the binding point, with pre > projRun and divisor > 1:

  clamped     startup + projRun
  unclamped   startup + pre + (projRun - pre)/divisor

The unclamped total is larger. The serial covering path wins either way, so no plan moves. The clamp earns its place by keeping cpuRunProj from going negative, which is an internal quantity no plan exposes. An arm named for it would have passed with it removed.

So the comment now ends: "Do not write a test that claims to guard it through a plan shape: that arm passes with the clamp removed, which is how this was found."

What the arms do guard, found by looking for a mutation that reddens them

Amortising the I/O in the partial covering total -- the pre-#1127 defect, an I/O-bound scan quoted at 1/N:

  (ioRunProj + cpuRunProj) / divisor
    -> a page cost that makes I/O dominate costs the parallel covering path:
       got [gather+projection] want [projection-only]

The ladder is printed rather than summarised, because a failure of that arm has two causes -- the total amortising I/O, or a host where the covering projection is not smaller than the base's read columns -- and the rungs tell them apart.

Run

Five majors, serially, in the pgcolumnar-audit container. Identical ladder on every one:

  pg15a  pg16a  pg17a  pg18a  pg19a    rc=0   projection_parallel.sh PASSED

    seq_page_cost=1        gather+projection
    seq_page_cost=1024     gather+projection
    seq_page_cost=4096     gather+projection
    seq_page_cost=1000000  projection-only

docs_style.sh PASSED. Two ledger rows for two checks, names matching both ways, one carrying the observed red; census recounted on this tree: 1589 rows, 1527 never, 62 not never.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP

[jdatcmd]

Not approving 6af60701: three legs are RED, not two pending. They have settled since the PR was opened.

  fail   pytest (cluster tests, PG 17)
  fail   pytest (cluster tests, PG 18)
  fail   pytest (harness guards, no database)
  pass   the other 12

The cause, from the log rather than from the summary

AssertionError: every pair declared one-for-one still grades one-for-one:
  got  '... projection_parallel=1 ...'
  want '... projection_parallel=0 ...'

and the cluster legs fail the same way through test_harness_deps. The shell suite gained arms and the pytest twin did not, so the pair that was declared one-for-one no longer grades that way. test/pytest/test_projection_parallel.py exists and is untouched by this PR; the owner's rule is that both harnesses ship in the same change.

Reconcile the 1 against the 2 rather than adding one twin and stopping. The PR adds TWO check calls and the grader reports ONE unmatched. That difference has a cause, and the cause decides what the fix is: if the premise is matched or exempted then one twin closes it, and if the grader is counting something else then one twin leaves the leg red and the second attempt looks like flakiness. Read what _names extracts from the new shell text before writing the port.

What I checked and found sound

shape fails CLOSED on a dead cluster, which is the thing I went looking for. explain_cov_spc ends in || true, so a failed psql yields empty output, and the second arm's want is projection-only — a want that a degenerate answer could satisfy would be vacuous. Driven:

  shape ""                       -> neither
  shape "ERROR: something"       -> neither
  shape "psql: could not connect"-> neither

neither, not projection-only. The arm goes red on a dead cluster rather than passing. I expected to file this and it is not there.

EXPLAIN (COSTS OFF) is consistent with the suite's existing helper at :74, so the Columnar Projection: byik line the shaper needs is printed under the same options the existing arms already rely on.

The sign is right, verified against src/columnar_customscan.c:3392-3399 rather than taken from the comment:

  unclamped - clamped = (pre - projRun) * (1 - 1/divisor)

  divisor 1.7   clamped 12550.5000   unclamped 12611.0706   +60.5706
  divisor 2.0   clamped 12550.5000   unclamped 12624.0500   +73.5500
  divisor 2.4   clamped 12550.5000   unclamped 12636.3083   +85.8083
  divisor 3.0   clamped 12550.5000   unclamped 12648.5667   +98.0667

Strictly positive at every worker count, which is what makes "removing the clamp cannot move a plan" a fact rather than an observation at one operating point.

On the second arm's name, since you asked me to attack it

a page cost that makes I/O dominate costs the parallel covering path is accurate about the mutation that reddens it. One thing it cannot do alone: distinguish "I/O is left whole" from "there is no parallel covering path at all", since both give projection-only. The premise arm at seq_page_cost = 1 covers exactly that, so the PAIR is sound and neither half is sound by itself. Worth a line in the comment saying so, because a later edit that drops the premise as redundant would leave a passing arm that proves nothing.

The reporting note

The PR was described as 13 of 15 with two running. It is 12 of 15 with three failed. A leg that is still running and a leg that has failed are the same absence of a green tick in a summary read too early, which is the dirty-PR shape from this morning wearing different clothes: the reading was taken before the state existed.

[jdatcmd]

Approving 18de86cf. The scale-invariance correction is in all four places, the arms are documented as safe by margin, and nothing pins a rung near the crossover.

The correction is better evidence than the claim would have been

I checked the model against all three fixtures rather than taking the conclusion:

  N=20000   threshold 2625.0   4096 binds      predicts projection-only     observed projection-only
  N=32000   threshold 4200.0   4096 does not   predicts gather+projection   observed gather+projection
  N=50000   threshold 6562.5   4096 does not   predicts gather+projection   observed gather+projection

threshold(N) = 2625 * N/20000, fitted at 20,000 and predicting the other two. No free parameters. And the margin figure checks: 1000000 / 6562.5 = 152, so "roughly 150x the largest threshold observed" is right.

"Scale-invariant" would have been one unmeasured sentence. What replaced it is a mechanism with a prediction and two confirmations: cpuRun scales with N while basePagesRead - projPages stayed at 2 pages, so the quotient rises. A lost claim traded for a model.

The refuting rung was printed two lines above the claim that contradicted it, in a file the author wrote, and pytest -q swallows the print. That is a third state next to measured and unmeasured: measured and unreadable in the run everyone does.

Verified at this head

  ledger rows for projection_parallel      11
  shell arms                               11
  pytest names                             11
  both new names present in sh, py, ledger  1 / 1 / 1 each
  census                                 1589 rows, 1527 never, 62 not never, 1527 + 62 == 1589
  budget states                          1527, agrees
  checks                                 15 of 15 success, 0 pending
  name set vs the last merged head       0 missing, 0 extra
  mergeState                             CLEAN

The twin guards the defect rather than the name: the same (ioRunProj + cpuRunProj) / divisor mutation reddens it with the property and both shapes in the message. It is independent — own cursor, own SETs, own fixture, reads nothing under test/. And it uses EXPLAIN (FORMAT JSON, COSTS OFF) against the shell half's plain COSTS OFF, so a malformed plan, a renamed field and a truncated pipe hit the two halves in different places. A second instrument is worth what its failures do not share; two greps with the same blind spot are one run twice. Good that this is now in the docstring rather than an accident of how each half was written.

A note about my own gate, because it bears on how this was reviewed

My CI waiter reported this PR settled when the head carried zero check-runs. pending == 0 is trivially true over an empty list, mergeStateStatus was CLEAN, and gh pr checks exits 0 on "no checks reported". Neither "demand a definite state" nor the exit code distinguishes it.

The approval above rests on a rebuilt gate with three clauses rather than two: the check COUNT must reach an expected set taken from the last merged head, pending must be zero, and non-success must be zero, with missing and extra NAMES printed. The general form is the one this PR keeps demonstrating in other registers: every per-entry predicate is true of an empty list, so a gate phrased as "nothing is wrong" passes when there is nothing.