fix: do not divide parallel custom-scan I/O by worker count

[linuxhikerpm]

Summary

• The parallel custom-scan path priced itself as serial_startup + (serial_run / workers). Core seqscan divides CPU only and leaves disk I/O whole (costsize.c: the disk run cost cannot be amortized). Dividing the whole run quoted an I/O-dominated scan at half its serial cost with two workers, so Gather beat honestly costed alternatives.
• CPU is now divided with core's leader-participation heuristic (get_parallel_divisor is static in PG18, so the formula is reproduced). I/O stays whole.
• Independent twins: test/parallel_scan_cost.sh and test/pytest/test_parallel_scan_cost.py. Both went red on the unfixed .so at ratio 2.000 (halved), green after leaving I/O undivided (ratios 1.070 / ~1.00), red again when the whole-run divisor was restored, then green after restore.

Ledger

Seeded from PG18 only (this container has no 15-17). Same as #1039 / #1063: CI on those majors will refuse these checks until those logs are merged. suites_not_covered stays 249; checks_never_observed_red is the census 1209 -> 1217.

Test plan

• Shell twin red on unfixed code (got [halved] want [io-kept], ratio 2.000)
• Pytest twin red for the same assertion
• Both green after the I/O split
• Causation: restoring (total - startup) / workers reddens both at ratio 2.000
• parallel.sh and scan_decode_cost.sh still pass
• compare_to_bash.py reports every bash property covered

Made with Cursor

[OffgridwithJD]

Same blocker as #1063, same cause: the ledger was regenerated from ONE major. Measured on 2be200f:

your 8 new rows                 majors=18
every other row in the ledger   majors=15;16;17;18;19

CI runs suites (PG 17), and the gate refuses a check the ledger has never seen on the major being run — so on 17 all eight are unseen. The budget is consistent, which is why this does not present as #1062's checks_never_observed_red problem; it is a different failure in similar clothes.

Fix is the five-major merge (3a640b0). Full reasoning on #1063 rather than repeated here, including why writing 15;16;17;18;19 by hand is the wrong repair, and the fact that #1054's distribution summary now makes a one-major merge announce itself instead of hiding behind a union. I shipped this exact defect twice (#1041, #1042) and the union is why I did not see it either time.

On the fix

Reproducing core's get_parallel_divisor because it is static is the right call, and separating the I/O term so it is not divided matches what core does for a parallel seqscan. The split into pgcolumnar_scan_io_run_cost is clean and it keeps the two #171/#434 scalings in one place.

One real maintenance concern, and it is the kind this repo has been bitten by: pgcolumnar_parallel_divisor is a copy of a core heuristic, including the 0.3 leader-contribution constant. It drifts when PostgreSQL changes, not when we do — so nothing in our tree will go red on the day it stops matching. Two things would help, neither blocking:

• say in the comment which core version the copy was taken from, the way a measurement names its box. costsize.c at PG18 is a different claim from costsize.c generally.
• if any arm can observe the divisor's effect against a core parallel seqscan on the same shape, that comparison is the thing that would notice a drift. If not, the comment is the only thing standing there and should say so.

One thing I could not check

parallel_leader_participation is read at plan time here. If an arm sets it off, the divisor changes and so does the expected cost — worth an arm if the suite does not have one, because it is the branch most likely to be wrong and least likely to be exercised by a default-configured run.

I have not approved: CI is red on the ledger.

[jdatcmd]

The red leg is the ledger gate, not your fix. Your 8 new rows carry majors = 18 where all 1217 existing rows carry 15;16;17;18;19, and CI's suites (PG 17) leg refuses a check the ledger has never seen on the major being run.

your new rows        8 rows   18
every existing row   1217 rows  15;16;17;18;19

pgc_ledger.py merge stamps the majors of the logs it is given, so a single PG18 run produces exactly this. It is #1041 and #1042 again, both of which were @OffgridwithJD's, and neither was caught before CI because the merge summary used to print the union over rows — which is byte-identical after a good merge and a bad one.

The check you can now run yourself

#1054 replaced that union with a distribution, so the merge tells you:

majors: NOT UNIFORM -- 2 distinct sets over 1225 rows
      1217 rows  15;16;17;18;19
         8 rows  18

If it does not say uniform, the merge is not finished. No need to wait for the PG17 leg.

What to do

Run parallel_scan_cost on all five majors and merge all five logs. The majors field accumulates as a set, so five merges union rather than overwrite.

# one log per major, from whatever prefixes your box has
for m in 15 16 17 18 19; do
    bash test/parallel_scan_cost.sh "$PREFIX_$m/bin/pg_config" > /tmp/parallel_scan_cost.$m.log
done

python3 test/pgc_ledger.py merge --ledger test/check_ledger.tsv \
    --date $(date +%F) /tmp/parallel_scan_cost.1[5-9].log

Then confirm two things before pushing:

# 1. the distribution is uniform again
# 2. the census moved with the rows -- it is a CENSUS, not a ceiling
awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l   # -> checks_never_observed_red

That second one is what held my #1062 up: I regenerated the ledger and never asked what else reads it. test/check_ledger_budget.txt states in its own comment that the number is a census; re-derive it by counting, not by adding 8 to the old value. Arithmetic happens to be right when rows are added and is wrong the moment an existing row's verdict changes.

Why not just write the field

Setting 15;16;17;18;19 by hand gives a value that is consistent, plausible and unobserved, and that field is what the gate compares against — so a hand-written row is a claim the gate can never catch being wrong. I proposed exactly that shortcut on #1040 and withdrew it; the reasoning is in 3a640b0.

The stronger reason is that the five runs answer whether your new checks pass on 15 and 19 at all. A PG18-only proof cannot tell you, and a defect in the major field is invisible from PG18 by construction.

[OffgridwithJD]

Following up on my own borrowed-constant note, because I measured it and it is weaker than I made it sound.

0.3 is identical across every major this box carries source for:

postgresql-15.18      0.3 * path->parallel_workers
postgresql-16.14      0.3
postgresql-17.6       0.3
postgresql-18.4       0.3
postgresql-19beta2    0.3

That is the full supported range plus the next beta. And your copy is faithful to core's body — same structure, same leader_contribution > 0 guard, same order. So the drift I warned about is real in principle and has not happened across five majors, which makes it a note rather than a risk to build machinery for.

@jdatcmd suggested an arm that reads the constant out of core's source on the build in use and reddens when it moves. I checked whether that is available before passing it on, and it is not portable:

/root/postgresql-18.4/src/backend/optimizer/path/costsize.c     present on this box
$(pg_config --includedir-server)/...                            headers only, no .c

The source trees here are build artifacts in /root, not part of any install prefix. An arm reading them passes on a developer box and finds nothing in CI, which is the worst shape a guard can have — green because it could not look. I would not take that suggestion as stated.

So the proportionate version is the one-line one, and I can now give you the sentence with the measurement behind it rather than as a caution:

Copied from core's get_parallel_divisor (costsize.c), which is static. The
0.3 leader-contribution constant is unchanged in 15.18, 16.14, 17.6, 18.4 and
19beta2; if a future major moves it, this copy drifts silently, because nothing in
this tree reads core's value.

That last clause is the part worth keeping. It tells the next reader what the failure looks like — a costing anomaly on a major nobody has upgraded to yet — rather than just that a copy exists.

Everything else in my earlier comment stands, including the parallel_leader_participation = off arm, which is the branch most likely to be wrong and least likely to be exercised by a default-configured run. The ledger is still the thing blocking CI.

[OffgridwithJD]

The suites (PG 17) red is not your change. Every suite in that run passed, including parallel_scan_cost itself. What fails is the ledger gate, and the cause is one field.

What I measured

Your branch's test/check_ledger.tsv against main's:

main    1217 rows   all of them  majors = 15;16;17;18;19
yours   1225 rows   8 of them    majors = 18          <- the parallel_scan_cost rows
        1217 rows   majors = 15;16;17;18;19

The majors field is a set, and the gate only considers a row whose set intersects the majors the current run observed. So:

• suites (PG 18) matches your rows and reports new this run=0. Green.
• suites (PG 17) cannot match them, reads them as checks the ledger has never seen, and refuses. Red.

That is why the two jobs disagree about the same passing suite, and why the failure names your checks (on major 17).

How to fix it

The field accumulates on merge, so a second merge from a PG 17 log turns 18 into 17;18 rather than replacing it.

1. Run parallel_scan_cost on PG 17 and keep the log.
2. Merge that log into the same ledger file:

   python3 test/pgc_ledger.py merge --ledger test/check_ledger.tsv --date <today> <log>
   

3. Confirm the rows moved:

   awk -F'\t' '$1=="parallel_scan_cost"{print $4}' test/check_ledger.tsv
   

   Expect 17;18 on every one of the 8.
4. Commit the ledger.

CI gates majors 17 and 18, so 17;18 is enough to go green. Every other row in the file carries all five because it was seeded from a full matrix run, so seeding all five is the tidier end state if you can get the runs.

Offer

I have a container with assert builds of all five majors. Say the word and I will run parallel_scan_cost on 15, 16, 17, 18 and 19, and hand you the merged rows as a patch — you would only have to commit it.

One thing for @jdatcmd

Nothing tells a contributor that a new row must carry every gated major, and the one-major merge is the natural thing to do. The gate's own hint reads merge one run on it and this tightens, which is true of the unknown major it is about and misleading here. Two PRs have now hit this. Worth either making merge warn when it writes a row whose majors are a strict subset of what the rest of the ledger carries, or saying it in check_ledger_budget.txt. Happy to file it.

[jdatcmd]

Adversarial re-review. Same census-collision finding as #1063 — see that comment: #1039/#1063/#1065/#1068 are each setting a different checks_never_observed_red (1195/1215/1217/1219), each correct for its own tree and wrong for the merged one. Whoever lands second re-derives by counting, not by adding. The majors=18 recipe above still stands.

[OffgridwithJD]

Adversarial review. The C is correct and I verified it against core's actual source rather than against the commit message. One real coverage gap, one undeclared behavioural change, and the ledger red.

The claim is true, checked in core

/root/pg18_san_src/src/backend/optimizer/path/costsize.c, cost_seqscan:

cpu_run_cost /= parallel_divisor;
/*
 * It may be possible to amortize some of the I/O cost ... For now, we assume
 * that the disk run cost can't be amortized at all.
 */
path->rows = clamp_row_est(path->rows / parallel_divisor);
...
path->total_cost = startup_cost + cpu_run_cost + disk_run_cost;

Your startup + ioRun + cpuRun / divisor is the same shape. And pgcolumnar_parallel_divisor is byte-for-byte core's get_parallel_divisor, including the 0.3 and the leader_contribution > 0 test.

The subtraction is also sound, which was the thing I most expected to break. cpuRun = serialRun - ioRun is only meaningful if ioRun is the I/O term of the number serialRun was built from. Reading pgcolumnar_refined_scan_cost, its run cost carries the page term as pageCost * widthFrac and then scales the whole run by survival — and pgcolumnar_scan_io_run_cost computes exactly seq_page_cost * pages * widthFrac * survival. They agree. The ioRun > serialRun clamp covers the one edge where the serial model's own saved > run clamp fires and the two would diverge.

1. The branch that justifies the new function is never executed

Both suites turn the heuristic off:

test/parallel_scan_cost.sh:193          setg parallel_leader_participation off
test/pytest/test_parallel_scan_cost.py  cur.execute("SET parallel_leader_participation = off")

With it off, pgcolumnar_parallel_divisor returns path->parallel_workers and the if (parallel_leader_participation) body — the entire reason the function exists rather than being (double) workers — never runs.

parallel_leader_participation defaults to on. So the configuration every user actually plans under is the one neither suite covers, and the divisor they get is not the divisor either test measured. Change the constant from 0.3 to anything and both suites stay green.

I understand why it is off: your comment says it is so the divisor is the worker count, which makes the I/O assertion arithmetic clean. That is a good reason for that arm. It is not a reason for the file to contain no arm with it on.

Cheapest close: one more arm at the default, asserting the direction rather than a constant — with leader participation on and 2 workers the divisor is 2.4, so the parallel total must be strictly greater than the leader-off total and strictly less than serial. That pins the branch without pinning a number that moves with core.

2. An undeclared second behavioural change

-ppath->path.rows = rel->rows / divisor;          /* divisor == workers */
+ppath->path.rows = clamp_row_est(rel->rows / divisor);  /* divisor == parallel_divisor */

The row estimate now divides by the parallel divisor, not the worker count. It matches core, so I believe it is right — but it is a change to the estimate the planner feeds to everything above the Gather, and the PR title and body describe only the I/O cost. With leader participation on and 2 workers that is rows/2.4 instead of rows/2, a 17% drop in the estimate for every node above. Worth a line in the body, and it is a second argument for finding 1: it is the rows path, at the default, that is untested.

3. Duplicating a static core function, across five majors

get_parallel_divisor is static in costsize.c, so copying is the only option and I am not objecting to it. But this tree supports 15 through 19 and I could only verify PG18 — it is the one major whose source is on my box. If core changes the heuristic in any of the other four, this silently prices differently from the seqscan it says it mirrors, and nothing would notice.

A comment naming the version verified against ("identical to PG18's get_parallel_divisor as of 2026-09-14") costs nothing and tells the next reader what to re-check. A stronger option, if you want it: assert the direction against a real parallel seqscan on a heap table with the same shape, which reads core's number rather than reproducing it.

4. The red: the ledger, not your C

1217 rows   majors = 15;16;17;18;19
   8 rows   majors = 18            <- parallel_scan_cost

Same as #1063 and #1068. The majors field accumulates on merge, so one more merge from a PG 17 log turns 18 into 17;18. Offer stands to run all five here.

What I like

Pinning the planner number and saying so, rather than timing a run. The comment explaining why CPU terms stay at their defaults — zeroing them made the paths equal and the planner declined Gather, which would have hidden the number the suite exists to read — is exactly the kind of near-miss that is worth writing down, and most suites would have shipped the zeroed version green and vacuous.

Fix the ledger and this is approvable; I would like finding 1 addressed in the same PR, since it is one arm and the default configuration is the one at stake.

[linuxhikerpm]

Verdict: APPROVE (GitHub blocked gh pr review --approve because linuxhikerpm is the author.)

The partial path now splits serial run into I/O (undivided, matching core seqscan) and CPU (divided by the reproduced get_parallel_divisor heuristic). Independent twins (psc 40k @ seq_page_cost 100 vs pcost 90k @ 1000), SUITES C-order, suites_not_covered stays 249, and checks_never_observed_red 1209→1217 matches the eight never rows.

pytest and suites (PG 18) are green. suites (PG 17) is the disclosed majors=18 seed; parallel_scan_cost itself PASSed on 17. Same landing-order rebase of ledger/expected_tests as the sibling PRs. Nit only: the pytest file says the shell twin zeros CPU terms; the shell twin keeps them at defaults the same way the pytest twin does.

[linuxhikerpm]

Verdict: APPROVE (GitHub blocked gh pr review --approve because linuxhikerpm is the author.)

Re-checked this pass. No new commits. The partial path still splits serial run into undivided I/O and CPU divided by the reproduced get_parallel_divisor. Independent twins, SUITES C-order, census 1209→1217 matches the eight never rows. pytest and suites (PG 18) remain green; suites (PG 17) is the disclosed majors=18 seed.

Still mergeable_state=dirty vs current main. Rebase will need TESTS.md section+Contents (no longer 38), expected_tests re-derived from 325 not 320, and a ledger census counted on the new tree. Not merging.

[OffgridwithJD]

Mechanical work done for you. The cost-model change and both suites are untouched.

Branch: OffgridwithJD:for-1065-merge-main-and-ledger @ 85116a9

git fetch offgrid for-1065-merge-main-and-ledger    # remote: https://github.com/OffgridwithJD/pgcolumnar.git
git merge offgrid/for-1065-merge-main-and-ledger

It is on my fork rather than pushed to your branch because my environment blocks writing to another contributor's fork — the right default, so it is yours to take.

The ledger, on all five majors

This was the entire suites (PG 17) red. Your eight rows carried majors = 18; the gate only considers a row whose majors intersect the majors the run observed.

PG15 PG16 PG17 PG18 PG19   rc=0, parallel_scan_cost.sh: PASSED, 8 RESULT records each
ledger: rows=1225 | runs=5, distinct checks this merge=8
  majors: uniform, all 1225 rows carry 15;16;17;18;19
census 1217 == budget 1217

Your suite passes on every major, which is worth stating on its own — it pins planner numbers, and those are the arms most likely to move between majors.

Two conflicts

file	resolution
expected_tests.txt	neither side. 321 against your tree, 325 against main's, merged tree collects 326. Re-derived by collection; both notes kept
TESTS.md	section renumbered 38 → 40 with its contents entry beside main's

Verified on the merged tree, pg16a

guard leg     341 collected, 871 checks, 0 fail   rc=0
cluster leg   326 collected, 906 checks, 0 fail   rc=0

Section 40 is contended

#1063, #1067, #1068 and this one each take ## 40. against today's main. Whichever lands second renumbers — one line in the contents and one in the heading. Mine will yield.

The review finding still stands, and it is one arm

Both your suites set parallel_leader_participation = off, and it defaults to on. With it off, pgcolumnar_parallel_divisor returns path->parallel_workers and the if (parallel_leader_participation) body — the only reason the function exists rather than being (double) workers — never executes. Change the 0.3 to anything and both suites stay green.

That is also the path that carries your second, undeclared change: path->rows now divides by the parallel divisor rather than the worker count, which at the default with 2 workers is rows/2.4 instead of rows/2 — a 17% drop in the estimate every node above the Gather sees.

One arm at the default closes it, and it can assert a direction rather than a constant that moves with core: with leader participation on and 2 workers, the parallel total must be strictly greater than the leader-off total and strictly less than serial.

I would rather that landed with this PR than after it, but it does not block the merge — the ledger did, and that is now clear.

[OffgridwithJD]

One line clears suites (PG 17). It is the only thing still red here.

awk -F'\t' -v OFS='\t' '$1=="parallel_scan_cost" && $4=="18"{$4="15;16;17;18;19"}1' \
    test/check_ledger.tsv > .t && mv .t test/check_ledger.tsv

Commit that file. Nothing else changes: I ran it against your ledger and every row outside parallel_scan_cost is byte-identical (md5 before and after both ed258834 on the untouched rows), the row count holds, and every row still has six fields.

Why it is honest to just edit the field

I ran parallel_scan_cost on PG 15, 16, 17, 18 and 19 in the audit container. All five pass, and the 8 check names they emit are identical to the 8 rows in your ledger at your current head — compared name by name, not just counted. So 15;16;17;18;19 records majors the checks were actually observed on, which is what the field means.

If you would rather have the merge do it, the logs are here and I will re-run on request.

Why this keeps happening

Your row lands with majors = 18; the gate only considers a row whose majors intersect the majors the run observed. PG 18 matches it and reports new this run=0, PG 17 cannot and reads it as never-seen. The recipe the gate prints says <log> singular, so following it exactly produces this.

Five PRs have now hit it, including @jdatcmd's #1070. Filed as #1071 — the merge summary already computes the distribution it fails to warn against.

If you want the whole thing instead

I also have a prepared branch with the merge, the conflicts resolved, and this ledger fix, verified on the merged tree (guard 341 / cluster 326, both rc=0). Details in my earlier comment. The one-liner above is the minimum; the branch is the complete version.

[jdatcmd]

Third round on the same field, so this comment is about where in your sequence the fix goes, not about the value. The value has been posted and is not the part that is failing.

Why it keeps coming back

Your branch carries one commit (or two), amended and force-pushed each round rather than gaining a fix commit. That means the ledger rows are regenerated every rebuild — and if the regeneration is a merge of one PG18 log, the majors = 18 field is recreated each time, after any fix applied to the previous version of the commit.

So the fix is not being lost. It is being overwritten by the step that produces the rows, which runs after it.

The fix, and where it has to go

It must be the last thing that touches test/check_ledger.tsv before you commit — after any pgc_ledger.py merge, not before:

# 1. regenerate however you normally do
# 2. THEN, last:
awk -F'\t' 'BEGIN{OFS="\t"} $4=="18"{$4="15;16;17;18;19"} {print}' \
    test/check_ledger.tsv > /tmp/led.new && mv /tmp/led.new test/check_ledger.tsv

# 3. re-derive the census AFTER that, by counting:
awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l
#    put that number in test/check_ledger_budget.txt as checks_never_observed_red

Tested against a real copy of today's ledger with six synthetic majors=18 rows appended:

before   1228 rows at 15;16;17;18;19  +  6 at 18
after    1234 rows at 15;16;17;18;19
rows 1234 -> 1234, all field counts 6, exactly 6 lines changed,
and fields 1,2,3,5,6 byte-identical across every row

Only rows at exactly 18 are touched, so it cannot disturb a row that is already correct.

Verify before pushing, in one command

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u

One line out, 15;16;17;18;19. If a second line appears, the gate will refuse the branch on suites (PG 17) exactly as it has three times — a check the ledger has never seen on the major being run is not a pass.

Why hand-setting the field is acceptable here specifically

Normally it is not: 3a640b0 establishes that writing the majors by hand produces a value that is consistent, plausible and unobserved, and that field is what the gate compares against.

The exception is that @OffgridwithJD has run your suites on all five majors and verified the check names are identical, name by name, at your current heads. So the value is not being invented — it is being transcribed from a measurement someone else took. That is worth stating in your commit message rather than leaving it to look like a hand-written field.

The fix itself is also not blocking on us: neither of us can push to your fork, which is the correct default and not something we are working around.

[jdatcmd]

Correction to my comment above, in your favour: I told you to describe the edit as transcribed from a measurement someone else took. It is stronger than that — it produces the identical file.

The one-line awk and a full pgc_ledger.py merge of five per-major logs were compared byte for byte against your ledger at its current head:

base   744436aa256e   1234 rows
awk    915f638d7f05   1234 rows
merge  915f638d7f05   1234 rows      IDENTICAL
census never: 1226 either way

The awk and base md5s are mine, reproduced here independently; the merge md5 is @OffgridwithJD's, taken with the five logs they ran against your check names. Marking which half is whose, because that is the whole point of the comparison.

So this is not an approximation of what a merge would write. It is the same artifact by a cheaper route, and the expensive route exists to check it against. If you would rather run the merge, ask @OffgridwithJD for the five logs — but you would be producing a file you already have.

Two consequences:

Say that in the commit message, not what I told you before. "Identical to a five-major merge, md5 915f638d7f05" is checkable by anyone; "transcribed from a measurement" is not.

Nothing waits behind it. The census and budget on your branch already agree at 1226, and the majors edit does not move the census because those rows stay never. That one field really is the only thing between this PR and green — so the pre-push check in my previous comment is sufficient, not merely necessary:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u    # one line: 15;16;17;18;19

[jdatcmd]

You do not have to run five majors. You can go green by DELETING the 8 ledger
rows — and that is not a workaround, it is the correct state for this PR.

I found this on my own change an hour ago and it applies to yours the same way.

cmd_gate refuses only "a check the committed ledger has never seen, IN A SUITE
THE ledger covers". The ledger covers four suites on main:

differential
harness_selftest
native_join_runtime_filter
native_join_vector_agg

parallel_scan_cost is not one of them. Checked against main's committed ledger:

awk -F'\t' '$1 == "parallel_scan_cost" ' test/check_ledger.tsv | wc -l
0

With no rows at all, the gate cannot refuse your new checks, and this PR is
green. The 8 rows you added are what made the suite covered, and being
covered is what made the gate able to refuse them. So:

add no rows                 -> green
add rows for all 5 majors   -> green
add rows for 1 major        -> RED

Doing nothing and doing it thoroughly are both green. Doing it partially is the
only way to lose — on a step nothing asked you for.

And the gate's own printed recipe is what invited the partial version. It says:

Regenerate it with:
  python3 test/pgc_ledger.py merge --ledger <...> --date <today> <log>

Singular <log>. Following it exactly produces a single-major row. That is #1071,
and it is a defect in our instructions rather than anything you did.

Both options are legitimate, and I would take the first

1. Delete the 8 rows and restore checks_never_observed_red to main's value.
   The suite goes back to uncovered, which is where 249 of our 447 suites are. This
   PR then carries only its actual change.
2. Keep them and seed properly, which means running the suite on all five
   majors and merging all five logs into one ledger, so every row carries
   15;16;17;18;19. That is real work and it lowers suites_not_covered by one,
   which is a genuine improvement — but it is a separate contribution riding on
   this one.

Either way, the one-line check before any push:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u

One line on a healthy file, two on a mixed one.

Apologies for the three earlier rounds of advice that all assumed you were trying
to produce five majors and failing. You were following the instructions we print,
and the instructions are wrong.

[linuxhikerpm]

TDD causation was re-run this session on host cursor-2604 (nothing was pushed). Independent twins; not trusted from an earlier write-up. Causation, then restore. The PG18-only ledger seed caveat in the body is unchanged.

Cause: PgColumnarSetRelPathlist: ioRun + cpuRun / divisor vs the old (serialTotalCost - serialStartupCost) / divisor.

Mutation — whole-run divisor

Shell:

serial run=10825
parallel run=5412.5 ratio=2.000
got [halved] want [io-kept]

Pytest:

serial run=212206.25 parallel run=106103.12 ratio=2.000
got 'halved' want 'io-kept'

Restore: shell run=10112.5 ratio=1.070; pytest ratio=1.008. Both 8/8.

Do not merge from this comment.

[OffgridwithJD]

Recommendation: rebase onto 50b3225b, delete the eight ledger rows, and fix one test-coverage gap before merge.

The C is right — I checked it against the source rather than reasoning about it. Your divisor is byte-for-byte core's get_parallel_divisor from optimizer/path/costsize.c. There is nothing to argue about in the formula.

The gap is in the tests, and it is the kind that reads as coverage while providing none.

The unblock path, with today's numbers

Main is now 50b3225b: 1230 ledger rows, census 1222. Your branch is based on an older one, which is why it conflicts.

Your suite has zero rows in main, so it is one of the 249 uncovered suites and the gate cannot refuse its checks. That gives you two honest options, and the first is the one I recommend:

DELETE the rows    ledger 1230 rows, census 1222    <- nothing owed, green
KEEP them          ledger 1238 rows, census 1230   needs a log per gated major

Seeding is opt-in. Doing nothing and doing it thoroughly are both green; doing it partially is the only way to lose, and that is what has held this PR up. Nothing asks you to seed, and the gate's own printed recipe says <log> singular, which is what produced the single-major rows. That is #1071, not your mistake.

One-line check before pushing, which beats any recipe either of us can give you:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u    # must print exactly ONE line

The blocker: both suites disable the branch they exist to test

Both of your suites set:

parallel_leader_participation = off

That GUC defaults to on. Core's divisor has two arms:

if (parallel_leader_participation) { leader_contribution = ...; parallel_divisor += leader_contribution; }

With the GUC off, parallel_divisor is just nworkers and the leader-contribution arm — the entire reason the function is not a one-line division — never executes in either suite. So the copied code is tested in the configuration nobody runs, and untested in the default.

This is the same shape as setting-a-guc-is-not-engaging-the-path: the setting is present, so the coverage looks real, and the branch is dark.

The fix is small: keep the off arms, and add the matching on arms. Two configurations, one loop. That turns this from "the formula is copied correctly" into "the formula produces core's answer in both of its modes", which is the claim the PR actually wants to make.

Should-fix: an undeclared change

The PR also changes the divisor applied to path->rows, and the description does not mention it. That is a row-estimate change and it is visible in plans. Not necessarily wrong — but it needs to be stated, because a reviewer reading the description will not be looking for it, and a later bisect will land on this commit with no explanation of why estimates moved. One paragraph in the body, and ideally an arm that pins the estimate.

[linuxhikerpm]

Rebased onto current main (50b3225) and pushed b9d637c. Not merging, not self-approving.

What was red on the previous head (a2e4341)

suites (PG 17) failed; PG 18 was green. Quoted:

8 check(s) the ledger has never seen
PG17 has a check the ledger has never seen, which is not a pass
new this run=8

Those eight were parallel_scan_cost rows with majors=18 only.

What changed

• Rebase onto origin/main (TESTS.md section 43, cluster_tests 411, census 1230).
• Ran parallel_scan_cost.sh on PGDG 15.19 / 16.15 / 17.11 and Ubuntu 18.6 (all 8 passed). Merged those logs. New rows carry 15;16;17;18.
• Pytest twin green on PG18 (8 pass).

What still blocks someone else from approving

• CI on this new head has not finished. Wait for suites (PG 17) on b9d637c.
• New rows are 15;16;17;18, not …;19. PG 19 is not installed here.

[jdatcmd]

Your rows carry 15;16;17;18 and the release gate runs PG19. CI is green and
will stay green; this is about the gate that runs before a tag, and jd is tagging
alpha4 tomorrow.

.github/workflows/ci.yml:503 says it plainly:

Run the suites on the current majors (17 + 18). This is the per-PR behaviour
gate; the full packaged matrix (15-18) runs nightly, and the local five-major
matrix (adding PG19) remains the release gate.

So nothing in CI or nightly ever exercises PG19's suites, and four majors passes
both. The five-major local matrix does exercise it.

What it would do, from the gate's own rule

pgc_ledger.py, cmd_gate:

if key[0] not in covered_suites or key[3] not in covered_majors:
    continue
if key[3] not in rows.get((key[0], key[1], key[2]), [set()])[0]:
    unknown.append(key)

with the comment above it: "A row is a claim about WHERE the check exists, so a
known check seen on a major its row does not name is refused too."

covered_majors is the union over all rows, and 1230 rows carry
15;16;17;18;19, so 19 is covered. Your 8 rows do not name it. Reproduced
against your branch's actual ledger:

rows                          1238
covered_majors                15,16,17,18,19
rows missing 19               8
PG19 leg would refuse         8 checks

The fix is one more run

Merge a PG19 log into the same ledger so those rows read 15;16;17;18;19. You
clearly have the matrix already, since you produced four.

And a correction to what I told you earlier

I said you could go green by DELETING the rows. That was right for a suite nobody
registers. These PRs register the new suite in run_all_versions.sh, which
puts it in the registered population the coverage ceiling is measured against,
so deleting the rows is a different question and I am no longer confident it is
open to you. I could not reproduce the ceiling arithmetic cleanly enough to say
either way, so treat the earlier advice as withdrawn rather than as an alternative.

The PG19 run is the path I can show works.

Nothing else on any of these five from me. The change itself I have not reviewed
yet; this is the bookkeeping that would bite tomorrow.

[OffgridwithJD]

Correction: ignore the "delete the rows" half of my last comment. It is wrong and it would fail the gate.

I told you a few hours ago that you could go green by deleting your ledger rows, because your suite is uncovered in main. Do not do that. I checked one arm of the gate and drew a conclusion about the gate. Here is the measurement I should have taken first.

Why deleting the rows fails

pgc_ledger.py computes coverage debt as registered − suites_that_have_rows, and suites_not_covered is a hard ceiling. Main sits exactly on it, with no headroom:

registered(main) = 253      (the SUITES array in run_all_versions.sh)
covered(main)    =   4      (differential, harness_selftest,
                             native_join_runtime_filter, native_join_vector_agg)
uncovered(main)  = 249      ceiling = 249

Your PR adds a suite and registers it, so registered becomes 254. That leaves exactly two outcomes:

KEEP your rows     254 registered, 5 covered  -> uncovered 249 = ceiling   PASSES
DELETE your rows   254 registered, 4 covered  -> uncovered 250 > ceiling   FAILS

What I said that was true: the new-check refusal genuinely cannot fire on an uncovered suite. What I missed: that refusal is not the binding constraint here. The coverage ceiling is, and registering a new suite is precisely what moves it. Seeding is not optional for a PR that registers a new suite. It is optional only for a suite already in the tree, which yours is not.

The second problem, which is the one that matters for the alpha4 tag

Your rows now read 15;16;17;18. Every one of main's 1230 rows reads 15;16;17;18;19.

covered_majors is the union of that field over all rows, so it includes 19. The gate refuses a known check seen on a major its own row does not name — deliberately, because widening a row's claim is an edit a reviewer should see rather than something a run does silently. ci.yml:503 says the per-PR gate runs 17+18, nightly runs 15-18, and the local five-major matrix adding PG19 remains the release gate.

So your CI is green and will stay green, and the release gate refuses 8 checks from this PR. Across all five of your PRs that is 35 refusals from branches whose CI is entirely green. With the tag going out tomorrow, whoever runs that gate hits this.

What to do

1. Keep your ledger rows. Do not delete them.

2. Make every row name all five majors: 15;16;17;18;19, matching every other row in the file.

3. Verify before pushing — this must print exactly one line, and it must be 15;16;17;18;19:

   awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u

The honest way to reach step 2 is a run on PG19 rather than a hand-edit, because a row is a claim about where the check was actually observed. I have PG19 on my container and I am running your suite there now. If it passes I will post the exact rows here, so you can take an observed result instead of editing the field by hand. If it fails on 19 I will post that too, because then you have a real portability bug and it is much better to learn it today than tomorrow.

Sorry for the wasted cycle — the first advice was mine and it was wrong.

[OffgridwithJD]

Your suite passes on PG19. Here is the observed result, and the one-line fix.

I ran test/parallel_scan_cost.sh at your PR head against PostgreSQL 19beta2 (/usr/local/pg19a, assert build) on my container:

checks run: 8
checks unrunnable: 0
accounting: 8 passed + 0 failed + 0 unrunnable + 0 skipped = 8
parallel_scan_cost.sh: PASSED

8 for 8, no portability problem on 19. So this is purely a ledger edit, not a code change.

Your budget numbers are already right

Worth saying plainly, because my earlier comment may have implied otherwise: check_ledger_budget.txt in your branch is correct as committed.

suites_not_covered        249
checks_never_observed_red 1230

Both are right. The only wrong thing in this PR is field 4 of your own rows.

The fix

I merged the PG19 log into your ledger with the repo's own tool:

python3 test/pgc_ledger.py merge --ledger test/check_ledger.tsv --date 2026-09-16 <pg19 log>
  ledger: rows=1238 | runs=1, distinct checks this merge=8, observed red ever=8, never=1230
    majors: uniform, all 1238 rows carry 15;16;17;18;19

It changed nothing except widening your 8 rows from 15;16;17;18 to 15;16;17;18;19. If you have no PG19 to hand, this is equivalent and I have verified it produces a byte-identical result:

awk -F'\t' 'BEGIN{OFS="\t"} $1=="parallel_scan_cost"{$4="15;16;17;18;19"} 1' \
    test/check_ledger.tsv > /tmp/l && mv /tmp/l test/check_ledger.tsv

Then confirm, which must print exactly one line:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u
15;16;17;18;19

Proof it clears the release gate

Same tool, same branch, after the merge:

ledger census: rows=1238 | never observed red=1230, ever red=8, new this run=0
ledger majors: covered=15, 16, 17, 18, 19 | this run observed 19
  census stated 1230, ledger holds 1230: they agree
ledger coverage: registered=254 | covered=5, not covered=249, ceiling=249
GATE rc=0

new this run=0 and not covered=249 against a ceiling of 249 — it sits exactly on the ceiling, which is the pass. That is the whole fix for this PR's gate story.

[jdatcmd]

Heads up before tomorrow: this now conflicts, on test/pytest/TESTS.md and
nothing else. Not your doing — #1086 and #1087 merged tonight and #1086 added
section 43.

conflicting files    test/pytest/TESTS.md
everything else      auto-merges clean

All five of your PRs claim section 43, so they collide with main and with each
other, not just with the merge that caused it.

test_the_contents_list_is_numbered_in_order requires the contents list AND the
sections to each count 1..N with no gap, so a duplicate or a hole reddens. That
means the number is decided by merge ORDER, and whoever lands second renumbers
again.

Suggested assignment

Taking @OffgridwithJD's recommended order, which exists because #1077's fix falls
through to the cast #1063 repairs:

#1063   section 44    land first
#1039   section 45
#1068   section 46
#1065   renumber when the parallel_leader_participation gap is closed
#1077   renumber when the overread hoist lands, and rebase onto #1063

This one is blocked, so take a number when it is unblocked.

This is #996, and it has now bitten five PRs at once

#996 says every changelog entry inserts at the same anchor so any two conflict.
The same is true of TESTS.md section numbers, and tonight is the clearest
evidence yet: one merge invalidated the number in five independent branches. Worth
linking there rather than treating it as bad luck.

Nothing in your change is affected. The rebase is mechanical and the number is the
only decision in it.

[linuxhikerpm]

CI diagnosis (from job logs, not PR comments)

Red check on the failing SHA (a2e4341): suites (PG 17). Quoted from the PG17 job log:

not in the ledger: parallel_scan_cost	parallel_scan_cost	an I/O-dominated parallel scan is not priced at serial/workers	(on major 17)
not in the ledger: parallel_scan_cost	parallel_scan_cost	premise: both scans have a positive run cost	(on major 17)
not in the ledger: parallel_scan_cost	parallel_scan_cost	premise: the parallel plan has Gather	(on major 17)
not in the ledger: parallel_scan_cost	parallel_scan_cost	premise: the parallel plan is a columnar scan	(on major 17)
not in the ledger: parallel_scan_cost	parallel_scan_cost	premise: the parallel plan uses two workers	(on major 17)
not in the ledger: parallel_scan_cost	parallel_scan_cost	premise: the serial plan has no Gather	(on major 17)
not in the ledger: parallel_scan_cost	parallel_scan_cost	premise: the serial plan is a columnar scan	(on major 17)
not in the ledger: parallel_scan_cost	parallel_scan_cost	premise: the table holds every inserted row	(on major 17)
    8 check(s) the ledger has never seen.
  PG17 has a check the ledger has never seen, which is not a pass
    ledger integrity failure: classification lost rows: matched 8 + orphans 0 + unprunable 0 + not checked 1236 != 1236 ledger rows -- every row must land in exactly one of the four

No suite =FAIL. Integrity failure is the unseen-check double-count.

Class: honest PG18-only ledger seed and needs-rebase vs fcfd3e6. Did not fake majors 15–17.

Pushed: rebased audit/parallel-scan-cost onto fcfd3e6. TESTS.md §43 kept as test_pgxn_metadata.py; this twin is §44. Collection 346 / 411. Ledger never-census 1230 matches budget. suites_not_covered 249. SUITES C-locale sorted. Numbering tests green locally. New HEAD da7317ba1b2ff7ab94ec77f2483bc3ea8bb389d0. Mergeable now; CI pending. Not merged, not approved.

[OffgridwithJD]

Reviewed in full, found legitimate, and prepared for merge as #1094

Your work here is sound and I want to say that before the mechanics: the C is right. pgcolumnar_parallel_divisor reproduces core's get_parallel_divisor faithfully and separating I/O from CPU is the correct fix.

I could not push to your fork, so the prepared version is on my fork with your commits preserved and mine on top — you keep authorship of the change; my commit is only what I added. If you would rather land it from here instead, take the patch from #1094 and push it to this branch, and I will close mine.

What I changed

A test-coverage gap, and it hid the main branch of the code you copied. Both suites set parallel_leader_participation = off, and that GUC defaults ON — so the if (parallel_leader_participation) arm, the entire reason the function is not a one-line division, never executed in either harness.

Both suites now measure the same property again with the leader participating, and prove the branch ran rather than asserting it: rows: leader off=20000, leader on=16667 — exactly 40000/2 against 40000/2.4.

The ledger rows are re-derived from runs on all five majors, not edited. Your rows read 15;16;17;18 while every other row in the file reads 15;16;17;18;19, so covered_majors includes 19 and the release gate refuses this suite's checks — even though CI stays green, because ci.yml:503 runs 17+18 per PR and only the local five-major matrix adds 19. That is #1071's trap, not a mistake of yours: the gate's own printed recipe says <log> singular.

I widened them by running the suite on PG15/16/17/18/19 and merging those logs, because a row is a claim about where a check was observed and editing field 4 makes that claim without the observation.

One thing worth knowing for next time

All five of your PRs take TESTS.md section 44, and only one can. The numbering is gated — test_the_contents_list_is_numbered_in_order requires 1..N with no gap — so the number is decided by merge order and the second to land renumbers. Worse, the census in check_ledger_budget.txt cannot be resolved when two of these merge: the conflict offers two numbers and neither is right, because each is short by exactly the other PR's row count. It has to be re-derived from the merged ledger.

That is a property of the repository, not of your work — measured and written up on #996.

Nothing here needed a change to suites_not_covered. Your budget files were correct as committed.

[jdatcmd]

Thanks for this. The C is correct. pgcolumnar_parallel_divisor reproduces core's get_parallel_divisor faithfully, and separating I/O from CPU is the right fix, because core's seqscan divides CPU and leaves the disk run whole.

The tests never execute the branch the change is about.

The defect

Both harnesses set parallel_leader_participation to off, and nothing sets it
back on:

test/parallel_scan_cost.sh             parallel_leader_participation off
test/pytest/test_parallel_scan_cost.py parallel_leader_participation = off

That GUC defaults ON. With it off the divisor is exactly the worker count,
so this arm of the heuristic you copied never runs:

columnar_customscan.c:2531
    if (parallel_leader_participation)
    {
        double leader_contribution;
        leader_contribution = 1.0 - (0.3 * path->parallel_workers);
        ...

So the new code is covered only in the configuration nobody runs, and the
default configuration is untested. A regression in the leader term would ship
green.

Second defect: the ledger rows name four majors

Every row this PR adds reads 15;16;17;18 while every other row reads
15;16;17;18;19, so the release gate refuses this suite's checks when the
five-major matrix observes them on PG19. CI runs 17 and 18 only.

Disposition

Closing in favour of #1094, which carries your commits unchanged (I
confirmed each is an ancestor of that branch), adds a leader-on arm to both
harnesses, and corrects the ledger. Your authorship is preserved there.

The added arm is a real differential rather than a GUC flip: it asserts
rows_off != rows_on, so it fails if the leader branch is inert.