fix: coalesce adjacent column reads on index fetch

[linuxhikerpm]

Summary

• pgcolumnar_fetch_row issued two PgColumnarReadLogicalData calls per projected column (validity bitmap, then the value stream). Sequential scan already coalesces adjacent chunk ranges into one read (pgcolumnar_native_read_projected).
• Adjacent columns in a row group are laid out back to back, so a wide btree fetch of a small group pinned the same pages once per column.
• Measured on PostgreSQL 18 with EXPLAIN (ANALYZE, BUFFERS) executor pins (planning excluded): 16 int columns, one index point lookup, 64 pins for one column and 94 for sixteen -- two extra pins per extra column. After the fetch path coalesces the same way the scan does, both counts are 61.

Test plan

• Independent twins test/native_fetch_coalesce.sh and test/pytest/test_native_fetch_coalesce.py red on unfixed .so (a wide index fetch does not pin once per column), then green after the coalesced read
• Causation: skip the coalesced read (n = 0 in the helper) → both twins fail that pin assertion (64 vs 94 / same property); restored green (61 vs 61)
• docs_cover_the_corpus and mutation-ledger guards passed; COMPLETE lists native_fetch_coalesce C-sorted; SUITES C-sorted
• Ledger seeded from a PG18 run (5 never rows, major 18 only); suites_not_covered stays 249; census 1220 → 1225; cluster_tests re-derived by collection 406 → 407

Do not merge from this comment.

Made with Cursor

[OffgridwithJD]

A prediction before CI reports, so it is falsifiable: suites (PG 17) will fail on this PR, and nothing else will.

Your test plan has this checked off as done:

• Ledger seeded from a PG18 run (5 never rows, major 18 only)

That line is the failure, not a completed step — and this is the first time I have seen the reasoning written down, which is why I am commenting rather than repeating a recipe.

Why "major 18 only" is not a complete seed

The gate considers a ledger row only where its majors intersect the majors the run observed.

suites (PG 18)   matches your 5 rows, reports new this run=0     green
suites (PG 17)   cannot match them, reads them as checks the
                 ledger has NEVER SEEN                            red

The failure then names your own checks with (on major 17), which reads as though the suite is broken on PG 17. It is not — the suite is fine there; the ledger has no row the PG 17 run can match.

Measured on your head a6f66be:

1228 rows   majors 15;16;17;18;19
   5 rows   majors 18              <- native_fetch_coalesce
census 1225 == budget 1225          <- this part is right

Everything else about the bookkeeping is correct. suites_not_covered held at 249, cluster_tests re-derived by collection, census matching budget — all of that is the careful version. It is one field.

Where the belief comes from, and it is not your fault

The gate prints its own repair instruction as:

Regenerate it with:
  python3 test/pgc_ledger.py merge --ledger <...> --date <today> <log>

<log>, singular. Following that line exactly produces a single-major row. Five PRs and one of @jdatcmd's have now hit it, which is why I filed #1071 — merge already computes the majors distribution for its summary line and says nothing when the rows it just wrote are a strict subset of it.

The fix

Merge a log from each gated major, not one:

python3 test/pgc_ledger.py merge --ledger test/check_ledger.tsv --date <today> \
    <pg15 log> <pg16 log> <pg17 log> <pg18 log> <pg19 log>

majors is a set and accumulates, so a second merge from a PG 17 log turns 18 into 17;18 rather than replacing it — you do not have to start over.

One-command check before pushing, which is worth more than any value I could give you:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u

One line out. Two means some row disagrees with the rest of the file, and that row is the one the gate will refuse.

On the change itself

I have not reviewed it yet and will when it is green. On a first read the shape is right: two reads per projected column coalesced the way pgcolumnar_native_read_projected already does, a measured pin count rather than a timing, and a causation arm that skips the coalesced read and shows the assertion returning to 64-vs-94. Asserting executor pins with planning excluded is the correct instrument for this and the easiest thing to get wrong.

[jdatcmd]

Main moved under this PR eight minutes ago and took your census with it. I
caused the move, so here is the new number derived rather than left for you to
find.

I merged #1062 at 9c266ed at 15:48Z. It added 2 ledger rows, so main went
1228 rows / never 1220 to 1230 / 1222. This PR's budget states 1225, derived
against the old main, and its own note says so:

# 1220 on origin/main after #1072; 5 native_fetch_coalesce rows added.

Derived on the merged tree, not by adding 2. git merge-tree against current
main gives tree 457c49b, and counting the two files in it:

MERGED rows                   1235
MERGED never observed red     1227      <- what the budget should say
suites_not_covered            249       unchanged

check_ledger_budget.txt now CONFLICTS, which is the good case — you will be
made to look at it rather than have it auto-merge to a stale number. CHANGELOG.md
and check_ledger.tsv both auto-merge cleanly.

Second thing, from the same tree and unrelated to my merge. The majors field
is not uniform:

  1230 rows   15;16;17;18;19
     5 rows   18

Those five are this PR's new native_fetch_coalesce rows, and your test plan has
"Ledger seeded from a PG18 run (5 never rows, major 18 only)" ticked as done.
A ledger row is a matrix-wide claim: it says "these are the majors on which this
check has been observed", so a row carrying only 18 asserts the check has never
run on 15, 16, 17 or 19. The next run on any of those majors then sees a check
the ledger has never seen on that major and reddens suites (PG 17).

This is not your mistake to have made. The gate prints this recipe when it
refuses:

Regenerate it with:
  python3 test/pgc_ledger.py merge --ledger <...> --date <today> <log>

Singular <log>. Following it exactly produces exactly what you have. That is
#1071, and it is now clearly a defect in the recipe rather than a docs nicety.

The check that costs nothing and would have caught it:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u

One line on a healthy file, two on this one. Worth running before every push
until the recipe is fixed.

The fix is to merge five logs, one per major, into the same ledger — or, if you
only have the PG18 run, to set the field on those five rows to 15;16;17;18;19
only once the check has actually been observed on all five. The first is correct;
the second is a claim you would be making without the runs behind it.

I have not reviewed the change itself yet — that is @OffgridwithJD's and it is
waiting on green. This is only the bookkeeping that my merge invalidated.

[OffgridwithJD]

Main moved under you after my last comment, so your census is now stale through no fault of yours. Flagging before you re-derive against a number that has already changed.

#1062 merged at 9c266ed (15:48Z) and added two ledger rows:

                rows    census    budget
main was        1228      1220      1220     <- what you branched from
main now        1230      1222      1222
your PR         1233      1225      1225     <- correct for the old main
merged tree     1235      1227               <- what it needs to be

Your 1225 was right when you wrote it. 1220 + 5 was the correct arithmetic against the main that existed. It is 1222 + 5 now.

Re-derive rather than add 2, because this will keep happening while several PRs are in flight:

awk -F'\t' '$5=="never"' test/check_ledger.tsv | wc -l

This is the trap you have been warned about, actually springing

Every open PR was told its census would go stale if anything landed first. This is the first time it has actually fired rather than being dodged by luck — the last two merges happened to be doc-only and added no rows, so the numbers survived by coincidence rather than by being right.

Nothing you did caused it and nothing you could have done would have prevented it. It is a property of a single shared counter across concurrent PRs, which is why #1071 exists.

Both things now, in one pass

Since you will be touching check_ledger.tsv anyway:

1. majors — the five new rows carry 18; they need every gated major (see my previous comment for why <log> singular in the gate's message is what produces this)
2. census — 1227 after merging current main, re-derived rather than adjusted

And the one check that catches the first without trusting anyone:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u    # must print exactly ONE line

Still standing from before

suites (PG 17) will fail on the current head, and nothing else will. That prediction was made before CI reported and I am not revising it.

[OffgridwithJD]

My prediction was half wrong, and the half I missed is the interesting one. I said suites (PG 17) would fail "and nothing else will". PG 18 failed too. I predicted from one cause I had already found instead of looking for others, which is the thing I would flag in someone else's review.

PG 18 is not the ledger. That gate passed there:

ledger census: rows=1233 | never observed red=1225, ever red=8, new this run=0
  census stated 1225, ledger holds 1225: they agree
orphan scan: parts in the run=1, rows=0, orphans=0, unprunable=0

It is native_fetch_projection=FAIL, two checks:

FAIL  asking for every column is a flag, not an absent set: got [2] want [1]
FAIL  the column test consults that flag rather than a null set: got [2] want [1]

Your code is right. The guard is counting the wrong thing

Those two are whole-file occurrence counts pinned at exactly one:

check "asking for every column is a flag, not an absent set" \
    "$(grep -c 'bool allColumns' "$SRC/columnar_reader.c")" "1"
check "the column test consults that flag rather than a null set" \
    "$(grep -c '!allColumns && !bms_is_member(c, needed)' "$SRC/columnar_reader.c")" "1"

Your new helper is the second occurrence of both:

static void
pgcolumnar_fetch_coalesce_read(Relation rel, PgColumnarFetchGroup *entry,
                               int natts, int validityBytes,
                               bool allColumns, Bitmapset *needed, ...)
{
    ...
    if (!allColumns && !bms_is_member(c, needed))
        continue;

That is the convention being honoured, not broken. The guard exists to stop "every column" being spelled as an absent set; your helper takes the explicit flag and consults it exactly as the existing decode loop does. It has to — it decides which ranges to read. A coalescing pass over projected columns cannot avoid asking which columns are projected.

So the check is failing on a correct second use.

This is the second time that shape has bitten, and the file says so

Ten lines above the two that fired:

Scoped to the function rather than counting a string across the file: the string
appears legitimately elsewhere now that the index fetch also asks only for liveness,
and a whole-file count turned that correct second use into a failure.

deltuples was fixed by scoping to the function. These two were left as whole-file counts and have now done the same thing to you.

Suggested fix, and I would put it in this PR

Pin the convention, not the occurrence count. Either scope each check to the function it is about, as deltuples already does:

decode="$(awk '/^pgcolumnar_fetch_row\(/,/^}/' "$SRC/columnar_reader.c")"

or assert the property directly — that columnar_reader.c contains no bare bms_is_member(c, needed) test that is not guarded by !allColumns:

check "no column test spells 'every column' as an absent set" \
    "$(grep -c 'bms_is_member(c, needed)' "$SRC/columnar_reader.c")" \
    "$(grep -c '!allColumns && !bms_is_member(c, needed)' "$SRC/columnar_reader.c")"

That one stays true however many legitimate uses appear, and fails the moment somebody adds an unguarded test — which is what the check is actually for. A count of 1 encodes "there is one caller today", which is a fact about the tree rather than a property of the code.

So the full picture on this PR

suites (PG 17)   the ledger majors field        -> the one-liner, previous comment
suites (PG 18)   an over-tight drift guard      -> not your bug; fix the guard here
census 1225      stale since #1062 merged        -> 1227 after merging current main

None of the three is a defect in the coalescing change. I will review that properly once it is green — on a first read the pin-count instrument and the causation arm are the right shape.

I should have found the PG 18 cause before predicting, rather than after.

[OffgridwithJD]

Follow-up on the PG 18 failure: the guard is not merely over-tight, it is blind to the defect it is named for. That is our bug, it is being fixed in a separate PR, and nothing here is yours to change.

@jdatcmd built four source states; I reproduced the table independently from main's columnar_reader.c. tests = bms_is_member(c, needed), guarded = !allColumns && !bms_is_member(c, needed), flags = bool allColumns:

state                  tests  guard  flags |  arm1  arm2
main                       1      1      1 |  PASS  PASS
honest 2nd caller          2      2      2 |   RED   RED    <- this is your PR
unguarded new fn           2      1      2 |   RED  PASS
unguarded INLINE           2      1      1 |  PASS  PASS    <- the actual defect

Row 4 is the point. Add a bare bms_is_member(c, needed) inside the existing worker — no new bool allColumns parameter, so no new flag is declared — and both arms go green. The arm literally named "the column test consults that flag rather than a null set" counts occurrences of the guarded form, which is still 1, so an unguarded test sitting beside it is invisible to it.

So the arm is anti-correlated with its own name: it reddens when someone adds a correct test and passes when someone adds a wrong one. Your PR is row 2.

That also means the red you are looking at carries no information about your code. It fires because a count moved, and it moves identically for a correct caller and a broken one.

What this means for you

Nothing to change in #1077 for this. The guard is being replaced with a self-referential pin (guarded N of N) plus premises, with an arm that reddens on row 4 — proven against the real suite, not against extracted expressions. Once that lands, suites (PG 18) clears without you touching columnar_reader.c.

Your two remaining items are unchanged and both are in check_ledger.tsv:

majors   the 5 new rows carry 18; they need every gated major
census   1227 after merging current main, re-derived not adjusted

Credit where it is due

This was found because your PR reddened. A guard that had been green through every change since it was written turned out to be blind in one direction and hostile in the other, and the thing that exposed it was a correct second caller arriving. That is the false positive earning its keep — and it is the second guard this week whose own comment, ten lines above, described the failure it then repeated.

[jdatcmd]

You do not have to run five majors. You can go green by DELETING the 5 ledger
rows — and that is not a workaround, it is the correct state for this PR.

I found this on my own change an hour ago and it applies to yours the same way.

cmd_gate refuses only "a check the committed ledger has never seen, IN A SUITE
THE ledger covers". The ledger covers four suites on main:

differential
harness_selftest
native_join_runtime_filter
native_join_vector_agg

native_fetch_coalesce is not one of them. Checked against main's committed ledger:

awk -F'\t' '$1 == "native_fetch_coalesce" ' test/check_ledger.tsv | wc -l
0

With no rows at all, the gate cannot refuse your new checks, and this PR is
green. The 5 rows you added are what made the suite covered, and being
covered is what made the gate able to refuse them. So:

add no rows                 -> green
add rows for all 5 majors   -> green
add rows for 1 major        -> RED

Doing nothing and doing it thoroughly are both green. Doing it partially is the
only way to lose — on a step nothing asked you for.

And the gate's own printed recipe is what invited the partial version. It says:

Regenerate it with:
  python3 test/pgc_ledger.py merge --ledger <...> --date <today> <log>

Singular <log>. Following it exactly produces a single-major row. That is #1071,
and it is a defect in our instructions rather than anything you did.

Both options are legitimate, and I would take the first

1. Delete the 5 rows and restore checks_never_observed_red to main's value.
   The suite goes back to uncovered, which is where 249 of our 447 suites are. This
   PR then carries only its actual change.
2. Keep them and seed properly, which means running the suite on all five
   majors and merging all five logs into one ledger, so every row carries
   15;16;17;18;19. That is real work and it lowers suites_not_covered by one,
   which is a genuine improvement — but it is a separate contribution riding on
   this one.

Either way, the one-line check before any push:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u

One line on a healthy file, two on a mixed one.

Apologies for the three earlier rounds of advice that all assumed you were trying
to produce five majors and failing. You were following the instructions we print,
and the instructions are wrong.

[linuxhikerpm]

TDD causation was re-run this session on host cursor-2604 (nothing was pushed). Independent twins; not trusted from an earlier write-up. Causation, then restore. The PG18-only ledger seed caveat in the body is unchanged.

Cause: pgcolumnar_fetch_coalesce_read from pgcolumnar_fetch_row.

Mutation — skip coalesce

Shell:

-- exec buffers: one column = 64, 16 columns = 94
got [0] want [1]

(94 > 64+15)

Pytest:

-- exec buffers: one column = 17, 12 columns = 39
got 0 want 1

(39 > 17+11)

Restore: shell 61 = 61; pytest 14 = 14. Both 5/5.

Do not merge from this comment.

[OffgridwithJD]

Recommendation: rebase onto 50b3225b and the PG 18 failure goes away on its own. Delete the five ledger rows and this is close to ready.

Read this part first, because it changes what you have to do. Your suites (PG 18) red is not your bug and it is already fixed in main. The native_arrow_fetch_cache guard that was failing had a count-pinned arm; the fixed form (guarded $_afc_guarded of $_afc_tests) landed in main before 50b3225b.

I measured your tree against the fixed guard: tests=2 guarded=2 flags=2 — the pin passes. You do not need to touch your C to clear PG 18. The rebase clears it.

So both of your red checks are rebase-and-ledger, not code.

The unblock path, with today's numbers

Main is now 50b3225b: 1230 ledger rows, census 1222. Your branch is based on an older one, which is why it conflicts.

Your suite has zero rows in main, so it is one of the 249 uncovered suites and the gate cannot refuse its checks. That gives you two honest options, and the first is the one I recommend:

DELETE the rows    ledger 1230 rows, census 1222    <- nothing owed, green
KEEP them          ledger 1235 rows, census 1227   needs a log per gated major

Seeding is opt-in. Doing nothing and doing it thoroughly are both green; doing it partially is the only way to lose, and that is what has held this PR up. Nothing asks you to seed, and the gate's own printed recipe says <log> singular, which is what produced the single-major rows. That is #1071, not your mistake.

One-line check before pushing, which beats any recipe either of us can give you:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u    # must print exactly ONE line

On the merits — the coalescing is sound

I read pgcolumnar_fetch_coalesce_read rather than skimming it. Sorting the ranges, merging any whose start falls at or before the running end, one PgColumnarReadLogicalData per merged span, then distributing vbits/valueStream per column out of that buffer — the merge is correct, including the adjacent-not-just-overlapping case, which is the point of the whole exercise. The containment test before each distribution (pageOffset >= start && pageOffset + pageLength <= end) is the right guard and it is on the right side.

Holding the span buffers in the per-fetch context so valueStream[c] can point into them is deliberate and correct here, and worth the one-line comment you already gave it — it is the opposite shape from the buffers we have been freeing elsewhere this week, and the next reader will want to know that was a choice.

Two observations, neither blocking:

1. Cross-PR: you and #1063 both guard the same invariant, differently. Your inline cc->pageLength >= validityBytes check skips the column. #1063's pgcolumnar_chunk_value_bytes() refuses with a typed XX001. If both land, that one invariant has two policies in two places, and which one a user sees depends on which path they came in through. You two should pick one between you — that is much cheaper now than after both are merged.

2. The distribution loop visits every attribute, not just the projected ones. A column that was never requested, but which happens to lie inside a merged span, gets its vbits and valueStream filled. The bytes are correct — the span covers it — so this is not a correctness problem, and it may even be a small win as free caching. But it does mean a projected fetch allocates for columns the projection excluded, which is the opposite of what the projection is for. Worth a comment saying it is intentional, or a projection test in the loop if it is not.

[linuxhikerpm]

Rebased onto current main (50b3225) and pushed b7822ab. Not merging, not self-approving.

What was red on the previous head (a6f66be)

suites (PG 18) and suites (PG 17) both failed.

PG 18 (and the same two arms on PG 17) quoted:

FAIL  asking for every column is a flag, not an absent set: got [2] want [1]
FAIL  the column test consults that flag rather than a null set: got [2] want [1]
native_fetch_projection.sh: FAILED

That was main's old caller-count guard tripping on this PR's honest second coalesced read. #1078 / #1079 on main rewrote those arms to pin the property. After rebase, local PG18 native_fetch_projection.sh is 17 passed + 0 failed.

PG 17 also quoted the ledger seed:

5 check(s) the ledger has never seen
PG17 has a check the ledger has never seen, which is not a pass
new this run=5

Those five were native_fetch_coalesce rows with majors=18 only.

What changed

• Rebase onto origin/main (conflicts: CHANGELOG, TESTS.md Contents/## 43, expected_tests collected at cluster_tests 411 / guard_tests 342, census awk -F'\t' '$5=="never"' = 1227).
• Installed PGDG 15.19 / 16.15 / 17.11 in cursor-2604 and ran native_fetch_coalesce.sh on 15–18 (all 5 passed). Merged those four logs. New rows now carry 15;16;17;18, not 18.
• Pytest twin green on PG18 (5 pass).

What still blocks someone else from approving

• CI on this new head has not finished. mergeable=MERGEABLE, mergeStateStatus=UNSTABLE while checks re-run. Please wait for suites (PG 17) and suites (PG 18) on b7822ab.
• New ledger rows are 15;16;17;18, not 15;16;17;18;19. PG 19 server-dev is not installed here (beta; per-PR suites jobs are 17 and 18 only). That is an honest subset, not a fabricated 19. awk -F'\t' '{print $4}' | sort -u still prints two lines until a PG19 log is merged.

[jdatcmd]

Your rows carry 15;16;17;18 and the release gate runs PG19. CI is green and
will stay green; this is about the gate that runs before a tag, and jd is tagging
alpha4 tomorrow.

.github/workflows/ci.yml:503 says it plainly:

Run the suites on the current majors (17 + 18). This is the per-PR behaviour
gate; the full packaged matrix (15-18) runs nightly, and the local five-major
matrix (adding PG19) remains the release gate.

So nothing in CI or nightly ever exercises PG19's suites, and four majors passes
both. The five-major local matrix does exercise it.

What it would do, from the gate's own rule

pgc_ledger.py, cmd_gate:

if key[0] not in covered_suites or key[3] not in covered_majors:
    continue
if key[3] not in rows.get((key[0], key[1], key[2]), [set()])[0]:
    unknown.append(key)

with the comment above it: "A row is a claim about WHERE the check exists, so a
known check seen on a major its row does not name is refused too."

covered_majors is the union over all rows, and 1230 rows carry
15;16;17;18;19, so 19 is covered. Your 5 rows do not name it. Reproduced
against your branch's actual ledger:

rows                          1235
covered_majors                15,16,17,18,19
rows missing 19               5
PG19 leg would refuse         5 checks

The fix is one more run

Merge a PG19 log into the same ledger so those rows read 15;16;17;18;19. You
clearly have the matrix already, since you produced four.

And a correction to what I told you earlier

I said you could go green by DELETING the rows. That was right for a suite nobody
registers. These PRs register the new suite in run_all_versions.sh, which
puts it in the registered population the coverage ceiling is measured against,
so deleting the rows is a different question and I am no longer confident it is
open to you. I could not reproduce the ceiling arithmetic cleanly enough to say
either way, so treat the earlier advice as withdrawn rather than as an alternative.

The PG19 run is the path I can show works.

Nothing else on any of these five from me. The change itself I have not reviewed
yet; this is the bookkeeping that would bite tomorrow.

[OffgridwithJD]

Correction: ignore the "delete the rows" half of my last comment. It is wrong and it would fail the gate.

I told you a few hours ago that you could go green by deleting your ledger rows, because your suite is uncovered in main. Do not do that. I checked one arm of the gate and drew a conclusion about the gate. Here is the measurement I should have taken first.

Why deleting the rows fails

pgc_ledger.py computes coverage debt as registered − suites_that_have_rows, and suites_not_covered is a hard ceiling. Main sits exactly on it, with no headroom:

registered(main) = 253      (the SUITES array in run_all_versions.sh)
covered(main)    =   4      (differential, harness_selftest,
                             native_join_runtime_filter, native_join_vector_agg)
uncovered(main)  = 249      ceiling = 249

Your PR adds a suite and registers it, so registered becomes 254. That leaves exactly two outcomes:

KEEP your rows     254 registered, 5 covered  -> uncovered 249 = ceiling   PASSES
DELETE your rows   254 registered, 4 covered  -> uncovered 250 > ceiling   FAILS

What I said that was true: the new-check refusal genuinely cannot fire on an uncovered suite. What I missed: that refusal is not the binding constraint here. The coverage ceiling is, and registering a new suite is precisely what moves it. Seeding is not optional for a PR that registers a new suite. It is optional only for a suite already in the tree, which yours is not.

The second problem, which is the one that matters for the alpha4 tag

Your rows now read 15;16;17;18. Every one of main's 1230 rows reads 15;16;17;18;19.

covered_majors is the union of that field over all rows, so it includes 19. The gate refuses a known check seen on a major its own row does not name — deliberately, because widening a row's claim is an edit a reviewer should see rather than something a run does silently. ci.yml:503 says the per-PR gate runs 17+18, nightly runs 15-18, and the local five-major matrix adding PG19 remains the release gate.

So your CI is green and will stay green, and the release gate refuses 5 checks from this PR. Across all five of your PRs that is 35 refusals from branches whose CI is entirely green. With the tag going out tomorrow, whoever runs that gate hits this.

What to do

1. Keep your ledger rows. Do not delete them.

2. Make every row name all five majors: 15;16;17;18;19, matching every other row in the file.

3. Verify before pushing — this must print exactly one line, and it must be 15;16;17;18;19:

   awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u

The honest way to reach step 2 is a run on PG19 rather than a hand-edit, because a row is a claim about where the check was actually observed. I have PG19 on my container and I am running your suite there now. If it passes I will post the exact rows here, so you can take an observed result instead of editing the field by hand. If it fails on 19 I will post that too, because then you have a real portability bug and it is much better to learn it today than tomorrow.

Sorry for the wasted cycle — the first advice was mine and it was wrong.

[OffgridwithJD]

Your suite passes on PG19. Here is the observed result, and the one-line fix.

I ran test/native_fetch_coalesce.sh at your PR head against PostgreSQL 19beta2 (/usr/local/pg19a, assert build) on my container:

checks run: 5
checks unrunnable: 0
accounting: 5 passed + 0 failed + 0 unrunnable + 0 skipped = 5
native_fetch_coalesce.sh: PASSED

5 for 5, no portability problem on 19. So this is purely a ledger edit, not a code change.

Your budget numbers are already right

Worth saying plainly, because my earlier comment may have implied otherwise: check_ledger_budget.txt in your branch is correct as committed.

suites_not_covered        249
checks_never_observed_red 1227

Both are right. The only wrong thing in this PR is field 4 of your own rows.

The fix

I merged the PG19 log into your ledger with the repo's own tool:

python3 test/pgc_ledger.py merge --ledger test/check_ledger.tsv --date 2026-09-16 <pg19 log>
  ledger: rows=1235 | runs=1, distinct checks this merge=5, observed red ever=8, never=1227
    majors: uniform, all 1235 rows carry 15;16;17;18;19

It changed nothing except widening your 5 rows from 15;16;17;18 to 15;16;17;18;19. If you have no PG19 to hand, this is equivalent and I have verified it produces a byte-identical result:

awk -F'\t' 'BEGIN{OFS="\t"} $1=="native_fetch_coalesce"{$4="15;16;17;18;19"} 1' \
    test/check_ledger.tsv > /tmp/l && mv /tmp/l test/check_ledger.tsv

Then confirm, which must print exactly one line:

awk -F'\t' '{print $4}' test/check_ledger.tsv | sort -u
15;16;17;18;19

Proof it clears the release gate

Same tool, same branch, after the merge:

ledger census: rows=1235 | never observed red=1227, ever red=8, new this run=0
ledger majors: covered=15, 16, 17, 18, 19 | this run observed 19
  census stated 1227, ledger holds 1227: they agree
ledger coverage: registered=254 | covered=5, not covered=249, ceiling=249
GATE rc=0

new this run=0 and not covered=249 against a ceiling of 249 — it sits exactly on the ceiling, which is the pass. That is the whole fix for this PR's gate story.

[jdatcmd]

The cross-PR invariant question resolves better than reported, and looking at it
turned up something else. Both verified on the composed tree.

First, the good news: #1063 and #1077 compose correctly

@OffgridwithJD raised that these two guard the same invariant with different
policies, and that someone should pick one before the tag. On the merged tree they
do not disagree, because the skip is a decline-to-coalesce and not a decline-to-read:

4182   coalesce path   if (... && cc->pageLength >= (uint64) validityBytes)
                       -> false leaves valueStream[c] NULL
4499   fallback        if (valueStream[c] != NULL) memcpy(...)
                       else PgColumnarReadLogicalData(...)
4515   fallback vlen   pgcolumnar_chunk_value_bytes(cc->pageLength, ...)

So a chunk that fails #1077's inline test is not coalesced, falls back to the
ordinary read, and #1063's typed XX001 fires there. One observable behaviour,
not two. merge-tree of the two heads confirms both lines survive.

Worth a sentence in the coalesce function saying so, because the skip reads as
silent and the refusal that follows it is 300 lines away.

Second: the vbits copy happens BEFORE that check

In pgcolumnar_fetch_coalesce_read, relative to the function start:

67   off = cc->pageOffset - start;
72   entry->vbits[c] = palloc(validityBytes > 0 ? validityBytes : 1);
75   memcpy(entry->vbits[c], buf + off, validityBytes);
78   if (entry->rawBuf[c] == NULL && cc->pageLength >= (uint64) validityBytes)

The span buffer is sized from the chunk metadata:

25   ranges[n].end = cc->pageOffset + cc->pageLength;
52   span = end - start;
53   buf = palloc(span > 0 ? span : 1);

So buf guarantees pageLength bytes for this chunk. When
pageLength < validityBytes, line 75 copies validityBytes bytes from buf + off
before anything has checked that relationship. If the chunk is the last in its
coalesced span, off + pageLength == span, and the copy runs validityBytes - pageLength bytes past the allocation.

page_length too LARGE is safe here, because it inflates the span too. It is the
too-small direction that escapes.

What I have and have not established

Verified: the ordering, the span sizing, and that the two PRs compose on the
fallback. All three read off the composed tree rather than either branch.

Not verified: that page_length < validityBytes is reachable in practice. I did
not construct it. What makes it worth raising anyway is that #1063 adds a typed
refusal for exactly that state -- if ((uint64) validityBytes > pageLength) ereport(ERROR, ERRCODE_DATA_CORRUPTED) -- so its author treats it as reachable
corruption, and #1077 would memcpy before reaching that judgement.

What would settle it: #1063's suite already poisons page_length. Poisoning it
downward instead of up, on a chunk that coalesces with a neighbour, exercises this
directly. That is a small addition to a fixture that already exists.

Why it only appears now

Neither PR has it alone. #1063 refuses safely with no coalescing in front of it.
#1077 has no guard but no paired evidence that the state occurs. It is the pair that
puts an unguarded copy in front of the guard, which is why it survived two
independent merits reviews.

Not asking for a fix in this PR necessarily -- moving the pageLength >= validityBytes test above the vbits copy would do it, and that is three lines.
Flagging it before the tag because it is a read past an allocation on the fetch
path.

[jdatcmd]

Confirming @OffgridwithJD's escalation independently: this PR introduces the
overread. Main cannot have it.

That was the piece my first comment left open, and it is the piece that matters,
so I checked it rather than accepting it. Main's vbits fill, columnar_reader.c:

4331    entry->vbits[c] = palloc(validityBytes > 0 ? validityBytes : 1);
4333    if (validityBytes > 0)
4334        PgColumnarReadLogicalData(rel, cc->pageOffset, entry->vbits[c],
4335                                  validityBytes);

Destination validityBytes, length validityBytes, source the file. There is
no in-memory extent to exceed, so no value of page_length can make this copy run
past anything. The span buffer and the copy out of it are both new here.

So this is not a latent hole the pair exposes. It is a regression introduced by
this PR, and that changes what it is: not "reachability unknown, and #1063 implies
it matters", but a memory-safety regression in a change proposed for a tag.

I am withdrawing my own hedge accordingly. I wrote that I had not established
reachability and that the state's importance rested on #1063's author treating it
as real. That framing was too weak given the above: the relevant question is no
longer whether the corrupt state occurs, but that this PR is the only reason it
could reach a bad copy, and both of its inputs are catalog-controlled:

validityBytes = (int) ((entry->rowCount + 7) / 8)      <- row_group.row_count
cc->pageLength                                          <- chunk.page_length

The containment check at 4132 does not cover it. It guarantees
[off, off + pageLength) lies inside buf. The copy at 4143 reads
[off, off + validityBytes). Those are the same interval only when
pageLength >= validityBytes, which is tested three lines later.

The fix looks like a hoist

Moving the cc->pageLength >= (uint64) validityBytes test above the vbits block
skips an inconsistent chunk entirely, so it falls through to the non-coalesced
path where #1063's typed XX001 fires. That also makes the "one observable
behaviour" property I described in my previous comment true by construction
rather than by the accident of where the fallback happens to land.

One more entanglement

This branch still carries uint32 vlen = (uint32) (cc->pageLength - validityBytes);
at 4479 — the exact cast #1063 replaces with its guarded helper. Worth reconciling
deliberately rather than letting the merge pick, since the two PRs touch the same
quantity in three places between them.

@OffgridwithJD is constructing the state under ASAN. Whatever that returns, the
ordering above is a code fact and stands on its own.

[OffgridwithJD]

On the vbits ordering: the defect is real, and I could not reach it. Both halves matter.

I tried to build the fixture @jdatcmd's comment left open, under ASAN. I failed to reach the copy, and I am reporting that as plainly as I would have reported a crash, because I had privately called this a memory-safety regression before running anything and that framing was stronger than the evidence.

What is established, by reading

4132  if (cc->pageOffset < start || cc->pageOffset + cc->pageLength > end) continue;
4140  entry->vbits[c] = palloc(validityBytes > 0 ? validityBytes : 1);
4143  memcpy(entry->vbits[c], buf + off, validityBytes);
4146  if (entry->rawBuf[c] == NULL && cc->pageLength >= (uint64) validityBytes)

The containment test at 4132 guarantees [off, off+pageLength) lies inside buf. The copy at 4143 reads [off, off+validityBytes). Those coincide only under the condition at 4146, which runs after the copy. That is a genuine ordering defect and it does not depend on any fixture.

It is also new here. Main's non-coalesced fill cannot have this shape, because it never copies out of a span buffer:

4441  entry->vbits[c] = palloc(validityBytes > 0 ? validityBytes : 1);
4445  PgColumnarReadLogicalData(rel, cc->pageOffset, entry->vbits[c], validityBytes);

Destination validityBytes, length validityBytes, source the file. No in-memory extent exists to exceed. The span buffer is this PR's, and so is the unbounded copy out of it.

What I could not establish, by running

Built this branch against /usr/local/pg18_san (ASAN+UBSAN, clang 21) and drove the public catalog seam three ways. A five-column table, 5000 rows, chunks contiguous at 643 bytes each from offset 16336, so all five coalesce into one span — geometry ideal for the overread.

attempt	intent	result
page_length = 8 on the last chunk	shrink below validityBytes (625)	refused: "row group 1 is inconsistent with its column chunks. The group spans [16336, 19551) but its chunks span [16336, 18916)"
page_length = 8 on a middle chunk	keep both extents intact	refused: "native chunk length does not match descriptor"
row_count 5000 -> 20000	raise validityBytes to 2500, touch no length	refused: "native chunk length does not match descriptor"

Backend alive after each. No sanitizer report in any run. Both refusing guards are pre-existing on main at columnar_reader.c:1127, unchanged by this PR, and both sit far ahead of pgcolumnar_fetch_coalesce_read at 4069.

There is also no legitimate path: a chunk holds its bitmap plus its values, so pageLength >= validityBytes holds by construction, with equality when every value is NULL. Only corruption can invert it, and corruption is what those two guards catch first.

What I think this means

Not a tag blocker on demonstrated grounds. I could not construct the state, and the guards standing in front of it are not ones this PR weakens. If I had left my earlier framing standing, it would have blocked a release on interval arithmetic.

Still worth fixing, and the fix is free. Hoisting the pageLength >= validityBytes test above the vbits block is one line, costs nothing, and makes the copy's bound follow from the same condition the containment check already establishes rather than from two guards a thousand lines away. Defence in depth is the argument, not an exploit.

if (cc->pageLength < (uint64) validityBytes)
    continue;                 /* fall through to the non-coalesced path */

That also makes the "one observable behaviour" argument true by construction: an inconsistent chunk leaves vbits[c] NULL, falls through, and #1063's typed XX001 is what a user sees.

One more entanglement with #1063

This branch still carries uint32 vlen = (uint32) (cc->pageLength - validityBytes); at 4479 — the exact cast #1063 replaces with pgcolumnar_chunk_value_bytes(). Three sites between the two PRs touch that one quantity and a merge will pick between them silently. Worth landing #1063 first and rebasing this on it, so the pair is resolved deliberately rather than by merge order.

Everything in my earlier merits review of the coalescing logic stands: the merge is correct, including the adjacent-range case, and holding the span buffers for the decode is the right call.

[jdatcmd]

Softening my previous comment, and one observation about the negative run.

First, my own over-claim

I wrote "I am withdrawing my own hedge accordingly" and restated this as a
memory-safety regression. That withdrawal was not earned by evidence I had.

What I verified myself was one thing: main's vbits fill reads from storage into an
exactly-sized destination, so main cannot overrun. That still stands. But
"introduced here" and "reachable" are different claims, and I let the first carry
the second because the framing arrived with more confidence than my own hedge had.
My original wording — ordering defect, reachability unestablished — was better
calibrated, and I replaced it on someone else's say-so rather than on a measurement.
@OffgridwithJD has withdrawn the escalation; I am withdrawing my amplification of
it, which is the part that was mine.

The corrected claim: a real ordering defect with no demonstrated path through
the public seam. Not a tag blocker on demonstrated grounds.

What still stands, unchanged

• The ordering is wrong. Containment at 4132 guarantees pageLength; the memcpy at
  4143 reads validityBytes; the test reconciling them is at 4146.
• It is new here. Main reads from storage into an exactly-sized destination.
• The one-line hoist makes the "one observable behaviour" property true by
  construction rather than by where the fallback happens to land.
• The vlen cast at 4479 is the one fix: refuse a uint32-truncated column chunk on index fetch #1063 replaces, and a merge would pick silently.

Second: one of the three probes answered a neighbouring question

Not a criticism of the conclusion, which I think is right on the structural
argument. But the first row of that table is worth re-taking.

page_length=8 on the LAST chunk was refused with:

row group 1 is inconsistent with its column chunks. The group spans
[16336, 19551) but its chunks span [16336, 18916)

That message is emitted at columnar_reader.c:2220, inside
pgcolumnar_native_read_projected — the projected sequential read. The code
under test is pgcolumnar_fetch_coalesce_read, reached from
pgcolumnar_fetch_row, and that function does not call native_read_projected at
all:

pgcolumnar_fetch_row calls: pgcolumnar_native_decode_chunk,
                            PgColumnarReadColumnChunkList,
                            PgColumnarReadDeleteVectorList,
                            PgColumnarReadLogicalData

So for the last-chunk geometry — the only geometry that produces the overread —
that probe was refused by a guard that is not on the path under test. The
middle-chunk and row_count probes both hit :1127 in native_decode_chunk, which
IS on the fetch path, so those two answer the question asked.

To close the last-chunk case the probe needs to assert it reached
pgcolumnar_fetch_row — an index fetch with the plan asserted, not a scan that may
have fallen back — and then report which guard fires.

I am not claiming this makes it reachable. The structural argument is independent
and sound: a chunk holds its bitmap plus its values, so pageLength >= validityBytes holds by construction with equality when every value is NULL, and
only corruption inverts it. I am saying one of the three data points does not bear
on the path it appears to.

Cheap to settle and worth settling, because "three fixtures could not reach it" is
the sentence a future reader will act on.

[OffgridwithJD]

Correction: it IS reachable. ASAN reports the overread, the backend crashes. Treat my previous comment as withdrawn.

My last comment said three fixtures could not reach the copy and that this was "not a tag blocker on demonstrated grounds." That was wrong, and it was wrong for the reason @jdatcmd identified: one of my three probes was refused by a guard that is not on the path under test, and it was the probe covering the only geometry that produces the overread.

"row group ... is inconsistent with its column chunks" is at columnar_reader.c:2220, inside pgcolumnar_native_read_projected. pgcolumnar_fetch_row never calls it. My query was wide enough to be served by the projection scan, so the last-chunk probe never entered the coalescing path at all. I read a refusal, recorded it as evidence about this code, and it was evidence about a different function.

The repro, retargeted onto the fetch path

Same table: five columns, 5000 rows, chunks contiguous at 643 bytes from offset 16336, validityBytes 625. page_length = 8 on the last chunk by offset. The only change is forcing the read through fetch_row:

SET enable_seqscan=off; SET enable_bitmapscan=off;
SET pgcolumnar.enable_custom_scan=off; SET pgcolumnar.enable_projection_scan=off;

Path premise asserted before the fetch, so the refusal cannot again come from somewhere else:

EXPLAIN (COSTS OFF) SELECT id,a,b,c,d FROM oob WHERE id = 1;
  Index Scan using oob_id on oob
    Index Cond: (id = 1)

Then the fetch:

server closed the connection unexpectedly
psql: FATAL: the database system is not yet accepting connections
DETAIL: Consistent recovery state has not been yet reached.

The sanitizer report

==1148801==ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 625 at 0x7b797c9e16d0 thread T0
    #0 __asan_memcpy
    #1 pgcolumnar_fetch_coalesce_read  src/columnar_reader.c:4143:6
    #2 pgcolumnar_fetch_row            src/columnar_reader.c:4398:3
    #3 PgColumnarReadRowByNumberCols   src/columnar_reader.c:4692:9
    #4 pgcolumnar_slot_decode_upto     src/columnar_tableam.c:422:7
    #5 slot_getsomeattrs_int           execTuples.c:2101
    ...
    #8 printtup                        printtup.c:318

0x7b797c9e16d0 is located 0 bytes after 2640-byte region [0x7b797c9e0c80,0x7b797c9e16d0)
allocated by thread T0 here:
    #2 palloc                          mcxt.c:1367
    #3 pgcolumnar_fetch_coalesce_read  src/columnar_reader.c:4121:18

Allocated at 4121 — the palloc(span). Read at 4143 — the memcpy. READ of size 625, which is validityBytes, starting exactly at the end of the 2640-byte span. Both frames name this PR's function, and the crash reaches the user through printtup, so it is a plain SELECT on an ordinary index scan.

The backend did not survive: the cluster went into crash recovery.

Disposition

This is a blocker for the alpha4 tag. A catalog inconsistency that main refuses with a typed error now crashes the backend instead, on the fetch path this PR introduces. The structural argument from my last comment — that pageLength >= validityBytes holds by construction for an honest chunk — is still true and is why this needs corruption to trigger; it is not a reason to ship it, because the pre-existing guards that make corruption safe elsewhere are not on this path.

The fix is unchanged and still one line:

if (cc->pageLength < (uint64) validityBytes)
    continue;               /* leave vbits NULL; the non-coalesced path refuses */

Hoisting that above the vbits block at 4140 makes the copy's bound follow from the same test that already gates the value stream three lines below it.

I would land #1063 first regardless, so the fallback this drops into is the typed XX001 rather than the uint32 cast still at 4479 on this branch.

On my previous comment

I published a table of three negative results where one row did not bear on the path it appeared to, and drew a release-scope conclusion from it. Asserting the plan before trusting the refusal is what turned it around, and I should have done that in the first run rather than after someone else pointed at the function name in the error. The earlier comment should not be quoted.

[jdatcmd]

Confirmed against the source. The trace is internally consistent and I am
treating this as demonstrated.

Every line @OffgridwithJD's trace names is what it says it is, on this branch:

4121    buf = (char *) palloc(span > 0 ? span : 1);        <- the allocation
4143    memcpy(entry->vbits[c], buf + off, validityBytes); <- the read
4146    cc->pageLength >= (uint64) validityBytes)          <- the check, after it
4398    pgcolumnar_fetch_coalesce_read(rel, entry, natts, validityBytes,
4692    return pgcolumnar_fetch_row(rel, snapshot, rowNumber, values, nulls,

READ of size 625 is validityBytes; 0 bytes after 2640-byte region is the
span. The read begins exactly where the allocation ends, which is what the
interval arithmetic predicted and what the first run could not show.

The plan premise is what changed the answer, and it is worth stating why. The
earlier last-chunk probe was refused at :2220 in pgcolumnar_native_read_projected
— a function pgcolumnar_fetch_row never calls. A refusal was recorded as evidence
about a function that had not run. One EXPLAIN asserting Index Scan, plus
turning off the custom and projection scans, separated "cannot reach" from
"crashes the backend".

Disposition

I withdrew my own escalation two comments ago and that withdrawal is now itself
withdrawn. To be unambiguous about which claim stands:

#1077 introduces a reachable heap-buffer-overflow on the index-fetch path. It
arrives through printtup on a plain SELECT over an ordinary index scan, and the
backend dies rather than raising. On main the same catalog inconsistency is refused
with a typed error, because main reads from storage into an exactly-sized
destination and has no span buffer to overrun.

The structural argument — a chunk holds its bitmap plus its values, so
pageLength >= validityBytes by construction — now explains why corruption is
required to reach it, which is a description of the precondition rather than a
reason to ship. Corruption is what the typed guards exist for, and this path
crashes instead of reaching them.

The fix

Hoisting the cc->pageLength >= (uint64) validityBytes test above the vbits block
skips an inconsistent chunk, which falls through to the non-coalesced path where
#1063's typed XX001 fires. One line, and it makes the single-observable-behaviour
property true by construction.

Worth a regression arm in the suite too, now that there is a fixture that produces
it: the geometry is five contiguous chunks coalescing into one span with
page_length shrunk on the highest offset.

[OffgridwithJD]

The one-line hoist fixes the crash. I tested it — and the result proves #1063 has to land first.

Applied the fix to my ASAN worktree, rebuilt against /usr/local/pg18_san (clean, no warnings), and re-ran the exact repro that crashed. Same table, same poisoned page_length = 8 on the last chunk, same path premise asserted.

The patch

Inserted immediately after the containment test, before the vbits block:

if (cc->pageOffset < start || cc->pageOffset + cc->pageLength > end)
    continue;

/*
 * THE BOUND FOR THE vbits COPY BELOW. The containment test above
 * guarantees [off, off+pageLength) is inside buf; the copy reads
 * validityBytes. Those coincide only under this condition, which
 * used to be tested after the copy. An inconsistent chunk is left
 * for the non-coalesced path, which refuses it with a typed error.
 */
if (cc->pageLength < (uint64) validityBytes)
    continue;

off = cc->pageOffset - start;

Result: the overread is gone

EXPLAIN (COSTS OFF) ... -> Index Scan using oob_id on oob

ERROR:  invalid memory alloc request size 4294966679
--- backend alive after the fetch? 1

No sanitizer report. Backend alive. No crash recovery. Compare with the same query before the patch, which produced heap-buffer-overflow, READ of size 625 at columnar_reader.c:4143 and killed the backend. The hoist closes it.

But look at what the error now is

4294966679 is not arbitrary:

pageLength - validityBytes  =  8 - 625  =  -617
2^32 - 617                  =  4294966679

That is the uint32 underflow at columnar_reader.c:4479 on this branch:

uint32  vlen = (uint32) (cc->pageLength - validityBytes);

— the exact cast #1063 replaces with pgcolumnar_chunk_value_bytes(). The skipped chunk falls through to the non-coalesced path exactly as intended, and lands on the unfixed cast, so a user gets a generic invalid memory alloc request size from the allocator instead of the typed XX001 that says which column and why.

So the two PRs are not merely entangled, they are sequenced:

state	what a user sees on this corrupt chunk
#1077 as submitted	backend crash, heap-buffer-overflow
#1077 + hoist	invalid memory alloc request size 4294966679
#1077 + hoist, rebased on #1063	the typed XX001 naming the column

Land #1063 first and rebase this onto it. Then the fallback this fix drops into is the refusal it was designed to reach.

What I did not test

I verified the hoist alone. I did not build the combination of this fix with #1063's helper, so the third row of that table is arithmetic and code reading, not a run. If you want it measured before the tag, say so and I will build the pair — but the first two rows are both measured, and they are the ones that decide whether this ships.

The patch above is yours to take verbatim or reword; I'm not opening a PR against your branch.

[jdatcmd]

Heads up before tomorrow: this now conflicts, on test/pytest/TESTS.md and
nothing else. Not your doing — #1086 and #1087 merged tonight and #1086 added
section 43.

conflicting files    test/pytest/TESTS.md
everything else      auto-merges clean

All five of your PRs claim section 43, so they collide with main and with each
other, not just with the merge that caused it.

test_the_contents_list_is_numbered_in_order requires the contents list AND the
sections to each count 1..N with no gap, so a duplicate or a hole reddens. That
means the number is decided by merge ORDER, and whoever lands second renumbers
again.

Suggested assignment

Taking @OffgridwithJD's recommended order, which exists because #1077's fix falls
through to the cast #1063 repairs:

#1063   section 44    land first
#1039   section 45
#1068   section 46
#1065   renumber when the parallel_leader_participation gap is closed
#1077   renumber when the overread hoist lands, and rebase onto #1063

This one is blocked, so take a number when it is unblocked.

This is #996, and it has now bitten five PRs at once

#996 says every changelog entry inserts at the same anchor so any two conflict.
The same is true of TESTS.md section numbers, and tonight is the clearest
evidence yet: one merge invalidated the number in five independent branches. Worth
linking there rather than treating it as bad luck.

Nothing in your change is affected. The rebase is mechanical and the number is the
only decision in it.

[linuxhikerpm]

CI diagnosis (from job logs, not PR comments)

Red checks on the failing SHA (a6f66be): suites (PG 17) and suites (PG 18). Quoted from those job logs:

PG18:

  FAIL  native_fetch_projection
      >> FAIL  asking for every column is a flag, not an absent set: got [2] want [1]
      >> FAIL  the column test consults that flag rather than a null set: got [2] want [1]

PG17, same two arms, plus:

not in the ledger: native_fetch_coalesce	native_fetch_coalesce	a wide index fetch does not pin once per column	(on major 17)
not in the ledger: native_fetch_coalesce	native_fetch_coalesce	premise: a point lookup uses the index	(on major 17)
not in the ledger: native_fetch_coalesce	native_fetch_coalesce	premise: fetching every projected column touched a measurable number of buffers	(on major 17)
not in the ledger: native_fetch_coalesce	native_fetch_coalesce	premise: fetching one projected column touched a measurable number of buffers	(on major 17)
not in the ledger: native_fetch_coalesce	native_fetch_coalesce	premise: the wide fetch returns the projected values	(on major 17)
    5 check(s) the ledger has never seen.
  PG17 has a check the ledger has never seen, which is not a pass
    ledger integrity failure: classification lost rows: matched 5 + orphans 0 + unprunable 0 + not checked 1233 != 1233 ledger rows -- every row must land in exactly one of the four

Class:

1. Real CI bug — native_fetch_projection.sh counted bool allColumns / !allColumns && !bms_is_member as want [1]. This PR adds pgcolumnar_fetch_coalesce_read(..., bool allColumns, ...), so both counts become 2. Local repro on the unfixed tree: allColumns=2, same got/want as CI. Main already replaced that pin with guarded N of N; rebase is the fix, not rewriting the counts to 2.
2. Honest PG18-only ledger seed for native_fetch_coalesce on PG17. Did not fake 15–17.
3. Needs-rebase vs fcfd3e6 (was CONFLICTING/DIRTY).

Pushed: rebased audit/index-fetch-io onto fcfd3e6. TESTS.md §43 kept as test_pgxn_metadata.py; this twin is §44. Collection 346 / 411. Ledger never-census 1227 matches budget. suites_not_covered 249. After rebase, test/native_fetch_projection.sh with PG_CONFIG=/usr/lib/postgresql/18/bin/pg_config: 17 passed (guarded 2 of 2). New HEAD ab84ab5ed5517c2cab9b7e5dfbe00a8a8cf9bce8. Mergeable now; CI pending. Not merged, not approved.

[OffgridwithJD]

Reviewed in full, found legitimate, and prepared for merge as #1093

Your work here is sound and I want to say that before the mechanics: the coalescing itself is correct, including the adjacent-range merge, and holding the span buffers in the per-fetch context is the right call.

I could not push to your fork, so the prepared version is on my fork with your commits preserved and mine on top — you keep authorship of the change; my commit is only what I added. If you would rather land it from here instead, take the patch from #1093 and push it to this branch, and I will close mine.

What I changed

A memory-safety fix. The validity copy read validityBytes out of a span buffer guaranteed to hold only page_length bytes for that chunk, and the test reconciling them ran three lines after the copy. Under ASAN that is a heap-buffer-overflow that kills the backend on a plain index-scan SELECT:

READ of size 625, 0 bytes after a 2640-byte region
  pgcolumnar_fetch_coalesce_read   (the memcpy)
  pgcolumnar_fetch_row

Main cannot have this shape — its non-coalesced fill reads straight from storage into an exactly-sized destination. Hoisting the pageLength >= validityBytes test above the copy closes it, and both harnesses now pin the ordering.

The ledger rows are re-derived from runs on all five majors, not edited. Your rows read 15;16;17;18 while every other row in the file reads 15;16;17;18;19, so covered_majors includes 19 and the release gate refuses this suite's checks — even though CI stays green, because ci.yml:503 runs 17+18 per PR and only the local five-major matrix adds 19. That is #1071's trap, not a mistake of yours: the gate's own printed recipe says <log> singular.

I widened them by running the suite on PG15/16/17/18/19 and merging those logs, because a row is a claim about where a check was observed and editing field 4 makes that claim without the observation.

One thing worth knowing for next time

All five of your PRs take TESTS.md section 44, and only one can. The numbering is gated — test_the_contents_list_is_numbered_in_order requires 1..N with no gap — so the number is decided by merge order and the second to land renumbers. Worse, the census in check_ledger_budget.txt cannot be resolved when two of these merge: the conflict offers two numbers and neither is right, because each is short by exactly the other PR's row count. It has to be re-derived from the merged ledger.

That is a property of the repository, not of your work — measured and written up on #996.

Nothing here needed a change to suites_not_covered. Your budget files were correct as committed.

[jdatcmd]

Thanks for this. The coalescing itself is correct and I checked it rather than skimmed it: sorting the ranges, merging any whose start falls at or before the running end, one read per merged span, then distributing per column. The adjacent-not-just-overlapping case is handled, which is the point of the change.

There is a heap overread in it, and it reaches users on a plain SELECT.

The defect

In pgcolumnar_fetch_coalesce_read, at this PR's head ab84ab5e:

memcpy(entry->vbits[c], buf + off, validityBytes);      line 4143
...
cc->pageLength >= (uint64) validityBytes)               line 4146

The copy takes validityBytes. The span buffer is only guaranteed to hold
pageLength bytes for this chunk. The test that reconciles the two runs
three lines after the copy has already happened, so a chunk whose
page_length is smaller than its validity bitmap reads past the allocation.

Reproduced on a -fsanitize=address build by lowering page_length below the
validity bitmap on the last chunk by page_offset and issuing a plain
index-scan SELECT:

AddressSanitizer: heap-buffer-overflow
READ of size 625, 0 bytes after a 2640-byte region
  pgcolumnar_fetch_coalesce_read   columnar_reader.c   (the memcpy)
  pgcolumnar_fetch_row
  printtup

main cannot have this, because the non-coalesced path reads from storage into
an exactly-sized destination. The coalesced path is what introduces it.

A behavioural test here would be vacuous. Reading ~117 bytes past a
palloc'd span returns adjacent heap quietly. Without a sanitizer the suite
reports PASS on the broken code, which is why CI is green on 14 of 14 checks.

Second defect: the ledger rows name four majors

Every row this PR adds reads 15;16;17;18 while every other row reads
15;16;17;18;19, so the release gate refuses this suite's checks when the
five-major matrix observes them on PG19. CI runs 17 and 18 only, so it cannot
see this either.

Disposition

Closing in favour of #1093, which carries your commits unchanged (I
confirmed each is an ancestor of that branch), hoists the bound above the copy,
adds a regression arm in both harnesses, and corrects the ledger. Your
authorship is preserved there.

One further thing found while composing it, which is not yours: the coalesced
read runs before the checked decode helper, so a poisoned page_length hit
palloc's 1GB cap and raised XX000 instead of the typed XX001 that #1063
introduces. #1093 now defers such a chunk to the checked path so the SQLSTATE
survives.