Skip to content

test: pin the projection guard's property, not its caller count (#1077) - #1078

Merged
jdatcmd merged 1 commit into
mainfrom
test/1078-guard-the-property-not-the-caller-count
Sep 16, 2026
Merged

jdatcmd merged 1 commit into
mainfrom
test/1078-guard-the-property-not-the-caller-count

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

native_fetch_projection.sh protected one convention -- "every column" is an
explicit flag, never an absent set -- with two arms that counted a string across
columnar_reader.c and compared it against a literal 1. That asserts HOW MANY
honest callers exist, which is a fact about today's tree rather than a property of
the code.

It errs in both directions, and the second one is why this is a fix rather than
a widening.
Four source states, the same two arms:

state              tests guarded flags | OLD arm1  OLD arm2 | NEW
main                   1       1     1 | PASS      PASS     | PASS
honest caller          2       2     2 | RED       RED      | PASS
unguarded fn           2       1     2 | RED       PASS     | RED
unguarded INLINE       2       1     1 | PASS      PASS     | RED

Row 2 is #1077's coalescing read. It takes the flag and tests it exactly as the
convention demands, and was failed for existing.

Row 4 is the finding. Add a bare bms_is_member(c, needed) inside the existing
worker, so no new bool allColumns is declared, and BOTH arms pass. arm2 counts
the GUARDED form, which is still 1, so an unguarded test beside it is invisible to
the arm named "the column test consults that flag rather than a null set". The
arm is anti-correlated with its own name: a correct addition reddens it, a wrong
one does not.

Row 3 shows the one red the old arms do produce carries no information either.
arm1 reddens there for the same reason it reddens on row 2 -- the count moved --
so it cannot tell a correct caller from a broken one.

The change

The arms pin the property against itself: every needed-set membership test
consults the flag, guarded N of N. Honest callers move both counts together; an
unguarded one moves only the total. Both numbers sit inside the compared strings,
so the arm's message is the reconciliation rather than a bare verdict.

Anchored, not asserted: on this tree the pin reads guarded 1 of 1.

Two premises sit beside it because guarded 0 of 0 also satisfies equality.
Against an empty file:

pin        actual[guarded 0 of 0] expected[guarded 0 of 0]   PASS
premise 1  there is a membership test to guard               RED
premise 2  the flag is declared                              RED

A literal match cannot tell "written differently" from "written wrongly", and both
readings are live: reversed operands, a renamed variable, a pgindent wrap, or the
positive form all redden a correct tree. Failing closed is the right direction, so
the message now names both readings instead of one. It is emitted BEFORE the check
and only on mismatch, so the check's NAME stays the ledger key.

Removal proof, against the real suite

mutation applied   md5 b3cbbc36426a -> 06be33a15eab, counts 1 -> 2
suite run          14 passed + 1 failed + 0 unrunnable + 0 skipped = 15
the one red        every needed-set membership test consults that flag rather
                   than a null set: got [guarded 1 of 2] want [guarded 2 of 2]
both premises      stayed green, correctly
restored           md5 b3cbbc36426a, counts 1 1, git diff empty

The note printed above the red, and printed nothing on the clean tree.

Five majors, this exact tree

Own make clean, own build and own .so per major:

PG15 15 PASS   PG16 15 PASS   PG17 15 PASS   PG18 15 PASS   PG19 15 PASS

Each log carries exactly its own major. The suite's check names diffed against the
file: 14 of 15 identical, the one difference being the wide fixture has $((NCOLS + 1)) columns expanding to 41, which is the log doing its job.

No ledger change, and that is checked rather than assumed

native_fetch_projection has no rows in the ledger -- it is one of the 249
uncovered suites -- and cmd_gate refuses only "a check the committed ledger has
never seen, IN A SUITE THE ledger covers"
.

I found that by merging the five logs expecting +3 rows and getting +15, 1230
to 1245, then restoring. Merging would have SEEDED the suite, which is a different
change and not this one. There are no orphans to prune either: the two removed
names were never in the ledger.

Not in this PR

Seeding native_fetch_projection from the five logs would drop
suites_not_covered 249 to 248. It is a clean standalone change; putting it here
would make a 19-line test fix carry a 15-row ledger diff a reviewer has to check
separately. The logs are kept.

Unblocks #1077, whose change is correct and was failed by this guard.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK

`native_fetch_projection.sh` protected one convention -- "every column" is an
explicit flag, never an absent set -- with two arms that counted a string across
`columnar_reader.c` and compared it against a literal `1`. That asserts HOW MANY
honest callers exist, which is a fact about today's tree rather than a property
of the code.

It errs in BOTH directions, and the second is why this is a fix rather than a
widening. Four source states, the same two arms, verified by running them:

    state              tests guarded flags | OLD arm1  OLD arm2 | NEW
    main                   1       1     1 | PASS      PASS     | PASS
    honest caller          2       2     2 | RED       RED      | PASS
    unguarded fn           2       1     2 | RED       PASS     | RED
    unguarded INLINE       2       1     1 | PASS      PASS     | RED

Row 2 is #1077's coalescing read: it takes the flag and tests it exactly as the
convention demands, and was failed for existing.

Row 4 is the defect the arms exist to catch, added inside the existing worker so
no new flag is declared -- and BOTH arms pass it. `arm2` counts the GUARDED form,
which is still 1, so a bare `bms_is_member(c, needed)` beside it is invisible to
the arm named "the column test consults that flag rather than a null set". It is
anti-correlated with its own name: a correct addition reddens it, a wrong one
does not.

Row 3 shows the one red the old arms do produce carries no information either.
`arm1` reddens there for the same reason it reddens on row 2 -- the count moved --
so it cannot distinguish a correct caller from a broken one.

The arms now pin the property against itself: every needed-set membership test
consults the flag, `guarded N of N`. Honest callers move both counts together; an
unguarded one moves only the total. Both numbers are inside the compared strings,
so the arm's message IS the reconciliation. Today the tree reads `guarded 1 of 1`.

Two premises sit beside it because `guarded 0 of 0` also satisfies equality: that
a membership test exists to guard, and that the flag is declared. Against an empty
file the pin passes and both premises redden.

A literal match cannot tell "written differently" from "written wrongly", and
both readings of a mismatch are live -- reversed operands, a renamed variable, a
`pgindent` wrap, or the positive form all redden a correct tree. Failing closed is
right, but the message now names both readings so the next reader does not hunt a
caller that does not exist. It is emitted BEFORE the check and only on mismatch,
so the check's NAME stays the ledger key.

Removal proof, against the real suite rather than extracted expressions:

    mutation applied   md5 b3cbbc36426a -> 06be33a15eab, counts 1 -> 2
    suite run          14 passed + 1 failed + 0 unrunnable + 0 skipped = 15
    the one red        every needed-set membership test consults that flag rather
                       than a null set: got [guarded 1 of 2] want [guarded 2 of 2]
    both premises      stayed green, correctly
    restored           md5 b3cbbc36426a, counts 1 1, git diff empty

Five majors, own build and own `.so` each, on this exact tree:

    PG15 15 PASS   PG16 15 PASS   PG17 15 PASS   PG18 15 PASS   PG19 15 PASS

NO LEDGER CHANGE, and that is checked rather than assumed. `native_fetch_projection`
has no rows in the ledger -- it is one of the 249 uncovered suites -- and `cmd_gate`
refuses only "a check the committed ledger has never seen, IN A SUITE THE ledger
covers". Merging the five logs would have SEEDED the suite: +15 rows, 1230 -> 1245,
not the +3 I expected. That is a different change and is not in this one. There are
no orphans to prune either; the two removed names were never in the ledger.

This is the third arm in this file repaired for counting a string across a whole
file. The `deltuples` comment 15 lines above records the first, fixed by scoping;
these two were left as whole-file counts and did the same thing again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at ae56f72. 14/14 green, and I verified the thing this PR exists to do rather than the thing it says.

The acceptance test: does it unblock #1077

Ran these arms verbatim against the two real trees, not against extracted expressions:

main                 tests=1 guarded=1 flags=1 | premise1=yes premise2=yes | pin=PASS
#1077's real tree    tests=2 guarded=2 flags=2 | premise1=yes premise2=yes | pin=PASS

So suites (PG 18) clears on #1077 without that PR touching columnar_reader.c, which is the whole point. And main anchors at guarded 1 of 1, so the false-positive budget in the body is a measurement.

The removal proof, on a real tree rather than a synthetic one

Planted the row-4 defect — a bare bms_is_member(c, needed) inside the existing worker, no new flag declared — into #1077's actual source:

#1077 unmodified                      t=2 g=2 f=2   pin=PASS
+ unguarded INLINE test (the defect)  t=3 g=2 f=2   pin=RED
mutation asserted applied: +1 membership test, no new flag declared

And the blindness it replaces, on main, where the old arms are not already saturated:

main + the same defect                t=2 g=1 f=1   OLD1=PASS  OLD2=PASS  pin=RED

Both old arms pass the defect on a clean tree. That is the finding, and it is why this is a fix and not a widening.

I reproduced the four-state table independently

Cell for cell, from main's source, before reading your version — including the vacuity case your premises exist for:

tests=0 guarded=0 flags=0 -> pin PASS (equality satisfied by nothing), premises RED

guarded 0 of 0 is the case that would have made this a decorative arm, and both premises catch it.

The note

Emitted before the check and only on mismatch, so the check label stays the ledger key — that is the detail that made it safe to reuse the earlier logs, and diffing the names against the file to prove it is the check I would have asked for.

It also says the thing I care about most: a literal match cannot distinguish written differently from written wrongly. I measured five correct rewrites that redden exactly as the defect does — reversed operands, extra spacing, a wrapped line, the positive form, a renamed variable — so both readings are live, and pgindent wraps on width. Without the note the next reader spends an afternoon hunting a caller that does not exist.

The ledger: correctly nothing

native_fetch_projection has zero rows in main, so it is one of the 249 uncovered suites and the gate cannot refuse its checks. Removing two check names orphans nothing, because neither was ever in the ledger. suites_not_covered stays 249.

You caught that by merging five logs, expecting +3 rows and getting +15, then restoring. That is worth more than the fix: it is the same trap five other PRs are sitting in, from the opposite side, and it is what produced the finding that seeding is opt-in and a partial seed is the only way to lose.

One thing for later, not for this PR

This is the third arm in this file repaired for counting a string across a whole file, and the deltuples comment 15 lines above records the first. Two were left after one was fixed. Worth a sweep of the other suites for grep -c ... "1" against a literal — the shape is mechanical to find and it fails in both directions wherever it appears.

Not a condition of this approval, and I would not hold a 19-line fix for it.

@jdatcmd
jdatcmd merged commit 8e88f42 into main Sep 16, 2026
14 checks passed
@jdatcmd
jdatcmd deleted the test/1078-guard-the-property-not-the-caller-count branch September 16, 2026 16:25
@OffgridwithJD

Copy link
Copy Markdown
Collaborator

A sweep for the same shape elsewhere, since this PR's body says "the third arm in this file". On the numbers it is the third, fourth and fifth — and the file is not the only one.

Not a condition of the approval above; #1078 is right as it stands. Recording it so the class is countable rather than anecdotal.

Filtering test/*.sh and test/selftest/*.sh to the exact shape this PR repairs — a whole-file grep -c over a source file, compared against a literal >= 1, not awk-scoped to a function:

whole-file, want >= 1     12     <- the #1078 shape
scoped (awk-bounded)       3     <- the deltuples pattern, correct
whole-file, want = 0       3     <- "must not appear"; safe, a second honest
                                    caller cannot break a zero

Two of the twelve are this PR. Two more are in the same file

native_fetch_projection.sh   grep -c 'PgColumnarRowIsLive(rel, snap, baseRow)'          "1"
native_fetch_projection.sh   grep -c 'PgColumnarReadRowByNumberCols(rel, snap, baseRow' "1"

Both are call-site counts rather than definitions, so both fail in the same two directions this PR documents: a second honest caller reddens them, and a call written with different arguments is invisible to them. They are the arms named "index deletion asks whether the row is live" and "the reconstruct caller asks only for uncovered columns" — and neither asserts that, it asserts there is exactly one such call today.

The deltuples comment recording the original instance sits fifteen lines above all four.

The rest of the class

native_fetch_cache.sh      4    (one pinned at "2" -- same shape, different constant)
native_fetch_position.sh   2
decode_interrupts.sh       1    (its two siblings ARE awk-scoped: the same file
                                 got it right twice and wrong once)
native_saop_pushdown.sh    1

decode_interrupts.sh is the useful one to look at, because it shows the correct pattern and the incorrect one side by side in one file — awk '/^pgcolumnar_native_load_group\(/,/^}/' | grep -c scopes to the function, and the #define arm counts the whole file.

Not all twelve are wrong

A pinned count is right where the count is the property — exactly one definition, one entry point, one #define. decode_interrupts.sh's ^#define COLUMNAR_DECODE_INTERRUPT(i) is plausibly that. The distinction is whether a second occurrence would be a defect or just a second caller, and it has to be made per arm rather than swept.

Happy to take these as a separate change if nobody else wants them.

OffgridwithJD pushed a commit to OffgridwithJD/pgcolumnar that referenced this pull request Sep 16, 2026
My own new arms tripped selftest/080: five sites piped a captured string into
`grep -q`, which exits on its first match and closes the pipe under its writer.
Caught by @jdatcmd on commandprompt#1081's CI, with the sites rather than a count.

The answer was already in the file family I was working in --
native_fetch_projection.sh:155 uses a `case` over a captured variable. No
subprocess, no pipe, and it reads better than the `&& echo yes || echo no` tail.
Patterns are quoted so the brackets in valOffset[c][present] stay literal rather
than becoming glob character classes.

AND THE DISCARD ARMS GAINED THE PREMISE THEY WERE MISSING. As written they could
not tell "this function does not discard the cache" from "the awk range matched
nothing" -- the same both-readings problem I raised against commandprompt#1078's literal match,
in my own arm one day later. Each function's extraction is now captured, asserted
non-empty, and then matched.

Re-verified rather than assumed, because the arms changed:

    selftest/080   no suite pipes a captured string into an early-exit reader  PASS
    suites         native_fetch_cache 26 passed, native_fetch_position 8 passed
    matrix         identical verdicts to the pre-fix run, every restore md5-checked
    five majors    PG15 PG16 PG17 PG18 PG19, both suites, PASSED

The pre-existing `| sort -n | head -1` sites in the same file are correctly NOT
flagged: `sort` reads all its input before writing, so `head` closing early reaches
`sort` rather than the producer. The detector discriminates rather than banning pipes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012RSw4qMHS7ByE7PY8Ns4cs
jdatcmd pushed a commit that referenced this pull request Sep 16, 2026
#1078 repaired two arms in native_fetch_projection.sh that compared a whole-file
`grep -c` against a literal. A sweep of test/ found TWELVE sites of that shape.
These are six of the remaining ten.

THE REPAIR DIFFERS PER ARM BECAUSE THE FAILURE DIRECTION DOES, which is the part
worth reading:

    the entry key    both directions   -> self-referential, keyed N of N
    the cid reject   noise only        -> scoped to the function that must hold it
    the geometry     blind to 3 of 4   -> membership over all four compared fields
    the discard      proxy for "where" -> the two functions named
    rank, valOffset  noise only        -> scoped to pgcolumnar_fetch_row

TWO OF THE EIGHT I WAS ASSIGNED ARE CORRECT AND ARE LEFT ALONE. A pinned count is
right where the count IS the property: `^#define COLUMNAR_DECODE_INTERRUPT(i)`
appearing twice would be a redefinition, and native_saop_pushdown's premise is
load-bearing for an `awk` range that would silently concatenate two expressions
into one `guard` string. Classifying them took longer than fixing the six.

MEASURED. Every mutation compiles, so the suite rebuilds and runs end to end --
the harness refuses a source/.so mismatch, correctly, and there is no shortcut:

    case                        OLD arms            NEW arms
    unkeyed group lookup        key=1     PASS      RED  keyed 1 of 2
    second keyed lookup         key=2     RED       24 passed
    rowCount dropped            geom=1    PASS      RED  rowCount
    executor-end discard gone   discard=1 RED       RED  names the function
    third discard call          discard=3 RED       24 passed
    rank replaced by a walk     rank=0    RED       RED  rank prefix

The two OLD-PASS rows are the case for the change: an unkeyed lookup and a dropped
geometry field both leave the old arms green. The two OLD-RED-NEW-PASS rows are
what fired on #1077 and cost a correct PR a red.

MY FIRST MUTATION MATRIX WAS WRONG AND I ALMOST SHIPPED IT. It reported the
second-keyed-lookup case reddening an unrelated `natts` arm. Run alone that case
is 24/24. The harness asserted each mutation APPLIED and never asserted it was
RESTORED, so one case was measuring two mutations. Re-run with an md5 restore
assertion per case, and the contamination is gone. A mutation harness that does
not check its own restore produces exactly the false finding I would have filed.

FIVE MAJORS, both suites, own `make clean` each:

    PG15 PG16 PG17 PG18 PG19    native_fetch_cache PASSED, native_fetch_position PASSED

No ledger change. `native_fetch_cache` and `native_fetch_position` have zero rows,
so they are two of the 249 uncovered suites and no check name here is a ledger key.
Checked, not inherited from #1078.

Four of the twelve remain, in native_fetch_cache (0 left), decode_interrupts (1,
correct), native_saop_pushdown (1, correct) and the two #1078 already fixed --
leaving native_fetch_cache's siblings done and nothing outstanding in these two files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012RSw4qMHS7ByE7PY8Ns4cs
jdatcmd added a commit that referenced this pull request Sep 16, 2026
…-not-the-caller-count

Pin six more guards to their property, not their caller count (#1078)
jdatcmd added a commit that referenced this pull request Sep 16, 2026
CHANGELOG.md only. #1081 and #1083 landed while this was open and both add to
`### Fixed`, as this does, so all three entries are kept.

Verified by count rather than by reading the diff:

    conflict markers left                0
    each entry present exactly once      #1077 sweep, #1075, #1080, #1081,
                                         and #1078's, which was already there
    bodiless headings in [Unreleased]    0

The suite file is untouched by the merge: its patch md5 against main is unchanged
from before it, and the 17 check names are identical by sorted diff, so no ledger
key moves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants