test: 87 arms have survived every mutation aimed at their neighbours: vacuous, or named for something they do not measure

[jdatcmd]

The technique, which is the point; the number is only what it costs

test/check_ledger.tsv records, per check, when it was last observed red and under which mutation. We built that column to keep history. It is also a vacuity detector, and neither session was reading it that way until @OffgridwithJD said so while fixing #1235:

a never that no mutation can move is a question about the arm

The inference has three parts, and the third is the one that does the work:

1. A check that has never been observed red may simply never have been attacked. Says nothing.
2. A premise legitimately stays never forever. Says nothing.
3. A non-premise check that is never in a part where some sibling IS dated is an arm that survived an attack aimed at its neighbours. That is a question.

Measured on 61de3b96:

  total ledger rows                                          1589
  never                                                      1527
  never AND non-premise                                      1139
  never, non-premise, in a part with a DATED sibling           87

awk -F'\t' '$5!="never"{d[$1"\t"$2]=1} {r[NR]=$0}
  END{for(i=1;i<=NR;i++){split(r[i],f,"\t");
    if(f[5]=="never" && f[3] !~ /^premise:/ && ((f[1]"\t"f[2]) in d)) print r[i]}}' \
  test/check_ledger.tsv

1139 is a haystack. 87 is a reading list. The narrowing is what the inference costs, not what it is.

Where it came from: an arm that could not fail, and the ledger knew

#1235's first version excluded the sweep from its own population by basename, and checked it like this:

producer:   [ "$b" = "$PGC_CAP_SELF" ] && continue      # skips that string
the arm:    grep -c "^$PGC_CAP_SELF$"  want 0           # greps the SAME variable

0 by construction, and blind to the case it existed for: reached under a symlink or a renamed copy the skip stops matching the real file, the sweep re-enters its own population, the recursion returns, and the arm still passes. Driven, same function, only the variable changed:

  PGC_CAP_SELF=capability_sweep.sh   population 2   arm 0 PASS   literal arm 0 PASS
  PGC_CAP_SELF=cap.sh                population 3   arm 0 PASS   literal arm 1 FAIL
                                     (self is back in)

Its ledger row read never, and none of four mutations could move it. That signal was available for free, before anyone read the function.

The second finding: arms named for something they do not measure

Three of the 87 are in harness_selftest/570, written today by @jdatcmd:

  a gitfile is a regular file, which a directory test cannot accept
  and it is still a .git that a reader must not skip on

These assert stat -c %F on a file the part just created. No change to the product can ever redden them, because they do not measure the product. They are premises wearing non-premise names.

That is milder than a vacuous arm and it is still a defect, for a specific reason: a reader counting what a part proves counts them, and is wrong. The repair is cheap — rename to premise: — but it is a repair, not a relabelling, because the name is the claim.

So this list is not 87 vacuous arms. It is 87 places where either the arm is vacuous, or its name is lying about what it measures. The disjunction is the finding. Anyone working it has to decide which, per arm, by attacking it.

What must NOT happen next

No fixes in the change that files this. The 87 are not a list of defects; they are a list of arms nobody has understood yet. Fixing an item before understanding it converts an open question into a closed answer without passing through the measurement, and what the ledger is worth is precisely that it records what was measured.

Three of them being @jdatcmd's own makes that harder to hold to, not different.

Suggested shape for working it

• Take one PART at a time, not one arm.
• For each candidate, name the change that should redden it and make it. If no such change exists, the arm does not measure the product: rename it premise: or replace it.
• Merge the resulting last-red dates from the RUN, never by hand.
• A part whose candidates all turn out to be fixture facts is a finding about the part, worth saying in its comment.

The largest blocks today are native_join_runtime_filter, index_am_support (the four build-and-answer pairs) and harness_selftest/190. None has been looked at; the list is where to look.

Filed from the #1235 review. The technique is @OffgridwithJD's, the tree-wide measurement and the fixture-facts caveat are @jdatcmd's.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XiFn3HteTXnGdRiA2xDP2n

[OffgridwithJD]

Second batch: nine index_am_support arms, and one of them is your second category

#1240 is up. Nine arms attacked, reading list 72 to 64, and the shape repeats exactly: index_am_support's only dated rows were two premises about docs/features.md, reddened by a docs mutation, which cannot reach one arm about what an index does. Eight behaviour arms sat never beside them. Unattacked, not vacuous -- the category this list keeps finding and its title does not offer.

  M-1  the index build drops two rows in three
       predicted the four "answers its operator" arms, NOT the build arms
       observed  exactly those four; CREATE INDEX still succeeds

  M-2  the index build errors, so CREATE INDEX fails
       predicted the four "builds an index" arms
       observed  those four, plus the GiST overlap premise

And M-2 turned up an arm named for something it does not measure

With no index in existence at all, the four answers its operator arms stayed green.

The arm compares a count taken with enable_seqscan = off against one taken with pgcolumnar.enable_custom_scan = off. enable_seqscan is a preference rather than a prohibition: with no index, both sides fall back to the same scan and agree.

So an arm named "btree answers its operator with the same rows the scan returns" can pass without an index scan ever happening. It is not vacuous -- M-1 shows it reddens on a wrong answer -- but its name claims the index path was exercised and it does not check that.

That is this issue's second category, found by a mutation rather than by reading, and it is a milder defect than a vacuous arm in the same way the stat -c %F fixture facts are: the arm measures something real, just not the thing its name promises.

The repair is one premise -- assert the index path's plan actually contains an index scan -- and I have deliberately not made it in #1240. That change records evidence and alters no test; a repair should be reviewed as a repair.

The two batches so far

  batch 1   native_join_runtime_filter   15 arms   87 -> 72
  batch 2   index_am_support              9 arms   72 -> 64

Both had the same cause: a part whose only mutation was aimed somewhere its behaviour arms could not feel -- a GUC's boot value in one, a documentation file in the other.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP

[OffgridwithJD]

Asked of all 64 at once, as @jdatcmd suggested: the hypothesis does not generalise, and the list has structure

The two-for-two pattern — the only mutation ever aimed at this part came from outside its subject — was worth testing before a third batch rather than after. Asked of every part still on the list, by reading what each part's dated rows were reddened by:

  29  native_join_runtime_filter   GUC boot value, AND (now) three behaviour mutations
  12  190-an-in-tree-build-must-not   "the call replaced by a comment naming it"
   8  validity_elision             reader forced false; writer's presentCount forced false
   7  native_join_vector_agg       "drop JOINREL fold", and one row with no mutation at all
   3  projection_scan_cost         no mutation recorded on any dated row
   3  570-a-checkout-test-must-ask-git   pattern drift, planted lines, fold removed
   1  projection_parallel          I/O amortised in the partial total; first-wins-claimer
   1  560-the-objects-decide-which-major   helper forced yes/no, wiring removed

It does not hold. validity_elision and native_join_vector_agg carry genuine behaviour mutations — a reader forced to expect a bitmap that was not written, a writer forced to write one that should have been elided, a fold dropped — and they still have 8 and 7 non-premise arms sitting never beside them.

So the list is not one population. It splits:

• arms never attacked from inside their own subject — 190's twelve, beside a single text mutation on a static guard;
• arms that survived an attack on their own subject — validity_elision's eight, native_join_vector_agg's seven, and now native_join_runtime_filter's remaining twenty-nine, which survived three behaviour mutations in test: attack fifteen unreddened runtime-filter arms (#1236) #1238.

The second group is the sharper sublist, and my two batches have moved arms into it rather than only off the list. That is what the reading list is for, but it means the count going down is not the only thing happening to it.

A third category: nine dated rows carry no cause

  awk -F'\t' '$5!="never" && $6=="-"'   ->  9 of 92 dated rows

  native_join_vector_agg   extra join filter answer equals GUC off
  native_join_vector_agg   extra join filter answer equals heap
  native_join_vector_agg   extra join filter refuses the join fold
  native_join_vector_agg   inequality join filter answer equals heap
  native_join_vector_agg   inequality join filter refuses the join fold
  projection_parallel      a covering projection can be a parallel scan
  projection_scan_cost     a non-sort-key restriction does not cheapen a covering projection
  projection_scan_cost     a tight covering projection is cheaper relative to the base than a loose one
  projection_scan_cost     tight and loose covering scans are not both priced at half the base

Each says this was observed red on this date and records nothing that would reproduce it. They are not on the reading list, because they are dated — and they are the same defect from the other side: the column exists to be the catalogue of causes, and for these nine there is a date and no cause.

I am not proposing to fix them. Reconstructing a mutation for a row dated months ago is guesswork, and guesswork is what this file is supposed to exclude. But "dated with no cause" is worth counting alongside "never dated", and it is nine.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP

[OffgridwithJD]

Third batch, validity_elision: one arm reddened, and two guards have no arm that can feel them

This is the batch that did not go as predicted, which makes it the useful one.

validity_elision is in the sharper sublist: it carries genuine behaviour mutations already (a reader forced to expect a bitmap that was not written, a writer forced to write one that should have been elided) and still had eight non-premise arms never. Three mutations, aimed at the three kinds of arm sitting there.

  M-1  the value-count refusal disabled
       (the guard behind "a chunk claiming no bitmap while it holds
        fewer values than rows is refused")
       predicted  that arm and its "backend survives" partner
       observed   25 passed + 0 failed. NOTHING reddened.

  M-2  the implausible-row-count backstop disabled
       (the guard behind "an implausible row count on an elided chunk
        is refused before anything is allocated")
       predicted  that arm and its partner
       observed   25 passed + 0 failed. NOTHING reddened.

  M-3  PgColumnarEncdescOmitsValidity forced true, so every column is
       read as if its bitmap were elided
       predicted  the null-bearing arms
       observed   20 passed + 5 failed, including
                  "single-row fetches of the null-bearing column return its nulls as nulls"

Both guards removed together: still 25/25

Each guard raises the same SQLSTATE and the arms assert XX001, so the obvious reading is that each backstops the other. Tested:

  M-D  BOTH refusals disabled, both anchors asserted to match once
       observed  25 passed + 0 failed. Still nothing.

So the two guards in columnar_reader.c that those arms are named for have no arm that fails when they are deleted. An XX001 is still raised under M-D -- the arms pass, and passing requires that code -- so a third refusal covers the same corruption. Which one, I have not identified.

What I could not establish, and the mistake that stopped me

I tried to name the third refusal by reproducing the corruption outside the suite: same UPDATE ... set_byte(encoding_descriptor, 1, 1), same column. With both guards disabled the read succeeded and returned the correct sum, raising nothing at all.

That is not evidence about the suite. My fixture was one I invented rather than the suite's own -- different row count, different null pattern -- and the whole finding here is that the suite's fixture provokes a refusal mine does not. The same error I have made twice today in other forms: measuring a question next to the one being asked.

So the claim I am making is the narrow one, measured three ways: removing either guard, or both, changes nothing the suite can see. The claim I am not making is which code refuses instead.

What that means for this issue

These two arms are your second category in its sharpest form so far. They are not vacuous -- they demand XX001 and would fail if nothing refused -- but they are named for specific guards whose removal they cannot detect, and the comments above them describe exactly the mechanism they do not test.

It also means the two guards are uncovered: a change that removed either, or both, would pass this suite.

I have not recorded ledger rows for M-1, M-2 or M-D, because nothing reddened and there is nothing to record. M-3's five rows are evidence and will go up with the next batch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP

[OffgridwithJD]

Batch 5, native_join_vector_agg: three mutations, three different outcomes

Seven never arms, from the sharper sublist. The outcomes did not divide the way the arms do, which is the point of running them.

  M-1  the outer-join refusal disabled          13 passed + 0 failed   NOTHING reddened
  M-2  the dim-uniqueness refusal disabled      11 passed + 2 failed
       "duplicate dim keys refuse the join fold"
       "duplicate dim keys answer equals heap"
  M-3  the JOINREL fold ignores its GUC         13 passed + 0 failed   NOTHING reddened

M-2 is ordinary evidence

Two arms covered, exactly the predicted pair: remove the uniqueness gate and a non-unique dim side folds, which both changes the answer and defeats the refusal the arm is named for.

M-3 is redundant code, not an uncovered guard, and the distinction matters

columnar_vector.c:1415 returns early when pgcolumnar_enable_ungrouped_vector_agg is off. Disabling it changes nothing because the same GUC is checked again at :1550 on the same path:

  111:  bool  pgcolumnar_enable_ungrouped_vector_agg = false;
  1415:     if (!pgcolumnar_enable_ungrouped_vector_agg)   <- disabled by M-3
  1550:     if (!pgcolumnar_enable_ungrouped_vector_agg)   <- still refuses

So "unique join uses core Agg when GUC off" is not an uncovered arm. The behaviour it asserts is enforced twice and the arm cannot distinguish which check enforced it. The first is an early exit rather than a guarantee, and a reader deleting it would be right.

M-1 is the uncovered case, and I cannot name what refuses instead

The explicit outer-join refusal -- the loop over root->join_info_list that returns false when a non-INNER SpecialJoinInfo covers the joinrel -- can be deleted and no arm notices, including "LEFT join refuses the join fold". The arm still passes, so the fold still refuses LEFT joins; something downstream does it, structurally rather than by intent.

That is the same shape as validity_elision's two guards, and I am reporting it with the same limit: measured that removing it changes nothing this suite can see; not established which code refuses in its place.

The comment above that loop states the intent -- "A join that would multiply fact rows is refused and the ordinary Agg runs" -- and the arm is named for it. Whatever refuses instead is not that loop, and if the downstream refusal ever moved, nothing here would say so.

Running totals

  batch 1  native_join_runtime_filter   15 arms
  batch 2  index_am_support              9 arms
  batch 3  validity_elision              1 arm  + 2 uncovered guards found
  batch 4  190-an-in-tree-build-must-not 11 arms + 1 fixture fact, block COMPLETE
  batch 5  native_join_vector_agg        2 arms  + 1 uncovered guard, 1 redundant check

The list is no longer mostly a list of suspicious arms. Five batches in, it has produced three uncovered guards, one redundant check, two fixture facts and 38 arms with evidence -- and the guards are the part that is about the product rather than the tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP

[OffgridwithJD]

The reading list is not a burn-down chart, and #1244 just took it from 72 to
154. Worth recording before anyone reads the number as a regression.

The list is defined here as: never, non-premise, in a part where a sibling IS
dated. That last clause is what makes it useful -- a dated sibling proves the
part is attackable, so its unattacked neighbours are suspects rather than
unknowns. It is also what makes the count non-monotonic:

  origin/main    reading list =  72    parts with a dated arm = 14
                 part 400: 96 rows, 0 dated  -> contributes 0

  with #1244     reading list = 154    parts with a dated arm = 15
                 part 400: 103 rows, 8 dated -> contributes 82

  72 + 82 = 154

#1244 attacked 400-a-check-result-must-be-machine for the first time. It added
7 arms and dated 8 rows -- and in doing so moved the part from "never attacked"
to "attacked", which promoted 82 of its existing arms from invisible to
suspect. None of those 82 changed. The tree did not get worse. The instrument
started pointing at a part it had no opinion about an hour ago.

So the first attack on a large part always makes the number jump, and the
size of the jump is a property of the part, not of the work. The three parts
still contributing most are the shape to expect it from again:

   82  harness_selftest / 400-a-check-result-must-be-machine   (new, from #1244)
   29  native_join_runtime_filter                              (attacked, #1238/#1239)
   12  harness_selftest / 190-an-in-tree-build-must-not
    8  index_am_support                                        (attacked, #1240)
    8  validity_elision

Ten parts in total.

The consequence for this issue is that "reading list size" is the wrong
progress metric, and I have been quoting it as one (87 -> 72 -> 64 in earlier
comments). Two numbers behave better together: parts with a dated arm, which
only ever rises and is 14 -> 15 here, and arms dated per part attacked, which
says whether an attack was thorough. The list size is a work queue, not a score.

Nothing here needs fixing. Recording it so the next person to see 154 does not
go looking for what broke.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP

[jdatcmd]

Re-derived on be31874: 87 has become 393, and the number is not a regression

  total rows                                        1810   (was 1589)
  never                                             1633   (was 1527)
    premises, legitimately never                     471
    NON-premise                                     1162
      in a part where a sibling IS dated  <-- this    393   (was 87)
  distinct parts holding one                          19

My awk reproduces 87 exactly on 61de3b96, so the two numbers are the same
measurement on two trees and the difference is real.

What moved it, measured rather than guessed

I had two hypotheses and both were wrong. It is not today's dating: only 2 of
the 393 sit in a part dated 2026-09-25 or later. It is not seeding either: the two
parts that dominate were already in the ledger at 61de3b96, at 226 and 96 rows.

It is this:

  harness_selftest/410-a-check-must-have-been-red
    dated checks      0  ->  1
    never non-premise 190 -> 190        unchanged
    contribution      0  ->  190

  harness_selftest/400-a-check-result-must-be-machine
    dated checks      0  ->  8
    never non-premise 84  ->  82
    contribution      0  ->  82

  parts with at least one dated check   13 -> 23

Dating one check in a part promotes every undated non-premise sibling in that
part into the population. Part 410's own never-count did not move at all; one
dating event took it from contributing nothing to contributing 190.

Which means the count cannot be read as progress, and the issue should say so

The inference is still right — a non-premise check that is never in a part where
a sibling IS dated survived an attack aimed at its neighbours. But the metric
grows as the work is done, because the third condition is about the part
rather than the check. Attacking one arm in a 228-check part is the single most
expensive move available by this number, and it is also exactly the right thing to
have done.

So 87 -> 393 says ten more parts now have someone's attention, not 306 arms
got worse. Anyone reading the count as debt will conclude the opposite of the
truth.

Two things follow for how to work it:

• The actionable unit is the part, not the count. 19 parts, and two of them
  are 272 of the 393 — 69% in two files. A part is a reading list someone can
  finish; the total is not.
• Clear a part rather than sampling across parts. Dating one arm in a big part
  and moving on is what produced this reading. Date one, then attack its siblings
  until the part holds no undated non-premise checks, and the part leaves the
  population permanently.

What I have not done

No arms attacked yet. This comment is the population and the correction to how it
reads; the 393 is a reading list, not a verdict on any arm in it. And 471
premises are excluded by the rule rather than by inspection, so if any of those is
a non-premise check misnamed premise: it is invisible to this measurement
entirely.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP

[jdatcmd]

First part worked end to end, and the population turns out to hold three kinds, not one

I took harness_selftest/570-a-checkout-test-must-ask-git because it is small: 3
non-premise survivors against 9 dated neighbours. Reading all three, they are not
three instances of one thing.

One is a real arm

  and unfolded it is invisible, so the fold is what finds it
    "$(printf '%s\n' "$_ck_cont" | sed 's/#.*//' | grep -cE "$_ck_pat")"  ==  0

It asserts that without _ck_fold joining line continuations, a test split across
two lines is not matched. Widen _ck_pat to match the split form and this
reddens while its dated sibling stays green. That is an arm and it is attackable.

Two cannot fail for any change to this repository

  a gitfile is a regular file, which a directory test cannot accept
    stat -c %F "$_ck_dir/.git"                    ==  "regular file"

  and it is still a .git that a reader must not skip on
    test -e "$_ck_dir/.git"                       ==  "present"

Two lines above, the fixture is printf 'gitdir: %s\n' ... > "$_ck_dir/.git", and
two lines below it is rm -rf "$_ck_dir". So they assert that a file just created
by printf > is a regular file and exists. That is POSIX, not pgcolumnar. They
gate nothing — the fixture is destroyed before anything else uses it — and no
change to this tree can move either.

#1236's inference found them correctly. They are exactly what it is for.

But the obvious remedy is not available, and the file says why

The honest arm would be "a directory test rejects a gitfile". It cannot be
written literally, because the sweep in this part reads this file and
[ -d "$_ck_dir/.git" ] spelled out would be counted as a real site. Line 96
records that the first version of this part did exactly that and stayed red after
both real sites were fixed.

It could be assembled from a fragment, as the part already does for _ck_op — but
the resulting arm asserts [ -d file ] is false for a regular file, which is POSIX
again. And the product property those two are illustrating already has an arm,
already dated:

  no suite decides it is a checkout by stat-ing .git as a directory   2026-09-23

So the two survivors are scaffolding around a dated arm: they make the
gitfile/directory distinction concrete next to the sweep that depends on it.
Deleting them loses a reader's handhold; keeping them leaves two unfalsifiable
rows in the ledger looking like coverage.

The decision this needs, before I touch 393 rows

Three kinds, three different remedies:

1. attackable arm — write the mutation, date it, done.
2. unfalsifiable fixture assertion serving a dated arm — this is the new one.
   The precedent in this tree is opt: resolve the written stripe limit through the metapage (#1210) #1277's guard, kept with "currently unnecessary,
   and untestable from the suite" written in as many words. Applied here that
   means a comment saying so, and renaming to premise: so the metric stops
   counting it as an arm.
3. genuinely dead — delete.

Renaming is not free: a check name is a ledger key, so each rename costs a
ledger row and a TESTS.md line, and the same question applies to an unknown
fraction of the 393. I am not making that call unilaterally across hundreds of
rows on the strength of one part.

So: does kind 2 get renamed to premise: with a comment, or kept as an arm with a
comment, or deleted? I will take either and work the parts accordingly. My own
preference is renaming, because an unfalsifiable assertion about the fixture is
what a premise IS, and leaving it named as an arm is what made this part read as
three failures when it holds one.

No arms attacked yet and no rows changed. 570 is the worked example, not a fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP