test/pytest: port the rest of the differential suite, and a fourth arm that cannot fail (#432)

[OffgridwithJD]

The whole of test/differential.sh now has a pytest twin. Parts 1 and 2 were #1020 and
this PR's first commits; this completes parts 3 to 7.

85 tests, 232 checks, 17 seconds. Every arm ported by name so compare_to_bash.py can
diff the two harnesses by property.

A fourth unfalsifiable arm, and the subtlest of the four

textbloom collate-mismatch probes tk = 'k100'. The column is
'k' || ((g*2654435761)%50000) over 16,000 rows of a 50,000-wide domain:

rows with tk = k100:  0
distinct tk values:   16000
domain coverage:      32.0%

The arm exists to catch a bloom wrongly pushed under a mismatched explicit COLLATE. A
wrongly-pushed bloom skips the chunks holding the match and returns 0. The absent value
also returns 0. So the one defect the arm is built to detect produces exactly the answer it
expects, and the arm passes either way.

That is worse than the three in #1020. Those compared nothing to nothing; this one compares the
right answer to the wrong answer's answer.

Found by a premise arm I added asking whether the probe value exists -- it failed, and the
failure is the finding. The port derives a present value, asserts it is there exactly once, and
adds the direction the arm must fail in: the mismatched collation must return the row rather
than skip it.

Fixture premises asserted where they decide what is under test

This is the part I would want reviewed hardest, because each of these is a way the suite could
be green and mean nothing:

• the bloom fixture's key spread -- each chunk's key range must span at least 90,000 of the
  domain. If the keys were ordered, every bloom arm would pass on zone maps alone and say
  nothing about blooms, and no existing arm would report it.
• the dictionary cardinalities -- 4 and 6 against an md5 per row, so the per-column verdict
  is visible rather than assumed.
• the delete removed exactly 400 rows -- a DELETE matching nothing leaves part 4's
  per-group fallback untested while every arm stays green.
• the update leaves a count neither operation alone would give -- which is the arithmetic
  the metadata count path has to get right and a plain scan gets right for free.

Two smaller judgements

Float aggregates use min/max, not sum. A float sum has no single right answer -- part 1
measures three from heap alone by row order -- so sum there would be asserting a coincidence.

SET on the connection, not ALTER DATABASE. The bash suite uses the latter because each
psql invocation is a new session. This one is not, and copying the workaround would have been
copying a fact about psql into a harness it is not true of.

The load_pair fix, which is the other reason this PR matters

Part 1 is merged and regenerated the fixture into both tables instead of generating once and
copying, which lib.sh's load_pair does deliberately -- "regardless of any volatile
generators". Measured on part 3's own generator, 2000 rows:

generator run twice      rows differing: 2000
generated once, copied   rows differing: 0

Parts 1 and 2 were right by luck, because every load in them is deterministic. Part 3 uses
random(), so with the old helper the oracle would have compared two different fixtures and
reported a columnar defect that does not exist -- on the suite whose whole purpose is to be
believed when it says the two access methods disagree.

I ported the assertions faithfully and rebuilt the fixture my own way, and the fixture
construction is where the previous author had buried the lesson.

Test plan

• 85 tests, 232 checks, 0 fail, on pg18a with the driver.
• Each failure along the way was a premise arm, not an arm: the 61-column count (caught my
  information_schema query returning 81), and the collate probe value (caught the bash
  arm above).
• test_docs_cover_the_corpus passes; all parts named in TESTS.md.
• docs_style.sh PASSED.
• cluster_tests 166 -> 191 in expected_tests.txt, which ci: run the cluster-half pytest tests, and make both jobs assert their size (#1016) #1018 now enforces -- so this
  is the first port slice whose tests are gated from the commit that adds them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[OffgridwithJD]

Added a fix for a latent defect in part 1, which is merged. Found while reading the
helpers part 3 needs rather than by a failing test.

lib.sh's load_pair generates the data ONCE into heap and then copies it:

psql_run "INSERT INTO t_heap $body;"
psql_run "INSERT INTO t_col SELECT * FROM t_heap;"

with its reason stated: "both hold byte-identical logical contents regardless of any
volatile generators."

My helper ran the generator twice, once per table. Measured, 2000 rows,
(1000 + sum(random() - 0.5) OVER (ORDER BY g))::float8:

generator run twice      rows differing: 2000
generated once, copied   rows differing: 0

Every load in parts 1 and 2 is deterministic -- generate_series, arithmetic, md5 -- so it
was right by luck rather than by construction. Part 3's encoding fixture uses random(),
and with the old helper the oracle would have compared two different fixtures and reported a
columnar defect that did not exist, on the suite whose whole purpose is to be believed when
it says the two disagree.

Both loaders now generate once and copy: the module fixture from part 1 and the per-test
helper here. Part 2 still passes, 76 tests and 183 checks, so the change is a no-op on
deterministic loads as expected.

The generalisable bit: a test is its assertions AND its fixture. I ported the assertions
faithfully, by name, and rebuilt the fixture my own way -- and the fixture construction is
where the previous author had buried the lesson. load_pair is four lines and one of them
was load-bearing.

[linuxhikerpm]

The fourth unfalsifiable arm is real: tk = 'k100' matches nothing, and a bloom wrongly pushed under a mismatched collation also returns 0. Generating once into heap and copying is the right load_pair lesson; part 3's random() walk would have compared two fixtures. The bloom spread premise, the 400-row delete, and the 18462 covering-count arithmetic all look load-bearing. CI is green on 5a2abd3bbd88dd2f1828b426b8028accbcdb2fca, including pytest-cluster at cluster_tests 191.

Two leftovers in TESTS.md before I would approve:

• The section 30 intro now says this is the whole suite, then still says "The boundary, encoding, bloom and aggregate parts are separate slices."
• The heading and TOC still say "type matrix".

Do not merge until those match what the file contains. I will re-review the SHA that fixes them.

[OffgridwithJD]

Both fixed at the new head, and they were the same defect twice: a name that outran what it
described.

TOC      30. test_differential.py: the heap oracle, type matrix  ->  all seven parts
heading  ## 30. ... type matrix                                  ->  all seven parts
intro    "whole suite ported" ... "boundary, encoding, bloom and aggregate parts are
         separate slices"                                         ->  names the seven, then
                                                                      scopes the matrix
                                                                      paragraph to part 1

The intro now lists the parts in the order the bash suite runs them, and the twenty-column
paragraph is explicitly part 1's rather than reading as the whole file's.

Checked the consistency mechanically rather than by eye, since that is the thing that went
wrong: 30 TOC entries, 30 sections, symmetric difference empty, no duplicate section
numbers. test_docs_cover_the_corpus and docs_style.sh both pass.

Worth saying plainly: this is the same name-overran-its-content defect I spent the afternoon
finding in other people's work -- the BOGUS arm asserting a field count, administration.md states it beside the advice passing 402 lines away, every committed row has five fields
checking six. I wrote the section as parts 1 and 2 and then extended the file without
re-reading the heading I had written. Caught by you rather than by any arm, which is the part
worth noting: the corpus has an arm for "is every test named in TESTS.md" and none for "does
TESTS.md describe what it contains", and the second is not obviously automatable.

That is the gap #1024 is about in a narrower form -- it proposes dropping the section numbers
and adding a count arm. A count arm would not have caught this one.

[jdatcmd]

Approving. The whole of differential.sh now has a twin, and the fourth arm is the sharpest finding either of us has produced on #432.

The collate-mismatch arm, verified arithmetically here

tk = 'k' || ((g * 2654435761) % 50000), g in 1..16000

distinct values   16000 of a 50000 domain (32.0%)
is 100 among them FALSE -- no g produces k100

So the probe value does not exist, and that is worse than the three in #1020 for the reason you give. Those compared nothing to nothing. This one compares the right answer to the wrong answer's answer: a bloom wrongly pushed under a mismatched COLLATE skips the chunks holding the match and returns 0, and the absent value returns 0. The one defect the arm exists to detect produces exactly the result it asserts.

And the heap oracle cannot save it either — heap returns 0 correctly, columnar returns 0 whether correct or wrongly skipping, so diff_query agrees at 0 in both worlds. An arm whose oracle cannot distinguish the defect from the expected answer is a strictly stronger failure than one that asserts nothing.

The fixture premises are the right things to have asserted

The bloom key-spread one especially:

If the keys were ordered, every bloom arm would pass on zone maps alone and say nothing about blooms, and no existing arm would report it.

That is the same shape as the collate arm one level up — a whole family of arms green for a reason unrelated to what they name. Asserting the spread is what makes the family mean anything, and nothing in the bash suite does it.

the delete removed exactly 400 rows is the other one I would keep: a DELETE matching nothing leaves part 4's per-group fallback untested while every arm stays green.

The load_pair fix is the reason this PR matters beyond the port

generator run twice      rows differing: 2000
generated once, copied   rows differing: 0

Parts 1 and 2 right by luck because every load in them is deterministic, and part 3 uses random(). Without it the oracle compares two different fixtures and reports a columnar defect that does not exist — on the suite whose entire purpose is to be believed when it says the two access methods disagree. Correcting your own merged work rather than building on it is the call I would want made.

Two judgements I agree with and would have got wrong

Float aggregates as min/max rather than sum. Part 1 measured three answers from heap alone by row order, so a sum arm there asserts a coincidence about summation order.

SET on the connection rather than ALTER DATABASE. The bash suite needs the latter because each psql is a new session; copying it here would be copying a fact about psql into a harness where it is not true. That is the kind of thing a port carries across without noticing.

Merging when CI settles.

[linuxhikerpm]

Approving d392b80ff954fbad09436a7d2bfca4fab7e27ae4.

The two leftovers from 5a2abd3b are gone: TOC, heading and intro now say the whole suite, all seven parts. CI is green, including pytest-cluster at the re-derived cluster_tests after merging main.

The collate-mismatch arm, the one-shot load_pair copy, and the fixture premises still look load-bearing. I will not merge it.