From abe5ea6da7f643974a33ded1f5fa813abb8572f3 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Tue, 22 Sep 2026 10:37:26 -0600 Subject: [PATCH 1/3] test: native_reclaim_cycles could not reach the defect it guards (#1138) @OffgridwithJD measured it: delete the #84 fix, rebuild, and the suite reported 12 passed + 0 failed, arm for arm, including `compact_rewrite cycle N returns a count (no self-conflict)` -- the arm named after the defect. #84 needs ONE COMMAND to allocate from the free list MORE THAN ONCE. `pgcolumnar.reclaim_coalesce` defaults ON, so compaction merges adjacent freed ranges and the free list holds one or two rows however much is freed: cycle 1 2 3 4 5 free rows 0 1 2 2 2 One row is not two allocations. The just-consumed row was never re-selected, so the missing CommandCounterIncrement cost nothing observable. THE FIXTURE NOW FRAGMENTS THE FREE LIST, and says so. Coalescing off in the CLUSTER CONFIG rather than by SET -- every psql_run here is its own session, so a SET would last one statement and the writing session would not have it. Then 30,000 rows in groups of 1,000 and a CONTIGUOUS block of whole groups freed at once, which is what puts many separate reusable ranges on the list. A rotating slice frees a little from every group and coalesces back to one range. TWO PREMISES, because the old suite's silence came from an unasserted precondition: the table has several row groups to rewrite read from the catalog the free list is fragmented 18 rows, floor of 5 FOUR CELLS, PG17, `make clean` between builds: suite build result new clean 14 passed + 0 failed new MUTATED 9 passed + 5 FAILED old MUTATED 12 passed + 0 failed <- the defect, on the SAME .so old clean 12 passed + 0 failed The mutation is the #84 fix itself, the CommandCounterIncrement in PgColumnarAllocateFreeSpace whose comment predicts exactly this. Under it the new suite reddens with the defect's own words: FAIL compact_rewrite cycle 1 returns a count (no self-conflict): got [bad:ERROR: tuple already updated by self] want [ok] on every cycle. The old suite passed on binary b4705762baf8; the new one reddens on it. Five majors, 14 passed + 0 failed each, free list 18 rows on every one: pg15 pg16 pg17 pg18_nc pg19 No ledger rows: native_reclaim_cycles is not one of the covered suites. The pytest twin already had this fixture; @OffgridwithJD left the shell side deliberately, rather than edit a suite inside a PR about the port. harness_selftest 1092 passed + 0 failed shellcheck -S error -s bash test/*.sh test/selftest/*.sh rc=0 Closes #1138. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XiFn3HteTXnGdRiA2xDP2n --- CHANGELOG.md | 23 +++++++++++++ test/native_reclaim_cycles.sh | 61 ++++++++++++++++++++++++++++++++--- 2 files changed, 80 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 198c08ba..b25680f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -112,6 +112,29 @@ true until the next version shipped. ### Fixed +- `native_reclaim_cycles` could not reach the defect it guards (#1138). + + It is the declared regression guard for #84. Deleting the #84 fix left it reporting + `12 passed + 0 failed`, arm for arm, including the arm named after the defect. + + #84 needs one command to allocate from the free list more than once. + `pgcolumnar.reclaim_coalesce` defaults on, so compaction merges adjacent freed + ranges and the free list holds one or two rows however much is freed. Measured on + the old fixture, `free_space` rows before each `compact_rewrite`: + + | cycle | 1 | 2 | 3 | 4 | 5 | + | --- | ---: | ---: | ---: | ---: | ---: | + | free rows | 0 | 1 | 2 | 2 | 2 | + + One row is not two allocations, so the just-consumed row was never re-selected and + the missing `CommandCounterIncrement` cost nothing observable. + + The suite now runs with coalescing off and frees a contiguous block of whole groups. + It **asserts the free list is fragmented before relying on it**: 18 rows, against a + floor of 5. With the #84 fix removed it now reddens five arms with + `ERROR: tuple already updated by self`, on the same binary the old fixture passed + on. Found by @OffgridwithJD, whose pytest twin already had the fixture. + - A pytest run killed with `SIGTERM` leaked its throwaway cluster (#1170). Python does not run `finally` blocks when the default `SIGTERM` disposition diff --git a/test/native_reclaim_cycles.sh b/test/native_reclaim_cycles.sh index 05e9c09e..f66174d9 100755 --- a/test/native_reclaim_cycles.sh +++ b/test/native_reclaim_cycles.sh @@ -21,15 +21,41 @@ # Written fresh for pgColumnar. set -uo pipefail + +# COALESCING OFF, OR THIS SUITE CANNOT REACH THE DEFECT IT GUARDS (#1138). #84 +# needs ONE COMMAND to allocate from the free list MORE THAN ONCE. +# `pgcolumnar.reclaim_coalesce` defaults ON, and compaction then merges adjacent +# freed ranges, so the free list holds one or two rows however much is freed. +# Measured on the old fixture, free_space rows before each compact_rewrite: +# +# cycle 1 2 3 4 5 +# free rows 0 1 2 2 2 +# +# One row is not two allocations, so the just-consumed row was never re-selected +# and the missing CommandCounterIncrement cost nothing observable. Delete the #84 +# fix, rebuild, and the old suite reported 12 passed + 0 failed, arm for arm -- +# including `compact_rewrite cycle N returns a count (no self-conflict)`, the arm +# named after the defect. Found by @OffgridwithJD. +# +# IN THE CLUSTER CONFIG, NOT A `SET`. Every psql_run here is its own session, so a +# SET would last exactly one statement and the writing session would not have it. +PGC_EXTRA_CONF="${PGC_EXTRA_CONF:-} +pgcolumnar.reclaim_coalesce=off" +export PGC_EXTRA_CONF . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" pgc_setup "${1:-/usr/local/pg17/bin/pg_config}" -# 8000 rows in 8 groups of 1000, so each compaction rewrites several groups and -# each command performs several free-space allocations. -GEN="SELECT g AS id, (g % 100) AS v, md5(g::text) AS payload FROM generate_series(1, 8000) g" +# 30,000 rows in groups of 1,000, then a CONTIGUOUS block of whole groups freed +# at once. That is what puts many separate reusable ranges on the free list; a +# rotating slice frees a little from every group and coalesces back to one range. +ROWS=30000 +GROUP=1000 +DEL_LO=6001 +DEL_HI=24000 +GEN="SELECT g AS id, (g % 100) AS v, md5(g::text) AS payload FROM generate_series(1, $ROWS) g" psql_run "CREATE TABLE h (id int, v int, payload text);" psql_run "CREATE TABLE n (id int, v int, payload text) USING pgcolumnar;" -psql_run "SELECT pgcolumnar.set_options('n', stripe_row_limit => 1000, chunk_group_row_limit => 1000);" +psql_run "SELECT pgcolumnar.set_options('n', stripe_row_limit => $GROUP, chunk_group_row_limit => $GROUP);" psql_run "INSERT INTO h $GEN;" psql_run "INSERT INTO n $GEN;" @@ -37,6 +63,33 @@ fsize() { q "SELECT pg_relation_size('n');"; } hash_n() { pgc_set_hash 'SELECT id, v, payload FROM n'; } hash_h() { pgc_set_hash 'SELECT id, v, payload FROM h'; } +free_rows() { q "SELECT count(*) FROM pgcolumnar.free_space + WHERE storage_id = pgcolumnar.get_storage_id('n');"; } + +# SEVERAL GROUPS, OR ONE COMMAND CANNOT ALLOCATE TWICE. Read back from the +# catalog rather than assumed from the option that asked for it. +_groups="$(q "SELECT count(*) FROM pgcolumnar.storage s + JOIN pgcolumnar.row_group rg USING (storage_id) + WHERE s.relation_oid = 'n'::regclass;")" +check "premise: the table has several row groups to rewrite" \ + "$([ "${_groups:-0}" -ge 2 ] && echo "many ($_groups)" || echo "TOO FEW ($_groups)")" \ + "many ($_groups)" + +# Free a large CONTIGUOUS block of whole groups and compact, which is what puts +# many separate reusable ranges on the free list. +psql_run "DELETE FROM h WHERE id BETWEEN $DEL_LO AND $DEL_HI;" +psql_run "DELETE FROM n WHERE id BETWEEN $DEL_LO AND $DEL_HI;" +psql_run "SELECT pgcolumnar.compact('n');" +_free="$(free_rows)" +echo " (free_space rows after the block delete: $_free)" + +# THE PRECONDITION FOR #84, ASSERTED RATHER THAN HOPED FOR. With a free list of +# one row -- which is what coalescing produces -- no command allocates from it +# twice, and every arm below passes on a build with the fix removed. +check "premise: the free list is fragmented, so one command allocates from it more than once" \ + "$([ "${_free:-0}" -ge 5 ] && echo "fragmented ($_free)" || echo "TOO FEW ($_free)")" \ + "fragmented ($_free)" + check "initial parity" "$(hash_n)" "$(hash_h)" # Repeated {delete a rotating slice, compact_rewrite}. No inserts, so the only From 13ed40f7917f8559d28e17ecf7329fcf867ab196 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Tue, 22 Sep 2026 10:53:37 -0600 Subject: [PATCH 2/3] test: the explanation was wrong -- it is the GUC, not the fragmentation (#1138) @OffgridwithJD separated the two factors and my comment told the next maintainer to preserve the wrong property. `reclaim_coalesce` does TWO things: it merges adjacent freed ranges, AND it carries its own CommandCounterIncrement on the free path (columnar_metadata.c:792, guarded by `if (pgcolumnar_reclaim_coalesce)`). That second one does the visibility work the #84 fix would otherwise do. ISOLATED BY FRAGMENTING WITHOUT THE GUC: free ALTERNATE whole groups, which fragments by non-adjacency and leaves coalescing at its default. Re-run here on the same mutated .so (b21ed48bc490, the #84 fix removed): alternate groups, coalesce=on free list 15, rewrote 15 CLEAN contiguous block, coalesce=off free list 18, rewrote 12 tuple already updated by self MY FIRST VERSION OF THAT CELL REWROTE 0 GROUPS and I nearly reported it. After `compact('t')` nothing exceeds the 2% dead fraction, so `compact_rewrite` returned 0 -- an allocation that never happened cannot self-conflict, so "clean" meant nothing. Dirtying the surviving groups first makes it rewrite 15, which is the cell that carries the argument. So coalesce=off is necessary and sufficient, and the free-list count is a property of the fixture rather than the thing that arms the suite. THE PREMISE NOW ASSERTS WHAT IT CLAIMS. It reads the GUC back from the SERVER, because the value lives in the cluster config and a conf line that stops taking effect returns this suite to exactly the state #1138 is about. The free-list count is printed instead of asserted. premise: coalescing is off, which is what lets this suite reach #84 Removal proof: set it back to the shipped default and that arm alone reddens, `got [on] want [off]`, 13 passed + 1 failed. native_reclaim_cycles 14 passed + 0 failed, PG17 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XiFn3HteTXnGdRiA2xDP2n --- CHANGELOG.md | 39 ++++++++++++++------------ test/native_reclaim_cycles.sh | 52 ++++++++++++++++++++++------------- 2 files changed, 55 insertions(+), 36 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b25680f0..8954e26d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -117,23 +117,28 @@ true until the next version shipped. It is the declared regression guard for #84. Deleting the #84 fix left it reporting `12 passed + 0 failed`, arm for arm, including the arm named after the defect. - #84 needs one command to allocate from the free list more than once. - `pgcolumnar.reclaim_coalesce` defaults on, so compaction merges adjacent freed - ranges and the free list holds one or two rows however much is freed. Measured on - the old fixture, `free_space` rows before each `compact_rewrite`: - - | cycle | 1 | 2 | 3 | 4 | 5 | - | --- | ---: | ---: | ---: | ---: | ---: | - | free rows | 0 | 1 | 2 | 2 | 2 | - - One row is not two allocations, so the just-consumed row was never re-selected and - the missing `CommandCounterIncrement` cost nothing observable. - - The suite now runs with coalescing off and frees a contiguous block of whole groups. - It **asserts the free list is fragmented before relying on it**: 18 rows, against a - floor of 5. With the #84 fix removed it now reddens five arms with - `ERROR: tuple already updated by self`, on the same binary the old fixture passed - on. Found by @OffgridwithJD, whose pytest twin already had the fixture. +`pgcolumnar.reclaim_coalesce` defaults on, and it does two things: it merges + adjacent freed ranges, and it carries its own `CommandCounterIncrement` on the free + path. That second one does the visibility work the #84 fix would otherwise do, so + with coalescing on the defect is masked. + + Isolated by freeing alternate whole groups, which fragments by non-adjacency and + leaves the option at its default. On the same mutated build: + + | fixture | coalesce | free list | rewrote | result | + | --- | --- | ---: | ---: | --- | + | alternate groups | on | 15 | 15 | clean | + | contiguous block | off | 18 | 12 | `tuple already updated by self` | + + Fifteen fragments with coalescing on does not reach it. The option is necessary and + sufficient, and the free-list count is a property of the fixture. The suite runs with coalescing + off and **asserts that, read back from the server**. The value is set in the cluster + config. A conf line that stops taking effect returns the suite to the state this + issue is about. With the #84 fix removed it reddens five + arms with `ERROR: tuple already updated by self`. + + Found by @OffgridwithJD, whose pytest twin already had the fixture and who then + separated the two factors. - A pytest run killed with `SIGTERM` leaked its throwaway cluster (#1170). diff --git a/test/native_reclaim_cycles.sh b/test/native_reclaim_cycles.sh index f66174d9..affb7865 100755 --- a/test/native_reclaim_cycles.sh +++ b/test/native_reclaim_cycles.sh @@ -22,20 +22,30 @@ set -uo pipefail -# COALESCING OFF, OR THIS SUITE CANNOT REACH THE DEFECT IT GUARDS (#1138). #84 -# needs ONE COMMAND to allocate from the free list MORE THAN ONCE. -# `pgcolumnar.reclaim_coalesce` defaults ON, and compaction then merges adjacent -# freed ranges, so the free list holds one or two rows however much is freed. -# Measured on the old fixture, free_space rows before each compact_rewrite: +# COALESCING OFF, OR THIS SUITE CANNOT REACH THE DEFECT IT GUARDS (#1138). +# Delete the #84 fix, rebuild, and the old fixture reported 12 passed + 0 failed, +# arm for arm -- including `compact_rewrite cycle N returns a count (no +# self-conflict)`, the arm named after the defect. Found by @OffgridwithJD. # -# cycle 1 2 3 4 5 -# free rows 0 1 2 2 2 +# THE OPERATIVE PROPERTY IS THE GUC, NOT A FRAGMENTED FREE LIST, and the first +# version of this comment said the opposite. `reclaim_coalesce` does two things: +# it merges adjacent freed ranges, AND it carries its own CommandCounterIncrement +# on the free path (columnar_metadata.c:792, `if (pgcolumnar_reclaim_coalesce)`). +# That second one does the visibility work the #84 fix would otherwise do, so +# with coalescing on the defect is masked however fragmented the list is. # -# One row is not two allocations, so the just-consumed row was never re-selected -# and the missing CommandCounterIncrement cost nothing observable. Delete the #84 -# fix, rebuild, and the old suite reported 12 passed + 0 failed, arm for arm -- -# including `compact_rewrite cycle N returns a count (no self-conflict)`, the arm -# named after the defect. Found by @OffgridwithJD. +# Isolated by freeing ALTERNATE whole groups, which fragments by non-adjacency +# and leaves the GUC at its default. On the same mutated .so: +# +# alternate groups, coalesce=on free list 15, rewrote 15 CLEAN +# contiguous block, coalesce=off free list 18, rewrote 12 tuple already +# updated by self +# +# Fifteen fragments with coalescing on does not reach it. So coalesce=off is +# necessary and sufficient, and the free-list count is a property of the fixture +# rather than the thing that arms the suite. @OffgridwithJD separated the two +# factors; the first version of this comment would have told the next maintainer +# to preserve the wrong one. # # IN THE CLUSTER CONFIG, NOT A `SET`. Every psql_run here is its own session, so a # SET would last exactly one statement and the writing session would not have it. @@ -81,14 +91,18 @@ psql_run "DELETE FROM h WHERE id BETWEEN $DEL_LO AND $DEL_HI;" psql_run "DELETE FROM n WHERE id BETWEEN $DEL_LO AND $DEL_HI;" psql_run "SELECT pgcolumnar.compact('n');" _free="$(free_rows)" -echo " (free_space rows after the block delete: $_free)" -# THE PRECONDITION FOR #84, ASSERTED RATHER THAN HOPED FOR. With a free list of -# one row -- which is what coalescing produces -- no command allocates from it -# twice, and every arm below passes on a build with the fix removed. -check "premise: the free list is fragmented, so one command allocates from it more than once" \ - "$([ "${_free:-0}" -ge 5 ] && echo "fragmented ($_free)" || echo "TOO FEW ($_free)")" \ - "fragmented ($_free)" +# THE PRECONDITION FOR #84, ASSERTED RATHER THAN HOPED FOR, AND IT IS THE GUC. +# Read back from the SERVER, because the value is set in the cluster config: a +# conf line that stops taking effect leaves every arm below passing on a build +# with the fix removed, which is the state this suite was in before #1138. +check "premise: coalescing is off, which is what lets this suite reach #84" \ + "$(q "SHOW pgcolumnar.reclaim_coalesce;")" "off" + +# The free list's shape, PRINTED rather than asserted. It is a property of the +# fixture and not what arms the suite: 15 fragments with coalescing ON do not +# reach the defect. +echo " (free_space rows after the block delete: $_free)" check "initial parity" "$(hash_n)" "$(hash_h)" From bf254a9f7516183c723a4ee5848b4adad6077bfb Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Tue, 22 Sep 2026 11:19:01 -0600 Subject: [PATCH 3/3] test: the pytest twin still taught the mechanism I corrected (#1138) The parity grader refused the pair and it was right: I corrected the shell side and left the port encoding the account @OffgridwithJD disproved. FAILED test_compare_to_bash.py::test_the_ported_suites_in_this_tree_are_graded_one_for_one got ... native_reclaim_cycles=1 ... want ... native_reclaim_cycles=0 ... Every other pair was 0, and the two cluster legs were the SAME failure re-reported: test_the_guard_half_of_the_corpus_runs_without_a_database_driver runs the guard half in a subprocess and asserts zero failures. WHAT THE GRADER CAUGHT IS A DIVERGENCE IN MEANING, NOT IN NAMES. The twin set the GUC and never asserted it took, and still asserted `premise: the free list is fragmented, so one command allocates from it more than once` as the property that arms it, with a docstring giving the fragmentation account. Had the guard only compared counts, the twin would have kept teaching the wrong mechanism while grading 0. The twin now makes the same two moves the shell side made: it asserts the option READ BACK FROM THE SERVER under the same name, and prints the free-list count. literal matches: 4 | template matches: 2 | missing: 0 VERDICT: every bash property is covered REMOVAL PROOF, because `missing: 0` proves the name is in the TEXT and not that anything records under it. Delete the `SET` and nothing else: AssertionError: premise: coalescing is off, which is what lets this suite reach #84: got 'on' want 'off' Mutant asserted to still parse, source restored byte-identical (6001df91aa7d), __pycache__ cleared on both sides. pytest guard leg 398 passed, 1100 checks, 0 fail the twin on PG17 15 checks, 1 passed `cluster_tests` does not move: the file gains arms, not test functions. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01XiFn3HteTXnGdRiA2xDP2n --- CHANGELOG.md | 5 ++ test/pytest/test_native_reclaim_cycles.py | 58 +++++++++++++---------- 2 files changed, 38 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8954e26d..e135dc82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -137,6 +137,11 @@ true until the next version shipped. issue is about. With the #84 fix removed it reddens five arms with `ERROR: tuple already updated by self`. + The pytest twin says the same thing under the same name. It asserts the option read + back from the server, rather than assumed from the `SET` that asked for it. The parity + grader is what caught the divergence. It graded the pair `1` while every other pair + was `0`, on a difference of meaning rather than of counts. + Found by @OffgridwithJD, whose pytest twin already had the fixture and who then separated the two factors. diff --git a/test/pytest/test_native_reclaim_cycles.py b/test/pytest/test_native_reclaim_cycles.py index 73f6e4e5..265c2994 100644 --- a/test/pytest/test_native_reclaim_cycles.py +++ b/test/pytest/test_native_reclaim_cycles.py @@ -16,27 +16,31 @@ That keeps the two harnesses independent by construction, and a failure prints the rows that differ instead of two unequal hashes. -THE FIXTURE IS FRAGMENTED ON PURPOSE, AND THAT IS THE WHOLE DIFFERENCE. The shell -suite's fixture cannot reach the defect it is the regression guard for. Measured: -delete the #84 fix, and the shell suite reports 12 passed / 0 failed, unchanged. - -The reason is `pgcolumnar.reclaim_coalesce`, which defaults ON. Compaction then merges -adjacent freed ranges, so the free list holds one or two rows however much is freed, -one command allocates from it at most once, and the just-consumed row is never -re-selected. Measured on the shell suite's own fixture, per cycle: - - free_space rows before compact_rewrite: 0, 1, 2, 2, 2 - -With coalescing OFF the same workload keeps the ranges separate and the precondition -holds. Two cells, each built from its own source and printing its own `.so` hash: - - fix present (.so e95880e45673) 3 cycles, no error, free list steady at 18 - fix removed (.so 42bb17933a55) first compact_rewrite raises - "tuple already updated by self" - -So this file runs its cycles with coalescing off, and asserts the free list is -actually fragmented before relying on it. That premise is what keeps the suite from -going quietly vacuous again if the allocator's shape changes. +COALESCING OFF IS THE WHOLE DIFFERENCE, AND IT IS NOT THE FRAGMENTATION. Delete the +#84 fix and the old shell fixture reported 12 passed / 0 failed, unchanged. An earlier +version of this docstring said a one-row free list was the reason, and that account is +wrong. + +`pgcolumnar.reclaim_coalesce` does TWO things. It merges adjacent freed ranges, and it +carries its own `CommandCounterIncrement` on the free path -- `columnar_metadata.c:792`, +guarded by `if (pgcolumnar_reclaim_coalesce)`. That second one does the visibility work +the #84 fix would otherwise do, so with coalescing on the defect is masked however +fragmented the list is. + +Isolated by freeing ALTERNATE whole groups, which fragments by non-adjacency and leaves +the option at its default. On the same mutated build: + + alternate groups, coalesce=on free list 15, rewrote 15 CLEAN + contiguous block, coalesce=off free list 18, rewrote 12 tuple already + updated by self + +Fifteen fragments with coalescing on does not reach it. So the option is necessary and +sufficient, and the free-list count is a property of the fixture rather than the thing +that arms the suite. + +So this file runs its cycles with coalescing off and asserts THAT, read back from the +server rather than assumed from the `SET` that asked for it. The free-list count is +printed. The shell twin asserts the same property under the same name. """ import psycopg @@ -106,10 +110,14 @@ def test_native_reclaim_cycles(pgc_conn, expect): # THE PRECONDITION FOR #84, ASSERTED RATHER THAN HOPED FOR. With a free list of # one row -- which is what coalescing produces -- no command allocates from it # twice and every arm below passes on a build with the fix removed. - expect.at_least( - free, 5, - "premise: the free list is fragmented, so one command allocates from it " - "more than once", + # THE PRECONDITION FOR #84, AND IT IS THE OPTION. Read back from the server, + # because a `SET` that silently stopped applying leaves every arm below passing + # on a build with the fix removed -- which is the state this suite was in before + # #1138. The free-list count above is printed rather than asserted: 15 fragments + # with coalescing ON do not reach the defect. + expect.text( + _one(pgc_conn, "SHOW pgcolumnar.reclaim_coalesce"), "off", + "premise: coalescing is off, which is what lets this suite reach #84", ) expect.rows(_rows(pgc_conn, "n"), _rows(pgc_conn, "h"), "initial parity")