test: port analyze_reltuples, projection_update and projection_drop_column

CHANGELOG.md

@@ -18,6 +18,49 @@ true until the next version shipped.
 
 ### Fixed
 
+- Three suites ported to pytest, and the queue re-derived (#432).
+
+  `analyze_reltuples`, `projection_update` and `projection_drop_column`, 21 names,
+  each graded `missing: 0` by `compare_to_bash.py`. Ports take 17 to 20 of the
+  corpus, and the batch is one pull request rather than three because the
+  per-PR cost -- the census, the counts, TESTS.md and its anchor -- is paid once
+  per review and not once per suite.
+
+  EACH PORT ASSERTS SOMETHING ITS BASH ORIGINAL DOES NOT, which is the point of
+  porting rather than translating:
+
+      analyze_reltuples        the helper builds a columnar and a heap table from
+                               two separate inserts and compares their row
+                               estimates without checking they hold the same rows.
+                               Both counts are asserted now. And the arm named
+                               `20 stripes: within 5% of actual` never read the
+                               geometry -- a single-group table passes that 5%
+                               check too -- so the port reads pgcolumnar.row_group.
+
+      projection_update        an UPDATE whose WHERE matched nothing leaves both
+                               sides identical and every arm below it green,
+                               having exercised no fan-out at all. The affected
+                               row counts are pinned.
+
+      projection_drop_column   the two non-owner arms exist to show a stranger
+                               cannot tell a projected column from an unprojected
+                               one. The bash suite asserts each against the
+                               literal 42501 and leaves the reader to notice they
+                               match; the port compares them to each other.
+
+  THE SECOND ONE CAUGHT MY OWN PREMISE. I first asserted the non-owner could read
+  the table, and the run returned 42501: USAGE on the schema resolves the NAME,
+  while SELECT is a separate grant. The property the arms below it need is only
+  that the name is not invisible, because 42P01 would mean they were asserting
+  ownership against a table the role cannot see. Corrected to test for that.
+
+  Row sets are compared as sorted tuples in Python rather than through
+  `pgc_set_hash`, so the two harnesses stay independent by construction and a
+  failure prints the rows that differ instead of two unequal hashes.
+
+  No bash suite changes, so no ledger row moves and the census does not.
+  `cluster_tests` 418 -> 421, re-derived by collection.
+
 - A covering projection was priced by clauses that merely mention its sort key,
   rather than by clauses it can prune on (#1126, the remainder of #1107).
 

test/pytest/TESTS.md

@@ -98,6 +98,9 @@ behaviour, the source of that number is named.
 - [50. test_collation_pinned.py: comm's inputs must be sorted the same way](#50-test_collation_pinnedpy-comms-inputs-must-be-sorted-the-same-way)
 - [51. test_projection_scan_cost.py: a covering projection is not priced at half](#51-test_projection_scan_costpy-a-covering-projection-is-not-priced-at-half)
 - [52. test_record_names_its_major.py: a record must name its major](#52-test_record_names_its_majorpy-a-record-must-name-its-major)
+- [53. test_analyze_reltuples.py: ANALYZE must estimate the row count, not zero](#53-test_analyze_reltuplespy-analyze-must-estimate-the-row-count-not-zero)
+- [54. test_projection_update.py: UPDATE must fan the new row number out to projections](#54-test_projection_updatepy-update-must-fan-the-new-row-number-out-to-projections)
+- [55. test_projection_drop_column.py: DROP COLUMN must not invalidate a projection](#55-test_projection_drop_columnpy-drop-column-must-not-invalidate-a-projection)
 
 ## 1. How to read a test in here
 
@@ -4639,3 +4642,70 @@ The load-bearing assertion is the last one's bound. The runner has many later
 `verfail=1` lines, so an unbounded search finds one whatever the guard does --
 which is exactly how the shell twin's first version of that arm stayed green
 against the line removed. Mutation testing caught it.
+
+## 53. test_analyze_reltuples.py: ANALYZE must estimate the row count, not zero
+
+Port of `analyze_reltuples.sh` (#432). A block was mapped to its row group by comparing
+offsets that had `COLUMNAR_FIRST_LOGICAL_OFFSET` subtracted from one side and not the
+other, so `reltuples` came back 0 for a 10,000-row table and the planner believed every
+columnar table smaller than one stripe was empty.
+
+heap is the oracle: ANALYZE samples, so the columnar estimate is not required to be exact,
+only as close as heap's on the same data.
+
+Two assertions the bash suite does not make. Its helper builds the columnar and heap
+tables with separate inserts and compares their estimates without checking they hold the
+same rows, so this port asserts both counts. And the arm named `20 stripes: within 5% of
+actual` sets `pgcolumnar.stripe_row_limit` but checks only the estimate, which a
+single-group table also passes; this port reads `pgcolumnar.row_group` and fails first if
+the fixture is not the shape the name claims.
+
+### Every arm
+
+| test | what it holds |
+| --- | --- |
+| `test_analyze_estimates_the_row_count` | every arm, from the four paired columnar/heap sizes through the multi-stripe geometry, the delete, and the clustered `n_distinct` |
+
+## 54. test_projection_update.py: UPDATE must fan the new row number out to projections
+
+Port of `projection_update.sh` (#432). UPDATE is delete-old plus insert-new, and the
+insert half used to skip fan-out, so `read_projection` returned no rows and a covering
+projection scan answered as if the updated rows had been deleted.
+
+Row sets are compared as sorted tuples in Python rather than through `pgc_set_hash`. That
+keeps the two harnesses independent by construction, and a failure prints the rows that
+differ instead of two unequal hashes.
+
+The port also pins the affected row counts. An UPDATE whose WHERE matched nothing leaves
+both sides identical and every arm below it green, having exercised no fan-out at all.
+
+### Every arm
+
+| test | what it holds |
+| --- | --- |
+| `test_update_fans_the_new_row_number_out_to_projections` | every arm: the covering scan before and after updating a projected column, then a non-projected one, with `read_projection` and `reconstruct_via_projection` |
+
+## 55. test_projection_drop_column.py: DROP COLUMN must not invalidate a projection
+
+Port of `projection_drop_column.sh` (#432). Projections are extension metadata rather
+than `pg_depend` objects, so PostgreSQL accepted `DROP COLUMN` on a projected column and
+the next INSERT failed in `lookup_type_cache`.
+
+SQLSTATE rather than message text: `2BP01` for the dependency refusal and `42501` for the
+non-owner. The bash suite gets there by running `psql` with `VERBOSITY verbose` and
+extracting the code with `sed`; the port reads `exc.sqlstate`.
+
+`SET ROLE` rather than a login role, which is the opposite of
+`test_projection_privilege.py` and for the reason that file gives: real logins are needed
+when the ACL layers are the subject, and only add ways to fail when ownership is.
+
+The two non-owner arms exist to show a stranger cannot tell a projected column from an
+unprojected one. The bash suite asserts each against the literal `42501`; the port also
+compares them to each other, so indistinguishability is asserted rather than implied.
+
+### Every arm
+
+| test | what it holds |
+| --- | --- |
+| `test_drop_column_is_refused_while_a_projection_depends_on_it` | every arm: the two non-owner refusals, the dependency refusal, writability afterwards, the unrelated column, and the partitioned parent |
+

test/pytest/expected_tests.txt

@@ -293,4 +293,8 @@ guard_tests 380
 # After rebase onto origin/main (ab8feef), re-derived by collection on this
 # tree, not by keeping the auto-merged 418: guard_tests 374 tests collected;
 # cluster_tests 418 tests collected. guard_tests did not move.
-cluster_tests 418
+# 418 -> 421 with the three ports in #432's analyze/projection batch:
+# test_analyze_reltuples.py, test_projection_update.py and
+# test_projection_drop_column.py, one collected test each. Re-derived BY COLLECTION on
+# this tree, never by adding three.
+cluster_tests 421

test/pytest/test_analyze_reltuples.py

@@ -0,0 +1,134 @@
+"""ANALYZE must estimate the row count, not zero (#432, port of analyze_reltuples.sh).
+
+A block was mapped to its row group by comparing the block's logical offset against
+the group's `file_offset`, but the block's offset had `COLUMNAR_FIRST_LOGICAL_OFFSET`
+subtracted from it while the group's had not. Every block compared two blocks low, so
+a table whose only row group starts at the beginning matched no block at all and
+`reltuples` came back 0 for 10,000 rows.
+
+Zero is the sharp case and this file is built around it: an estimate that is merely
+imprecise costs a plan, an estimate of zero costs the shape of every plan that joins
+the table.
+
+**heap is the oracle.** ANALYZE samples, so the columnar estimate is not required to
+be exact. It is required to be as close as heap's is on the same data, which at these
+sizes is exact.
+
+TWO THINGS THIS PORT ASSERTS THAT THE BASH SUITE DOES NOT.
+
+**The geometry the arm is named after.** `20 stripes: within 5% of actual` sets
+`pgcolumnar.stripe_row_limit` and then checks only the estimate. The limit does take
+effect today -- measured, 20 row groups with it against 1 without -- but nothing in
+that arm would notice if it stopped, and a single-group table also passes a 5% check.
+The premise here reads `pgcolumnar.row_group` and fails first if the fixture is not
+the shape the name claims.
+
+**That the two access methods hold the same data.** The bash helper builds the
+columnar and heap tables from two separate `INSERT`s and compares their estimates. If
+one insert silently wrote fewer rows the comparison still runs, against a different
+population. A count premise makes that a failure rather than a quiet change of
+subject.
+"""
+
+
+def _reltuples(cur, table):
+    cur.execute(f"SELECT reltuples::bigint FROM pg_class WHERE oid = '{table}'::regclass")
+    return cur.fetchone()[0]
+
+
+def _rows(cur, table):
+    cur.execute(f"SELECT count(*) FROM {table}")
+    return cur.fetchone()[0]
+
+
+def _groups(cur, table):
+    cur.execute(
+        "SELECT count(*) FROM pgcolumnar.row_group "
+        f"WHERE storage_id = pgcolumnar.get_storage_id('{table}')"
+    )
+    return cur.fetchone()[0]
+
+
+def _closeness(got, want, tol):
+    """-> "close", or how far off, so a failure names the distance like the bash suite."""
+    d = abs(got - want)
+    return "close" if d <= tol else f"off by {d}"
+
+
+def _both(cur, expect, lab, n, ddl, ex):
+    """Build the same data in both access methods and compare the estimate.
+
+    Separate statements per table, as in the bash suite, so a failure building one
+    side cannot stop the other from being built and leave the comparison silently
+    one-sided.
+    """
+    cur.execute(f"DROP TABLE IF EXISTS ar_c; CREATE TABLE ar_c ({ddl}) USING pgcolumnar")
+    cur.execute(f"INSERT INTO ar_c SELECT {ex} FROM generate_series(1,{n}) g")
+    cur.execute(f"DROP TABLE IF EXISTS ar_h; CREATE TABLE ar_h ({ddl})")
+    cur.execute(f"INSERT INTO ar_h SELECT {ex} FROM generate_series(1,{n}) g")
+    cur.execute("ANALYZE ar_c")
+    cur.execute("ANALYZE ar_h")
+
+    # EXTRA, not in the bash suite: the two sides must hold the same population, or
+    # the comparison below is between two different tables.
+    expect.num(_rows(cur, "ar_c"), n, f"premise: {lab} loaded every columnar row")
+    expect.num(_rows(cur, "ar_h"), n, f"premise: {lab} loaded every heap row")
+
+    c = _reltuples(cur, "ar_c")
+    h = _reltuples(cur, "ar_h")
+
+    expect.text("nonzero" if c > 0 else "ZERO", "nonzero",
+                f"{lab}: estimate is not zero")
+    expect.text(_closeness(c, n, n * 0.05), "close",
+                f"{lab}: within 5% of actual (heap says {h})")
+
+
+def test_analyze_estimates_the_row_count(pgc_conn, expect):
+    with pgc_conn.cursor() as cur:
+        # A single row group starting at the first logical offset is the case that
+        # reported zero; the larger sizes cover the partial shift.
+        _both(cur, expect, "10k rows", 10000, "id int, v int", "g, g*2")
+        _both(cur, expect, "50k rows", 50000, "id int, v int", "g, g*2")
+        _both(cur, expect, "200k rows", 200000, "id int, v int", "g, g*2")
+        _both(cur, expect, "text column", 50000, "id int, v text", "g, 'x' || g")
+
+        # Several stripes, so more than one group has to be mapped.
+        #
+        # RESET AFTERWARDS, which the bash suite gets for free and a port does not.
+        # Each `psql_run` there is its own session, so the GUC dies with it. Here one
+        # connection carries every arm, and leaving the limit set would silently
+        # change the geometry of every table built below it.
+        cur.execute("SET pgcolumnar.stripe_row_limit = 1000")
+        cur.execute("DROP TABLE IF EXISTS ar_s; CREATE TABLE ar_s (id int, v int) USING pgcolumnar")
+        cur.execute("INSERT INTO ar_s SELECT g, g FROM generate_series(1,20000) g")
+        cur.execute("ANALYZE ar_s")
+
+        # EXTRA: the arm below is named for a geometry it never checks.
+        expect.num(_groups(cur, "ar_s"), 20,
+                   "premise: the multi-stripe fixture really has twenty row groups")
+
+        expect.text(_closeness(_reltuples(cur, "ar_s"), 20000, 1000), "close",
+                    "20 stripes: within 5% of actual")
+
+        # The estimate has to follow deletes down, not just up.
+        cur.execute("DELETE FROM ar_s WHERE id % 2 = 0")
+        cur.execute("ANALYZE ar_s")
+        expect.text(_closeness(_reltuples(cur, "ar_s"), 10000, 500), "close",
+                    "after deleting half, the estimate follows")
+
+        # The sampling quality this mapping exists to protect must survive the fix: a
+        # clustered column still has to report its true n_distinct rather than the
+        # per-group count, which is what whole-group sampling would give.
+        cur.execute("DROP TABLE IF EXISTS ar_n; CREATE TABLE ar_n (id int, k int) USING pgcolumnar")
+        cur.execute("INSERT INTO ar_n SELECT g, g / 20 FROM generate_series(1,20000) g")
+        cur.execute("ANALYZE ar_n")
+        cur.execute("RESET pgcolumnar.stripe_row_limit")
+
+        cur.execute(
+            "SELECT CASE WHEN n_distinct < 0 THEN 'ratio' "
+            "WHEN n_distinct > 500 THEN 'many' ELSE 'few:' || n_distinct END "
+            "FROM pg_stats WHERE tablename = 'ar_n' AND attname = 'k'"
+        )
+        row = cur.fetchone()
+        expect.text(row[0] if row else "no row in pg_stats", "many",
+                    "a clustered column still reports many distinct values")

test/pytest/test_compare_to_bash.py

@@ -85,9 +85,12 @@
 #
 # The shape is `SHELL_REFERENCES`' in `test_harness_deps.py`, asserted in both
 # directions for the same reason: a one-way list rots into a permanent exemption.
-COMPLETE = ["differential", "hilbert_cluster", "hilbert_locality",
+COMPLETE = ["analyze_reltuples",
+            "differential", "hilbert_cluster", "hilbert_locality",
             "native_chunk_length_bound", "native_fetch_coalesce", "native_ownership", "native_projection", "parallel_am_scan",
+            "projection_drop_column",
             "projection_privilege",
+            "projection_update",
             "projection_scan_cost",
             "projections",
             "sorted_pathkeys", "stats_privilege",