test: derive the wide-table ANALYZE bound from the run and floor it (#1252)

[OffgridwithJD]

Closes #1252.

What

The arm read an_ms < scan_ms * 20. It went red on a CI runner against #1251 --
five test and docs files that cannot reach the ANALYZE path -- and passed when
the same jobs were re-run at the same commit.

Both halves of that ratio were wrong, measured

pgcolumnar-audit, pg17a, PostgreSQL 17.6, five interleaved rounds, fixture
rebuilt each round with its row count asserted at 40000 before any timing
prints. empty is the same psql_run wrapper around SELECT 1, so it is the
floor every other column sits on:

  round    empty    insert   scan     analyze
  1        8        328      20       116
  2        7        294      20       109
  3        9        320      21       117
  4        8        319      19       107
  5        7        322      20       127

  mean     7.8      316.6    20.0     115.2

  scan     ~12.2 ms of work above the floor   ->  1.5x its own noise
  insert  ~308.8 ms of work above the floor   ->   40x it

The scan is 39% process startup, so the ratio measured the floor as much as
the scan. And the two terms did not move together when the machine changed:

                this box     the failing CI run
  analyze        115 ms           353 ms          3.1x
  scan            20 ms            14 ms          0.7x

The bound

  cap_ms = max(5000, insert_ms * 20)

The insert reads and writes the same rows the ANALYZE samples, so it tracks the
machine and the data volume. The floor stops a future faster write path
squeezing the bound into a flake -- which is exactly how the old one arrived
here: its 20x was justified in the file against "341 ms against a 155 ms
scan", and no machine has measured a 155 ms scan since (mine 20, CI 14, #1091's
four majors 22-33).

The bug is a hang, not a slowdown, so the bound does not need to be tight.
Removal proof against f82bdcd's parent, same fixture, each tree building its
own .so: the unfixed build did not complete within 180 s -- an unbounded
run was killed at 10m52s of 99.9% CPU, still running -- against 206 ms
fixed. At least 3000x.

Real run on this branch: ANALYZE 121 ms against a 6340 ms cap (insert 317 ms),
a 52x margin against the old frame's real 3.5x.

The decision is a function, driven with literals

So the branch a healthy machine never takes is judged here rather than by a box
that happens to be slow. Six arms drive pgc_analyze_cap_ms directly, including
both sides of the tie: 250 -> 5000 (floor wins) and 251 -> 5020 (derived
wins).

Four mutations, and A and B are disjoint

Worth stating, because the last two changes I shipped had mutations that were
not:

  A  the floor removed          3 arms x 5 majors
  B  the scaling removed        2 arms x 5 majors   disjoint from A
  C  the input guard removed    1 arm  x 5 majors   isolates what A also hits
  D  the floor moved by one     4 arms x 5 majors   the only one reaching the tie

C reddened nothing on its first run

The guard was uncovered, not redundant -- #1236's taxonomy, and the
measurement is what chose between the two readings. 0, "", "abc" and a
negative all reach the floor anyway, because $(( )) reads them as 0. The input
it actually defends is a partially numeric token:

  pgc_analyze_cap_ms 12abc     with the guard  5000
                               without it      arithmetic error, no output

So an arm was added to cover it rather than the guard deleted.

analyze_stats enters the mutation ledger, which is most of this diff

It had zero rows on main. Recording that the new arms redden means merging a
run of the suite they live in, and the tool refuses to attribute a mutation
across partial or multiple runs -- so registering all 40 of its checks is the
mechanism, not scope added on top.

  ledger rows   1700 + 40 = 1740
  never         1540 + 34 = 1574     32 pre-existing arms + 2 of the new 8
  dated          160 +  6 =  166     the 6 arms the four mutations redden
  1574 + 166 == 1740

  suites_not_covered  249 -> 248     20 suites covered before, 21 now

suites_not_covered falls by exactly one, which is the direction that ceiling
is allowed to move.

Two arms arrive never, and I will not pretend otherwise

• premise: the cap is exposed as a function is a type -t check, and every
  mutation replaces a body while keeping the name.
• ANALYZE on a wide table completes well inside its derived cap has a
  removal proof rather than a mutation -- the unfixed build, which is a
  different tree. Not expressible as a same-tree mutation, so the row stays
  never rather than being dated on evidence of another kind.

Verified

  analyze_stats       40 checks  PASSED  pg17a / pg18a
  harness_selftest  1198 checks  PASSED  pg17a / pg18a
  docs_style          55 checks  PASSED  pg17a / pg18a
  rc=0, zero FAIL rows throughout

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP

[jdatcmd]

I drove pgc_analyze_cap_ms rather than reading it, and every arithmetic
claim in the file is exact. Holding my approval only until suites (PG 17)
and suites (PG 18) land — they are the two that actually run this suite, and
13 of 15 are green.

INPUT        GUARDED    UNGUARDED
[1]          5000       5000
[250]        5000       5000
[251]        5020       5020
[1000]       20000      20000
[0]          5000       5000
[]           5000       5000
[abc]        5000       5000
[12abc]      5000       <none>
[-5]         5000       5000

12abc is the only ordinary input where the two differ, exactly as your comment
says, and the crossover is where the arms put it: 250*20 = 5000 is not greater
than the floor, 251*20 = 5020 is. The boundary pair is doing real work.

One input your comment does not name, and it widens your own case

[99999999999999999999]   guarded 5000   unguarded 7751640039368425452

The guard also catches integer overflow, not only a partially numeric token.
$(( )) wraps silently and hands back a positive number that sails past the
floor, so the unguarded function would return a 7.7-quintillion-millisecond cap
— a bound that can never fail. That is a worse failure than the arithmetic error
you documented, because the arithmetic error is loud and this one looks like a
passing test.

No insert will ever take 10^20 ms, so this changes nothing operationally. It is
worth a clause in the comment because "the one input it actually defends is a
PARTIALLY numeric token" is the sentence a later reader would use to justify
deleting the guard, and it is not quite true.

The assumption I would watch, which is not a blocker

The change rests on insert time tracking ANALYZE time across machines. Your own
numbers are the argument for it and also the warning: between your box and that
CI runner, ANALYZE went 3.1x and the scan went 0.7x. That is a measured
divergence for the scan and an argued convergence for the insert — you have
local insert numbers, not CI ones. The reasoning is sound (same rows read and
written, same hardware) and I would not hold the PR for it.

What makes me comfortable is the floor rather than the ratio. With ANALYZE at
121 ms against a 5000 ms floor there is about 41x of headroom before the derived
term matters at all, so on any machine near this one the guard is the floor and
the tracking assumption is not load-bearing yet. If a future write path makes
inserts much faster, the floor keeps holding while the derived term goes away —
which is the failure direction you want.

Two small things

c > 0 in the awk is now dead. cap_ms cannot be less than 5000, so that
conjunct is unreachable-false. It was meaningful in the old form, where scan_ms
really could be 0. Harmless, but by the argument you just made for the input
guard, a condition that cannot fire either gets covered or gets removed — and
this one cannot be covered.

_as_t0 starts before the DROP/CREATE, not at the INSERT. So ins_ms
is a superset of the insert. That inflates the cap, which is the safe direction,
and the name is the only thing that is slightly off.

On the ledger arithmetic

1700 + 40 = 1740, 1540 + 34 = 1574, 160 + 6 = 166, and 1574 + 166 = 1740.
That reconciles against main at 345823d9, which I recounted after merging
#1257: 1700 rows = 1540 never + 160 dated. Merging a whole run because the tool
refuses partial ones is the right call — and saying plainly that two arms arrive
never, with which two and why, is the part that makes the number readable.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XiFn3HteTXnGdRiA2xDP2n

[jdatcmd]

Approving. suites (PG 17) and suites (PG 18) are both green, which were the
two I was waiting on — 15 of 15, and the check-name set is identical to the last
merged PR, so no job was added and silently skipped.

The bound comes out of the run, the floor stops it tightening as the baseline
improves, and the boundary pair (250 -> 5000, 251 -> 5020) drives the tie
with literals rather than waiting for a slow machine to wander into it. I
verified the whole table by running the function rather than reading it.

The overflow note in my comment is a comment-accuracy point, not a blocker: the
guard defends more than the partially numeric token you documented, and the
sentence naming that token as "the one input it actually defends" is the one a
later reader would use to justify deleting it. Worth a clause next time this
file is open; not worth another round now.

[OffgridwithJD]

On whether the insert and the ANALYZE really track together — I tried to test it, the instrument failed, and the assumption turns out not to be load-bearing

@jdatcmd named the right thing to attack: the insert tracking the ANALYZE across
machines is the assumption the derived half rests on, and my own numbers show
the old pair moving independently by a factor of 4 between this box and CI.

What I tried, and why it proved nothing

A controlled slowdown on one box: same data, same code, different speed. Three
CPU budgets through the container:

  allowance    empty  insert  scan  analyze
  8 cores (none)   8     297    21      113
  40%              7     303    21      125
  20%              8     309    20      114

Identical. That is not a result, it is a broken instrument, and the control
says so — a fixed CPU spin, timed at each setting:

  8 cores, no allowance    226 ms
  1 core,  no allowance    222 ms
  1 core,  25% allowance   225 ms
  1 core,  10% allowance   225 ms

Flat. The throttle never bit, so every row above was taken at full speed. Had I
not timed the spin I would have published "the terms track together across a 5x
slowdown" from three measurements of the same machine. Config restored to
limits.cpu=8, spin back at 237 ms.

And CI cannot supply the number either. The suite prints
-- wide-table ANALYZE ... (insert ...), but the harness suppresses per-suite
detail on a PASS: I pulled both suites (PG 17) and suites (PG 18) from this
PR's green run and the line is absent from all 273 KB.

So the assumption is untested across machines. I am not going to claim
otherwise.

Why it is not load-bearing anyway

The question that matters is not whether the cap is accurate but whether it
can flake, and the floor bounds that direction regardless of how the terms
diverge.

  old:  flake when  an_ms >= scan_ms * 20
        CI scan 14 ms  ->  threshold  280 ms,  CI analyze was 353 ms  ->  RED

  new:  flake when  an_ms >= max(5000, ins_ms * 20)
        floor alone    ->  threshold 5000 ms,  CI analyze  353 ms  ->  14.2x headroom

If the insert diverges downward relative to ANALYZE -- the case
@jdatcmd is worried about -- the derived term shrinks, the floor takes over, and
the bound stops at 5000 ms. If it diverges upward, the cap grows, which is
safe. Divergence can only make the guard more permissive, never flakier.

That is the answer to the challenge: the assumption affects how tight the
bound is on a slow machine, not whether it produces a false red. The old frame
had no such bound and was already inside its flaking range on CI at 353 against
280.

What would settle it, for the record

Once this lands, the first CI run where analyze_stats fails for any reason
prints insert beside analyze from a different machine, which is the
cross-machine pair I could not get today. That is a worse instrument than a
designed experiment and I would rather name it than imply I have the number.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP