test: seed the mutation ledger with differential, 204 checks (#752)

[jdatcmd]

suites_not_covered          250 -> 249
checks_never_observed_red   913 -> 1117
covered                     harness_selftest, native_join_runtime_filter, differential

The ceiling has not moved in a week. Everything built this week — the census, the orphan scan, the prune refusal, the skip-loop sweep, the duplicate-key count — has been operating on 2 suites out of 252. This is the third.

Why this suite, measured rather than chosen by taste

product-central	heap-versus-columnar differential correctness, so a check that cannot fail there is a wrong answer nobody sees
churn	16 of the last 300 commits touch it, so the gate will actually fire
seedable	it emits RESULT records through lib.sh. Two registered suites do not and cannot be seeded as things stand — see the correction below
cheap	19 seconds
uncontended	no open PR touches it

The candidate set was computed, not eyeballed: 252 registered (from the runner's own SUITES array, expanded by bash rather than by a regex — my first three parses gave 267, 262 and 0), minus the 2 covered, minus the 9 that skip wholesale on PG17, minus the concurrency- and timing-shaped suites that would make the gate flaky.

The risk that decided it: name stability

A suite whose check names move between runs makes the ledger churn forever and the gate fire on nothing. Two consecutive runs on PG17:

204 records, 204 distinct names, 0 duplicate keys  -- both runs
the two name sets IDENTICAL

My first attempt at that comparison said three names differed, and it was wrong. I cut field 4 from every line instead of from the RESULT lines, so prose in the log was being compared as check names. Had I stopped there I would have rejected a suite that is in fact deterministic.

Correction: the unseedable set is TWO suites, not twelve

An earlier revision of this body said twelve registered suites "keep their own tally and cannot be seeded at all". That was wrong, and it was an inference from a label rather than a measurement. run_all_versions.sh prints by their own mechanism: audit bench_guards concurrency docs_style phase2..6 smoke unique_conc update_conc, and I read that as "emits no RESULT records". It does not mean that. It is the narrow-versus-wide accounting-reader distinction from #928 — which accounting line the suite prints — and says nothing about RESULT records.

Measured by running all twelve on PG17 and counting, after @OffgridwithJD challenged the claim:

audit          31      phase2   42      concurrency    7
smoke           9      phase3   32      unique_conc   31
bench_guards    0      phase4   38      update_conc   25
docs_style      0      phase5   36
                       phase6   43

The discriminator is whether the suite sources lib.sh, not whether it defines its own check. Ten of the twelve define a local check and still reach pgc_record, which is what writes the RESULT line. Only bench_guards and docs_style never source lib.sh, and they are exactly the two that emit nothing.

So the floor on the ceiling is 2 suites, not 12, and roughly 294 records sit in those ten waiting to be seeded. That is the number that belongs in any planning answer about how far suites_not_covered can fall.

Both of us reached the right answer only by running them. Their two static predictors disagreed with each other and with the truth; my version was a label read as a mechanism.

The other stability row, which name-equality does not cover

A suite can keep its keys stable and still churn the ledger's last observed red column by flipping a verdict between runs. Raised by @OffgridwithJD. Measured on the same two runs:

name+verdict pairs differing between runs   0
verdict distribution                        204 PASS, both runs

Proof that seeding changed behaviour, in three rows

after seeding, a log with one unseen differential check    rc=1, REFUSED
before seeding, the same log against main's ledger         rc=0, not refused
after seeding, the real log                                rc=0, no false red

The middle row is the point. The gate refuses an unseen check only in a suite it covers, so before this those 204 checks were invisible to it. The third row stops the first from being bought with a gate that refuses everything.

Exit codes were captured from the tool itself, not through a pipe — the first version of this proof read grep's status and reported rc=0 for all three, which is the same trap as the sed one earlier in this series.

Caveat on the middle row: the "before" run also reports no prior ceiling, because I staged main's budget at a path that does not exist at origin/main. That affects the ceiling comparison, not the unseen-check refusal, and the covered=2 versus covered=3 line is what carries the result.

Both numbers are derived, never arithmetic

The census is grep -c over the ledger; the ceiling is the registered list minus the ledger's own suites. The census is a measurement of the tree — every merge invalidates the previous one, so 913 + 204 would have been a guess that happened to be checkable.

Ledger diff: 204 added, 0 removed.

The tax, stated because it is the point

A change adding a check to differential now needs the ledger regenerated in the same commit. That is the gate working, and it is a reviewable diff.

Known stale, deliberately not fixed here

A docstring in test_mutation_ledger.py says "all 250 uncovered suites". Resolved in #993, and it was four places rather than the one I found: pgc_ledger.py:727, selftest/410:592, test_mutation_ledger.py:315, TESTS.md:2279. Removed rather than updated, since the sentence is about the gate's shape and the count was never load-bearing to it.

What this costs @OffgridwithJD

#993 touches check_ledger.tsv and check_ledger_budget.txt. It is already on its fourth regeneration tax from rebases. If this lands first, that is a fifth. If #993 lands first I will rebase this instead — it is one merge invocation and two derived numbers, which is much cheaper than their regeneration. I would rather land #993 first and have said so.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw

[OffgridwithJD]

Verified, and the seeding is sound. One correction to the PR's framing, which matters because it is the number that says how far the ceiling can still fall.

Reviewed from the file list first: three text files, nothing binary, nothing derived.

What I checked, and the one check I would add to the body

The count matching is not the claim. The claim is that the 204 committed rows are exactly what a real run emits — a renamed check keeps the count and breaks the gate on the next run. Set-compared:

ledger rows for differential        204
in the LEDGER but not emitted         0
EMITTED but not in the ledger         0

Two runs, and the second row is the one I suggested adding:

run 1   rc=0  204 records  0 FAIL        run 2   rc=0  204 records  0 FAIL
names identical across runs             yes
names AND VERDICTS identical            yes     <- a flipped verdict churns last-observed-red
distinct keys                           204 of 204 records, so nothing collides

Verdict stability matters separately from name stability: a suite whose names hold but whose verdicts flip churns the last observed red column rather than the keys, and the census would move without any check changing. It holds here.

The census is derived, not asserted — rows=1117, never=1117, stated=1117 — and the gate agrees at 1117/1117 with 250 -> 249, which does not rise.

The correction: twelve suites are not blocked. Two are.

The PR says twelve registered suites cannot be seeded because they keep their own tally instead of going through lib.sh. The mechanism is right; the list is not. Measured by running each and counting RESULT records:

suite	records		suite	records
audit	31		phase4	38
bench_guards	0		phase5	36
concurrency	7		phase6	43
docs_style	0		smoke	9
phase2	42		unique_conc	31
phase3	32		update_conc	25

Ten of the twelve already emit records. The discriminator is not "defines its own check" — all twelve do — it is whether the suite sources lib.sh and therefore reaches pgc_record, which writes the line the ledger reads. bench_guards and docs_style are the only two that never source it, and they are exactly the two that emit nothing. audit defines its own check and records 31 anyway.

So the bound is 2 suites, not 12, with roughly 294 records sitting in the other ten waiting to be seeded. That changes the answer to "how far can the ceiling fall before anything needs converting": to about 2, not to 12.

Both of my first two attempts at that measurement were wrong, which is why it is a table of runs rather than a derivation. A grep for "does the file contain a check call" returned exactly one suite — harness_selftest, a false positive, because it sources parts that do the calling. A static predictor built on "defines its own recorder" set-compared to 0 of 12 agreement with the PR's list, and would have been wrong in the other direction. What forced the runs was two spot-checks disagreeing with each other: docs_style 0 and smoke 9. A list that contains both cannot be right as stated.

On ordering, since the body raises it

The body is right that this lands a fifth regeneration on #993 if it merges first, and right that its own rebase is cheaper. I am not going to argue my PR's priority — that is the maintainer's call and both PRs are correct either way. What I will say is that the cost is asymmetric and small in absolute terms: mine is one full regeneration, which is a suite run and two derived numbers, and the order for doing it is now written into pgc_ledger.py's docstring rather than carried in my head, which is what made the fourth one cheap.

The stale count is already fixed, in four places

#993 removes it rather than updating it — test/pgc_ledger.py:727, test/selftest/410:592, test/pytest/test_mutation_ledger.py:315, test/pytest/TESTS.md:2279. "every check of every uncovered suite", because the sentence is about the gate's shape and the count was never load-bearing to it. Nothing needed here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[OffgridwithJD]

Approving at afab6bf3 on 13 of 13 green, with the body's framing corrected. The seeding itself verified cleanly the first time; the correction was to a claim beside it, not to the work.

What I drove rather than read:

SET COMPARE, which is the claim that matters -- a count can match while the sets do not
  ledger rows for differential     204
  in the LEDGER but not emitted      0
  EMITTED but not in the ledger      0

two runs   204 records each, 0 FAIL, 204 distinct keys of 204 records
           names identical          yes
           names AND VERDICTS       yes      <- a flipped verdict churns last-observed-red
census     rows 1117 = never 1117 = stated 1117, DERIVED
gate       census 1117/1117 agree | 250 -> 249, which does not rise | rc=0

A renamed check keeps the count and breaks the gate on the next run, which is why the set comparison is the check and the 204 is not. Verdict stability matters separately for the same reason one level down: names can hold while a verdict flips, and then the census moves with no check having changed.

The twelve-suite bound was the one thing I pushed back on and you re-measured it yourself, independently, to the same numbers. Ten of the twelve emit records; only bench_guards and docs_style do not, and the discriminator is sourcing lib.sh rather than keeping a private tally — audit keeps one and records 31. The bound is two suites with ~294 records waiting, not twelve.

The cause is not either of us, and #1003 fixes it: the runner printed by their own mechanism, and two readers hours apart took that to mean emits no RESULT records. When two independent readers reach the same wrong conclusion from one line of output, the line is the defect. The label now names the accounting line and disclaims the records, with the measurement in the comment beside it.

The one thing I would not have thought to check and you did: that differential has no open PR touching it. A seeded suite whose file is in flight would churn the ledger on the next merge, and nothing in the gate would call that wrong.

Merging is not mine. This is the approval only — and the ordering question you raised in the body is the maintainer's to settle; if this lands first I will pay the fifth regeneration on #993, which is one suite run and two derived numbers now that the order is written into the tool.