test: read a suite's own recorders, and refuse the one it cannot (#1053)

[OffgridwithJD]

Closes #1053.

#1051 taught the extractor the recorders lib.sh shares. A suite may also define its own — and there are two shapes, only one of which is a gap.

COMPOSE   check "non-owner refused: ${1%%(*}"   the definition states a TEMPLATE naming the property
FORWARD   check_text "$label" "$got" "$want"    the definition states nothing; the NAME is at the call sites

A composing wrapper is already read correctly: non-owner refused: {} covers all nine of native_ownership's call sites, which is why that pair grades one-for-one today.

A forwarding wrapper's definition reduces to the bare template {}, and 17 of those were being published — a "property" with no content, sitting in MISSING where no port can ever assert it, and matching a port name that is entirely one interpolation. A wrong name is worse than an absent one, which is the argument #1051 turned on.

What changed

The grader derives each suite's own recorders by the rule that already works for lib.sh — a function forwarding a bare positional into a known recorder's name slot, transitively, seeded from pgc_record — reads the call sites of the forwarding ones, drops the bare {}, and leaves composers alone.

names the reader gains   145 across 14 suites
every graded pair        unchanged
refused corpus-wide      1 of 264, and it has no twin

sorted_pathkeys gains 18, and they are not a random 18: that suite pairs every "... plans no Sort" with an "and ... still answers correctly", so the grader could see every claim about the plan and none about the answer.

And it refuses what it cannot read. hilbert_curve.sh defines two helpers taking a newline-separated LIST of names in one argument and looping read -r over it, so no rule about argument positions can read them. main exits 2 naming both, and prints no verdict, rather than grading the rest.

Measured before building, and it changed the design

Refusing on "the name position is not a bare positional" also refuses every composing wrapper — 32 suites, including hilbert_cluster, hilbert_locality and native_ownership, three pairs that are COMPLETE today — to fix nothing. The refuse half is right in principle and, aimed at that population, it breaks green pairs. @jdatcmd independently reproduced the 32.

Removal proof

__pycache__ cleared before every arm, the runtime asserted, md5 applied and restored.

	mutation	result
	CONTROL	rc=0, 0 failed — runtime (110 names, ans+ansp)
M1	do not read the call sites	rc=1, 3 failed — runtime 110 → 92
M2	composers treated as unreadable	rc=1, 3 failed
M3	do not refuse, just skip	rc=1, 1 failed
M4	publish the bare {}	rc=1, 2 failed — runtime 110 → 113
M5	seed without the primitive	rc=1, 1 failed

M3 and M5 reddened nothing at first. The refusal in main was exercised by no arm — only the classifier feeding it was — and the pgc_record seed was covered by no arm either, though it is the whole difference between reading 145 names and 89. Two guards nothing exercised, in a change about a grader that was not reading what it claimed to. Both now have an arm and both reddens isolate to it.

The refused set is pinned by name, not counted

checks_never_observed_red is this repo's worked example of the other shape: a census every legitimate addition broke, so the only way to land one was to raise a number the design said may only fall — which retires the guard the first time it is inconvenient. A count tells a reviewer that something moved; a set tells them what. Asserted in both directions, so an entry cannot outlive its cause.

One arm was correctly caught by another guard

Naming a real test_*.py as the control made test_harness_deps.py classify this file as cluster-bound — a file that drives a cluster-bound file needs whatever that file needs, and it was right to. The arm is about the grader's refusal, not about any port, so it supplies its own fixture port instead.

Verification

guard leg    329 passed, 830 checks, 0 fail
cluster leg  320 passed, 885 checks, 0 fail   (--pg-config=/usr/local/pg17a)

Every graded pair re-graded: differential 86/54 and iceberg_fdw 76/2 unchanged (both declared INCOMPLETE), the other seven at missing: 0.

guard_tests 322 → 329 by collection. #1054 moves it to 324 on its own tree, so whichever lands second re-derives — two independent 322s collected 323 earlier today and nothing conflicted.

🤖 Generated with Claude Code

https://claude.ai/code/session_012RSw4qMHS7ByE7PY8Ns4cs

[jdatcmd]

Verified at f239e98 on my own instrument. Everything reproduces, including the number I could not reach before.

The corpus, measured with your shipped reader

suites scanned   264
REFUSED          ['hilbert_curve']  -> ['arms_failed', 'arms_unrunnable']
raw call-site names               147
after the bare-{} drop            145 across 14 suites
dropped                             2   native_groupagg_batch: agree_in -> '$1'
                                        sorted_pathkeys:       ansp     -> '$1'

145 across 14, exact. And the two dropped are precisely the forwarder-calling-forwarder cases your own correction identified — agree into agree_in, ans into ansp — so that fix is doing visible work rather than being asserted.

Every graded pair unchanged

differential 54   iceberg_fdw 2   the other seven all missing: 0
native_ownership  literal 2 | template 1     <- the composer still readable
hilbert_locality  literal 19 | template 11

The composers survive, which is the whole point of the split.

Both guards you flagged are load-bearing now

Verified by mutation, caches cleared, each asserting it applied:

main no longer refuses            1 failed
pgc_record seeded at $1 not $2    2 failed  (extractor arm + planted-wrapper arm)

One of my own mutations did not apply and I nearly read it as a missing guard. My first attempt at the seed mutation targeted a "pgc_record": 2, table entry that does not exist — the seed is a _derive_recorders default argument. The sed matched nothing, the run came back 30 passed, and "30 passed" is exactly what "this seed is unguarded" looks like. Caught only because my "applied" line never printed. That is the fourth time this week the not-running has looked like an answer.

The budget arm

expect.text(", ".join(refused), "hilbert_curve", ...) is the right shape. It names the suite, reddens in both directions, and makes a legitimate new refusal a one-line reviewable diff rather than a number to bump.

That you reached the checks_never_observed_red precedent from the other side — that it is the same argument as INCOMPLETE carrying a reason rather than a licence — is the connection I would keep. A count tells a reviewer something moved; a set tells them what.

On the diagnosis exchange

Worth recording that you diagnosed my instrument by the failure you had just had yourself — right family, wrong level. It was a good guess and testing it is what found the real cause, which was pgc_record naming its check at $2 while my pattern read every seed member at $1. Three instances in one day across two sessions of the same rule being stated and then not implemented.

Before merge

#1054 landed as f5fec97, so main has moved under you. guard_tests was 324 there and is 329 here; re-derive both keys on the merged tree rather than carrying either side, and note cluster_tests is 320 on both — the agreeing-numbers case that produced 323 out of two independent 322s earlier today.

Approving once the legs land; nothing in the substance is outstanding.

[jdatcmd]

Approving at 9b714ef. Re-verified on the rebased tree rather than assumed to survive it, and both keys re-derived on my own instrument.

guard   collected 331   tracked 331   331 passed, 841 checks, 0 fail, 0 unrun
cluster collected 320   tracked 320   320 passed, 885 checks, 0 fail, 0 unrun   (PG16)

all nine graded pairs unchanged   differential 54, iceberg_fdw 2, seven at 0
REFUSED                           ['hilbert_curve'] -> arms_failed, arms_unrunnable
names the reader gains            145 across 14 suites

331 reproduces, and so does your point that 322 + 7 + 2 reaching it is a coincidence of this merge rather than a method — the two deltas were measured against different trees, which is the precise thing that produced a number no tree collected this morning.

The observation worth keeping is yours

The two keys failed differently in the same merge operation.

guard_tests    moved on both sides   ->  git CONFLICTED and demanded an answer
cluster_tests  320 on both sides     ->  merged SILENTLY

Silence is not agreement, it is the absence of a signal. cluster_tests happened to be right; two independent 322s merged just as silently earlier today and the tree collected 323. The conflicting case is the safe one because it forces the question — the quiet one is where the wrong number ships.

Putting that in the file's comment rather than the PR body is the right call. The file is where the next person is standing, and a PR body is read once.

On the near-miss

Agreed on the framing, and it is the part I would keep from this whole exchange: it is the stale .pyc in a different coat. Both are "the mutation did not reach the runtime", and in both a green run means two opposite things depending on why.

did it apply?      md5 moved
did it take effect? probe the runtime

This is the first time one of those fired on something real rather than being asserted as a habit, and it fired on me about your branch. Without the applied-check I would have sent you "your pgc_record seed is unguarded" with a clean 30-passed run behind it.

Merging

Substance is done; I will merge when the legs land. Then #1045 class 2 is yours — same shape one harness over, and you have the instrument for it.