A named skip cannot be ported: 66 suites and 1281 names have no expressible counterpart

[OffgridwithJD]

Split out of #432 because #1040, which carried this as "phase 0b", is closed. It blocks 30% of the remaining port and nothing tracks it.

The number

remaining unported suites                              241
  carrying a NAMED gate (check_skip/pgc_skip/pgc_fail)  66   1281 names   30%
  clean                                                175   2917 names

Measured on a95c360, over the registered suites with no pytest twin, counting distinct names with _names_in (not _bash_names — see my correction in #432 about the two disagreeing enough to reorder the queue).

The top of the queue is in the blocked set: parquet_export_stats, 70 names, three gate calls.

The mechanism

check_skip, pgc_skip and pgc_fail record under the name they are given. expect.cannot_run(reason, detail) records under the reason CODE, validated against a closed list:

if reason not in UNRUNNABLE_REASONS:
    raise VacuityError(...)
self.unrunnable = (reason, detail)

So a port cannot emit the string at all, and compare_to_bash.py reports it MISSING. iceberg_fdw is the one pair that hit this and it is declared INCOMPLETE for exactly this reason, which is the precedent this issue generalises.

The entry there also refused the obvious workaround, and I agree with the refusal: naming a passing check could not read the server major, so the gate below cannot be trusted would read as an assertion that the major is unreadable.

There is a second thing going on, and it may be the real one

The grader is static; the gate is dynamic.

if [ "$PGC_MAJOR" -lt 18 ]; then
    check_skip "the differential analyze path" "SKIP  ... needs pg_restore_attribute_stats (PG18+)"
    pgc_summary
fi

On PG18 and PG19 that branch is not taken, so analyze_differential.sh never records the name. It records it on PG15/16/17 as a SKIP. On the majors where a port actually runs, the bash suite emits no such name — and the grader is asking the port to cover a branch that did not execute.

Three resolutions, not equivalent

1. cannot_run carries a name as well as its reason code. Unblocks all 1281 at once and keeps the reason keyable for the stream. Largest change to pgc_vacuity.py.
2. The grader learns about version-gated branches: a name reachable only under a PGC_MAJOR test is not required of a port running on a major where the branch is dead. Smallest change — but it asks the grader to reason about reachability in shell, which is the class this tree has been burned by four times (A sweep over shell source must strip comments AND decide about string literals: four guards got this wrong in one week #1123), so it needs a false-positive budget before it is trusted.
3. Declare all 66 INCOMPLETE, each with a reason, as iceberg_fdw is. Costs nothing to build and is honest. It also means "one-for-one" stops describing a third of the corpus, and the INCOMPLETE dict becomes the place ports go to stop being graded.

My own preference is 1, weakly, because it is the only one that leaves the grader strict and the ports complete. But this is @jdatcmd's tool and the layer's own escape-hatch design, so I am filing the measurement rather than the patch.

Why it matters now

jd wants the pytest suite released before beta1. Whatever "released" turns out to mean, 30% of the remaining names sitting behind a design question with no open issue is the wrong state to discover late. Filed so it can be answered deliberately.

🤖 Generated with Claude Code

https://claude.ai/code/session_012RSw4qMHS7ByE7PY8Ns4cs

[OffgridwithJD]

Twenty-four hours, no replies, and the cost is now concrete rather than projected. Posting
the queue position because "30% of the remainder" is easy to defer and "the next eight
suites I would take" is not.

Where the port is, on 6082c5a

remaining suites   233
  clean            167    2,872 names
  BLOCKED HERE      66    1,281 names

Five batches have landed since I filed this (#1133, #1136, #1142, and #1148 open), and
every one of them was drawn from the clean 167 — not because they were the best next
suites, but because they were the reachable ones.

What is blocked in the near queue

Of the forty smallest remaining suites, eight cannot reach missing: 0 today:

  4  iceberg_fdw_projection
  5  fuzz_arrow
  5  fuzz_parquet
  5  native_parquet_dict_oob
  5  temporal
  6  iceberg_fdw_estimate
  6  native_parquet_codecs
  7  arrow_nested

That is 43 names sitting in the cheapest part of the queue. The Parquet and Arrow ones
matter beyond their size: they are the file-format decode surface, and they are the suites
whose absence from the pytest side leaves the corpus with no independent read of that
decode path at all.

temporal is the sharpest example of the cost. #1147 has just documented temporal
constraint support on docs/features.md for the first time, on the strength of that very
suite — and the suite cannot be ported, so the documented behaviour has one harness
asserting it rather than two.

The decision, restated in one line each

1. Let cannot_run carry a name. The reason code stays; a name is added so the port
   can emit the string the bash suite records. Smallest change, and it makes 66 suites
   portable.
2. Teach compare_to_bash that a gate name has no counterpart — an exemption list, or
   a rule that a name matching a check_skip in the bash source needs no pytest twin.
   No harness change, but the grader stops being a one-to-one check for those names.
3. Declare all 66 INCOMPLETE, as iceberg_fdw already is. Honest, costs nothing
   today, and leaves a permanent 1,281-name hole that nothing ever closes.

My weak preference remains 1, and the argument against 3 is that INCOMPLETE was designed
as a declared gap with a reason, not as a bucket for a quarter of the corpus.

I am not asking for the right answer, only for one. Any of the three unblocks the same
66 suites, and I will implement whichever is chosen. What I cannot do is pick it myself:
it changes the contract between the two harnesses, and CONTEXT.md makes that the owner's.

There is a second question underneath, which I would rather not answer by accident either:
the grader is static and the gate is dynamic. On PG18 and PG19 a version branch is never
taken, so a bash suite emits no such name on the majors a port actually runs on — which
means for some of the 66 the "missing" name may never be emitted in CI at all. That
argues for 2 more than it argues for 1, and it is worth someone deciding both at once.

[jdatcmd]

Decided: option 1 — cannot_run carries a name as well as its reason code. The owner's call, taken after the three options were put with the analysis below. @OffgridwithJD is unblocked on all 66 suites and 1,281 names and will implement it.

Shape of it, as filed: the name alongside the reason; the closed reason list unchanged so the stream stays keyable; the outcome still UNRUN rather than PASS. That last point preserves the refusal the iceberg_fdw entry records — naming a passing check after a missing dependency would read as an assertion that the dependency is absent, and it stays refused.

Why 1 and not 2, which changes the second question rather than answering it

This issue's second half argued that "the grader is static; the gate is dynamic" favours teaching the grader about version-gated branches. Checked rather than taken:

_py_names     parses the port's SOURCE with ast
_bash_names   regexes the suite's TEXT

Both sides are static. Neither runs anything. A port whose source contains the name therefore matches the bash suite's check_skip name regardless of which branch executes on which major. Option 1 dissolves the dynamic question rather than needing a special case for it. What remains is whether the port's gate is faithful at run time, which is ordinary test correctness and not a property of the grading contract.

What option 2 would have cost, for the record:

• missing: 0 stops being a property of the pair and becomes a property of the run, since the same pair would grade differently on PG17 and PG18.
• a static grader is asked to reason about reachability in shell, which is the class this tree has been burned by four times (A sweep over shell source must strip comments AND decide about string literals: four guards got this wrong in one week #1123).

Option 3 stays available as the escape hatch it was designed to be, for a genuine one-off gap. Applying it 66 times would have made a hatch into the answer for a third of the corpus, with 1,281 names permanently ungraded.

One sequencing note

temporal is in the blocked 66, and #1147 documents temporal constraint support on docs/features.md for the first time on the strength of that suite. Until it has a twin, a published claim rests on one harness. Asked for early; not a condition.

[jdatcmd]

A correction to the scope I stated when I put the options up. I described resolution 1 as unblocking all 1,281 names. @OffgridwithJD found the boundary by implementing it and hitting it, and their sentence is the accurate one:

Resolution 1 unblocks a bash gate name whose PRECONDITION EXISTS on the pytest side. A gate on a dependency the port does not have stays unportable, and that is correct rather than a shortfall.

The end-to-end proof on iceberg_fdw is literal 74 | missing: 2 before and literal 75 | missing: 1 after. The remaining name is python3 is needed, and the port cannot honestly carry it: it has no python3 dependency, because it reads the committed warehouses directly.

So the decision stands and its reach is narrower than I said. That matters for planning the queue: some of the 66 will reach missing: 0, and some will reveal — as this one did — that the bash gate is the thing that is wrong.

Which is what happened here, and I verified it independently (#1150)

test/iceberg_fdw.sh:22   python3 -c 'import json' 2>/dev/null || pgc_skip python "python3 is needed"

That is the only occurrence of python3 in the file. Every .json after it is a filename reached with ls and handed to SQL; nothing in the shell parses JSON. The gate guards nothing, and removing it takes that pair to missing: 0 and leaves the tree with no declared-incomplete pairs.

Two measurements for whoever takes #1150:

• It moves no ledger rows. grep -c '^iceberg_fdw' test/check_ledger.tsv is 0 — the suite is not ledger-covered, and the gate only records when python3 is absent, so no clean run has ever emitted it.
• The harness still uses python3, just not that suite. lib.sh:748 shells to it for the source fingerprint and reports rather than swallows a miss. The change should say the suite has no python3 dependency, not that the tree has none.

[OffgridwithJD]

Implemented and gating. jd chose resolution 1; this records the limit of what it
buys, because I stated the scope too widely when I asked for the decision and the boundary
only became visible by hitting it.

The scope, in one sentence

Resolution 1 unblocks a bash gate name whose PRECONDITION EXISTS on the pytest side. A
gate on a dependency the port does not have stays unportable — and that is correct rather
than a shortfall.

I put "66 suites and 1,281 names" to the decision as though all of them clear. They do not.
What clears is every gate where both harnesses can meet, or fail to meet, the same
precondition — which is the great majority, and is the right scope. A gate guarding
something only the shell needs was never a porting problem; it is a statement about the
shell harness.

The proof, on the pair that was declared INCOMPLETE for exactly this

before   literal 74 | missing: 2     both gate names unexpressible
after    literal 75 | missing: 1     only `python3 is needed` left

iceberg warehouse data files are missing is now carried by
expect.cannot_run(..., name=...) and matched. python3 is needed is not, and the port
should not pretend otherwise: it has no python3 dependency, it reads the committed
warehouses directly, so a refusal named after a missing python3 would assert a
precondition that is always met.

That last one turns out not to be a harness question at all — the shell suite does not use
python3 either. It is the only occurrence in the file and nothing after it parses JSON in
the shell. Filed as #1150; remove the gate there and the pair reaches missing: 0 with no
further harness work.

The shape that landed

cannot_run(reason, detail="", *, name=None)

• name is keyword-only. cannot_run(reason, detail) is called positionally across
  the corpus; a name parameter in second position would silently reinterpret every
  existing detail as a check name, with nothing red to say so.
• The reason list is unchanged and still validated, and the code still rides in
  self.unrunnable, so the UNRUN line and anything keying on the code are untouched.
• The outcome is still UNRUN, not PASS. A name that arrived by turning the declaration
  into a pass would satisfy the grader while asserting the missing dependency is present.
• _NAME_ARG["cannot_run"] is now None, so an unnamed declaration states no property
  rather than publishing its reason code as a check name — the rule _names_in already
  applies when it drops a bare {}.

One test was retired rather than kept: test_cannot_run_names_its_reason_not_its_detail
pinned "the reason CODE is the name", which was the least-bad answer while there was no
name to harvest. Its successor asserts that an unnamed declaration states nothing, and the
reasoning is written into the file so the change is not mistaken for a relaxation.

Also worth knowing before porting the other 65: compare_to_bash resolves string
LITERALS at the call site. A gate name held in a module constant is a name the grader
cannot see — measured, the iceberg pair still reported missing: 2 until the string was
written where expect.cannot_run(...) is called.

[OffgridwithJD]

Closing: the blocker this issue tracks is resolved, shipped and exercised. The
port work it was blocking belongs to #432 and is moving.

The mechanism is in main

test/pytest/pgc_vacuity.py:

def cannot_run(self, reason, detail="", *, name=None):
    ...
    self._record(name=name or reason, verdict="UNRUN", reason=detail)

Resolution 1 as @jdatcmd decided it: the name alongside the reason, the closed
reason list unchanged so the stream stays keyable, the outcome still UNRUN
rather than PASS. name is keyword-only, and the docstring gives the reason —
cannot_run(reason, detail) is called positionally across the corpus, so a
positional name in second place would silently reinterpret every existing
detail as a check name.

And it is used, not merely present

  cannot_run call sites        45
  passing name=                12

Six of the seven suites this issue named as blocked now have pytest twins:
analyze_differential, temporal, iceberg_fdw, range_pruning,
native_groupagg, advisory_lock_class. parquet_export_stats is still
unported — that is port queue, not blockage.

A correction in my own measurement, because it nearly became a finding

My first count said 2, and I was about to report that the mechanism had
shipped unexercised — a capability nothing consumes, which is a real defect
shape and one I filed against someone else's change earlier today.

It was my grep:

  grep 'cannot_run(' *.py | grep -c 'name='     ->   2
  paren-balanced scan over the call text        ->  12

These calls span lines. A line-oriented pattern sees name= only where it lands
on the same line as cannot_run(, which for a four-argument call across four
lines is almost never. The two it did find were the definition and a comment —
so the grep was matching the documentation rather than the calls, which is the
same trap as counting pgc_skip in arrow_import.sh and matching the sentence
saying why not to use it.

The limit still stands and is not a shortfall

From the boundary I recorded when I implemented this:

Resolution 1 unblocks a bash gate name whose PRECONDITION EXISTS on the pytest
side. A gate on a dependency the port does not have stays unportable — and
that is correct rather than a shortfall.

iceberg_fdw's remaining python3 is needed is that case: the port reads the
committed warehouses directly and has no python3 dependency to be missing.
Claiming the name would assert a precondition it does not have.

Reopen if the blocked set turns out to be larger than the gate mechanism
reaches. The remaining 196 unported suites are #432's.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP