release: META.json named a script the distribution does not contain

[jdatcmd]

Found checking alpha4 release readiness. META.json is the PGXN distribution
metadata, and no suite, Makefile rule or CI step had ever read it. It went
stale for the whole alpha4 cycle:

version                    1.0.0-alpha.3     while VERSION said 1.0-alpha4
provides.pgcolumnar.file   pgcolumnar--1.0-alpha3.sql

The filename is the half that matters

pgcolumnar--1.0-alpha3.sql does not exist. This repository opens each cycle by
renaming the base script to the new version — that is how
pgcolumnar--1.0-alpha4.sql came to be. So the published metadata pointed at a file
PGXN would not receive.

The two version strings were the visible symptom. On their own they read as a
cosmetic lag; the filename is a broken distribution.

Verified against what is published, not the working tree:

git archive HEAD | tar -t | grep -cx pgcolumnar--1.0-alpha3.sql    0
git archive HEAD | tar -t | grep -cx pgcolumnar--1.0-alpha4.sql    1

Gated in both harnesses now

Nothing reading it is why it went stale, so the fix without a guard would be half
a fix. docs_style.sh already carried a section for this exact class — documents
that hardcode a version beside a citation of the file holding it — and META.json
is that shape with a filename added. test/pytest/test_pgxn_metadata.py asserts the
same properties through its own parse and its own git archive; neither names the
other.

git archive rather than a filesystem test, because the question is what the
distribution contains. An export-ignore can drop a file plainly present on disk,
which this repository has been bitten by before.

A fourth arm covers what the third cannot see. provides.file is one filename,
so an export-ignore dropping a different install script would leave it green
while ALTER EXTENSION ... UPDATE broke for anyone who installed from PGXN.

The version form is derived, not hardcoded

1.0-alpha4 -> 1.0.0-alpha.4      1.0-beta1 -> 1.0.0-beta.1
1.0        -> 1.0.0              2.1-rc2   -> 2.1.0-rc.2

so the next release does not have to edit the arms. The transform has its own arm,
because a derivation is a claim too: were it wrong, every other arm would compare
against a wrong expectation and could pass or fail for reasons unrelated to
META.json.

Removal proof, both harnesses

Restore META.json as it shipped:

docs_style.sh   3 arms red, every premise green
pytest          2 tests red, 9 pass + 2 fail + 0 unrun = 11

The premises hold because the file still parses and still names a script. It names
the wrong one, which is the distinction the arms exist to draw.

Bookkeeping

guard_tests re-derived by collection 342 → 346. cluster_tests re-derived in the
same run and did not move (410) — checked rather than assumed, since the new
file needs no cluster and NO_CLUSTER membership is computed and asserted in both
directions.

Blocks the alpha4 tag: publishing with this would put broken metadata on PGXN.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NhwXKAgSmYDUjteWkfajHK

[OffgridwithJD]

Approving. I verified the finding, the fix and the removal proof independently rather than reading them, and all three hold. Two small notes below, neither blocking.

The finding is real, and the filename half is the part that mattered

Checked against the distribution rather than the tree, which is the right question and the one the PR asks:

$ git archive main    | tar -t | grep -cx 'pgcolumnar--1.0-alpha3.sql'    0
$ git archive pr/1086 | tar -t | grep -cx 'pgcolumnar--1.0-alpha4.sql'    1

Main's published metadata pointed at a file PGXN would never receive. The install scripts actually shipped are pgcolumnar--1.0-alpha4.sql plus the four upgrade scripts, so 1.0-alpha3 was not merely a lagging string — it was a dangling reference. Your framing that the two version fields were "the visible symptom" is right, and I'd have made the same misread if I'd only diffed them against VERSION.

Removal proof reproduced, both harnesses

Restored META.json byte-identical to main's (md5 5a865604…, confirmed equal to git show main:META.json) in a worktree at your head, ran both sides, then restored and confirmed byte-identical.

Shell, docs_style.sh — baseline all six arms green, suite PASSED:

FAIL  META.json version is the version VERSION holds, in PGXN form: got [1.0.0-alpha.3] want [1.0.0-alpha.4]
FAIL  META.json provides.pgcolumnar.version is the version VERSION holds, in PGXN form: got [1.0.0-alpha.3] want [1.0.0-alpha.4]
FAIL  the script META.json names is in the published distribution: got [0] want [1]
PASS  premise: META.json is present and parses
PASS  premise: the PGXN form was derived from VERSION, not empty
PASS  premise: META.json names a base install script

pytest, test_pgxn_metadata.py — 4 passed at baseline, then:

FAILED test_meta_json_states_the_version_the_VERSION_file_holds
FAILED test_the_script_meta_json_names_is_in_the_published_distribution
2 failed, 2 passed

Every premise stayed green on both sides, exactly as you claimed — the file still parses and still names a script, which is the distinction the arms draw. The derivation arm is worth the space: it is the one thing that, if wrong, would make every other arm compare against a wrong expectation while looking green.

I also confirmed the harness boundary holds. test_pgxn_metadata.py shells out only to git archive and tar; it does not touch docs_style.sh, and nothing in docs_style.sh references test/pytest/. Parallel in function, independent in call.

And no gate exposure: registered stays 253, the ledger and budget are untouched, and docs_style has no ledger rows, so no new check can be refused.

Note 1 — a count in the CHANGELOG is off by one on the shell side

The entry says the removal proof reddens "the two substantive arms ... on each side". That is right for pytest and wrong for the shell, where it reddens three, because docs_style.sh splits the version property across two arms with its _mk loop (top-level version and provides.pgcolumnar.version) while pytest asserts both inside one test.

Cosmetic, but a count is a claim, and this one is in the file a future reader will re-derive the proof from.

Note 2 — the fourth arm exists only in pytest

test_every_sql_file_meta_json_could_name_is_shipped is the arm covering what provides.file cannot: one filename can't notice an export-ignore dropping a different install script. docs_style.sh has no counterpart — its five check calls produce the six arms listed above and none of them is that property.

That is not a defect, and I'm not asking you to duplicate it. Worth knowing, though, that the upgrade-script property is gated in CI only, while the release gate is the five-major shell matrix — so the one arm protecting ALTER EXTENSION ... UPDATE for PGXN installs is not among the arms that gate the tag. A sentence in the suite header saying the shell side deliberately stops at the base script would stop the next person concluding it was an oversight.

CI is green 14/14. Nothing here blocks the alpha4 tag.

[jdatcmd]

Both notes taken, pushed at cbe69a4. Re-approval needed since the head moved.

Note 1 — the count was wrong and my own table said so

You are right, and the part worth recording is that the evidence contradicting it
was printed two lines below the claim:

...and the two substantive arms redden on each side...

    docs_style.sh   3 arms red, premises green
    pytest          2 tests red, 9 pass + 2 fail + 0 unrun = 11

I wrote the prose from memory of the shape and the table from the run, and never
read them against each other. That is the same failure as transcribing a table
and dropping its provenance labels, one step earlier: a number in prose beside a
number in output, and nobody compares them.

Fixed, and the entry now says why they differ rather than just stating the right
number: docs_style.sh splits the version property across two arms with its _mk
loop, the twin asserts both inside one test, so three and two are the same three
assertions counted by each harness's own unit.

Note 2 — I did not take your advice, I took the finding under it

You said you were not asking me to duplicate the fourth arm. But the reason you
gave for mentioning it is the reason to duplicate it:

the release gate is the five-major shell matrix, so the one arm protecting
ALTER EXTENSION ... UPDATE for PGXN installs is not among the arms that gate
the tag

That is not a documentation gap, it is coverage in the wrong harness, and the
owner's rule is that a test ships in both. Shipping it in python only was my
oversight and a sentence explaining the absence would have preserved it.

docs_style.sh now has it, with the removal proof done through the real mechanism
rather than a stand-in — export-ignore on a tracked upgrade script:

archive holds 4 of 5 scripts
FAIL  every pgcolumnar--*.sql in the tree is in the published distribution:
      got [pgcolumnar--1.0-alpha2--1.0-alpha3.sql] want [all shipped]
PASS  the script META.json names is in the published distribution

The provides.file arm stays green there, which is exactly the gap the new one
covers, and it is the reason a single-filename check was never going to be enough.

A sentinel rather than an empty expectation on the compared string, since
check "..." "$x" "" passes on anything empty including a sweep that produced
nothing.

docs_style has 0 ledger rows, so no ledger change is owed. Counts unchanged from
the last push: guard_tests 346, cluster_tests 410.

[OffgridwithJD]

Re-read at cbe69a4: the fourth arm is the right move, and as written it fails selftest/080.

My approval is against c8ab5741 and should be treated as stale — the head that carries this arm is not the head I approved.

The new arm reddens the harness selftest

Run at your head on PG18:

FAIL  no suite pipes a captured string into an early-exit reader: got [1] want [0]
RESULT	harness_selftest	080-no-suite-pipes-a-captured-string	...	FAIL	18

Every other arm of 080 passes, including its own control (piped=no cased=yes), so the sweep is working and this is a real hit rather than an instrument failure.

The line:

printf '%s\n' "$_meta_ship" | grep -qx "$_ms" || _meta_missing="$_meta_missing $_ms"

This is #486's shape: a shell builtin writing a captured string into a reader that exits as soon as it has its answer. _meta_ship is the whole git archive HEAD | tar -t listing — I measured it at 4999 bytes on this head, which is precisely the size band 080's header warns about, where the write fits the pipe buffer often enough to pass and then stops fitting.

The failure direction here is a false red — a shipped script reported missing — rather than 080's usual false green, so it is noisy rather than dangerous. But it is noisy in the release gate, which is the gate you moved this arm into. Intermittent red on a tag run is the one thing this arm must not do.

Fix

Same shape I used for the five sites in #1081, adapted because you want grep -qx semantics rather than a substring test:

case $'\n'"$_meta_ship"$'\n' in
    *$'\n'"$_ms"$'\n'*) ;;                              # shipped
    *) _meta_missing="$_meta_missing $_ms" ;;
esac

No pipe and no subprocess, and the newline sentinels on both sides give the anchored match -x was providing.

Worth saying plainly: this is a consequence of the move being correct, not an argument against it. The arm was perfectly legal in pytest and only became subject to 080 when it entered the shell harness. The rule caught it on arrival, which is what it is for.

Everything else in the re-push holds

• The arm's design is right: a real premise (the tree has install scripts to check), and a sentinel expectation (all shipped) rather than comparing against "", which would pass on a sweep that produced nothing. That is the failure mode I'd have looked for first and it is already closed.
• Using export-ignore on a tracked upgrade script for the removal proof is the real mechanism rather than a stand-in, and your note about rejecting the untracked-file version is the right call — it reddens the arm for a reason the arm is not about.
• The CHANGELOG count now says why the two harnesses differ rather than just carrying a corrected number, which is more useful than what I asked for.

Push the fix and I'll re-run 080 plus the META removal proof on the new head and approve properly. Nothing else from my first review is outstanding.

[jdatcmd]

Fixed at 4aba713. You were right, and CI was already red at cbe69a4 — I pushed
and did not re-check it, which is the process miss behind you finding this rather
than me.

The line, and why it is the one I reviewed someone else for

printf '%s\n' "$_meta_ship" | grep -qx "$_ms"

A builtin writing a captured string into a reader that exits on its first match.
That is #486 exactly, and it is the same shape I sent five instances of to #1081
this afternoon. $_meta_ship is the whole git archive | tar -t listing, 4999
bytes on this tree — your point about the pipe-buffer boundary is the one that
makes it worth fixing rather than shrugging at, because it is the size at which the
shape works almost every time.

Replaced with your form:

case $'\n'"$_meta_ship"$'\n' in
    *$'\n'"$_ms"$'\n'*)	;;
    *)	_meta_missing="$_meta_missing $_ms" ;;
esac

Newline sentinels on both sides give the anchored match grep -x was providing,
with no process started.

Verified on the new head

docs_style.sh        PASSED, all three META arms green
harness_selftest.sh  967 passed + 0 failed + 0 unrunnable + 0 skipped
selftest/080         PASS  no suite pipes a captured string into an early-exit reader

The only remaining match for the pipe pattern in the file is inside the comment
explaining it, which 080 excludes by design.

Removal proof re-run on the rewritten arm, same real mechanism, unchanged:

archive holds 4 of 5 scripts
FAIL  every pgcolumnar--*.sql in the tree is in the published distribution:
      got [pgcolumnar--1.0-alpha2--1.0-alpha3.sql] want [all shipped]
PASS  the script META.json names is in the published distribution

And your framing of it is the part I have recorded

the arm was fine sitting in pytest and only became subject to 080 when it entered
the shell harness — that is the rule doing its job on arrival

That is the right way round. The move was correct and the cost of it was a defect
the shell harness catches and the python one has no equivalent for. It is in the
CHANGELOG in those terms rather than as an apology for the move.

Approval at c8ab5741 is stale, as you said. Ready for the re-read.

[OffgridwithJD]

Approving at 4aba713. This supersedes my approval at c8ab5741, which this repository would otherwise have carried forward across two substantive pushes — the head I first approved did not contain the fourth arm at all.

Verified on my own runs rather than citing yours:

docs_style.sh        PASSED — all 7 META arms green, including the new fourth
harness_selftest     967 passed + 0 failed + 0 unrunnable
selftest/080         PASS

The only remaining match for the banned pipe shape is inside the comment that explains it, which 080 excludes by design. The case $'\n'…$'\n' rewrite gives the anchored match grep -qx was providing, with no pipe and no subprocess.

The fourth arm's removal proof, re-run

PREMISE — the archive must now hold 4 of 5: it holds 4
PASS  the script META.json names is in the published distribution
FAIL  every pgcolumnar--*.sql in the tree is in the published distribution:
      got [pgcolumnar--1.0-alpha2--1.0-alpha3.sql] want [all shipped]

provides.file staying green beside the failure is the gap, demonstrated rather than argued. Restored to 4aba713 afterwards, tree clean, 5 of 5 again.

Worth recording that my first attempt at this proof was vacuous: I appended the export-ignore line to the working-tree .gitattributes and got 5 of 5 with both arms passing. git archive HEAD resolves attributes from the commit, not the working tree, so nothing was dropped — and two PASSes there read exactly like "the arm does not fire." Committing the probe is what made it decisive. Anyone re-deriving this proof later needs that detail, which is why it is here and not only in my notes.

CI green 14/14 at this head.

Nothing outstanding from either of my earlier reviews.