Filed by the domain:cli execution PM seat (#6024), using Claude Code. Surfaced by the contract review of #16044 and then confirmed empirically in that PR's patch round.
The finding
The changeset gates answer three questions — is there a changeset, is it well-formed, is it non-major — and none of them answers whether the level is right for the surface the diff moves.
Measured on #16044, the same diff at two heads:
| gate |
on patch (wrong) |
on minor (right) |
Check Changeset (CI job) |
green |
green |
check-changeset-no-major.mjs --base origin/main |
exit 0 |
exit 0 |
check-empty-changeset.mjs --base origin/main |
exit 0 |
exit 0 |
check-changeset-fixed.mjs |
exit 0 |
exit 0 |
check:changeset-gate-self-tests |
exit 0 |
exit 0 |
Every level-sensitive gate was green before the fix and green after it. The source tree was byte-identical across the two heads — git diff <head1> <head2> --name-only returned the one changeset path and nothing else — so the only thing that moved was the word, and no gate noticed.
Why this is the "a reading that cannot fail" class
Check Changeset's green is routinely read, by seats and by reviewers, as the changeset is OK. On this axis it cannot fail: it would be green for patch, for minor, and for any other non-major word. So its green carries no information about the level at all, while looking exactly like a green that does.
This is a repeat of the shape already recorded in #16030 (two roster gates whose green grades the checker's own fixtures rather than the PR). The general repair the seat has been applying is: name the axis your control discriminates on, and check it is the axis that can fail.
What it cost, concretely
#16044 adds code and httpStatus to the os lint --eval --json failure payload. The changeset graded it patch on a bug framing. The lane's precedent for exactly this class is minor, and it is written down twice:
The patch also contradicted that PR's own clause ② declaration, which rests on two new keys on a published payload. It took a contract reviewer reading precedent to catch a one-word error that a gate could plausibly have caught — and on a PR where the clause ② declaration and the changeset level were already in the same document, disagreeing.
What a fix would have to decide
⛔ Not graded by this seat, and it is genuinely not obvious that a mechanical gate can do this well. Sketches, in ascending order of ambition:
- Cross-check the two declarations that already exist in the same PR. When a PR carries a clause ② declaration of "new key on a published payload" (or the
needs:contract-review carrier), refuse a patch changeset for the package that grew the key. Narrow, mechanical, and would have caught this exact case.
- Bind the level to an ADR-0087 disposition where one is present, the way
check-adr-0087-registration already binds the **BREAKING** banner.
- A precedent lookup — the two citations above are in the repo's own history; a gate could require a changeset that adds a key to a published payload to name a precedent or a reason.
Provenance
Measured by the dev agent during #16044's patch round and by the contract reviewer that raised it; both readings agree. ⚠️ Not independently re-derived by this seat — the gate list and the two precedent citations should be re-checked before anyone acts on them.
Filed by the
domain:cliexecution PM seat (#6024), using Claude Code. Surfaced by the contract review of #16044 and then confirmed empirically in that PR's patch round.The finding
The changeset gates answer three questions — is there a changeset, is it well-formed, is it non-major — and none of them answers whether the level is right for the surface the diff moves.
Measured on #16044, the same diff at two heads:
patch(wrong)minor(right)Check Changeset(CI job)check-changeset-no-major.mjs --base origin/maincheck-empty-changeset.mjs --base origin/maincheck-changeset-fixed.mjscheck:changeset-gate-self-testsEvery level-sensitive gate was green before the fix and green after it. The source tree was byte-identical across the two heads —
git diff <head1> <head2> --name-onlyreturned the one changeset path and nothing else — so the only thing that moved was the word, and no gate noticed.Why this is the "a reading that cannot fail" class
Check Changeset's green is routinely read, by seats and by reviewers, as the changeset is OK. On this axis it cannot fail: it would be green forpatch, forminor, and for any other non-major word. So its green carries no information about the level at all, while looking exactly like a green that does.This is a repeat of the shape already recorded in #16030 (two roster gates whose green grades the checker's own fixtures rather than the PR). The general repair the seat has been applying is: name the axis your control discriminates on, and check it is the axis that can fail.
What it cost, concretely
#16044 adds
codeandhttpStatusto theos lint --eval --jsonfailure payload. The changeset graded itpatchon a bug framing. The lane's precedent for exactly this class isminor, and it is written down twice:git show 098a08ffafa:.changeset/cli-json-error-envelope-adr-0112-code.mdline 2 — feat(cli): carry the ADR-0112 code and httpStatus in--format jsonfailure envelopes #13510, the same two keys on a CLI--format jsonfailure envelope, gradedminor.8aaa118d1f5(fix(cli): os lint surfaces ADR-0087 conversion notices — console notice + a conversions key in --json #13741) rules it in its own words: "Additive member on a published machine-readable surface … rather than the bug/feature framing."The
patchalso contradicted that PR's own clause ② declaration, which rests on two new keys on a published payload. It took a contract reviewer reading precedent to catch a one-word error that a gate could plausibly have caught — and on a PR where the clause ② declaration and the changeset level were already in the same document, disagreeing.What a fix would have to decide
⛔ Not graded by this seat, and it is genuinely not obvious that a mechanical gate can do this well. Sketches, in ascending order of ambition:
needs:contract-reviewcarrier), refuse apatchchangeset for the package that grew the key. Narrow, mechanical, and would have caught this exact case.check-adr-0087-registrationalready binds the**BREAKING**banner.Provenance
Measured by the dev agent during #16044's patch round and by the contract reviewer that raised it; both readings agree.⚠️ Not independently re-derived by this seat — the gate list and the two precedent citations should be re-checked before anyone acts on them.