Measured on PR #16243 (head 484601894), where it produced a false "authored to green" claim that stood for about three hours.
The reading
scripts/pm/dispatch-gates.mjs maps a change set's paths to the check:* families that watch them. It maps packages/qa/dogfood/** to nothing.
On that PR the change set included packages/qa/dogfood/test/authz-probe-blind-spot.census.ts:
$ node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack | wc -l
102
$ node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack | grep -i 'dogfood\|qa'
(no output)
102 families derived, 102 runnable, and not one of them runs a dogfood test. On the same head Dogfood Regression Gate (3/3) was red (run 34019739422, job 101450083462, failed 07:50:38Z) on test/authz-probe-blind-spot.test.ts.
So a dev can follow the documented loop exactly — derive the families, run all of them, reconcile with --ran, get 102 derived famil(ies) accounted for, 102 run, 0 NOT-MEASURED — and truthfully report a full green while a test in the very package they edited is failing in CI. The reconciliation cannot catch it: --ran reconciles against what the derivation produced, and the derivation never named the gate.
Why this is not the same as the known family
There is a family of derived-coverage findings (#13813, #15501, #15553, #13511, #15312 and others) where a gate exists but a particular change set does not reach it. This one is wider in a way worth separating: the derivation has no edge at all from a first-party test package to the CI job that runs its tests. It is not a missed path literal inside a routed family — packages/qa/dogfood is routed nowhere, so every card that touches it inherits the blind spot, not just cards with an unusual shape.
The general shape is also worth stating once: packages/** test suites are run in CI by sharded workflow jobs (Dogfood Regression Gate, Test Core) rather than by a check:* script, and the derivation's vocabulary is check:* families. Any package whose CI coverage is a workflow job rather than a gate script is invisible to it the same way.
Suggested direction (not a decision)
Either teach the derivation to emit a package-test command for a changed package (pnpm --filter PKG test) alongside the check:* families, or have it name the workflow job that covers the path so the dev at least knows what they have not run. The second is cheaper and keeps the tool's vocabulary honest — it is a derivation of what watches this path, and a workflow job watches it just as much as a gate script does.
⛔ Not fixed in #16243 — that PR's scope is the metadata.endpoints.* switch radii. Filed per the out-of-scope rule; unassigned for PM triage.
Found while repairing #16243 (cards #15542 / #15854).
Measured on PR #16243 (head
484601894), where it produced a false "authored to green" claim that stood for about three hours.The reading
scripts/pm/dispatch-gates.mjsmaps a change set's paths to thecheck:*families that watch them. It mapspackages/qa/dogfood/**to nothing.On that PR the change set included
packages/qa/dogfood/test/authz-probe-blind-spot.census.ts:102 families derived, 102 runnable, and not one of them runs a dogfood test. On the same head
Dogfood Regression Gate (3/3)was red (run34019739422, job101450083462, failed 07:50:38Z) ontest/authz-probe-blind-spot.test.ts.So a dev can follow the documented loop exactly — derive the families, run all of them, reconcile with
--ran, get102 derived famil(ies) accounted for, 102 run, 0 NOT-MEASURED— and truthfully report a full green while a test in the very package they edited is failing in CI. The reconciliation cannot catch it:--ranreconciles against what the derivation produced, and the derivation never named the gate.Why this is not the same as the known family
There is a family of derived-coverage findings (#13813, #15501, #15553, #13511, #15312 and others) where a gate exists but a particular change set does not reach it. This one is wider in a way worth separating: the derivation has no edge at all from a first-party test package to the CI job that runs its tests. It is not a missed path literal inside a routed family —
packages/qa/dogfoodis routed nowhere, so every card that touches it inherits the blind spot, not just cards with an unusual shape.The general shape is also worth stating once:
packages/**test suites are run in CI by sharded workflow jobs (Dogfood Regression Gate,Test Core) rather than by acheck:*script, and the derivation's vocabulary ischeck:*families. Any package whose CI coverage is a workflow job rather than a gate script is invisible to it the same way.Suggested direction (not a decision)
Either teach the derivation to emit a package-test command for a changed package (
pnpm --filter PKG test) alongside thecheck:*families, or have it name the workflow job that covers the path so the dev at least knows what they have not run. The second is cheaper and keeps the tool's vocabulary honest — it is a derivation of what watches this path, and a workflow job watches it just as much as a gate script does.⛔ Not fixed in #16243 — that PR's scope is the
metadata.endpoints.*switch radii. Filed per the out-of-scope rule; unassigned for PM triage.Found while repairing #16243 (cards #15542 / #15854).