ci: red the Test Core shard when its predicted time stops matching the measured one

[claude]

Part of #16173. This lands the split that removes the shard-kill hazard now —
@objectstack/cli reaches the Test Core matrix as file-level slices instead of one
indivisible 1231s package — together with a fully-tested --check-drift mode in the
partitioner whose CI wiring is deliberately withheld until the dataset refresh
lands with it. Two follow-ups are named at the bottom; #16173 stays open for them.

⭐ What changes on merge: the worst Test Core shard goes from ~1445s (80% of the
30-minute wall) to ~1059s (59%) with the timings dataset untouched, because the CLI's
real cost is halved across two runners instead of falling on one. That is what ends the
run of cancelled shard-1/6 legs, and it does not wait on anything.

What was measured, before anything was written

The card's premise holds, and then some. Two independent readings:

reading	source	predicted	measured	ratio
@objectstack/cli alone	run 34009395649 attempt 2, job 101427282674 (Duration 1231.52s, 265 files / 3159 passed)	458.15s	1231.52s	2.69x
Test Core (1/6), full package list	run 34013842594, a green merge_group build, step Run this shard's tests	672s	1168s	1.74x

The second reading is the load-bearing one: it is the full package list, on a
build that went green, at the base of this branch. Its five sibling shards ran
462s / 630s / 714s / 776s / 793s against the same 672s prediction — 0.69x to
1.18x. So the imbalance is not an artefact of turbo ls --affected on one PR,
and it is not runner noise: one shard is the outlier, and it is the one carrying
the repo's largest suite.

Meanwhile partition-test-shards.mjs --self-test reported max/mean 1.00x, and
every step of that job was green. A perfectly balanced split of stale weights
is still perfectly balanced. Nothing in the repo compared a prediction to an
outcome, so the only instrument that ever noticed was a job killed by the wall —
and a killed shard is precisely the state that yields no reading while the
rollup reads green (#16157 remains open on that half; it is not touched here).

Half 1 — the drift gate (code in, CI wiring withheld)

scripts/partition-test-shards.mjs gains --check-drift, which reads back the
summary a shard's --summarize has just written and reports:

shard-timing-drift: DRIFT -- Test Core (1/6), 1282.1s measured vs 507.4s predicted
across 5 package(s) = 2.53x (bound 1.5x).
  Heaviest overshoots:
    @objectstack/cli: predicted 458.1s, measured 1231.5s (2.69x, +773.4s)
    ...

Three properties worth review:

• It reuses the generator's extractor (samplesFromSummary) rather than
  parsing summaries a second time. A cache HIT replays a stored log in
  milliseconds; read as a measurement it says the suite got ~1000x faster than
  predicted, which would make a fully-cached shard vouch for a rotted dataset.
• The comparison is over an intersection. A package measured here but absent
  from the dataset was weighed by the test-file-count estimate, so charging it
  to the dataset would red on brand-new packages; it is excluded and named. A
  dataset entry this shard never ran is excluded too, or the ratio would become
  a function of the diff.
• NOT MEASURED is a first-class verdict, not a quiet pass. A shard whose
  test tasks were all replays has said nothing, and reporting that as OK is the
  check:react-declaration-parity 是唯一没接进任何 workflow 的源码审计门禁,且无 MANIFEST 时静默 skip 退出 0 —— 它现在永远不可能红 #4690 shape — a check that read nothing reporting as a check that found
  nothing wrong.

Why the bound is 1.5x

MAX_MEASURED_OVER_PREDICTED = 1.5, and the separation is measured rather than
picked: on the green merge_group build above, the five healthy shards top out
at 1.18x and the drifted one sits at 1.74x. 1.5 is the smallest round
factor that lands between them, and it also satisfies both ends the constant's
header states — comfortably below the 2.69x this card was filed on, and above
MAX_SHARD_OVER_MEAN (1.3), because a dataset accurate to within the balance
bound cannot be the thing that breaks balance.

Half 2 — sharding @objectstack/cli below package granularity

A correct refresh reds this script's own balance pins, by design. Substituting
the measured 1231.52s and re-partitioning the committed dataset:

bins 1232/716/714/714/714/714s   mean 801s   max/mean 1.54x  (bound 1.30x)
floor (heaviest single package) 1232s  >  1.3 x 801s = 1041s

Sharding is BY PACKAGE, so no split across six bins can put shard 1 under its
heaviest package — and pin 3 (partition-test-shards.mjs, #10149 / #4859) names
the only remedy itself: "Splitting that suite below package granularity, not a
different shard count, is the only thing that moves this." That is what this
does, in the shape the Dogfood job has run since #4859 — vitest's own
--shard=k/n, applied to one named package.

The arithmetic for n, derived rather than picked

n is the smallest integer for which one slice fits under the acceptance bound
against the mean the refresh produces. Slicing moves no weight in or out, so
with the measurement substituted the other 70 packages total 3572.66s and
the mean is fixed at (3572.66 + 1231.52) / 6 = 800.70s; the bound is
1.3 x 800.70 = 1040.91s:

n	one slice	vs the 1040.91s bound
1	1231.52s	over	RED — this is today
2	615.76s	under	the derived answer

and the split it produces is bins 801/801/801/801/801/800s, max/mean 1.00x,
floor 615.8s. Two is not a floor to sit on quietly either: solving
C/2 <= (1.3/6)(3572.66 + C) for the CLI's whole cost C says n = 2 holds until
that suite reaches ~2732s, a further 2.2x. Past that, pin 3 reds again naming
the floor, and the remedy is to raise n — never the bound.

FILE_SHARDED_PACKAGES in partition-test-shards.mjs is the one place this is
configured, and three self-test cases pin the derivation: the sliced refresh
meets the ratio bound, it meets the floor bound, and — the case that makes the
other two non-vacuous — the same refresh unsliced still breaches the floor.
If the CLI ever comes back under the bound on its own, that third case reds and
says the slicing is now a decision to revisit rather than leaving a pin that
cannot fail.

The one recorded objection to --shard, and why it does not reach here

partition-test-shards.mjs's header refuses vitest --shard workspace-wide,
and the reason is exact: on vitest 4 a package with fewer test files than the
shard count hard-fails, and --passWithNoTests converts that into running
nothing on every shard. Three workspace packages own one test file each.
Applied to one package with 268 it cannot arise — and rather than leave that
as a comment, sliceCountFor() refuses a configuration that would slice a
package below its own test-file count. It refuses; it does not clamp, because a
silently-reduced n hands back a split that balances a quantity CI cannot run.

What a shard item is now, and the four readers of it

An item is a package (@objectstack/spec) or a package plus a slice
(@objectstack/cli 1/2). formatShardItem / parseShardItem are the only place
that spelling is written or read, so the four consumers cannot drift apart:

1. ci.yml's turbo construction. --shard=k/n is passed through by turbo as
   a run-level argument, so it would reach every package in the run — which
   is the single-test-file failure above. The whole packages therefore share one
   invocation as they always have, and each slice gets its own, filtered to the
   one package. Each leg tees to its own log (run-with-stall-guard opens with
   'w', so two legs on one path would truncate the first), and the logs are
   concatenated even when a leg failed, because a red suite is when the
   completeness guard earns its keep. A failing leg stops the remaining ones,
   the way turbo stops scheduling on the first failure inside one run: carrying
   on would add a second full suite to a job already inside a 30-minute wall,
   turning an informative red into a killed job with no attestation — the
   CI: the shard-timings file is stale for the CLI package — 672s predicted vs 28m46s measured against a 30-minute timeout, so Test Core shard 1/6 is one slow run from being killed on any PR touching the CLI #16173 failure mode itself.
2. check-test-completeness.mjs --scheduled. Q2 is asked per package, and
   the reported set is keyed by the name turbo prints, which carries no slice.
   Items are folded to package names (scheduledPackages) and duplicates
   collapse. Unfolded, @objectstack/cli 1/2 reaches describe() as a name the
   turbo ls document has never listed and the guard refuses the whole shard —
   a false red, which is the machine that file's header spends its length
   warning about. A self-test case asserts the unfolded item still throws,
   so the pin proves the fold is what fixes it rather than describe() having
   gone lenient.
   parseRunCompleted now folds every roster line rather than the last: a
   shard with a slice runs two invocations, and a completed second one would
   otherwise vouch for a first that stopped early — which Rule B turns into a red
   on every package the abort left unreached.
3. --check-drift (code only in this PR — see Expected CI behaviour).
   A slice is charged its slice-sized prediction
   (predictedSecondsFor = dataset entry / n). Charging it the whole entry reads
   as a 0.50x under-run — measured, below — which would not only hide the drift
   but dilute any genuine overshoot elsewhere on the same shard.
   The n it divides by comes from what the summary records the run as
   (cliArguments), not from FILE_SHARDED_PACKAGES. The two agree on a Test
   Core shard and only the observed one is right anywhere else: a developer
   running the CLI suite locally runs it whole, and charging that a half-sized
   prediction reported a ~2x drift that was purely the arithmetic. Measured both
   ways — a local whole run at 470s against the 458.15s entry now reads
   OK ... 1.03x, where the config-driven version would have said 2.05x DRIFT.
   The config remains the default for callers with no run in hand.
4. The attestation roster: verified UNCHANGED, and not touched. The roster
   counts shard legs (--leg "test/6", --emit --job test --shard N --total 6). A sliced item changes what a leg attests to, not the leg count, so
   there are still six legs and check:shard-attestation and
   check:required-contexts are green with no edit. No job was renamed and the
   shard count is still 6.

The generator had to learn slices too — otherwise the fix re-creates the bug

The dataset stores each package's whole cost. Once the CLI is sharded, a
green queue build leaves n summaries each holding ~1/n of it, and the median
rule — correct for repeat measurements of one package — would write one
slice's duration as the whole package's weight. That is #16173 again, in the
same silent direction, caused by the fix for it.

turbo records a run's passthrough argv on every task record as cliArguments.
Verified on turbo 2.10.10 against a real executed summary, not only a
--dry=json plan:

pnpm exec turbo run test --filter=@objectstack/types --only --summarize -- --shard=1/2
  -> summary task: "cliArguments": ["--shard=1/2"]
  -> child really received: pnpm run test --shard=1/2

So measure-test-shard-timings.mjs reassembles a package's slices — sum
within a run, median across runs, the two rules composed — and an incomplete
slice set is never summed: summing 1 of 2 slices would record half a suite as
the whole of it. The package drops out of packages (falling back to the
test-file estimate like any unmeasured package), is named in a new
skippedIncompleteSlices field, and raises a loud warning, so a refresh built
on five of six artifacts is visible in the file rather than inferred later from
the split going strange.

The dataset refresh — why it is not here, and the recipe for it

The refresh is blocked on the channel, not on the data. The six
test-core-run-summary-N-of-6 artifacts of green merge_group run
34013842594 exist and are unexpired (ids 9983534660 / 9983445661 /
9983395767 / 9983459531 / 9983425025 / 9983454944), but
GET /actions/artifacts/{id}/zip redirects to a blob host this session's egress
policy denies — curl exit 56, CONNECT tunnel failed, response 403. Probed
once here and not retried (a policy denial is not a transient). The generator's
other documented path, a local full turbo run test, is ~4800s of contended
suite on the shared 4-vCPU agent box — an order of magnitude past the foreground
budget, and it would monopolise a box several agents share. Hand-editing the
dataset is refused by the file's own contract and by the ruling on #16173.

So the mechanism ships validated end-to-end through the generator's own code
path on synthetic samples (six summaries mirroring the sliced layout; the two
CLI slices at 615.80s and 615.72s are reassembled to exactly 1231.52), and the
refresh is one local command for someone whose network can reach the artifacts:

gh run download 34013842594 -R objectstack-ai/objectstack \
  -p 'test-core-run-summary-*-of-6' -D /tmp/shard-summaries
node scripts/measure-test-shard-timings.mjs /tmp/shard-summaries/*/*.json \
  --out scripts/test-shard-timings.json
node scripts/partition-test-shards.mjs --self-test    # must stay green

⚠️ Use a run whose six shards all uploaded a summary — the generator now says
so out loud if a slice set is short, and with the CLI sliced a missing artifact
costs the heaviest package in the workspace its measured weight.

Either side of this PR works, and the generator tells them apart on its own: run
34013842594 PREDATES the slicing, so the CLI appears there as one whole sample and
is recorded directly; a run from after this lands carries two CLI slices, which the
generator sums back into one whole-package weight before the median rule sees it.

The upload step is unchanged (path: .turbo/runs/), so a shard that carries a
slice now publishes two summaries in its artifact — one per turbo invocation.
That needs no special handling and the glob above already takes both: the whole
packages come from one, the slice from the other, and the generator joins them.

What the split alone buys, before the dataset is refreshed at all

Worth stating because the affected-set radius is wider than the card's title
suggests — turbo ls --affected lists 75 of 78 packages for a packages/spec
change, so the CLI is on nearly every PR's shard set, not only the CLI-touching
ones. Arithmetic (not a CI measurement): take each bin's predicted total and
substitute the CLI's real 1231.52s for whatever the stale dataset predicted for
its slice(s):

TODAY   (stale dataset, CLI unsliced)     actual bins 1445*/672/671/672/672/672s
                                          worst shard 1445s = 24.1 min = 80% of the wall
THIS PR (stale dataset, CLI sliced 1/2+2/2) actual bins 672/672/672/671/1059*/1058*s
                                          worst shard 1059s = 17.6 min = 59% of the wall

The split is what moves that number, and it moves it without the refresh,
because the slices are placed by weight like any other item and the CLI's real
cost is halved across two runners instead of falling on one. It does not make
the PR landable on its own — see below — but it is why the split is not merely
bookkeeping ahead of the refresh.

Expected CI behaviour

⭐ A balanced shard set and NO drift red. The drift step is not wired in this
PR, so nothing here can red on the stale dataset; what merges is the split, and
the shard that used to carry the whole CLI now carries half of it. On a
merge_group run expect six Test Core legs of comparable length, the CLI's two
slices on separate shards (measured here: 1/2 on shard 5, 2/2 on shard 6), and
the cancelled-shard-1/6 pattern gone.

⛔ What this PR deliberately does NOT do: it does not make the dataset honest.
scripts/test-shard-timings.json still says 458.15s for a suite that costs
1231.52s, and nothing reds about that until the follow-up below. That is the
trade this sequencing accepts: the silent-rot detector waits one commit so the
active shard-kill hazard stops today.

The two follow-ups, in this order

1. Refresh the dataset with the recipe above. The partitioner stays green on
   the refreshed numbers — verified here by substituting the measurement: bins
   801/801/801/801/801/800s, max/mean 1.00x, and a CLI slice at 615.76s against
   a 1040.91s bound.
2. Wire the drift step. Restore a step in the Test Core job running
   --check-drift over .turbo/runs/*.json with --label "Test Core (${{ matrix.shard }}/6)", no
   if: and no continue-on-error, placed above the attestation pair.
   ci.yml carries a comment at exactly that spot spelling this out, so the
   mode does not read as forgotten. Verified it goes green once the dataset is
   true: the same sliced shard reads OK ... 1.000x against a refreshed dataset.

#16157 remains open (the rollup that reads green over a cancelled shard) and
#16222 remains open (no scheduled refresh); neither is addressed here.

Verification

• node scripts/partition-test-shards.mjs --self-test — exit 0, 9 batteries,
  71 measured packages -> 72 shard items, floor 404s, max/mean 1.00x. The floor
  moved from 458s to 404s because the CLI is no longer the heaviest single item.
• node scripts/measure-test-shard-timings.mjs --self-test — exit 0 (34 cases).
• node scripts/check-test-completeness.mjs --self-test — exit 0 (79 cases).
• The two halves agree, driven end to end. The sliced shard against the
  stale dataset: DRIFT ... @objectstack/cli: predicted 229.1s, measured 615.8s (2.69x) — the slice charged 458.15/2, and the same 2.69x the card was
  filed on. Against a refreshed dataset the identical reading is OK 1.000x.
  An all-replayed sliced shard is NOT MEASURED, exit 0.
• The ci.yml step's shell executed against four shard shapes with a stubbed
  runner: mixed (one whole-packages leg + one -- --shard=1/2 leg filtered to
  the CLI), slice-only, packages-only (byte-identical argv to today's), and a
  failing first leg (second leg not run, combined log still assembled, status
  propagated).
• Ablation, three legs, on the committed tree. Each mutation was proved to
  reach disk by a removed-text / injected-marker grep-count pair (an editor's
  exit code is 0 on zero matches), and each restore proved by blob hash against
  HEAD plus an empty git diff HEAD — never by an exit code. Every leg turned
  red, and two of them also moved the production path, which is what shows
  the pins guard the shipped behaviour rather than themselves:
  • delete the new battery's opener ⇒ self-test battery "file-level slice items (#16173)" DID NOT RUN — 0 cases registered, 18 pinned.
  • make expandSlices stop slicing ⇒ self-test reds on the slice-derivation
    pin, and the scheduled items go from @objectstack/cli 1/2 on shard 5 +
    2/2 on shard 6 back to the whole @objectstack/cli alone on shard 1 —
    CI: the shard-timings file is stale for the CLI package — 672s predicted vs 28m46s measured against a 30-minute timeout, so Test Core shard 1/6 is one slow run from being killed on any PR touching the CLI #16173's exact state.
  • make the generator forget slices ⇒ self-test reds, and regenerating from
    the six synthetic summaries records @objectstack/cli at 615.76s instead
    of 1231.52 — the silent halving, reproduced on demand.
• The 52-command gate union (re-run in full at head d911e6b55, after the
  drift step was withheld) derived by scripts/pm/dispatch-gates.mjs --changed --commands and reconciled with --ran, plus the two self-tests the edited
  scripts owe; verdicts in the report on CI: the shard-timings file is stale for the CLI package — 672s predicted vs 28m46s measured against a 30-minute timeout, so Test Core shard 1/6 is one slow run from being killed on any PR touching the CLI #16173. The three self-tests are green at
  that head: 71 measured packages -> 72 shard items ... floor 404s, 34 generator
  cases, 79 completeness cases. check-step-collectors correctly drops 415 -> 414
  run: steps, which is the removed drift step and nothing else.

skip-changeset: nothing on this diff ships from a package.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vbw3RPgdtqesx4azk9SbW8

---

Generated by Claude Code

[claude]

Parked as draft — accepted half, not yet landable (PM seat domain:devx @ objectstack, 2026-09-06T06:5xZ). This PR's pull_request CI is green only because the --affected subset excludes @objectstack/cli; on a merge_group run the new --check-drift gate will red Test Core (1/6) with shard-timing-drift: DRIFT — … @objectstack/cli … 2.69x — that is the gate working on a real drift, so this PR must not enter the queue before the refresh and the CLI split land. Expected-red list (landing-operations C): exactly that one verdict on shard 1/6 of a merge_group run. Release condition: the continuation on this same branch adds the CLI file-level sharding and the dataset refresh (ruling on #16173), after which the body's Part of #16173 becomes Fixes #16173 and the PR is flipped by the PM. ⛔ Not to be flipped ready or armed before then. #16157 remains open and untouched.

---

Generated by Claude Code