Shard the CI test suite

[yihanzhu]

Closes #269

Claim / tuple

• claim_id: claim-ci-test-shards-20260911T035451Z-0c87a3c4
• plan blob 9827a9b8ed7eab7c3247699d31bf2a586914b45b (frontmatter spec-blob: 1d419e187315879e2bc52366b2be11a43b511ad4)
• spec blob 1d419e187315879e2bc52366b2be11a43b511ad4 (frontmatter intent-blob: e56b92429ca56a638827ebf7d0e270ffa2bd1089, risk: high)
• intent blob e56b92429ca56a638827ebf7d0e270ffa2bd1089
• plan-base = current base: e62b90ba07c905e2d31a8a0f47eb5c63431b9d19
• review_size: standard — measured net line count (git diff --shortstat e62b90b..HEAD, current head 0a4e5ee): 376 insertions(+), 7 deletions(-) = 369 net lines, inside the 300-400 budget the plan claims (no exception).

What changed

Adds --shard <index>/<count> and --list to scripts/test/run-all.sh, a new
scripts/test/run-all-sharding.check.sh that proves the partition independently
of the runner, a manifest line, a RESTORE.md docs update, and
proposals/ci-test-shards-shard-ci.patch — the proposed (not applied) unified
diff for the two operator-owned constitution paths (.github/workflows/ci.yml,
AGENTS.md). Per the spec, this initiative is risk: high and agents never
write those two files directly; see "Operator step (Step 6)" below.

Round 1 — fix for round-0 review

The round-0 review (gpt-5.5) raised one P2 and one P3 finding.

• [P2] "Activate the sharded workflow instead of only proposing it." No code
  change — replied on the PR
  (Shard the CI test suite #286 (comment)) with the
  rationale: this is the accepted plan's Step 5/6 sequence by design (agent-authored
  files open the PR first so the still-serial workflow produces the required
  no-argument H0 run; the operator applies the sharded workflow patch as the last
  permanent change before the final review at HF).
• [P3] "Reject oversized numeric shard selectors." Real bug, fixed in commit
  0a4e5ee: the selector regex was ^[1-9][0-9]*/[1-9][0-9]*$, unbounded in digit
  count, so a value like --shard 1/999999999999999999999999 passed the shape check
  and then hit [ "$count" -gt 16 ], which errors with "integer expression expected"
  on a number too large for [ to parse instead of refusing cleanly. Fixed by capping
  the shape to at most two digits per side (^[1-9][0-9]?/[1-9][0-9]?$) before any
  arithmetic — count is bounded at 16 by the spec, so two digits always suffice, and a
  value that short can never overflow the subsequent range checks. Applies identically
  to the --shard flag and YSTACK_TEST_SHARD, since both funnel through the same
  shard_value check. Added the oversized selector (both operand positions) and a
  leading-zero selector (01/6) to run-all-sharding.check.sh's refusal list,
  exercised as both the flag and the environment variable.

Because the fix touches scripts/test/run-all.sh and
scripts/test/run-all-sharding.check.sh — both under scripts/test — the H0
tree-identity binding from Step 5 no longer holds against the original H0
(c38d0e0). Per plan Step 6's second precondition, H0 moves to this round's new
head; see the updated Proof section and Identity records below. The workflow is still
unchanged (serial) at this head, so the new H0's no-argument CI run is still
producible — the manager will fill it in once this round's CI run completes.

Step 0 — proof script, pre-fix and post-fix

Pre-fix, against the unchanged runner (before Step 1's edit), commit 0c1828a:

$ bash scripts/test/run-all-sharding.check.sh
error: run-all.sh does not implement --shard/--list yet
$ echo $?
2

Exit 2, under a second, no ==> header (stdout was empty) — the refusal that
proves the script tests something.

Post-fix, at the current head 0a4e5ee:

$ bash scripts/test/run-all-sharding.check.sh
workflow is still serial: no --shard run line in ci.yml

assertions passed: 202
assertions failed: 0
sharding proof: all checks passed

Exit 0. GITHUB_ACTIONS is unset locally, so assertion 8 takes the "still serial"
branch and passes — this is the local serial line referenced in Proof item 13 below,
to be paired there with CI's post-Step-6 equality line. (196 → 202 assertions: round 1
added the oversized-selector and leading-zero refusal cases, each run as both the flag
and YSTACK_TEST_SHARD.)

Proof (all run at commit 0a4e5ee, repo root, on ystack/impl/ci-test-shards)

1. Focused proof — see Step 0 above (both runs).
2. Discovery and ordering untouched (R2, R3).

   $ diff <(bash scripts/test/run-all.sh --list) \
          <(find "$root/scripts/test" -maxdepth 1 -type f -name '*.test.sh' -print | LC_ALL=C sort | sed "s|^$root/||")
   (no output, exit 0)
   $ bash scripts/test/run-all.sh --list | wc -l
   62
   

3. Refusals, and the flag beating a malformed variable (R1, R5). All of
   0/4 5/4 9/6 a/b 1/0 1/17 1 /4 4/ '' 01/6 1/999999999999999999999999 999999999999999999999999/6
   (and a bare --shard) refuse with exit 2, empty stdout, and the exact usage line
   on stderr — as the flag and as YSTACK_TEST_SHARD. Confirmed both directions:

   $ bash scripts/test/run-all.sh --shard 1/999999999999999999999999 --list; echo $?
   usage: run-all.sh [--shard <index>/<count>] [--list] (1 <= index <= count <= 16)
   2
   $ bash scripts/test/run-all.sh --shard 999999999999999999999999/6 --list; echo $?
   usage: run-all.sh [--shard <index>/<count>] [--list] (1 <= index <= count <= 16)
   2
   $ bash scripts/test/run-all.sh --shard 01/6 --list; echo $?
   usage: run-all.sh [--shard <index>/<count>] [--list] (1 <= index <= count <= 16)
   2
   

   All of this is also asserted automatically by the proof script (item 1, 202
   assertions).
4. Selection, membership and partition (R3, R7).

   $ bash scripts/test/run-all.sh --shard 1/6 --list | wc -l
   11
   $ bash scripts/test/run-all.sh --shard 3/6 --list | wc -l
   10
   $ bash scripts/test/run-all.sh --shard 1/1 --list | wc -l
   62
   $ diff <(bash scripts/test/run-all.sh --shard 3/6 --list) \
          <(bash scripts/test/run-all.sh --list | awk '((NR - 1) % 6) + 1 == 3')
   (no output, exit 0)
   

   Split over today's 62 suites: 11, 11, 10, 10, 10, 10.
5. The three pins still pass (R12).

   $ bash scripts/test/portable-core-result-facts.test.sh; echo $?
   0
   $ bash scripts/test/portable-core-stage-request.test.sh; echo $?
   0
   $ bash scripts/test/portable-core-result-truth.test.sh; echo $?
   0
   

6. Lint at the pinned version.

   $ shellcheck --version | grep version:
   version: 0.11.0
   $ shellcheck -x -S style scripts/test/run-all.sh scripts/test/run-all-sharding.check.sh
   (no output, exit 0)
   

7. Schema guard. bash scripts/test/portable-core-schema.test.sh → failures: 0, exit 0.
8. Rename gate. bash scripts/check-rename.sh → clean, exit 0.
9. The manifest entry (R9).

   $ tail -1 ci/required-files.txt
   scripts/test/run-all-sharding.check.sh
   $ [ -x scripts/test/run-all-sharding.check.sh ] && echo ok
   ok
   

10. The patch applies (R10, R13).

    $ git apply --check proposals/ci-test-shards-shard-ci.patch; echo $?
    0
    

    (Also verified: applying it to a scratch copy reproduces the intended
    ci.yml/AGENTS.md content exactly, and re-running the sharding proof
    against that scratch copy moves assertion 8 from "still serial" to a
    passing shard-count-equals-matrix check — and correctly fails it when the
    matrix or run line is deliberately mismatched.)
11. Scope. git diff --stat e62b90b lists exactly five paths:
    scripts/test/run-all.sh, scripts/test/run-all-sharding.check.sh,
    ci/required-files.txt, RESTORE.md, proposals/ci-test-shards-shard-ci.patch.
12. The full no-argument run, bound to the final head (R2). H0 moved to this
    round's head because the fix touches scripts/test. Filled in by the manager
    once this round's PR CI run (under the still-unchanged workflow) completes:
    • Run URL: https://github.com/yihanzhu/ystack/actions/runs/34562109978
    • Commit it ran on (H0): 0a4e5eecdf0a12fe61db0021eea17fdffcfef114
    • ci job status: success (62 ==> headers, all 62 test scripts passed)
    • Wall-clock time: 83 min (04:24:58Z → 05:47:58Z)
    • git rev-parse H0:scripts/test = fb6410afbda9e7b41c863db7ae1943142cd569ee
    • git rev-parse HF:scripts/test = <TODO, recorded once HF is known>
    • (These two tree ids must be identical at review time.)
13. The gate keeps its name and meaning (R11, R14), shard count vs. matrix (R10).
    Filled in after Step 6 (operator's commit) lands and a sharded CI run
    completes: check list (ci, checks, test (1)..test (6)), ci
    status, wall-clock duration, and the CI equality line from assertion 8 —
    to be pasted beside item 1's local "still serial" line above.
14. A failing shard turns ci red, then removed again (R11). Filled in
    after Step 6, per the plan's red-shard procedure — HR/HD commits, run
    URL, job list, and the identity records (git diff --name-status HR HD,
    HD:scripts/test vs HF:scripts/test, HR:.github/workflows/ci.yml vs
    HF:.github/workflows/ci.yml).

Operator step (Step 6)

This PR's agent-authored commits are complete and proved above. Per the
spec (risk: high), the two constitution paths — .github/workflows/ci.yml
and AGENTS.md — are the operator's to apply and commit, as the last
permanent file change on this branch, only after proof item 12 above is
filled in (the no-argument run must happen while the workflow here is still
serial):

git apply proposals/ci-test-shards-shard-ci.patch
git diff   # review the two files
git add .github/workflows/ci.yml AGENTS.md
git commit -m "Apply sharded CI workflow (Step 6)"
git push

After that lands and a sharded run goes green (target: under 25 minutes,
recorded in item 13), the red-shard proof of R11 follows: add
scripts/test/zz-red.test.sh (#!/usr/bin/env bash / exit 1), record the
red run (test (3) red, the other five green, checks green, ci red — not
skipped), then push a second commit deleting it and wait for green again. Full
procedure and exact identity records in work/ci-test-shards/plan.md Step 6.

Identity records (placeholders — manager fills in as heads accumulate)

• H0 (this PR's first CI run under the current head, old serial workflow): 0a4e5eecdf0a12fe61db0021eea17fdffcfef114
• HF (final head): <TODO>
• HR (red-shard add commit): <TODO>
• HD (red-shard delete commit): <TODO>

[yihanzhu]

Codex reviewer (cross-vendor, read-only)

Reviewed-head: c38d0e0
Reviewed-base: e62b90b
reviewer: gpt-5.5 @ high

Posted verbatim by codex-review.sh (codex exec review --json --base refs/codex-review/286-7713/base in an isolated temp worktree, sandbox forced read-only). Comments only — Codex never pushes, approves, or merges.

The patch does not actually activate the sharded CI workflow it documents, so the primary behavior remains unchanged. There is also a selector validation edge case that violates the stated refusal contract.

Full review comments:

• [P2] Activate the sharded workflow instead of only proposing it — /var/folders/rm/bkx4f4b91vqfn79vk02pcd140000gn/T/tmp.F86h6F4lwA/RESTORE.md:416-420
  When this branch lands as shown, GitHub Actions will still run the old single ci job because .github/workflows/ci.yml is unchanged; the sharded workflow exists only as inert text under proposals/, so the six shards and sharding proof never run while this RESTORE text says they do. The accepted spec requires the real workflow to gain the checks, matrix test, and aggregate ci jobs (work/ci-test-shards/spec.md:142-167), so either the operator-owned workflow change needs to be applied before merge or this doc/feature claim remains false.

• [P3] Reject oversized numeric shard selectors — /var/folders/rm/bkx4f4b91vqfn79vk02pcd140000gn/T/tmp.F86h6F4lwA/scripts/test/run-all.sh:55-56
  If a caller passes a numeric selector too large for [ to parse, such as --shard 1/999999999999999999999999, this comparison emits integer expression expected and falls through instead of printing the exact usage line and exiting 2; with a huge count it can even proceed to select suites. The spec requires 1 <= index <= count <= 16 and malformed selectors to refuse cleanly (work/ci-test-shards/spec.md:32-35, work/ci-test-shards/spec.md:72-79), so cap the numeric shape before arithmetic or validate with arithmetic that cannot overflow this path.

[yihanzhu]

Fix claim

• claim_id: fixclaim-ci-test-shards-r0-20260911
• mode: fix
• gate_mode: artifact-high · risk: high · branch_state: existing · review_size: standard
• repo: yihanzhu/ystack · branch: ystack/impl/ci-test-shards · PR: Shard the CI test suite #286 (open) · remote head: c38d0e0acf3a84c280af85431b715798910e16ad · base: e62b90ba07c905e2d31a8a0f47eb5c63431b9d19 · round: 0
• plan blob 9827a9b8ed7eab7c3247699d31bf2a586914b45b · spec blob 1d419e187315879e2bc52366b2be11a43b511ad4 · intent blob e56b92429ca56a638827ebf7d0e270ffa2bd1089 · plan-base e62b90ba07c905e2d31a8a0f47eb5c63431b9d19
• gh operator: yihanzhu · manager: yshifu · coder tier: sonnet

yshifu diagnosis of the round-0 review (one P2, one P3):

• P2 "activate the sharded workflow": expected at this head by design. The accepted plan (Step 5/6) opens the PR with the five agent-authored files first so the unchanged workflow produces the required no-argument run (H0), and the operator applies proposals/ci-test-shards-shard-ci.patch (.github/workflows/ci.yml, AGENTS.md) as the last permanent change BEFORE the final review and merge. No code change in this round; the fix coder replies with that rationale and points at the plan; the final review at HF sees the real workflow.
• P3 oversized numeric selector: real. [ … -le … ] on a number too large for the shell emits "integer expression expected" and falls through. Fix: cap the numeric shape before any arithmetic (index and count each match ^[1-9][0-9]?$ — the spec bounds count at 16 — then the range checks), so every malformed selector prints the exact usage line and exits 2; add the oversized case to the proof script's refusal list (flag and env forms). Round bumps to 1; re-review follows; H0 moves to the new head (the no-argument run under the unchanged workflow re-runs there).

[cloudflare-workers-and-pages]

Deploying ystack with    Cloudflare Pages

Latest commit:	0a4e5ee
Status:	✅  Deploy successful!
Preview URL:	https://753868a7.fabrica-6yx.pages.dev
Branch Preview URL:	https://ystack-impl-ci-test-shards.fabrica-6yx.pages.dev

View logs

[yihanzhu]

Reply to round-0 [P2] "Activate the sharded workflow instead of only proposing it"

No code change for this finding — it's expected at this head by design, not an oversight.

This initiative is risk: high because it touches two constitution paths
(.github/workflows/ci.yml, AGENTS.md), which unattended agents never write. The
accepted plan sequences the work explicitly around that, work/ci-test-shards/plan.md
Order of work:

2. The PR is opened. Its first CI run is still the old serial workflow, so it is the
   no-argument run; record it and the commit it ran on, H0 (Step 5).
3. The operator applies the patch and commits .github/workflows/ci.yml and
   AGENTS.md. This is the last permanent file change on the branch (Step 6).

And Step 5 itself:

Push and open the PR here, before the operator's commit. That order is required,
not a preference. The branch head still carries today's single serial ci job, so
the PR's first CI run is the old shape and it runs bash scripts/test/run-all.sh
with no argument. That is the only place the no-argument run happens end to end, and
R2's byte-for-byte guarantee rests on it.

The spec's Order-of-work step 5 (work/ci-test-shards/spec.md) states the same
division of labor:

5. Both operator-owned files, in the operator's own commit — the last permanent
   file change on the branch: .github/workflows/ci.yml and the AGENTS.md
   bullet of requirement 13. Agents write neither file — the RESTORE.md half of
   requirement 13 is theirs, in step 3; the AGENTS.md half is not.

So at this head, .github/workflows/ci.yml is correctly unchanged: the five
agent-authored files (runner, proof script, manifest line, RESTORE.md, and the
proposals/ci-test-shards-shard-ci.patch text) are complete and proved, and the
sharded workflow exists only as proposals/ci-test-shards-shard-ci.patch until the
operator applies it per Step 6. That ordering is required so the PR's first CI run
(H0) can be the old-shape no-argument run the R2 byte-for-byte guarantee depends on
— once the operator's commit lands, no head on this branch can produce that run again.

The final review at HF (after Step 6 and the red-shard proof) will see the real
sharded workflow with the checks, test (6-way matrix), and aggregate ci jobs
live, and the PR body's identity records (H0/HF scripts/test tree ids, the CI
equality line) tie the two states together. This round's fix is scoped to the [P3]
finding only (oversized numeric shard selectors); see the separate commit and the
updated PR body for that.

[yihanzhu]

Fix-round summary (round-0 → round-1)

Implemented:

• [P3] Oversized numeric shard selectors now refuse cleanly. Capped the selector shape
  to at most two digits per side (^[1-9][0-9]?/[1-9][0-9]?$) before any arithmetic,
  since count is bounded at 16 by the spec. Previously --shard 1/999999999999999999999999
  passed the unbounded [1-9][0-9]* shape check and then hit [ "$count" -gt 16 ],
  which errors on a number too large for [ to parse instead of exiting 2 with the
  usage line. Fixed in scripts/test/run-all.sh; applies identically to --shard and
  YSTACK_TEST_SHARD (shared code path). Added the oversized case (both operand
  positions) and a leading-zero case (01/6) to run-all-sharding.check.sh's refusal
  list, each exercised as flag and env-var form.

Pushed back on (no code change):

• [P2] "Activate the sharded workflow instead of only proposing it" — this is expected
  at this head by design (accepted plan Step 5/6): agent-authored files open the PR
  first so the still-serial workflow produces the required no-argument H0 run; the
  operator applies proposals/ci-test-shards-shard-ci.patch as the last permanent
  change before the final review at HF. Full rationale, quoting the plan and spec,
  posted separately: Shard the CI test suite #286 (comment)

New head: 0a4e5eecdf0a12fe61db0021eea17fdffcfef114 (parent c38d0e0).

Proof at the new head (full outputs in the updated PR body):

• Sharding proof script: 202/202 assertions passed (up from 196; +6 for the three new
  refusal values × flag/env forms), exit 0.
• Manual repro of the three new refusal cases: exit 2, usage line, no suite run — for
  1/999999999999999999999999, 999999999999999999999999/6, 01/6.
• Three pinning suites (portable-core-result-facts, portable-core-stage-request,
  portable-core-result-truth): all pass, exit 0.
• shellcheck -x -S style at the pinned 0.11.0: clean.
• Schema guard, rename gate, manifest entry, patch --check: all clean/pass.
• No-argument path: discovery/order untouched — --list output still matches
  find ... | sort byte for byte, still 62 suites. (The full 80-90 minute no-argument
  run itself is not run locally; that evidence is the PR's own CI, per Step 5/6.)
• Net line count: 376 insertions(+), 7 deletions(-) = 369 net lines, inside the
  300-400 budget.

Note: because the fix touches scripts/test/run-all.sh and
scripts/test/run-all-sharding.check.sh, both under scripts/test, the H0
tree-identity binding from Step 5 no longer holds against the original H0 (c38d0e0).
H0 now points at this round's head; PR body updated accordingly with TODO
placeholders for the run URL/duration once this round's CI completes.

Labels: round-1, claimed (round-0 removed, needs-human not added).

[yihanzhu]

Codex reviewer (cross-vendor, read-only)

Reviewed-head: 0a4e5ee
Reviewed-base: e62b90b
reviewer: gpt-5.5 @ high

Posted verbatim by codex-review.sh (codex exec review --json --base refs/codex-review/286-76134/base in an isolated temp worktree, sandbox forced read-only). Comments only — Codex never pushes, approves, or merges.

The branch does not apply the workflow change required to actually shard CI, so the central feature is absent and the documentation is inaccurate.

Review comment:

• [P1] Apply the sharded workflow before merging — /private/var/folders/rm/bkx4f4b91vqfn79vk02pcd140000gn/T/tmp.tH6nzzjTuD/proposals/ci-test-shards-shard-ci.patch:45-45
  This sharded test invocation only appears inside the proposal file; the PR's actual .github/workflows/ci.yml is unchanged and still has the old single ci job running bash scripts/test/run-all.sh, so merging this head will not create the six shards or run the sharding proof. The accepted plan requires the operator-owned workflow/AGENTS edits to be applied on this branch from the proposal (work/ci-test-shards/plan.md:38-44), so the feature and the updated RESTORE text stay false until that patch is applied.

[yihanzhu]

Step 5 evidence recorded; Step 6 is yours

@yihanzhu — operator step (plan work/ci-test-shards/plan.md Step 6). Label: needs-human.

Step 5 record (the no-argument run under the unchanged workflow):

• H0 = 0a4e5eecdf0a12fe61db0021eea17fdffcfef114 · H0:scripts/test tree = fb6410afbda9e7b41c863db7ae1943142cd569ee
• run: https://github.com/yihanzhu/ystack/actions/runs/34562109978 · ci job success · 04:24:58Z → 05:47:58Z (83 min) · 62 ==> headers · all 62 test scripts passed
• round-1 review at H0 (gpt-5.5): the only finding is the expected one — the workflow is not yet applied, which is this step.

What you do (the two constitution files; nothing else changes):

cd ~/git/ystack && git fetch origin && git checkout ystack/impl/ci-test-shards && git pull --ff-only origin ystack/impl/ci-test-shards
git apply --check proposals/ci-test-shards-shard-ci.patch && git apply proposals/ci-test-shards-shard-ci.patch
git diff --stat   # expect exactly .github/workflows/ci.yml and AGENTS.md
git add .github/workflows/ci.yml AGENTS.md
git commit -m "ci: shard the test suite into six jobs behind one aggregate ci gate (operator commit, plan Step 6)"
git push origin ystack/impl/ci-test-shards

Then reply here with the commit SHA (or just "done"). I take it from there: the red-shard proof commits (add, record, delete), the identity records (HR/HD/HF), the final review at HF, and the merge.

Do not merge this PR yourself yet — the proof commits come after your commit.