[Mikola Lysenko (mikolalysenko)]

Self-picked from the CI performance dashboard (#1182). No ci-perf issue was eligible this run: #1670, #1512 and #1248 are settings issues, and #1720 is agent:needs-human. This retries #1768 the way its closing comment asked, with a Linux coverage timing sample.

Problem

coverage (2) sets the merge-queue critical path: it finished last in 9 of 10 merge_group runs on the dashboard at 10-29 00:16. The coverage shards use the count split, and shard 2 is much slower than shard 1:

run	coverage (1) job / test step	coverage (2) job / test step
mg 40782886803	3.40 min / 179 s	5.98 / 335 s
mg 40775177102	3.98 / 176	6.35 / 307
this PR, count split (40801155956)	3.27 / 173	4.52 / 248

#1768 balanced on scripts/ci-test-durations.json and saved nothing, because that sample was measured on Windows and Windows times don't predict instrumented Linux runs. Two examples: e2e_vendor_pypi_build takes 35–72 s under Linux coverage but 0.0 s in the Windows sample, and e2e_vex_lockfile takes 11–42 s on Linux vs 119 s.

Change

• scripts/ci-test-shard.py forwards cargo's merged output byte for byte and records each test binary's finished in time. At the end of the shard it prints one ci-test-shard: timings {...} JSON line with the build time. The log API returns only the last 5,000 lines, so this is the only way to read a full sample back. It also applies to the Windows, macOS and release shards, which makes refreshing their samples possible too.
• New scripts/ci-coverage-durations.json: all 234 targets, from this PR's first coverage run (40801155956). unit_seconds covers the unit-test runtime plus building the unit-test harnesses, and was tuned once against run 2. compile_seconds is 0.3, fitted from the two shards' build and report times.
• coverage partitions on that sample, using the longest-first split the Windows test shards already use, unless CI_TEST_TIMINGS is set. An empty CI_TEST_TIMINGS keeps the count split.
• Review fixes from /code-review high:
  • Raw bytes are passed through, so Windows can't fail with UnicodeEncodeError on cp1252.
  • The runner reads cargo's pipe for at most 10 s after cargo exits, so a daemon left holding the pipe can't hang the shard.
  • ANSI colour is stripped before matching.
  • Tests no longer print to the console.
• No workflow file changes, because Retry setup-python on a transient download failure #1803 is open on ci.yml.

Expected saving

Balancing the shards' test steps at about (173 + 248) / 2 ≈ 210 s should cut ~0.5 min per PR and merge-queue run on fast hosts. Slow hosts should save more, since shard 2 had the slow, network-bound targets: in the mg runs above, its step took 307–335 s. Job-minutes stay about the same.

Measured result

Same PR, same workflow, fast Depot hosts:

run	split	coverage (1) job / step	coverage (2) job / step	slowest coverage job	run created → done
40801155956	count	3.27 / 173 s	4.52 / 248 s	4.52	4.80 min
40802064188	Linux sample v1	4.23 / 229 s	3.80 / 202 s	4.23	5.07 min
40802643957	Linux sample v2 (unit_seconds +27)	3.75 / 201 s	4.15 / 223 s	4.15	4.45 min
40804462397	v2 + drain fix (489db94)	3.80 / 203 s	4.55 / 245 s	4.55	4.80 min

• Result: with the sample, the slowest coverage job averaged 4.31 min over 3 runs (4.23, 4.15, 4.55). The count split took 4.52 min on this PR, so the saving is ~0.2 min. That is smaller than the 0.5 min expected, but each balanced run beat or matched the count split. Coverage job-minutes stayed about the same (7.79 → 7.90–8.35).
• Run-to-run noise: shard 1 is now steady (201–229 s), but shard 2 varies by about 40 s (202–245 s). Its network-bound e2e builds drive that. In run 4 they took 16 s more than in the sample: e2e_vendor_pypi_build 39.9 vs 34.9 s, e2e_vendor_pnpm_build 15.2 vs 12.6 s. The saving should be larger on slow hosts, where shard 2 took 307–335 s with the count split.
• Still to verify: merge_group effects show only after merge. The profiler will check that coverage (2) no longer sets the mg critical path and that mg ci-ok p50 (6.53 at 10-29 00:16) drops.

Where each test runs

Unchanged. Every integration target still runs in exactly one coverage shard (test_ci_test_shard.py). Targets added after the sample fall back to default_seconds, and a test fails once more than 10% of targets are missing from the sample.

Review findings not fixed

• Hyphenated target names would be recorded under the binary's underscore name. No target has a hyphen today (cargo metadata).
• The doctest invocation's time isn't in the timings line. Coverage runs no doctests.
• The sample is a single run and was refreshed by hand. The measured result above is the check on it. If the noisy, network-bound targets drift, refresh it from any coverage log's timings line.

Risk

Low. Exit statuses are unchanged and all 380 scripts/tests tests pass. All three CI runs on this PR were green.

https://claude.ai/code/session_011f9aeQpFXjqvowuU1RFbdQ

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 489db94. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Ready for review at 489db94.

• CI: 50/50 checks green (31 success, 19 skipped) on 489db94; mergeable, no conflicts.
• Bugbot: reviewed 489db94; its one thread (scripts/ci-test-shard.py pipe drain) was fixed and is resolved.
• Reviewer focus: the coverage split now depends on the hand-refreshed scripts/ci-coverage-durations.json sample; measured saving is ~0.2 min on fast hosts (see the table in the description).

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Final review brief

What it does: Coverage shards were split by test-target count, which left coverage (2) on the merge-queue critical path. This adds a Linux coverage runtime sample (scripts/ci-coverage-durations.json) and makes the coverage shards use the same longest-first split the Windows test shards already use. ci-test-shard.py now also streams cargo's output through a reader that records each test binary's finished in time and prints a ci-test-shard: timings {...} line, so the sample can be refreshed from any shard log. No workflow files change.

Risk: Low. It's CI-only. Exit statuses are unchanged, and each target still runs in exactly one shard. The only behaviour change outside coverage is that cargo's stderr is now merged into stdout and forwarded by a Python thread. The measured saving is modest (~0.2 min on fast hosts).

Look here:

• scripts/ci-test-shard.py:119-156: run(), which forwards raw bytes, keeps draining after a broken stdout, and waits at most 10 s for the pipe after cargo exits
• scripts/ci-test-shard.py:60: the Running ... (deps/<name>-<hash>) regex (handles Windows \ and .exe)
• scripts/ci-test-shard.py:168: coverage defaults to the Linux sample, and an empty CI_TEST_TIMINGS keeps the count split (macOS still passes '', so it stays on count)
• scripts/tests/test_ci_test_shard.py:199: a guard that fails once more than 10% of targets are missing from the sample

Verified:

• Read the full diff (3 files, +451/−15). CHANGELOG.md and ci.yml are untouched.
• Ran scripts/tests/test_ci_test_shard.py locally on 489db94: 21/21 pass, including the sample-coverage check against real cargo metadata.
• The new run() tests would fail on main, since the function doesn't exist there.
• The partition is deterministic, so both shards compute the same split.
• ci-ok, clippy and Cursor Bugbot are green on the head. Mergeable, with 0 unresolved threads.

Changes I made: none.

Open questions: none blocking. The sample comes from one run and was refreshed by hand. The PR already lists that and the hyphenated-target-name limitation as known.

Auto-merge is armed, so approving sends this straight to the merge queue.

---

Generated by Claude Code