CI performance dashboard #1182

Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.

Headline (run 2026-10-29 04:16 UTC)

Window: 2026-10-28 04:16 → 2026-10-29 04:16. 300 runs listed (3 pages, back to 10-28 14:06). The merge queue has been idle since 22:56: no merge_group or push CI runs since 00:16, so the 24h merge-queue numbers are carried from 00:16. This slice had 12 non-stub PR CI runs, all green, with no Depot slow-starts (0/276). PR feedback was the fastest it's been in this history.

Since the last run (00:16 → 04:16)

• Merge queue: 0 merges, 0 entries. 24h stays at 32 merged, 0 evictions (carried).
• PR feedback: green full runs created → ci-ok 4.79 / 5.12 (n=12, 4.45–6.48), vs 5.67 / 7.47 at 00:16. coverage (2) finished last in 9 of 12 runs and the Gradle hosted e2e leg in 3. The 6.48 outlier (run) had a 2.67-min e2e-build; the other 11 took 1.12–1.40. That's 1 of 12, so it's a watch item.
• Depot: slow-start 0% (0/276 Depot jobs, boot p50 5 s, p90 6 s, max 8 s), vs 47% at 00:16. Queue wait p50 1 s, max 7 s. CI perf (settings): slow Depot 16-core hosts since 10-13 15:20 add ~1.5–2.5 min to the merge-queue critical path (~0.6k Linux job-min/day) #1512 is quiet this slice.
• Open ci-perf PR #1806 (coverage shards balanced on a Linux sample): on its own 4 runs the slowest coverage job went from 4.52 (count split) to 4.23 / 4.15 / 4.55 (~0.2 min). Other PRs' count-split runs this slice: coverage (2) 3.80–4.92 (p50 4.50), coverage (1) 3.00–4.23. The merge_group effect will show after it merges.
• Open ci-janitor PR #1803 (retry setup-python) covers the 00:16 Setup Python hang watch item. 0 hangs this slice: the longest of 132 Setup Python steps took 7 s. (The 13 attempt 1/attempt 2 steps are Retry setup-python on a transient download failure #1803's own runs.)

24h table

Top steps since 00:16 (job-min): Run e2e tests 157 (144 legs), Run tests with coverage 85 (24), Install Rust 16 (108), yarn berry/classic VEX steps 15 + 14, Run hosted-mode production e2e 12, Compile the CLI and every CLI test target 10.5, Checkout 10 (288).

Merge-queue settings: not re-read this run (last read 10-28 20:16: build 5 / merge 5, min 1, min wait 5 min, ALLGREEN, timeout 90, SQUASH).

History

Open ci-perf issues by ROI

ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).

This run (04:16): filed none, commented on none, closed none. The queue was idle and Depot had no slow-starts, so there were no new numbers for #1512 or #1248.

Open ci-perf / ci-janitor PRs: #1806 balance coverage shards on Linux test times (~0.2 min per run on its own PR runs); #1803 retry setup-python on a transient download failure.

Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect

Method notes

• GraphQL and search/issues are blocked in this environment. Merge-queue state comes from REST timelines of merged PRs. "Enqueue → merge" uses the PR's first added_to_merge_queue event in the window. "Slot wait" = enqueue → the PR's first merge_group CI run created_at.

• Jobs are fetched with filter=all. Queue wait = job started_at − created_at; skipped jobs are excluded. "Slow start" = job started_at → first step started_at over 15 s, Depot jobs only.

• Critical path = merge_group run created_at → ci-ok completed_at of the latest attempt.

• "Discarded" merge_group runs are every CI run except each merged PR's last one.

• Hang cost (~330 entry-min) = for each entry held, its merge time minus when its own group run finished (or, for the Fix Deno global npm tools missed by -g (#444) #1719 hang, minus ~15:30, when the held groups' runs had finished).

• Run 10-26 20:16: 300 of 313 runs listed (3-page cap), and jobs for all 108 CI runs. "PR full runs" leave out 1-job stub runs (ci-ok in under 1 min).

• Runs 10-27 00:16, 04:16, 08:16 and 12:17: the queue was idle, so the 24h merge and merge_group numbers are carried from 20:16. Only the new 4-hour slice was fetched each time. The 12:17 run dropped the 10-24 12:16 history row (18-row cap).

• Run 10-27 16:17: 197 runs in the window (2 pages). Jobs fetched for every CI run since 12:17. The 24h merge numbers combine the 10-26 post-16:19 slice (carried) with today's 9 merges. The oldest history row (10-24 16:16) was dropped (18-row cap).

• Run 10-27 20:16: 157 runs in the window. Jobs were fetched for every CI run since 16:17, plus all merge_group/push CI runs in the window, to get ci-ok start gaps (started_at → first step). The oldest history row (10-24 20:16) was dropped (18-row cap).

• Run 10-28 00:16: 190 runs in the window. Jobs were fetched for every non-stub CI run since 20:16. The 24h enqueue→merge p50 combines the 9 carried 13:07–14:58 values (5.2–8.5, p50 6.1) with Build(deps): Bump zod from 3.25.76 to 4.6.6 in /npm/socket-patch #1702, Build(deps): Bump rustls from 0.23.45 to 0.23.46 #1744 and fix: vendor hosted patches directly from lockfiles #1648 (last enqueue); it's approximate. The oldest history row (10-25 00:16) was dropped (18-row cap).

• Run 10-28 04:16: 227 runs in the window (3 pages). No merge_group or push CI runs since 00:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 11 completed full PR CI runs since 00:16. The oldest history row (10-25 04:16) was dropped (18-row cap).

• Run 10-28 08:17: 283 runs in the window (3 pages). No merge_group or push CI runs since 00:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 24 non-stub CI runs since 04:16. The oldest history row (10-25 08:15) was dropped (18-row cap).

• Run 10-28 12:17: 288 runs in the window (3 pages). No merge_group or push CI runs since 10-27 21:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 17 CI runs since 08:17. The oldest history row (10-25 12:15) was dropped (18-row cap).

• Run 10-28 16:18: 300 of 369 runs listed (3-page cap; the list reaches back to 10-27 22:41, so the 10-27 20:05–21:16 merges are carried). Jobs were fetched for all 39 non-stub CI runs since 12:17 (20 mg, 11 push, 8 PR) and timelines for 15 of the 20 merged PRs (~60 REST requests). Enqueue → merge p50/p90 is over those 15. The oldest history row (10-25 16:15) was dropped (18-row cap).

• Run 10-28 20:16: 300 of 417 runs listed (3-page cap, back to 04:51). Jobs were fetched for all 20 non-stub CI runs since 16:18 and timelines for Keep Go hosted pins after go mod tidy drops the zip line #1748 and Read Go workspaces the way go builds them #1763. The CI perf: release-readiness — 10-second gate waits up to 5 min for a Depot boot in front of coverage and e2e (~1–3 min per run in slow-start periods) #1795 replay keeps each run's measured clippy end and per-job boots and puts release-readiness at its own work time + 6 s; jobs needing clippy too move only as far as clippy allows. The oldest history row (10-25 20:15) was dropped (18-row cap).

• Run 10-29 00:16: 300 of 463 runs listed (3-page cap, back to 10-28 13:44, so all 32 merge_group runs since the 13:56 burst are in it). Jobs were fetched for all 31 non-stub CI runs since 20:16, timelines for the 10 PRs merged since 20:16, and 1 job log. GraphQL is still blocked (403). ~50 REST requests. The oldest history row (10-26 00:16) was dropped (18-row cap).

• Watch item, not filed: one uv/composer e2e leg on a PR (job) spent 18.7 min in Setup Python (3.12.10 isn't in the Depot tool cache, so every leg downloads it in 5–8 s; this one hit getaddrinfo EAI_AGAIN raw.githubusercontent.com at 3.5 min and then hung until 19.3 min). It cost the PR ~20 min and a 24-min re-run. That's 1 of 200 Setup Python steps in 4 h. A step timeout-minutes plus a retry, like the rust-cache cap (Cap hung rust-cache restores so they cannot stall the merge queue #1730), would bound it. It'll be filed if it recurs.

• Not filed: coverage shard balance. coverage (2)'s test step still runs 241–264 s vs 161–179 s for shard 1 on fast hosts, but the Gradle e2e leg ends at ~4.9–5.0 min, so balancing gives at most ~0.3–0.5 min on fast hosts. Balance coverage shards on measured test time #1768's retry with a Linux timing sample stays a low-ROI option.

• Run 10-29 04:16: 300 runs listed (3-page cap, back to 10-28 14:06). No merge_group or push CI runs since 00:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 12 non-stub CI runs since 00:16 (~25 REST requests, no job logs). The oldest history row (10-26 04:16) was dropped (18-row cap).

• Watch item, not filed: one e2e-build took 2.67 min (run), vs 1.12–1.40 in the other 11 runs, and it pushed that run's ci-ok to 6.48. It'll be filed if it recurs.

Generated by Claude Code