ci: fail fast on stalled git fetch so checkout's own retry can recover - #517
Merged
Conversation
"Optimize the slow checkout" turned out to be the wrong frame. Across 170 checkout steps in the last 40 publish.yml runs, 137 finished in 12-19s. The bad tail does not smear across a range - it piles up on 599s and 899s, which ARE the 10- and 15-minute timeout-minutes values. A step ending exactly at the cap did not run slowly; it hung. Evidence: run 32407671265 logs `git fetch` at 19:15:42 and the next line is "The operation was canceled" at 19:30:41 - 15 minutes of zero output. Shrinking the clone is not an available lever. Every job except _hugo.yml/build already checks out at the default --depth=1, and run 32414801788 hung 10 minutes on a shallow fetch anyway. The one job needing fetch-depth: 0 needs it for enableGitInfo -> .Lastmod, so it cannot go shallow either. actions/checkout ALREADY retries a failed fetch 3 times (src/git-command-manager.ts wraps fetch in retryHelper) - it just never fires, because a hang is not a failure. So make the hang a failure: http.lowSpeedLimit=1000 + http.lowSpeedTime=30 abort a transfer that stops moving, and checkout's own retry recovers. No new action, no dependency, no per-step retry scaffolding. Two mechanics that would have silently no-op'd this: - GIT_CONFIG_* env, NOT `git config --global` - checkout overrides HOME before running git, so an earlier step's global config never reaches it. - Reusable workflows do not inherit the caller's env, and neither do sibling workflow files, so all six checkout-bearing workflows carry the block. Verified both directions against the real remote, not just read back: at an absurd 100 MB/s floor a healthy fetch aborts with `curl 28 Operation too slow` (exit 128, the error retryHelper catches); at the shipped 1 KB/s floor a real fetch of master completes clean. All six workflows re-parse with the guard at workflow level and jobs: intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci-gates.md already documented the checkout stall as upstream; this adds what the session actually learned beyond "it stalls": - The duration histogram DECIDES slow-vs-stalled without guessing. 137/170 steps at 12-19s, bad tail on 599s/899s = the timeout-minutes values. A duration landing exactly on the cap never finished. - "Just shallow-clone it" is not an available lever - every job but _hugo.yml/build is already --depth=1 and stalls anyway; the one that cannot go shallow needs enableGitInfo for .Lastmod. - The guard, and the two mechanics that would silently no-op it (GIT_CONFIG_* vs --global under checkout's HOME override; reusable workflows not inheriting caller env). - A config you can only read back is not a guard - verified by breaking it (100 MB/s floor -> curl 28) and by confirming the shipped floor passes. Bundle validates conformant (--strict, 0 errors). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reported problem was the wrong frame
Asked to "optimize the slow checkout" on a 7-minute
Asset Pipelinejob. The histogram answered before any code changed — across 170 checkout steps in the last 40publish.ymlruns:A healthy checkout of this repo is 12–19s. The bad tail doesn't smear across a range — it piles up on 599s and 899s, which are the 10- and 15-minute
timeout-minutesvalues. A step ending exactly at the cap didn't run slowly; it hung.Evidence — run 32407671265:
15 minutes of zero output inside
git fetch.Shrinking the clone is not an available lever
Paul asked whether reducing checkout to the last commit would help. It wouldn't, and mostly it's already done:
_hugo.yml / buildis already at the default--depth=1— and they stall anyway. Run 32414801788 hung 10 minutes on--depth=1.fetch-depth: 0forenableGitInfo→.Lastmod→article:modified_time+ schemadateModified(config/_default/hugo.toml:7). It already carriesfilter: blob:none, and stalled at 482s with it applied.Nothing left to shrink.
The fix works with the grain of the tool
actions/checkoutalready retries a failed fetch three times —src/git-command-manager.tswrapsfetchinretryHelper(3 attempts). It just never fires, because a hang is not a failure.So make the hang a failure.
http.lowSpeedLimit=1000+http.lowSpeedTime=30abort a transfer that has stopped moving; checkout's own retry recovers. A stall costs ~30s instead of the whole job. No new action, no dependency, no per-step retry scaffolding.Two mechanics that would have silently no-op'd this:
GIT_CONFIG_*env, notgit config --global— checkout logsTemporarily overriding HOME=...before it runs git, so a global config set by an earlier step is invisible to it.Verification — broke the guard, watched it fail
A config you can only read back is not a guard; reading the value back "passes" even if the knob is inert. Tested both directions against the real remote:
error: RPC failed; curl 28 Operation too slow— exit 128, the errorretryHelpercatches--depth=1fetch of master completes cleanAlso confirmed git honours
GIT_CONFIG_*on git 2.55.0 (the runner's version) including under the HOME override, and that all six workflows re-parse with the guard at workflow level andjobs:intact.Gates
CI-config + docs only — no
themes/,layouts/, CSS, or content, so the visual suites don't apply (content-only rule). OKF bundle validates conformant, 0 errors under--strict.Note for the reviewer
The repo's agent 4-eyes pre-commit gate was not run — this session is configured not to spawn agents unless asked. Flagging rather than silently skipping; happy to run it if you want it before merge.
🤖 Generated with Claude Code