Skip to content

ci: fail fast on stalled git fetch so checkout's own retry can recover - #517

Merged
pftg merged 2 commits into
masterfrom
ci-stalled-fetch-guard
Aug 20, 2026
Merged

ci: fail fast on stalled git fetch so checkout's own retry can recover#517
pftg merged 2 commits into
masterfrom
ci-stalled-fetch-guard

Conversation

@pftg

@pftg pftg commented Aug 20, 2026

Copy link
Copy Markdown
Member

The reported problem was the wrong frame

Asked to "optimize the slow checkout" on a 7-minute Asset Pipeline job. The histogram answered before any code changed — across 170 checkout steps in the last 40 publish.yml runs:

Checkout duration Steps
< 30s 137
30–120s 13
2–5 min 7
5–8 min 3
> 8 min 10

A healthy checkout of this repo is 12–19s. The bad tail doesn't smear across a range — it piles up on 599s and 899s, which are the 10- and 15-minute timeout-minutes values. A step ending exactly at the cap didn't run slowly; it hung.

Evidence — run 32407671265:

19:15:42  [command]/usr/bin/git -c protocol.version=2 fetch ...
19:30:41  ##[error]The operation was canceled.

15 minutes of zero output inside git fetch.

Shrinking the clone is not an available lever

Paul asked whether reducing checkout to the last commit would help. It wouldn't, and mostly it's already done:

  • Every job except _hugo.yml / build is already at the default --depth=1 — and they stall anyway. Run 32414801788 hung 10 minutes on --depth=1.
  • The one job that can't go shallow needs fetch-depth: 0 for enableGitInfo.Lastmodarticle:modified_time + schema dateModified (config/_default/hugo.toml:7). It already carries filter: blob:none, and stalled at 482s with it applied.

Nothing left to shrink.

The fix works with the grain of the tool

actions/checkout already retries a failed fetch three timessrc/git-command-manager.ts wraps fetch in retryHelper (3 attempts). It just never fires, because a hang is not a failure.

So make the hang a failure. http.lowSpeedLimit=1000 + http.lowSpeedTime=30 abort a transfer that has stopped moving; checkout's own retry recovers. A stall costs ~30s instead of the whole job. No new action, no dependency, no per-step retry scaffolding.

Two mechanics that would have silently no-op'd this:

  • GIT_CONFIG_* env, not git config --global — checkout logs Temporarily overriding HOME=... before it runs git, so a global config set by an earlier step is invisible to it.
  • Reusable workflows don't inherit the caller's env, and neither do sibling workflow files — so all six checkout-bearing workflows carry the block.

Verification — broke the guard, watched it fail

A config you can only read back is not a guard; reading the value back "passes" even if the knob is inert. Tested both directions against the real remote:

Threshold Result
absurd 100 MB/s floor error: RPC failed; curl 28 Operation too slowexit 128, the error retryHelper catches
shipped 1 KB/s floor real --depth=1 fetch of master completes clean

Also confirmed git honours GIT_CONFIG_* on git 2.55.0 (the runner's version) including under the HOME override, and that all six workflows re-parse with the guard at workflow level and jobs: intact.

Gates

CI-config + docs only — no themes/, layouts/, CSS, or content, so the visual suites don't apply (content-only rule). OKF bundle validates conformant, 0 errors under --strict.

Note for the reviewer

The repo's agent 4-eyes pre-commit gate was not run — this session is configured not to spawn agents unless asked. Flagging rather than silently skipping; happy to run it if you want it before merge.

🤖 Generated with Claude Code

pftg and others added 2 commits August 20, 2026 23:57
"Optimize the slow checkout" turned out to be the wrong frame. Across 170
checkout steps in the last 40 publish.yml runs, 137 finished in 12-19s. The
bad tail does not smear across a range - it piles up on 599s and 899s, which
ARE the 10- and 15-minute timeout-minutes values. A step ending exactly at the
cap did not run slowly; it hung.

Evidence: run 32407671265 logs `git fetch` at 19:15:42 and the next line is
"The operation was canceled" at 19:30:41 - 15 minutes of zero output.

Shrinking the clone is not an available lever. Every job except
_hugo.yml/build already checks out at the default --depth=1, and run
32414801788 hung 10 minutes on a shallow fetch anyway. The one job needing
fetch-depth: 0 needs it for enableGitInfo -> .Lastmod, so it cannot go
shallow either.

actions/checkout ALREADY retries a failed fetch 3 times
(src/git-command-manager.ts wraps fetch in retryHelper) - it just never fires,
because a hang is not a failure. So make the hang a failure:
http.lowSpeedLimit=1000 + http.lowSpeedTime=30 abort a transfer that stops
moving, and checkout's own retry recovers. No new action, no dependency, no
per-step retry scaffolding.

Two mechanics that would have silently no-op'd this:
- GIT_CONFIG_* env, NOT `git config --global` - checkout overrides HOME before
  running git, so an earlier step's global config never reaches it.
- Reusable workflows do not inherit the caller's env, and neither do sibling
  workflow files, so all six checkout-bearing workflows carry the block.

Verified both directions against the real remote, not just read back: at an
absurd 100 MB/s floor a healthy fetch aborts with `curl 28 Operation too slow`
(exit 128, the error retryHelper catches); at the shipped 1 KB/s floor a real
fetch of master completes clean. All six workflows re-parse with the guard at
workflow level and jobs: intact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci-gates.md already documented the checkout stall as upstream; this adds what
the session actually learned beyond "it stalls":

- The duration histogram DECIDES slow-vs-stalled without guessing. 137/170
  steps at 12-19s, bad tail on 599s/899s = the timeout-minutes values. A
  duration landing exactly on the cap never finished.
- "Just shallow-clone it" is not an available lever - every job but
  _hugo.yml/build is already --depth=1 and stalls anyway; the one that cannot
  go shallow needs enableGitInfo for .Lastmod.
- The guard, and the two mechanics that would silently no-op it
  (GIT_CONFIG_* vs --global under checkout's HOME override; reusable
  workflows not inheriting caller env).
- A config you can only read back is not a guard - verified by breaking it
  (100 MB/s floor -> curl 28) and by confirming the shipped floor passes.

Bundle validates conformant (--strict, 0 errors).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 90299a01-b23c-4667-ad5a-b6e3ec7ed323


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@pftg
pftg merged commit 165f470 into master Aug 20, 2026
5 checks passed
@pftg
pftg deleted the ci-stalled-fetch-guard branch August 20, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant