From 48a03fa8743510bb118eef7647c232c9fd5b109c Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 3 Sep 2026 11:49:24 +0000 Subject: [PATCH] fix(ci): Cap check-container cores and raise the timeout floor Two remedies from the contention analysis of run 32574134229, where all seven timeouts sat at the 1200 s floor on 4-core shards running at load 5-13. Check containers now run with _R_CHECK_LIMIT_CORES_=TRUE, the value CRAN's own check machines use: a cluster or fork call asking for more than 2 workers errors instead of spawning them. Without it a test suite sizing itself from parallel::detectCores() sees the runner's 4 cores and fans out 4 workers per check, and four concurrent checks amplified each other into exactly the hot windows the timeouts died in. Both halves get the identical setting, so the old-vs-new comparison is unaffected. The timeout floor rises from 20 to 30 minutes: a saturated shard runs each check at roughly half speed (measured queue-vs-pair inflation median 1.5x, p90 1.8x, up to ~3x in the hottest windows), and the old floor killed four packages that finish in 7-12 minutes uncontended. The cost is bounded -- a genuinely hung check holds a worker ten extra minutes. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF --- .github/workflows/revdep4.yaml | 10 +++++++--- .github/workflows/revdepx/README.md | 16 ++++++++++++++-- .github/workflows/revdepx/check-half.sh | 13 +++++++++++++ 3 files changed, 34 insertions(+), 5 deletions(-) diff --git a/.github/workflows/revdep4.yaml b/.github/workflows/revdep4.yaml index 072323334a7..e42f563d951 100644 --- a/.github/workflows/revdep4.yaml +++ b/.github/workflows/revdep4.yaml @@ -216,9 +216,13 @@ env: # layers and stale system packages forever. REVDEPX_IMAGE_MAX_AGE_DAYS: ${{ vars.REVDEPX_IMAGE_MAX_AGE_DAYS || '14' }} # Per-check timeout: factor times the package's CRAN check time, but never - # below the floor -- CRAN's machines are not these runners. + # below the floor -- CRAN's machines are not these runners. The floor is + # 30 min because a saturated shard runs each check at roughly half speed + # (measured queue-vs-pair inflation: median 1.5x, p90 1.8x, and up to + # ~3x in the hottest windows): run 32574134229 killed four packages at + # the old 20-min floor that finish in 7-12 min uncontended. REVDEPX_TIMEOUT_FACTOR: ${{ vars.REVDEPX_TIMEOUT_FACTOR || '1.5' }} - REVDEPX_TIMEOUT_MIN_MINUTES: ${{ vars.REVDEPX_TIMEOUT_MIN_MINUTES || '20' }} + REVDEPX_TIMEOUT_MIN_MINUTES: ${{ vars.REVDEPX_TIMEOUT_MIN_MINUTES || '30' }} REVDEPX_DEADLINE_MINUTES: ${{ vars.REVDEPX_DEADLINE_MINUTES || '300' }} jobs: @@ -949,7 +953,7 @@ jobs: GH_TOKEN: ${{ github.token }} SHARD: ${{ matrix.shard }} TIMEOUT_FACTOR: ${{ vars.REVDEPX_TIMEOUT_FACTOR || '1.5' }} - TIMEOUT_MIN_MINUTES: ${{ vars.REVDEPX_TIMEOUT_MIN_MINUTES || '20' }} + TIMEOUT_MIN_MINUTES: ${{ vars.REVDEPX_TIMEOUT_MIN_MINUTES || '30' }} DEADLINE_MINUTES: ${{ vars.REVDEPX_DEADLINE_MINUTES || '300' }} steps: diff --git a/.github/workflows/revdepx/README.md b/.github/workflows/revdepx/README.md index 6ddea498e5c..626019ef12e 100644 --- a/.github/workflows/revdepx/README.md +++ b/.github/workflows/revdepx/README.md @@ -171,7 +171,11 @@ Repository variables (`vars.*`) shared by both: `REVDEPX_SHARD_BUDGET_MINUTES`, `REVDEPX_MAX_PARALLEL`, `REVDEPX_SHARD_CAPACITY_MINUTES`, `REVDEPX_BASELINE_MAX_AGE_DAYS`, `REVDEPX_IMAGE_MAX_AGE_DAYS`, `REVDEPX_TIMEOUT_FACTOR`, -`REVDEPX_TIMEOUT_MIN_MINUTES`, `REVDEPX_DEADLINE_MINUTES`, +`REVDEPX_TIMEOUT_MIN_MINUTES` +(the per-check timeout floor, default 30 min: +a saturated shard runs each check at roughly half speed, +so a floor sized for uncontended times kills healthy checks), +`REVDEPX_DEADLINE_MINUTES`, `REVDEPX_COMMIT_REPORT`, `REVDEPX_MEMORY_PER_CHECK` (per-check container memory cap, default 6g; both engines honor it, and each derives a machine-sized cap when it is @@ -181,7 +185,15 @@ of the package under test, default `-g0` — template-heavy Stan/TMB translation units spend most of their compiler memory on debug info; set `-g` to restore CRAN's own flags), `REVDEPX_CHECK_MAKEFLAGS` (MAKEFLAGS inside the check container, -default `-j1`: the memory cap is sized for one compiler process); +default `-j1`: the memory cap is sized for one compiler process). +Check containers also run with `_R_CHECK_LIMIT_CORES_=TRUE` +(overridable through the environment), +the value CRAN's own machines use: +a test suite sizing itself from `parallel::detectCores()` +would otherwise fan out one worker per runner core, +and four such checks side by side +drove 4-core shards to load 5–13 — +the direct cause of the floor timeouts in run 32574134229; queue engine only: `REVDEPX_WORKERS`. Script-level environment variables are documented in the header of each script. diff --git a/.github/workflows/revdepx/check-half.sh b/.github/workflows/revdepx/check-half.sh index 6774757d847..331bf9f364f 100755 --- a/.github/workflows/revdepx/check-half.sh +++ b/.github/workflows/revdepx/check-half.sh @@ -48,6 +48,17 @@ # -j1): one compiler process per check, so the # memory cap is sized for one cc1plus, not for a # package Makefile's idea of parallelism. +# _R_CHECK_LIMIT_CORES_ - passed into the container, default TRUE, the +# value CRAN's own check machines use: a cluster or +# fork call asking for more than 2 workers errors +# instead of spawning them. Without it a test suite +# that sizes itself from parallel::detectCores() +# sees the runner's 4 cores and fans out 4 workers +# per check -- with 4 concurrent checks, run +# 32574134229's 4-core shards ran at load 5-13, +# and every floor timeout sat in exactly those +# windows. Identical for both halves, so the +# comparison is unaffected either way. # # Leaves // holding the .Rcheck directory, `driver.log` (what # R CMD check said, each line stamped with elapsed seconds), `status` (the @@ -178,6 +189,8 @@ run_args=( # and a package Makefile that would fan out -j$(nproc) compilers under a # 6g cap trades one OOM-killed compiler for several. -e MAKEFLAGS="${REVDEPX_CHECK_MAKEFLAGS:--j1}" + # CRAN parity, and the shard's own tranquillity: see the header note. + -e _R_CHECK_LIMIT_CORES_="${_R_CHECK_LIMIT_CORES_:-TRUE}" -v "${tarball}:/revdepx/src/${src_name}:ro" -v "${out}:/revdepx/out" -v "${out}/tmp:/tmp"