@@ -32,21 +32,12 @@ jobs:
3232 # push runs (whose caches feed production image builds) or with trusted
3333 # internal-PR runs.
3434 #
35- # node_modules ALSO keys on the lockfile hash, the other two deliberately
36- # do not. A sticky disk is a persistent mutable volume, so every branch
37- # sharing one key shares one node_modules tree, and `bun install
38- # --frozen-lockfile` installs what the lockfile needs without pruning what
39- # it no longer references. Branches cut before a dependency bump therefore
40- # wrote their older copies onto the shared disk and every other branch
41- # inherited them — that is how builds on next 16.2.11 ended up loading a
42- # stale @next/swc 16.2.6 that appears nowhere in bun.lock. Hashing the
43- # lockfile gives each distinct dependency set its own disk, so a tree can
44- # never mix versions from two different lockfiles.
45- #
46- # The bun download cache is content-addressed by package+version and the
47- # Turbo cache is keyed by task hash, so both are safe to share across
48- # lockfiles — and sharing them is what keeps a fresh node_modules disk
49- # cheap to populate.
35+ # node_modules also keys on the lockfile hash: a sticky disk is a mutable
36+ # volume, and `bun install --frozen-lockfile` adds what the lockfile needs
37+ # without pruning what it dropped, so branches on different lockfiles were
38+ # contaminating each other (a stale @next/swc 16.2.6 outlived the 16.2.11
39+ # bump). The bun and Turbo caches are content/hash-addressed, so they stay
40+ # shared — that is what keeps a fresh node_modules disk cheap to fill.
5041 - name : Mount Bun cache
5142 uses : ./.github/actions/cache-mount
5243 with :
@@ -209,21 +200,16 @@ jobs:
209200 # Next.js production build, in parallel with lint + tests. Sticky disks are
210201 # cloned from the last committed snapshot per job and committed last-writer-
211202 # wins, so concurrent mounts are safe. The bun/node_modules disks are shared
212- # with test-build — and because the node_modules key now includes the lockfile
213- # hash, the two jobs only ever share a disk when they really do resolve the
214- # same dependency tree, so the LWW loss really is harmless. The Turbo cache
215- # gets its own key: with a shared key, only
216- # the last committer's new entries survive each run, so the test and build
217- # Turbo entries would evict each other nondeterministically.
218- # Same next build as the app image, so it needs a large runner. Peak RSS
219- # tracks Turbo/Turbopack cache warmth, and it is the COLD case that sizes the
220- # runner: warm peaks ~12 GB, partial ~28 GB, and a cold full rebuild peaked
221- # 51 GB. The 8vcpu tier only has 30.4 GB, so cold-cache runs OOM-killed the VM
222- # (oom_count 1, memory p100 ~30.5 GB, ~99% of the tier) — the job burned 8-15
223- # min and died, while warm runs finished under 8 min. NODE_OPTIONS'
224- # --max-old-space-size only caps Node's JS heap; the bulk is native Turbopack
225- # worker memory, so the cap cannot prevent this. 16vcpu = 60.8 GB fits the
226- # 51 GB cold peak with headroom. Keep the test job on 8vcpu; its memory is fine.
203+ # with test-build (the lockfile-hashed key means they only ever share when the
204+ # dependency tree really is identical, so LWW loss is harmless), but the Turbo
205+ # cache gets its own key: with a shared key, only the last committer's new
206+ # entries survive each run, so the test and build Turbo entries would evict
207+ # each other nondeterministically.
208+ #
209+ # Runner is sized for the COLD-cache build, which is what OOM-killed the 8vcpu
210+ # tier (23 kills / 1074 runs at 98% of its 30.4 GB): warm peaks ~12 GB, cold
211+ # peaked 51 GB. NODE_OPTIONS' --max-old-space-size caps only Node's JS heap,
212+ # not the native Turbopack workers that dominate, so it cannot prevent this.
227213 build :
228214 name : Build App
229215 runs-on : ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-16vcpu-ubuntu-2404' || 'linux-x64-8-core' }}
@@ -276,6 +262,20 @@ jobs:
276262 key : ${{ github.repository }}-nextjs-cache-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}
277263 path : ./apps/sim/.next/cache
278264
265+ # A cold build peaks ~51 GB, and running out of RAM kills the whole VM —
266+ # which surfaces only as "the runner has received a shutdown signal", with
267+ # no mention of memory. Fail up front with the actual numbers instead, so
268+ # an undersized runner (any CI_PROVIDER, including the GitHub fallback)
269+ # is a one-line diagnosis rather than a 12-minute mystery.
270+ - name : Check runner has enough memory
271+ run : |
272+ TOTAL_GB=$(awk '/MemTotal/ {printf "%d", $2/1048576}' /proc/meminfo)
273+ echo "Runner memory: ${TOTAL_GB} GB (cold-cache build peaks ~51 GB)"
274+ if [ "$TOTAL_GB" -lt 40 ]; then
275+ echo "::error::Runner has ${TOTAL_GB} GB; this build needs ~51 GB on a cold cache and will be OOM-killed. Use a larger runner for the Build App job."
276+ exit 1
277+ fi
278+
279279 - name : Install dependencies
280280 run : bun install --frozen-lockfile
281281
0 commit comments