Skip to content

Commit 9221ec8

Browse files
claude[bot]claude
andauthored
fix(tooling): raise the re-measure heap ceiling to 6144 and give the job the memory (A1) (#14805)
* chore(tooling): TEMPORARY runner-memory probe for the tsc heap ceiling (#14569) Reverted before this PR's final diff. `CI_TSC_HEAP_CEILING_MB` may only move on a reading taken where the verdict is taken -- the `Type Check · debt ledger` job on `ubuntu-latest` -- and this container cannot download job logs. Check-run ANNOTATIONS are readable over REST, so the probe emits its readings as `::notice` workflow commands from that job: - the runner's MemTotal/MemAvailable/Swap, image, nproc, and the gate process's own V8 `heap_size_limit` (the runner's default old space); - what else is resident at the point the re-measure starts (`ps` RSS census); - the `packages/qa/http-conformance` TEST_DEBT program -- the same generated project `measureTestDebt` writes -- run with `--extendedDiagnostics` under `--max-old-space-size=4096` and under `6144`, reporting tsc's own "Memory used", peak RSS, and the machine's minimum MemAvailable during each run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WLJQhde67SeTccsmnBVarV * Revert "chore(tooling): TEMPORARY runner-memory probe for the tsc heap ceiling (#14569)" This reverts commit d415f4f. * docs(tooling): record the runner measurement beside CI_TSC_HEAP_CEILING_MB The pin's provenance was archaeology through a failed job's GC trace, which bracketed the runner's old space into [4040, 4148] MB. It is now a first-hand reading, taken where the verdict is taken -- inside the `Type Check · debt ledger` job, by a temporary probe step (reverted in the previous commit) that emitted its numbers as `::notice` annotations: runner ubuntu24 20260831.293.1, 4 vCPU, MemTotal 16,373,452 kB (~15.6 GiB) -- not the 7 GB the finding assumed consumers 153 processes / 940,316 kB (~918 MB); the job's steps are sequential, so nothing runs beside the re-measure this gate heap_size_limit 4144 MB with NODE_OPTIONS unset -- the runner's V8 default, confirming the 4096 MB old space directly heaviest qa/http-conformance's TEST_DEBT program under two caps: program 4096 -> 4,077,718K used, 4,212,904 kB peak RSS, 26.84s check 6144 -> 4,420,706K used, 4,545,500 kB peak RSS, 21.90s check The pair is the headroom reading the finding asked for: 343 MB more heap keeps 343 MB more live and finishes ~5s sooner, so under 4096 the program is paying GC pressure to fit. The constant does NOT move on it, and the measurement is why: the scarce resource is V8's default old space (4096 MB), not the runner's memory, and this number describes that default exactly. The comment also records what the measurement made mechanically visible -- raising the pin alone cannot deliver a roomier run. `remeasureHeapCeiling` minimises over the pin and the running process's own limit, so a 6144 pin under the runner's default still chooses 4144, and the `stale` arm then refuses the run outright: `--re-measure` exits 1 before the first tsc. Reproduced against a 4144 MB process. Delivering a raise needs the gate PROCESS given the memory first, which is a workflow decision and is escalated on #14569. The self-test row for "a box shaped like CI" gains a note that its `+ 48` is now the measured runner rather than a construction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WLJQhde67SeTccsmnBVarV * fix(tooling): pin the re-measure heap ceiling at 6144 and give the job the memory Ruling A1 (#14569, 2026-09-03). The 02:25Z ruling A — raise on a recorded measurement — is completed the only way the measurement allows: the pin and the process's actual old space move together. - .github/workflows/lint.yml: the `typecheck-debt` job's re-measure step now runs under `NODE_OPTIONS: --max-old-space-size=6144`, so the process running tsc really has the old space the pin describes. V8's default there is 4096 MB, measured on the runner. - CI_TSC_HEAP_CEILING_MB: 4096 -> 6144, with the runner measurement already written beside it kept as the evidence. - Two new `remeasureHeapCeiling` self-test rows pin both directions of the pairing: the runner as the workflow now starts it (6192 reported, caller cap tying the pin, chosen ceiling 6144 named as the CI pin), and the same runner WITHOUT the workflow line (its measured 4144 MB default, refused). The `stale` arm is untouched and still refuses any pin above the process's own limit — that refusal is what caught the bare constant raise, and it is what keeps the two halves inseparable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zGPuVVX3deAx9LdjK8jCk --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 35e94c9 commit 9221ec8

2 files changed

Lines changed: 153 additions & 7 deletions

File tree

.github/workflows/lint.yml

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4808,7 +4808,33 @@ jobs:
48084808
- name: Build the ledgered packages' dependencies
48094809
run: pnpm exec turbo run build --filter='./packages/*' --filter='./packages/*/*'
48104810

4811+
# ⚠️ `NODE_OPTIONS` here is HALF of the heap ceiling this gate runs tsc
4812+
# under, and the two halves ship together or not at all (#14569, ruled
4813+
# 2026-09-03). The other half is `CI_TSC_HEAP_CEILING_MB` in
4814+
# scripts/check-type-check-coverage.mjs, which is a DESCRIPTION of the old
4815+
# space the process running tsc really has -- and the gate chooses the
4816+
# MINIMUM of that pin and this process's actual limit. So without this
4817+
# line the pin buys nothing (V8's default old space on this runner is
4818+
# 4096 MB, measured) and the gate's `stale` arm refuses the run outright,
4819+
# before the first tsc, on every PR and on `main`. That refusal is the
4820+
# pairing's enforcement, and it is deliberate.
4821+
#
4822+
# Why 6144 and not more, measured on this runner rather than reasoned
4823+
# (run 33708954003, job 100504131338, 2026-09-03): the heaviest ledger
4824+
# program keeps 4,077,718K live under a 4096 cap and 4,420,706K under
4825+
# 6144 while finishing ~5s sooner -- i.e. under 4096 it was paying GC
4826+
# pressure to fit -- and 10,562,192 kB of the box's 16,373,452 kB stayed
4827+
# available at the tightest moment of either run. The raise is headroom
4828+
# the box really has, not a promise it cannot keep; a ceiling above the
4829+
# box's real memory buys nothing and converts a recoverable heap error
4830+
# into an exit-137 SIGKILL with no diagnostic.
4831+
#
4832+
# ⛔ Do not change this number without changing the constant, or the
4833+
# reverse. The constant's comment carries the full reading and both
4834+
# directions are pinned in that file's `--self-test`.
48114835
- name: Re-measure the type-check DEBT / TEST_DEBT ledger
4836+
env:
4837+
NODE_OPTIONS: --max-old-space-size=6144
48124838
run: pnpm check:type-check-debt
48134839

48144840
# Lane 4 of 4 behind the required `TypeScript Type Check` context. The

scripts/check-type-check-coverage.mjs

Lines changed: 127 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2792,7 +2792,7 @@ function countTscErrors(output, { dropRootDirDiagnostics = false } = {}) {
27922792
// same tree. ⚠️ The asymmetry IS the defect: a local pass was never a claim
27932793
// about CI, and nothing said so out loud.
27942794
//
2795-
// ## Where the number comes from -- CI, never this box
2795+
// ## Where the runner's DEFAULT old space comes from -- CI, never this box
27962796
//
27972797
// Read off the CI runner itself: run 33136681083, job `Type Check · debt
27982798
// ledger`, at 6d097a604, Node v22.23.2. The `packages/qa/http-conformance`
@@ -2812,19 +2812,105 @@ function countTscErrors(output, { dropRootDirDiagnostics = false } = {}) {
28122812
// with `NODE_OPTIONS` on a box under this file's own eyes), so a 4096 old space
28132813
// reports 4144 and commits the 4147.5 above it.
28142814
//
2815-
// ⚠️ If 4096 is wrong, it is wrong DOWNWARD -- the only safe direction. This
2816-
// number's entire job is to be no HIGHER than CI's ceiling. A pin ABOVE CI's is
2817-
// worse than no pin at all: it makes local runs pass where CI still OOMs, which
2818-
// is exactly this defect with extra confidence attached.
2815+
// ## Re-measured FIRST-HAND on the runner, 2026-09-03 (#14569)
2816+
//
2817+
// The bracket above is archaeology through a failed job's GC trace. #14569
2818+
// asked for a raise to 6144 to be taken on a measurement rather than on a
2819+
// typed number, so the reading was taken where the verdict is taken: inside
2820+
// the `Type Check · debt ledger` job itself, by a temporary probe step that
2821+
// emitted its numbers as `::notice` annotations (run 33708954003, job
2822+
// 100504131338, image `ubuntu24 20260831.293.1`, Node v22.23.2, 4 vCPU).
2823+
//
2824+
// the runner MemTotal 16,373,452 kB (~15.6 GiB) plus 3,145,724 kB of
2825+
// swap -- NOT the 7 GB #14569 assumed. MemAvailable at the
2826+
// point the re-measure starts: 14,329,064 kB.
2827+
// other consumers 153 processes holding 940,316 kB (~918 MB) altogether:
2828+
// Runner.Worker 144 MB, Runner.Listener 98 MB, provjobd
2829+
// 96 MB, dockerd 73 MB, containerd 43 MB. The job's steps
2830+
// are sequential, so nothing in it runs BESIDE the
2831+
// re-measure -- the ledger's tsc has the box to itself.
2832+
// this gate's own `heap_size_limit` 4144 MB with `NODE_OPTIONS` unset --
2833+
// ceiling the runner's V8 default, read directly rather than
2834+
// inferred. It confirms the 4096 MB old space the GC trace
2835+
// above could only bracket.
2836+
// the heaviest `packages/qa/http-conformance`'s TEST_DEBT program (906
2837+
// program files, 692,003 lines of definitions, 7,328,937
2838+
// instantiations) under `--extendedDiagnostics`, twice:
2839+
//
2840+
// cap 4096 Memory used 4,077,718K peak RSS 4,212,904 kB
2841+
// check 26.84s
2842+
// cap 6144 Memory used 4,420,706K peak RSS 4,545,500 kB
2843+
// check 21.90s
2844+
//
2845+
// Neither OOMs, and the pair IS the headroom finding
2846+
// #14569 asked for: handed 343 MB more heap the same
2847+
// program keeps 343 MB more live and finishes ~5s sooner,
2848+
// so under 4096 it is paying GC pressure to fit rather
2849+
// than fitting. Lowest MemAvailable seen at any point
2850+
// during either run: 10,562,192 kB.
2851+
//
2852+
// The scarce resource is therefore NOT the runner's memory -- 15.6 GiB with
2853+
// ~918 MB of it spoken for -- but V8's DEFAULT old space on that runner, which
2854+
// the reading above pins at 4096 MB from two directions.
2855+
//
2856+
// ## The raise, on that measurement (#14569, ruled A then A1, 2026-09-03)
2857+
//
2858+
// A default is not a budget. The ledger's heaviest program was paying GC
2859+
// pressure to fit inside 4096 rather than fitting, and the tripwire (spec
2860+
// declaration growth) is a weekly event, so the ruling raises the ceiling --
2861+
// on the measurement above, never on a typed number. What that raise is NOT
2862+
// is a bigger promise about the box. It is the pair below, and ⛔ neither
2863+
// half is shippable alone:
2864+
//
2865+
// the workflow `.github/workflows/lint.yml`, job `typecheck-debt`, step
2866+
// "Re-measure the type-check DEBT / TEST_DEBT ledger", now
2867+
// runs under `NODE_OPTIONS: --max-old-space-size=6144`. That
2868+
// is the half that actually hands the process the old space:
2869+
// V8's default there is 4096 and no constant in this file can
2870+
// move it.
2871+
// this constant 6144 -- a description of the old space that step now
2872+
// really has, exactly as 4096 described the default before it.
2873+
//
2874+
// ⛔ Raising this constant ALONE cannot buy the ledger a roomier run --
2875+
// measured on 2026-09-03, not reasoned. `remeasureHeapCeiling` below takes the
2876+
// MINIMUM of this pin and the limit the running process actually has, so with
2877+
// the pin at 6144 and the gate started under the runner's DEFAULT the chosen
2878+
// ceiling is still 4144 -- and the `stale` arm below then refuses the run
2879+
// outright: `--re-measure` exits 1 before the first tsc ("the pin is now ABOVE
2880+
// the ceiling it claims to describe"), on every PR and on `main`. That
2881+
// refusal is the pairing's enforcement -- it is what caught the bare raise
2882+
// when it was attempted -- and both directions are pinned as self-test rows
2883+
// below ("the runner as the workflow now starts it" and "the same runner
2884+
// WITHOUT it"). Delete the `NODE_OPTIONS` line and the lane says so, loudly,
2885+
// on the runner.
2886+
//
2887+
// ⚠️ If 6144 is wrong, it is wrong DOWNWARD -- the only safe direction. This
2888+
// number's entire job is to be no HIGHER than the ceiling the process running
2889+
// tsc on CI really has. A pin ABOVE it is worse than no pin at all: it makes
2890+
// local runs pass where CI still OOMs, which is exactly this defect with extra
2891+
// confidence attached.
28192892
//
28202893
// ⛔ Do not raise this to make a local measurement complete. `--re-measure`
28212894
// OOMing under this ceiling is the gate WORKING -- it is CI's failure,
28222895
// reproduced on your box before you push. What grew is the type graph, not the
28232896
// memory CI has. (`packages/spec/tsup.config.ts` carries the other half of this
28242897
// lesson from the build side: a ceiling above the box's real memory does not
28252898
// buy a bigger run, it converts a recoverable heap error into an exit-137
2826-
// SIGKILL that carries no diagnostic at all.)
2827-
const CI_TSC_HEAP_CEILING_MB = 4096;
2899+
// SIGKILL that carries no diagnostic at all.) The 6144 is not an exception to
2900+
// that rule, it is an application of it: the runner was MEASURED to carry the
2901+
// heaviest program under a 6144 cap (4,420,706K used, 4,545,500 kB peak RSS,
2902+
// 10,562,192 kB still available at the tightest moment) before it was pinned.
2903+
//
2904+
// ⚠️ One protection the pair costs, recorded here so nobody rediscovers it as
2905+
// a surprise. With `NODE_OPTIONS` set explicitly on that step,
2906+
// `heap_size_limit` there reads 6192 whatever the runner's physical memory
2907+
// does -- so on THAT job the `stale` arm can no longer notice the runner
2908+
// shrinking; it now only notices a pin above a DEFAULTED process. The margin
2909+
// is what makes that acceptable: the pin asks for 6144 MB where the
2910+
// measurement found 10,562,192 kB available at the heaviest moment, ~1.7x. If
2911+
// that margin is ever in doubt the answer is a fresh runner measurement and a
2912+
// smaller number in BOTH places, ⛔ never a bigger one here.
2913+
const CI_TSC_HEAP_CEILING_MB = 6144;
28282914

28292915
/**
28302916
* The last `--max-old-space-size` in a `NODE_OPTIONS` string, in MB, or null.
@@ -5169,10 +5255,44 @@ function selfTest() {
51695255
expect: { mb: CI_TSC_HEAP_CEILING_MB, stale: false },
51705256
},
51715257
{
5258+
// `+ 48` is the RUNNER's offset, not a construction: V8 reports the old
5259+
// space plus a fixed ~48 MB of other spaces, measured on the `Type Check
5260+
// · debt ledger` job itself (4144 for a 4096 old space, 2026-09-03,
5261+
// #14569). So this row is the machine whose limit EQUALS the pin with no
5262+
// caller flag in play -- and the row above it is every box roomier than
5263+
// that one.
51725264
label: 'on a box shaped like CI the ceiling is a no-op that still names itself',
51735265
where: { heapLimitMb: CI_TSC_HEAP_CEILING_MB + 48, onCi: true },
51745266
expect: { mb: CI_TSC_HEAP_CEILING_MB, stale: false },
51755267
},
5268+
{
5269+
// THE RUNNER AS THE WORKFLOW NOW STARTS IT (#14569). `lint.yml`'s
5270+
// re-measure step sets `NODE_OPTIONS: --max-old-space-size=6144`, so the
5271+
// gate process reports 6192 AND carries a caller cap EQUAL to the pin.
5272+
// Both candidates tie, the tie-break keeps the CI ceiling's name, and
5273+
// that name is what the job's log then prints. Pinned because an
5274+
// off-by-one in either direction here reads as a caller cap overriding
5275+
// the pin on the one machine whose verdict counts.
5276+
label: "the workflow's own NODE_OPTIONS ties the pin and is not read as a tighter caller cap",
5277+
where: {
5278+
heapLimitMb: CI_TSC_HEAP_CEILING_MB + 48,
5279+
nodeOptions: `--max-old-space-size=${CI_TSC_HEAP_CEILING_MB}`,
5280+
onCi: true,
5281+
},
5282+
expect: { mb: CI_TSC_HEAP_CEILING_MB, stale: false },
5283+
},
5284+
{
5285+
// THE OTHER HALF OF THE PAIR, and the row that keeps the two halves
5286+
// inseparable. 4144 is the runner's DEFAULT `heap_size_limit`, measured
5287+
// on that job 2026-09-03. Take the `NODE_OPTIONS` line back out of
5288+
// `lint.yml` and this is the reading the gate gets: refused outright,
5289+
// before the first tsc, on every PR and on `main`. A bare raise of the
5290+
// constant was attempted and this is what caught it, so the pin above
5291+
// cannot quietly outlive the workflow line that pays for it.
5292+
label: 'the same runner WITHOUT the workflow NODE_OPTIONS -- its 4144 MB default -- is refused',
5293+
where: { heapLimitMb: 4144, onCi: true },
5294+
expect: { mb: 4144, stale: true },
5295+
},
51765296
{
51775297
// Never RAISE. Promising V8 memory the box does not have trades a
51785298
// recoverable heap error for a kernel SIGKILL that says nothing.

0 commit comments

Comments
 (0)