build(ci): fix OOM errors on macOS runner#21176
Conversation
mikehardy
left a comment
There was a problem hiding this comment.
I think the approach is solid - question on whether we can make the 3 non-magical
If answer is "nope", then merge at will
If answer is "yep", then after pulling the 3 dynamically, merge away with gusto
| if (ciBuild) { | ||
| // #21168: The `macos-14` CI runner has only 7GB RAM and OOMs (exit 134) so bound by RAM | ||
| val totalRamGb = sysctl("hw.memsize") / (1024 * 1024 * 1024) | ||
| val reservedMemory = 3 // 3GB (org.gradle.jvmargs) + OS |
There was a problem hiding this comment.
is there a way to get that 3GB value dynamically?
concern: we change the 3GB at some point to 4GB (it does creep up over time...) and now we still OOM even with 2 workers because the formula should be kicking out 1 vs 2 ?
There was a problem hiding this comment.
Thanks! I chose to parse -Xmx3072M"
Runtime.getRuntime().maxMemory() is dependent on GC flags
| GC | maxMemory() for -Xmx3072M | / GiB (floor) |
|---|---|---|
| G1 (JDK 21 default) | 3221225472 | 3 |
| ParallelGC | 2863661056 | 2 |
| SerialGC | 3113877504 | 2 |
...by taking physical memory into account when deciding how many test processes to spawn. Even though `macos-14` has 3 CPUs, there is only 7GB of memory. Each test process has a max heap of 2GB, with the addition of Gradle this is enough to OOM the machine. So we limit the number of test processes based on RAM `gradleDaemonHeapBytes` is used because `Runtime.getRuntime().maxMemory()` is GC-dependent (default => 3GB; ParallelGC => 2GB) Now: * 3GB reserved for Gradle (org.gradle.jvmargs=-Xmx3072M) and the OS * 4GB remaining * Heap size: 2GB * => 2 forks Fixes 21168 Assisted-by: Claude Opus 4.8 - diagnosis & some code
|
@mikehardy Could I have an approve? Request changes -> I believe I can only rebase directly, rather than use the merge queue |
mikehardy
left a comment
There was a problem hiding this comment.
Runtime.getRuntime().maxMemory() is dependent on GC flags
Why can't anything be easy 😆
Gnarly code but also based on the flag-dependency of the easy way - an optimal solution
And so it goes - nifty fix, thank you!
...by taking physical memory into account when deciding how many test processes to spawn.
Note
Assisted-by: Claude Opus 4.8 - diagnosis & some code
Purpose / Description
Even though
macos-14has 3 CPUs, there is only 7GB of memory.Each test process has a max heap of 2GB, with the addition of Gradle this is enough to OOM the machine.
Fixes
Approach
How Has This Been Tested?
CI-only
Learning (optional, can help others)
macos-14runner. 3 CPUs, 7GB RAMChecklist