diff --git a/cc-perf-campaign/codex/REPORT_arena_trigger_fix.md b/cc-perf-campaign/codex/REPORT_arena_trigger_fix.md new file mode 100644 index 0000000000..252c1baff3 --- /dev/null +++ b/cc-perf-campaign/codex/REPORT_arena_trigger_fix.md @@ -0,0 +1,86 @@ +# Arena trigger promotion assertion follow-up + +Implementation SHA: `0fa6101bbcca31605b0bdad0e2c3740f19149a74` + +Branch: `perf/arena-trigger-excludes-nursery` (push target: `fork` only) + +## Root cause + +This is case (b): the in-place promotion path deliberately reserves one fresh +block after transferring the filled nursery blocks. + +- `crates/perry-runtime/src/arena/promote.rs:338-352` installs every captured + in-use nursery block into `OLD_ARENA`, adds its used bytes to old-gen + occupancy, and subtracts the same reserved capacity from + `NURSERY_RESERVED_BYTES`. It does not change `ARENA_TOTAL_BYTES`; that part + is a capacity transfer. +- The isolated fixture starts with the reusable empty Eden head left by its + preparatory minor. `fill_dead_nursery_to(16 MiB + BLOCK_SIZE)` consumes that + head and allocates until all 18 reserved 1 MiB Eden blocks have nonzero + offsets. `retag_young_for_in_place_promotion` therefore captures all 18. +- `crates/perry-runtime/src/arena/promote.rs:563-587` must leave Eden with a + usable allocator head. After all 18 filled blocks become tombstones, it calls + `install_fresh_block(BLOCK_SIZE)`. +- `crates/perry-runtime/src/arena/block.rs:654-689` accounts that new nursery + block through `arena_reserved_bytes_add`; `block.rs:1044-1049` correctly adds + 1 MiB to both `ARENA_TOTAL_BYTES` and `NURSERY_RESERVED_BYTES`. + +The reported numbers account exactly: 18 MiB before promotion becomes 19 MiB +whole-arena capacity after promotion, comprising 18 MiB old space plus the +mandatory empty 1 MiB Eden head. Because that head is present in both total and +nursery counters, it cancels out of `old_space_total_bytes()`. No nursery +counter creation/reset/release arm is missing. + +The failing assertion was therefore wrong at +`crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs:152-157`: promotion +transfers all existing capacity, but restoring the required Eden allocator head +necessarily reserves one new block. + +## Base-difference audit + +`git log 504e180d0..616a2cb84 -- crates/perry-runtime/src/arena \ +crates/perry-runtime/src/gc/policy.rs crates/perry-runtime/src/gc/promote.rs \ +crates/perry-runtime/src/gc/reset.rs` returned no commits. The one-block result +is not caused by drift between perrymaster's `504e180d0` base and this branch's +`616a2cb84` base. + +## Change + +`crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs:133-167` now: + +- proves the fixture promotes every reserved nursery block, making the fresh + Eden head deterministic; +- expects whole-arena reservation to increase by exactly `BLOCK_SIZE` and + identifies that block in the assertion message; +- verifies the post-promotion nursery reservation is exactly the replacement + Eden head; and +- verifies old-space pacing increases by exactly the promoted reservation, so + the replacement nursery head remains excluded. + +Runtime bookkeeping was not changed because its total, nursery, and old-space +quantities are already internally consistent. + +## Validation + +Every Cargo invocation was preceded by `df -g /`; free space was 27-65 GB, +above the binding 12 GB floor. Cargo ran through +`cc-perf-campaign/measure_lock.sh --build` with `-j4`. + +- Exact test: passed (1 passed, 0 failed). +- Runtime lib suite, non-PTY and single-threaded: passed (3261 passed, 0 + failed, 4 ignored). +- Archive feature-set build: `cargo build --release -j4 -p perry-runtime + --features wasm-host` passed. +- Default compiler build: `cargo build --release -j4 -p perry` passed. +- `rustfmt --edition 2021` on the changed Rust file: passed. +- `git diff --check`: passed. +- `bash scripts/check_file_size.sh`: passed. + +One earlier runtime-suite attempt was launched through a PTY and produced 3260 +passed / 1 failed / 4 ignored; the sole failure was +`tty::tests::columns_undefined_when_not_tty`, because the runner had supplied a +TTY. The required non-PTY rerun above was fully green. + +Not run: a separate `nm` audit or the external performance campaign. This +follow-up changes only test assertions; the prescribed runtime, feature-set, +and compiler gates all ran locally. diff --git a/cc-perf-campaign/codex/REPORT_arena_trigger_nursery.md b/cc-perf-campaign/codex/REPORT_arena_trigger_nursery.md new file mode 100644 index 0000000000..85186cfd49 --- /dev/null +++ b/cc-perf-campaign/codex/REPORT_arena_trigger_nursery.md @@ -0,0 +1,145 @@ +# ArenaBytes excludes the copying nursery + +Implementation SHA: `008fa075d542d9c09646a974593db03c6023ebc0` + +Branch: `perf/arena-trigger-excludes-nursery` (push target: `fork` only) + +## Mechanism map + +- `crates/perry-runtime/src/arena/block.rs:1008-1064`: `ARENA_TOTAL_BYTES` + is the cached reserved capacity of every arena region. The new adjacent + `NURSERY_RESERVED_BYTES` is the reserved-capacity share owned by Eden and + both survivor semispaces. Initial/fresh blocks update both counters according + to `HeapGeneration`; reset/release and quarantine detach subtract in the same + place as their cost. +- `crates/perry-runtime/src/arena/walk.rs:321-338`: `arena_total_bytes()` is + still the whole-arena diagnostic/RSS quantity. `old_space_total_bytes()` is + `arena_total - copying_nursery_reserved`, i.e. long-lived plus old reserved + capacity. The `[gc] blocks` `general` count is Eden; `non_general` combines + both survivor semispaces with long-lived and old, so neither printed bucket + alone was an old-space quantity. +- `crates/perry-runtime/src/arena/promote.rs:334-357`: in-place promotion moves + blocks from Eden/survivor ownership into `OLD_ARENA`. It now subtracts the + transferred capacity from the nursery counter without changing + `ARENA_TOTAL_BYTES`, making promotion visible as old-space growth. +- `crates/perry-runtime/src/gc/policy.rs:83-124,3144-3165`: + `next_arena_trigger_base()` is the armed `GC_NEXT_TRIGGER_BYTES` value (or + the device-derived absolute ceiling before the first arm). The ArenaBytes + predicate now compares `old_space_total_bytes()` with that base. Nursery + occupancy remains owned by the independent `young_scavenge_cap_due()` arm + and is handed to a copying minor. +- `crates/perry-runtime/src/gc/heap_budget.rs:119-132` and + `crates/perry-runtime/src/gc/policy.rs:2316-2508`: rebaselining uses the same + old-space capacity. Its default headroom floor remains exactly 16 MiB (scaled + only by the pre-existing constrained-device policy); adaptive step, ceiling, + and promotion-runway credit are unchanged. `next_base` is therefore based on + `old_total`, never nursery capacity. This is a unit change, not threshold + tuning. +- `crates/perry-runtime/src/gc/policy.rs:1570-1637,1831-1839`: + `old_reclaimable = old_gen_in_use - old_free`. Its baseline is the last + reclaim/growth decision's old occupancy; proven-live promotion is credited + to it so the signal remains reclaimable growth rather than absolute old + occupancy. ArenaBytes no longer needs to duplicate that evidence gate. +- `crates/perry-runtime/src/gc/arena_right_size.rs:142-176` and + `crates/perry-runtime/src/gc/idle_reclaim.rs:406-450`: idle right-sizing now + evaluates old-space live bytes against old-space capacity. The existing + sustained <=50% utilization rule, two-full bound, hysteresis, and idle-window + `arena_right_size` start remain unchanged. +- `crates/perry-runtime/src/gc/oldgen.rs:1405-1427,1487-1650` and + `crates/perry-runtime/src/arena/walk.rs:118-195`: a budgeted full's arena + cursor includes Eden, both survivor regions, long-lived, and old blocks. The + sweep therefore visits nursery objects and accounts dead Eden bytes in + place; block cleanup calls `arena_reset_empty_blocks` and survivor reclaim. + `crates/perry-runtime/src/arena/reset.rs:173-251` shows what the subsequent + copying minor normally does instead: after rewriting survivors it resets + Eden plus the active survivor from-space and flips semispaces. Thus the + measured major was doing nursery-death work before the next minor could. + +## Change + +The selected structural fix is to exclude all reserved copying-nursery +capacity from ArenaBytes arming and from every consumer of that trigger's +units: rebaseline, debt scaling, tiny-parse pressure/bump, and the explicit +memory-pressure clamp. The old-space ownership boundary is exact and already +has a separate nursery-cap consumer; adding a second reclaim-evidence policy +would overlap `old_reclaimable` and make arming depend on two baselines. + +`old_space_live_allocated_bytes()` subtracts the running from-space live census +from the running whole-arena live census, so #9838's tiny-parse guard no longer +treats a large nursery as pressure. The explicit safepoint deferral slack valve +remains whole-arena by design because it bounds RSS growth while a collection +waits; it does not arm ArenaBytes. + +With `PERRY_GC_DIAG=1`, `[gc-trigger]` now prints `arena_total`, the actual +`old_total` compared, and `nursery_excluded`. `[gc-budgeted] start` prints an +`arming_reason` plus the same totals. `[gc-arena-rebaseline]` prints the +old-space total and excluded nursery capacity on one complete line. + +## Sabotage-able tests + +- `crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs:57-115` fills + dead nursery objects to the real cap, proves whole-arena bytes crossed the + base while old bytes did not, asserts the incremental-start counter does not + move, then proves the precise copying-minor counter advances and from-space + shrinks. Restoring `arena_total_bytes()` in the ArenaBytes comparison starts + budgeted work and fails the status/counter assertion. +- `crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs:119-181` + transfers more than the unchanged headroom from nursery to old with the + nursery empty, then asserts exactly one ArenaBytes budgeted start. Removing + the nursery-counter transfer from `finish_in_place_promotion` keeps the old + quantity below the base and fails the start counter. +- `crates/perry-runtime/src/gc/tests/idle_reclaim.rs:153-226` retains the + bounded idle-window right-size test and names its sabotage: deleting the + `arena_right_size::owed()` start arm prevents the dedicated + `arena_right_size_starts` counter from advancing. +- Existing trigger, rebaseline, debt-pacer, copying-survival, host-safepoint, + arena-right-size, and idle-reclaim coverage was updated only where its unit + is now old-space capacity. + +## Validation + +No Cargo command was invoked. The required pre-Cargo `df -g /` check reported +0 GB available (97% capacity), below the binding 12 GB floor, so the build and +test gates were not run and no build lock was acquired or waited on. + +Non-Cargo checks run: + +- `rustfmt --edition 2021` on every modified Rust file: passed. +- `git diff --check` (working tree and alternate-index commit): passed. +- `bash scripts/check_file_size.sh`: passed (`OK: no Rust source files exceed + 2000 lines`). + +Not run locally; request these perrymaster gates from the pushed SHA: + +1. `cargo test -p perry-runtime --release --lib -- --test-threads=1` +2. `cargo build --release -p perry-runtime --features wasm-host` +3. `cargo build --release -p perry` +4. Coordinator label: `run-extended-tests` (GC-adjacent change). + +## Predictions and exact perrymaster request + +Prediction: budgeted cycles armed by ArenaBytes while `old_reclaimable` is at +baseline fall to 0 per run; in-turn `[gc-charge] budgeted-done total_us` falls +from 169-213 ms to approximately 0; minors per turn and `old_in_use` are +unchanged; idle-window reclaim is unchanged. The expected paired bound is +approximately -2% at 3300 characters from removing about 50 ms/turn, larger at +400 characters, with peak and settled RSS unchanged. RSS +1-10% remains +acceptable. Because the change removes majors that a larger nursery was +arming, the NS2 n64 arm needs a runtime-only rerun to determine whether its +previous +13% peak moves. + +Exact request: + +> From the pushed SHA, run the three release gates listed above and apply the +> `run-extended-tests` label. Relink that runtime on the I7-view tree +> (runtime-only). Run one 4-turn graceful 3300-character reply and one +> 400-character reply with `PERRY_GC_DIAG=1`; use `ns2_majors_an.py` and +> `ns2_alloc_an.py`. Verify ArenaBytes budgeted cycles with +> `old_reclaimable` at baseline are 0/run, in-turn `[gc-charge] +> budgeted-done total_us` moves from 169-213 ms to approximately 0, minors per +> turn are unchanged, `old_in_use` is unchanged, and idle-window reclaim is +> unchanged. Then run paired 5x3300 + 3x400 versus I7-view; expect about -2% +> at 3300 from the roughly 50 ms/turn removal, a larger win at 400, and +> unchanged peak/settled RSS. Finally rerun the NS2 n64 arm on this runtime to +> see whether its +13% peak moves now that the larger nursery no longer arms +> those majors. diff --git a/crates/perry-runtime/src/arena/block.rs b/crates/perry-runtime/src/arena/block.rs index 96e64195d1..3e29ceddb5 100644 --- a/crates/perry-runtime/src/arena/block.rs +++ b/crates/perry-runtime/src/arena/block.rs @@ -556,7 +556,7 @@ impl Arena { space, initial.object_starts_ptr(), ); - ARENA_TOTAL_BYTES.with(|t| t.set(t.get() + initial.size)); + arena_reserved_bytes_add(generation, initial.size); Arena { blocks: vec![initial], current: 0, @@ -686,7 +686,7 @@ impl Arena { } }; self.current = new_idx; - ARENA_TOTAL_BYTES.with(|t| t.set(t.get() + fresh_size)); + arena_reserved_bytes_add(self.generation, fresh_size); } fn alloc_fresh_block(&mut self, size: usize, align: usize) -> *mut u8 { @@ -1004,6 +1004,13 @@ thread_local! { /// alloc into a tombstone slot or the end, and release inside /// `arena_reset_empty_blocks`). pub(crate) static ARENA_TOTAL_BYTES: std::cell::Cell = const { std::cell::Cell::new(0) }; + + /// Cached nursery share of [`ARENA_TOTAL_BYTES`]: Eden plus BOTH survivor + /// semispaces. The ArenaBytes policy subtracts this value so the copying + /// nursery cannot arm old-space work merely by filling its from-space. + /// Maintained beside the whole-arena counter so the allocation trigger + /// remains O(1). + pub(crate) static NURSERY_RESERVED_BYTES: std::cell::Cell = const { std::cell::Cell::new(0) }; } crate::perry_thread_local! { @@ -1034,6 +1041,29 @@ crate::perry_thread_local! { } +#[inline] +pub(crate) fn arena_reserved_bytes_add(generation: HeapGeneration, bytes: usize) { + ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_add(bytes))); + if generation == HeapGeneration::Nursery { + NURSERY_RESERVED_BYTES.with(|nursery| nursery.set(nursery.get().saturating_add(bytes))); + } +} + +#[inline] +pub(crate) fn arena_reserved_bytes_sub(generation: HeapGeneration, bytes: usize) { + ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_sub(bytes))); + if generation == HeapGeneration::Nursery { + nursery_reserved_bytes_sub(bytes); + } +} + +/// Transfer already-accounted capacity out of the nursery without changing +/// the whole-arena total (whole-block promotion). +#[inline] +pub(crate) fn nursery_reserved_bytes_sub(bytes: usize) { + NURSERY_RESERVED_BYTES.with(|nursery| nursery.set(nursery.get().saturating_sub(bytes))); +} + // `ARENA` and `INLINE_STATE` below stay raw: they are NAMED `HotTls` fields // (`arena`, `inline_state`) whose addresses `tls_hot::fill` reads through the // providers further down, and a named field is one dependent load cheaper diff --git a/crates/perry-runtime/src/arena/mod.rs b/crates/perry-runtime/src/arena/mod.rs index 90a45f0c66..3d0043e008 100644 --- a/crates/perry-runtime/src/arena/mod.rs +++ b/crates/perry-runtime/src/arena/mod.rs @@ -37,12 +37,12 @@ pub(crate) use allocators::{ #[cfg(test)] pub(crate) use block::old_gen_in_use_bytes_slot_index; pub(crate) use block::{ - arena_cell_alloc, arena_cell_try_alloc_current, drain_block_pool_if_requested, - new_object_start_bitmap, old_gen_in_use_bytes_sub, release_arena_block, - request_block_pool_drain, Arena, ArenaBlock, ArenaBlockRelease, BlockPoolDrainStats, - ACTIVE_SURVIVOR, ARENA, ARENA_TOTAL_BYTES, BLOCK_SIZE, FRESH_GENERAL_BLOCK_MIN_USED_BYTES, - INLINE_STATE, LONGLIVED_ARENA, OBJECT_START_SHIFT, OLD_ARENA, OLD_GEN_IN_USE_BYTES, - SURVIVOR_ARENA_0, SURVIVOR_ARENA_1, + arena_cell_alloc, arena_cell_try_alloc_current, arena_reserved_bytes_sub, + drain_block_pool_if_requested, new_object_start_bitmap, nursery_reserved_bytes_sub, + old_gen_in_use_bytes_sub, release_arena_block, request_block_pool_drain, Arena, ArenaBlock, + ArenaBlockRelease, BlockPoolDrainStats, ACTIVE_SURVIVOR, ARENA, ARENA_TOTAL_BYTES, BLOCK_SIZE, + FRESH_GENERAL_BLOCK_MIN_USED_BYTES, INLINE_STATE, LONGLIVED_ARENA, NURSERY_RESERVED_BYTES, + OBJECT_START_SHIFT, OLD_ARENA, OLD_GEN_IN_USE_BYTES, SURVIVOR_ARENA_0, SURVIVOR_ARENA_1, }; /// #7469 hot-TLS plumbing — see `crate::tls_hot`. The `*_hot_addr` half is /// consumed by `tls_hot::fill`; the `hot_*` half is the cached accessor the @@ -92,8 +92,8 @@ pub(crate) use walk::ArenaRegionTelemetry; pub use walk::{ arena_block_count, arena_in_use_bytes, arena_total_bytes, arena_walk_objects, arena_walk_objects_addr_sorted, arena_walk_objects_filtered, - arena_walk_objects_with_block_index, general_block_count, longlived_end, - old_arena_walk_objects, ArenaResetStats, + arena_walk_objects_with_block_index, copying_nursery_reserved_bytes, general_block_count, + longlived_end, old_arena_walk_objects, old_space_total_bytes, ArenaResetStats, }; pub(crate) use walk::{ arena_block_snapshots, arena_telemetry_snapshot, general_block_in_recent_window, @@ -136,7 +136,9 @@ pub use stats::{ arena_live_allocated_bytes, js_arena_stats, longlived_in_use_bytes, old_gen_in_use_bytes, pointer_in_nursery, pointer_in_old_gen, }; -pub(crate) use stats::{arena_live_from_space_bytes, record_arena_live_census}; +pub(crate) use stats::{ + arena_live_from_space_bytes, old_space_live_allocated_bytes, record_arena_live_census, +}; #[cfg(test)] pub(crate) use stats::{old_gen_in_use_bytes_recomputed, old_gen_in_use_bytes_resync}; diff --git a/crates/perry-runtime/src/arena/promote.rs b/crates/perry-runtime/src/arena/promote.rs index 12debf1b57..441c51131d 100644 --- a/crates/perry-runtime/src/arena/promote.rs +++ b/crates/perry-runtime/src/arena/promote.rs @@ -91,16 +91,14 @@ pub(crate) struct InPlacePromotion { blocks: Vec, /// Reserved bytes the young generation is about to lose to old-gen. /// - /// This is NOT bookkeeping trivia — it is the whole of the pacing contract. - /// The arena-bytes trigger fires on `arena_total_bytes()`, and the runway - /// between two collections is therefore "Eden's free capacity + the trigger - /// step". A copying minor recycles Eden's blocks, so that capacity survives - /// the collection and the runway stays wide. Promotion HANDS THOSE BLOCKS - /// AWAY, so without giving the capacity back the runway collapses to the - /// bare step. Measured on `retain.ts`: Eden's high-water fell 57 MB → 18 MB, - /// collections went 6 → 12, and the extra pressure bought a second full - /// collection at 690 ms — a 0.81 s → 1.52 s regression from a change that - /// made every individual promotion cheaper. + /// This is NOT bookkeeping trivia — it is part of the pacing contract. + /// Whole-block promotion removes reusable Eden capacity. Although that + /// capacity now becomes visible in the old-space ArenaBytes quantity, the + /// young allocation runway still needs to be restored separately or it + /// collapses to the bare trigger step. Measured on `retain.ts`: Eden's + /// high-water fell 57 MB → 18 MB, collections went 6 → 12, and the extra + /// pressure bought a second full collection at 690 ms — a 0.81 s → 1.52 s + /// regression from a change that made every individual promotion cheaper. /// /// The capacity is given back as trigger HEADROOM /// (`gc::note_promoted_young_capacity`), not as eagerly mapped blocks. @@ -348,6 +346,10 @@ pub(crate) fn finish_in_place_promotion( retag_block_space(base, size, HeapGeneration::Old, HeapSpace::Old); } }); + // Ownership moved from Eden/survivor to old-gen, but the whole-arena + // reservation did not change. Move only the cached nursery share so the + // ArenaBytes old-space reading sees promotion as old-space growth. + nursery_reserved_bytes_sub(promotion.reserved_bytes); // Both young regions are empty now. Eden needs a usable current block for // the inline bump allocator; the survivor flip keeps the semispace diff --git a/crates/perry-runtime/src/arena/quarantine.rs b/crates/perry-runtime/src/arena/quarantine.rs index 37986e7e06..ea99e7c2ca 100644 --- a/crates/perry-runtime/src/arena/quarantine.rs +++ b/crates/perry-runtime/src/arena/quarantine.rs @@ -483,7 +483,7 @@ fn push_set_and_evict(blocks: Vec) -> Vec { } } else { // Poisoned registry. These blocks are already detached from the arena, - // unregistered, and subtracted from `ARENA_TOTAL_BYTES`, and + // unregistered, and subtracted from arena/nursery reservation totals, and // `QuarantinedBlock` has no `Drop` — dropping them here would leak the // whole from-space. Recycle immediately instead. for block in blocks { @@ -611,7 +611,7 @@ unsafe fn detach_used_blocks(arena: &mut Arena) -> Vec<(*mut u8, usize, usize)> let size = block.size; let used = block.offset; unregister_block_generation(base, size); - ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_sub(size))); + arena_reserved_bytes_sub(arena.generation, size); detached.push((block.data, size, used)); block.data = std::ptr::null_mut(); block.size = 0; diff --git a/crates/perry-runtime/src/arena/reset.rs b/crates/perry-runtime/src/arena/reset.rs index e53918b18d..81b4ba8b2a 100644 --- a/crates/perry-runtime/src/arena/reset.rs +++ b/crates/perry-runtime/src/arena/reset.rs @@ -413,7 +413,7 @@ pub fn arena_reset_empty_blocks(block_has_live: &[bool]) -> ArenaResetStats { removed_ranges.push((base, size)); let release = release_arena_block(block.data, block.size); release_stats.record_block_release(size, release); - ARENA_TOTAL_BYTES.with(|t| t.set(t.get().saturating_sub(block.size))); + arena_reserved_bytes_sub(arena.generation, block.size); block.data = std::ptr::null_mut(); block.size = 0; block.object_starts = Box::new([]); @@ -735,7 +735,7 @@ impl ArenaResetEmptyBlocksState { let size = block.size; unregister_block_generation(base, size); let release = release_arena_block(block.data, block.size); - ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_sub(size))); + arena_reserved_bytes_sub(arena.generation, size); block.data = std::ptr::null_mut(); block.size = 0; block.object_starts = Box::new([]); @@ -940,7 +940,7 @@ impl SurvivorArenaReclaimState { let size = block.size; unregister_block_generation(base, size); let release = release_arena_block(block.data, block.size); - ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_sub(size))); + arena_reserved_bytes_sub(arena.generation, size); block.data = std::ptr::null_mut(); block.size = 0; block.object_starts = Box::new([]); @@ -1255,7 +1255,7 @@ impl OldArenaReclaimDeadBlocksState { unregister_block_generation(base, size); let release = release_arena_block(block.data, block.size); - ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_sub(size))); + arena_reserved_bytes_sub(arena.generation, size); block.data = std::ptr::null_mut(); block.size = 0; block.object_starts = Box::new([]); @@ -1345,7 +1345,7 @@ pub(crate) fn old_arena_reclaim_dead_blocks(block_has_live: &[bool]) -> ArenaRes unregister_block_generation(base, size); let release = release_arena_block(block.data, block.size); - ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_sub(size))); + arena_reserved_bytes_sub(arena.generation, size); block.data = std::ptr::null_mut(); block.size = 0; block.object_starts = Box::new([]); @@ -1444,7 +1444,7 @@ pub(crate) fn old_arena_reclaim_selected_dead_blocks( unregister_block_generation(base, size); let release = release_arena_block(block.data, block.size); - ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_sub(size))); + arena_reserved_bytes_sub(arena.generation, size); block.data = std::ptr::null_mut(); block.size = 0; block.object_starts = Box::new([]); @@ -1540,7 +1540,7 @@ fn reclaim_dead_survivor_arena_blocks( let size = block.size; unregister_block_generation(base, size); let release = release_arena_block(block.data, block.size); - ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_sub(size))); + arena_reserved_bytes_sub(arena.generation, size); block.data = std::ptr::null_mut(); block.size = 0; block.object_starts = Box::new([]); diff --git a/crates/perry-runtime/src/arena/stats.rs b/crates/perry-runtime/src/arena/stats.rs index 00c67c339d..a875e41515 100644 --- a/crates/perry-runtime/src/arena/stats.rs +++ b/crates/perry-runtime/src/arena/stats.rs @@ -141,6 +141,15 @@ pub fn arena_live_allocated_bytes() -> usize { }) } +/// Header-inclusive live bytes outside the copying nursery. +/// +/// Both operands advance from the same exact post-collection census, so their +/// difference excludes nursery allocation growth between collections without +/// adding accounting to the generated bump-allocation fast path. +pub(crate) fn old_space_live_allocated_bytes() -> usize { + arena_live_allocated_bytes().saturating_sub(arena_live_from_space_bytes()) +} + /// Get arena memory statistics: (heap_used, heap_total). /// `heap_used` is live allocated object bytes; `heap_total` is total reserved /// block capacity. Allocation high-water remains separately available through diff --git a/crates/perry-runtime/src/arena/walk.rs b/crates/perry-runtime/src/arena/walk.rs index a460808116..6f70ef9de2 100644 --- a/crates/perry-runtime/src/arena/walk.rs +++ b/crates/perry-runtime/src/arena/walk.rs @@ -321,6 +321,21 @@ pub fn arena_total_bytes() -> usize { ARENA_TOTAL_BYTES.with(|t| t.get()) } +/// Reserved copying-nursery capacity: Eden plus both survivor semispaces. +/// This is the exact committed-capacity share excluded from old-space pacing. +#[inline] +pub fn copying_nursery_reserved_bytes() -> usize { + NURSERY_RESERVED_BYTES.with(|bytes| bytes.get()) +} + +/// Reserved non-nursery capacity (long-lived plus old generation). +/// ArenaBytes arming, its rebaseline, and idle right-sizing all use this one +/// definition so nursery growth cannot schedule a whole-heap sweep indirectly. +#[inline] +pub fn old_space_total_bytes() -> usize { + arena_total_bytes().saturating_sub(copying_nursery_reserved_bytes()) +} + /// Get allocation high-water bytes (sum of `block.offset` across blocks). /// /// This includes swept holes in partially-live blocks and is deliberately a diff --git a/crates/perry-runtime/src/gc/arena_right_size.rs b/crates/perry-runtime/src/gc/arena_right_size.rs index 6b892dac61..b38542cc97 100644 --- a/crates/perry-runtime/src/gc/arena_right_size.rs +++ b/crates/perry-runtime/src/gc/arena_right_size.rs @@ -146,7 +146,7 @@ fn current_usage(live_bytes: usize) -> ArenaUsage { } ArenaUsage { live_bytes, - capacity_bytes: crate::arena::arena_total_bytes(), + capacity_bytes: super::policy::arena_trigger_total_bytes(), } } diff --git a/crates/perry-runtime/src/gc/diag_sites.rs b/crates/perry-runtime/src/gc/diag_sites.rs index 5d7776bfab..954525cc44 100644 --- a/crates/perry-runtime/src/gc/diag_sites.rs +++ b/crates/perry-runtime/src/gc/diag_sites.rs @@ -11,7 +11,8 @@ //! by hand. These lines print the inputs at the decision: //! //! * `[gc-trigger] site=… kind=…` — every predicate input the trigger policy -//! reads (`arena_total` vs the armed base trigger, from-space occupancy vs +//! reads (old-space capacity vs the armed base trigger, excluded nursery +//! capacity, from-space occupancy vs //! the nursery cap, old-gen reclaimable pressure vs its baseline and band, //! the malloc-count pair, the pending/retaining flags), emitted at each //! site that decides to collect. @@ -39,6 +40,8 @@ pub(super) fn trigger_decision(site: &'static str, kind: &'static str) { return; } let arena_total = crate::arena::arena_total_bytes(); + let nursery_excluded = policy::arena_trigger_nursery_excluded_bytes(); + let old_total = policy::arena_trigger_total_bytes(); let next_base = policy::next_arena_trigger_base(); let armed = policy::GC_TRIGGER_ARMED.with(Cell::get); let from_space = crate::arena::copying_from_space_in_use_bytes(); @@ -55,7 +58,7 @@ pub(super) fn trigger_decision(site: &'static str, kind: &'static str) { let old_in_use = crate::arena::old_gen_in_use_bytes(); let old_free = old_free_bytes(); eprintln!( - "[gc-trigger] site={site} kind={kind} arena_total={arena_total} next_base={next_base} armed={armed} \ + "[gc-trigger] site={site} kind={kind} arena_total={arena_total} old_total={old_total} nursery_excluded={nursery_excluded} next_base={next_base} armed={armed} \ from_space={from_space} nursery_cap={nursery_cap} old_in_use={old_in_use} old_free={old_free} \ old_reclaimable={old_reclaimable} external_side={external} old_baseline={old_baseline} \ old_band={old_band} old_threshold={old_threshold} old_pending={old_pending} retaining={retaining} \ @@ -170,12 +173,25 @@ pub(super) fn budgeted_started( GcCollectionKind::Full => "full", GcCollectionKind::Minor => "minor", }; + let arming_reason = match trigger { + GcTriggerKind::ArenaBytes => "old_space_capacity", + GcTriggerKind::MallocCount => "malloc_count", + GcTriggerKind::OldGenBytes => "old_reclaim", + GcTriggerKind::SurvivorPromotionBytes => "survivor_promotion", + GcTriggerKind::Emergency => "emergency", + GcTriggerKind::Manual => "manual", + GcTriggerKind::Direct => "direct", + GcTriggerKind::IdleReclaim => "idle_reclaim", + GcTriggerKind::IdleCompact => "idle_compact", + }; eprintln!( - "[gc-budgeted] start trigger={trigger:?} kind={collection} progress={} old_reclaimable={} old_baseline={} arena_total={}", + "[gc-budgeted] start trigger={trigger:?} arming_reason={arming_reason} kind={collection} progress={} old_reclaimable={} old_baseline={} arena_total={} old_total={} nursery_excluded={}", progress.as_str(), policy::old_gen_reclaimable_pressure_bytes(), policy::GC_LAST_OLD_RECLAIM_IN_USE_BYTES.with(Cell::get), - crate::arena::arena_total_bytes() + crate::arena::arena_total_bytes(), + policy::arena_trigger_total_bytes(), + policy::arena_trigger_nursery_excluded_bytes() ); BUDGETED.with(|b| { *b.borrow_mut() = Some(BudgetedCycleDiag { diff --git a/crates/perry-runtime/src/gc/policy.rs b/crates/perry-runtime/src/gc/policy.rs index fed40c7b31..f4edc8cb29 100644 --- a/crates/perry-runtime/src/gc/policy.rs +++ b/crates/perry-runtime/src/gc/policy.rs @@ -9,7 +9,7 @@ crate::perry_thread_local! { pub(super) static GC_FLAGS: Cell = const { Cell::new(0) }; } -/// Threshold: run GC when total arena bytes exceed this. +/// Threshold: run GC when reserved old-space bytes exceed this. /// /// Current app-pattern tuning: 128 MB. The earlier 64 MB setting reduced /// peak RSS on JSON round-trip style workloads, but it also forced a @@ -27,7 +27,7 @@ pub(super) const GC_THRESHOLD_INITIAL_BYTES: usize = 128 * 1024 * 1024; // 128 M /// guardrail is `GC_TRIGGER_ABSOLUTE_CEILING` below. pub(super) const GC_THRESHOLD_MAX_BYTES: usize = 1024 * 1024 * 1024; // 1 GB -/// Hard ceiling on the next-GC trigger (arena_total bytes), independent +/// Hard ceiling on the next-GC trigger (old-space reserved bytes), independent /// of how productive recent sweeps have been. Without this, the /// >90%-freed branch doubles the step on every productive collection, /// > and `next_trigger = new_total + step` lets peak nursery occupancy @@ -42,11 +42,11 @@ pub(super) const GC_THRESHOLD_MAX_BYTES: usize = 1024 * 1024 * 1024; // 1 GB /// > to roughly initial + one iter's allocation buffer + headroom for /// > non-arena overhead. /// -/// Floor: even if `arena_total` is already near or past the ceiling +/// Floor: even if old-space capacity is already near or past the ceiling /// (large old-gen + longlived combined live set), keep at least the /// 16 MB step floor as headroom — `next_trigger = max(new_total + 16 MB, /// min(new_total + step, ceiling))`. This avoids GC thrash when the -/// non-nursery component of arena_total alone exceeds the ceiling. +/// old-space component alone exceeds the ceiling. /// /// 2026-05-02 raise from 64 MB → 128 MB: ECS perf-comprehensive's /// allocation-heavy benches (10k two-comp + sync, 5k × 3 cmds) hit @@ -78,9 +78,8 @@ pub(super) const GC_TRIGGER_ABSOLUTE_CEILING: usize = 128 * 1024 * 1024; /// device-derived ceiling while the cell still holds its desktop-default /// const initializer. /// The adaptive/armed arena trigger WITHOUT the scavenge nursery cap: -/// compared against `arena_total_bytes()` (all generations), as it always -/// was. The cap is deliberately not part of this value — it is -/// young-generation-scoped and lives in [`young_scavenge_cap_due`]. +/// compared against [`crate::arena::old_space_total_bytes`]. Nursery capacity +/// has its own young-generation trigger in [`young_scavenge_cap_due`]. pub(super) fn next_arena_trigger_base() -> usize { if GC_TRIGGER_ARMED.with(|a| a.get()) { GC_NEXT_TRIGGER_BYTES.with(|c| c.get()) @@ -91,6 +90,42 @@ pub(super) fn next_arena_trigger_base() -> usize { } } +/// Committed capacity priced by the ArenaBytes arm. +/// +/// In the generational collector this is old/long-lived capacity only: Eden +/// and both survivor semispaces are owned by the copying minor's independent +/// cap. The non-generational escape hatch has no copying nursery, so it keeps +/// the historical whole-arena basis. +#[inline] +pub(super) fn arena_trigger_total_bytes() -> usize { + if gen_gc_enabled() { + crate::arena::old_space_total_bytes() + } else { + crate::arena::arena_total_bytes() + } +} + +/// Live-allocation twin of [`arena_trigger_total_bytes`], used by tiny-parse +/// pressure and idle right-sizing so neither path reintroduces the copying +/// nursery into whole-heap work. +#[inline] +pub(super) fn arena_trigger_live_bytes() -> usize { + if gen_gc_enabled() { + crate::arena::old_space_live_allocated_bytes() + } else { + crate::arena::arena_live_allocated_bytes() + } +} + +#[inline] +pub(super) fn arena_trigger_nursery_excluded_bytes() -> usize { + if gen_gc_enabled() { + crate::arena::copying_nursery_reserved_bytes() + } else { + 0 + } +} + /// True when the young generation (Eden + active survivor space) has /// reached the effective scavenge nursery cap. /// @@ -344,7 +379,7 @@ crate::perry_thread_local! { pub(super) static GC_TRIGGER_BUMPED: std::cell::Cell = const { std::cell::Cell::new(false) }; - /// Issue #745: snapshot of `arena_total_bytes()` at the most + /// Issue #745: snapshot of `old_space_total_bytes()` at the most /// recent `gc_suppress` call. Used by `gc_bump_malloc_trigger` /// to compute the suppressed window's arena growth. pub(super) static GC_PRE_SUPPRESS_BYTES: std::cell::Cell = @@ -404,7 +439,7 @@ pub(super) fn tiny_parse_pressure_headroom_bytes(step: usize) -> usize { /// while the adaptive step sat at its 1 GiB maximum saying "back off" and /// nothing consulted it. The growth clause is what that step is for. /// -/// `base` is `arena_in_use_bytes()` as the last collection ended +/// `base` is `old_space_live_allocated_bytes()` as the last collection ended /// (`GC_TINY_PARSE_PRESSURE_BASE_BYTES`); `in_use` is the same reading now. pub(super) fn tiny_parse_pressure_due_with( in_use: usize, @@ -1066,11 +1101,10 @@ crate::perry_thread_local! { /// reading and still escalate. Nursery garbage is not. pub(super) static GC_LAST_COLLECTION_POST_IN_USE_BYTES: Cell = const { Cell::new(0) }; - /// #9831: `arena_in_use_bytes()` as the most recent collection of ANY kind - /// ended — the base the tiny-parse pressure guard measures growth from. - /// The bump-offset reading rather than the live census above, because the - /// guard compares against `arena_in_use_bytes()` at every parse boundary, - /// and mixing the two would count every swept hole as growth. + /// #9831: `old_space_live_allocated_bytes()` as the most recent collection + /// of ANY kind ended — the base the tiny-parse pressure guard measures + /// growth from. The same running live-census accessor is read at parse + /// boundaries, so nursery bump growth and swept holes are both excluded. pub(super) static GC_TINY_PARSE_PRESSURE_BASE_BYTES: Cell = const { Cell::new(0) }; /// Yield-adaptive backoff for major-GC pacing (#7726). /// @@ -1364,10 +1398,10 @@ pub fn gc_suppress() { { crate::arena::arena_start_fresh_general_block(); } - // Issue #745: snapshot arena_total at suppress-start so the + // Issue #745: snapshot old-space capacity at suppress-start so the // matching `gc_bump_malloc_trigger` can size the suppressed // window's parse growth and gate the bytes-trigger bump on it. - GC_PRE_SUPPRESS_BYTES.with(|c| c.set(crate::arena::arena_total_bytes())); + GC_PRE_SUPPRESS_BYTES.with(|c| c.set(arena_trigger_total_bytes())); GC_FLAGS.with(|f| f.set(f.get() | GC_FLAG_SUPPRESSED)); } @@ -1434,8 +1468,7 @@ impl Drop for GcSuppressScope { /// of four. pub fn gc_bump_malloc_trigger() { let current = MALLOC_STATE.with(|s| s.borrow().objects.len()); - use crate::arena::arena_total_bytes; - let bytes_now = arena_total_bytes(); + let bytes_now = arena_trigger_total_bytes(); let is_tiny_parse = gc_bump_malloc_trigger_with_snapshot(current, bytes_now); if is_tiny_parse { let use_gen_gc = gen_gc_enabled(); @@ -1443,7 +1476,7 @@ pub fn gc_bump_malloc_trigger() { // EVERY tiny parse on a program whose live set never drops below it. // The guard now also requires the arena to have grown past the // productivity-priced headroom since the last collection ended. - let in_use = crate::arena::arena_in_use_bytes(); + let in_use = arena_trigger_live_bytes(); if !tiny_parse_pressure_due(in_use, tiny_parse_in_use_trigger_for_mode()) { return; } @@ -1493,13 +1526,13 @@ pub fn gc_collect_pending_suppressed_parse() { // since then has moved the base, and re-pricing here is what keeps the // boundary collection from stacking a second minor on top of it. A request // nothing has satisfied is still due and still collects. - let in_use = crate::arena::arena_in_use_bytes(); + let in_use = arena_trigger_live_bytes(); if !tiny_parse_pressure_due(in_use, tiny_parse_in_use_trigger_for_mode()) { return; } diag_tiny_parse_forced_collection("parse_boundary", in_use); - let total = crate::arena::arena_total_bytes(); + let total = arena_trigger_total_bytes(); GC_NEXT_TRIGGER_BYTES.with(|trigger| { if trigger.get() > total { trigger.set(total); @@ -1524,7 +1557,7 @@ pub fn gc_schedule_parse_boundary_collection_if_pressure() { // #9831: priced the same way as the post-parse guard above — see // `tiny_parse_pressure_due_with`. if !tiny_parse_pressure_due( - crate::arena::arena_in_use_bytes(), + arena_trigger_live_bytes(), gc_tiny_parse_in_use_trigger_dyn_bytes(), ) { return; @@ -2023,8 +2056,8 @@ pub(super) fn pacing_arena_in_use_bytes() -> usize { /// | cell | unit | read by | /// |---|---|---| /// | `GC_LAST_COLLECTION_POST_IN_USE_BYTES` | `pacing_arena_in_use_bytes()` — the LIVE census (`arena_live_allocated_bytes`), test-injectable | `arena_growth_full_escalation_due` | -/// | `GC_TINY_PARSE_PRESSURE_BASE_BYTES` (#9831) | `arena_in_use_bytes()` — BUMP OFFSETS, the same reading the guard takes at each parse boundary | `tiny_parse_pressure_due_with` | -/// | `GC_NEXT_TRIGGER_BYTES` (#9840) | `arena_total_bytes()` — COMMITTED bytes, which is what `next_arena_trigger_base()` is compared against | `gc_budgeted_due_trigger`'s `ArenaBytes` arm | +/// | `GC_TINY_PARSE_PRESSURE_BASE_BYTES` (#9831) | `old_space_live_allocated_bytes()` — live non-nursery bytes, the same reading the guard takes at each parse boundary | `tiny_parse_pressure_due_with` | +/// | `GC_NEXT_TRIGGER_BYTES` (#9840) | `old_space_total_bytes()` — committed non-nursery bytes, which is what `next_arena_trigger_base()` is compared against | `gc_budgeted_due_trigger`'s `ArenaBytes` arm | /// /// The third is *not* written here, and that is deliberate rather than an /// omission: re-arming the arena trigger needs the collection's productivity @@ -2036,14 +2069,14 @@ pub(super) fn pacing_arena_in_use_bytes() -> usize { /// collection finishers call; #9840 is the change that made that "both" /// true, and it is the same symmetry #9831 applied to the cell above. /// Mixing the units — re-baselining a committed-bytes trigger from a bump- -/// offset or live-census base — would arm it below the arena's own total and -/// make the arm due the instant it re-armed. +/// offset or live-census base would mix incompatible units. pub(super) fn note_collection_finished_arena_occupancy(full: bool) { let bytes = pacing_arena_in_use_bytes(); GC_LAST_COLLECTION_POST_IN_USE_BYTES.with(|cell| cell.set(bytes)); // #9831: the same moment, in the units the tiny-parse guard reads. - GC_TINY_PARSE_PRESSURE_BASE_BYTES.with(|cell| cell.set(crate::arena::arena_in_use_bytes())); - super::arena_right_size::note_collection_finished(bytes, full); + let old_live = arena_trigger_live_bytes(); + GC_TINY_PARSE_PRESSURE_BASE_BYTES.with(|cell| cell.set(old_live)); + super::arena_right_size::note_collection_finished(old_live, full); } /// The arena reading [`arena_growth_full_escalation_due`] tests — see @@ -2221,7 +2254,7 @@ fn gc_rebaseline_malloc_trigger_to_survivors(mstep: usize) { GC_NEXT_MALLOC_TRIGGER.with(|c| c.set(survivors + mstep)); } -/// Which nursery-collection finisher is re-baselining the whole-arena trigger. +/// Which nursery-collection finisher is re-baselining the old-space trigger. /// Diagnostic attribution only — the arithmetic is identical for both. #[derive(Clone, Copy)] enum ArenaRebaselineArm { @@ -2254,8 +2287,8 @@ impl ArenaRebaselineArm { /// still here (see `gc_finish_arena_trigger_collection`): an arena minor that /// skipped the malloc sweep must not move the malloc trigger. /// -/// **Unit.** The base is `arena_total_bytes()` — COMMITTED bytes, all -/// generations — because that is the quantity `next_arena_trigger_base()` is +/// **Unit.** The base is `old_space_total_bytes()` — COMMITTED non-nursery +/// bytes — because that is the quantity `next_arena_trigger_base()` is /// compared against in `gc_budgeted_due_trigger`. It is deliberately neither /// of the two post-collection occupancy readings published at /// [`note_collection_finished_arena_occupancy`] (a live census, and #9831's @@ -2263,7 +2296,7 @@ impl ArenaRebaselineArm { /// Re-baselining this cell from either of those would arm the trigger below /// the arena's own total and make the arm due the instant it re-armed. /// -/// **What is in scope.** The two *nursery-trigger* finishers, whichever +/// **What is in scope.** The two allocation-trigger finishers, whichever /// collection their arm ended up running — an `ArenaBytes` or `MallocCount` /// trigger that `arena_growth_full_escalation_due()` escalated to a full still /// finishes here, exactly as the arena arm's escalated fulls already did before @@ -2427,10 +2460,10 @@ fn gc_rebaseline_arena_trigger_after_collection( ); } } - let new_total = crate::arena::arena_total_bytes(); + let new_total = arena_trigger_total_bytes(); // C4b-δ-tune: hard cap on next_trigger so the >90%-freed - // step-doubling can't drive peak nursery past the initial - // threshold. Floor: at least 16 MB of headroom past + // step-doubling can't leave the old-space trigger unbounded. Floor: at + // least 16 MB of headroom past // `new_total` so a workload whose post-GC live set already // approaches the ceiling doesn't thrash on every fresh // allocation. @@ -2451,10 +2484,10 @@ fn gc_rebaseline_arena_trigger_after_collection( let capped = stepped.min(gc_trigger_absolute_ceiling_bytes()); let floor = new_total.saturating_add(gc_trigger_headroom_floor_bytes()); // #7742: whole-block promotion hands Eden's blocks to old-gen instead of - // recycling them, so the free young capacity that would have carried the - // mutator to the next collection is gone from `new_total`. Give it back as - // headroom (consumed once) rather than by re-reserving the blocks, which - // would map memory the program may never reach. + // recycling them. `new_total` now sees those blocks as old space, but the + // free young capacity that would have carried the mutator to the next + // collection is still gone. Give that runway back as headroom (consumed + // once) rather than by re-reserving blocks the program may never reach. let next_trigger = std::cmp::max(capped, floor).saturating_add(super::take_promoted_young_capacity_credit()); GC_NEXT_TRIGGER_BYTES.with(|c| c.set(next_trigger)); @@ -2465,10 +2498,12 @@ fn gc_rebaseline_arena_trigger_after_collection( // every `sweep_freed=`/`freed_bytes=` it can grep, so a second line // carrying those keys would double-count reclaim in the ratchet. eprintln!( - "[gc-arena-rebaseline] arm={} next_trigger={} total={} headroom={} pct={}% step={}→{}", + "[gc-arena-rebaseline] arm={} next_trigger={} total={} old_total={} nursery_excluded={} headroom={} pct={}% step={}→{}", arm.label(), next_trigger, new_total, + new_total, + arena_trigger_nursery_excluded_bytes(), next_trigger.saturating_sub(new_total), scored_pct_freed, old_step, @@ -2596,8 +2631,9 @@ fn gc_finish_malloc_trigger_collection( if outcome.malloc_swept { GC_NEXT_MALLOC_TRIGGER.with(|c| c.set(survivors + mstep)); } - // #9840: this collection swept the nursery too, so the whole-arena trigger - // is measured from after it — see `gc_rebaseline_arena_trigger_after_collection`. + // #9840: this collection can promote into or reclaim old space too, so the + // old-space trigger is measured from after it — see + // `gc_rebaseline_arena_trigger_after_collection`. if arena_rebaseline_all_enabled() { gc_rebaseline_arena_trigger_after_collection( pre_in_use, @@ -3115,12 +3151,10 @@ fn gc_budgeted_due_trigger() -> Option { return Some(BudgetedGcTrigger::OldReclaim); } - // Two separately-scoped arena arms (see `young_scavenge_cap_due` for why - // they must not share a basis): the adaptive base trigger against the - // whole arena, and the scavenge nursery cap against the young generation - // only. - let total = crate::arena::arena_total_bytes(); - if total >= next_arena_trigger_base() { + // Two separately-scoped arms: ArenaBytes is old-space capacity only; the + // scavenge cap owns nursery occupancy and hands it to the copying minor. + let old_total = arena_trigger_total_bytes(); + if old_total >= next_arena_trigger_base() { return Some(BudgetedGcTrigger::ArenaBytes); } if young_scavenge_cap_due() { diff --git a/crates/perry-runtime/src/gc/pressure.rs b/crates/perry-runtime/src/gc/pressure.rs index ab907dbd5c..b12f40e337 100644 --- a/crates/perry-runtime/src/gc/pressure.rs +++ b/crates/perry-runtime/src/gc/pressure.rs @@ -64,9 +64,12 @@ pub extern "C" fn js_gc_memory_pressure(level: u32) -> u32 { // Pull the arena trigger down to "collect at the next check" and arm // it so the un-armed budget ceiling substitution doesn't override the // clamp (see `effective_next_arena_trigger`). - let total = crate::arena::arena_total_bytes(); + // ArenaBytes is paced in old-space units. Keep this explicit-pressure + // clamp in those same units so a reserved copying nursery cannot affect + // the trigger that the next allocation-side check evaluates. + let trigger_total = arena_trigger_total_bytes(); GC_NEXT_TRIGGER_BYTES.with(|c| { - let clamp = total.saturating_add(1024 * 1024); + let clamp = trigger_total.saturating_add(1024 * 1024); if c.get() > clamp { c.set(clamp); GC_TRIGGER_ARMED.with(|a| a.set(true)); @@ -98,7 +101,10 @@ pub extern "C" fn js_gc_memory_pressure(level: u32) -> u32 { // reads, so the safepoint drain runs a full mark-sweep rather than a minor. if roots::shadow_stack_has_active_frame() { if !GC_SAFEPOINT_PENDING.with(std::cell::Cell::get) { - GC_SAFEPOINT_DEFER_ARENA_BASE.with(|base| base.set(total)); + // The deferral slack valve is an RSS safety bound, not an + // ArenaBytes trigger. It intentionally watches whole-arena growth + // while this request waits for a precise safepoint. + GC_SAFEPOINT_DEFER_ARENA_BASE.with(|base| base.set(crate::arena::arena_total_bytes())); // Through the helper, never the `Cell`: it also arms the global // shadow that codegen's inline poll check reads. See // `policy::set_safepoint_pending`. diff --git a/crates/perry-runtime/src/gc/telemetry.rs b/crates/perry-runtime/src/gc/telemetry.rs index ac62c3b605..e35f03dfb1 100644 --- a/crates/perry-runtime/src/gc/telemetry.rs +++ b/crates/perry-runtime/src/gc/telemetry.rs @@ -715,27 +715,18 @@ pub(super) struct GcDebtSnapshot { impl GcDebtSnapshot { #[inline] pub(super) fn current() -> Self { - let total = crate::arena::arena_total_bytes(); - // #6950: read the SAME trigger the arming path compares against. - // `gc_budgeted_due_trigger` uses `effective_next_arena_trigger()`, which - // substitutes the device/`PERRY_GC_HEAP_LIMIT`-derived ceiling while the - // raw cell still holds its 128 MB desktop-default const initializer - // (`GC_TRIGGER_ARMED == false`). Reading the raw cell here made the two - // disagree: a cycle armed at a 2 MB effective trigger measured its own - // debt against 128 MB and therefore reported ZERO debt, so - // `gc_mutator_assist_scaled_work_units` never scaled past its 256-unit - // floor and the budgeted cycle crawled without ever completing — - // 300k escaping allocations / 330 MB RSS with ZERO collections. That is - // exactly the unbounded-growth failure the debt-proportional pacing was - // introduced to prevent. - let next_arena_trigger = effective_next_arena_trigger(); + let old_total = arena_trigger_total_bytes(); + // Read the SAME old-space base the ArenaBytes arm compares against. + // Nursery occupancy has its own scavenge trigger and must not inflate + // the debt that scales whole-heap budgeted work. + let next_arena_trigger = next_arena_trigger_base(); let malloc_count = malloc_object_count(); let next_malloc_trigger = GC_NEXT_MALLOC_TRIGGER.with(|c| c.get()); let old_in_use = crate::arena::old_gen_in_use_bytes(); let old_baseline = GC_LAST_OLD_RECLAIM_IN_USE_BYTES.with(|bytes| bytes.get()); Self { - arena_debt_bytes: total.saturating_sub(next_arena_trigger) as u64, + arena_debt_bytes: old_total.saturating_sub(next_arena_trigger) as u64, malloc_debt_objects: malloc_count.saturating_sub(next_malloc_trigger) as u64, old_reclaim_debt_bytes: gc_old_reclaim_debt_bytes(old_in_use, old_baseline), } diff --git a/crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs b/crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs new file mode 100644 index 0000000000..b88155f296 --- /dev/null +++ b/crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs @@ -0,0 +1,201 @@ +use super::super::*; +use super::support::*; + +const NURSERY_OBJECT_BYTES: usize = 8 * 1024; + +struct TriggerArmedTestGuard(bool); + +impl TriggerArmedTestGuard { + fn new() -> Self { + Self(GC_TRIGGER_ARMED.with(std::cell::Cell::get)) + } +} + +impl Drop for TriggerArmedTestGuard { + fn drop(&mut self) { + GC_TRIGGER_ARMED.with(|armed| armed.set(self.0)); + } +} + +struct MajorPacingReadingTestGuard(Option); + +impl MajorPacingReadingTestGuard { + fn keep_minor() -> Self { + Self(super::super::policy::test_set_pacing_arena_in_use(Some(0))) + } +} + +impl Drop for MajorPacingReadingTestGuard { + fn drop(&mut self) { + super::super::policy::test_set_pacing_arena_in_use(self.0); + } +} + +fn reset_old_reclaim_pressure() { + let old_in_use = + old_gen_reclaimable_pressure_bytes().saturating_add(external_side_live_bytes()); + GC_LAST_OLD_RECLAIM_IN_USE_BYTES.with(|bytes| bytes.set(old_in_use)); + GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); +} + +fn fill_dead_nursery_to(bytes: usize) { + while crate::arena::copying_from_space_in_use_bytes() < bytes { + let _ = crate::arena::arena_alloc_gc(NURSERY_OBJECT_BYTES, 8, crate::gc::GC_TYPE_STRING); + } +} + +fn collect_unrooted_nursery() { + gc_collect_forced_evacuating_minor(GcTriggerSnapshot::capture(GcTriggerKind::Direct)) + .emit_after_current(); +} + +/// Sabotage: compare `arena_total_bytes()` instead of +/// `arena_trigger_total_bytes()` in `gc_budgeted_due_trigger`. The filled +/// nursery then crosses `next_base`, increments the budgeted-start counter, +/// and makes the first assertion fail. +#[test] +fn full_dead_nursery_does_not_arm_budgeted_cycle_and_next_minor_reclaims_it() { + let _guard = CopyingNurseryTestGuard::new(0); + let _moving = force_moving_gc_pacing(); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + let _armed = TriggerArmedTestGuard::new(); + let _minor = MajorPacingReadingTestGuard::keep_minor(); + reset_old_reclaim_pressure(); + + let old_total = arena_trigger_total_bytes(); + let next_base = old_total.saturating_add(gc_trigger_headroom_floor_bytes()); + GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.set(next_base)); + GC_TRIGGER_ARMED.with(|armed| armed.set(true)); + + fill_dead_nursery_to(gc_scavenge_nursery_cap_bytes()); + let from_space_before = crate::arena::copying_from_space_in_use_bytes(); + assert!( + from_space_before >= gc_scavenge_nursery_cap_bytes(), + "fixture must fill the copying nursery to its cap" + ); + assert_eq!( + arena_trigger_total_bytes(), + old_total, + "nursery allocation must not grow the old-space trigger quantity" + ); + assert!( + crate::arena::arena_total_bytes() >= next_base, + "SABOTAGE WITNESS: restoring whole-arena bytes must make the old predicate due" + ); + + let starts_before = super::super::instruments::incremental_cycle_starts(); + let mut result = JsGcStepResult::default(); + assert_eq!( + js_gc_step_work_units(1, &mut result), + JS_GC_STEP_STATUS_IDLE, + "a full nursery with old space below next_base must not start budgeted work" + ); + assert_eq!( + super::super::instruments::incremental_cycle_starts(), + starts_before, + "the budgeted-cycle start counter prices the work this test excludes" + ); + + let copied_before = copying_minor_cycles(); + assert!( + gc_safepoint_moving_minor(), + "the precise safepoint must consume the deferred nursery arm" + ); + assert_eq!( + copying_minor_cycles(), + copied_before + 1, + "the replacement copying minor must actually run" + ); + assert!( + crate::arena::copying_from_space_in_use_bytes() < from_space_before, + "the next minor must reclaim the dead nursery" + ); +} + +/// Sabotage: omit `nursery_reserved_bytes_sub` from +/// `finish_in_place_promotion`. The promoted blocks remain excluded, so the +/// ArenaBytes arm stays below `next_base` and the start counter does not move. +#[test] +fn promotion_past_old_space_headroom_arms_one_budgeted_cycle_with_empty_nursery() { + let _guard = CopyingNurseryTestGuard::new(0); + let _legacy = force_legacy_gc_pacing(); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + let _armed = TriggerArmedTestGuard::new(); + + collect_unrooted_nursery(); + assert_eq!(crate::arena::copying_from_space_in_use_bytes(), 0); + + let old_total_before = arena_trigger_total_bytes(); + let next_base = old_total_before.saturating_add(gc_trigger_headroom_floor_bytes()); + GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.set(next_base)); + GC_TRIGGER_ARMED.with(|armed| armed.set(true)); + + fill_dead_nursery_to(gc_trigger_headroom_floor_bytes() + crate::arena::BLOCK_SIZE); + let total_before_promotion = crate::arena::arena_total_bytes(); + let nursery_before_promotion = crate::arena::copying_nursery_reserved_bytes(); + let promotion = crate::arena::retag_young_for_in_place_promotion(false); + let promoted_reserved_bytes = promotion.reserved_bytes(); + assert!( + promoted_reserved_bytes > gc_trigger_headroom_floor_bytes(), + "fixture must promote more than the unchanged headroom" + ); + assert_eq!( + promoted_reserved_bytes, nursery_before_promotion, + "fixture must promote every reserved nursery block so the mandatory \ + replacement Eden head is the only new reservation" + ); + crate::arena::finish_in_place_promotion( + promotion, + crate::arena::PromotionLiveness::AssumeAllLive, + ); + + assert_eq!( + crate::arena::arena_total_bytes(), + total_before_promotion.saturating_add(crate::arena::BLOCK_SIZE), + "promotion transfers its existing capacity; only the mandatory empty \ + Eden allocator head reserves one new block" + ); + assert_eq!( + crate::arena::copying_nursery_reserved_bytes(), + crate::arena::BLOCK_SIZE, + "the replacement Eden head must remain accounted as nursery capacity" + ); + assert_eq!( + arena_trigger_total_bytes(), + old_total_before.saturating_add(promoted_reserved_bytes), + "promotion must transfer exactly its reserved capacity into old-space \ + pacing; the replacement Eden head stays excluded" + ); + assert_eq!( + crate::arena::copying_from_space_in_use_bytes(), + 0, + "the promoted nursery must be empty before the arming decision" + ); + assert!( + arena_trigger_total_bytes() >= next_base, + "promoted capacity must become visible to old-space pacing" + ); + reset_old_reclaim_pressure(); + + let starts_before = super::super::instruments::incremental_cycle_starts(); + let mut result = JsGcStepResult::default(); + assert_eq!( + js_gc_step_work_units(1, &mut result), + JS_GC_STEP_STATUS_ACTIVE + ); + assert_eq!(result.trigger_kind, GcTriggerKind::ArenaBytes.ffi_code()); + assert_eq!( + super::super::instruments::incremental_cycle_starts(), + starts_before + 1, + "old-space growth past the headroom must arm exactly one cycle" + ); + assert_eq!( + complete_budgeted_gc_cycle().status, + JS_GC_STEP_STATUS_COMPLETED + ); + + // The promoted fixture is deliberately unrooted; clean it out of old-gen + // so this sabotage test leaves no large live-looking cohort behind. + gc_collect_full_mark_sweep_with_trigger(GcTriggerSnapshot::capture(GcTriggerKind::Manual)) + .emit_after_current(); +} diff --git a/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs b/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs index aed0b6c681..e07e9cefe2 100644 --- a/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs +++ b/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs @@ -389,7 +389,7 @@ fn test_budgeted_malloc_minor_rebaselines_the_whole_arena_trigger() { ); // Arena arm armed but NOT due (1 MB of headroom); malloc pressure due. - let arena_total_before = crate::arena::arena_total_bytes(); + let arena_total_before = arena_trigger_total_bytes(); let stale_trigger = arena_total_before + 1024 * 1024; GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.set(stale_trigger)); trigger_guard.make_malloc_sweep_due(); @@ -422,7 +422,7 @@ fn test_budgeted_malloc_minor_rebaselines_the_whole_arena_trigger() { ); // (1) The budgeted finisher re-baselined the whole-arena trigger too. - let arena_total_after = crate::arena::arena_total_bytes(); + let arena_total_after = arena_trigger_total_bytes(); let next_trigger = GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.get()); assert!( next_trigger >= arena_total_after + gc_trigger_headroom_floor_bytes(), @@ -446,7 +446,7 @@ fn test_budgeted_malloc_minor_rebaselines_the_whole_arena_trigger() { )); } assert!( - crate::arena::arena_total_bytes() > stale_trigger, + arena_trigger_total_bytes() > stale_trigger, "the filler must grow the arena total past the PRE-collection trigger, \ or the assertion below cannot distinguish the two behaviours" ); diff --git a/crates/perry-runtime/src/gc/tests/debt_pacer.rs b/crates/perry-runtime/src/gc/tests/debt_pacer.rs index dfb1361181..49734c11d4 100644 --- a/crates/perry-runtime/src/gc/tests/debt_pacer.rs +++ b/crates/perry-runtime/src/gc/tests/debt_pacer.rs @@ -68,7 +68,7 @@ fn arena_threshold_debt_starts_bounded_assist_without_monolithic_collection() { assert_string_bytes(live_after, b"arena_debt_live"); } assert!( - GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.get()) > crate::arena::arena_total_bytes(), + GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.get()) > arena_trigger_total_bytes(), "completed arena debt cycle should rebaseline the heap goal" ); } @@ -430,7 +430,7 @@ fn direct_malloc_minor_also_rebaselines_the_whole_arena_trigger() { // The arena arm is ARMED BUT NOT DUE — 1 MB of headroom left. This is the // state the cc captures show at the start of a `MallocCount` run: the arena // arm re-baselined a while ago and the total has not yet reached it. - let arena_total_before = crate::arena::arena_total_bytes(); + let arena_total_before = arena_trigger_total_bytes(); GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.set(arena_total_before + 1024 * 1024)); // ...and malloc pressure IS due, so the direct minor takes the MallocCount arm. trigger_guard.make_malloc_sweep_due(); @@ -452,7 +452,7 @@ fn direct_malloc_minor_also_rebaselines_the_whole_arena_trigger() { // (1) The whole-arena trigger is measured from what THIS collection left // behind, with the same headroom floor the arena finisher applies. - let arena_total_after = crate::arena::arena_total_bytes(); + let arena_total_after = arena_trigger_total_bytes(); let next_trigger = GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.get()); assert!( next_trigger >= arena_total_after + gc_trigger_headroom_floor_bytes(), @@ -479,11 +479,11 @@ fn direct_malloc_minor_also_rebaselines_the_whole_arena_trigger() { )); } assert!( - crate::arena::arena_total_bytes() > arena_total_before + 1024 * 1024, + arena_trigger_total_bytes() > arena_total_before + 1024 * 1024, "the filler must grow the arena total past the PRE-collection trigger \ value, or the second assertion cannot distinguish the two behaviours \ (total={}, pre-collection trigger={})", - crate::arena::arena_total_bytes(), + arena_trigger_total_bytes(), arena_total_before + 1024 * 1024 ); reset_old_reclaim_pressure(); @@ -537,7 +537,7 @@ fn direct_malloc_minor_arena_rebaseline_kill_switch_restores_the_stale_threshold js_shadow_slot_set(0, ptr_bits(live_malloc as usize)); let churn_headers = allocate_dead_malloc_churn_headers(128); - let arena_total_before = crate::arena::arena_total_bytes(); + let arena_total_before = arena_trigger_total_bytes(); let stale_trigger = arena_total_before + 1024 * 1024; GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.set(stale_trigger)); trigger_guard.make_malloc_sweep_due(); @@ -579,9 +579,9 @@ fn direct_malloc_minor_arena_rebaseline_kill_switch_restores_the_stale_threshold )); } assert!( - crate::arena::arena_total_bytes() > stale_trigger, + arena_trigger_total_bytes() > stale_trigger, "the filler must cross the stale threshold (total={}, stale_trigger={})", - crate::arena::arena_total_bytes(), + arena_trigger_total_bytes(), stale_trigger ); reset_old_reclaim_pressure(); @@ -608,6 +608,10 @@ fn mutator_assist_work_units_scale_with_debt() { let _legacy_pacing = crate::gc::policy::force_legacy_gc_pacing(); let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + let _old_pressure_subject = + crate::arena::arena_alloc_gc_old(2 * 1024 * 1024, 8, GC_TYPE_STRING); + reset_old_reclaim_pressure(); + // Suppressed triggers (usize::MAX) → both debts read zero → base budget. assert_eq!( gc_mutator_assist_scaled_work_units(), @@ -615,7 +619,7 @@ fn mutator_assist_work_units_scale_with_debt() { ); // Arena debt scales at GC_ASSIST_DEBT_BYTES_PER_WORK_UNIT bytes per unit. - let total = crate::arena::arena_total_bytes(); + let total = arena_trigger_total_bytes(); let arena_debt = (2 * 1024 * 1024).min(total); GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.set(total - arena_debt)); let expected = GC_MUTATOR_ASSIST_WORK_UNITS @@ -655,8 +659,14 @@ fn debt_scaled_assists_cannot_be_outrun_by_allocation() { let _ = young_leaf(); } + // ArenaBytes debt is old-space debt now; keep the large nursery as the + // collection workload, but arm the assist with real old capacity. + let _old_pressure_subject = + crate::arena::arena_alloc_gc_old(4 * 1024 * 1024, 8, GC_TYPE_STRING); + reset_old_reclaim_pressure(); + // Simulate the collector having fallen far behind: several MB of debt. - let total = crate::arena::arena_total_bytes(); + let total = arena_trigger_total_bytes(); let debt = (total / 2).max(1); GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.set(total - debt)); assert!( @@ -1026,10 +1036,17 @@ fn test_arena_debt_measured_against_effective_trigger_not_raw_cell() { GC_NEXT_TRIGGER_BYTES.with(|c| c.set(usize::MAX / 2)); let ceiling = gc_trigger_absolute_ceiling_bytes(); let overshoot = 64 * 1024 * 1024; - crate::arena::ARENA_TOTAL_BYTES.with(|c| c.set(ceiling.saturating_add(overshoot))); + let nursery = crate::arena::copying_nursery_reserved_bytes(); + crate::arena::ARENA_TOTAL_BYTES.with(|c| { + c.set( + nursery + .saturating_add(ceiling) + .saturating_add(overshoot), + ) + }); let effective = effective_next_arena_trigger(); - let due = crate::arena::arena_total_bytes() >= effective; + let due = arena_trigger_total_bytes() >= effective; let debt = GcDebtSnapshot::current().arena_debt_bytes; let units = gc_mutator_assist_scaled_work_units(); diff --git a/crates/perry-runtime/src/gc/tests/host_safepoints.rs b/crates/perry-runtime/src/gc/tests/host_safepoints.rs index 743c49c015..0474f0d629 100644 --- a/crates/perry-runtime/src/gc/tests/host_safepoints.rs +++ b/crates/perry-runtime/src/gc/tests/host_safepoints.rs @@ -131,7 +131,7 @@ fn repeated_runtime_safepoints_complete_cycle_rebaseline_debt_and_preserve_roots assert_eq!(completed.completed, 1); assert_eq!(completed.arena_debt_bytes, 0); assert!( - GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.get()) > crate::arena::arena_total_bytes(), + GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.get()) > arena_trigger_total_bytes(), "completed safepoint cycle should rebaseline the arena trigger" ); @@ -333,7 +333,7 @@ fn an_active_budgeted_cycle_locks_out_the_moving_minor_and_keeps_the_barrier_arm // assertion below is equally satisfied by a fixture with nothing due — // CLAUDE.md, the fourth way a gate cannot fail. assert!( - crate::arena::arena_total_bytes() >= GC_NEXT_TRIGGER_BYTES.with(|t| t.get()), + arena_trigger_total_bytes() >= GC_NEXT_TRIGGER_BYTES.with(|t| t.get()), "fixture must present a due arena trigger" ); @@ -450,8 +450,8 @@ fn a_nursery_cap_only_trigger_is_deferred_to_the_collector_that_can_discharge_it ); // ...and it is the ONLY thing due, so the refusal below can only be about it. assert!( - crate::arena::arena_total_bytes() < super::super::policy::next_arena_trigger_base(), - "the whole-arena trigger must NOT be due in phase 1" + arena_trigger_total_bytes() < super::super::policy::next_arena_trigger_base(), + "the old-space trigger must NOT be due in phase 1" ); assert!( malloc_object_count() < GC_NEXT_MALLOC_TRIGGER.with(|trigger| trigger.get()), @@ -509,8 +509,8 @@ fn a_nursery_cap_only_trigger_is_deferred_to_the_collector_that_can_discharge_it // cycle must start here, or phase 1 proves nothing. trigger_guard.make_arena_trigger_due(); assert!( - crate::arena::arena_total_bytes() >= super::super::policy::next_arena_trigger_base(), - "control phase must present a due whole-arena trigger" + arena_trigger_total_bytes() >= super::super::policy::next_arena_trigger_base(), + "control phase must present a due old-space trigger" ); let started = gc_runtime_safepoint(); diff --git a/crates/perry-runtime/src/gc/tests/idle_reclaim.rs b/crates/perry-runtime/src/gc/tests/idle_reclaim.rs index 63f609223d..1b8cc9b42a 100644 --- a/crates/perry-runtime/src/gc/tests/idle_reclaim.rs +++ b/crates/perry-runtime/src/gc/tests/idle_reclaim.rs @@ -154,8 +154,12 @@ fn idle_reclaim_runs_a_full_at_the_park_when_owed() { /// the right-sizer must grant observation two without demanding new mutator /// activity. The episode then disarms, so the bypass cannot become a periodic /// full-GC loop. +/// +/// Sabotage: delete the `arena_right_size::owed()` start arm in `park_hook`. +/// The second idle-window start never reaches `note_started`, so the dedicated +/// `arena_right_size_starts` counter assertion below fails. #[test] -fn sustained_arena_slack_gets_one_bounded_followup_without_mutator_activity() { +fn idle_reclaimer_still_runs_one_bounded_right_size_reclaim_in_idle_window() { let _guard = CopyingNurseryTestGuard::new(1); let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); let _reducer = IdleReclaimTestGuard::new(0); diff --git a/crates/perry-runtime/src/gc/tests/mod.rs b/crates/perry-runtime/src/gc/tests/mod.rs index a698870282..002da17c20 100644 --- a/crates/perry-runtime/src/gc/tests/mod.rs +++ b/crates/perry-runtime/src/gc/tests/mod.rs @@ -1,5 +1,6 @@ mod alloc; mod arena_right_size; +mod arena_trigger_old_space; mod array_pointer_slot_enumeration; mod barrier; mod barrier_arming; diff --git a/crates/perry-runtime/src/gc/tests/support.rs b/crates/perry-runtime/src/gc/tests/support.rs index 130c0f7a08..4ecf6923c5 100644 --- a/crates/perry-runtime/src/gc/tests/support.rs +++ b/crates/perry-runtime/src/gc/tests/support.rs @@ -511,12 +511,18 @@ impl GcTriggerThresholdTestGuard { } pub(super) fn make_arena_trigger_due(&self) { + // ArenaBytes is old-space-only under the copying collector. Give a + // pristine test thread one real old block so `just_due` carries one + // byte of measurable debt instead of relying on nursery capacity. + if arena_trigger_total_bytes() == 0 { + let _ = crate::arena::arena_alloc_gc_old(8, 8, GC_TYPE_STRING); + } // Just-due, not zero: with debt-proportional assist pacing // (`gc_mutator_assist_scaled_work_units`), trigger=0 would read the // ENTIRE arena as debt and scale the first assist into a monolithic // collection — these tests want "trigger due, collector keeping up" // (debt ≈ 1 byte). Pacing-specific tests inflate debt explicitly. - let just_due = crate::arena::arena_total_bytes().saturating_sub(1); + let just_due = arena_trigger_total_bytes().saturating_sub(1); GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.set(just_due)); } }