gc(policy): the ArenaBytes major trigger and its consumers measure old space, not the copying nursery's reserved capacity

cc-perf-campaign/codex/REPORT_arena_trigger_fix.md

@@ -0,0 +1,86 @@
+# Arena trigger promotion assertion follow-up
+
+Implementation SHA: `0fa6101bbcca31605b0bdad0e2c3740f19149a74`
+
+Branch: `perf/arena-trigger-excludes-nursery` (push target: `fork` only)
+
+## Root cause
+
+This is case (b): the in-place promotion path deliberately reserves one fresh
+block after transferring the filled nursery blocks.
+
+- `crates/perry-runtime/src/arena/promote.rs:338-352` installs every captured
+  in-use nursery block into `OLD_ARENA`, adds its used bytes to old-gen
+  occupancy, and subtracts the same reserved capacity from
+  `NURSERY_RESERVED_BYTES`. It does not change `ARENA_TOTAL_BYTES`; that part
+  is a capacity transfer.
+- The isolated fixture starts with the reusable empty Eden head left by its
+  preparatory minor. `fill_dead_nursery_to(16 MiB + BLOCK_SIZE)` consumes that
+  head and allocates until all 18 reserved 1 MiB Eden blocks have nonzero
+  offsets. `retag_young_for_in_place_promotion` therefore captures all 18.
+- `crates/perry-runtime/src/arena/promote.rs:563-587` must leave Eden with a
+  usable allocator head. After all 18 filled blocks become tombstones, it calls
+  `install_fresh_block(BLOCK_SIZE)`.
+- `crates/perry-runtime/src/arena/block.rs:654-689` accounts that new nursery
+  block through `arena_reserved_bytes_add`; `block.rs:1044-1049` correctly adds
+  1 MiB to both `ARENA_TOTAL_BYTES` and `NURSERY_RESERVED_BYTES`.
+
+The reported numbers account exactly: 18 MiB before promotion becomes 19 MiB
+whole-arena capacity after promotion, comprising 18 MiB old space plus the
+mandatory empty 1 MiB Eden head. Because that head is present in both total and
+nursery counters, it cancels out of `old_space_total_bytes()`. No nursery
+counter creation/reset/release arm is missing.
+
+The failing assertion was therefore wrong at
+`crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs:152-157`: promotion
+transfers all existing capacity, but restoring the required Eden allocator head
+necessarily reserves one new block.
+
+## Base-difference audit
+
+`git log 504e180d0..616a2cb84 -- crates/perry-runtime/src/arena \
+crates/perry-runtime/src/gc/policy.rs crates/perry-runtime/src/gc/promote.rs \
+crates/perry-runtime/src/gc/reset.rs` returned no commits. The one-block result
+is not caused by drift between perrymaster's `504e180d0` base and this branch's
+`616a2cb84` base.
+
+## Change
+
+`crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs:133-167` now:
+
+- proves the fixture promotes every reserved nursery block, making the fresh
+  Eden head deterministic;
+- expects whole-arena reservation to increase by exactly `BLOCK_SIZE` and
+  identifies that block in the assertion message;
+- verifies the post-promotion nursery reservation is exactly the replacement
+  Eden head; and
+- verifies old-space pacing increases by exactly the promoted reservation, so
+  the replacement nursery head remains excluded.
+
+Runtime bookkeeping was not changed because its total, nursery, and old-space
+quantities are already internally consistent.
+
+## Validation
+
+Every Cargo invocation was preceded by `df -g /`; free space was 27-65 GB,
+above the binding 12 GB floor. Cargo ran through
+`cc-perf-campaign/measure_lock.sh --build` with `-j4`.
+
+- Exact test: passed (1 passed, 0 failed).
+- Runtime lib suite, non-PTY and single-threaded: passed (3261 passed, 0
+  failed, 4 ignored).
+- Archive feature-set build: `cargo build --release -j4 -p perry-runtime
+  --features wasm-host` passed.
+- Default compiler build: `cargo build --release -j4 -p perry` passed.
+- `rustfmt --edition 2021` on the changed Rust file: passed.
+- `git diff --check`: passed.
+- `bash scripts/check_file_size.sh`: passed.
+
+One earlier runtime-suite attempt was launched through a PTY and produced 3260
+passed / 1 failed / 4 ignored; the sole failure was
+`tty::tests::columns_undefined_when_not_tty`, because the runner had supplied a
+TTY. The required non-PTY rerun above was fully green.
+
+Not run: a separate `nm` audit or the external performance campaign. This
+follow-up changes only test assertions; the prescribed runtime, feature-set,
+and compiler gates all ran locally.

cc-perf-campaign/codex/REPORT_arena_trigger_nursery.md

@@ -0,0 +1,145 @@
+# ArenaBytes excludes the copying nursery
+
+Implementation SHA: `008fa075d542d9c09646a974593db03c6023ebc0`
+
+Branch: `perf/arena-trigger-excludes-nursery` (push target: `fork` only)
+
+## Mechanism map
+
+- `crates/perry-runtime/src/arena/block.rs:1008-1064`: `ARENA_TOTAL_BYTES`
+  is the cached reserved capacity of every arena region. The new adjacent
+  `NURSERY_RESERVED_BYTES` is the reserved-capacity share owned by Eden and
+  both survivor semispaces. Initial/fresh blocks update both counters according
+  to `HeapGeneration`; reset/release and quarantine detach subtract in the same
+  place as their cost.
+- `crates/perry-runtime/src/arena/walk.rs:321-338`: `arena_total_bytes()` is
+  still the whole-arena diagnostic/RSS quantity. `old_space_total_bytes()` is
+  `arena_total - copying_nursery_reserved`, i.e. long-lived plus old reserved
+  capacity. The `[gc] blocks` `general` count is Eden; `non_general` combines
+  both survivor semispaces with long-lived and old, so neither printed bucket
+  alone was an old-space quantity.
+- `crates/perry-runtime/src/arena/promote.rs:334-357`: in-place promotion moves
+  blocks from Eden/survivor ownership into `OLD_ARENA`. It now subtracts the
+  transferred capacity from the nursery counter without changing
+  `ARENA_TOTAL_BYTES`, making promotion visible as old-space growth.
+- `crates/perry-runtime/src/gc/policy.rs:83-124,3144-3165`:
+  `next_arena_trigger_base()` is the armed `GC_NEXT_TRIGGER_BYTES` value (or
+  the device-derived absolute ceiling before the first arm). The ArenaBytes
+  predicate now compares `old_space_total_bytes()` with that base. Nursery
+  occupancy remains owned by the independent `young_scavenge_cap_due()` arm
+  and is handed to a copying minor.
+- `crates/perry-runtime/src/gc/heap_budget.rs:119-132` and
+  `crates/perry-runtime/src/gc/policy.rs:2316-2508`: rebaselining uses the same
+  old-space capacity. Its default headroom floor remains exactly 16 MiB (scaled
+  only by the pre-existing constrained-device policy); adaptive step, ceiling,
+  and promotion-runway credit are unchanged. `next_base` is therefore based on
+  `old_total`, never nursery capacity. This is a unit change, not threshold
+  tuning.
+- `crates/perry-runtime/src/gc/policy.rs:1570-1637,1831-1839`:
+  `old_reclaimable = old_gen_in_use - old_free`. Its baseline is the last
+  reclaim/growth decision's old occupancy; proven-live promotion is credited
+  to it so the signal remains reclaimable growth rather than absolute old
+  occupancy. ArenaBytes no longer needs to duplicate that evidence gate.
+- `crates/perry-runtime/src/gc/arena_right_size.rs:142-176` and
+  `crates/perry-runtime/src/gc/idle_reclaim.rs:406-450`: idle right-sizing now
+  evaluates old-space live bytes against old-space capacity. The existing
+  sustained <=50% utilization rule, two-full bound, hysteresis, and idle-window
+  `arena_right_size` start remain unchanged.
+- `crates/perry-runtime/src/gc/oldgen.rs:1405-1427,1487-1650` and
+  `crates/perry-runtime/src/arena/walk.rs:118-195`: a budgeted full's arena
+  cursor includes Eden, both survivor regions, long-lived, and old blocks. The
+  sweep therefore visits nursery objects and accounts dead Eden bytes in
+  place; block cleanup calls `arena_reset_empty_blocks` and survivor reclaim.
+  `crates/perry-runtime/src/arena/reset.rs:173-251` shows what the subsequent
+  copying minor normally does instead: after rewriting survivors it resets
+  Eden plus the active survivor from-space and flips semispaces. Thus the
+  measured major was doing nursery-death work before the next minor could.
+
+## Change
+
+The selected structural fix is to exclude all reserved copying-nursery
+capacity from ArenaBytes arming and from every consumer of that trigger's
+units: rebaseline, debt scaling, tiny-parse pressure/bump, and the explicit
+memory-pressure clamp. The old-space ownership boundary is exact and already
+has a separate nursery-cap consumer; adding a second reclaim-evidence policy
+would overlap `old_reclaimable` and make arming depend on two baselines.
+
+`old_space_live_allocated_bytes()` subtracts the running from-space live census
+from the running whole-arena live census, so #9838's tiny-parse guard no longer
+treats a large nursery as pressure. The explicit safepoint deferral slack valve
+remains whole-arena by design because it bounds RSS growth while a collection
+waits; it does not arm ArenaBytes.
+
+With `PERRY_GC_DIAG=1`, `[gc-trigger]` now prints `arena_total`, the actual
+`old_total` compared, and `nursery_excluded`. `[gc-budgeted] start` prints an
+`arming_reason` plus the same totals. `[gc-arena-rebaseline]` prints the
+old-space total and excluded nursery capacity on one complete line.
+
+## Sabotage-able tests
+
+- `crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs:57-115` fills
+  dead nursery objects to the real cap, proves whole-arena bytes crossed the
+  base while old bytes did not, asserts the incremental-start counter does not
+  move, then proves the precise copying-minor counter advances and from-space
+  shrinks. Restoring `arena_total_bytes()` in the ArenaBytes comparison starts
+  budgeted work and fails the status/counter assertion.
+- `crates/perry-runtime/src/gc/tests/arena_trigger_old_space.rs:119-181`
+  transfers more than the unchanged headroom from nursery to old with the
+  nursery empty, then asserts exactly one ArenaBytes budgeted start. Removing
+  the nursery-counter transfer from `finish_in_place_promotion` keeps the old
+  quantity below the base and fails the start counter.
+- `crates/perry-runtime/src/gc/tests/idle_reclaim.rs:153-226` retains the
+  bounded idle-window right-size test and names its sabotage: deleting the
+  `arena_right_size::owed()` start arm prevents the dedicated
+  `arena_right_size_starts` counter from advancing.
+- Existing trigger, rebaseline, debt-pacer, copying-survival, host-safepoint,
+  arena-right-size, and idle-reclaim coverage was updated only where its unit
+  is now old-space capacity.
+
+## Validation
+
+No Cargo command was invoked. The required pre-Cargo `df -g /` check reported
+0 GB available (97% capacity), below the binding 12 GB floor, so the build and
+test gates were not run and no build lock was acquired or waited on.
+
+Non-Cargo checks run:
+
+- `rustfmt --edition 2021` on every modified Rust file: passed.
+- `git diff --check` (working tree and alternate-index commit): passed.
+- `bash scripts/check_file_size.sh`: passed (`OK: no Rust source files exceed
+  2000 lines`).
+
+Not run locally; request these perrymaster gates from the pushed SHA:
+
+1. `cargo test -p perry-runtime --release --lib -- --test-threads=1`
+2. `cargo build --release -p perry-runtime --features wasm-host`
+3. `cargo build --release -p perry`
+4. Coordinator label: `run-extended-tests` (GC-adjacent change).
+
+## Predictions and exact perrymaster request
+
+Prediction: budgeted cycles armed by ArenaBytes while `old_reclaimable` is at
+baseline fall to 0 per run; in-turn `[gc-charge] budgeted-done total_us` falls
+from 169-213 ms to approximately 0; minors per turn and `old_in_use` are
+unchanged; idle-window reclaim is unchanged. The expected paired bound is
+approximately -2% at 3300 characters from removing about 50 ms/turn, larger at
+400 characters, with peak and settled RSS unchanged. RSS +1-10% remains
+acceptable. Because the change removes majors that a larger nursery was
+arming, the NS2 n64 arm needs a runtime-only rerun to determine whether its
+previous +13% peak moves.
+
+Exact request:
+
+> From the pushed SHA, run the three release gates listed above and apply the
+> `run-extended-tests` label. Relink that runtime on the I7-view tree
+> (runtime-only). Run one 4-turn graceful 3300-character reply and one
+> 400-character reply with `PERRY_GC_DIAG=1`; use `ns2_majors_an.py` and
+> `ns2_alloc_an.py`. Verify ArenaBytes budgeted cycles with
+> `old_reclaimable` at baseline are 0/run, in-turn `[gc-charge]
+> budgeted-done total_us` moves from 169-213 ms to approximately 0, minors per
+> turn are unchanged, `old_in_use` is unchanged, and idle-window reclaim is
+> unchanged. Then run paired 5x3300 + 3x400 versus I7-view; expect about -2%
+> at 3300 from the roughly 50 ms/turn removal, a larger win at 400, and
+> unchanged peak/settled RSS. Finally rerun the NS2 n64 arm on this runtime to
+> see whether its +13% peak moves now that the larger nursery no longer arms
+> those majors.

crates/perry-runtime/src/arena/block.rs

@@ -556,7 +556,7 @@ impl Arena {
             space,
             initial.object_starts_ptr(),
         );
-        ARENA_TOTAL_BYTES.with(|t| t.set(t.get() + initial.size));
+        arena_reserved_bytes_add(generation, initial.size);
         Arena {
             blocks: vec![initial],
             current: 0,
@@ -686,7 +686,7 @@ impl Arena {
             }
         };
         self.current = new_idx;
-        ARENA_TOTAL_BYTES.with(|t| t.set(t.get() + fresh_size));
+        arena_reserved_bytes_add(self.generation, fresh_size);
     }
 
     fn alloc_fresh_block(&mut self, size: usize, align: usize) -> *mut u8 {
@@ -1004,6 +1004,13 @@ thread_local! {
     /// alloc into a tombstone slot or the end, and release inside
     /// `arena_reset_empty_blocks`).
     pub(crate) static ARENA_TOTAL_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
+
+    /// Cached nursery share of [`ARENA_TOTAL_BYTES`]: Eden plus BOTH survivor
+    /// semispaces. The ArenaBytes policy subtracts this value so the copying
+    /// nursery cannot arm old-space work merely by filling its from-space.
+    /// Maintained beside the whole-arena counter so the allocation trigger
+    /// remains O(1).
+    pub(crate) static NURSERY_RESERVED_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
 }
 
 crate::perry_thread_local! {
@@ -1034,6 +1041,29 @@ crate::perry_thread_local! {
 
 }
 
+#[inline]
+pub(crate) fn arena_reserved_bytes_add(generation: HeapGeneration, bytes: usize) {
+    ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_add(bytes)));
+    if generation == HeapGeneration::Nursery {
+        NURSERY_RESERVED_BYTES.with(|nursery| nursery.set(nursery.get().saturating_add(bytes)));
+    }
+}
+
+#[inline]
+pub(crate) fn arena_reserved_bytes_sub(generation: HeapGeneration, bytes: usize) {
+    ARENA_TOTAL_BYTES.with(|total| total.set(total.get().saturating_sub(bytes)));
+    if generation == HeapGeneration::Nursery {
+        nursery_reserved_bytes_sub(bytes);
+    }
+}
+
+/// Transfer already-accounted capacity out of the nursery without changing
+/// the whole-arena total (whole-block promotion).
+#[inline]
+pub(crate) fn nursery_reserved_bytes_sub(bytes: usize) {
+    NURSERY_RESERVED_BYTES.with(|nursery| nursery.set(nursery.get().saturating_sub(bytes)));
+}
+
 // `ARENA` and `INLINE_STATE` below stay raw: they are NAMED `HotTls` fields
 // (`arena`, `inline_state`) whose addresses `tls_hot::fill` reads through the
 // providers further down, and a named field is one dependent load cheaper

crates/perry-runtime/src/arena/mod.rs

@@ -37,12 +37,12 @@ pub(crate) use allocators::{
 #[cfg(test)]
 pub(crate) use block::old_gen_in_use_bytes_slot_index;
 pub(crate) use block::{
-    arena_cell_alloc, arena_cell_try_alloc_current, drain_block_pool_if_requested,
-    new_object_start_bitmap, old_gen_in_use_bytes_sub, release_arena_block,
-    request_block_pool_drain, Arena, ArenaBlock, ArenaBlockRelease, BlockPoolDrainStats,
-    ACTIVE_SURVIVOR, ARENA, ARENA_TOTAL_BYTES, BLOCK_SIZE, FRESH_GENERAL_BLOCK_MIN_USED_BYTES,
-    INLINE_STATE, LONGLIVED_ARENA, OBJECT_START_SHIFT, OLD_ARENA, OLD_GEN_IN_USE_BYTES,
-    SURVIVOR_ARENA_0, SURVIVOR_ARENA_1,
+    arena_cell_alloc, arena_cell_try_alloc_current, arena_reserved_bytes_sub,
+    drain_block_pool_if_requested, new_object_start_bitmap, nursery_reserved_bytes_sub,
+    old_gen_in_use_bytes_sub, release_arena_block, request_block_pool_drain, Arena, ArenaBlock,
+    ArenaBlockRelease, BlockPoolDrainStats, ACTIVE_SURVIVOR, ARENA, ARENA_TOTAL_BYTES, BLOCK_SIZE,
+    FRESH_GENERAL_BLOCK_MIN_USED_BYTES, INLINE_STATE, LONGLIVED_ARENA, NURSERY_RESERVED_BYTES,
+    OBJECT_START_SHIFT, OLD_ARENA, OLD_GEN_IN_USE_BYTES, SURVIVOR_ARENA_0, SURVIVOR_ARENA_1,
 };
 /// #7469 hot-TLS plumbing — see `crate::tls_hot`. The `*_hot_addr` half is
 /// consumed by `tls_hot::fill`; the `hot_*` half is the cached accessor the
@@ -92,8 +92,8 @@ pub(crate) use walk::ArenaRegionTelemetry;
 pub use walk::{
     arena_block_count, arena_in_use_bytes, arena_total_bytes, arena_walk_objects,
     arena_walk_objects_addr_sorted, arena_walk_objects_filtered,
-    arena_walk_objects_with_block_index, general_block_count, longlived_end,
-    old_arena_walk_objects, ArenaResetStats,
+    arena_walk_objects_with_block_index, copying_nursery_reserved_bytes, general_block_count,
+    longlived_end, old_arena_walk_objects, old_space_total_bytes, ArenaResetStats,
 };
 pub(crate) use walk::{
     arena_block_snapshots, arena_telemetry_snapshot, general_block_in_recent_window,
@@ -136,7 +136,9 @@ pub use stats::{
     arena_live_allocated_bytes, js_arena_stats, longlived_in_use_bytes, old_gen_in_use_bytes,
     pointer_in_nursery, pointer_in_old_gen,
 };
-pub(crate) use stats::{arena_live_from_space_bytes, record_arena_live_census};
+pub(crate) use stats::{
+    arena_live_from_space_bytes, old_space_live_allocated_bytes, record_arena_live_census,
+};
 #[cfg(test)]
 pub(crate) use stats::{old_gen_in_use_bytes_recomputed, old_gen_in_use_bytes_resync};
 

crates/perry-runtime/src/arena/promote.rs

@@ -91,16 +91,14 @@ pub(crate) struct InPlacePromotion {
     blocks: Vec<PromotedBlock>,
     /// Reserved bytes the young generation is about to lose to old-gen.
     ///
-    /// This is NOT bookkeeping trivia — it is the whole of the pacing contract.
-    /// The arena-bytes trigger fires on `arena_total_bytes()`, and the runway
-    /// between two collections is therefore "Eden's free capacity + the trigger
-    /// step". A copying minor recycles Eden's blocks, so that capacity survives
-    /// the collection and the runway stays wide. Promotion HANDS THOSE BLOCKS
-    /// AWAY, so without giving the capacity back the runway collapses to the
-    /// bare step. Measured on `retain.ts`: Eden's high-water fell 57 MB → 18 MB,
-    /// collections went 6 → 12, and the extra pressure bought a second full
-    /// collection at 690 ms — a 0.81 s → 1.52 s regression from a change that
-    /// made every individual promotion cheaper.
+    /// This is NOT bookkeeping trivia — it is part of the pacing contract.
+    /// Whole-block promotion removes reusable Eden capacity. Although that
+    /// capacity now becomes visible in the old-space ArenaBytes quantity, the
+    /// young allocation runway still needs to be restored separately or it
+    /// collapses to the bare trigger step. Measured on `retain.ts`: Eden's
+    /// high-water fell 57 MB → 18 MB, collections went 6 → 12, and the extra
+    /// pressure bought a second full collection at 690 ms — a 0.81 s → 1.52 s
+    /// regression from a change that made every individual promotion cheaper.
     ///
     /// The capacity is given back as trigger HEADROOM
     /// (`gc::note_promoted_young_capacity`), not as eagerly mapped blocks.
@@ -348,6 +346,10 @@ pub(crate) fn finish_in_place_promotion(
             retag_block_space(base, size, HeapGeneration::Old, HeapSpace::Old);
         }
     });
+    // Ownership moved from Eden/survivor to old-gen, but the whole-arena
+    // reservation did not change. Move only the cached nursery share so the
+    // ArenaBytes old-space reading sees promotion as old-space growth.
+    nursery_reserved_bytes_sub(promotion.reserved_bytes);
 
     // Both young regions are empty now. Eden needs a usable current block for
     // the inline bump allocator; the survivor flip keeps the semispace