perf(gc): pre-size the per-minor dirty-scan covered set instead of rebuilding it from empty

changelog.d/9827-gc-trigger-path-hot-tls.md

@@ -0,0 +1,55 @@
+**The GC trigger path and the dirty-page barrier stop paying `_tlv_get_addr`
+per read, and the policy gate that let them stop paying it now counts the
+thing it is bounding.**
+
+`gc_check_trigger` runs on every `gc_malloc`, and its predicate
+(`gc_budgeted_due_trigger`) resolved eleven raw `thread_local!` declarations
+one out-of-line call at a time. Measured with `sample` on the compiled
+claude-code TUI streaming a 3300-char reply (14,578 active main-thread
+samples, callers resolved by an explicit ancestor walk rather than
+nearest-symbol labels): `_tlv_get_addr` was 380 main-thread leaf samples,
+**71 of them with `gc_budgeted_due_trigger` as the immediate caller**, 36 in
+`old_page_account_dirty_slots`, 31 in `scan_dirty_object_slots`, 27 in
+`gc_malloc_header_is_tracked`. `crates/perry-runtime/src/tls_hot.rs` has
+existed to abolish exactly this since #7469; the allocation path's *fields*
+were covered and the trigger path never was.
+
+Sixty-seven declarations across `gc/policy.rs`, `gc/malloc.rs`, `gc/old_free.rs`,
+`gc/tenuring.rs`, `gc/trace.rs`, `gc/barrier/mod.rs`, `arena/block.rs` and
+`arena/page_meta.rs` move to `crate::perry_thread_local!` — same syntax, same
+`.with()` at every call site, the address served from this thread's hot cache
+instead of a libdyld call.
+
+**Why they were still cold is a measurement bug in the gate, not an oversight
+anyone could have noticed.** `scripts/check_thread_locals.py` ratchets on the
+number of raw `thread_local!` **blocks** per file, while `thread_local! { … }`
+holds any number of declarations — so `gc/policy.rs` counted as **6** while
+declaring **28**, and adding a `static` to an already-recorded block passed
+the gate silently. Counted in the same unit as the hot side, `main` was **318
+hot declarations against 339 cold ones** — cold was the majority, reported as
+a 2.6:1 minority. The gate now ratchets on declarations (`385 hot / 272
+cold`), and `--self-test` grew a seventh direction that fails when a `static`
+is added to a recorded block; restoring the block count makes that case, and
+only that case, fail.
+
+Three declarations stay deliberately raw and say so at their declaration:
+`ARENA_TOTAL_BYTES`, `BLOCK_POOL` and `BLOCK_POOL_BYTES` are read from
+`Arena::new`, which runs as `tls_hot::fill`'s **first** provider, so a
+`HotKey` there re-enters `fill` — which by design has not yet written the
+`temp_roots` field it gates on — and re-runs `ARENA`'s initializer without
+bound. It is a stack overflow at thread start, not a slow path, and it is the
+first documented instance of the rule that a declaration read from inside a
+`fill` provider cannot use the macro. `gc::tests::tls_fill_reentrancy` is the
+standing guard, and it is sabotage-proved: moving `ARENA_TOTAL_BYTES` alone
+into the neighbouring hot block aborts that test with `fatal runtime error:
+stack overflow`.
+
+`gc::tests::trigger_path_tls` is the runtime half of the gate: it drives
+`gc_check_trigger` on a fresh thread and asserts every trigger-path
+declaration owns a hot slot and that the path publishes slots at all.
+Reverting any one of them to a raw `thread_local!` removes `slot_index` and
+breaks the build at that declaration's own name. It is a test that can fail
+and did: the first run rejected `GC_DEFERRED_REQUEST` with `index 4294967295`,
+correctly — `defer_gc_request` reads it only while a root lock is held, so it
+is not a fast-path read and never claims a slot. The list is what the fast
+path reads, not what the module declares.

changelog.d/9835-dirty-scan-presize.md

@@ -0,0 +1,36 @@
+**The per-minor dirty-scan covered set is pre-sized instead of being rebuilt
+from empty**, removing the hashbrown growth ladder every copying minor walked.
+
+`dirty_scan_covered` is created with `new_ptr_hash_set()` at the top of every
+copying minor and filled during the dirty-slot scan. Measured with
+`[gc-dirty-covered]` (added here), it reaches **~119,000 entries** on a
+3300-character claude-code reply — not the ~1,000 the `[gc-restore-coverage]`
+`objects_skipped` figure suggested — so it walked hashbrown's capacity ladder
+(1,792 → 14,336 → 57,344 → 114,688 → 229,376) and paid a
+`RawTable::reserve_rehash` at each boundary, re-hashing and re-copying the whole
+table. `reserve_rehash` was **217 leaf samples, 1.49 % of the turn**, 111 of
+them under `PtrHashSet::insert` and the rest under `run_copied_minor_attempt`
+and `restore_surviving_dirty_coverage`.
+
+The set is now pre-sized from the previous minor's count, the same treatment and
+the same justification as `PREVIOUS_SURVIVOR_ESTIMATE` immediately above it: the
+count is autocorrelated between adjacent cycles, over-estimating costs only
+untouched reserved bytes, under-estimating falls back to ordinary growth, and
+the estimate shares that constant's cap so one huge cycle cannot make every
+later cycle reserve unboundedly.
+
+`reserve_rehash` falls **217 → 167 leaf samples (1.49 % → 1.24 % of the turn)**.
+The rig is flat, as expected of a 1.5 % item — 400-character turn CPU 4.05 min
+against 4.14, 3300 17.86 against 17.71 — with settled footprint and peak RSS
+improving at 400 (557 → 457 MB, 604 → 563 MB) and flat at 3300. The ground
+claimed is **work permanently removed**, counted rather than inferred:
+`[gc-dirty-covered]` reports `len`, `capacity` and `presized_to` per minor, so
+the pre-size can be seen tracking rather than assumed to.
+
+**A high-water estimate was tried and rejected.** It is better on the mechanism
+— under-shoots fall from 57 of 96 minors to 21 of 97 — but reserving the peak on
+every minor cost settled footprint 763 → 1165 MB and peak RSS 974 → 1250 MB at
+3300 characters for no measurable time difference (167 vs 182 leaf samples,
+inside run-to-run noise). Trading footprint for CPU is rejected, and here it did
+not even buy CPU. The rejection is recorded at the function so the next person
+does not re-derive it.

crates/perry-runtime/src/arena/block.rs

@@ -122,6 +122,29 @@ impl Drop for BlockPool {
     }
 }
 
+// ---------------------------------------------------------------------------
+// These three stay RAW, and not because nobody got to them: they are read from
+// inside `Arena::new`, which runs as `ARENA`'s lazy initializer — and `ARENA`
+// is the FIRST provider `tls_hot::fill` resolves. Routing them through the hot
+// cache closes a cycle that ends in a stack overflow at thread start:
+//
+//     HotKey::get -> hot() -> hot_uncached -> hot_via_tls
+//       -> (temp_roots still null) fill()
+//         -> arena_hot_addr() -> ARENA.with(..) -> Arena::new
+//           -> ARENA_TOTAL_BYTES / BLOCK_POOL{,_BYTES}.with(..)
+//             -> HotKey::get -> hot() -> ... (temp_roots STILL null) ...
+//
+// `fill` writes `temp_roots` last precisely so a re-entrant reader cannot see a
+// half-filled cache as ready — which makes the nesting re-run `fill`, and
+// re-run `ARENA`'s initializer, without bound. It bites on any thread where
+// the first `hot()` precedes the first arena touch, i.e. every freshly spawned
+// one, and it fails as a crash rather than as a slow path.
+//
+// The rule this is an instance of: a declaration read from the dynamic extent
+// of a `tls_hot::fill` provider cannot use `crate::perry_thread_local!`. Today
+// `ARENA` is the only provider whose initializer runs code, so this is the
+// whole set.
+// ---------------------------------------------------------------------------
 thread_local! {
     static BLOCK_POOL: RefCell<BlockPool> = const { RefCell::new(BlockPool {
         blocks: Vec::new(),
@@ -981,7 +1004,9 @@ thread_local! {
     /// alloc into a tombstone slot or the end, and release inside
     /// `arena_reset_empty_blocks`).
     pub(crate) static ARENA_TOTAL_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
+}
 
+crate::perry_thread_local! {
     /// Cached running sum of `block.offset` across the old-gen arena —
     /// the delta-maintained twin of `ARENA_TOTAL_BYTES` above, same
     /// rationale: `gc_budgeted_due_trigger()` reads the old-gen in-use
@@ -1007,8 +1032,18 @@ thread_local! {
     /// the OldReclaim trigger.
     pub(crate) static OLD_GEN_IN_USE_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
 
+}
+
+// `ARENA` and `INLINE_STATE` below stay raw: they are NAMED `HotTls` fields
+// (`arena`, `inline_state`) whose addresses `tls_hot::fill` reads through the
+// providers further down, and a named field is one dependent load cheaper
+// than a claimed slot. Their neighbours in this block were never migrated.
+thread_local! {
     pub(crate) static ARENA: UnsafeCell<Arena> =
         UnsafeCell::new(Arena::new(HeapGeneration::Nursery, HeapSpace::NurseryEden));
+}
+
+crate::perry_thread_local! {
 
     /// Segregated long-lived arena (issue #179). Holds objects that are
     /// intentionally pinned for the lifetime of the program by explicit
@@ -1056,6 +1091,9 @@ thread_local! {
     pub(crate) static OLD_ARENA: UnsafeCell<Arena> =
         UnsafeCell::new(Arena::new_lazy(HeapGeneration::Old, HeapSpace::Old));
 
+}
+
+thread_local! {
     /// Inline allocator state — a cache of the current arena block's
     /// `(data, offset, size)` tuple, exposed via a stable pointer so
     /// codegen can emit inline bump-allocate IR without going through
@@ -1072,6 +1110,12 @@ thread_local! {
     }) };
 }
 
+/// Hot-cache slot claimed by `OLD_GEN_IN_USE_BYTES`. See above.
+#[cfg(test)]
+pub(crate) fn old_gen_in_use_bytes_slot_index() -> u32 {
+    OLD_GEN_IN_USE_BYTES.slot_index()
+}
+
 // --- #7469 hot-TLS address providers. See `crate::tls_hot`. ---
 
 /// Address of this thread's `ARENA`. Resolving it once and caching it is what

crates/perry-runtime/src/arena/mod.rs

@@ -46,6 +46,8 @@ pub(crate) use block::{
 /// allocation path uses instead of a per-access `_tlv_get_addr`.
 pub(crate) use block::{arena_hot_addr, hot_arena, hot_inline_state, inline_state_hot_addr};
 #[cfg(test)]
+pub(crate) use block::old_gen_in_use_bytes_slot_index;
+#[cfg(test)]
 pub(crate) use block::{
     block_pool_bytes_for_test, block_pool_explicit_drained_bytes_for_test, block_pool_put,
     force_next_block_alloc_failure, gc_trigger_arena_borrow_depth, gc_trigger_arena_calls,

crates/perry-runtime/src/arena/page_meta.rs

@@ -319,13 +319,20 @@ pub(crate) struct OldArenaSourceBlockSelection {
     pub(crate) pages: crate::fast_hash::PtrHashSet<usize>,
 }
 
+// `PAGE_GENERATIONS` and `PAGE_GENERATION_CACHE` stay raw because they are
+// NAMED fields of `HotTls` (`page_generations`, `page_generation_cache`), whose
+// providers below hand `tls_hot::fill` their addresses. A named field is one
+// dependent load cheaper than a claimed slot, which is why the closed set
+// exists; everything else in this file was simply never migrated.
 thread_local! {
     static PAGE_GENERATIONS: RefCell<PageGenerationMap> =
         RefCell::new(crate::fast_hash::new_ptr_hash_map());
 
     static PAGE_GENERATION_CACHE: UnsafeCell<PageGenerationCacheSet> =
         const { UnsafeCell::new(PageGenerationCacheSet::empty()) };
+}
 
+crate::perry_thread_local! {
     static OLD_GEN_PAGE_OBJECTS: RefCell<OldGenPageObjectMap> =
         RefCell::new(crate::fast_hash::new_ptr_hash_map());
 
@@ -1138,7 +1145,7 @@ pub(crate) fn register_old_object_pages(header_addr: usize, total_size: usize) {
 // which fails if a new toucher of either table appears without one.
 // ---------------------------------------------------------------------------
 
-thread_local! {
+crate::perry_thread_local! {
     /// Old-object page registrations not yet folded into `OLD_GEN_PAGE_OBJECTS`.
     /// Entries are `(header_addr, total_size)`; nothing here is dereferenced, so
     /// a deferred entry never keeps an object alive and is not a GC root — and

crates/perry-runtime/src/fast_hash.rs

@@ -107,6 +107,14 @@ pub fn new_ptr_hash_set<T: std::hash::Hash + Eq>() -> PtrHashSet<T> {
     HashSet::with_hasher(PtrHasher)
 }
 
+/// The same, pre-sized. A set that is rebuilt from empty on every collection
+/// pays a `RawTable::reserve_rehash` at each power-of-two boundary on the way
+/// up, and the whole table is re-hashed and re-copied each time.
+#[inline]
+pub fn new_ptr_hash_set_with_capacity<T: std::hash::Hash + Eq>(cap: usize) -> PtrHashSet<T> {
+    HashSet::with_capacity_and_hasher(cap, PtrHasher)
+}
+
 #[inline]
 pub fn new_ptr_hash_map<K: std::hash::Hash + Eq, V>() -> PtrHashMap<K, V> {
     HashMap::with_hasher(PtrHasher)

crates/perry-runtime/src/gc/barrier/mod.rs

@@ -697,6 +697,10 @@ pub(super) unsafe fn scan_dirty_object_slots(
 // retained only as a test fallback for the previous object-level
 // HashSet behavior.
 
+// The three declarations below stay raw: they are NAMED `HotTls` fields
+// (`incremental_mark_valid_ptrs`, `birth_extra_flags`,
+// `incremental_mark_minor_only`), one dependent load cheaper than a claimed
+// slot. Everything after them was simply never migrated.
 thread_local! {
     /// Active incremental mark barrier state (Full AND budgeted Minor
     /// cycles — a Minor cycle sliced across mutator turns has exactly the
@@ -737,7 +741,9 @@ thread_local! {
     /// later. Old children need no shading in a minor anyway: minors never
     /// collect live old-gen objects.
     pub(super) static INCREMENTAL_MARK_BARRIER_MINOR_ONLY: Cell<bool> = const { Cell::new(false) };
+}
 
+crate::perry_thread_local! {
     /// Dirty old-generation pages that have received a YOUNG-gen
     /// pointer since the last collection. This is Perry's compact
     /// modbuf: barriers log bounded page regions, and minor GC scans
@@ -1761,7 +1767,7 @@ fn mark_dirty_old_page_uncached(page: usize) -> bool {
     inserted
 }
 
-thread_local! {
+crate::perry_thread_local! {
     /// PERRY_GC_VERIFY_EVACUATION diagnostic only: every old page EVER marked
     /// dirty over the process lifetime (never cleared). Lets the verifier's
     /// missing-edge report say whether the slot's page was recorded at some

crates/perry-runtime/src/gc/copying.rs

@@ -215,6 +215,37 @@ static PREVIOUS_SURVIVOR_ESTIMATE: std::sync::atomic::AtomicUsize =
 /// reserve 100 MB of pointers.
 const SURVIVOR_ESTIMATE_CAP: usize = 1 << 21;
 
+/// Previous minor's dirty-scan covered-set size, for pre-sizing the next one.
+/// Capped for the same reason as the survivor estimate: a one-off huge cycle
+/// must not make every later cycle reserve unboundedly.
+static PREVIOUS_DIRTY_COVERED_ESTIMATE: std::sync::atomic::AtomicUsize =
+    std::sync::atomic::AtomicUsize::new(0);
+
+pub(super) fn previous_dirty_covered_estimate() -> usize {
+    PREVIOUS_DIRTY_COVERED_ESTIMATE.load(std::sync::atomic::Ordering::Relaxed)
+}
+
+/// LAST-VALUE. **Do not "just reserve the peak" — that was tried and it cost
+/// 400 MB of settled footprint for no time gain.**
+///
+/// LAST-VALUE, and a high-water mark was tried and REJECTED.
+///
+/// `[gc-dirty-covered]` shows this set is far more volatile than the survivor
+/// count this pattern was copied from: it ramps 1,028 -> ~119,000 over a turn
+/// and swings between adjacent minors, so a last-value estimate under-shoots on
+/// 57 of 96 minors. A high-water mark fixes that on the mechanism — under-shoots
+/// fall to 21 of 97 — and was still rejected: reserving the peak on EVERY minor
+/// cost settled footprint 763 -> 1165 MB and peak RSS 974 -> 1250 MB at 3300
+/// characters, for no measurable time difference (`reserve_rehash` 167 vs 182
+/// leaf samples, inside run-to-run noise). Trading footprint for CPU is
+/// rejected, and here it did not even buy CPU.
+pub(super) fn note_dirty_covered_for_presizing(count: usize) {
+    PREVIOUS_DIRTY_COVERED_ESTIMATE.store(
+        count.min(SURVIVOR_ESTIMATE_CAP),
+        std::sync::atomic::Ordering::Relaxed,
+    );
+}
+
 pub(super) fn note_survivor_count_for_presizing(count: usize) {
     PREVIOUS_SURVIVOR_ESTIMATE.store(
         count.min(SURVIVOR_ESTIMATE_CAP),
@@ -1356,7 +1387,20 @@ pub(super) fn run_copied_minor_attempt(
     let snapshot = remembered_dirty_snapshot();
     // #9754: objects whose every slot the dirty scan visited in-body — the
     // post-cycle coverage restore skips them (see `scan_dirty_object_slots`).
-    let mut dirty_scan_covered = crate::fast_hash::new_ptr_hash_set();
+    // #9835: this set is rebuilt from EMPTY on every minor and reaches ~1,000
+    // entries (`[gc-restore-coverage] objects_skipped=1026..1116`), so it walked
+    // hashbrown's growth ladder and paid a `RawTable::reserve_rehash` at each
+    // power-of-two boundary — measured 217 leaf samples in `reserve_rehash` on a
+    // 3300-char claude-code reply (1.5 % of the turn), 111 of them under
+    // `PtrHashSet::insert` and the rest under this function and
+    // `restore_surviving_dirty_coverage`.
+    //
+    // Same treatment, and the same justification, as `PREVIOUS_SURVIVOR_ESTIMATE`
+    // above: the count is strongly autocorrelated between adjacent cycles (it is
+    // the same program in the same phase), over-estimating costs only untouched
+    // reserved bytes, and under-estimating falls back to ordinary growth.
+    let mut dirty_scan_covered =
+        crate::fast_hash::new_ptr_hash_set_with_capacity(previous_dirty_covered_estimate());
     if !untraced {
         let _phase = super::pin::CopyingWalkPhaseGuard::enter("remembered_set");
         let remembered_stats = scan_remembered_dirty_slots_copying(
@@ -1617,6 +1661,21 @@ pub(super) fn run_copied_minor_attempt(
     if !collector.skip_remembering {
         restore_surviving_dirty_coverage(&snapshot, &dirty_scan_covered, "copying_minor");
     }
+    // The mechanism, counted rather than assumed: with the pre-size working,
+    // `capacity` is already >= `len` on entry and hashbrown never grows the
+    // table, so `reserve_rehash` disappears from this path. A capacity that
+    // keeps climbing across minors would say the estimate is not tracking.
+    if crate::gc::gc_diag_enabled() {
+        eprintln!(
+            "[gc-dirty-covered] len={} capacity={} presized_to={}",
+            dirty_scan_covered.len(),
+            dirty_scan_covered.capacity(),
+            previous_dirty_covered_estimate(),
+        );
+    }
+    note_dirty_covered_for_presizing(dirty_scan_covered.len());
+    {
+    }
     let malloc_freed_bytes = if malloc_sweep_due {
         let phase_start = trace_phase_start(trace);
         let freed = sweep_malloc_objects();

crates/perry-runtime/src/gc/malloc.rs

@@ -196,7 +196,7 @@ pub(super) const MALLOC_STATE_HEAVY_LEN_THRESHOLD: usize = 64 * 1024;
 /// 7/8 load factor without further rehashes).
 pub(super) const MALLOC_STATE_HEAVY_CAPACITY: usize = 256 * 1024;
 
-thread_local! {
+crate::perry_thread_local! {
     pub(crate) static MALLOC_STATE: RefCell<MallocState> = RefCell::new(MallocState {
         objects: Vec::with_capacity(MALLOC_STATE_INITIAL_CAPACITY),
         set: crate::fast_hash::PtrHashSet::with_capacity_and_hasher(
@@ -209,12 +209,25 @@ thread_local! {
         heavy_capacity_reserved: false,
         kind_telemetry: [MallocKindTelemetry::zero(); MALLOC_KIND_BUCKET_COUNT],
     });
+}
 
+// `ARENA_FREE_LIST` / `ARENA_FREE_LIST_NONEMPTY` stay raw: they are NAMED
+// `HotTls` fields (`arena_free_list`, `arena_free_list_nonempty`), one
+// dependent load cheaper than a claimed slot.
+thread_local! {
     pub(crate) static ARENA_FREE_LIST: RefCell<Vec<(*mut u8, usize)>> = const { RefCell::new(Vec::new()) };
     pub(crate) static ARENA_FREE_LIST_NONEMPTY: std::cell::Cell<bool> =
         const { std::cell::Cell::new(false) };
 }
 
+/// Hot-cache slot claimed by `MALLOC_STATE`, whose length is the
+/// `MallocCount` trigger's basis on every `gc_malloc`. Liveness
+/// instrumentation for `gc::tests::trigger_path_tls`.
+#[cfg(test)]
+pub(crate) fn malloc_state_slot_index() -> u32 {
+    MALLOC_STATE.slot_index()
+}
+
 pub fn gc_malloc(size: usize, obj_type: u8) -> *mut u8 {
     let total = GC_HEADER_SIZE + size;
     let layout = Layout::from_size_align(total, 8).unwrap();
@@ -480,7 +493,7 @@ pub(super) fn malloc_sweep_revalidate_header(
     })
 }
 
-thread_local! {
+crate::perry_thread_local! {
     pub(super) static MALLOC_REGISTRY_REBUILD_COUNT: Cell<u64> = const { Cell::new(0) };
 }