perf(gc): route the trigger path through hot TLS — and fix the gate that counts blocks where the cost is per declaration

changelog.d/9827-gc-trigger-path-hot-tls.md

@@ -0,0 +1,55 @@
+**The GC trigger path and the dirty-page barrier stop paying `_tlv_get_addr`
+per read, and the policy gate that let them stop paying it now counts the
+thing it is bounding.**
+
+`gc_check_trigger` runs on every `gc_malloc`, and its predicate
+(`gc_budgeted_due_trigger`) resolved eleven raw `thread_local!` declarations
+one out-of-line call at a time. Measured with `sample` on the compiled
+claude-code TUI streaming a 3300-char reply (14,578 active main-thread
+samples, callers resolved by an explicit ancestor walk rather than
+nearest-symbol labels): `_tlv_get_addr` was 380 main-thread leaf samples,
+**71 of them with `gc_budgeted_due_trigger` as the immediate caller**, 36 in
+`old_page_account_dirty_slots`, 31 in `scan_dirty_object_slots`, 27 in
+`gc_malloc_header_is_tracked`. `crates/perry-runtime/src/tls_hot.rs` has
+existed to abolish exactly this since #7469; the allocation path's *fields*
+were covered and the trigger path never was.
+
+Sixty-seven declarations across `gc/policy.rs`, `gc/malloc.rs`, `gc/old_free.rs`,
+`gc/tenuring.rs`, `gc/trace.rs`, `gc/barrier/mod.rs`, `arena/block.rs` and
+`arena/page_meta.rs` move to `crate::perry_thread_local!` — same syntax, same
+`.with()` at every call site, the address served from this thread's hot cache
+instead of a libdyld call.
+
+**Why they were still cold is a measurement bug in the gate, not an oversight
+anyone could have noticed.** `scripts/check_thread_locals.py` ratchets on the
+number of raw `thread_local!` **blocks** per file, while `thread_local! { … }`
+holds any number of declarations — so `gc/policy.rs` counted as **6** while
+declaring **28**, and adding a `static` to an already-recorded block passed
+the gate silently. Counted in the same unit as the hot side, `main` was **318
+hot declarations against 339 cold ones** — cold was the majority, reported as
+a 2.6:1 minority. The gate now ratchets on declarations (`385 hot / 272
+cold`), and `--self-test` grew a seventh direction that fails when a `static`
+is added to a recorded block; restoring the block count makes that case, and
+only that case, fail.
+
+Three declarations stay deliberately raw and say so at their declaration:
+`ARENA_TOTAL_BYTES`, `BLOCK_POOL` and `BLOCK_POOL_BYTES` are read from
+`Arena::new`, which runs as `tls_hot::fill`'s **first** provider, so a
+`HotKey` there re-enters `fill` — which by design has not yet written the
+`temp_roots` field it gates on — and re-runs `ARENA`'s initializer without
+bound. It is a stack overflow at thread start, not a slow path, and it is the
+first documented instance of the rule that a declaration read from inside a
+`fill` provider cannot use the macro. `gc::tests::tls_fill_reentrancy` is the
+standing guard, and it is sabotage-proved: moving `ARENA_TOTAL_BYTES` alone
+into the neighbouring hot block aborts that test with `fatal runtime error:
+stack overflow`.
+
+`gc::tests::trigger_path_tls` is the runtime half of the gate: it drives
+`gc_check_trigger` on a fresh thread and asserts every trigger-path
+declaration owns a hot slot and that the path publishes slots at all.
+Reverting any one of them to a raw `thread_local!` removes `slot_index` and
+breaks the build at that declaration's own name. It is a test that can fail
+and did: the first run rejected `GC_DEFERRED_REQUEST` with `index 4294967295`,
+correctly — `defer_gc_request` reads it only while a root lock is held, so it
+is not a fast-path read and never claims a slot. The list is what the fast
+path reads, not what the module declares.

crates/perry-runtime/src/arena/block.rs

@@ -122,6 +122,29 @@ impl Drop for BlockPool {
     }
 }
 
+// ---------------------------------------------------------------------------
+// These three stay RAW, and not because nobody got to them: they are read from
+// inside `Arena::new`, which runs as `ARENA`'s lazy initializer — and `ARENA`
+// is the FIRST provider `tls_hot::fill` resolves. Routing them through the hot
+// cache closes a cycle that ends in a stack overflow at thread start:
+//
+//     HotKey::get -> hot() -> hot_uncached -> hot_via_tls
+//       -> (temp_roots still null) fill()
+//         -> arena_hot_addr() -> ARENA.with(..) -> Arena::new
+//           -> ARENA_TOTAL_BYTES / BLOCK_POOL{,_BYTES}.with(..)
+//             -> HotKey::get -> hot() -> ... (temp_roots STILL null) ...
+//
+// `fill` writes `temp_roots` last precisely so a re-entrant reader cannot see a
+// half-filled cache as ready — which makes the nesting re-run `fill`, and
+// re-run `ARENA`'s initializer, without bound. It bites on any thread where
+// the first `hot()` precedes the first arena touch, i.e. every freshly spawned
+// one, and it fails as a crash rather than as a slow path.
+//
+// The rule this is an instance of: a declaration read from the dynamic extent
+// of a `tls_hot::fill` provider cannot use `crate::perry_thread_local!`. Today
+// `ARENA` is the only provider whose initializer runs code, so this is the
+// whole set.
+// ---------------------------------------------------------------------------
 thread_local! {
     static BLOCK_POOL: RefCell<BlockPool> = const { RefCell::new(BlockPool {
         blocks: Vec::new(),
@@ -981,7 +1004,9 @@ thread_local! {
     /// alloc into a tombstone slot or the end, and release inside
     /// `arena_reset_empty_blocks`).
     pub(crate) static ARENA_TOTAL_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
+}
 
+crate::perry_thread_local! {
     /// Cached running sum of `block.offset` across the old-gen arena —
     /// the delta-maintained twin of `ARENA_TOTAL_BYTES` above, same
     /// rationale: `gc_budgeted_due_trigger()` reads the old-gen in-use
@@ -1007,8 +1032,18 @@ thread_local! {
     /// the OldReclaim trigger.
     pub(crate) static OLD_GEN_IN_USE_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
 
+}
+
+// `ARENA` and `INLINE_STATE` below stay raw: they are NAMED `HotTls` fields
+// (`arena`, `inline_state`) whose addresses `tls_hot::fill` reads through the
+// providers further down, and a named field is one dependent load cheaper
+// than a claimed slot. Their neighbours in this block were never migrated.
+thread_local! {
     pub(crate) static ARENA: UnsafeCell<Arena> =
         UnsafeCell::new(Arena::new(HeapGeneration::Nursery, HeapSpace::NurseryEden));
+}
+
+crate::perry_thread_local! {
 
     /// Segregated long-lived arena (issue #179). Holds objects that are
     /// intentionally pinned for the lifetime of the program by explicit
@@ -1056,6 +1091,9 @@ thread_local! {
     pub(crate) static OLD_ARENA: UnsafeCell<Arena> =
         UnsafeCell::new(Arena::new_lazy(HeapGeneration::Old, HeapSpace::Old));
 
+}
+
+thread_local! {
     /// Inline allocator state — a cache of the current arena block's
     /// `(data, offset, size)` tuple, exposed via a stable pointer so
     /// codegen can emit inline bump-allocate IR without going through
@@ -1072,6 +1110,12 @@ thread_local! {
     }) };
 }
 
+/// Hot-cache slot claimed by `OLD_GEN_IN_USE_BYTES`. See above.
+#[cfg(test)]
+pub(crate) fn old_gen_in_use_bytes_slot_index() -> u32 {
+    OLD_GEN_IN_USE_BYTES.slot_index()
+}
+
 // --- #7469 hot-TLS address providers. See `crate::tls_hot`. ---
 
 /// Address of this thread's `ARENA`. Resolving it once and caching it is what

crates/perry-runtime/src/arena/mod.rs

@@ -46,6 +46,8 @@ pub(crate) use block::{
 /// allocation path uses instead of a per-access `_tlv_get_addr`.
 pub(crate) use block::{arena_hot_addr, hot_arena, hot_inline_state, inline_state_hot_addr};
 #[cfg(test)]
+pub(crate) use block::old_gen_in_use_bytes_slot_index;
+#[cfg(test)]
 pub(crate) use block::{
     block_pool_bytes_for_test, block_pool_explicit_drained_bytes_for_test, block_pool_put,
     force_next_block_alloc_failure, gc_trigger_arena_borrow_depth, gc_trigger_arena_calls,

crates/perry-runtime/src/arena/page_meta.rs

@@ -319,13 +319,20 @@ pub(crate) struct OldArenaSourceBlockSelection {
     pub(crate) pages: crate::fast_hash::PtrHashSet<usize>,
 }
 
+// `PAGE_GENERATIONS` and `PAGE_GENERATION_CACHE` stay raw because they are
+// NAMED fields of `HotTls` (`page_generations`, `page_generation_cache`), whose
+// providers below hand `tls_hot::fill` their addresses. A named field is one
+// dependent load cheaper than a claimed slot, which is why the closed set
+// exists; everything else in this file was simply never migrated.
 thread_local! {
     static PAGE_GENERATIONS: RefCell<PageGenerationMap> =
         RefCell::new(crate::fast_hash::new_ptr_hash_map());
 
     static PAGE_GENERATION_CACHE: UnsafeCell<PageGenerationCacheSet> =
         const { UnsafeCell::new(PageGenerationCacheSet::empty()) };
+}
 
+crate::perry_thread_local! {
     static OLD_GEN_PAGE_OBJECTS: RefCell<OldGenPageObjectMap> =
         RefCell::new(crate::fast_hash::new_ptr_hash_map());
 
@@ -1138,7 +1145,7 @@ pub(crate) fn register_old_object_pages(header_addr: usize, total_size: usize) {
 // which fails if a new toucher of either table appears without one.
 // ---------------------------------------------------------------------------
 
-thread_local! {
+crate::perry_thread_local! {
     /// Old-object page registrations not yet folded into `OLD_GEN_PAGE_OBJECTS`.
     /// Entries are `(header_addr, total_size)`; nothing here is dereferenced, so
     /// a deferred entry never keeps an object alive and is not a GC root — and

crates/perry-runtime/src/gc/barrier/mod.rs

@@ -697,6 +697,10 @@ pub(super) unsafe fn scan_dirty_object_slots(
 // retained only as a test fallback for the previous object-level
 // HashSet behavior.
 
+// The three declarations below stay raw: they are NAMED `HotTls` fields
+// (`incremental_mark_valid_ptrs`, `birth_extra_flags`,
+// `incremental_mark_minor_only`), one dependent load cheaper than a claimed
+// slot. Everything after them was simply never migrated.
 thread_local! {
     /// Active incremental mark barrier state (Full AND budgeted Minor
     /// cycles — a Minor cycle sliced across mutator turns has exactly the
@@ -737,7 +741,9 @@ thread_local! {
     /// later. Old children need no shading in a minor anyway: minors never
     /// collect live old-gen objects.
     pub(super) static INCREMENTAL_MARK_BARRIER_MINOR_ONLY: Cell<bool> = const { Cell::new(false) };
+}
 
+crate::perry_thread_local! {
     /// Dirty old-generation pages that have received a YOUNG-gen
     /// pointer since the last collection. This is Perry's compact
     /// modbuf: barriers log bounded page regions, and minor GC scans
@@ -1761,7 +1767,7 @@ fn mark_dirty_old_page_uncached(page: usize) -> bool {
     inserted
 }
 
-thread_local! {
+crate::perry_thread_local! {
     /// PERRY_GC_VERIFY_EVACUATION diagnostic only: every old page EVER marked
     /// dirty over the process lifetime (never cleared). Lets the verifier's
     /// missing-edge report say whether the slot's page was recorded at some

crates/perry-runtime/src/gc/malloc.rs

@@ -196,7 +196,7 @@ pub(super) const MALLOC_STATE_HEAVY_LEN_THRESHOLD: usize = 64 * 1024;
 /// 7/8 load factor without further rehashes).
 pub(super) const MALLOC_STATE_HEAVY_CAPACITY: usize = 256 * 1024;
 
-thread_local! {
+crate::perry_thread_local! {
     pub(crate) static MALLOC_STATE: RefCell<MallocState> = RefCell::new(MallocState {
         objects: Vec::with_capacity(MALLOC_STATE_INITIAL_CAPACITY),
         set: crate::fast_hash::PtrHashSet::with_capacity_and_hasher(
@@ -209,12 +209,25 @@ thread_local! {
         heavy_capacity_reserved: false,
         kind_telemetry: [MallocKindTelemetry::zero(); MALLOC_KIND_BUCKET_COUNT],
     });
+}
 
+// `ARENA_FREE_LIST` / `ARENA_FREE_LIST_NONEMPTY` stay raw: they are NAMED
+// `HotTls` fields (`arena_free_list`, `arena_free_list_nonempty`), one
+// dependent load cheaper than a claimed slot.
+thread_local! {
     pub(crate) static ARENA_FREE_LIST: RefCell<Vec<(*mut u8, usize)>> = const { RefCell::new(Vec::new()) };
     pub(crate) static ARENA_FREE_LIST_NONEMPTY: std::cell::Cell<bool> =
         const { std::cell::Cell::new(false) };
 }
 
+/// Hot-cache slot claimed by `MALLOC_STATE`, whose length is the
+/// `MallocCount` trigger's basis on every `gc_malloc`. Liveness
+/// instrumentation for `gc::tests::trigger_path_tls`.
+#[cfg(test)]
+pub(crate) fn malloc_state_slot_index() -> u32 {
+    MALLOC_STATE.slot_index()
+}
+
 pub fn gc_malloc(size: usize, obj_type: u8) -> *mut u8 {
     let total = GC_HEADER_SIZE + size;
     let layout = Layout::from_size_align(total, 8).unwrap();
@@ -480,7 +493,7 @@ pub(super) fn malloc_sweep_revalidate_header(
     })
 }
 
-thread_local! {
+crate::perry_thread_local! {
     pub(super) static MALLOC_REGISTRY_REBUILD_COUNT: Cell<u64> = const { Cell::new(0) };
 }
 

crates/perry-runtime/src/gc/old_free.rs

@@ -47,7 +47,7 @@
 
 use super::*;
 
-thread_local! {
+crate::perry_thread_local! {
     /// total_size -> user_ptrs of swept holes of exactly that size.
     static OLD_FREE_MAP: RefCell<crate::fast_hash::PtrHashMap<usize, Vec<usize>>> =
         RefCell::new(crate::fast_hash::new_ptr_hash_map());
@@ -60,6 +60,14 @@ pub(crate) fn old_free_bytes() -> usize {
     OLD_FREE_BYTES.with(Cell::get)
 }
 
+/// Hot-cache slot claimed by `OLD_FREE_BYTES`, which
+/// `gc_budgeted_due_trigger` reads on every `gc_malloc`. Liveness
+/// instrumentation for `gc::tests::trigger_path_tls`.
+#[cfg(test)]
+pub(crate) fn old_free_bytes_slot_index() -> u32 {
+    OLD_FREE_BYTES.slot_index()
+}
+
 fn old_free_push(user_ptr: usize, total_size: usize) {
     if user_ptr == 0 || total_size < GC_HEADER_SIZE {
         return;

crates/perry-runtime/src/gc/policy.rs

@@ -5,7 +5,7 @@ pub(super) const GC_FLAG_IN_ALLOC: u8 = 0b01;
 /// Bit 1 of GC_FLAGS — suppression flag (JSON.parse).
 pub(super) const GC_FLAG_SUPPRESSED: u8 = 0b10;
 
-thread_local! {
+crate::perry_thread_local! {
     pub(super) static GC_FLAGS: Cell<u8> = const { Cell::new(0) };
 }
 
@@ -292,7 +292,7 @@ thread_local! {
     static GC_NURSERY_CAP_TEST_SUPPRESSED: Cell<bool> = const { Cell::new(false) };
 }
 
-thread_local! {
+crate::perry_thread_local! {
     /// Lower bound for the next GC trigger. Bumped after each
     /// `gc_collect_inner` based on collection effectiveness (see the
     /// adaptive logic in `gc_check_trigger`).
@@ -401,7 +401,7 @@ pub(super) const GC_MALLOC_COUNT_STEP_INITIAL: usize = 100_000;
 pub(super) const GC_MALLOC_COUNT_STEP_MAX: usize = 2_000_000;
 pub(super) const GC_MALLOC_COUNT_STEP_MIN: usize = 10_000;
 
-thread_local! {
+crate::perry_thread_local! {
     /// Per-program adaptive malloc-count step. Mirrors `GC_STEP_BYTES`
     /// behaviour: doubles when mostly-garbage, halves when mostly-live.
     pub(super) static GC_MALLOC_COUNT_STEP: std::cell::Cell<usize> =
@@ -431,7 +431,7 @@ thread_local! {
 ///   the Map/Set side-allocation finalizers) instead of leaking.
 const GC_EXTERNAL_SIDE_ALLOC_STEP: usize = 16 * 1024 * 1024;
 
-thread_local! {
+crate::perry_thread_local! {
     static GC_EXTERNAL_SIDE_ALLOC_PENDING: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
     static GC_EXTERNAL_SIDE_LIVE_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
 }
@@ -942,7 +942,7 @@ impl GcTriggerSnapshot {
     }
 }
 
-thread_local! {
+crate::perry_thread_local! {
     pub(super) static GC_DEFERRED_REQUEST: Cell<DeferredGcRequest> =
         const { Cell::new(DeferredGcRequest::None) };
     pub(super) static GC_OLD_RECLAIM_PENDING: Cell<bool> = const { Cell::new(false) };
@@ -2655,7 +2655,7 @@ enum BudgetedGcTrigger {
     MallocCount,
 }
 
-thread_local! {
+crate::perry_thread_local! {
     static GC_BUDGETED_CYCLE: RefCell<Option<BudgetedGcCycle>> = const { RefCell::new(None) };
     static GC_BUDGETED_CYCLE_ACTIVE: Cell<bool> = const { Cell::new(false) };
     static GC_BUDGETED_STEP_ACTIVE: Cell<bool> = const { Cell::new(false) };
@@ -2690,6 +2690,56 @@ pub(super) fn gc_old_reclaim_debt_bytes(old_in_use: usize, baseline: usize) -> u
     old_in_use.saturating_sub(trigger) as u64
 }
 
+/// The thread-locals [`gc_budgeted_due_trigger`] reads on its fast path, with
+/// the hot-cache slot each one claimed.
+///
+/// Enumerating them in the module that reads them is what makes the coverage
+/// test able to fail: reverting any declaration below to a raw
+/// `thread_local!` removes `slot_index` and breaks this function's build at
+/// the declaration's own name, rather than leaving a silently slow path — the
+/// failure mode `tls_hot.rs` was written to abolish and that this path
+/// nonetheless kept for three years.
+#[cfg(test)]
+pub(crate) fn trigger_path_hot_slot_indices() -> Vec<(&'static str, u32)> {
+    // Touch each one first: a slot is claimed on first read, not at
+    // declaration, so an unread declaration reports the unassigned sentinel.
+    //
+    // `GC_DEFERRED_REQUEST` is deliberately absent: `defer_gc_request` reads it
+    // only when `GC_ROOT_LOCK_DEPTH` is non-zero, so on the fast path it is
+    // never touched and never claims a slot. Listing it made this function's
+    // own test fail with `index 4294967295` on the first run — which is the
+    // evidence that the test can fail, and the reason the list is "what the
+    // fast path reads" rather than "what the module declares".
+    let _ = gc_budgeted_due_trigger();
+    vec![
+        ("GC_OLD_RECLAIM_PENDING", GC_OLD_RECLAIM_PENDING.slot_index()),
+        (
+            "GC_LAST_OLD_RECLAIM_IN_USE_BYTES",
+            GC_LAST_OLD_RECLAIM_IN_USE_BYTES.slot_index(),
+        ),
+        ("GC_NEXT_MALLOC_TRIGGER", GC_NEXT_MALLOC_TRIGGER.slot_index()),
+        ("GC_NEXT_TRIGGER_BYTES", GC_NEXT_TRIGGER_BYTES.slot_index()),
+        ("GC_TRIGGER_ARMED", GC_TRIGGER_ARMED.slot_index()),
+        (
+            "GC_EXTERNAL_SIDE_LIVE_BYTES",
+            GC_EXTERNAL_SIDE_LIVE_BYTES.slot_index(),
+        ),
+        ("GC_MAJOR_PACING_RETAINING", GC_MAJOR_PACING_RETAINING.slot_index()),
+        ("GC_FLAGS", GC_FLAGS.slot_index()),
+        (
+            "GC_BUDGETED_CYCLE_ACTIVE",
+            GC_BUDGETED_CYCLE_ACTIVE.slot_index(),
+        ),
+        ("GC_BUDGETED_STEP_ACTIVE", GC_BUDGETED_STEP_ACTIVE.slot_index()),
+        (
+            "OLD_GEN_IN_USE_BYTES",
+            crate::arena::old_gen_in_use_bytes_slot_index(),
+        ),
+        ("OLD_FREE_BYTES", super::old_free_bytes_slot_index()),
+        ("MALLOC_STATE", super::malloc_state_slot_index()),
+    ]
+}
+
 fn gc_budgeted_due_trigger() -> Option<BudgetedGcTrigger> {
     let old_pending = GC_OLD_RECLAIM_PENDING.with(Cell::get);
     // #6010: external Map/Set side-buffer bytes escalate to OldReclaim too.