perf(gc): negative-cache the page classes the map does not hold — reach is 52.8-62.1% of remaining misses, not #9852's 90.7%

changelog.d/9827-gc-trigger-path-hot-tls.md

@@ -0,0 +1,55 @@
+**The GC trigger path and the dirty-page barrier stop paying `_tlv_get_addr`
+per read, and the policy gate that let them stop paying it now counts the
+thing it is bounding.**
+
+`gc_check_trigger` runs on every `gc_malloc`, and its predicate
+(`gc_budgeted_due_trigger`) resolved eleven raw `thread_local!` declarations
+one out-of-line call at a time. Measured with `sample` on the compiled
+claude-code TUI streaming a 3300-char reply (14,578 active main-thread
+samples, callers resolved by an explicit ancestor walk rather than
+nearest-symbol labels): `_tlv_get_addr` was 380 main-thread leaf samples,
+**71 of them with `gc_budgeted_due_trigger` as the immediate caller**, 36 in
+`old_page_account_dirty_slots`, 31 in `scan_dirty_object_slots`, 27 in
+`gc_malloc_header_is_tracked`. `crates/perry-runtime/src/tls_hot.rs` has
+existed to abolish exactly this since #7469; the allocation path's *fields*
+were covered and the trigger path never was.
+
+Sixty-seven declarations across `gc/policy.rs`, `gc/malloc.rs`, `gc/old_free.rs`,
+`gc/tenuring.rs`, `gc/trace.rs`, `gc/barrier/mod.rs`, `arena/block.rs` and
+`arena/page_meta.rs` move to `crate::perry_thread_local!` — same syntax, same
+`.with()` at every call site, the address served from this thread's hot cache
+instead of a libdyld call.
+
+**Why they were still cold is a measurement bug in the gate, not an oversight
+anyone could have noticed.** `scripts/check_thread_locals.py` ratchets on the
+number of raw `thread_local!` **blocks** per file, while `thread_local! { … }`
+holds any number of declarations — so `gc/policy.rs` counted as **6** while
+declaring **28**, and adding a `static` to an already-recorded block passed
+the gate silently. Counted in the same unit as the hot side, `main` was **318
+hot declarations against 339 cold ones** — cold was the majority, reported as
+a 2.6:1 minority. The gate now ratchets on declarations (`385 hot / 272
+cold`), and `--self-test` grew a seventh direction that fails when a `static`
+is added to a recorded block; restoring the block count makes that case, and
+only that case, fail.
+
+Three declarations stay deliberately raw and say so at their declaration:
+`ARENA_TOTAL_BYTES`, `BLOCK_POOL` and `BLOCK_POOL_BYTES` are read from
+`Arena::new`, which runs as `tls_hot::fill`'s **first** provider, so a
+`HotKey` there re-enters `fill` — which by design has not yet written the
+`temp_roots` field it gates on — and re-runs `ARENA`'s initializer without
+bound. It is a stack overflow at thread start, not a slow path, and it is the
+first documented instance of the rule that a declaration read from inside a
+`fill` provider cannot use the macro. `gc::tests::tls_fill_reentrancy` is the
+standing guard, and it is sabotage-proved: moving `ARENA_TOTAL_BYTES` alone
+into the neighbouring hot block aborts that test with `fatal runtime error:
+stack overflow`.
+
+`gc::tests::trigger_path_tls` is the runtime half of the gate: it drives
+`gc_check_trigger` on a fresh thread and asserts every trigger-path
+declaration owns a hot slot and that the path publishes slots at all.
+Reverting any one of them to a raw `thread_local!` removes `slot_index` and
+breaks the build at that declaration's own name. It is a test that can fail
+and did: the first run rejected `GC_DEFERRED_REQUEST` with `index 4294967295`,
+correctly — `defer_gc_request` reads it only while a root lock is held, so it
+is not a fast-path read and never claims a slot. The list is what the fast
+path reads, not what the module declares.

changelog.d/9845-gc-page-class-direct-table.md

@@ -0,0 +1,63 @@
+**The page-generation cache becomes a direct-indexed table over the arena's
+1 MiB address classes, so a classification is a bounds compare and one load
+instead of a four-way probe that missed one call in five.**
+
+`classify_heap_generation` and `classify_heap_space_in_range` sit under three
+callers with no cheaper predicate of their own. The write barrier's
+`remembered_child_needs_tracking` runs **35,871,391 times per turn** on the
+compiled claude-code TUI and **95.23 %** of those take its cheapest arm — one
+cached classification and a compare — so there is no barrier predicate left to
+fix: what remains after the predicate is already optimal is the classification
+itself. `mark_addr` (233 of 760 `classify*` leaf samples) and the side-table
+prunes pay the same cost.
+
+The structure in front of the authoritative `PageGenerationMap` was a **4-way
+round-robin set**. Measured with a dedicated counter on a 3300-char streaming
+reply: **440 M lookups per turn at 20.0–21.6 % miss**, with **59.7–61.8 % of
+misses on a key evicted within the last 64 evictions** — capacity, not conflict —
+against a working set of **402–432 registered classes**. `ways_distinct_max` was
+4, so every way was already in use and the shortfall is ~120x.
+
+**Widening it was not an option, and the reason is on the record.** #7469
+measured 16 ways as an **8.6 % regression** on the same row (0/7 pairs) for 1.5 %
+fewer misses, and five further associativity changes measured flat. The rule
+those produced — *associativity pays only when a miss is expensive* — says that a
+miss which is just a hash lookup wants the cache to become **unnecessary**, not
+larger.
+
+It can be. The registered classes occupy a span of **1,018–1,021 classes at
+~40 % density**, so a table over that span holds every one of them in **160 KB**
+and answers with one bounds compare and one load. `PageGenerationMap` stays
+authoritative and every miss falls through to it exactly as before; the change is
+confined to `PageGenerationCacheSet` and its two callers.
+
+Four things the measurement did not settle, each handled explicitly and each
+pinned by a test that fails when its guard is removed — a wrong answer here is a
+misclassified pointer, so none of them is left to inference:
+
+* **The base moves per process** (`0x43daa2` vs `0x57e3c2` on two runs — ASLR).
+  It is taken from the first insert, never compiled in.
+* **The span can grow** (1,018 → 1,021 across two runs of one binary). An insert
+  outside the table rebases it, up to a 16,384-class cap; past the cap the key is
+  left uncached and falls through to the map rather than being mis-indexed.
+* **The sizing is not obvious.** With base `first_key - S` and a table of `N`,
+  the span covered is `min(S + 1, N - S)`, maximised at `S = N / 2`. The natural
+  pairing `N = 4096, S = 1024` covers **1,025** classes — four above the measured
+  span — while `S = N / 2` covers **2,048** for the same memory. A `const` assert
+  now fails the build for any pairing covering less than twice the measured span.
+* **A key match is not an address match.** A class can hold more than one range,
+  so a hit still requires `range.contains(addr)`.
+
+Invalidation is an epoch bump: O(1), and the same "clear everything" contract the
+4-way set met by being reset wholesale. That contract matters more here, because
+the table holds ~2,000 entries where the set held 4 — a missing invalidation the
+old structure survived by luck would be a live misclassification — so all three
+`PageGenerationMap` mutation sites were enumerated and each ends with an
+unconditional `invalidate_generation_cache()`.
+
+The arm is a plain `u8` field in the set's first cache line rather than the env
+`OnceLock`: this path runs 440 M times per turn, and an acquire load on each
+would have been charged to both arms of the A/B — hiding it in the comparison
+that was meant to isolate it — while still being paid against main.
+`PERRY_GC_PAGE_CLASS_TABLE=0` restores the 4-way set in the same binary, which is
+how the numbers above and below were taken.

changelog.d/9852-gc-page-class-negative.md

@@ -0,0 +1,23 @@
+### Fixed
+
+- **gc:** the page-class table now remembers that a 1 MiB class holds no
+  registered range, so a candidate address in no GC block is answered from the
+  table instead of a failing hash lookup in the authoritative
+  `PageGenerationMap` (#9852).
+
+  After the direct-indexed table (#9853) took classification misses from
+  18.5 % to ~6 % of lookups, what remained was dominated by addresses the map
+  cannot answer either: conservative-scan candidates, interior pointers and
+  non-heap words that every classification path re-asks about millions of
+  times per turn. Counted on the compiled claude-code TUI, 3300-char reply,
+  three runs: **52.8-62.1 % of every remaining miss** is a class the map does
+  not hold at all, at 41.5-66.2 % over two 400-char runs.
+
+  A negative entry is an ordinary table entry carrying the current epoch, so
+  the registration that would make it wrong is the same event that already
+  invalidates every positive entry — no new invalidation obligation. It is
+  written **only** where `pages.get(&key)` is itself `None`; a class that holds
+  a range the address happens to miss is a different case (16-29 % of the same
+  population) and caching "absent" for it would answer wrongly for the
+  addresses the class really covers. `PERRY_GC_PAGE_CLASS_NEGATIVE=0` restores
+  the previous behaviour in the same binary.

crates/perry-runtime/src/arena/block.rs

@@ -122,6 +122,29 @@ impl Drop for BlockPool {
     }
 }
 
+// ---------------------------------------------------------------------------
+// These three stay RAW, and not because nobody got to them: they are read from
+// inside `Arena::new`, which runs as `ARENA`'s lazy initializer — and `ARENA`
+// is the FIRST provider `tls_hot::fill` resolves. Routing them through the hot
+// cache closes a cycle that ends in a stack overflow at thread start:
+//
+//     HotKey::get -> hot() -> hot_uncached -> hot_via_tls
+//       -> (temp_roots still null) fill()
+//         -> arena_hot_addr() -> ARENA.with(..) -> Arena::new
+//           -> ARENA_TOTAL_BYTES / BLOCK_POOL{,_BYTES}.with(..)
+//             -> HotKey::get -> hot() -> ... (temp_roots STILL null) ...
+//
+// `fill` writes `temp_roots` last precisely so a re-entrant reader cannot see a
+// half-filled cache as ready — which makes the nesting re-run `fill`, and
+// re-run `ARENA`'s initializer, without bound. It bites on any thread where
+// the first `hot()` precedes the first arena touch, i.e. every freshly spawned
+// one, and it fails as a crash rather than as a slow path.
+//
+// The rule this is an instance of: a declaration read from the dynamic extent
+// of a `tls_hot::fill` provider cannot use `crate::perry_thread_local!`. Today
+// `ARENA` is the only provider whose initializer runs code, so this is the
+// whole set.
+// ---------------------------------------------------------------------------
 thread_local! {
     static BLOCK_POOL: RefCell<BlockPool> = const { RefCell::new(BlockPool {
         blocks: Vec::new(),
@@ -981,7 +1004,9 @@ thread_local! {
     /// alloc into a tombstone slot or the end, and release inside
     /// `arena_reset_empty_blocks`).
     pub(crate) static ARENA_TOTAL_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
+}
 
+crate::perry_thread_local! {
     /// Cached running sum of `block.offset` across the old-gen arena —
     /// the delta-maintained twin of `ARENA_TOTAL_BYTES` above, same
     /// rationale: `gc_budgeted_due_trigger()` reads the old-gen in-use
@@ -1007,8 +1032,18 @@ thread_local! {
     /// the OldReclaim trigger.
     pub(crate) static OLD_GEN_IN_USE_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
 
+}
+
+// `ARENA` and `INLINE_STATE` below stay raw: they are NAMED `HotTls` fields
+// (`arena`, `inline_state`) whose addresses `tls_hot::fill` reads through the
+// providers further down, and a named field is one dependent load cheaper
+// than a claimed slot. Their neighbours in this block were never migrated.
+thread_local! {
     pub(crate) static ARENA: UnsafeCell<Arena> =
         UnsafeCell::new(Arena::new(HeapGeneration::Nursery, HeapSpace::NurseryEden));
+}
+
+crate::perry_thread_local! {
 
     /// Segregated long-lived arena (issue #179). Holds objects that are
     /// intentionally pinned for the lifetime of the program by explicit
@@ -1056,6 +1091,9 @@ thread_local! {
     pub(crate) static OLD_ARENA: UnsafeCell<Arena> =
         UnsafeCell::new(Arena::new_lazy(HeapGeneration::Old, HeapSpace::Old));
 
+}
+
+thread_local! {
     /// Inline allocator state — a cache of the current arena block's
     /// `(data, offset, size)` tuple, exposed via a stable pointer so
     /// codegen can emit inline bump-allocate IR without going through
@@ -1072,6 +1110,12 @@ thread_local! {
     }) };
 }
 
+/// Hot-cache slot claimed by `OLD_GEN_IN_USE_BYTES`. See above.
+#[cfg(test)]
+pub(crate) fn old_gen_in_use_bytes_slot_index() -> u32 {
+    OLD_GEN_IN_USE_BYTES.slot_index()
+}
+
 // --- #7469 hot-TLS address providers. See `crate::tls_hot`. ---
 
 /// Address of this thread's `ARENA`. Resolving it once and caching it is what

crates/perry-runtime/src/arena/mod.rs

@@ -46,13 +46,15 @@ pub(crate) use block::{
 /// allocation path uses instead of a per-access `_tlv_get_addr`.
 pub(crate) use block::{arena_hot_addr, hot_arena, hot_inline_state, inline_state_hot_addr};
 #[cfg(test)]
+pub(crate) use block::old_gen_in_use_bytes_slot_index;
+#[cfg(test)]
 pub(crate) use block::{
     block_pool_bytes_for_test, block_pool_explicit_drained_bytes_for_test, block_pool_put,
     force_next_block_alloc_failure, gc_trigger_arena_borrow_depth, gc_trigger_arena_calls,
     reset_gc_trigger_arena_probe,
 };
 pub(crate) use page_meta::{
-    address_span_overlaps_pages, defer_old_object_page_registration,
+    address_span_overlaps_pages, defer_old_object_page_registration, page_class_table_report,
     register_block_space_with_object_starts, register_old_object_pages,
     unregister_block_generation, unregister_old_block_pages, OLD_GEN_RECLAIM_POOLED_BYTES,
     OLD_GEN_RECLAIM_RETURNED_BYTES, OLD_GEN_RECLAIM_REUSABLE_BYTES,