fix(gc): the tenuring occupancy rule may not claim promote-on-first-copy (#9851)

changelog.d/9851-lock-rates-one-cohort.md

@@ -0,0 +1,49 @@
+### Fixed
+
+- **The tenuring survival-rate lock now rates one fresh cohort, not the whole
+  survivor space — a well-formed ratio that stopped describing what it is named
+  after as soon as the threshold it sets rose above 2.**
+
+  The lock exists to answer "did an aging round filter anything?" and, when the
+  answer is no, to promote on first copy. It tested
+
+  ```
+  prev_copied >= substantial && survivor_live_bytes * 10 >= prev_copied * 9
+  ```
+
+  where `survivor_live_bytes` is every live byte leaving the from-survivor space
+  this cycle, of any age, and `prev_copied` is the previous cycle's whole intake
+  into that space. Those two scopes match — the survivor spaces are a strict
+  semispace pair, so the from-space holds exactly what the last cycle copied —
+  and the ratio cannot exceed 1. **The defect is not the arithmetic; it is which
+  population the ratio rates, and that is chosen by the very threshold the lock
+  sets.** At a threshold of 2 the space holds one fresh cohort and the ratio is
+  one aging round's survival. At 3 or 4 it also holds objects that have already
+  survived a round and are therefore selected for longevity, so the aggregate
+  clears the 90 % bar while a fresh cohort does not. The rule reads its own
+  setting back as evidence.
+
+  This was invisible while the occupancy rule sealed the loop at S=1, because
+  there `copied_bytes` is 0 and the lock's guard can never be satisfied.
+  Removing that seal handed the lock its guard back, and it became the dominant
+  route to promote-on-first-copy.
+
+  Measured on the compiled claude-code TUI, one binary, three arms via
+  `PERRY_GC_TENURING_SURVIVALS`, 3300-character replies, 4 turns in one process:
+
+  | arm | minors | promoted | S=1 share of promotion | reached 1 via the lock |
+  |---|---|---|---|---|
+  | `=1` (pre-clamp equivalent) | 356 | 1055 MB | 100 % | - |
+  | occupancy clamp only | 368 / 384 | 982 / 980 MB | 85 % / 84 % | **8 / 12** |
+  | `=2` (positive control) | 380 | 785 MB | 0 % | n/a |
+
+  The copier now also accounts the fresh half of each cycle: `eden_copied_bytes`
+  (bytes copied out of *Eden* into the to-survivor space, no re-copies) and
+  `survivor_first_round_live_bytes` (live bytes leaving the from-survivor space
+  whose stored survival age is 1, i.e. members of exactly the cohort the
+  previous cycle's `eden_copied_bytes` counted). The lock rates those two. Both
+  are on the `[gc-copy-minor]` diagnostic line, so first-round mortality is
+  readable from any build rather than only from an instrumented one.
+
+  Reaching 1 still belongs to the paths that measure mortality; what changes is
+  that the measurement is now of one aging round at every threshold.

changelog.d/9851-occupancy-may-not-promote-on-first-copy.md

@@ -0,0 +1,67 @@
+### Fixed
+
+- **The adaptive tenuring loop's occupancy rule can no longer conclude
+  "promote on first copy" — a claim about lifetime that it has no evidence
+  for, and which destroys the evidence that would refute it.**
+
+  `retune_after_scavenge` picks a survival threshold from
+  `S = 1 + desired / influx`: the largest S whose projected survivor occupancy
+  `(S-1) x influx` fits the desired survivor size. With integer division, any
+  influx above `desired` yields exactly **1** — there is no rung at 2 or 3.
+  Measured on the compiled claude-code TUI, the first drop reads
+  `eden_live_bytes=12075344` against `desired=1048576`.
+
+  S=1 does not reduce the surviving data; it relocates it, from the survivor
+  space — where the next minor re-examines it for free — to the old generation,
+  which only a full collection can reclaim. The occupancy formula has no term
+  for that. And S=1 is **self-sealing**: nothing is copied, so `copied_bytes` is
+  0, so next cycle `prev_copied` is 0, so the survival-rate lock's guard
+  (`prev_copied >= substantial`) is false forever. Both remaining exits — the
+  occupancy recompute and `PROMOTE_LOCK`'s unlock — are *quiet-influx* exits,
+  which say nothing about lifetime.
+
+  Measured, 4 streamed turns in one process, both arms from one binary via the
+  diagnostic knob `PERRY_GC_TENURING_SURVIVALS`, 3300-character replies:
+
+  | | adaptive | pinned S=2 |
+  |---|---|---|
+  | minors at S=1 | **351 of 352**, carrying 100 % of promotion | 0 |
+  | threshold transitions in the whole run | **1** | 7 |
+  | survivor-round mortality samples | **1** | **393** |
+  | median mortality | **0.9 %** | **26.1 %** |
+  | ...in steady turns 2 / 3 / 4 | not measurable | 26.1 / 26.1 / 26.1 % |
+  | promoted | 1057 MB | 792 MB |
+
+  The loop takes its one and only mortality measurement on the **first minor of
+  the process** — before any steady state, when the cohort really is immortal —
+  reads 99.1 % survival, drops to 1, and can never sample again. In steady
+  state an aging round filters about **a quarter** of each cohort.
+
+  The occupancy rule now stops at the lowest threshold that still *produces*
+  that measurement. That value is 2 by construction, not by tuning: at S=1
+  nothing enters the survivor space, at S=2 exactly one cohort does. The
+  arithmetic is untouched — `compute_target_survivals` still computes 1, and
+  its test asserts so byte-identically; only what the loop may do with the
+  result changes.
+
+  **Reaching 1 still belongs to the two paths that measure mortality** — the
+  survival-rate lock (a substantial cohort of which >= 90 % came back alive)
+  and the sweep seed (the mark-sweep's own Eden live/dead split). Both are
+  untouched, so the rule is self-limiting: on a workload whose cohort genuinely
+  does not die, the lock fires after one cohort's copy and takes the loop back
+  to 1.
+
+  On claude-code it **does** fire, 8-12 times per four-turn run, and the
+  companion entry below is why: once the clamp lets the ladder climb past 2 the
+  lock is rating a population its own threshold selected. An earlier version of
+  this entry claimed the opposite ("5 of 358 substantial cohorts sit under the
+  lock's threshold, so the clamp holds rather than oscillating"); that figure
+  was measured with the threshold *pinned*, where every cohort the lock can
+  rate is a first-round cohort, and it does not describe the rule running.
+
+  Two existing tests change their expected value from 1 to 2 and keep their
+  names, structure and invariants: `drops_immediately_and_rises_debounced`
+  protects the *asymmetric response* (immediate drop, debounced rise), which
+  4 -> 2 demonstrates exactly as well as 4 -> 1; and
+  `steady_heavy_influx_is_a_fixed_point` protects *fixed-pointness*, which is
+  unchanged with 2 as the fixed point.

crates/perry-runtime/src/gc/copying.rs

@@ -617,8 +617,29 @@ impl CopyingNurseryCollector {
         // moved somewhere at any threshold), which is what makes the loop's
         // fixed point stable.
         match ptr.kind {
-            CopyingPointerKind::Eden => self.stats.eden_live_bytes += total,
-            _ => self.stats.survivor_live_bytes += total,
+            CopyingPointerKind::Eden => {
+                self.stats.eden_live_bytes += total;
+                // #9851 follow-up: the fresh half of `copied_bytes`. The
+                // survival-rate lock's denominator must be the intake of ONE
+                // cohort; `copied_bytes` also carries survivor residents being
+                // re-copied, which at a threshold above 2 is most of it.
+                if !promote {
+                    self.stats.eden_copied_bytes += total;
+                }
+            }
+            _ => {
+                self.stats.survivor_live_bytes += total;
+                // ...and the matching numerator. A from-survivor object whose
+                // stored age is 1 entered from Eden on the previous cycle, so
+                // it is a member of exactly the cohort `eden_copied_bytes`
+                // counted then. Ages above 1 have already survived a round and
+                // are a population selected for longevity; including them is
+                // what made the ratio drift above the lock's bar as the
+                // threshold rose.
+                if prior_age == 1 {
+                    self.stats.survivor_first_round_live_bytes += total;
+                }
+            }
         }
         new_user as usize
     }
@@ -1867,14 +1888,24 @@ pub(super) fn run_copied_minor_attempt(
             .copied_objects
             .saturating_add(collector.stats.promoted_objects),
     );
-    retune_after_scavenge(
-        collector.stats.eden_live_bytes,
+    // #9851 follow-up: the survival-rate lock is fed the FRESH cohort's intake
+    // and that same cohort's survival, not the whole survivor space's. See
+    // `retune_after_scavenge`.
+    #[cfg(test)]
+    test_record_cohort_split(
         collector.stats.copied_bytes,
+        collector.stats.eden_copied_bytes,
         collector.stats.survivor_live_bytes,
+        collector.stats.survivor_first_round_live_bytes,
+    );
+    retune_after_scavenge(
+        collector.stats.eden_live_bytes,
+        collector.stats.eden_copied_bytes,
+        collector.stats.survivor_first_round_live_bytes,
     );
     if crate::gc::gc_diag_enabled() {
         eprintln!(
-            "[gc-copy-minor] ran in_place={} untraced={} untraced_cycles={} untraced_objects={} in_place_blocks={} in_place_dead_bytes={} sparse_blocks={} survival_permille={} copied_objects={} copied_bytes={} promoted_objects={} promoted_bytes={} freed_bytes={} tenuring_survivals={} eden_live_bytes={} trigger={:?} declared_safepoint={}",
+            "[gc-copy-minor] ran in_place={} untraced={} untraced_cycles={} untraced_objects={} in_place_blocks={} in_place_dead_bytes={} sparse_blocks={} survival_permille={} copied_objects={} copied_bytes={} promoted_objects={} promoted_bytes={} freed_bytes={} tenuring_survivals={} eden_live_bytes={} eden_copied_bytes={} survivor_live_bytes={} survivor_first_round_live_bytes={} trigger={:?} declared_safepoint={}",
             collector.stats.in_place_promotion,
             untraced,
             super::untraced_promotion_cycles(),
@@ -1890,6 +1921,9 @@ pub(super) fn run_copied_minor_attempt(
             freed_bytes,
             collector.stats.tenuring_survivals,
             collector.stats.eden_live_bytes,
+            collector.stats.eden_copied_bytes,
+            collector.stats.survivor_live_bytes,
+            collector.stats.survivor_first_round_live_bytes,
             _trigger_kind,
             super::policy::GC_AT_DECLARED_SAFEPOINT.with(std::cell::Cell::get)
         );
@@ -1908,6 +1942,41 @@ pub(super) fn run_copied_minor_attempt(
     }))
 }
 
+/// Test-only witness for the #9851 follow-up: the whole-space pair against the
+/// fresh-cohort pair, as the copier computed them for one cycle. Without this
+/// the change is unfalsifiable from a test — the two quantities are equal on
+/// every heap whose survivor space holds a single generation, which is every
+/// heap at a threshold of 2 or below.
+#[cfg(test)]
+thread_local! {
+    static LAST_COHORT_SPLIT: std::cell::Cell<(usize, usize, usize, usize)> =
+        const { std::cell::Cell::new((0, 0, 0, 0)) };
+}
+
+#[cfg(test)]
+fn test_record_cohort_split(
+    copied_bytes: usize,
+    eden_copied_bytes: usize,
+    survivor_live_bytes: usize,
+    first_round_live_bytes: usize,
+) {
+    LAST_COHORT_SPLIT.with(|c| {
+        c.set((
+            copied_bytes,
+            eden_copied_bytes,
+            survivor_live_bytes,
+            first_round_live_bytes,
+        ))
+    });
+}
+
+/// `(copied_bytes, eden_copied_bytes, survivor_live_bytes, first_round_live_bytes)`
+/// from the most recent copying minor on this thread.
+#[cfg(test)]
+pub(super) fn test_last_cohort_split() -> (usize, usize, usize, usize) {
+    LAST_COHORT_SPLIT.with(std::cell::Cell::get)
+}
+
 fn finalize_dead_copied_minor_from_space_side_allocations() {
     crate::map::finalize_dead_copied_minor_from_space_maps();
     crate::set::finalize_dead_copied_minor_from_space_sets();

crates/perry-runtime/src/gc/telemetry.rs

@@ -275,6 +275,18 @@ pub(super) struct CopyingNurseryTraceStats {
     /// Live bytes re-copied/promoted out of the from-survivor space this
     /// cycle — the re-copy tax the adaptive loop exists to bound.
     pub(super) survivor_live_bytes: usize,
+    /// #9851 follow-up: the FRESH half of `copied_bytes` — bytes copied out of
+    /// Eden into the to-survivor space this cycle, excluding survivor-space
+    /// residents being re-copied. This is the intake of exactly one cohort,
+    /// and it is the denominator the survival-rate lock must use.
+    pub(super) eden_copied_bytes: usize,
+    /// The matching numerator: live bytes moved out of the from-survivor space
+    /// this cycle whose stored survival age was 1 — i.e. objects that entered
+    /// the survivor space from Eden on the PREVIOUS cycle, and nothing older.
+    /// `survivor_live_bytes` rates the whole space, whose composition changes
+    /// with the threshold; this rates one aging round of one fresh cohort,
+    /// which is what the lock's conclusion is about.
+    pub(super) survivor_first_round_live_bytes: usize,
     pub(super) large_excluded_objects: usize,
     pub(super) large_excluded_bytes: usize,
     pub(super) reset_blocks: usize,
@@ -1151,6 +1163,8 @@ impl GcCycleTrace {
             "tenuring_survivals": self.copying_nursery.tenuring_survivals,
             "eden_live_bytes": self.copying_nursery.eden_live_bytes,
             "survivor_live_bytes": self.copying_nursery.survivor_live_bytes,
+            "eden_copied_bytes": self.copying_nursery.eden_copied_bytes,
+            "survivor_first_round_live_bytes": self.copying_nursery.survivor_first_round_live_bytes,
             "large_excluded_objects": self.copying_nursery.large_excluded_objects,
             "large_excluded_bytes": self.copying_nursery.large_excluded_bytes,
             "reset_blocks": self.copying_nursery.reset_blocks,