fix(gc): the tenuring occupancy rule may not claim promote-on-first-copy (#9851) - #9861
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…opy (PerryTS#9851) The adaptive tenuring loop takes its one and only survivor-round mortality sample on the FIRST minor of the process -- when the cohort really is immortal (99.1 % survival) -- drops the threshold to 1, and thereby destroys its ability to ever sample again: n=1 across 352 minors. In steady state an aging round filters 26.1 % of each cohort, and the loop cannot see it. `retune_after_scavenge` picks the threshold from `S = 1 + desired / influx`, the largest S whose projected survivor occupancy `(S-1) x influx` fits the desired survivor size. With integer division, any influx above `desired` yields exactly 1 -- there is no rung at 2 or 3. On the compiled claude-code TUI the first drop reads `eden_live_bytes=12075344` against `desired=1048576`. S=1 does not reduce the surviving data; it relocates it, from the survivor space -- where the next minor re-examines it for free -- to the old generation, which only a full can reclaim. The occupancy formula has no term for that. And S=1 is self-sealing: nothing is copied, so `copied_bytes` is 0, so next cycle `prev_copied` is 0, so the survival-rate lock's guard (`prev_copied >= substantial`) is false forever. Both remaining exits -- the occupancy recompute and PROMOTE_LOCK's unlock -- are QUIET-INFLUX exits, which say nothing about lifetime. The loop concludes "long-lived" from a premise about space and then removes its ability to check. Measured, 4 streamed turns in one process, both arms from one binary via the diagnostic knob PERRY_GC_TENURING_SURVIVALS, 3300-character replies: adaptive pinned S=2 minors at S=1 351 of 352 (100 % of promotion) 0 threshold transitions 1 7 mortality samples 1 393 median mortality 0.9 % 26.1 % ...steady turns 2 / 3 / 4 not measurable 26.1 / 26.1 / 26.1 % substantial cohorts < 10 % 1/1 5/358 promoted 1057 MB 792 MB The occupancy rule now stops at the lowest threshold that still PRODUCES that measurement. 2 is forced by the requirement rather than tuned: at S=1 nothing enters the survivor space, at S=2 exactly one cohort does. The clamp is at the USE SITE, not inside `compute_target_survivals`: that pure function has a second caller, `full_seed_promotes_on_first_copy`, which gates the sweep seed on `... != 1`. Clamping the shared function would silently disarm the sweep seed, which is one of the two paths that IS allowed to reach 1. Reaching 1 still belongs to the survival-rate lock and the sweep seed, which measure mortality; both are untouched, so the rule is self-limiting -- on a workload whose cohort genuinely does not die the lock fires after one cohort's copy and takes the loop back to 1. On claude-code it correctly does not: 5 of 358 substantial cohorts sit under the lock's 90 % bar, so the clamp holds rather than oscillating. Tests. `target_formula_matches_projected_occupancy` is byte-identical -- the arithmetic is untouched, and that test is the proof. Four tests move an expected value 1 -> 2 and keep their names, structure and invariants: `drops_immediately_and_rises_debounced` (asymmetric response: immediate drop, debounced rise -- 4 -> 2 shows it as well as 4 -> 1), `steady_heavy_influx_is_a_fixed_point` (fixed-pointness, now at 2), `heavy_influx_lowers_threshold_and_promotes_next_cycle` (its promotion half is untouched: the cohort was copied once, so `next_age` is 2 on cycle 2 and it still tenures exactly when the test says) and `quiet_cycles_restore_power_on_threshold_debounced` (the debounced restore is asserted structurally and survives). Two new tests: the two-phase attributed pair -- occupancy alone holds at the floor and has not taken the lock's route, then a substantial fully-surviving cohort still reaches 1 through the lock -- and a dying-cohort test at claude-code's measured 74 % survival.
… survivor space Follow-up to the previous commit, and caused by it. PerryTS#9851's clamp stops the occupancy rule concluding "promote on first copy", and measuring the relinked candidate showed it buys -7 % of promotion where the pinned control buys -26 %: 85 % of promotion still happens at S=1, now reached through the survival-rate lock 8-12 times per four-turn run. That is a consequence of the clamp, not a coincidence. At S=1 nothing is copied, so `prev_copied` is 0 and the lock's guard can never be satisfied -- the previous commit's own argument. Removing the seal hands the lock its guard back, and the lock then reaches 1 by itself. The lock tested prev_copied >= substantial && survivor_live_bytes * 10 >= prev_copied * 9 where `survivor_live_bytes` is every live byte leaving the from-survivor space this cycle, of any age, and `prev_copied` is the previous cycle's whole intake. Those two scopes MATCH: the survivor spaces are a strict semispace pair (to-space reset before the minor, everything copied into it, then flip), so the from-space at cycle N holds exactly what cycle N-1 copied. The ratio is well-formed and cannot exceed 1. The defect is not the arithmetic. The defect is which POPULATION the ratio rates, and that is chosen by the very threshold the lock sets. At S <= 2 the space holds one fresh cohort (age-2 is promoted) and the ratio is one aging round's survival -- 74 % on the compiled claude-code TUI, under the 90 % bar. At S = 3-4 it also holds age-2 and age-3 objects, which have already survived a round and are therefore selected for longevity, so the aggregate clears 90 % while a fresh cohort does not. The rule reads its own setting back as evidence. The clamp is what lets the debounced rise reach 3 and 4, which is why this only became visible once the seal was gone. The copier now accounts the fresh half of each cycle. `eden_copied_bytes` is what this cycle copied out of EDEN into the to-survivor space (no re-copies) -- one cohort's intake. `survivor_first_round_live_bytes` is what came back out of the from-survivor space alive with a stored survival age of 1, i.e. members of exactly the cohort the previous cycle's `eden_copied_bytes` counted; the age is already in the header at copy time (`copied_survival_age`), so no new per-object state is needed. `retune_after_scavenge` keeps its arity and its two lock parameters are redefined to those, which is the whole change at the policy end: both sides of the ratio are now scoped to one cohort at every threshold. Both new counts are on the `[gc-copy-minor]` diagnostic line next to the whole-space ones, so first-round mortality is readable from ANY build rather than only from an instrumented branch -- the measurement this policy is about should not require a custom binary. Measured, one binary, three arms via `PERRY_GC_TENURING_SURVIVALS`, 3300-char replies, 4 turns in one process, macOS arm64: arm minors promoted S=1 share via the lock =1 (pre-clamp equivalent) 356 1055 MB 100 % - clamp only, run 1 368 982 MB 85 % 8 clamp only, run 2 384 980 MB 84 % 12 =2 (positive control) 380 785 MB 0 % n/a Tests. No existing expected value moves -- all 1,069 gc tests pass unchanged, which is itself the finding: nothing in the suite distinguished the two scopes, because they are equal on every heap whose survivor space holds one generation, and that is every heap at a threshold of 2 or below. So the premise gets a test of its own on a real heap: two rooted objects introduced one cycle apart at the power-on threshold, asserting that the two numbers AGREE while only one generation is resident and then DIFFER once an aged resident joins it, with the aged object in the whole-space number and not in the cohort number. A test-only witness (`test_last_cohort_split`) reports the pair the copier computed. The two lock tests keep their values and gain the scoping in their names and comments; `a_cohort_that_dies_in_its_round_holds_at_the_occupancy_floor` now states that this same heap locks if the call site passes the whole space, which is what it used to pass. The previous commit's changelog fragment claimed the lock correctly stays out on claude-code (5 of 358 substantial cohorts under the bar). That figure was taken with the threshold PINNED, where every cohort the lock can rate is a first-round cohort; it does not describe the rule running, and the fragment is corrected rather than left to be read as a result. Claude-Session: https://claude.ai/code/session_014knX724SYDogwzsXybCGxp
566601b to
828974c
Compare
…d is measured The symmetric half of PerryTS#9851. That commit stopped the occupancy rule concluding "promote on first copy" -- a claim about LIFETIME derived from a measurement of SPACE. The same formula makes the same category error at the other end: compute_target_survivals = 1 + desired / influx (capped at the ceiling) returns the ceiling for a tiny influx AND for a zero one. On the first minors of a process -- heap nearly empty, no cohort ever followed -- occupancy therefore claims the MAXIMUM, before a single object has been given the chance to die. It is the expensive direction of the error, because every survivor is then copied up to three times before it may be promoted. Measured on the landing base (main5 + PerryTS#9881, one binary, four env arms, two rounds of 4 turns at 3300 and 400, quiet host), this startup excursion is the WHOLE difference between the adaptive loop and a pinned threshold: * unset vs pinned S=1: turn-1 CPU 3.41 s vs 3.02 s at 3300 (+0.35..0.45 s both rounds) and 1.05 s vs 0.72 s at 400 (+50 %), while the sum over turns 2-4 is within noise (6.18-6.23 vs 6.35-6.41); * the adaptive arm's transitions are `4 -> 2 (occupancy) -> 1 (lock)` and ALL of them land inside turn 1; turns 2-4 run at S=1 with nothing copied. So the adaptive policy's only cost on this workload was a startup claim it had no evidence for, and its steady state was already the pinned one. The rule is now symmetric: **until one survivor round has actually been rated, the occupancy rule holds at `OCCUPANCY_MIN_SURVIVALS`.** That value is not a tuning choice; it is the lowest threshold that PRODUCES the measurement the rule needs in order to say anything -- at 1 nothing enters the survivor space, at 2 exactly one cohort does. The power-on threshold becomes the same value for the same reason: starting at the ceiling is a lifetime claim made before the process has run. `SURVIVOR_ROUND_MEASURED` is set the moment a cohort the previous cycle copied becomes rateable, so the gate lifts after about two minors and the ladder is unchanged from then on -- it delays the claim until evidence exists, it does not remove the ladder. The two paths that MEASURE mortality are untouched: the survival-rate lock and the sweep seed may still reach 1 whenever they have the evidence for it. `compute_target_survivals` is again left alone, and its test is again the proof: the arithmetic still returns the ceiling for a zero and a tiny influx. Only what the loop may do with that changes. Tests. A new two-phase test: eight startup-shaped minors (tiny influx, nothing copied) must leave the loop at the floor and out of the lock; then, once a cohort has gone through the survivor space and been followed, the debounced rise must still reach the ceiling. Sabotage: delete the gate, or restore the power-on value to the ceiling, and phase 1 fails. Two existing tests move with the power-on value and keep their properties: `drops_immediately_and_rises_debounced` is about the ladder's ASYMMETRY, so it now seeds a fully-dying cohort first (which rates a round without involving the lock) and then tests the same immediate-drop / debounced-rise behaviour; `sweep_seed_refuses_a_small_fully_live_eden` asserts the threshold is unchanged from power-on, which is the floor now. `survival_rate_lock_breaks_a_saturated_ pipeline` needs no change -- the lock firing implies a rated round, so its ladder recovery is unaffected. NOT COMPILED: the box is at 7 GB free, under this campaign's 12 GB build floor, so neither the build nor the suite has been run against this commit. The braces balance and the reasoning above is stated per test, but that is a review and not a check.
LAST_COHORT_SPLIT (cfg(test), copying.rs) is test_only; SURVIVOR_ROUND_MEASURED (tenuring.rs) is a boolean, not a GC pointer. Inventory only; no code change.
01ec430 to
2248fba
Compare
jdalton
left a comment
There was a problem hiding this comment.
Review of 2248fba564ce1dff248873757a80528fd31806b6 (2026-09-07).
The cohort-scoped numerator/denominator and two-phase tests address the occupancy-versus-survival confusion. Please add a phase-changing linked workload: start with a long-lived cohort that takes PROMOTE_LOCK, then switch to short-lived cohorts while influx stays high. The lock branch returns before the occupancy floor, and first-copy promotion removes the next cohort's survival sample, so the floor alone does not demonstrate recovery from a formerly justified lock. Report promotions, full collections, and settled memory for that transition. Treat this as a remaining limitation/validation request, not as evidence that the current steady-state measurements are wrong.
Validation scope: source/diff inspection; I have not run this PR's build or test suite locally.
Fixes #9851. Based on
644b9d362(maind36a1af0c+ #9838), two commits: the occupancy clamp, and the lock rewire the clamp's own measurement forced (see the last two sections).The adaptive tenuring loop takes its one and only survivor-round mortality sample on the first minor of the process — when the cohort really is immortal (99.1 % survival) — drops the threshold to 1, and thereby destroys its ability to ever sample again: n=1 across 352 minors. In steady state an aging round filters 26.1 % of each cohort, and the loop cannot see it.
Mechanism
retune_after_scavengepicks the threshold fromS = 1 + desired / influx— the largest S whose projected survivor occupancy(S-1) × influxfits the desired survivor size. With integer division, any influx abovedesiredyields exactly 1; there is no rung at 2 or 3. On the compiled claude-code TUI the first drop readseden_live_bytes=12075344againstdesired=1048576.S=1 does not reduce the surviving data. It relocates it — from the survivor space, where the next minor re-examines it for free, to the old generation, which only a full can reclaim. The occupancy formula has no term for that.
And S=1 is self-sealing: nothing is copied, so
copied_bytesis 0, so next cycleprev_copiedis 0, so the survival-rate lock's guard (prev_copied >= substantial) is false forever. Both remaining exits — the occupancy recompute andPROMOTE_LOCK's unlock — are quiet-influx exits, which say nothing about lifetime. The loop concludes "long-lived" from a premise about space, and then removes its ability to check.Measured
4 streamed turns in one process, both arms from one binary via the diagnostic knob
PERRY_GC_TENURING_SURVIVALS, 3300-character replies:update_major_pacing_backoffSteady state is required to see this, and a single turn inverts the verdict. The kill condition was registered in advance ("under 10 % mortality on substantial cohorts, the mortality reading is dead"). Turn 1 alone reads 3.6 % and would have fired it; turns 2–4 read 26.1 % flat. The transient and the steady state are different regimes and the quantity moves 7× between them — startup allocates a genuinely long-lived cohort, so any "does it die?" measurement reads low there.
The change
One clamp, at the use site:
2 is forced, not tuned: it is the lowest S at which
copied_bytes > 0, i.e. the lowest value that still produces the survivor-round measurement. At S=1 nothing enters the survivor space; at S=2 exactly one cohort does.Deliberately not inside
compute_target_survivals. That pure function has a second caller —full_seed_promotes_on_first_copy, which gates the sweep seed on... != 1("would occupancy alone already promote on first copy?"). Clamping the shared function would silently disarm the sweep seed, which is one of the two paths that is allowed to reach 1.Reaching 1 still belongs to the two paths that measure mortality — the survival-rate lock and the sweep seed — and both are untouched. So the rule is self-limiting: on a workload whose cohort genuinely does not die, the lock fires after one cohort's copy and takes the loop back to 1.
Tests
In this order, because the first one is the proof:
target_formula_matches_projected_occupancyis byte-identical. The arithmetic is untouched —compute_target_survivalsstill computes 1. Only what the loop may do with the result changes.drops_immediately_and_rises_debounced— protects the asymmetric response (immediate drop, debounced rise). 4 → 2 demonstrates it exactly as well as 4 → 1.steady_heavy_influx_is_a_fixed_point— protects fixed-pointness while the cap scale walks up underneath. The fixed point is now 2.heavy_influx_lowers_threshold_and_promotes_next_cycle— its promotion half is untouched: the cohort was copied once in cycle 1, sonext_ageis 2 on cycle 2 and it still tenures on exactly the cycle the test names.quiet_cycles_restore_power_on_threshold_debounced— the debounced restore is asserted structurally (ends at power-on, rises at most one step per cycle) and survives from 2.Sabotage: drop the
.max(..)and phase 1 of the new test fails (the loop reports 1 with no evidence); drop the lock instead and phase 2 fails.cargo test -p perry-runtime --release --lib -- --test-threads=1: 3179 passed, 0 failed, 4 ignored.What the candidate binary actually does — the clamp captures 27 % of the promotion, not all of it
Everything above is measured on the instrumented branch with the threshold pinned. The relinked candidate makes the honest comparison possible: one binary built from this branch, three arms via
PERRY_GC_TENURING_SURVIVALS(unset = this change;=1= the pre-change state, since base sat at S=1 on 351 of 352 minors;=2= the positive control), 3300-char replies, 4 turns in one process,PERRY_GC_DIAG=1, macOS arm64:=1(pre-change equivalent)=2(positive control)Two things follow.
The earlier table reproduces on a second, independently built binary.
=1here (356 minors, 1055 MB) against the instrumented adaptive arm (352, 1057 MB), and=2here (380, 785 MB, 1059 MB copied) against the instrumented pinned arm (394, 792 MB, 1066 MB copied) — all within 2 %. The mortality finding stands.The convergence claim does not — and the size of the shortfall is now measured on a quiet host, which corrects this table's own arm. On the dev box (load 13–80) the clamp-only arm oscillated to S=3/4 and read −7 %. On the Mac mini (18 runs, load 1.87 mean) the clamp alone moves promotion by ~0 %: steady turns 2–4 at 3300, promoted is 803 MB at
=1, 798 MB with the clamp (−0.6 %), 601 MB at=2(−25.2 %); at 400 chars, 112 / 113 (+0.9 %) / 90. Repeat ranges overlap for clamp-vs-=1and are disjoint for=2.The mini's S-histogram says why, directly:
S=1: 318 minors carrying 1032 MB (98 %); S=2: 1 minor; S=4: 1 minor, with steady copied bytes 0.0/0.0/0.0. The clamp fires once in turn 1, the state returns to S=1, and stays. The −7 % is retracted; the shortfall is total, not partial.Why: the clamp re-arms a lock whose input is only a mortality reading at S ≤ 2
At S=1 nothing is copied, so
prev_copiedis 0 and the survival-rate lock's guard can never be satisfied — that is this PR's own argument. Removing that seal hands the lock back its guard, and the lock then fires 8–12 times per four-turn run (4->1,3->1), which is where the residual 85 % comes from. Not the occupancy rule: occupancy now correctly stops at 2 (4->2(occupancy), where the pre-change arm logged4->1(occupancy)). The clamp does exactly what it says; something else reaches 1.The lock's test is
where
survivor_live_bytesis accumulated in the copier as every live byte moved out of the from-survivor space this cycle, of any age (copying.rs:_ => self.stats.survivor_live_bytes += total), andprev_copiedis the previous cycle's whole intake into the to-survivor space.The survivor spaces are a strict semispace pair — the to-space is reset before each minor, everything copied goes into it, and the spaces flip — so the from-space at cycle N holds exactly what cycle N−1 copied. The ratio is therefore well-formed: numerator and denominator have the same scope, and it cannot exceed 1. What is wrong is not the arithmetic, it is which population it rates — and that is set by the very threshold this rule controls:
So the lock rates an aged, self-selected mixture and applies the conclusion — promote on first copy — to first-round cohorts. The clamp is what lets the debounced rise reach 3 and 4, so the change itself moves the lock into the regime where its input stops being about first-round mortality. This is the same defect shape the PR is about, one level up: the rule's own setting decides what its evidence measures, so it reads its setting back as evidence.
Draft — and now for a second reason
The original reason stands: +1054 MB of survivor copying against −265 MB of promotion and 5 fewer fulls on the pinned arm, and the box these counts came from ran at load 13–80, so its CPU cannot price it. The pricing runs on a quiet host are set up.
The second reason is the table above: as it stands this change buys −7 % of promotion, so it should not land on the strength of the −25 % the pinned arm shows. The two candidate next steps, in order:
566601b07, described below.Whichever is chosen, the falsifier registered for the quiet-host run is unchanged in shape but its expected values are not: promoted −25 % is already refuted for this change alone (−7 %), so what the quiet host is being asked is now whether −7 % of promotion, bought with 457–518 MB of copying, is CPU-neutral — turn CPU flat-to-better across turns 2–4, peak and settled RSS within the base spread, and the
[gc-major-pacing] shift=read. Both arms come from the one binary via the knob, so binary identity is not a variable.One caution on the full-reclaim numbers: three hosts sit in three regions of the same policy — this box 13–51 %, perrymaster's A arm 14 %, the regex lane's box 0.2–0.3 %. They are not merged.
Both commits are required — the clamp is not superseded
Asked explicitly, because the two commits look like alternative fixes to the same sentence. They are not: without the clamp, the cohort-scoped lock has nothing to rate.
Drop the
.max(OCCUPANCY_MIN_SURVIVALS)and the occupancy rule computes 1 on any influx abovedesired(integer division), so the loop sets S=1, so nothing is copied, soeden_copied_bytesis 0, soprev_cohort_copiedis 0, and the lock's guardprev_cohort_copied >= substantialis false forever — the same self-sealing this PR's first paragraph is about, unchanged by rescoping what the lock measures.So the two are complementary and sequential: the clamp is what lets a cohort exist; the lock fix is what stops the lock misreading it once it does. The second commit is only reachable as a defect because the first one removed the seal — which is why it was invisible until the clamp was measured. Keeping both.
Second commit — the lock rates one fresh cohort
566601b07. The copier now accounts the fresh half of every cycle:eden_copied_bytes— bytes copied out of Eden into the to-survivor space, excluding survivor residents being re-copied. One cohort's intake.survivor_first_round_live_bytes— live bytes leaving the from-survivor space whose stored survival age is 1, i.e. members of exactly the cohort the previous cycle'seden_copied_bytescounted.The per-object survival age is already in the header at copy time (
copied_survival_age), so the cohort is identifiable with no new per-object state, and the weaker "refuse to rate at S ≥ 3" fallback was not needed.retune_after_scavengekeeps its arity; its two lock parameters are redefined to these, so both sides of the ratio are scoped to one cohort at every threshold. Both counts are also on the[gc-copy-minor]diagnostic line, so first-round mortality is readable from any build rather than only from an instrumented branch — the measurement this policy is about should not require a custom binary.Tests, stated plainly: no existing expected value moved. All 1,069 gc tests pass unchanged, and that is itself the finding — nothing in the suite distinguished the two scopes, because they are equal on every heap whose survivor space holds a single generation, which is every heap at a threshold of 2 or below. So the premise gets its own real-heap test: two rooted objects introduced one cycle apart at the power-on threshold, asserting the two numbers agree while only one generation is resident and then differ once an aged resident joins it, with the aged object in the whole-space number and not in the cohort number. A test-only witness (
test_last_cohort_split) reports the pair the copier computed.Sabotage — run, and it fails exactly where it should
A sabotage branch (
sabotage/9851-unscoped-lock, one commit on this branch's head, marked never to merge) restores the whole-space accounting, so both fields keep their names and hold their pre-fix contents:eden_copied_bytes == copied_bytesandsurvivor_first_round_live_bytes == survivor_live_bytes.retune_after_scavengethen receives exactly the pre-fix pair.Both trees, same command, on a second machine:
The failure is the predicted assertion, at
gc/tests/copying/survival_and_malloc.rs:959:— the two numbers equal, which is the failure mode the test was written for, and the test's other two assertions still pass (cycle 2's "the numbers agree while one generation is resident", which is the control that stops the test being satisfiable by "always different", and
first_round_3 > 0, which survives because the sabotaged counter is a superset).What this does and does not establish. The reversion is of the accounting, not the wiring, and deliberately so: the test reads the copier's counters through
test_last_cohort_split(), so passing the whole-space values toretune_after_scavengewhile leaving the counters correct would not be caught. So this proves the test guards the quantity, not the plumbing — a narrower claim than "any reversion is caught", and worth stating plainly rather than letting a green/red pair imply more than it shows.The end-to-end counterpart is unchanged: the clamp-only binary is the unscoped arm, at 8–12 lock transitions per four-turn run in the table above.
cargo test -p perry-runtime --release --lib -- --test-threads=1: 3180 passed, 0 failed, 4 ignored.Counter run on the corrected binary — the prediction was wrong on two of three points
One binary, three arms via the knob, 3300 × 4 turns in one process (dev box; counters and states only, this box cannot price CPU):
=1pre-change=2controlreached 1 via the lock≈ 0The mechanism is nevertheless confirmed, by a control rather than by argument
The two mortality scopes, same binary, same instrument, one threshold apart:
=2(single generation)At
=2the two scopes are identical, exactly as the semispace argument requires when the space holds one generation, and they diverge by 15 points where the ladder climbs above 2. The old lock's input sat about one point from its own 90 % trigger; the cohort input is sixteen points clear of it. 26.1 % has now reproduced on three independent captures across two binaries.What this branch is, honestly
A correct fix to the statistic and an incomplete fix to the policy. With the lock no longer pulling the threshold down for the wrong reason, nothing pulls it down at all, and the debounced occupancy rise parks the loop at 4 — where it copies survivors three times to avoid promoting them once.
The old lock was wrong as a survival statistic but was acting as a crude cost signal, and removing it left the policy with no economics. What the policy needs is a price, not a constant: continue aging only while
mortality > copy_cost / promote_cost. With 26 % first-round mortality, aging pays only if a promoted byte costs more than ~2.8× a copied byte over its life.That ratio cannot be measured yet, for two reasons, and this PR should not pretend otherwise:
=1and=2on the quiet host, +815 MB of copying and −202 MB of promotion cost +28.4 s. At 0.3 / 1.0 / 3.0 ns per byte, copying explains 0.9 % / 2.9 % / 8.6 % of that. ≥91 % is gc: the budgeted full step is starved by the shape-descriptor prune memmove — +59.5% peak RSS at tenuring threshold 2 #9871's stall, not copying. Pricing now would blame copying for a defect and conclude "S=1 was right all along" for the wrong reason.[gc-time]) is emitted from its exit block — which this campaign's driver was destroying withSIGKILLon every run. Fixed in the driver; the timing capture is running now.So this PR stays draft, and the tenuring question stays open rather than closed. It is blocked on #9871 for its memory column and on the cost ratio for its policy rule — and the rule it is heading for replaces the 90 % constant with a measured ratio.
The bar this has to clear is now memory, not just promotion
Pinned
=2is what "S floor 2" converges to, and on the quiet host it is not an acceptable landing point, three repeats disjoint at 3300 against=1:=1=2So if the corrected lock's steady state is S=2 and it inherits this profile, the branch is dead on the directive — CPU and memory both regress — and this PR should say so rather than land. That is the actual open question; the promotion number alone no longer decides it.
Where that cost comes from — counted, not narrated
Slicing the mini's diags by the driver's per-turn
diag_offset(whole-run greps return turn 4 silently — theS=<n>:line shape repeats per section):[gc-step][gc-general-reclaim]=1=2At
=2the full-cycle stepper and the general reclaimer effectively stop during turns 2–3 and release zero blocks, while RSS climbs ~1 GB. Turn 4 resumes (16 steps, 16 reclaims, 22 released) but the pages are already committed. The deferred work then lands in the idle window: footprint 1679 → 614 MB at 23–25 s of CPU, against 0.9–4.5 s on arms that had been releasing all along.Peak RSS and idle CPU are therefore one phenomenon, and it is deferred release, not extra garbage. The proof is that the ~800 MB gap collapses after the idle window: settled footprint is 609–618 MB at
=2against 500–573 MB at=1.What the steps that do run report is consistent with why they stop: at
=2, turns 2–3 steps readpct=0%(sweep_freed803 KB and 821 KB againstpre_in_use209 MB and 480 MB), where=1's turn-3 step frees 19 MB (pct=5%) and the clamp-only arm's frees 26 MB (pct=11%). A full sweeps the old gen; at=2the freshly-dead bytes are one aging round behind, in the survivor space, so the full finds nothing and is scored unproductive. I am not asserting the pacing backoff as the cause — this binary emits no[gc-major-pacing]line on this path, so that link is unmeasured here. It is the next measurement, not a finding.That makes the cost a separate defect from tenuring, now filed as #9871: a full collection is priced by what the old generation yields, so raising the tenuring threshold makes every full read unproductive and stalls page release. Falsifier there: pin the step cadence at
=2and peak RSS should fall toward the=1arm with idle CPU following, at the price of the fulls' own CPU — and if forcing the cadence does not move peak RSS, deferral is not the cause, #9871 is wrong and should be closed rather than reshaped.#9871 is the same defect class as this PR's two subjects and as the idle-reclaimer freshness re-arm: a decision priced by a yield that the decision itself delays.
The memory column is void until it is re-measured on the landing base
Every arm in this PR was measured on a pre-#9857 binary, and on that binary the workload is dominated by an artefact #9857 removes. Family counters name it: a burst of ~512 k single-key objects (the only key is
"0"), twice per turn, and when a minor lands relative to that burst decides the turn-CPU mode, the idle-window burn, and very probably the peak RSS itself — at=1the burst mostly dies in the nursery unless the coincidence carries it to promotion (one 2279 MB outlier against four draws at 1382–1495), while at=2aging keeps it alive every time (2231–2242, tight).So the memory numbers below are not this change's memory numbers; they are that burst's. The measurement this PR's memory column needs is
=1/ unset /=2on the I3 landing base with this commit cherry-picked, and it is queued. Until it lands, no memory claim here is load-bearing in either direction — including the ones that look bad for this branch.The promotion and lock-transition counters are less exposed (they are ratios over the same workload in both arms) but they are not clean either, for the same reason.
#9871 is closed, not deferred. It was filed off these arms as "a full is priced by old-gen yield, so a higher threshold stalls page release", rewritten once when a profile showed the idle half was actually #9881's
prune_dead_shape_keys→remove_descriptor_indexed_under→memmove, and then closed when its own registered falsifier fired against it: on the quiet host, restoring the full-cycle cadence completely (arm C, every repeat) raised peak RSS 3 % rather than lowering it, and peak RSS turned out to track collection volume rather than blocks released.One open question, recorded here rather than as an issue because it belongs to whatever tenuring work survives: why does
PERRY_GC_MAJOR_PACING_GROWTH=1cut promotion by 20 % and minors by 24 %? That arm changes what is live at the moment the full runs, which is a tenuring-adjacent effect, not a pacing one.This PR's memory outcome is a dependency, not a prediction
This branch converges on a survivor threshold of 2, so until #9871 is fixed it inherits the
=2profile above — +59.5 % peak RSS and ~24 s of idle CPU — regardless of what the tenuring logic does, because those costs come from pages not being released for two turns and not from the copying. On the directive's terms (neither CPU nor memory may regress) that means:A negative result on the turn-1 split
At
=2, turn 1 costs 34.62 / 35.89 / 17.05 s across three repeats — but the three runs' turn-1 counters are identical: 62 minors, 251 MB eden, 251 MB copied, 187 MB promoted, 17[gc-step], 4[gc-general-reclaim], 0 released. No GC counter in the diag distinguishes the 17 s run from the 35 s ones, so the split is not survivor accounting.It has since been explained elsewhere, and it is not this PR's subject — recorded here so nobody re-chases it. A perf diff of a 13 s against an 18 s draw puts 113 % of the delta in one symbol:
__memmove_avx512_unaligned_ermsundershapes::remove_descriptor_indexed_under←prune_dead_owner_side_tables_post_trace— same minors, same removals, 12× the bytes moved per removal, decided by where the dying owners' descriptors sit in the indexed list. Identical counters with 2× the time is a per-unit cost, which is exactly what an O(list) removal looks like. The gc-overhead lane is fixing the removal; #9857 removes cc's owners entirely (3300 → 6.9 s, unimodal).https://claude.ai/code/session_014knX724SYDogwzsXybCGxp
Test update for the power-on change (
e8dc4e402)d07b86071moved the adaptive threshold's power-on value from the ceiling (4) to the occupancy floor (2). Fourteen runtime tests assumed the old power-on value as a precondition and failed on the campaign's gate host. This commit adds a#[cfg(test)]-only scoped pin,set_survivals_for_test(n), and updates the tests: those that exercise a mechanism at a particular promotion age (promote on the fourth survival, a parent young for three survivals, a holder tracked across three moving minors, the old-to-young edge across minors, the promotable census walk, old-page accounting, the survivor-space-vs-cohort numbers, the two hook-dispatch handle tests) pin S=4 explicitly; those about power-on itself assert the new contract (start at the floor 2, a debounced rise to 3 after a measured round, a fully live sweep census may seed S=1, a 10 %-live eden keeps S=2). Each still fails under the sabotage named in the campaign report. Suite: 3241 passed, 0 failed, 4 ignored; default-feature compiler build green. No non-test code changed.Measured (perrymaster TN4, main
504e180d0+ this branch at2248fba56, runtime relinked on main's cache; 4-turn graceful runs, quiet box, every row stamped with foreign CPU / swap / clock)Gate: default build, archive feature set, runtime suite 3234 passed / 0 failed / 4 ignored with all 15 named tests green.
400: adaptive t1 1.07 vs pinned S=1 0.71 / S=2 0.89; turns 2–4 equal within noise.
Reading: the power-on fix changes nothing observable. The adaptive loop still ends at S=1 after a brief S=2 (two transitions inside turn 1), the turn-1 excursion is still there (+0.3…0.4 s at 3300 vs S=2, with the extra full collections in turns 1–2), turns 2–4 are equal across arms, and S=2 pinned remains the best steady total (−5…6 % over four turns vs adaptive) with the lowest peak and settled RSS and −30 % steady promotion. The excursion is the survival-rate lock's decision to promote on first copy, which the ceiling fix does not touch. This PR is correct as a startup-invariant fix but is not a CPU win by itself; the follow-up is an evidence-gated lock (see the campaign's
DESIGN_tenuring_steady_S2.md).