diff --git a/cc-perf-campaign/codex/REPORT_nursery_cap_evidence.md b/cc-perf-campaign/codex/REPORT_nursery_cap_evidence.md new file mode 100644 index 0000000000..2dc2bfc9cd --- /dev/null +++ b/cc-perf-campaign/codex/REPORT_nursery_cap_evidence.md @@ -0,0 +1,271 @@ +# Nursery-cap RSS evidence report + +Code SHA: `c18648332347cd4314e8d7a3255aa1223ab5e44a` + +Branch: `perf/nursery-cap-rss-evidence`, stacked directly on +`a771c33fce419fc04c1a811fd797f63e40cd8cbe` (#9949). + +## Code facts + +### Object denomination ends with the first completed copying minor + +The exact regime signal is the new per-thread `COPYING_MINOR_COMPLETED` cell in +`gc/tenuring.rs`. `note_surviving_object_census` sets it before either +zero-census early return. That call is made only after a copying attempt has +finished evacuation and immediately before the completed minor is retuned and +reported. It therefore flips on the first completed copying minor even if that +minor moved zero objects. + +This is deliberately not `OBJECT_CENSUS_SEEDED`: #8122's allocation walk sets +that before the first minor and must still denominate that first trace. It is +also deliberately not `SURVIVOR_ROUND_MEASURED`: that signal cannot become +true until a cohort admitted by one minor returns on a later minor, so it is +one minor too late. + +Before `COPYING_MINOR_COMPLETED`, `influx_driven_nursery_cap_bytes` retains the +existing 72 B reference, integer per-mille arithmetic, 500 per-mille floor and +one-sided 1000 per-mille clamp. Afterwards it returns exactly +`base * NURSERY_CAP_SCALE`. `note_surviving_object_census` still updates the +mean. Its diagnostic says `applied=false`; the pre-minor allocation-census +diagnostic says `applied=true`. + +There is no second production consumer of +`nursery_cap_object_scale_permille`: outside the cap's pre-copy branch, its +remaining calls are diagnostics and tests. There is likewise no production +reset of `NURSERY_CAP_SCALE`; its only reset is inside `reset_for_test` under +`cfg(test)`. The idle right-size does not reset it. + +The #7929 doc comment measured the cost of tracing/moving many small objects on +`deeplist` and `retain1`. Rule 1 preserves that measurement for the first +tracing band, including #8122's allocation seed. It deliberately does not +claim that survivor-object size prices the fixed scanner/remembered-set cost of +later copying minors. `deeplist`, `retain1`, and `tree_wide` are consequently +the probes most likely to expose a regression if that distinction is wrong. + +### Cap-due evidence is captured before evacuation + +`copying.rs::run_copied_minor_attempt` captures a `NurseryCapRating` before +eligibility checks or heap mutation can rewrite from-space. The snapshot holds +the current `copying_from_space_in_use_bytes`, the effective cap, and the +verdict returned by `policy::young_scavenge_cap_due()`. This is the same exact +comparison used to arm a cap collection; it does not infer anything from the +diagnostic trigger label. The label can still be `ArenaBytes` because the +effective arena trigger folds in the young cap. + +The completed attempt passes that snapshot as the fourth argument to +`retune_after_scavenge`, which passes it to `retune_nursery_cap_scale`. +Speculative attempts may take a snapshot, but only the attempt which completes +reaches the retune. A false verdict returns before touching either +`NURSERY_CAP_SCALE` or `CAP_GROW_STREAK` and emits one diagnostic: + +```text +[gc-tenuring] nursery cap scale: not rated (from_space=N cap=N) +``` + +An influx transition now carries all attribution: + +```text +[gc-tenuring] nursery cap scale 1x -> 2x (why=influx cap_due=true from_space=N cap=N eden_live_bytes=N) +``` + +The 4% threshold, two-rated-minor debounce, one-step `x2` growth, and `x4` +ceiling are unchanged. The survival-rate lock, occupancy rule, and +tenured-proportional cap term are unchanged. + +### No readable RSS-pressure predicate exists on this base + +Rule 3's shrink path is absent, as the design requires when no readable +predicate exists. `js_gc_memory_pressure` lowers `GC_NEXT_TRIGGER_BYTES` to +`arena_total + 1 MiB`, but neither that write nor the cell records its +provenance. `GC_TRIGGER_ARMED` is also set by ordinary re-arms and parse bumps. +Warning pressure may collect synchronously and have the normal finisher +overwrite the lowered trigger before any rated minor. Critical pressure sets +`GC_OLD_RECLAIM_PENDING`, but ordinary old-generation reclaim pressure sets +the same cell. The RSS evacuation threshold is an evacuation policy input, not +the arena-trigger clamp's active-state predicate. + +Using any of those as “pressure active” would invent or conflate a signal. +Therefore this patch ships rules 1, 2 and 4, retains idle page release, and does +not add `nursery_cap_scale_shrinks_under_memory_pressure` or a test override +for a predicate that production cannot read. + +## Fixture A replay: TN5 unset + +`old_cap` is the captured trigger cap. `new_cap` is the cap computed at the +same row by this policy. The first row retains the allocation-census mean of +66 B (`916` per mille); rows 2-4 are the existing ramp. From row 5 onward the +band is 64 MiB. Every cap-due row after the ramp is therefore exactly +67,108,864 B; the old 32-42 MiB rows disappear. + +| row | from-space B | old cap B | new cap B | mean after B | cap due | +|---:|---:|---:|---:|---:|:---:| +| 1 | 15,728,712 | 15,367,929 | 15,367,929 | 77 | yes | +| 2 | 17,407,296 | 16,777,216 | 16,777,216 | 84 | yes | +| 3 | 33,624,624 | 33,554,432 | 33,554,432 | 133 | yes | +| 4 | 34,829,760 | 33,554,432 | 33,554,432 | 84 | yes | +| 5 | 39,313,552 | 67,108,864 | 67,108,864 | 118 | no | +| 6 | 67,407,992 | 67,108,864 | 67,108,864 | 110 | yes | +| 7 | 67,230,456 | 67,108,864 | 67,108,864 | 61 | yes | +| 8 | 57,803,984 | 56,841,207 | 67,108,864 | 40 | yes | +| 9 | 37,882,104 | 37,245,419 | 67,108,864 | 47 | yes | +| 10 | 44,762,032 | 43,754,979 | 67,108,864 | 43 | yes | +| 11 | 40,614,704 | 40,063,991 | 67,108,864 | 36 | yes | +| 12 | 34,204,000 | 33,554,432 | 67,108,864 | 38 | yes | +| 13 | 35,524,488 | 35,366,371 | 67,108,864 | 80 | yes | +| 14 | 67,572,776 | 67,108,864 | 67,108,864 | 63 | yes | +| 15 | 59,188,240 | 58,720,256 | 67,108,864 | 45 | yes | +| 16 | 42,913,584 | 41,943,040 | 67,108,864 | 39 | yes | +| 17 | 36,981,552 | 36,305,895 | 67,108,864 | 84 | yes | +| 18 | 56,662,480 | 67,108,864 | 67,108,864 | 68 | no | +| 19 | 63,730,280 | 63,350,767 | 67,108,864 | 37 | yes | +| 20 | 34,583,744 | 34,426,847 | 67,108,864 | 39 | yes | +| 21 | 36,427,200 | 36,305,895 | 67,108,864 | 40 | yes | +| 22 | 37,347,336 | 37,245,419 | 67,108,864 | 73 | yes | +| 23 | 67,358,392 | 67,108,864 | 67,108,864 | 62 | yes | +| 24 | 58,724,472 | 57,780,731 | 67,108,864 | 38 | yes | +| 25 | 35,510,472 | 35,366,371 | 67,108,864 | 39 | yes | + +## Fixture B replay: NS3 n16 + +The fixture code contains all 26 `(from_space, cap, eden_live)` rows. Per the +fixture contract, the four transition values are copied exactly and other +`eden_live` values are `survival_permille * from_space / 1000`. Row 9 is the +only non-cap-due row and is skipped without clearing the accumulated state. + +| row | eden live B | cap due | recorded scale after row | new scale after row | +|---:|---:|:---:|---:|---:| +| 1 | 13,200,201 | yes | 1 | 1 | +| 2 | 3,208,856 | yes | 2 | 2 | +| 3 | 25,294,605 | yes | 2 | 2 | +| 4 | 9,684,456 | yes | 4 | 4 | +| 5 | 6,815,672 | yes | 4 | 4 | +| 6 | 542,697 | yes | 4 | 4 | +| 7 | 46,176 | yes | 2 | 4 | +| 8 | 34,602 | yes | 2 | 4 | +| 9 | 2,803,437 | no | 2 | 4 | +| 10 | 2,590,608 | yes | 2 | 4 | +| 11 | 32,928 | yes | 1 | 4 | +| 12 | 41,942 | yes | 1 | 4 | +| 13 | 125,733 | yes | 1 | 4 | +| 14 | 3,213,027 | yes | 1 | 4 | +| 15 | 3,102,415 | yes | 1 | 4 | +| 16 | 23,068 | yes | 1 | 4 | +| 17 | 23,068 | yes | 1 | 4 | +| 18 | 23,068 | yes | 1 | 4 | +| 19 | 459,049 | yes | 1 | 4 | +| 20 | 3,057,456 | yes | 1 | 4 | +| 21 | 3,155,564 | yes | 1 | 4 | +| 22 | 20,971 | yes | 1 | 4 | +| 23 | 20,971 | yes | 1 | 4 | +| 24 | 21,006 | yes | 1 | 4 | +| 25 | 44,094 | yes | 1 | 4 | +| 26 | 315,351 | yes | 1 | 4 | + +Thus growth remains exactly at rows 2 and 4. The recorded mortality +transitions at rows 7 and 11 disappear. + +## Tests and sabotage outcomes + +All five shipped named tests ran and passed. The sabotage outcomes below are +the discriminating assertion each test produces: + +- `steady_state_cap_is_not_object_denominated_after_first_copying_minor`: + letting the survivor mean feed the steady cap makes row 8 return 56,841,207 + B instead of 67,108,864 B, followed by the captured 32-42 MiB dips; the + suffix assertion fails. +- `first_minor_keeps_the_object_denomination`: deleting the pre-copy regime + test returns the raw 16,777,216 B band instead of 9,311,354 B; the exact-cap + assertion fails. +- `nursery_cap_scale_does_not_shrink_on_low_mortality`: restoring + `< cap / 100` first changes row 7 from scale 4 to 2, so the all-4 suffix + assertion fails. With the specified derived `eden_live` rows and cap-due + qualification, later high-influx pairs can re-grow it before later low + pairs shrink it again; the final row is 1, not 4. +- `nursery_cap_scale_still_grows_on_influx`: deleting or re-timing the grow + branch makes the exact 26-row vector first differ at row 2 or row 4. +- `non_cap_due_minor_does_not_rate_the_scale`: deleting the early cap-due + return makes its two forced high-influx minors grow the scale from 1 to 2; + both the scale and zero-streak assertions discriminate. +- `nursery_cap_scale_shrinks_under_memory_pressure`: not shipped or run, + because there is no production-readable active pressure-clamp predicate to + override without inventing a new signal. + +The #9949 startup-window, lock-entry, mortality-reset, and sweep-seed tests are +unchanged and green in both requested test gates. Existing object-denomination +integration coverage was updated only for the new completed-minor regime: the +census mean remains observable while the cap returns to its byte band. + +## Design contradictions and audit notes + +The requested “40 B means 500 per mille, bit-identical to today” assertion is +not compatible with the untouched arithmetic: + +```text +40 * 1000 / 72 = 555 per mille +16,777,216 * 555 / 1000 = 9,311,354 B +``` + +The 500-per-mille floor begins at 36 B and below. The first focused run exposed +this mismatch (and the old integration assertion that a completed minor still +denominated the cap); both tests now assert the preserved rule. Changing 40 B +to 500 per mille would have re-tuned the first-minor regime forbidden by rule +1. + +The design's shorthand that restoring mortality shrink makes “rows 7-8 shrink +it” describes the original diagnostic trace. Under the separately specified +test transcription (derive non-transition `eden_live` from total survival) and +the new cap-due filter, the first shrink is at row 7; row 8 starts the next +debounce, and subsequent derived high-influx rows can reset or re-grow it. The +test still fails at the first changed row and at the final scale. + +No other denomination consumer or scale reset was found. No local cc run was +made. The three #7929-sensitive probes may regress if survivor denomination +was in fact pricing steady copying work there; that is precisely the +perrymaster falsifier, not a reason to hide the policy distinction in a unit +test. + +## Gates + +The root filesystem reported 15 GB available immediately before every Cargo +command, above the binding 12 GB floor. All Cargo commands ran through the +shared build lock with `-j4`; this sandbox rejected `nice -n 19` with +`setpriority: Operation not permitted`, so the commands continued at the +inherited priority. + +- `cargo test -p perry-runtime --release --lib -j4 -- --test-threads=1 tenuring`: + green, 35 passed, 0 failed, 3,241 filtered out. +- `cargo test -p perry-runtime --release --lib -j4 -- --test-threads=1`: + green, 3,272 passed, 0 failed, 4 ignored. +- `cargo build --release -p perry-runtime --features wasm-host -j4`: green. +- `rustfmt --edition 2021 --check` on all four touched Rust files: green. +- `git diff --check`: green. +- Every touched file is below 2,000 lines (largest: `copying.rs`, 1,998). + +## Perrymaster falsifiers and predicted numbers + +Stage NC should compare #9949 (`c18648332^`) with this code SHA, relinking the +runtime only on TN5's cache: alternating two runs per arm, four 3,300-character +turns with diagnostics; one 400-character turn; then the gc-ratchet ladder. + +Predictions: + +- By minor 6 the arm is at 67,108,864 B, and every later cap-due trigger stays + there. Only the `1x -> 2x` and `2x -> 4x` ramp transitions appear. Survivor + means keep changing, with `applied=false`, without moving the band. +- Four-turn copying-minor count falls from 25 to about 20-21. At the measured + 45-73 ms fixed pause per avoided minor, five avoided minors predict roughly + 0.25-0.35 s, or 2-4%, off the 8.87 s total. Minors falling without CPU + falling falsifies the fixed-cost premise and must be reported, not averaged + away. +- Peak RSS stays within 0-3% because the 64 MiB nursery was already reserved + by minor 6; settled RSS after 120 seconds is equal; promoted bytes in turns + 2-4 stay within 2 MiB of control. +- The 400-character turn is flat or faster. +- Every gc-ratchet probe remains at or below baseline instructions and within + its RSS tolerance, especially `deeplist`, `retain1`, and `tree_wide`. A + regression in those three means steady denomination was still pricing the + work #7929 measured; the evidence-driven fallback is to weight it by the + preceding minor's copy-work share rather than restore it globally. + +Kill the change for any probe or cc row above +10% max RSS, or for fewer minors +combined with higher CPU. diff --git a/cc-perf-campaign/codex/REPORT_tenuring_evidence_lock.md b/cc-perf-campaign/codex/REPORT_tenuring_evidence_lock.md new file mode 100644 index 0000000000..3debf9aed5 --- /dev/null +++ b/cc-perf-campaign/codex/REPORT_tenuring_evidence_lock.md @@ -0,0 +1,152 @@ +# Tenuring steady-evidence lock report + +Code SHA: `d229d7d715cee4725776a83fa210ac5727ce1862` + +Branch: `perf/tenuring-evidence-lock` + +## Lock map before and after this change + +The copying collector builds the cohort-scoped signal in +`crates/perry-runtime/src/gc/copying.rs:621-640`: `eden_copied_bytes` counts +fresh Eden objects copied into the survivor space, while +`survivor_first_round_live_bytes` counts age-1 objects from that same cohort +which return alive on the next copying minor. The completed minor passes those +values to `retune_after_scavenge` at `copying.rs:1895-1909`. + +`crates/perry-runtime/src/gc/tenuring.rs:600-639` swaps the current +`eden_copied_bytes` into `PREV_COPIED_BYTES`. A rated round is, as before, a +cycle with `prev_cohort_copied > 0`: a previous copying minor admitted a fresh +cohort, so this minor can divide its `first_round_live_bytes` by that cohort's +intake. Before this change that single condition was also sufficient to make +the first startup cohort eligible for the lock. That is why TN4's startup +cohort qualified: the first minor copied it, the second minor returned it +nearly intact, and the code had no process-phase evidence. + +The entry decision is now at `tenuring.rs:676-694`. A rated round contributes +only when it is post-startup, its previous fresh intake is at least the existing +`desired / 4` substantial-volume bar, and its survival is at least the existing +90% bar. Any rated round that fails those conditions resets +`PROMOTE_LOCK_STREAK`. `PROMOTE_LOCK` latches only when that streak reaches +three. + +Startup is the first two **rated survivor cohorts** +(`tenuring.rs:195-205, 613-627`), not a wall-clock duration and not the +allocation-census flag. This uses the lock's own threshold-invariant evidence +and cannot re-pace collection. The allocation census is deliberately not the +marker: it seeds halfway to the first nursery cap, before the first copying +minor, so it was already true for precisely the TN4 startup cohort that must be +excluded. Two excluded ratings cover the observed two-minor startup phase; K=3 +is the smallest steady window that rejects a one-off or two-cycle phase +boundary while still reaching S=1 within five rated cohorts on a truly +non-dying workload (two startup plus three deciding rounds). + +The unlock path remains semantically unchanged at `tenuring.rs:642-664`: +while locked, substantial Eden influx holds S=1; two consecutive cycles below +`desired / 4` clear the lock and resume at S=2. The only added bookkeeping is +clearing the entry streak when unlocking, so stale entry evidence cannot be +reused after a later phase change. + +`seed_promote_lock_from_sweep` remains the same two-condition census rule +(occupancy computes S=1 and Eden survival is at least 90%), but +`tenuring.rs:838-890` now refuses even a qualifying census while fewer than two +survivor cohorts have been rated. After startup it still latches immediately +from those two conditions and hands off to the unchanged unlock path. + +## Prices recorded, not yet used to decide + +The 90% constant was not replaced. TN4 does not contain the two aligned unit +prices needed to validate the structural inequality, so changing the decision +rule now would merely substitute another assumption. + +Under `PERRY_GC_DIAG` only, each copying minor accumulates total copied and +promoted bytes (`copying.rs:1817-1824`, `instruments.rs:350-371`). The existing +cumulative minor pause is the copy-side numerator; cumulative +`step_us + remark_us` is the promote-side numerator +(`instruments.rs:391-402`). TN5 can therefore compute: + +```text +copy_cost = copy_pause_us / tenuring_copied_bytes +promote_cost = promote_us / tenuring_promoted_bytes +keep aging while mortality > copy_cost / promote_cost +``` + +The new byte atomics are not touched when diagnostics are off. + +## Diagnostic format + +Every adaptive S transition now has this format (`tenuring.rs:910-939`): + +```text +[gc-tenuring] survivals FROM -> TO (REASON, eden_live_bytes=N desired=N rounds_rated=N streak=N survival_permille=N copied_bytes=N startup=true|false copy_pause_us=N tenuring_copied_bytes=N promote_us=N tenuring_promoted_bytes=N) +``` + +For a sweep-seed transition, `copied_bytes` is the live Eden cohort the next +minor would otherwise copy; the adjacent `sweep-seed` line prints live/dead, +the two-condition verdict, and `startup=`. The process-exit block is +(`gc/mod.rs:1410-1419`): + +```text +[gc-time] wall_us=N step_us=N remark_us=N minor_us=N full_sync_us=N share_permille=N copy_pause_us=N tenuring_copied_bytes=N promote_us=N tenuring_promoted_bytes=N +``` + +## Tests and explicit sabotages + +- `startup_shaped_survivors_do_not_contribute_to_the_lock_streak`: remove the + `!startup` entry conjunct; the two startup ratings plus the first steady + rating reach K and latch S=1. +- `k_steady_fully_surviving_rounds_latch_promote_on_first_copy`: raise K or + stop advancing the streak; the exact-K final round fails to latch. +- `mortality_inside_the_steady_window_resets_the_lock_streak`: retain the + streak on a below-bar rated round; the final surviving cohort becomes the + cumulative Kth and latches S=1. +- `sweep_seed_cannot_latch_from_a_startup_census`: remove the sweep startup + conjunct; a census satisfying both original conditions immediately latches. +- Existing pinned-S coverage and + `occupancy_may_not_claim_the_ceiling_before_any_round_is_measured` were left + in place. They were not executable locally because of the disk gate below. + +## Gates + +Not run (zero tests/builds executed): + +- `cargo test -p perry-runtime --release --lib -- --test-threads=1` +- `cargo build --release -p perry-runtime --features wasm-host` +- `cargo build --release -p perry` + +Reason: `df -g /` was below the binding 12 GB floor before every possible +Cargo invocation. It was polled every 60 seconds for 30 minutes, declining +from 2 GB to 0 GB free. No Cargo command was invoked. + +Non-Cargo checks run and green: + +- `rustfmt --check` on all four changed Rust files +- `git diff --check` +- `python3 scripts/gc_runtime_root_holders.py` +- `python3 scripts/gc_runtime_root_holders.py --self-test` +- `scripts/check_file_size.sh` + +The new pointer-free TLS counters have explicit custody verdicts. The +`PASS1_MARKED` non-moving-snapshot pin was re-audited and refreshed because +`gc/mod.rs` changed only in the process-exit diagnostic path, outside its +mark-complete to sweep-entry window. + +## Exact perrymaster request and falsifiable predictions + +Request TN on **main6 + this branch**: arms `unset`, +`PERRY_GC_TENURING_SURVIVALS=1`, and +`PERRY_GC_TENURING_SURVIVALS=2`; workloads 3300 and 400 characters; two rounds +per arm/workload; four turns per process with graceful exit. Capture complete +`PERRY_GC_DIAG=1` output and run both `tn_summary` and `tenure_an`, including S +history, transition evidence, promoted bytes by turn, peak/settled RSS, and the +four cumulative price counters. + +Predictions to falsify: + +- On cc, `unset` shows no startup latch: S-history is S2 only, or any latch is + after a post-startup three-round steady window. +- Turn 1 equals pinned S2 within noise; turns 2-4 equal pinned S2. +- Promoted bytes in turns 2-4 are approximately 12 MB. +- Peak and settled RSS match the pinned-S2 arm (within the allowed 1-10% RSS + range). +- `k_steady_fully_surviving_rounds_latch_promote_on_first_copy` is the kill + condition proving the adaptive S1 lock still exists for non-dying workloads. diff --git a/changelog.d/9851-lock-rates-one-cohort.md b/changelog.d/9851-lock-rates-one-cohort.md new file mode 100644 index 0000000000..deb0ad2d56 --- /dev/null +++ b/changelog.d/9851-lock-rates-one-cohort.md @@ -0,0 +1,49 @@ +### Fixed + +- **The tenuring survival-rate lock now rates one fresh cohort, not the whole + survivor space — a well-formed ratio that stopped describing what it is named + after as soon as the threshold it sets rose above 2.** + + The lock exists to answer "did an aging round filter anything?" and, when the + answer is no, to promote on first copy. It tested + + ``` + prev_copied >= substantial && survivor_live_bytes * 10 >= prev_copied * 9 + ``` + + where `survivor_live_bytes` is every live byte leaving the from-survivor space + this cycle, of any age, and `prev_copied` is the previous cycle's whole intake + into that space. Those two scopes match — the survivor spaces are a strict + semispace pair, so the from-space holds exactly what the last cycle copied — + and the ratio cannot exceed 1. **The defect is not the arithmetic; it is which + population the ratio rates, and that is chosen by the very threshold the lock + sets.** At a threshold of 2 the space holds one fresh cohort and the ratio is + one aging round's survival. At 3 or 4 it also holds objects that have already + survived a round and are therefore selected for longevity, so the aggregate + clears the 90 % bar while a fresh cohort does not. The rule reads its own + setting back as evidence. + + This was invisible while the occupancy rule sealed the loop at S=1, because + there `copied_bytes` is 0 and the lock's guard can never be satisfied. + Removing that seal handed the lock its guard back, and it became the dominant + route to promote-on-first-copy. + + Measured on the compiled claude-code TUI, one binary, three arms via + `PERRY_GC_TENURING_SURVIVALS`, 3300-character replies, 4 turns in one process: + + | arm | minors | promoted | S=1 share of promotion | reached 1 via the lock | + |---|---|---|---|---| + | `=1` (pre-clamp equivalent) | 356 | 1055 MB | 100 % | - | + | occupancy clamp only | 368 / 384 | 982 / 980 MB | 85 % / 84 % | **8 / 12** | + | `=2` (positive control) | 380 | 785 MB | 0 % | n/a | + + The copier now also accounts the fresh half of each cycle: `eden_copied_bytes` + (bytes copied out of *Eden* into the to-survivor space, no re-copies) and + `survivor_first_round_live_bytes` (live bytes leaving the from-survivor space + whose stored survival age is 1, i.e. members of exactly the cohort the + previous cycle's `eden_copied_bytes` counted). The lock rates those two. Both + are on the `[gc-copy-minor]` diagnostic line, so first-round mortality is + readable from any build rather than only from an instrumented one. + + Reaching 1 still belongs to the paths that measure mortality; what changes is + that the measurement is now of one aging round at every threshold. diff --git a/changelog.d/9851-occupancy-may-not-promote-on-first-copy.md b/changelog.d/9851-occupancy-may-not-promote-on-first-copy.md new file mode 100644 index 0000000000..40d6d0a057 --- /dev/null +++ b/changelog.d/9851-occupancy-may-not-promote-on-first-copy.md @@ -0,0 +1,67 @@ +### Fixed + +- **The adaptive tenuring loop's occupancy rule can no longer conclude + "promote on first copy" — a claim about lifetime that it has no evidence + for, and which destroys the evidence that would refute it.** + + `retune_after_scavenge` picks a survival threshold from + `S = 1 + desired / influx`: the largest S whose projected survivor occupancy + `(S-1) x influx` fits the desired survivor size. With integer division, any + influx above `desired` yields exactly **1** — there is no rung at 2 or 3. + Measured on the compiled claude-code TUI, the first drop reads + `eden_live_bytes=12075344` against `desired=1048576`. + + S=1 does not reduce the surviving data; it relocates it, from the survivor + space — where the next minor re-examines it for free — to the old generation, + which only a full collection can reclaim. The occupancy formula has no term + for that. And S=1 is **self-sealing**: nothing is copied, so `copied_bytes` is + 0, so next cycle `prev_copied` is 0, so the survival-rate lock's guard + (`prev_copied >= substantial`) is false forever. Both remaining exits — the + occupancy recompute and `PROMOTE_LOCK`'s unlock — are *quiet-influx* exits, + which say nothing about lifetime. + + Measured, 4 streamed turns in one process, both arms from one binary via the + diagnostic knob `PERRY_GC_TENURING_SURVIVALS`, 3300-character replies: + + | | adaptive | pinned S=2 | + |---|---|---| + | minors at S=1 | **351 of 352**, carrying 100 % of promotion | 0 | + | threshold transitions in the whole run | **1** | 7 | + | survivor-round mortality samples | **1** | **393** | + | median mortality | **0.9 %** | **26.1 %** | + | ...in steady turns 2 / 3 / 4 | not measurable | 26.1 / 26.1 / 26.1 % | + | promoted | 1057 MB | 792 MB | + + The loop takes its one and only mortality measurement on the **first minor of + the process** — before any steady state, when the cohort really is immortal — + reads 99.1 % survival, drops to 1, and can never sample again. In steady + state an aging round filters about **a quarter** of each cohort. + + The occupancy rule now stops at the lowest threshold that still *produces* + that measurement. That value is 2 by construction, not by tuning: at S=1 + nothing enters the survivor space, at S=2 exactly one cohort does. The + arithmetic is untouched — `compute_target_survivals` still computes 1, and + its test asserts so byte-identically; only what the loop may do with the + result changes. + + **Reaching 1 still belongs to the two paths that measure mortality** — the + survival-rate lock (a substantial cohort of which >= 90 % came back alive) + and the sweep seed (the mark-sweep's own Eden live/dead split). Both are + untouched, so the rule is self-limiting: on a workload whose cohort genuinely + does not die, the lock fires after one cohort's copy and takes the loop back + to 1. + + On claude-code it **does** fire, 8-12 times per four-turn run, and the + companion entry below is why: once the clamp lets the ladder climb past 2 the + lock is rating a population its own threshold selected. An earlier version of + this entry claimed the opposite ("5 of 358 substantial cohorts sit under the + lock's threshold, so the clamp holds rather than oscillating"); that figure + was measured with the threshold *pinned*, where every cohort the lock can + rate is a first-round cohort, and it does not describe the rule running. + + Two existing tests change their expected value from 1 to 2 and keep their + names, structure and invariants: `drops_immediately_and_rises_debounced` + protects the *asymmetric response* (immediate drop, debounced rise), which + 4 -> 2 demonstrates exactly as well as 4 -> 1; and + `steady_heavy_influx_is_a_fixed_point` protects *fixed-pointness*, which is + unchanged with 2 as the fixed point. diff --git a/changelog.d/nursery-cap-rss-evidence.md b/changelog.d/nursery-cap-rss-evidence.md new file mode 100644 index 0000000000..d809cd2bad --- /dev/null +++ b/changelog.d/nursery-cap-rss-evidence.md @@ -0,0 +1,6 @@ +### Changed + +- Keep the adaptive copying nursery's steady-state ceiling byte-denominated: + survivor object-size censuses remain diagnostic after the first copying + minor, low mortality no longer shrinks the earned scale, and only minors + that were actually nursery-cap-due may rate influx growth. diff --git a/crates/perry-runtime/src/gc/copying.rs b/crates/perry-runtime/src/gc/copying.rs index 2eb134d64f..07b9f5d7c3 100644 --- a/crates/perry-runtime/src/gc/copying.rs +++ b/crates/perry-runtime/src/gc/copying.rs @@ -617,8 +617,29 @@ impl CopyingNurseryCollector { // moved somewhere at any threshold), which is what makes the loop's // fixed point stable. match ptr.kind { - CopyingPointerKind::Eden => self.stats.eden_live_bytes += total, - _ => self.stats.survivor_live_bytes += total, + CopyingPointerKind::Eden => { + self.stats.eden_live_bytes += total; + // #9851 follow-up: the fresh half of `copied_bytes`. The + // survival-rate lock's denominator must be the intake of ONE + // cohort; `copied_bytes` also carries survivor residents being + // re-copied, which at a threshold above 2 is most of it. + if !promote { + self.stats.eden_copied_bytes += total; + } + } + _ => { + self.stats.survivor_live_bytes += total; + // ...and the matching numerator. A from-survivor object whose + // stored age is 1 entered from Eden on the previous cycle, so + // it is a member of exactly the cohort `eden_copied_bytes` + // counted then. Ages above 1 have already survived a round and + // are a population selected for longevity; including them is + // what made the ratio drift above the lock's bar as the + // threshold rose. + if prior_age == 1 { + self.stats.survivor_first_round_live_bytes += total; + } + } } new_user as usize } @@ -1175,6 +1196,11 @@ pub(super) fn run_copied_minor_attempt( _trigger_kind: GcTriggerKind, may_speculate: bool, ) -> CopiedMinorAttempt { + // Capture before eligibility or evacuation can rewrite from-space. The + // safepoint folds nursery-cap and whole-arena triggers into the same + // `ArenaBytes` label, so the operands themselves are the only evidence + // that this collection was cap-due and may rate the scale. + let nursery_cap_rating = super::tenuring::NurseryCapRating::capture(); if let Some(trace) = trace.as_mut() { trace.copying_nursery = eligibility.trace_stats(); trace.legacy_copy_only_scanner_pinned = eligibility.legacy_root_stats; @@ -1796,6 +1822,10 @@ pub(super) fn run_copied_minor_attempt( // enable, so it releases the latch that suppressed a repeat handoff. note_copying_minor_completed(); super::instruments::note_copying_minor_pause_us(start.elapsed().as_micros() as u64); + super::instruments::note_tenuring_price_bytes( + collector.stats.copied_bytes, + collector.stats.promoted_bytes, + ); // #7604: the process-wide liveness counters. A copying minor ran, and this // is how much it actually relocated -- the only evidence that distinguishes // "the instrument was armed" from "the instrument fired". @@ -1852,11 +1882,10 @@ pub(super) fn run_copied_minor_attempt( // re-baseline sees post-collection live allocation rather than high-water. note_copying_minor_young_survival(collector.stats.young_survival_permille); maybe_schedule_old_reclaim_after_copied_minor(); - // #7929: the object denomination of the nursery constant band, fed BEFORE - // the tenuring loop so every number `retune_after_scavenge` derives from - // the effective cap (desired survivor occupancy, the cap-scale band) reads - // one consistent factor. Both tenuring ratios are representation-invariant - // by cancellation, so this only re-denominates the constant band itself. + // #7929/#8122: publish the survivor object-size census before the tenuring + // loop. This exact call is also the first-copying-minor completion witness: + // it ends object denomination after the tracing regime while keeping the + // measured mean observable in diagnostics. super::tenuring::note_surviving_object_census( collector .stats @@ -1867,14 +1896,25 @@ pub(super) fn run_copied_minor_attempt( .copied_objects .saturating_add(collector.stats.promoted_objects), ); - retune_after_scavenge( - collector.stats.eden_live_bytes, + // #9851 follow-up: the survival-rate lock is fed the FRESH cohort's intake + // and that same cohort's survival, not the whole survivor space's. See + // `retune_after_scavenge`. + #[cfg(test)] + test_record_cohort_split( collector.stats.copied_bytes, + collector.stats.eden_copied_bytes, collector.stats.survivor_live_bytes, + collector.stats.survivor_first_round_live_bytes, + ); + retune_after_scavenge( + collector.stats.eden_live_bytes, + collector.stats.eden_copied_bytes, + collector.stats.survivor_first_round_live_bytes, + nursery_cap_rating, ); if crate::gc::gc_diag_enabled() { eprintln!( - "[gc-copy-minor] ran in_place={} untraced={} untraced_cycles={} untraced_objects={} in_place_blocks={} in_place_dead_bytes={} sparse_blocks={} survival_permille={} copied_objects={} copied_bytes={} promoted_objects={} promoted_bytes={} freed_bytes={} tenuring_survivals={} eden_live_bytes={} trigger={:?} declared_safepoint={}", + "[gc-copy-minor] ran in_place={} untraced={} untraced_cycles={} untraced_objects={} in_place_blocks={} in_place_dead_bytes={} sparse_blocks={} survival_permille={} copied_objects={} copied_bytes={} promoted_objects={} promoted_bytes={} freed_bytes={} tenuring_survivals={} eden_live_bytes={} eden_copied_bytes={} survivor_live_bytes={} survivor_first_round_live_bytes={} trigger={:?} declared_safepoint={}", collector.stats.in_place_promotion, untraced, super::untraced_promotion_cycles(), @@ -1890,6 +1930,9 @@ pub(super) fn run_copied_minor_attempt( freed_bytes, collector.stats.tenuring_survivals, collector.stats.eden_live_bytes, + collector.stats.eden_copied_bytes, + collector.stats.survivor_live_bytes, + collector.stats.survivor_first_round_live_bytes, _trigger_kind, super::policy::GC_AT_DECLARED_SAFEPOINT.with(std::cell::Cell::get) ); @@ -1908,6 +1951,41 @@ pub(super) fn run_copied_minor_attempt( })) } +/// Test-only witness for the #9851 follow-up: the whole-space pair against the +/// fresh-cohort pair, as the copier computed them for one cycle. Without this +/// the change is unfalsifiable from a test — the two quantities are equal on +/// every heap whose survivor space holds a single generation, which is every +/// heap at a threshold of 2 or below. +#[cfg(test)] +thread_local! { + static LAST_COHORT_SPLIT: std::cell::Cell<(usize, usize, usize, usize)> = + const { std::cell::Cell::new((0, 0, 0, 0)) }; +} + +#[cfg(test)] +fn test_record_cohort_split( + copied_bytes: usize, + eden_copied_bytes: usize, + survivor_live_bytes: usize, + first_round_live_bytes: usize, +) { + LAST_COHORT_SPLIT.with(|c| { + c.set(( + copied_bytes, + eden_copied_bytes, + survivor_live_bytes, + first_round_live_bytes, + )) + }); +} + +/// `(copied_bytes, eden_copied_bytes, survivor_live_bytes, first_round_live_bytes)` +/// from the most recent copying minor on this thread. +#[cfg(test)] +pub(super) fn test_last_cohort_split() -> (usize, usize, usize, usize) { + LAST_COHORT_SPLIT.with(std::cell::Cell::get) +} + fn finalize_dead_copied_minor_from_space_side_allocations() { crate::map::finalize_dead_copied_minor_from_space_maps(); crate::set::finalize_dead_copied_minor_from_space_sets(); diff --git a/crates/perry-runtime/src/gc/instruments.rs b/crates/perry-runtime/src/gc/instruments.rs index 267a8bd1d5..d629dbc219 100644 --- a/crates/perry-runtime/src/gc/instruments.rs +++ b/crates/perry-runtime/src/gc/instruments.rs @@ -347,6 +347,10 @@ static REMARK_TOTAL_US: AtomicU64 = AtomicU64::new(0); static MINOR_TOTAL_US: AtomicU64 = AtomicU64::new(0); /// Total microseconds spent inside synchronous `js_gc_collect` calls. static FULL_TOTAL_US: AtomicU64 = AtomicU64::new(0); +/// Bytes copied into a survivor space, for the tenuring copy-price diagnostic. +static TENURING_COPIED_BYTES: AtomicU64 = AtomicU64::new(0); +/// Bytes promoted to old-gen, for the tenuring promotion-price diagnostic. +static TENURING_PROMOTED_BYTES: AtomicU64 = AtomicU64::new(0); /// Record one copying minor's pause duration. #[inline] @@ -354,6 +358,19 @@ pub(crate) fn note_copying_minor_pause_us(us: u64) { MINOR_TOTAL_US.fetch_add(us, Ordering::Relaxed); } +/// Record the byte denominators for the tenuring price experiment. +/// +/// These are diagnostic-only: when `PERRY_GC_DIAG` is off the two atomics are +/// untouched, so measuring the future decision rule adds no release-path cost. +#[inline] +pub(crate) fn note_tenuring_price_bytes(copied_bytes: usize, promoted_bytes: usize) { + if !crate::gc::gc_diag_enabled() { + return; + } + TENURING_COPIED_BYTES.fetch_add(copied_bytes as u64, Ordering::Relaxed); + TENURING_PROMOTED_BYTES.fetch_add(promoted_bytes as u64, Ordering::Relaxed); +} + /// Record one synchronous full collection's wall duration. #[inline] pub(crate) fn note_full_collect_us(us: u64) { @@ -371,6 +388,20 @@ pub fn gc_time_totals_us() -> (u64, u64, u64, u64) { ) } +/// `(copy pause us, copied bytes, step+remark us, promoted bytes)` cumulative +/// counters for pricing aging against promotion. Byte counters remain zero +/// unless diagnostics are enabled. +pub(crate) fn tenuring_price_counters() -> (u64, u64, u64, u64) { + ( + MINOR_TOTAL_US.load(Ordering::Relaxed), + TENURING_COPIED_BYTES.load(Ordering::Relaxed), + STEP_TOTAL_US + .load(Ordering::Relaxed) + .saturating_add(REMARK_TOTAL_US.load(Ordering::Relaxed)), + TENURING_PROMOTED_BYTES.load(Ordering::Relaxed), + ) +} + /// Microseconds since the first call in this process (the same epoch the /// mark-barrier timer uses) — the wall-clock denominator for the share line. pub(crate) fn wall_us_since_epoch() -> u64 { diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index 64409430cb..ec32b4438f 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -1409,10 +1409,12 @@ fn emit_incremental_liveness_diag() { /// disjoint by construction. fn emit_gc_time_share_diag() { let (step_us, remark_us, minor_us, full_us) = instruments::gc_time_totals_us(); + let (copy_pause_us, tenuring_copied_bytes, promote_us, tenuring_promoted_bytes) = + instruments::tenuring_price_counters(); let wall_us = instruments::wall_us_since_epoch().max(1); let pause_us = step_us + remark_us + minor_us; eprintln!( - "[gc-time] wall_us={wall_us} step_us={step_us} remark_us={remark_us} minor_us={minor_us} full_sync_us={full_us} share_permille={}", + "[gc-time] wall_us={wall_us} step_us={step_us} remark_us={remark_us} minor_us={minor_us} full_sync_us={full_us} share_permille={} copy_pause_us={copy_pause_us} tenuring_copied_bytes={tenuring_copied_bytes} promote_us={promote_us} tenuring_promoted_bytes={tenuring_promoted_bytes}", pause_us.saturating_mul(1000) / wall_us, ); } diff --git a/crates/perry-runtime/src/gc/telemetry.rs b/crates/perry-runtime/src/gc/telemetry.rs index ac62c3b605..e8c48255e1 100644 --- a/crates/perry-runtime/src/gc/telemetry.rs +++ b/crates/perry-runtime/src/gc/telemetry.rs @@ -275,6 +275,18 @@ pub(super) struct CopyingNurseryTraceStats { /// Live bytes re-copied/promoted out of the from-survivor space this /// cycle — the re-copy tax the adaptive loop exists to bound. pub(super) survivor_live_bytes: usize, + /// #9851 follow-up: the FRESH half of `copied_bytes` — bytes copied out of + /// Eden into the to-survivor space this cycle, excluding survivor-space + /// residents being re-copied. This is the intake of exactly one cohort, + /// and it is the denominator the survival-rate lock must use. + pub(super) eden_copied_bytes: usize, + /// The matching numerator: live bytes moved out of the from-survivor space + /// this cycle whose stored survival age was 1 — i.e. objects that entered + /// the survivor space from Eden on the PREVIOUS cycle, and nothing older. + /// `survivor_live_bytes` rates the whole space, whose composition changes + /// with the threshold; this rates one aging round of one fresh cohort, + /// which is what the lock's conclusion is about. + pub(super) survivor_first_round_live_bytes: usize, pub(super) large_excluded_objects: usize, pub(super) large_excluded_bytes: usize, pub(super) reset_blocks: usize, @@ -1151,6 +1163,8 @@ impl GcCycleTrace { "tenuring_survivals": self.copying_nursery.tenuring_survivals, "eden_live_bytes": self.copying_nursery.eden_live_bytes, "survivor_live_bytes": self.copying_nursery.survivor_live_bytes, + "eden_copied_bytes": self.copying_nursery.eden_copied_bytes, + "survivor_first_round_live_bytes": self.copying_nursery.survivor_first_round_live_bytes, "large_excluded_objects": self.copying_nursery.large_excluded_objects, "large_excluded_bytes": self.copying_nursery.large_excluded_bytes, "reset_blocks": self.copying_nursery.reset_blocks, diff --git a/crates/perry-runtime/src/gc/tenuring.rs b/crates/perry-runtime/src/gc/tenuring.rs index 1abb9dd99c..eb47a3d1cd 100644 --- a/crates/perry-runtime/src/gc/tenuring.rs +++ b/crates/perry-runtime/src/gc/tenuring.rs @@ -47,14 +47,14 @@ //! default — 40 bytes under) settles at S=2 and still copies every //! surviving byte exactly once for nothing, because 100% of each cohort //! survives its survivor round and gets promoted a cycle later anyway. -//! The **survival-rate lock** closes that: when last cycle's survivor -//! intake (`copied_bytes`) was substantial and ≥90% of it came back out -//! alive this cycle (`survivor_live_bytes`), the aging round demonstrably -//! filters nothing, so the threshold locks to 1 (promote on first copy) -//! until the influx goes quiet. The lock's exit signal — influx below -//! `desired/4` for two consecutive cycles — stays measurable while -//! locked, unlike survivor occupancy, which is zero at S=1 and would -//! leave the loop blind. +//! The **survival-rate lock** closes that, but only from steady evidence. The +//! first two rated cohorts belong to process startup and cannot decide the +//! lock. After that, three consecutive substantial survivor rounds must each +//! return ≥90% of the cohort copied in the preceding cycle before the threshold +//! locks to 1 (promote on first copy). Any rated round below that bar resets the +//! streak. The lock's exit signal — influx below `desired/4` for two consecutive +//! cycles — stays measurable while locked, unlike survivor occupancy, which is +//! zero at S=1 and would leave the loop blind. //! //! There is no env knob here (see CLAUDE.md's GC knob kill-policy): the //! loop is always on, and its neutral state — influx below `desired`, @@ -63,9 +63,11 @@ //! //! ## Seeding the lock from a non-copying collection (#7598) //! -//! The survival-rate lock above is correct but **one cycle late by -//! construction**: it keys on `prev_copied`, so a *previous copying minor* -//! must already have filled the survivor space. On a workload with one +//! The survival-rate measurement is **one cycle late by construction**: it +//! keys on `prev_copied`, so a *previous copying minor* must already have filled +//! the survivor space. After the startup evidence boundary, a sweep seed can +//! still establish the lock before a later copying minor pays that round trip. +//! On a workload with one //! long-lived burst (`json_pipeline`: `out.push({…})` 500k times) the first //! copying minor therefore always pays the wasted copy — measured 268 MB //! Eden→survivor on cycle 3 and the same 268 MB survivor→old on cycle 4, @@ -138,13 +140,70 @@ use super::*; -/// Ceiling and power-on value: the previous fixed threshold. +/// Ceiling and previous fixed threshold. pub(super) const GC_TENURING_SURVIVALS_MAX: u8 = GC_COPY_PROMOTION_SURVIVALS; +/// The lowest threshold the **occupancy rule** may select. +/// +/// Not a tuned number: it is the lowest S at which `copied_bytes > 0`, i.e. the +/// lowest value that still PRODUCES the survivor-round measurement. At S=1 +/// nothing enters the survivor space; at S=2 exactly one cohort does. +/// +/// Why the occupancy rule must not reach 1 (#9851). A threshold of 1 is a claim +/// about **lifetime** — "this cohort will not die, promote it on first copy" — +/// and the occupancy rule measures **space**: `(S-1) * influx <= desired` asks +/// only whether one cohort fits in the desired survivor size. When it does not, +/// S=1 does not reduce the surviving data; it relocates it, from the survivor +/// space (where the next minor re-examines it for free) to the old generation +/// (which only a full can reclaim). The formula has no term for that. +/// +/// Worse, S=1 is **self-sealing**: with nothing copied, `copied_bytes` is 0, so +/// next cycle `prev_copied` is 0, so the survival-rate lock's guard +/// (`prev_copied >= substantial`) is false forever. The state destroys the only +/// measurement that could refute it, and both remaining exits — the occupancy +/// recompute and `PROMOTE_LOCK`'s unlock — are *quiet-influx* exits, which say +/// nothing about lifetime. +/// +/// Measured on the compiled claude-code TUI, 4 streamed turns in one process, +/// both arms from one binary via `PERRY_GC_TENURING_SURVIVALS` (3300-char): +/// +/// | | adaptive | pinned S=2 | +/// |---|---|---| +/// | minors at S=1 | 351 of 352, carrying 100 % of promotion | 0 | +/// | survivor-round mortality samples | **1** | **393** | +/// | median mortality | **0.9 %** — the first minor of the process | **26.1 %** | +/// | ...in steady turns 2 / 3 / 4 | not measurable | 26.1 / 26.1 / 26.1 % | +/// | promoted | 1057 MB | 792 MB | +/// +/// The loop takes its one and only mortality sample on the first minor of the +/// process — before any steady state, when the cohort really is immortal — +/// concludes "nothing dies", and can never sample again. In steady state an +/// aging round filters about **a quarter** of the cohort. +/// +/// Reaching 1 still belongs to the two paths that actually MEASURE mortality: +/// the survival-rate lock (`prev_copied` substantial and >=90 % of it came back +/// alive) and the sweep seed (the mark-sweep's own Eden live/dead split). The +/// evidence gate below delays, rather than removes, those paths. So the rule is +/// self-limiting: on a workload whose steady cohorts genuinely do not die, the +/// lock reaches 1 after its bounded window. +pub(super) const OCCUPANCY_MIN_SURVIVALS: u8 = 2; + /// Consecutive cycles the computed target must exceed the current threshold /// before it is raised (by one step). const RAISE_DEBOUNCE_CYCLES: u8 = 2; +/// Rated survivor cohorts excluded as process startup. TN4 observed the lock +/// rating exactly the cohort formed by the first two copying minors, before the +/// first application turn. Counting rated cohorts (rather than wall time or an +/// allocation threshold) uses the lock's own threshold-invariant evidence and +/// cannot change collection pacing. +const STARTUP_RATED_ROUNDS: u64 = 2; + +/// Consecutive qualifying post-startup cohorts required to lock. Three is the +/// smallest window that both rejects a one-off/two-cycle phase boundary and +/// reaches S=1 promptly on a genuinely non-dying workload. +const PROMOTE_LOCK_RATED_ROUNDS: u8 = 3; + /// Ceiling for the influx-driven nursery cap scale: 16 MB × 4 = 64 MB. /// Bounds the young-gen RSS contribution on live-set-bound workloads while /// still cutting their collection count 4× (each collection carries a fixed @@ -153,19 +212,32 @@ const RAISE_DEBOUNCE_CYCLES: u8 = 2; const NURSERY_CAP_SCALE_MAX: u8 = 4; crate::perry_thread_local! { - static TENURING_SURVIVALS: Cell = const { Cell::new(GC_TENURING_SURVIVALS_MAX) }; + /// Power-on threshold. This is `OCCUPANCY_MIN_SURVIVALS`, not the ceiling: + /// see `SURVIVOR_ROUND_MEASURED`. Starting at the ceiling is a claim that + /// young objects live long, made before a single object has been given the + /// chance to die, and it is the expensive direction of that claim -- every + /// survivor is copied three times before it can be promoted. + static TENURING_SURVIVALS: Cell = const { Cell::new(OCCUPANCY_MIN_SURVIVALS) }; + /// Has any survivor round been RATED yet on this thread -- i.e. did some + /// cycle put a cohort into the survivor space that the next cycle could + /// then follow? Until this is true the loop has no lifetime evidence of + /// any kind, and the occupancy rule may not move off the floor. + static SURVIVOR_ROUND_MEASURED: Cell = const { Cell::new(false) }; static RAISE_STREAK: Cell = const { Cell::new(0) }; /// Survival-rate lock: promote-on-first-copy until influx goes quiet. static PROMOTE_LOCK: Cell = const { Cell::new(false) }; static UNLOCK_STREAK: Cell = const { Cell::new(0) }; + /// Number of survivor cohorts whose first round has been rated. + static RATED_ROUNDS: Cell = const { Cell::new(0) }; + /// Consecutive substantial, ≥90%-surviving post-startup rated cohorts. + static PROMOTE_LOCK_STREAK: Cell = const { Cell::new(0) }; /// Bytes the previous copying minor put into the to-survivor space — /// the denominator of this cycle's survival rate. static PREV_COPIED_BYTES: Cell = const { Cell::new(0) }; /// Influx-driven multiplier (1, 2, or 4) applied to the scavenge nursery - /// cap. Power of two; grows/shrinks one step at a time, debounced. + /// cap. Power of two; grows one step at a time, debounced. static NURSERY_CAP_SCALE: Cell = const { Cell::new(1) }; static CAP_GROW_STREAK: Cell = const { Cell::new(0) }; - static CAP_SHRINK_STREAK: Cell = const { Cell::new(0) }; /// #7929: mean size of the objects the last copying minor moved. Seeded at /// the calibration reference so a process with no completed copying minor /// paces exactly as it did before the object denomination existed — until @@ -177,6 +249,19 @@ crate::perry_thread_local! { /// allocation census — replaced the seed? Once true the allocation probe /// never runs again (its walk is paid at most once per process). static OBJECT_CENSUS_SEEDED: Cell = const { Cell::new(false) }; + /// Has this thread completed a copying minor? Unlike + /// `OBJECT_CENSUS_SEEDED`, this cannot be set by #8122's allocation walk; + /// unlike `SURVIVOR_ROUND_MEASURED`, it flips on the first completed minor + /// rather than waiting for a cohort to return on the second. + static COPYING_MINOR_COMPLETED: Cell = const { Cell::new(false) }; +} + +#[cfg(test)] +thread_local! { + /// Scoped threshold pin for tests of mechanisms that require a particular + /// promotion age. This is thread-local for the same reason as the adaptive + /// state: runtime tests share one process and may run on different threads. + static TENURING_SURVIVALS_TEST_OVERRIDE: Cell> = const { Cell::new(None) }; } /// The survivals threshold the next copying minor should promote at: @@ -184,12 +269,37 @@ crate::perry_thread_local! { /// original fixed policy, 1 promotes every live nursery object on first /// copy. pub(super) fn tenuring_survivals() -> u8 { + #[cfg(test)] + if let Some(forced) = TENURING_SURVIVALS_TEST_OVERRIDE.with(Cell::get) { + return forced; + } if let Some(forced) = tenuring_survivals_override() { return forced; } TENURING_SURVIVALS.with(Cell::get) } +/// Pin the promotion age for a threshold-sensitive test on this thread. +/// Restores the previous pin on drop; the adaptive policy continues to run +/// underneath it, but every copying minor snapshots the explicitly pinned age. +#[cfg(test)] +pub(super) fn set_survivals_for_test(survivals: u8) -> TenuringSurvivalsTestGuard { + assert!((1..=GC_TENURING_SURVIVALS_MAX).contains(&survivals)); + TenuringSurvivalsTestGuard( + TENURING_SURVIVALS_TEST_OVERRIDE.with(|cell| cell.replace(Some(survivals))), + ) +} + +#[cfg(test)] +pub(super) struct TenuringSurvivalsTestGuard(Option); + +#[cfg(test)] +impl Drop for TenuringSurvivalsTestGuard { + fn drop(&mut self) { + TENURING_SURVIVALS_TEST_OVERRIDE.with(|cell| cell.set(self.0)); + } +} + /// `PERRY_GC_TENURING_SURVIVALS=` pins the promotion age, overriding the /// adaptive threshold (#7432). Diagnostic only; unset means adaptive. /// @@ -263,12 +373,20 @@ pub(super) fn scavenge_nursery_cap_effective_bytes() -> usize { } /// The influx-driven half of the cap on its own: the configured base times -/// the debounced `NURSERY_CAP_SCALE`, **re-denominated in objects** by -/// [`nursery_cap_object_scale_permille`]. Named so the composition below reads -/// as the two-term policy it is. +/// the debounced `NURSERY_CAP_SCALE`. +/// +/// Before the first copying minor completes, #8122's allocation census +/// re-denominates this band in objects so the first tracing cycle sees the +/// object budget #7929 measured. Once a copying minor has completed, the +/// survivor census remains diagnostic evidence but no longer changes the byte +/// band: copying-minor cost is fixed plus O(bytes moved), and survivor size +/// does not price the fixed work or RSS. pub(super) fn influx_driven_nursery_cap_bytes() -> usize { let constant_band = gc_scavenge_nursery_cap_bytes().saturating_mul(NURSERY_CAP_SCALE.with(Cell::get) as usize); + if COPYING_MINOR_COMPLETED.with(Cell::get) { + return constant_band; + } // The multiply is done in u64 deliberately. `usize::saturating_mul` on an // ILP32 target (watchOS/visionOS are 32-bit) would saturate a 64 MB band // against a 1000-per-mille factor at `u32::MAX` and the following divide @@ -304,8 +422,9 @@ pub(super) const NURSERY_CAP_REFERENCE_OBJECT_BYTES: usize = 72; /// atypically small. const NURSERY_CAP_OBJECT_SCALE_MIN_PERMILLE: usize = 500; -/// #7929: how much of the byte-denominated constant band this representation -/// should get, in per mille, so the band buys a **constant number of objects**. +/// #7929: how much of the byte-denominated first tracing band this +/// representation should get, in per mille, so that band buys a **constant +/// number of objects**. /// /// The collector's trigger is denominated in bytes and its per-cycle cost is /// per object, so a fixed byte band silently buys more collector work as @@ -329,6 +448,11 @@ const NURSERY_CAP_OBJECT_SCALE_MIN_PERMILLE: usize = 500; /// crosses `GC_OLD_GEN_RECLAIM_THRESHOLD_BYTES` or lands in a #7909 budgeted /// stall. Every program at or above the reference — `retain_wide`, /// `retain_wide1`, `push_num`, `shapes` — is left bit-identical. +/// +/// [`influx_driven_nursery_cap_bytes`] consults this factor only before the +/// first copying minor completes. That preserves the traced-object cost this +/// comment measured without using survivor object size to multiply the fixed +/// cost of later copying minors. pub(super) fn nursery_cap_object_scale_permille(mean_surviving_object_bytes: usize) -> usize { if mean_surviving_object_bytes == 0 { return 1000; @@ -337,15 +461,20 @@ pub(super) fn nursery_cap_object_scale_permille(mean_surviving_object_bytes: usi .clamp(NURSERY_CAP_OBJECT_SCALE_MIN_PERMILLE, 1000) } -/// Feed one finished copying minor's move census into the object denomination. +/// Record one finished copying minor's move census and end the first-minor +/// object-denomination regime. /// -/// Called from the copying minor immediately before [`retune_after_scavenge`], -/// so everything that end-of-cycle computes is policy for the *next* cycle and -/// sees one consistent factor. A cycle that moved nothing carries the previous -/// estimate forward rather than resetting it — `tree`/`cycles` move single -/// digits of objects per minor and a zero denominator there is a missing -/// measurement, not a measurement of zero. +/// Called from the copying minor immediately before [`retune_after_scavenge`]. +/// The mean remains observable in diagnostics, but the next cycle's byte band +/// no longer reads it. A cycle that moved nothing carries the previous estimate +/// forward rather than resetting it — `tree`/`cycles` move single digits of +/// objects per minor and a zero denominator there is a missing measurement, +/// not a measurement of zero. pub(super) fn note_surviving_object_census(moved_bytes: usize, moved_objects: usize) { + // This call is the completion witness, including for a minor that moved no + // objects. Set it before the missing-census returns so such a cycle still + // ends the first-minor tracing regime exactly once. + COPYING_MINOR_COMPLETED.with(|completed| completed.set(true)); if moved_objects == 0 { return; } @@ -358,7 +487,7 @@ pub(super) fn note_surviving_object_census(moved_bytes: usize, moved_objects: us if previous != mean && crate::gc::gc_diag_enabled() { eprintln!( "[gc-tenuring] nursery cap object denomination: mean_surviving_object_bytes {} -> {} \ - (scale {} permille, band {} B)", + (scale {} permille, band {} B, applied=false)", previous, mean, nursery_cap_object_scale_permille(mean), @@ -390,12 +519,13 @@ pub(super) fn note_surviving_object_census(moved_bytes: usize, moved_objects: us /// point every program that will ever reach the cap passes exactly once — hop /// the young generation's headers (`arena::young_allocation_census`, ~1M /// instructions for 8 MB of small objects, paid ONCE per process) and install -/// `bytes / objects` as the mean. The first minor is then object-denominated -/// like every later one, and a smaller representation no longer buys the -/// collector a bigger first trace. The one-sided clamp still applies: a mean +/// `bytes / objects` as the mean. The first minor is then object-denominated, +/// and a smaller representation no longer buys the collector a bigger first +/// trace. The one-sided clamp still applies in this tracing regime: a mean /// above the reference leaves the 16 MB band untouched, so an array-dominated /// allocation stream cannot raise the cap. The collector's survivor census -/// overwrites this seed at the first minor, so steady state is unchanged. +/// overwrites this seed at the first minor for diagnostics, and that completed +/// minor ends object denomination for all later byte bands. /// /// Returns without walking when the base cap is not yet half full, when a /// census (either kind) already exists, or when the nursery is empty. @@ -421,7 +551,7 @@ pub(super) fn maybe_seed_object_census_from_allocation(from_space_in_use_bytes: if crate::gc::gc_diag_enabled() { eprintln!( "[gc-tenuring] nursery cap object denomination: allocation census seeded \ - mean_object_bytes {} -> {} ({} objects / {} B in from-space; scale {} permille, band {} B)", + mean_object_bytes {} -> {} ({} objects / {} B in from-space; scale {} permille, band {} B, applied=true)", previous, mean, objects, @@ -467,22 +597,103 @@ pub(super) fn compute_target_survivals(eden_live_bytes: usize, desired_bytes: us target.min(GC_TENURING_SURVIVALS_MAX as usize) as u8 } +/// The two operands and verdict of the nursery-cap comparison captured before +/// a copying attempt can rewrite from-space. The trigger kind is not evidence: +/// the precise safepoint folds the cap into `ArenaBytes`, while direct and +/// scheduled minors can reach the same copying path without the cap being due. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) struct NurseryCapRating { + pub(super) from_space_bytes: usize, + pub(super) cap_bytes: usize, + pub(super) cap_due: bool, +} + +impl NurseryCapRating { + pub(super) fn capture() -> Self { + let from_space_bytes = crate::arena::copying_from_space_in_use_bytes(); + let cap_bytes = scavenge_nursery_cap_effective_bytes(); + Self { + from_space_bytes, + cap_bytes, + cap_due: super::policy::young_scavenge_cap_due(), + } + } + + #[cfg(test)] + pub(super) const fn from_values(from_space_bytes: usize, cap_bytes: usize) -> Self { + Self { + from_space_bytes, + cap_bytes, + cap_due: from_space_bytes >= cap_bytes, + } + } +} + /// Feed one finished copying-minor cycle into the feedback loop. /// `eden_live_bytes` is the cycle's Eden survivor influx (bytes moved out -/// of Eden, whether copied to a survivor space or promoted); -/// `copied_bytes` is what this cycle put into the to-survivor space; -/// `survivor_live_bytes` is what came back out of the from-survivor space -/// alive (numerator of the survival rate against the *previous* cycle's -/// `copied_bytes`). +/// of Eden, whether copied to a survivor space or promoted). +/// +/// The other two are **one cohort's** intake and that same cohort's survival, +/// and they must stay that way (#9851 follow-up): +/// `eden_copied_bytes` is what this cycle copied out of *Eden* into the +/// to-survivor space — a fresh cohort, no re-copies — and +/// `first_round_live_bytes` is what came back out of the from-survivor space +/// alive with a stored age of 1, i.e. members of the cohort that the +/// *previous* cycle's `eden_copied_bytes` counted. +/// +/// Why not the whole space. The survivor spaces are a strict semispace pair, +/// so the from-space at cycle N holds exactly what cycle N-1 copied, and +/// `survivor_live_bytes / prev_copied_bytes` is a well-formed survival ratio — +/// of the whole space. But *what that space contains* is set by the very +/// threshold this loop controls: at S<=2 it is one fresh cohort, at S=3-4 it +/// also holds age-2 and age-3 objects, which have already survived a round and +/// are therefore selected for longevity. Rating that mixture and concluding +/// "the aging round filters nothing" applies a measurement of an aged, +/// self-selected population to first-round cohorts. Measured on the compiled +/// claude-code TUI: a fresh cohort survives at 74 %, and the loop still reached +/// the lock's 90 % bar 8-12 times per four-turn run once #9851's clamp let the +/// ladder climb past 2. pub(super) fn retune_after_scavenge( eden_live_bytes: usize, - copied_bytes: usize, - survivor_live_bytes: usize, + eden_copied_bytes: usize, + first_round_live_bytes: usize, + nursery_cap_rating: NurseryCapRating, ) { - retune_nursery_cap_scale(eden_live_bytes); + retune_nursery_cap_scale(eden_live_bytes, nursery_cap_rating); let desired = desired_survivor_bytes(); let substantial = desired / 4; - let prev_copied = PREV_COPIED_BYTES.with(|c| c.replace(copied_bytes)); + let prev_cohort_copied = PREV_COPIED_BYTES.with(|c| c.replace(eden_copied_bytes)); + // A cohort went into the survivor space last cycle, so THIS cycle is the + // one that could follow it: from here on the loop has lifetime evidence and + // the occupancy rule is allowed to move off the floor. + let rated = prev_cohort_copied > 0; + let rounds_rated = if rated { + SURVIVOR_ROUND_MEASURED.with(|m| m.set(true)); + RATED_ROUNDS.with(|rounds| { + let next = rounds.get().saturating_add(1); + rounds.set(next); + next + }) + } else { + RATED_ROUNDS.with(Cell::get) + }; + let startup = if rated { + rounds_rated <= STARTUP_RATED_ROUNDS + } else { + rounds_rated < STARTUP_RATED_ROUNDS + }; + let survival_permille = if rated { + first_round_live_bytes.saturating_mul(1000) / prev_cohort_copied + } else { + 0 + }; + let mut evidence = TransitionEvidence { + rounds_rated, + streak: PROMOTE_LOCK_STREAK.with(Cell::get), + survival_permille, + copied_bytes: prev_cohort_copied, + startup, + }; let current = TENURING_SURVIVALS.with(Cell::get); if PROMOTE_LOCK.with(Cell::get) { @@ -494,11 +705,13 @@ pub(super) fn retune_after_scavenge( if streak >= RAISE_DEBOUNCE_CYCLES { PROMOTE_LOCK.with(|l| l.set(false)); UNLOCK_STREAK.with(|s| s.set(0)); + PROMOTE_LOCK_STREAK.with(|s| s.set(0)); RAISE_STREAK.with(|s| s.set(0)); // Resume the ladder one step up rather than snapping to the // ceiling; the normal debounced rise takes it the rest of // the way if the workload stays quiet. - set_survivals(current, 2, eden_live_bytes, "unlock"); + evidence.streak = 0; + set_survivals(current, 2, eden_live_bytes, "unlock", evidence); } else { UNLOCK_STREAK.with(|s| s.set(streak)); } @@ -508,18 +721,65 @@ pub(super) fn retune_after_scavenge( return; } - // Survival-rate lock: last cycle's survivor intake was substantial and + // Survival-rate lock: last cycle's FRESH COHORT was substantial and // (nearly) all of it came back out alive, so the aging round filters // nothing — every copied byte is a byte that will be promoted anyway. - if prev_copied >= substantial && survivor_live_bytes.saturating_mul(10) >= prev_copied * 9 { + // + // Both sides are scoped to that one cohort (#9851 follow-up). Rating the + // whole survivor space instead makes the ratio rise with the threshold + // this rule sets, because a higher threshold is precisely what keeps + // already-aged objects in the space; the rule then reads its own setting + // back as evidence. See `retune_after_scavenge`'s header. + if rated { + let qualifies = !startup + && prev_cohort_copied >= substantial + && first_round_live_bytes.saturating_mul(10) + >= prev_cohort_copied.saturating_mul(PROMOTE_LOCK_LIVE_TENTHS); + let streak = if qualifies { + PROMOTE_LOCK_STREAK.with(Cell::get).saturating_add(1) + } else { + 0 + }; + PROMOTE_LOCK_STREAK.with(|s| s.set(streak)); + evidence.streak = streak; + } + if evidence.streak >= PROMOTE_LOCK_RATED_ROUNDS { PROMOTE_LOCK.with(|l| l.set(true)); UNLOCK_STREAK.with(|s| s.set(0)); RAISE_STREAK.with(|s| s.set(0)); - set_survivals(current, 1, eden_live_bytes, "lock"); + set_survivals(current, 1, eden_live_bytes, "lock", evidence); return; } - let target = compute_target_survivals(eden_live_bytes, desired); + // #9851: the occupancy rule measures SPACE and may not conclude 1, which is + // a claim about LIFETIME — see `OCCUPANCY_MIN_SURVIVALS`. Deliberately + // clamped HERE and not inside `compute_target_survivals`: that pure function + // has a second caller, `full_seed_promotes_on_first_copy`, which gates the + // sweep seed on `... != 1` ("would occupancy alone already promote on first + // copy?"). Clamping the shared function would silently disarm the sweep + // seed, which is one of the two paths that IS allowed to reach 1. + // + // The startup follow-up makes that rule SYMMETRIC. `1 + desired / influx` + // returns the ceiling for a tiny influx and for a zero one, so on the first + // minors of a process — when the heap is nearly empty and no cohort has + // ever been followed — occupancy claims the maximum. That is the same + // category error in the other direction: a claim about LIFETIME from a + // measurement of SPACE, made before any evidence exists, and the expensive + // one, because every survivor is then copied up to three times before it + // may be promoted. Measured on the compiled claude-code TUI, the whole + // adaptive-vs-pinned difference was this startup excursion — + // `4 -> 2 (occupancy) -> 1 (lock)` inside turn 1 and nothing afterwards, + // worth +0.35..0.45 s at 3300 chars and +50 % at 400. + // + // So until one survivor round has actually been rated, occupancy holds at + // the floor: the lowest threshold that PRODUCES the measurement it needs to + // say anything at all. Evidence, not the ladder, is what lets it move. + let measured = SURVIVOR_ROUND_MEASURED.with(Cell::get); + let target = if measured { + compute_target_survivals(eden_live_bytes, desired).max(OCCUPANCY_MIN_SURVIVALS) + } else { + OCCUPANCY_MIN_SURVIVALS + }; let next = if target < current { RAISE_STREAK.with(|s| s.set(0)); target @@ -536,46 +796,39 @@ pub(super) fn retune_after_scavenge( RAISE_STREAK.with(|s| s.set(0)); current }; - set_survivals(current, next, eden_live_bytes, "occupancy"); + set_survivals(current, next, eden_live_bytes, "occupancy", evidence); } /// Grow the nursery cap one ×2 step (to at most ×4) when survivor influx -/// exceeds 4% of the current effective cap for two consecutive cycles — -/// objects are surviving because they aren't getting time to die, so a -/// bigger Eden both cuts the collection count and lets them die young. -/// Shrink one step when influx falls below 1% for two consecutive cycles. -/// The 4%/1% band is wide enough that the scale cannot oscillate on a -/// steady workload (growing halves the observed ratio, 4%/2 = 2% > 1%). -fn retune_nursery_cap_scale(eden_live_bytes: usize) { +/// exceeds 4% of the current effective cap for two consecutive cap-due +/// copying minors. Objects are surviving because they are not getting time to +/// die, so a bigger Eden both cuts the collection count and lets them die +/// young. Mortality is not evidence for shrinking the byte/RSS bound. +fn retune_nursery_cap_scale(eden_live_bytes: usize, nursery_cap_rating: NurseryCapRating) { + if !nursery_cap_rating.cap_due { + if crate::gc::gc_diag_enabled() { + eprintln!( + "[gc-tenuring] nursery cap scale: not rated (from_space={} cap={})", + nursery_cap_rating.from_space_bytes, nursery_cap_rating.cap_bytes + ); + } + return; + } let cap = scavenge_nursery_cap_effective_bytes(); let scale = NURSERY_CAP_SCALE.with(Cell::get); if eden_live_bytes > cap / 25 { - CAP_SHRINK_STREAK.with(|s| s.set(0)); if scale < NURSERY_CAP_SCALE_MAX { let streak = CAP_GROW_STREAK.with(|s| s.get()).saturating_add(1); if streak >= RAISE_DEBOUNCE_CYCLES { CAP_GROW_STREAK.with(|s| s.set(0)); NURSERY_CAP_SCALE.with(|s| s.set(scale * 2)); - diag_cap_scale(scale, scale * 2, eden_live_bytes); + diag_cap_scale(scale, scale * 2, eden_live_bytes, nursery_cap_rating); } else { CAP_GROW_STREAK.with(|s| s.set(streak)); } } - } else if eden_live_bytes < cap / 100 { - CAP_GROW_STREAK.with(|s| s.set(0)); - if scale > 1 { - let streak = CAP_SHRINK_STREAK.with(|s| s.get()).saturating_add(1); - if streak >= RAISE_DEBOUNCE_CYCLES { - CAP_SHRINK_STREAK.with(|s| s.set(0)); - NURSERY_CAP_SCALE.with(|s| s.set(scale / 2)); - diag_cap_scale(scale, scale / 2, eden_live_bytes); - } else { - CAP_SHRINK_STREAK.with(|s| s.set(streak)); - } - } } else { CAP_GROW_STREAK.with(|s| s.set(0)); - CAP_SHRINK_STREAK.with(|s| s.set(0)); } } @@ -583,7 +836,7 @@ fn retune_nursery_cap_scale(eden_live_bytes: usize) { /// promote-on-first-copy lock: ≥90% of the Eden bytes the sweep classified /// must have been live. That is the "the aging round would filter nothing" /// proof, measured directly instead of inferred from a survivor round-trip. -const FULL_SEED_LIVE_TENTHS: usize = 9; +const PROMOTE_LOCK_LIVE_TENTHS: usize = 9; /// Would a completed mark-sweep's Eden census justify promote-on-first-copy? /// @@ -607,7 +860,9 @@ const FULL_SEED_LIVE_TENTHS: usize = 9; /// program whose nursery is atypically mostly-live at one sweep promotes one /// Eden's worth of short-lived objects and pays an old-gen reclaim to get them /// back. Exposure is bounded by one nursery cap and by the existing unlock -/// path; requiring BOTH conditions is what keeps it narrow. +/// path; requiring BOTH conditions is what keeps it narrow. This pure census +/// predicate intentionally does not know process phase; the stateful caller +/// adds the same post-startup gate as the survivor-round path. pub(super) fn full_seed_promotes_on_first_copy( eden_live_bytes: usize, eden_dead_bytes: usize, @@ -618,7 +873,7 @@ pub(super) fn full_seed_promotes_on_first_copy( } let classified = eden_live_bytes.saturating_add(eden_dead_bytes); classified > 0 - && eden_live_bytes.saturating_mul(10) >= classified.saturating_mul(FULL_SEED_LIVE_TENTHS) + && eden_live_bytes.saturating_mul(10) >= classified.saturating_mul(PROMOTE_LOCK_LIVE_TENTHS) } /// Feed one finished mark-sweep's Eden census into the loop. `eden_live_bytes` @@ -627,11 +882,17 @@ pub(super) fn full_seed_promotes_on_first_copy( /// /// Callers must exclude budgeted cycles and cycles that ran the conservative /// native-stack scan — see the module header for why those two inputs are not -/// sound liveness measurements. +/// sound liveness measurements. Even a qualifying census is refused until the +/// lock's first two survivor cohorts have been rated, so a startup sweep cannot +/// bypass the survivor path's startup exclusion. pub(super) fn seed_promote_lock_from_sweep(eden_live_bytes: usize, eden_dead_bytes: usize) { let already_locked = PROMOTE_LOCK.with(Cell::get); let desired = desired_survivor_bytes(); - let seeds = full_seed_promotes_on_first_copy(eden_live_bytes, eden_dead_bytes, desired); + let rounds_rated = RATED_ROUNDS.with(Cell::get); + let startup = rounds_rated < STARTUP_RATED_ROUNDS; + let census_qualifies = + full_seed_promotes_on_first_copy(eden_live_bytes, eden_dead_bytes, desired); + let seeds = !startup && census_qualifies; // Diagnostic, not a knob: print the census AND the verdict on every // mark-sweep, including refusals. A policy that silently declines is // indistinguishable from one that never ran (#7024/#7025), and the @@ -644,7 +905,7 @@ pub(super) fn seed_promote_lock_from_sweep(eden_live_bytes: usize, eden_dead_byt eden_live_bytes * 100 / classified }; eprintln!( - "[gc-tenuring] sweep-seed eden_live_bytes={eden_live_bytes} eden_dead_bytes={eden_dead_bytes} live_pct={pct} desired={desired} seeds={seeds} already_locked={already_locked}" + "[gc-tenuring] sweep-seed eden_live_bytes={eden_live_bytes} eden_dead_bytes={eden_dead_bytes} live_pct={pct} desired={desired} census_qualifies={census_qualifies} startup={startup} seeds={seeds} already_locked={already_locked}" ); } if already_locked || !seeds { @@ -656,45 +917,97 @@ pub(super) fn seed_promote_lock_from_sweep(eden_live_bytes: usize, eden_dead_byt RAISE_STREAK.with(|s| s.set(0)); // PREV_COPIED_BYTES is deliberately untouched: it is the survival-rate // lock's denominator, owned by the copying path. - set_survivals(current, 1, eden_live_bytes, "sweep-seed"); + let classified = eden_live_bytes.saturating_add(eden_dead_bytes); + let survival_permille = if classified == 0 { + 0 + } else { + eden_live_bytes.saturating_mul(1000) / classified + }; + set_survivals( + current, + 1, + eden_live_bytes, + "sweep-seed", + TransitionEvidence { + rounds_rated, + streak: PROMOTE_LOCK_STREAK.with(Cell::get), + survival_permille, + // The live Eden cohort is the volume the next copying minor would + // otherwise copy; the sweep reads that denominator one cycle early. + copied_bytes: eden_live_bytes, + startup, + }, + ); } -fn diag_cap_scale(from: u8, to: u8, eden_live_bytes: usize) { +fn diag_cap_scale(from: u8, to: u8, eden_live_bytes: usize, nursery_cap_rating: NurseryCapRating) { if crate::gc::gc_diag_enabled() { eprintln!( - "[gc-tenuring] nursery cap scale {from}x -> {to}x (eden_live_bytes={eden_live_bytes})" + "[gc-tenuring] nursery cap scale {from}x -> {to}x (why=influx cap_due={} from_space={} cap={} eden_live_bytes={eden_live_bytes})", + nursery_cap_rating.cap_due, + nursery_cap_rating.from_space_bytes, + nursery_cap_rating.cap_bytes, ); } } -fn set_survivals(current: u8, next: u8, eden_live_bytes: usize, why: &str) { +#[derive(Clone, Copy)] +struct TransitionEvidence { + rounds_rated: u64, + streak: u8, + survival_permille: usize, + copied_bytes: usize, + startup: bool, +} + +fn set_survivals( + current: u8, + next: u8, + eden_live_bytes: usize, + why: &str, + evidence: TransitionEvidence, +) { if next == current { return; } TENURING_SURVIVALS.with(|s| s.set(next)); if crate::gc::gc_diag_enabled() { + let (copy_pause_us, tenuring_copied_bytes, promote_us, tenuring_promoted_bytes) = + super::instruments::tenuring_price_counters(); eprintln!( - "[gc-tenuring] survivals {} -> {} ({why}, eden_live_bytes={} desired={})", + "[gc-tenuring] survivals {} -> {} ({why}, eden_live_bytes={} desired={} rounds_rated={} streak={} survival_permille={} copied_bytes={} startup={} copy_pause_us={} tenuring_copied_bytes={} promote_us={} tenuring_promoted_bytes={})", current, next, eden_live_bytes, - desired_survivor_bytes() + desired_survivor_bytes(), + evidence.rounds_rated, + evidence.streak, + evidence.survival_permille, + evidence.copied_bytes, + evidence.startup, + copy_pause_us, + tenuring_copied_bytes, + promote_us, + tenuring_promoted_bytes, ); } } #[cfg(test)] pub(super) fn reset_for_test() { - TENURING_SURVIVALS.with(|s| s.set(GC_TENURING_SURVIVALS_MAX)); + TENURING_SURVIVALS.with(|s| s.set(OCCUPANCY_MIN_SURVIVALS)); + SURVIVOR_ROUND_MEASURED.with(|m| m.set(false)); RAISE_STREAK.with(|s| s.set(0)); PROMOTE_LOCK.with(|l| l.set(false)); UNLOCK_STREAK.with(|s| s.set(0)); + RATED_ROUNDS.with(|s| s.set(0)); + PROMOTE_LOCK_STREAK.with(|s| s.set(0)); PREV_COPIED_BYTES.with(|c| c.set(0)); NURSERY_CAP_SCALE.with(|s| s.set(1)); CAP_GROW_STREAK.with(|s| s.set(0)); - CAP_SHRINK_STREAK.with(|s| s.set(0)); MEAN_SURVIVING_OBJECT_BYTES.with(|s| s.set(NURSERY_CAP_REFERENCE_OBJECT_BYTES)); OBJECT_CENSUS_SEEDED.with(|s| s.set(false)); + COPYING_MINOR_COMPLETED.with(|s| s.set(false)); } /// Test-only view of the seed flag (#8122): has any census replaced the @@ -704,13 +1017,49 @@ pub(super) fn object_census_seeded_for_test() -> bool { OBJECT_CENSUS_SEEDED.with(Cell::get) } +#[cfg(test)] +#[path = "tenuring_nursery_cap_evidence_tests.rs"] +mod nursery_cap_evidence_tests; + #[cfg(test)] mod tests { use super::*; const MB: usize = 1024 * 1024; - /// #7929: the constant band must buy a CONSTANT NUMBER OF OBJECTS. + /// Existing tenuring tests predate cap-due rating and intentionally test + /// the tenuring loop with a rated minor. Keep that premise explicit while + /// the fixture module beside this one exercises both rated and skipped + /// minors. + fn retune_after_scavenge( + eden_live_bytes: usize, + eden_copied_bytes: usize, + first_round_live_bytes: usize, + ) { + let cap = scavenge_nursery_cap_effective_bytes(); + super::retune_after_scavenge( + eden_live_bytes, + eden_copied_bytes, + first_round_live_bytes, + NurseryCapRating::from_values(usize::MAX, cap), + ); + } + + /// Advance past the startup boundary with cohorts that visibly die, so + /// tests below can exercise post-startup policy without contributing to + /// the promote-lock streak themselves. + fn finish_startup_with_mortality(desired: usize) { + let cohort = 3 * desired; + retune_after_scavenge(16 * desired, cohort, 0); + for _ in 0..STARTUP_RATED_ROUNDS { + retune_after_scavenge(16 * desired, cohort, 0); + } + assert_eq!(RATED_ROUNDS.with(Cell::get), STARTUP_RATED_ROUNDS); + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 0); + assert!(!PROMOTE_LOCK.with(Cell::get)); + } + + /// #7929: the first tracing band must buy a CONSTANT NUMBER OF OBJECTS. /// /// The discriminating quantity is deliberately `band / mean` (the object /// budget), not "the band changed". A test that only asserted the band @@ -789,15 +1138,17 @@ mod tests { ); note_surviving_object_census(56 * 1000, 1000); - let after_measurement = influx_driven_nursery_cap_bytes(); - assert_eq!(after_measurement, base * 777 / 1000); + assert_eq!(mean_surviving_object_bytes(), 56); + assert_eq!(influx_driven_nursery_cap_bytes(), base); note_surviving_object_census(0, 0); note_surviving_object_census(4096, 0); + assert_eq!(mean_surviving_object_bytes(), 56); assert_eq!( influx_driven_nursery_cap_bytes(), - after_measurement, - "a cycle with no moved objects is a missing measurement, not a zero one" + base, + "a cycle with no moved objects carries the diagnostic mean forward without \ + re-denominating the steady-state byte band" ); reset_for_test(); assert_eq!(influx_driven_nursery_cap_bytes(), base); @@ -846,13 +1197,18 @@ mod tests { /// The tenured-proportional term is representation-invariant by /// cancellation (`tenured_bytes / 2` is `tenured_objects / 2` objects), so - /// the object denomination must apply to the constant band ONLY. If it - /// leaked into `scavenge_nursery_cap_from` the proportional arm would be - /// scaled twice. + /// even the first-minor object denomination must apply to the constant band + /// only. If it leaked into `scavenge_nursery_cap_from` the proportional arm + /// would be scaled twice. #[test] fn only_the_constant_band_is_re_denominated() { reset_for_test(); - note_surviving_object_census(56 * 1000, 1000); + MEAN_SURVIVING_OBJECT_BYTES.with(|mean| mean.set(56)); + OBJECT_CENSUS_SEEDED.with(|seeded| seeded.set(true)); + assert_eq!( + influx_driven_nursery_cap_bytes(), + gc_scavenge_nursery_cap_bytes() * 777 / 1000 + ); let tenured = 512 * MB; assert_eq!( scavenge_nursery_cap_from(influx_driven_nursery_cap_bytes(), tenured), @@ -878,22 +1234,35 @@ mod tests { fn drops_immediately_and_rises_debounced() { reset_for_test(); let desired = desired_survivor_bytes(); - assert_eq!(tenuring_survivals(), 4); - - // Heavy influx: instant drop to 1. + // Power-on is the FLOOR now, not the ceiling (startup follow-up): the + // ladder may not claim a lifetime in either direction without evidence. + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + + // Give the loop its evidence, because this test is about the ladder's + // ASYMMETRY and not about the startup gate. Two cycles with a cohort + // that fully dies: the second rates the first, so a survivor round has + // been measured, and 0 % survival keeps the lock out of it. + retune_after_scavenge(desired * 2, 3 * desired, 0); + retune_after_scavenge(desired * 2, 3 * desired, 0); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + + // Heavy influx: instant drop, no debounce. #9851 changed the FLOOR this + // lands on (2, not 1 — the occupancy rule may not claim a lifetime), not + // the asymmetry this test is named for: 4 -> 2 in one cycle is the same + // "drops immediately" property that 4 -> 1 was. retune_after_scavenge(desired * 2, 0, 0); - assert_eq!(tenuring_survivals(), 1); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); // One quiet cycle: no rise yet (debounce). retune_after_scavenge(0, 0, 0); - assert_eq!(tenuring_survivals(), 1); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); // Second quiet cycle: rise by exactly one step, not to the target. retune_after_scavenge(0, 0, 0); - assert_eq!(tenuring_survivals(), 2); + assert_eq!(tenuring_survivals(), 3); // Heavy again: streak resets and threshold drops straight back. retune_after_scavenge(desired * 2, 0, 0); - assert_eq!(tenuring_survivals(), 1); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); // Sustained quiet recovers to the ceiling two cycles per step. for _ in 0..6 { @@ -911,10 +1280,13 @@ mod tests { // every cycle even while the cap scale walks up underneath it. An // influx only marginally above the base desired is a different case: // the growing cap re-classifies it as moderate, which is correct. + // #9851: the fixed point is now the occupancy floor (2) rather than 1. + // Fixed-POINTNESS is what this test protects — no oscillation while the + // cap scale walks up underneath — and that is unchanged. let heavy = gc_scavenge_nursery_cap_bytes(); for _ in 0..10 { retune_after_scavenge(heavy, 0, 0); - assert_eq!(tenuring_survivals(), 1); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); } assert_eq!( scavenge_nursery_cap_effective_bytes(), @@ -924,6 +1296,260 @@ mod tests { reset_for_test(); } + /// #9851, both halves of the rule in one test, in the #7909 two-phase shape + /// so the decline is ATTRIBUTED rather than merely absent. + /// + /// Phase 1 — the occupancy rule alone, on an influx far above `desired`, + /// must stop at 2 and NOT claim promote-on-first-copy. That is the whole + /// change: 2 is the lowest threshold that still puts a cohort through the + /// survivor space, so the loop keeps producing the measurement that could + /// refute it. + /// + /// Phase 2 — the same heap, once a substantial cohort HAS come back fully + /// alive, must still reach 1 through the survival-rate lock. The rule + /// removes an unmeasured conclusion, not the measured one, and this half is + /// what makes it self-limiting rather than a blanket floor. + /// + /// Sabotage: drop the `.max(OCCUPANCY_MIN_SURVIVALS)` in + /// `retune_after_scavenge` and phase 1 fails (the loop reports 1 with no + /// evidence). Drop the lock instead and phase 2 fails. + #[test] + fn occupancy_alone_never_claims_promote_on_first_copy_but_the_lock_still_can() { + reset_for_test(); + let d = desired_survivor_bytes(); + + // Phase 1: influx 16x the desired survivor size — the occupancy formula + // computes 1 (integer division: 1 + desired/influx). No cohort has been + // rated yet, so there is NO lifetime evidence on this heap. + assert_eq!( + compute_target_survivals(16 * d, d), + 1, + "precondition: the occupancy ARITHMETIC still computes 1 — this \ + change clamps what the loop may do with it, not the formula" + ); + for _ in 0..5 { + retune_after_scavenge(16 * d, 0, 0); + assert_eq!( + tenuring_survivals(), + OCCUPANCY_MIN_SURVIVALS, + "occupancy measures SPACE and must not conclude promote-on-first-copy" + ); + } + assert!( + !PROMOTE_LOCK.with(Cell::get), + "and it must not have taken the lock's route to get there" + ); + + // Phase 2: now substantial cohorts go through the survivor space and + // come back fully alive. Startup evidence is excluded, then exactly K + // steady ratings must still reach 1. + retune_after_scavenge(16 * d, 3 * d, 0); + for _ in 0..STARTUP_RATED_ROUNDS { + retune_after_scavenge(16 * d, 3 * d, 3 * d); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + } + for round in 1..=PROMOTE_LOCK_RATED_ROUNDS { + retune_after_scavenge(16 * d, 3 * d, 3 * d); + if round < PROMOTE_LOCK_RATED_ROUNDS { + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + } + } + assert_eq!( + tenuring_survivals(), + 1, + "K steady substantial cohorts that fully survive must still lock \ + promote-on-first-copy" + ); + assert!(PROMOTE_LOCK.with(Cell::get), "...through the lock"); + reset_for_test(); + } + + /// STARTUP FOLLOW-UP — the occupancy rule may not claim the CEILING either. + /// + /// `1 + desired / influx` returns the ceiling for a tiny influx and for a + /// zero one, so on the first minors of a process — heap nearly empty, no + /// cohort ever followed — occupancy claims the maximum. That is the same + /// category error as claiming 1: a statement about LIFETIME derived from a + /// measurement of SPACE, made before any evidence exists. It is also the + /// expensive direction, because every survivor is then copied up to three + /// times before it may be promoted. + /// + /// Measured on the compiled claude-code TUI, this was the WHOLE difference + /// between the adaptive loop and a pinned threshold: a single startup + /// excursion `4 -> 2 (occupancy) -> 1 (lock)` inside turn 1, nothing + /// afterwards, worth +0.35..0.45 s at 3300 characters and +50 % at 400. + /// + /// Sabotage: delete the `SURVIVOR_ROUND_MEASURED` gate in + /// `retune_after_scavenge` (or restore the power-on value to + /// `GC_TENURING_SURVIVALS_MAX`) and phase 1 fails — the loop reports the + /// ceiling on a heap where nothing has ever been rated. + #[test] + fn occupancy_may_not_claim_the_ceiling_before_any_round_is_measured() { + reset_for_test(); + let d = desired_survivor_bytes(); + + // Precondition: the ARITHMETIC still says "ceiling" for a startup-sized + // influx. This change gates what the loop may do with that, exactly as + // #9851 did at the other end of the range. + assert_eq!(compute_target_survivals(0, d), GC_TENURING_SURVIVALS_MAX); + assert_eq!( + compute_target_survivals(d / 64, d), + GC_TENURING_SURVIVALS_MAX + ); + + // Phase 1: power-on, then many startup-shaped minors — tiny influx, + // nothing copied, so nothing rateable. The loop must sit at the floor + // and never climb. + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + for _ in 0..8 { + retune_after_scavenge(d / 64, 0, 0); + assert_eq!( + tenuring_survivals(), + OCCUPANCY_MIN_SURVIVALS, + "no survivor round has been rated, so occupancy has no lifetime \ + evidence and may not leave the floor" + ); + } + assert!( + !PROMOTE_LOCK.with(Cell::get), + "and it must not have reached the floor via the lock either" + ); + + // Phase 2: once a cohort has actually gone through the survivor space + // and been followed, the ladder is allowed to move again. A cohort that + // fully dies keeps the lock out, so what is observed here is the + // occupancy rule being re-enabled and nothing else. + retune_after_scavenge(d / 64, 3 * d, 0); + retune_after_scavenge(d / 64, 3 * d, 0); + for _ in 0..8 { + retune_after_scavenge(d / 64, 0, 0); + } + assert_eq!( + tenuring_survivals(), + GC_TENURING_SURVIVALS_MAX, + "with a round measured and the influx quiet, the debounced rise must \ + still reach the ceiling — the gate delays the claim until there is \ + evidence, it does not remove the ladder" + ); + reset_for_test(); + } + + /// #9851: a cohort that DIES in its survivor round must keep the loop at the + /// occupancy floor rather than being locked to 1 — the case cc actually is. + /// Measured there: 26.1 % of each cohort dies in one survivor round, in + /// steady state, on 393 samples; the lock needs >=90 % survival, so it + /// correctly stays out. + /// + /// The arguments are the FRESH COHORT's intake and survival (#9851 + /// follow-up). Fed the whole survivor space instead — which is what the + /// call site used to pass — this same heap locks, because above a threshold + /// of 2 that space also holds objects already selected for longevity. That + /// is not a hypothetical: on cc the clamp alone left 85 % of promotion at + /// S=1, reached through this lock 8-12 times per four-turn run. + #[test] + fn a_cohort_that_dies_in_its_round_holds_at_the_occupancy_floor() { + reset_for_test(); + let d = desired_survivor_bytes(); + // Heavy influx (occupancy says 1) AND a substantial cohort of which + // ~26 % dies — cc's steady state, in miniature. + for _ in 0..8 { + retune_after_scavenge(16 * d, 4 * d, 3 * d); + } + assert!( + !PROMOTE_LOCK.with(Cell::get), + "74 % survival is below the lock's 90 % bar: the lock must stay out" + ); + assert_eq!( + tenuring_survivals(), + OCCUPANCY_MIN_SURVIVALS, + "so the loop holds at the occupancy floor and keeps aging the cohort" + ); + reset_for_test(); + } + + /// STARTUP EVIDENCE LOCK: two fully-surviving startup cohorts may not + /// contribute to the steady-state streak. The first post-startup survivor + /// therefore leaves the streak at one and S at the occupancy floor. + /// + /// Sabotage: remove `!startup` from the qualifying predicate. The two + /// startup ratings then count as the first two streak members and the final + /// call reaches K, latching S=1 and failing this test. + #[test] + fn startup_shaped_survivors_do_not_contribute_to_the_lock_streak() { + reset_for_test(); + let d = desired_survivor_bytes(); + let cohort = 3 * d; + + // Establish the denominator, then rate two startup cohorts at 100%. + retune_after_scavenge(16 * d, cohort, 0); + for _ in 0..STARTUP_RATED_ROUNDS { + retune_after_scavenge(16 * d, cohort, cohort); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + } + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 0); + + // One genuinely post-startup rating is not a K-round window. + retune_after_scavenge(16 * d, cohort, cohort); + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 1); + assert!(!PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + reset_for_test(); + } + + /// Kill condition from the design: the lock remains reachable, in bounded + /// time, for a workload whose steady cohorts genuinely do not die. + /// + /// Sabotage: raise `PROMOTE_LOCK_RATED_ROUNDS` or fail to advance the + /// streak; the exact-K final assertion fails. + #[test] + fn k_steady_fully_surviving_rounds_latch_promote_on_first_copy() { + reset_for_test(); + let d = desired_survivor_bytes(); + let cohort = 3 * d; + finish_startup_with_mortality(d); + + for round in 1..=PROMOTE_LOCK_RATED_ROUNDS { + retune_after_scavenge(16 * d, cohort, cohort); + if round < PROMOTE_LOCK_RATED_ROUNDS { + assert!(!PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + } + } + assert!(PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), 1); + reset_for_test(); + } + + /// A mortality observation breaks consecutiveness even when the process + /// has accumulated K qualifying observations in total. + /// + /// Sabotage: retain the streak on a rated below-bar round. The last survivor + /// becomes the Kth streak member and this test observes an S=1 latch. + #[test] + fn mortality_inside_the_steady_window_resets_the_lock_streak() { + reset_for_test(); + let d = desired_survivor_bytes(); + let cohort = 3 * d; + finish_startup_with_mortality(d); + + for _ in 0..PROMOTE_LOCK_RATED_ROUNDS - 1 { + retune_after_scavenge(16 * d, cohort, cohort); + } + assert_eq!( + PROMOTE_LOCK_STREAK.with(Cell::get), + PROMOTE_LOCK_RATED_ROUNDS - 1 + ); + + retune_after_scavenge(16 * d, cohort, cohort / 2); + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 0); + retune_after_scavenge(16 * d, cohort, cohort); + + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 1); + assert!(!PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + reset_for_test(); + } + #[test] fn survival_rate_lock_breaks_a_saturated_pipeline() { reset_for_test(); @@ -938,11 +1564,13 @@ mod tests { 2, "first cycle has no prior intake to rate, so occupancy decides" ); - retune_after_scavenge(influx, 3 * d, 3 * d); + for _ in 0..STARTUP_RATED_ROUNDS + u64::from(PROMOTE_LOCK_RATED_ROUNDS) { + retune_after_scavenge(influx, 3 * d, 3 * d); + } assert_eq!( tenuring_survivals(), 1, - "a substantial intake that fully survives its round must lock promote-on-first-copy" + "the bounded steady window must lock" ); // The lock holds through occupancy readings that would say S=2. for _ in 0..5 { @@ -963,7 +1591,7 @@ mod tests { } #[test] - fn cap_scale_grows_on_heavy_influx_and_shrinks_when_quiet() { + fn cap_scale_grows_on_heavy_influx_and_stays_earned() { reset_for_test(); let base = gc_scavenge_nursery_cap_bytes(); assert_eq!(scavenge_nursery_cap_effective_bytes(), base); @@ -981,15 +1609,13 @@ mod tests { retune_after_scavenge(base, 0, 0); } assert_eq!(scavenge_nursery_cap_effective_bytes(), base * 4); - // Quiet influx walks back one step at a time. - for _ in 0..2 { - retune_after_scavenge(0, 0, 0); - } - assert_eq!(scavenge_nursery_cap_effective_bytes(), base * 2); - for _ in 0..2 { + // Mortality does not shrink the RSS-bounded scale. In the absence of + // a readable pressure-clamp predicate, the earned ceiling remains. + for _ in 0..4 { retune_after_scavenge(0, 0, 0); } - assert_eq!(scavenge_nursery_cap_effective_bytes(), base); + assert_eq!(scavenge_nursery_cap_effective_bytes(), base * 4); + assert_eq!(CAP_GROW_STREAK.with(Cell::get), 0); reset_for_test(); } @@ -999,15 +1625,23 @@ mod tests { let d = desired_survivor_bytes(); // Medium-lived objects: a substantial intake of which only half // survives its survivor round. Aging is filtering — the lock must - // stay out and the occupancy ladder must decide. + // stay out and the occupancy ladder must age from the power-on floor. + let mut seen = Vec::new(); for _ in 0..6 { retune_after_scavenge(d / 2, d / 2, d / 4); assert!( - tenuring_survivals() >= 3, - "a cohort that dies in the survivor space must keep aging (got {})", - tenuring_survivals() + !PROMOTE_LOCK.with(Cell::get), + "50% survival is below the lock's 90% bar" ); + seen.push(tenuring_survivals()); } + assert_eq!( + seen, + [2, 2, 3, 3, 3, 3], + "power-on is the floor now, not the ceiling: after a survivor round \ + is measured, the debounced occupancy ladder must keep the dying \ + cohort aging rather than claim promote-on-first-copy" + ); reset_for_test(); } @@ -1015,7 +1649,7 @@ mod tests { // // #7596 added `max(influx_driven, old_gen_reclaimable / 2)` to // `scavenge_nursery_cap_effective_bytes` with no test of its own. The - // sibling `cap_scale_grows_on_heavy_influx_and_shrinks_when_quiet` looks + // sibling `cap_scale_grows_on_heavy_influx_and_stays_earned` looks // like coverage but is not: it asserts against the effective cap in a // unit-test thread whose old-gen is ~empty, so the proportional term // contributes 0 and the test stays green with that term deleted. These @@ -1079,26 +1713,49 @@ mod tests { // ── #7598's mark-sweep seed ───────────────────────────────────────── // - // The lock above cannot engage before the SECOND copying minor. These - // exercise the seed that reads the same proof off a completed mark-sweep, - // one collection earlier. + // The survivor lock needs a previous copying minor. These exercise the seed + // that reads the same proof off a completed mark-sweep after startup, one + // collection earlier, plus the startup refusal itself. + /// A sweep census may satisfy both of the seed's existing conditions and + /// still be forbidden to decide S=1 while the process is in startup. + /// + /// Sabotage: remove the startup conjunct from `seeds`; this qualifying + /// census immediately latches S=1 and fails both final assertions. #[test] - fn sweep_seed_decides_before_the_first_copying_minor_snapshots_the_threshold() { + fn sweep_seed_cannot_latch_from_a_startup_census() { + reset_for_test(); + let d = desired_survivor_bytes(); + let eden_live = 4 * d; + assert!( + full_seed_promotes_on_first_copy(eden_live, 0, d), + "precondition: both original sweep-seed conditions must qualify" + ); + + seed_promote_lock_from_sweep(eden_live, 0); + + assert!(!PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + reset_for_test(); + } + + #[test] + fn sweep_seed_decides_before_a_post_startup_copying_minor_snapshots_the_threshold() { // `copying.rs` snapshots `tenuring_survivals()` in - // `CopyingNurseryCollector::new`, so the only value that can change - // what the first big minor does is the one standing BEFORE any - // `retune_after_scavenge` for that cycle has run. That is precisely - // what the survival-rate lock cannot reach and this seed can. + // `CopyingNurseryCollector::new`; after startup, a completed sweep may + // still decide what the next copying minor does one cycle earlier than + // a survivor round-trip. reset_for_test(); let d = desired_survivor_bytes(); let eden_live = d * 4; assert_eq!( tenuring_survivals(), - 4, - "with no input the loop is at the ceiling: the wasted copy state" + OCCUPANCY_MIN_SURVIVALS, + "power-on is the floor now, not the ceiling: with no lifetime \ + evidence the loop may not claim either extreme" ); + finish_startup_with_mortality(d); seed_promote_lock_from_sweep(eden_live, eden_live / 50); assert_eq!( tenuring_survivals(), @@ -1118,6 +1775,7 @@ mod tests { let d = desired_survivor_bytes(); let eden_live = d * 4; let eden_dead = eden_live * 9; + finish_startup_with_mortality(d); assert_eq!( compute_target_survivals(eden_live, d), 1, @@ -1127,8 +1785,9 @@ mod tests { seed_promote_lock_from_sweep(eden_live, eden_dead); assert_eq!( tenuring_survivals(), - 4, - "10% Eden survival must not seed promote-on-first-copy" + OCCUPANCY_MIN_SURVIVALS, + "10% Eden survival must leave the loop at the power-on floor, not \ + seed promote-on-first-copy by claiming a threshold below 2" ); reset_for_test(); } @@ -1141,8 +1800,12 @@ mod tests { // reading, and seeding off it would lock every program at S=1. reset_for_test(); let d = desired_survivor_bytes(); + finish_startup_with_mortality(d); seed_promote_lock_from_sweep(d / 8, 0); - assert_eq!(tenuring_survivals(), 4); + // Unchanged from power-on, which is the floor now rather than the + // ceiling (startup follow-up). The property under test is that the + // sweep seed REFUSED — it left the threshold where it found it. + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); reset_for_test(); } @@ -1174,6 +1837,7 @@ mod tests { // differently at S=1 than at S=4 and would oscillate. reset_for_test(); let d = desired_survivor_bytes(); + finish_startup_with_mortality(d); seed_promote_lock_from_sweep(d * 4, 0); assert_eq!(tenuring_survivals(), 1); diff --git a/crates/perry-runtime/src/gc/tenuring_nursery_cap_evidence_tests.rs b/crates/perry-runtime/src/gc/tenuring_nursery_cap_evidence_tests.rs new file mode 100644 index 0000000000..3023157fa2 --- /dev/null +++ b/crates/perry-runtime/src/gc/tenuring_nursery_cap_evidence_tests.rs @@ -0,0 +1,200 @@ +use super::super::policy::ScavengeNurseryCapTestGuard; +use super::*; + +type FixtureARow = (usize, usize, usize, usize, usize, bool); + +/// TN5 unset, 25 copying minors. Columns are `(from_space_bytes, +/// nursery_cap_bytes, survival_permille, copied_bytes, +/// mean_surviving_object_bytes_after, cap_due)`. +const FIXTURE_A: [FixtureARow; 25] = [ + (15_728_712, 15_367_929, 839, 13_206_480, 77, true), + (17_407_296, 16_777_216, 936, 3_214_304, 84, true), + (33_624_624, 33_554_432, 757, 22_240_712, 133, true), + (34_829_760, 33_554_432, 396, 5_672_776, 84, true), + (39_313_552, 67_108_864, 506, 14_979_552, 118, false), + (67_407_992, 67_108_864, 261, 3_267_840, 110, true), + (67_230_456, 67_108_864, 60, 1_181_376, 61, true), + (57_803_984, 56_841_207, 30, 1_182_328, 40, true), + (37_882_104, 37_245_419, 39, 725_224, 47, true), + (44_762_032, 43_754_979, 28, 769_240, 43, true), + (40_614_704, 40_063_991, 16, 649_840, 36, true), + (34_204_000, 33_554_432, 26, 921_496, 38, true), + (35_524_488, 35_366_371, 72, 2_558_256, 80, true), + (67_572_776, 67_108_864, 56, 1_517_184, 63, true), + (59_188_240, 58_720_256, 33, 970_776, 45, true), + (42_913_584, 41_943_040, 30, 1_330_288, 39, true), + (36_981_552, 36_305_895, 86, 3_185_688, 84, true), + (56_662_480, 67_108_864, 68, 815_448, 68, false), + (63_730_280, 63_350_767, 22, 1_029_672, 37, true), + (34_583_744, 34_426_847, 32, 1_119_864, 39, true), + (36_427_200, 36_305_895, 19, 637_536, 40, true), + (37_347_336, 37_245_419, 90, 3_395_304, 73, true), + (67_358_392, 67_108_864, 60, 1_053_088, 62, true), + (58_724_472, 57_780_731, 24, 893_864, 38, true), + (35_510_472, 35_366_371, 31, 1_118_240, 39, true), +]; + +type FixtureBRow = (usize, usize, usize); + +/// NS3 n16, 26 copying minors. Columns are `(from_space_bytes, +/// nursery_cap_bytes, eden_live_bytes)`. The four transition rows retain the +/// exact diagnostic value; other rows use `survival_permille * from_space / +/// 1000`, as specified by the fixture contract. +const FIXTURE_B: [FixtureBRow; 26] = [ + (15_733_256, 15_367_929, 13_200_201), + (17_401_448, 16_777_216, 3_208_856), + (34_602_744, 33_554_432, 25_294_605), + (35_460_656, 33_554_432, 9_684_456), + (68_156_728, 67_108_864, 6_815_672), + (67_837_240, 67_108_864, 542_697), + (59_767_832, 59_592_671, 46_176), + (34_602_608, 33_554_432, 34_602), + (31_499_296, 33_554_432, 2_803_437), + (34_086_960, 33_554_432, 2_590_608), + (34_599_896, 33_554_432, 32_928), + (20_971_160, 20_287_508, 41_942), + (20_955_528, 20_479_960, 125_733), + (22_158_808, 21_317_124, 3_213_027), + (22_002_944, 21_367_748, 3_102_415), + (23_068_384, 22_920_104, 23_068), + (23_068_336, 22_938_412, 23_068), + (23_068_344, 22_956_972, 23_068), + (24_160_512, 23_127_680, 459_049), + (19_350_992, 19_195_148, 3_057_456), + (19_846_320, 19_196_388, 3_155_564), + (20_971_416, 20_775_896, 20_971), + (20_971_392, 20_793_888, 20_971), + (21_006_848, 20_793_888, 21_006), + (22_047_208, 21_018_128, 44_094), + (21_023_432, 21_018_128, 315_351), +]; + +fn cap_scale() -> u8 { + NURSERY_CAP_SCALE.with(Cell::get) +} + +fn cap_grow_streak() -> u8 { + CAP_GROW_STREAK.with(Cell::get) +} + +fn seed_allocation_mean(mean: usize) { + assert!(!COPYING_MINOR_COMPLETED.with(Cell::get)); + MEAN_SURVIVING_OBJECT_BYTES.with(|cell| cell.set(mean)); + OBJECT_CENSUS_SEEDED.with(|cell| cell.set(true)); +} + +fn replay_fixture_b() -> Vec { + let mut scales = Vec::with_capacity(FIXTURE_B.len()); + for &(from_space, cap, eden_live) in &FIXTURE_B { + note_surviving_object_census(NURSERY_CAP_REFERENCE_OBJECT_BYTES, 1); + super::retune_after_scavenge( + eden_live, + 0, + 0, + NurseryCapRating::from_values(from_space, cap), + ); + scales.push(cap_scale()); + } + scales +} + +/// Sabotage: let the survivor census feed the steady cap again; the expected +/// 64 MiB suffix reproduces TN5's 32--42 MiB dips instead. +#[test] +fn steady_state_cap_is_not_object_denominated_after_first_copying_minor() { + reset_for_test(); + let _cap_guard = ScavengeNurseryCapTestGuard::due_at_bytes(1); + let base = gc_scavenge_nursery_cap_bytes(); + seed_allocation_mean(66); + + let mut effective_caps = Vec::with_capacity(FIXTURE_A.len()); + for &(from_space, recorded_cap, survival_permille, copied_bytes, mean_after, cap_due) in + &FIXTURE_A + { + assert!(survival_permille <= 1000); + assert_eq!(cap_due, from_space >= recorded_cap); + effective_caps.push(influx_driven_nursery_cap_bytes()); + + note_surviving_object_census(mean_after, 1); + super::retune_after_scavenge( + copied_bytes, + 0, + 0, + NurseryCapRating::from_values(from_space, recorded_cap), + ); + } + + assert_eq!(effective_caps[0], 15_367_929); + assert_eq!(&effective_caps[1..4], &[base, base * 2, base * 2]); + for (row, &effective_cap) in FIXTURE_A.iter().zip(&effective_caps).skip(4) { + if row.5 { + assert_eq!(effective_cap, base * NURSERY_CAP_SCALE_MAX as usize); + } + } + assert_eq!(cap_scale(), NURSERY_CAP_SCALE_MAX); + reset_for_test(); +} + +/// Sabotage: replace the first-minor regime with the steady byte band; the +/// 40 B mean receives the raw 1000-per-mille band instead of today's 555. +#[test] +fn first_minor_keeps_the_object_denomination() { + reset_for_test(); + let _cap_guard = ScavengeNurseryCapTestGuard::due_at_bytes(1); + seed_allocation_mean(40); + + // 40 * 1000 / 72 = 555. The unchanged 500-per-mille floor begins below + // 36 B; claiming 500 here would silently re-tune the preserved regime. + assert_eq!(nursery_cap_object_scale_permille(40), 555); + assert_eq!(influx_driven_nursery_cap_bytes(), 9_311_354); + assert!(!COPYING_MINOR_COMPLETED.with(Cell::get)); + reset_for_test(); +} + +/// Sabotage: restore the `< cap / 100` mortality branch; the low-survival +/// suffix lowers the scale instead of leaving it at the ceiling. +#[test] +fn nursery_cap_scale_does_not_shrink_on_low_mortality() { + reset_for_test(); + let _cap_guard = ScavengeNurseryCapTestGuard::due_at_bytes(1); + let scales = replay_fixture_b(); + + assert_eq!(scales[3], NURSERY_CAP_SCALE_MAX); + assert!(scales[3..].iter().all(|&scale| scale == 4)); + assert_eq!(scales[25], 4); + reset_for_test(); +} + +/// Sabotage: remove or re-time the influx grow branch; this exact transition +/// vector changes at row 2 or row 4. +#[test] +fn nursery_cap_scale_still_grows_on_influx() { + reset_for_test(); + let _cap_guard = ScavengeNurseryCapTestGuard::due_at_bytes(1); + let scales = replay_fixture_b(); + let mut expected = vec![4; FIXTURE_B.len()]; + expected[0] = 1; + expected[1] = 2; + expected[2] = 2; + + assert_eq!(scales, expected, "growth must occur only at rows 2 and 4"); + reset_for_test(); +} + +/// Sabotage: delete the cap-due return; two high-influx forced minors advance +/// the streak and grow the scale to 2. +#[test] +fn non_cap_due_minor_does_not_rate_the_scale() { + reset_for_test(); + let _cap_guard = ScavengeNurseryCapTestGuard::due_at_bytes(1); + note_surviving_object_census(NURSERY_CAP_REFERENCE_OBJECT_BYTES, 1); + let base = gc_scavenge_nursery_cap_bytes(); + let not_due = NurseryCapRating::from_values(base - 1, base); + + super::retune_after_scavenge(base, 0, 0, not_due); + super::retune_after_scavenge(base, 0, 0, not_due); + + assert_eq!(cap_scale(), 1); + assert_eq!(cap_grow_streak(), 0); + reset_for_test(); +} diff --git a/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs b/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs index 9abfa5a6e2..8a48c5a176 100644 --- a/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs +++ b/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs @@ -27,22 +27,24 @@ fn heavy_influx_lowers_threshold_and_promotes_next_cycle() { let _guard = CopyingNurseryTestGuard::new(SLOTS); assert_eq!( crate::gc::tenuring::tenuring_survivals(), - GC_COPY_PROMOTION_SURVIVALS, - "guard must start every test at the power-on threshold" + crate::gc::tenuring::OCCUPANCY_MIN_SURVIVALS, + "guard must start every test at the power-on floor, not the ceiling" ); fill_slots_with_heavy_influx(); let before = (js_shadow_slot_get(0) & POINTER_MASK) as usize; assert!(crate::arena::pointer_in_nursery(before)); - // Cycle 1 runs at the power-on threshold: the cohort is copied into a - // survivor space (ages to 1), and its influx re-tunes the threshold down - // to promote-on-first-copy. + // Cycle 1 runs at the power-on floor: the cohort is copied into a survivor + // space (ages to 1), and heavy influx must not take occupancy below that + // floor by claiming promote-on-first-copy without lifetime evidence. let _ = gc_collect_minor(); assert_eq!( crate::gc::tenuring::tenuring_survivals(), - 1, - "a >desired Eden survivor influx must drop the threshold to 1" + crate::gc::tenuring::OCCUPANCY_MIN_SURVIVALS, + "a >desired Eden survivor influx must drop the threshold to the \ + occupancy floor (#9851: the occupancy rule measures space and may not \ + claim promote-on-first-copy, which is a claim about lifetime)" ); let after_first = (js_shadow_slot_get(0) & POINTER_MASK) as usize; assert!( @@ -51,7 +53,11 @@ fn heavy_influx_lowers_threshold_and_promotes_next_cycle() { ); // Cycle 2 promotes the whole cohort instead of re-copying it: this is - // the ping-pong the adaptive threshold exists to break. + // the ping-pong the adaptive threshold exists to break. #9851 did NOT + // weaken this half — the cohort was copied once in cycle 1, so its + // `next_age` here is 2, which still satisfies `next_age >= 2`. The test's + // named invariant ("lowers threshold AND promotes next cycle") is intact; + // only the literal threshold moved. let _ = gc_collect_minor(); for slot in 0..SLOTS { let addr = (js_shadow_slot_get(slot) & POINTER_MASK) as usize; @@ -62,8 +68,8 @@ fn heavy_influx_lowers_threshold_and_promotes_next_cycle() { } } -/// #7929: a real copying minor must feed its move census into the nursery -/// band's object denomination. +/// #7929: a real copying minor must feed its move census into diagnostics and +/// end the first-minor object denomination. /// /// The pure-function coverage lives in `gc::tenuring::tests`; that coverage /// passes with the `copying.rs` call site deleted, which is exactly the "the @@ -119,11 +125,9 @@ fn copying_minor_feeds_the_object_denomination_census() { "fixture must exercise the SCALING arm, not the one-sided clamp (mean {recorded} B)" ); - // And the band moved with it, proportionally. - assert_eq!( - crate::gc::tenuring::influx_driven_nursery_cap_bytes(), - base * crate::gc::tenuring::nursery_cap_object_scale_permille(recorded) / 1000 - ); + // The completed copying minor ends the tracing regime: its measured mean + // stays observable, while the steady-state band returns to bytes. + assert_eq!(crate::gc::tenuring::influx_driven_nursery_cap_bytes(), base); } /// #8122: BEFORE any copying minor has run, once the young generation is @@ -215,7 +219,13 @@ fn quiet_cycles_restore_power_on_threshold_debounced() { fill_slots_with_heavy_influx(); let _ = gc_collect_minor(); - assert_eq!(crate::gc::tenuring::tenuring_survivals(), 1); + // #9851: the occupancy floor, not 1. What this test protects — a DEBOUNCED + // restore, at most one step per cycle, ending at the power-on threshold — + // is asserted structurally below and is unchanged. + assert_eq!( + crate::gc::tenuring::tenuring_survivals(), + crate::gc::tenuring::OCCUPANCY_MIN_SURVIVALS + ); // Promote the cohort out of the nursery so later cycles are quiet. let _ = gc_collect_minor(); diff --git a/crates/perry-runtime/src/gc/tests/copying/promoted_remembered_7803.rs b/crates/perry-runtime/src/gc/tests/copying/promoted_remembered_7803.rs index 45c6c25f82..18963db4af 100644 --- a/crates/perry-runtime/src/gc/tests/copying/promoted_remembered_7803.rs +++ b/crates/perry-runtime/src/gc/tests/copying/promoted_remembered_7803.rs @@ -62,6 +62,8 @@ fn young_padded_closure_capturing(bits: u64) -> usize { #[test] fn drain_promoted_parent_keeps_its_young_child_edge_remembered() { let _guard = CopyingNurseryTestGuard::new(1); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); // parent captures a young leaf; intermediate captures parent. Only the // INTERMEDIATE is rooted, so the parent is reached — and, on the @@ -95,9 +97,8 @@ fn drain_promoted_parent_keeps_its_young_child_edge_remembered() { deref(capture_bits_of(spacer)) }; - // Age everyone to the brink of promotion (power-on threshold: promote on - // the fourth survival — pinned by - // `test_copying_minor_promotes_survivor_on_fourth_survival`). + // Age everyone to the explicitly pinned promotion boundary: the fourth + // survival. This test exercises drain promotion at S=4, not power-on. for _ in 0..3 { let _ = gc_collect_minor(); } diff --git a/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs b/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs index aed0b6c681..b6104a63f5 100644 --- a/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs +++ b/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs @@ -3,6 +3,8 @@ use super::*; #[test] fn test_copying_minor_promotes_survivor_on_fourth_survival() { let _guard = CopyingNurseryTestGuard::new(1); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let child = young_leaf(); js_shadow_slot_set(0, ptr_bits(child)); @@ -53,6 +55,8 @@ fn test_copying_minor_preserves_old_page_accounting_for_defrag_policy() { pinned_header: std::ptr::null_mut(), }; let _guard = CopyingNurseryTestGuard::new(1); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); clear_marks(); clear_mark_seeds(); @@ -176,6 +180,8 @@ fn test_copying_minor_preserves_old_page_accounting_for_defrag_policy() { #[test] fn test_copying_minor_sticky_old_to_survivor_edge_promotes_on_fourth_cycle() { let _guard = CopyingNurseryTestGuard::new(0); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let child = young_leaf(); let (old_arr, elements) = unsafe { alloc_old_test_array(1) }; unsafe { @@ -964,6 +970,8 @@ fn test_movable_regexp_evacuation_migrates_all_address_owned_state() { #[test] fn test_copied_minor_promotable_census_filtered_walk_matches_unfiltered() { let _guard = CopyingNurseryTestGuard::new(1); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let child = young_leaf(); js_shadow_slot_set(0, ptr_bits(child)); @@ -1065,3 +1073,85 @@ fn nursery_regexp_that_dies_young_is_finalized_by_the_copied_minor() { ); js_shadow_slot_set(0, 0); } + +/// #9851 follow-up — THE PREMISE OF THE LOCK REWIRE, on a real heap. +/// +/// The survival-rate lock used to rate `survivor_live_bytes` (every live byte +/// leaving the from-survivor space, of any age) against the previous cycle's +/// whole `copied_bytes`. Those two scopes match — the survivor spaces are a +/// strict semispace pair, so the from-space holds exactly what the last cycle +/// copied — and the ratio is well-formed. What is wrong is *which population* +/// it rates, and that is chosen by the threshold the lock itself sets: at a +/// threshold of 2 the space holds one fresh cohort, at 3 or 4 it also holds +/// objects that have already survived a round and are therefore selected for +/// longevity. +/// +/// This test pins the fact that makes the rewire meaningful rather than a +/// rename: **at a threshold above 2 the whole-space number and the fresh-cohort +/// number are different numbers**, with the aged resident in the first and not +/// in the second. On cc that difference is the whole finding — the aggregate +/// clears the lock's 90 % bar while a fresh cohort survives at 74 %. +/// +/// Shape: at an explicitly pinned threshold above 2 (promote on the 4th +/// survival) two rooted objects are introduced one cycle apart, so by the +/// third minor the from-survivor space holds one age-2 object and one age-1 +/// object. +#[test] +fn the_survivor_space_and_the_fresh_cohort_are_different_numbers_above_threshold_two() { + // TWO shadow slots: the test needs two independently rooted objects + // introduced one cycle apart, so that the survivor space holds two age + // classes at once. With one slot B is unrooted, dies immediately, and the + // fresh-cohort number is trivially zero. + let _guard = CopyingNurseryTestGuard::new(2); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); + + // Cycle 1: A enters the survivor space from Eden. The from-survivor space + // was empty, so both numbers are zero and the cohort is all of nothing. + let a = young_leaf(); + js_shadow_slot_set(0, ptr_bits(a)); + let _ = gc_collect_minor(); + let (_, _, survivor_live_1, first_round_1) = crate::gc::copying::test_last_cohort_split(); + assert_eq!( + (survivor_live_1, first_round_1), + (0, 0), + "cycle 1 evacuates Eden only: nothing came out of the survivor space" + ); + + // Cycle 2: A is re-copied (age 1 -> 2) and B enters from Eden. The + // from-survivor space held ONLY A, which is a first-round object, so the + // two numbers must still agree — this is the regime the lock was designed + // in, and the assertion that the split is not simply always different. + let b = young_leaf(); + js_shadow_slot_set(1, ptr_bits(b)); + let _ = gc_collect_minor(); + let (_, _, survivor_live_2, first_round_2) = crate::gc::copying::test_last_cohort_split(); + assert!( + survivor_live_2 > 0, + "A must have come back out of the survivor space" + ); + assert_eq!( + survivor_live_2, first_round_2, + "with a single generation resident the whole-space number IS the \ + fresh-cohort number — at threshold <= 2 the old rule was correct" + ); + + // Cycle 3: the from-survivor space now holds A (age 2) and B (age 1). + // `survivor_live_bytes` counts both; the fresh cohort is B alone. + let _ = gc_collect_minor(); + let (_, _, survivor_live_3, first_round_3) = crate::gc::copying::test_last_cohort_split(); + assert!( + first_round_3 > 0, + "B is a first-round survivor and must be counted as one" + ); + assert!( + survivor_live_3 > first_round_3, + "the aged resident A is in the whole-space number and must NOT be in \ + the fresh-cohort number: whole-space {survivor_live_3}, cohort \ + {first_round_3}. If these are equal the lock is still rating a \ + population its own threshold selected." + ); + + js_shadow_slot_set(0, 0); + js_shadow_slot_set(1, 0); +} diff --git a/crates/perry-runtime/src/gc/tests/copying/weak_holder_registry.rs b/crates/perry-runtime/src/gc/tests/copying/weak_holder_registry.rs index 581e22d237..1ea175c046 100644 --- a/crates/perry-runtime/src/gc/tests/copying/weak_holder_registry.rs +++ b/crates/perry-runtime/src/gc/tests/copying/weak_holder_registry.rs @@ -242,6 +242,8 @@ fn test_full_weak_processing_work_is_independent_of_unrelated_heap_size() { #[test] fn test_registry_tracks_holder_across_three_moving_minors() { let _guard = CopyingNurseryTestGuard::new(3); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let map = crate::weakref::js_weakmap_new(); let live_key = crate::object::js_object_alloc(0, 0); diff --git a/crates/perry-runtime/src/gc/tests/oldgen.rs b/crates/perry-runtime/src/gc/tests/oldgen.rs index 589adc3132..79b1453fd2 100644 --- a/crates/perry-runtime/src/gc/tests/oldgen.rs +++ b/crates/perry-runtime/src/gc/tests/oldgen.rs @@ -1328,6 +1328,8 @@ fn test_minor_skips_whole_heap_old_to_young_rebuild() { #[test] fn test_minor_preserves_old_to_young_edge_across_minors() { let _isolation = copying_nursery_isolation_lock(); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); let _barrier_guard = GeneratedWriteBarrierTestGuard::active(); reset_remembered_set(); diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots/hook_dispatch_handles.rs b/crates/perry-runtime/src/gc/tests/runtime_roots/hook_dispatch_handles.rs index 29217da550..cc65a5e865 100644 --- a/crates/perry-runtime/src/gc/tests/runtime_roots/hook_dispatch_handles.rs +++ b/crates/perry-runtime/src/gc/tests/runtime_roots/hook_dispatch_handles.rs @@ -131,6 +131,8 @@ fn test_timer_tick_roots_callback_args_and_previous_context_across_hooks() { let _async_hook_guard = AsyncHookRuntimeTestGuard::new(); let _guard = CopyingNurseryTestGuard::new(0); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); register_runtime_handle_root_scanner_for_tests(); gc_register_mutable_root_scanner(crate::async_hooks::scan_async_hooks_roots_mut); @@ -289,6 +291,8 @@ fn test_array_map_runtime_handles_survive_callback_copied_minor_gc() { #[test] fn test_map_materializers_runtime_handles_survive_copied_minor_gc() { let _guard = CopyingNurseryTestGuard::new(0); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); register_runtime_handle_root_scanner_for_tests(); diff --git a/crates/perry-runtime/src/gc/tests/support.rs b/crates/perry-runtime/src/gc/tests/support.rs index 130c0f7a08..d0cbecce02 100644 --- a/crates/perry-runtime/src/gc/tests/support.rs +++ b/crates/perry-runtime/src/gc/tests/support.rs @@ -414,9 +414,10 @@ pub(crate) struct CopyingNurseryTestGuard { } pub(super) fn reset_copying_nursery_runtime_test_state() { - // Age-sensitive tests assume the power-on tenuring threshold (promote at - // the 4th survival); pin it so a heavy-influx test earlier on the same - // thread cannot leak a lowered adaptive threshold in. + // Restore the adaptive policy to its power-on floor. Tests of mechanisms + // that require a particular promotion age pin it explicitly with + // `tenuring::set_survivals_for_test`, so a power-on policy change cannot + // silently change the mechanism they exercise. crate::gc::tenuring::reset_for_test(); // #7645: the young-pin latch is process-wide and monotone, so one // earlier pinning test would otherwise leave every later copying test diff --git a/scripts/gc_runtime_root_holders.json b/scripts/gc_runtime_root_holders.json index f1582b60c2..5ef98d06c2 100644 --- a/scripts/gc_runtime_root_holders.json +++ b/scripts/gc_runtime_root_holders.json @@ -266,6 +266,30 @@ "verdict": "not_a_gc_pointer", "why": "Boolean census request latch, set by census_arm and consumed at full-sweep entry; contains no address or JS value." }, + { + "file": "crates/perry-runtime/src/gc/copying.rs", + "name": "LAST_COHORT_SPLIT", + "verdict": "test_only", + "why": "Declared under #[cfg(test)] at crates/perry-runtime/src/gc/copying.rs:1951; this Cell<(usize, usize, usize, usize)> holds only the byte counts of the last survivor-space/fresh-cohort split for the tenuring lock tests. It is absent from shipped binaries." + }, + { + "file": "crates/perry-runtime/src/gc/tenuring.rs", + "name": "SURVIVOR_ROUND_MEASURED", + "verdict": "not_a_gc_pointer", + "why": "Declared at crates/perry-runtime/src/gc/tenuring.rs:225; this Cell records whether any survivor round has been rated on this thread, gating the occupancy rule off its floor. A boolean, never a heap pointer." + }, + { + "file": "crates/perry-runtime/src/gc/tenuring.rs", + "name": "RATED_ROUNDS", + "verdict": "not_a_gc_pointer", + "why": "Steady-state tenuring evidence counter. Cell stores only how many survivor cohorts have been rated; it never holds an address or NaN-boxed value." + }, + { + "file": "crates/perry-runtime/src/gc/tenuring.rs", + "name": "PROMOTE_LOCK_STREAK", + "verdict": "not_a_gc_pointer", + "why": "Tenuring evidence streak. Cell stores only the count of consecutive qualifying rated cohorts; it never holds an address or NaN-boxed value." + }, { "file": "crates/perry-runtime/src/gc/census.rs", "name": "LABEL", @@ -276,7 +300,7 @@ "file": "crates/perry-runtime/src/gc/census.rs", "name": "PASS1_MARKED", "verdict": "non_moving_snapshot", - "why": "Real GC header addresses, deliberately untraced so the diagnostic does not keep its observed objects alive. Populated only at the end of mark propagation of a synchronous full cycle; consumed at sweep entry in the same run_to_completion invocation. The intervening full-cycle phases do not relocate or run JS callbacks. The Vec is used for membership comparisons and dropped with the census before sweep. Budgeted and minor cycles skip both boundaries. Pin re-audited 2026-09-05 after #9760 touched `gc/mod.rs`: that change is `mod heap_stats;` plus a `pub(crate) use` re-export and alters no mark/sweep control flow. `heap_stats()` is reached only from `js_bun_jsc_heap_stats` (the JS-facing `bun:jsc.heapStats()`), i.e. from mutator code, never inside a cycle, and its own module contract forbids allocation or collection during its walk. The mark-complete \u2192 sweep-entry window is unchanged. Re-audited 2026-09-05 (train125) after #9769 and #9771 touched pinned files. #9769 adds one `reg_scanner!` registration to `gc/mod.rs`; #9771 adds a feature-gated `alloc_census_init()` there and a feature-gated Rust-heap dump inside `take_census`. `alloc-census` is not in the default feature set, and decisively: `census_take_if_armed_at_full_sweep_start` does `PASS1_MARKED.with(|p| p.borrow_mut().take())` BEFORE calling `take_census`, so the snapshot has already left the thread-local by the time #9771's code runs \u2014 it cannot affect the window. Neither change alters mark/sweep control flow. Re-audited 2026-09-06 after #9831 touched `gc/policy.rs`. Its hunks are (a) the tiny-parse pressure guard's pricing (`tiny_parse_pressure_headroom_bytes`, `tiny_parse_pressure_due*`, a `Cell` byte-count base) consulted from JSON.parse's mutator-side boundaries (`gc_bump_malloc_trigger`, `gc_collect_pending_suppressed_parse`, `gc_schedule_parse_boundary_collection_if_pressure`), none of which is reachable from inside a cycle, and (b) one extra `Cell` store in `note_collection_finished_arena_occupancy`, which runs from `publish_reclaim_outcome` in the Publish subphase \u2014 after `step_sweep` has already consumed the snapshot. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Re-audited 2026-09-05 (train126) after #9755 restructured `gc/cycle.rs`. Its hunks are all root-scan machinery (`RootScanSubphase`, `RootScanCycleState`, the mutable-scanner iteration state), which runs BEFORE mark propagation completes; `gc/mod.rs` gains only a `mod young_log;` declaration. The bracketing is unchanged \u2014 `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep` \u2014 and a synchronous full mark-sweep still moves nothing between them. Re-pinned 2026-09-05 for the #9740 hot-TLS conversion of this file: the sole change is `thread_local!` \u2192 `crate::perry_thread_local!`, a macro-name swap with identical declaration syntax and `.with()` call sites. No control flow, no phase boundary, and no storage semantics change. Re-audited 2026-09-06 (train128) after #9794's GC diagnostics touched `gc/mod.rs` and `gc/policy.rs`: both gain diagnostic module declarations and counters only \u2014 no mark/sweep control flow, and the census bracketing in `step_mark_propagation` / `step_sweep` is unchanged. Re-audited for #9794's GC diagnostics: `gc/mod.rs` gains `mod diag_sites;` / `mod survival_diag;`, a re-export, a `diag_sites::full_started(...)` call at TRIGGER time (before mark propagation begins), and exit-time reporting. Nothing executes between mark-complete and sweep-entry, so the window is unchanged. Re-audited 2026-09-06 for the retained array-growth verifier fix: the cycle.rs change passes the existing non-copying evacuation verifier an explicit all-forwarded policy. That call remains in minor finalization, outside the synchronous full-cycle census window; its root and heap reads do not allocate GC objects, move objects, or invoke JS callbacks. The mark-complete and sweep-entry boundaries are unchanged. Re-audited 2026-09-05 after #9830 touched `gc/policy.rs`. That change is (a) six `thread_local! {` blocks rewritten as `crate::perry_thread_local! {` and (b) one `#[cfg(test)]` accessor listing the trigger path's hot-slot indices. The macro keeps the same storage, the same `.with()` at every read and write, and the same destructor registration (the teardown guard exists exactly when `needs_drop` holds, which is what `std::thread_local!` already decided); no value, predicate or branch in the file changes, so no mark or sweep control flow does. The one new behaviour is on a declaration's FIRST read: `HotKey::resolve_and_cache` takes a mutex and allocates a key through the GLOBAL allocator. Even if a first read landed inside this window it would be sound \u2014 the window's contract is that nothing relocates and no JS callback runs, and a mimalloc allocation does neither. `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`; the bracketing is untouched. Re-audited 2026-09-06 (train132) after #9860 and #9845 touched `gc/mod.rs`. Both hunks are re-export lists and nothing else: #9860 adds `idle_reclaim_elapsed_starts` / `IDLE_RECLAIM_REARM_MS`, and #9845 adds `owner_is_dead_copied_minor_from_space_of_type`. No mark or sweep control flow changes. #9845's substantive work sits in `gc/oldgen.rs` and `gc/copying.rs`, neither pinned: the copying-minor arm (`finalize_dead_copied_minor_from_space_regexps`) runs on a MINOR, which skips both census boundaries; the full-cycle arm (`collect_dead_registered_regexps_post_trace`, from `with_dead_collection_finalize`) walks the RegExp registry building a Vec of addresses \u2014 no GC allocation, no JS callback, so it cannot relocate the snapshot's subjects \u2014 and it is reached from the sweep body, i.e. AFTER `census_take_if_armed_at_full_sweep_start` has already `take()`n the snapshot out of the thread-local. The mark-complete -> sweep-entry window is unchanged.", + "why": "Real GC header addresses, deliberately untraced so the diagnostic does not keep its observed objects alive. Populated only at the end of mark propagation of a synchronous full cycle; consumed at sweep entry in the same run_to_completion invocation. The intervening full-cycle phases do not relocate or run JS callbacks. The Vec is used for membership comparisons and dropped with the census before sweep. Budgeted and minor cycles skip both boundaries. Pin re-audited 2026-09-05 after #9760 touched `gc/mod.rs`: that change is `mod heap_stats;` plus a `pub(crate) use` re-export and alters no mark/sweep control flow. `heap_stats()` is reached only from `js_bun_jsc_heap_stats` (the JS-facing `bun:jsc.heapStats()`), i.e. from mutator code, never inside a cycle, and its own module contract forbids allocation or collection during its walk. The mark-complete \u2192 sweep-entry window is unchanged. Re-audited 2026-09-05 (train125) after #9769 and #9771 touched pinned files. #9769 adds one `reg_scanner!` registration to `gc/mod.rs`; #9771 adds a feature-gated `alloc_census_init()` there and a feature-gated Rust-heap dump inside `take_census`. `alloc-census` is not in the default feature set, and decisively: `census_take_if_armed_at_full_sweep_start` does `PASS1_MARKED.with(|p| p.borrow_mut().take())` BEFORE calling `take_census`, so the snapshot has already left the thread-local by the time #9771's code runs \u2014 it cannot affect the window. Neither change alters mark/sweep control flow. Re-audited 2026-09-06 after #9831 touched `gc/policy.rs`. Its hunks are (a) the tiny-parse pressure guard's pricing (`tiny_parse_pressure_headroom_bytes`, `tiny_parse_pressure_due*`, a `Cell` byte-count base) consulted from JSON.parse's mutator-side boundaries (`gc_bump_malloc_trigger`, `gc_collect_pending_suppressed_parse`, `gc_schedule_parse_boundary_collection_if_pressure`), none of which is reachable from inside a cycle, and (b) one extra `Cell` store in `note_collection_finished_arena_occupancy`, which runs from `publish_reclaim_outcome` in the Publish subphase \u2014 after `step_sweep` has already consumed the snapshot. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Re-audited 2026-09-05 (train126) after #9755 restructured `gc/cycle.rs`. Its hunks are all root-scan machinery (`RootScanSubphase`, `RootScanCycleState`, the mutable-scanner iteration state), which runs BEFORE mark propagation completes; `gc/mod.rs` gains only a `mod young_log;` declaration. The bracketing is unchanged \u2014 `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep` \u2014 and a synchronous full mark-sweep still moves nothing between them. Re-pinned 2026-09-05 for the #9740 hot-TLS conversion of this file: the sole change is `thread_local!` \u2192 `crate::perry_thread_local!`, a macro-name swap with identical declaration syntax and `.with()` call sites. No control flow, no phase boundary, and no storage semantics change. Re-audited 2026-09-06 (train128) after #9794's GC diagnostics touched `gc/mod.rs` and `gc/policy.rs`: both gain diagnostic module declarations and counters only \u2014 no mark/sweep control flow, and the census bracketing in `step_mark_propagation` / `step_sweep` is unchanged. Re-audited for #9794's GC diagnostics: `gc/mod.rs` gains `mod diag_sites;` / `mod survival_diag;`, a re-export, a `diag_sites::full_started(...)` call at TRIGGER time (before mark propagation begins), and exit-time reporting. Nothing executes between mark-complete and sweep-entry, so the window is unchanged. Re-audited 2026-09-06 for the retained array-growth verifier fix: the cycle.rs change passes the existing non-copying evacuation verifier an explicit all-forwarded policy. That call remains in minor finalization, outside the synchronous full-cycle census window; its root and heap reads do not allocate GC objects, move objects, or invoke JS callbacks. The mark-complete and sweep-entry boundaries are unchanged. Re-audited 2026-09-05 after #9830 touched `gc/policy.rs`. That change is (a) six `thread_local! {` blocks rewritten as `crate::perry_thread_local! {` and (b) one `#[cfg(test)]` accessor listing the trigger path's hot-slot indices. The macro keeps the same storage, the same `.with()` at every read and write, and the same destructor registration (the teardown guard exists exactly when `needs_drop` holds, which is what `std::thread_local!` already decided); no value, predicate or branch in the file changes, so no mark or sweep control flow does. The one new behaviour is on a declaration's FIRST read: `HotKey::resolve_and_cache` takes a mutex and allocates a key through the GLOBAL allocator. Even if a first read landed inside this window it would be sound \u2014 the window's contract is that nothing relocates and no JS callback runs, and a mimalloc allocation does neither. `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`; the bracketing is untouched. Re-audited 2026-09-06 (train132) after #9860 and #9845 touched `gc/mod.rs`. Both hunks are re-export lists and nothing else: #9860 adds `idle_reclaim_elapsed_starts` / `IDLE_RECLAIM_REARM_MS`, and #9845 adds `owner_is_dead_copied_minor_from_space_of_type`. No mark or sweep control flow changes. #9845's substantive work sits in `gc/oldgen.rs` and `gc/copying.rs`, neither pinned: the copying-minor arm (`finalize_dead_copied_minor_from_space_regexps`) runs on a MINOR, which skips both census boundaries; the full-cycle arm (`collect_dead_registered_regexps_post_trace`, from `with_dead_collection_finalize`) walks the RegExp registry building a Vec of addresses \u2014 no GC allocation, no JS callback, so it cannot relocate the snapshot's subjects \u2014 and it is reached from the sweep body, i.e. AFTER `census_take_if_armed_at_full_sweep_start` has already `take()`n the snapshot out of the thread-local. The mark-complete -> sweep-entry window is unchanged. Re-audited 2026-09-07 for the tenuring price counters: `gc/mod.rs` only loads diagnostic atomics in the process-exit reporting path, after collection has returned; it does not alter mark/sweep control flow or execute inside the mark-complete -> sweep-entry window.", "window": { "start": { "file": "crates/perry-runtime/src/gc/census.rs", @@ -293,7 +317,7 @@ "sources": { "crates/perry-runtime/src/gc/census.rs": "388414f9629f196e84673e91bebd04bdcdcabdaa180252d2dfe4b82d1b49ca5a", "crates/perry-runtime/src/gc/cycle.rs": "2e2f5adca2229f74409e01a1cb571e2147cd8a33f58d0976711fce98d4777309", - "crates/perry-runtime/src/gc/mod.rs": "43523b66595c61516ef6fcd4139d3ec5b4768a13c46ae1470c1d45481eacfdd9", + "crates/perry-runtime/src/gc/mod.rs": "2f8fa58d50aa95cd0156e68e01197ebb556d821a0e748ef238d1c5c407e8e77c", "crates/perry-runtime/src/gc/policy.rs": "dc9242ed40c0aa9c411d1ec0235c0219c6716dd82d56eb4d46578f7e889825d2", "crates/perry-runtime/src/gc/progress.rs": "a5ad3971bbe4047229ca57325234780daa85921dbc778e1c08dff4ad07ccfb96" }