Skip to content

gc: the budgeted full step is starved by the shape-descriptor prune memmove — +59.5% peak RSS at tenuring threshold 2 #9871

Description

@proggeramlug

Rewritten 2026-09-06 after profiling the idle window. The original idle-CPU half of this issue is struck: it was measuring a different, already-known defect. What remains is the peak-RSS half, restated around the mechanism the profile found.

Struck: the idle-CPU half

The issue originally claimed that at a tenuring threshold of 2 the idle window's 23–25 s of CPU was deferred page release. That is wrong. Two independent results:

  1. No copying happens in the idle window at all. Across eight arms on two protocols there is not one [gc-copy-minor] ran line inside any idle window, and copied bytes in the window are 0 on every arm — including the =2 arms that cost 27–65 s.
  2. The =1 and =2 idle windows run identical event inventories — same line kinds and the same counts of each (4 idle-reclaim, 3 full collections, 2 old-free, 3 general-reclaim, 3 restore-coverage, 15 young-log, 3 idle-compact, 1 copy-minor), with compaction pauses of 317 ms and 417 ms and comparable live heaps (387 MB, 367 MB) — at 5.5 s versus 64.8 s. Deferral would show as more collections or bigger ones. These are the same collections at the same size.

A 30 s main-thread sample of each idle window says what it is:

=1 =2
collector (by ancestor) 568 (2.4 %) 24,271 (100.0 %)
mutator 23,236 (97.6 %) 0
kevent — genuinely parked 23,212 0

At =1 the process is idle. At =2 it never idles: every sample is collector work. Inclusive: gc_budgeted_step_work_units_inner_with_progressGcCycleState::stepprune_dead_owner_side_tables_post_trace (23,426) → prune_dead_shape_keys (23,367). Self: _platform_memmove +22,820, which is 94 % of the whole difference, then scan_shape_table_rekey_mut +340, ShapeTableInner::facts_remove +223, remove_descriptor_indexed_under +150.

That is the same symbol under the same caller as the turn-CPU mode split (__memmove under remove_descriptor_indexed_underprune_dead_owner_side_tables_post_trace, identical counters carrying 2× the time) — the O(list) descriptor removal, not tenuring and not release. See #9881 and #9857.

What remains: peak RSS, restated

At =2, peak RSS is +59.5 % against =1 on the quiet host (2208–2244 MB vs 1384–1511 MB, three repeats, disjoint), and the budgeted steps in turns 2–3 run 13 and 2 times against 20 and 16, general-reclaim 0–1 times against 14–17, releasing zero blocks against 39 and 13.

The original explanation was "a full is priced by old-gen yield, so a higher threshold makes it read unproductive". The profile suggests a simpler one: those steps report pct=0% because each budgeted step spends its entire budget memmoving descriptor tails in prune_dead_shape_keys and never reaches the sweep that releases pages. The step is not mis-priced; it is starved — by the same O(list) removal as above. At =2 more objects are still live as owners when the full runs, so there is more to prune, so the starvation is worse.

Falsifier

=2 built with #9881's swap-remove (perf/family-swap-remove-main @ 01313fc2e) should release pages during turns 2–3 and bring peak RSS toward =1, on the mini.

  • Pages released in turns 2–3 goes from 0 to non-zero and [gc-step] pct rises above 0 → the starvation account holds.
  • Peak RSS does not move while the step cadence and pct do → the starvation is real but is not what holds the pages, and this issue is wrong a second time and should be closed rather than reshaped again.

Precondition (unchanged, step 0)

Run =1 and =2 on the host first and confirm peak RSS separates. On the dev box it does not (2298 vs 2230/2790, overlapping), so no arm there can decide this; the mini is the host where it separates. If the separation is absent, stop — every downstream number is undecidable, and that is a host result, not a refutation.

Method note

Both the copying-in-idle hypothesis and the deferred-release hypothesis were refuted from captures that already existed, by counting events rather than reasoning about them. The thing that finally named the cost was a profile, because the cost is per-unit and no counter the collector emits can see it. Third instance today; each time the profile named it, and twice now it named the same symbol.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions