coop/in_process: second Next app dies with TypeError: value is not a function (GC rooting, multi-heap)

[proggeramlug]

Summary

With #8528 merged, a Coop in_process daemon can load and initialize two
Next.js applications in one process — but the second application's HTTP
handler rejects at dispatch, so it serves 500 while the first serves 200.

Evidence

Measured on the quiet M1 mini, Perry 7c22189aa (0.5.1514), Coop in_process,
two deployments of the same Next fixture. Provider provenance verified by
content, not by recorded metadata: the merged fix deletes
ERR_PERRY_PATH_MODULE_THREAD, and strings on the packaged
libperry_runtime.dylib finds 0 occurrences of it (the string also appears
nowhere in main's source, so the check is meaningful rather than vacuous).

Both applications initialize cleanly:

INFO application library preloaded on dedicated Perry thread deployment="bench-000" load_ms=... init_ms=...
INFO application library preloaded on dedicated Perry thread deployment="bench-001" load_ms=749.8 init_ms=697.0

Then:

bench-000 -> 200  {"runtime":"next","iterations":100,"checksum":3726872593}
bench-001 -> 500  coop: 500 deployment dispatch failed

ERROR deployment dispatch failed deployment=bench-001
      error=Runtime(invoking deployment HTTP handler: handler promise rejected: )

The rejection reason is empty, which is itself worth fixing — an empty
handler promise rejected: gives an operator nothing to act on.

What this is NOT

• Not the path-module registry. Zero rejected duplicate path-module initializer messages in this run; both apps register their own initializers
  on their own dedicated Perry threads. (An earlier iteration of fix(runtime): make the path-module registry per-heap so one process can host several apps #8528 that kept
  the initializer-address map process-wide DID produce 115 such rejections;
  that is fixed.)
• Not app load or module init — both complete, with timings logged.
• Not ordering-dependent in an obvious way: the first deployment to be
  dispatched succeeds and the second fails.

Hypothesis

Some other process-global runtime/stdlib state assumes a single application per
process. Until #8528 the fail-closed thread guard made multi-app hosting
impossible, so none of that state has ever been exercised by two heaps. The
per-request registries are the obvious suspects — several are Mutex<HashMap<usize, ...>>
and are UNCOVERED by scripts/gc_runtime_root_holders.py, i.e. nobody has
determined whether they hold heap pointers:

REQUEST_REGISTRY, BLOB_REGISTRY, FETCH_RESPONSES, TRANSFORM_PAIRS,
STREAM_HANDLES (crates/perry-stdlib/src/fetch/mod.rs,
streams/transform.rs), plus buffer/own_props.rs's PROPS and
buffer/header.rs's EXTERNAL_*_REGISTRY (the latter keyed by RAW heap
address).

Suggested first steps

1. Make the rejection reason non-empty — surface the underlying JS error.
2. With that, name the failing operation, then check whether its backing table
   is process-global.

Reproduction

Needs a Coop checkout whose perry-main.lock points at a Perry with #8528, plus
coop-daemon's rusqlite bumped 0.37 -> 0.39 to match perry-stdlib (they link
the same native sqlite3; see the companion Coop change). Then
scripts/prepare-next-benchmark.sh and the resource_benchmark harness with
COOP_BENCH_APP_COUNTS=2, COOP_BENCH_EXECUTION_MODE=in_process.

[proggeramlug]

Re-tested on 0.5.1516 (3885ba491) — still fails, and #8552 is not the cause

I expected #8552 ("register mutable root scanners per thread") to fix this: a
process-global "scanner already registered" latch would mean the first heap
registers its scanners and every later heap silently skips, leaving app #2's
roots unscanned. That would have matched the symptom exactly.

It does not. Re-ran the full chain on main 3885ba491 (which includes #8552),
with provider provenance verified by content — the removed
ERR_PERRY_PATH_MODULE_THREAD string appears 0 times in the packaged
libperry_runtime.dylib, and nowhere in main's source, so the check is
meaningful rather than vacuous.

Result is byte-identical to the 0.5.1514 run:

application library preloaded on dedicated Perry thread   x2   (both apps init)
rejected duplicate path-module initializer                 0
bench-000 -> 200  {"runtime":"next","iterations":100,"checksum":3726872593}
bench-001 -> 500  coop: 500 deployment dispatch failed
ERROR deployment dispatch failed deployment=bench-001
      error=Runtime(invoking deployment HTTP handler: handler promise rejected: )

Which app fails varies between runs (it was bench-000 in the harness run and
bench-001 in the manual daemon run of the same build), consistent with "whichever
is second", not with anything specific to a deployment.

What this rules out

• per-thread scanner registration latches (fix(gc): register mutable root scanners per thread #8552)
• the path-module registry (fix(runtime): make the path-module registry per-heap so one process can host several apps #8528) — both apps register their own initializers,
  zero rejections
• app load / module init — both complete, with timings logged

Still the best hypothesis

Some other process-global runtime/stdlib state assumes one application per
process. The per-request registries remain the suspects, several of which are
Mutex<HashMap<usize, ...>> and UNCOVERED by the holders census:
REQUEST_REGISTRY, BLOB_REGISTRY, FETCH_RESPONSES, TRANSFORM_PAIRS,
STREAM_HANDLES, plus buffer/own_props.rs's PROPS and
buffer/header.rs's EXTERNAL_*_REGISTRY (keyed by RAW heap address).

The blocking next step is diagnosability

handler promise rejected: carries an empty reason. Until that surfaces the
underlying JS error there is nothing to attribute, and every further step here is
guesswork. I'd fix that first — it is likely a small change at the rejection
site, and it converts this issue from "one of ~7 suspects" into a named failure.

[proggeramlug]

The rejection reason, finally: TypeError: value is not a function

Coop was reporting an empty reason, and it turned out to be reporting it
wrongly. Two bugs on the host side, now fixed locally (PR: PerryTS/coop#14):

1. read_perry_string trusted a contract that does not exist. Its comment says
   js_get_string_pointer_unified "Returns 0 if not a string", but that function
   deliberately returns the payload for a POINTER_TAG (0x7ffd) value too
   ("used for cross-module returns", value/nanbox.rs:296-299). Handing it an
   Error object yields a non-null pointer that is not a StringHeader, and
   reading through it is a wild read — it fabricated a one-character reason
   ("\t") out of unrelated heap bytes.
2. The hex fallback only fired for None, so the garbage string took the
   "success" path and printed as if it were the reason.

With the tag checked and js_jsvalue_to_string used for non-strings, the real
error appears:

handler promise rejected: TypeError: value is not a function
                          [raw=0x7ffd0007ca3bf5f0 tag=0x7ffd]

What that changes

This is not a registry logic bug. TypeError: value is not a function is the
signature of a GC rooting failure — CLAUDE.md: "A 'GC value live but not rooted
across a collection point' bug is invisible at collection time … It surfaces one
or more cycles later, in a different function, as TypeError: value is not a function."

So: in a two-application in_process daemon, something live in the second
application's heap is being collected.

The reproducibility narrows it further

The failure is deterministic — the second application to be dispatched fails
every run (which application that is varies, so it tracks ordering, not identity).
Per the rule of thumb in the same docs, an unrooted register goes bad only
when a collection lands in its window and is therefore intermittent, while an
unrooted table goes bad at collection #0 and stays bad. A perfectly
reproducible failure points at a table, not a register — which is exactly the
process-global-registry hypothesis, now with evidence behind it rather than
suspicion.

Suggested instruments

The docs name the tools for this class, and they apply directly:

• PERRY_GC_SCHEDULE_SEED=<u64> PERRY_GC_PROTECT_FROMSPACE=1 to turn the stale
  deref into an immediate precise fault
• PERRY_GC_PROTECT_FROMSPACE_DEPTH raised, since the value may cross many
  collections between its last valid observation and the stale use
• scripts/gc_runtime_root_holders.py — the per-request registries that remain
  UNCOVERED are the shortlist: REQUEST_REGISTRY, BLOB_REGISTRY,
  FETCH_RESPONSES, TRANSFORM_PAIRS, STREAM_HANDLES, plus
  buffer/own_props.rs's PROPS and buffer/header.rs's EXTERNAL_*_REGISTRY

Note #8552 (per-thread scanner registration latches) is already in this build and
does not fix it, so the defect is not simply a skipped registration.

[proggeramlug]

Reopening — the close was a keyword accident, the bug is still live

#8562 carried Fixes #8546 and auto-closed this at 04:47 on 2026-08-22. #8562 is a regression test for #8552's per-thread scanner latch. My comment five hours later (09:18) established that #8552 is already in the failing build and does not fix this; PerryTS/coop#15, opened the same morning, says the same. Nothing that landed since claims to address it.

Where it stands after the two Coop-side fixes

• PerryTS/coop#14 — the rejection reason is no longer empty (the host was reading an Error object as a StringHeader).
• PerryTS/coop#15 — one real unrooted holder removed on the host side: await_promise cached the raw Promise pointer across perry_poll. PERRY_GC_SCHEDULE_SEED=12345 PERRY_GC_SCHEDULE_RATE=0.5 PERRY_GC_PROTECT_FROMSPACE=1 faulted on it directly (obj_type=5 = GC_TYPE_PROMISE, "the faulting instruction IS the stale use"); rooting it through js_ffi_root_scope_enter / js_ffi_root_push_nanbox / js_ffi_root_get_nanbox took the same seed from FAULT to 0 faults with 107 copying minors still running (anti-vacuity).

Both are merged. Two Next apps in_process still fail, but the failures are now named:

bench-000: TypeError: _onUserlandLoaded is not a function
bench-001: TypeError: value is not a function

_onUserlandLoaded is a Node bootstrap internal, so at least one more holder of the same class exists — a JS value cached across something that can run JS — and it is not on the Coop host side anymore, which only touches the Promise. The process-global stdlib registries from the original report remain the shortlist (REQUEST_REGISTRY, BLOB_REGISTRY, FETCH_RESPONSES, TRANSFORM_PAIRS, STREAM_HANDLES, buffer/own_props.rs PROPS, buffer/header.rs EXTERNAL_*_REGISTRY).

Measured against Perry 0.5.1512 (fc6b3378, Coop's current pin); next step is re-running the from-space-protect hunt against current main once Coop's pin and the paired rusqlite bump land.

[proggeramlug]

Reproduced on 0.5.1516 with coop#14/#15 in place — and the rule is now precise: only the app whose init finished LAST works

Coop main at 3e7180e (worker has #14's reason reporting and #15's Promise rooting), Perry 0.5.1516 (3885ba491) — chosen because on 0.5.1519 the fixture does not load at all (perry#8882). Harness patched locally to warm every app before judging (it used to stop at the first 500, which is what hid the pattern) and to relay the daemon's stdout/stderr live.

run	init-completion order	result
2 apps, preload_concurrency=1	000 → 001	000: 500, 001: 200
3 apps, preload_concurrency=1	000 → 001 → 002	000: 500, 001: 500, 002: 200
3 apps, concurrent	000 → 002 → 001 (by log order)	000: 500, 002: 500, 001: 200
2 apps, concurrent, inits finishing 0.5 ms apart	001 → 000	both 500

So it was never "the second one dispatched": every application except the last one to finish init fails, whatever order they are dispatched in. Two inits interleaving closely enough break both. That is last-writer-wins on process-global state written during module init.

Every failure is handler promise rejected: TypeError: value is not a function (once TypeError: _onUserlandLoaded is not a function).

What from-space protection says: nothing, and that is informative

PERRY_GC_SCHEDULE_SEED=12345 PERRY_GC_PROTECT_FROMSPACE=1 at RATE=0.5 DEPTH=64 and at RATE=1 DEPTH=256 (382 safepoints / 216 scheduled collections in the RATE=0.5 arm): no fault, same 500. Before #15 the same seed faulted immediately on the un-rooted Promise (obj_type=5, backtrace in invoke_buffer_handler), so the instrument is armed and #15 is confirmed as a real fix of a real instance. What remains does not deref retired from-space — the value it reads is live memory in the other application's heap, which the quarantine cannot object to. A cross-heap value, not a stale one.

Where it dies

_onUserlandLoaded is Next's, not Perry's: AppRouteRouteModule constructs new LazyModule(userland, (module1) => this._onUserlandLoaded(module1)), and the route bundle's userland is webpack's lazy () => __webpack_require__(3870) — evaluated on the first request, not at init. So the earlier-initialised app survives init, and dies the first time it resolves something lazily by name: the arrow's captured this (or the module it is handed) is the last-initialised image's, and _onUserlandLoaded is not a function on it.

That narrows the suspect list from "any of ~7 per-request registries" to "process-global tables keyed by name that module init writes with insert (last wins), whose values are heap pointers or ids used to resolve properties / modules / closures". In 0.5.1516 crates/perry-runtime/src/symbol.rs:116 SYMBOL_REGISTRY: Mutex<Option<HashMap<String, usize>>> is one such (a string → heap pointer map, process-global while the path-module initializer registry from #8528 is perry_thread_local!); I have not yet confirmed it is the one. Next step on my side is a census of name-keyed process-global statics written during init; on Perry's side, making whichever it is per-thread like #8528 did for initializers is the shape of the fix.

Harness changes (report every app; relay daemon output) are going into PerryTS/coop as a PR so this table is reproducible with one env var.

[proggeramlug]

Mechanism: the class registries are process-global and keyed by compile-time class id — N identical images overwrite each other

crates/perry-runtime/src/object/class_registry/state.rs (checked on 3885ba491 and unchanged on 41e8479a5):

pub static CLASS_VTABLE_REGISTRY: RwLock<Option<HashMap<u32, ClassVTable>>>        // methods/getters/setters by class id
pub static CLASS_STATIC_METHODS:  RwLock<Option<HashMap<u32, HashMap<String, (usize, u32, bool)>>>>
pub static CLASS_METHOD_BIND_LENGTHS / CLASS_STATIC_METHOD_BIND_LENGTHS / REGISTERED_CLASS_IDS / CLASS_STATIC_ACCESSORS

js_register_class_method does vtable.methods.insert(name, VTableMethodEntry { func_ptr, .. }) — last writer wins. Class ids are assigned by codegen (next_class_id threaded through lowering), so two dlopen'd copies of the same app dylib register the same ids with different func_ptrs (each image's own code addresses). After the last image's init, every class of every earlier image dispatches into the last image's code: earlier apps fail at their first by-name resolution (Next's lazy LazyModule → this._onUserlandLoaded), the last one serves. That is exactly the table from my previous comment, and it explains the from-space silence — a func_ptr is not a heap pointer. (CLASS_SYMBOL_METHODS / CLASS_SYMBOL_ACCESSORS in the same file are already perry_thread_local!; the path-initializer registry was made per-thread in #8528. These string-keyed tables were left behind.)

Three experiments, one patch each on a local 0.5.1516 provider (app image unchanged)

registration policy	serial 3 apps	reading
stock (insert, last wins)	000 ✗ 001 ✗ 002 ✓	last image owns every table
instance methods first-wins only	000 ✗ 001 ✗ 002 ✗	methods from image 1, getters/setters/statics from image 3 — every vtable is a mix, nobody works
every js_register_class_* / js_class_register_* entry point (24 of them) ignores registrations for a class id from any thread but the first to register it	000 boots; 001 dies during its own init — TypeError: Cannot read properties of undefined (reading 'createBaggage') — and takes the daemon down	later images cannot even initialise on another image's tables: the tables are per-image state that every image both writes and depends on

So the registries are not merely "polluted" by a second image — there is no ordering of writes to a shared table that works for more than one image. They have to be per image, and since Coop gives each application its own dedicated Perry thread, per-thread is the natural key — the same shape as #8528's fix for the path-initializer map.

Scope of the change

CLASS_VTABLE_REGISTRY has 50 use sites across class_registry/*.rs, proxy.rs, typed_feedback/guards.rs, value/dynamic_object.rs, object/{property_key,prop_plan,native_module,native_call_method,descriptors}.rs; the other five statics add ~45. Plus whatever backs js_register_class_constructor/_flags, js_class_register_capture_values (object/class_constructors.rs), js_register_class_{id,name} (class_meta.rs), js_register_class_parent{,_dynamic} (parent_static.rs), js_class_register_static_field, the class_meta_registry.rs four, data_view_registry.rs, and symbol/properties.rs's js_class_register_static_symbol — every one of the 24 entry points is keyed by class id. A perry_thread_local! conversion, or a per-image registry handle selected by the loading thread, are the two shapes I see; the second keeps the RwLock code paths intact.

Diagnostic patches were not committed anywhere; the reproduction is PerryTS/coop main + coop#18's harness (COOP_BENCH_APP_COUNTS=3 COOP_BENCH_PRELOAD_CONCURRENCY=1 COOP_BENCH_TRACE_DAEMON=1, path /api/benchmark) on any Perry that loads the fixture — i.e. 0.5.1516, until perry#8882 is fixed.