diff --git a/changelog.d/9846-iterator-next-override-probe-allocation.md b/changelog.d/9846-iterator-next-override-probe-allocation.md new file mode 100644 index 0000000000..b127d8fd7c --- /dev/null +++ b/changelog.d/9846-iterator-next-override-probe-allocation.md @@ -0,0 +1,55 @@ +### Fixed + +- **Every built-in iterator step allocated a `"next"` key string to learn that + nothing was patched.** `call_overridden_iterator_next` — the per-step probe + that lets a user replacement of `%ArrayIteratorPrototype%.next` (and the Map + / Set / String family prototypes) drive `for…of`, spread, `Array.from` and + manual `.next()` — ended in a by-name prototype lookup that minted a fresh + 4-byte `"next"` string on every call. One 32-byte allocation per iteration + step of every array, Map, Set and string iterator in the program. + + The existing early-out could not prevent it. `ITERATOR_PROTOTYPE_PTR == 0` + ("the tower was never materialized, so no override can exist") is **dead on + any program that has allocated one iterator**: every iterator allocator calls + `attach_iterator_prototype`, which calls `ensure_iterator_prototypes`, which + builds the tower. The guard is true exactly once and false forever after. + + Replaced by an allocation-free proof that runs on the path every real program + takes: the prototype's OWN `next` slot still holds a closure whose native + entry is the canonical thunk (the certified non-allocating own-field read, + #9480), AND no accessor descriptor is recorded for `"next"` on it (the + per-key Bloom bit `set_accessor_descriptor` sets before inserting, #6759 C2 — + needed because `defineProperty(proto, "next", {get})` leaves the old closure + in the data slot and puts the accessor in the side table). Anything else — + replaced, deleted, an accessor, a bound copy — takes the by-name path + unchanged. + + Affected files: + + - `crates/perry-runtime/src/object/iterator_prototypes.rs` — the + `prototype_next_is_canonical` probe, ahead of the by-name lookup. + + Measured: the 2026-09-06 claude-code allocation census ranked this site third + by count — ~122,880 allocations of 32 bytes per 400-character reply, 17.1 % + of the top-30 allocation count — and misattributed it to `Intl.Segmenter` + substring copying. Resolved by an explicit caller walk in the shipped binary: + `js_for_of_next+0xd0` → `dispatch_array_iterator_method_inner+0x218` (a `bl` + to `call_overridden_iterator_next`) → `+0x67c` (a `bl` to + `js_string_from_bytes_with_capacity`) → `string_storage_alloc`. + + Validation: `test-files/test_gap_iterator_prototype_next_patch.ts` drives a + replaced `next` through `for…of`, spread, `Array.from` and manual `.next()` + on all four families, and covers restore-by-identity, a second replace after + a restore, a bound copy of the original (which must NOT be mistaken for the + builtin), an accessor `next`, and a deleted `next`. The unit counter asserts + that 1,000 probes on an unpatched iterator with the tower materialized move + the arena by ZERO bytes, with the minor-cycle count pinned so a collection + inside the window cannot manufacture a zero delta. + + Counter on a relinked claude-code binary (this fix plus a measurement-only + hit/miss counter; before the fix every probe allocated, so `hits + byname` is + the pre-fix count and `byname` is what survives): a 400-character reply runs + **144,189 / 144,303** probes and a 3300-character reply **887,076**, with + **`byname = 0` on every one of the 173 per-minor reports across three runs** + — the proof answers 100 % of probes on a real program. At 32 B a string that + is 4.6 MB and 28.4 MB of allocation removed per process respectively. diff --git a/changelog.d/9860-intl-segmenter-view-mode.md b/changelog.d/9860-intl-segmenter-view-mode.md new file mode 100644 index 0000000000..99dbeec504 --- /dev/null +++ b/changelog.d/9860-intl-segmenter-view-mode.md @@ -0,0 +1,54 @@ +### Added + +- **`Intl.Segmenter` view mode: five runtime entry points that answer a + grapheme loop's questions without materialising a record or a substring.** + The compiler half (PR #9859) proves that a + `for (let {segment: O} of X.segment(q))` loop never lets the record or `O` + escape, and then drives a cursor instead of building either. + + ``` + js_segments_view_open(segmenter, input) -> cursor | 0.0 + js_segments_view_next(cursor) -> 1.0 | 0.0 (allocation-free) + js_segments_view_code_point_at(cursor, k) -> number | undefined (allocation-free) + js_segments_view_segment(cursor) -> string (materialise-on-miss) + js_segments_view_regexp_test(cursor, regex) -> true | false | undefined + ``` + + The cursor is an **ordinary GC object** whose slot 0 holds the input as a + traced value, so the collector rewrites it like any other field — no + registered root, no side table, no new scanner. Every entry point re-derives + its `&str` on entry and drops it before returning. + + `open` **declines with no observable effect**, in a fixed order: a + non-pristine `Intl.Segmenter`, a replaced `segment`, a non-grapheme + granularity, an input that is not ALREADY a string primitive (checked before + any coercion, because `build_segments` runs user `toString` and throws on a + Symbol), a non-UTF-8 (WTF-8 lone surrogate) input, or an empty one. It never + throws and never allocates before the final step; the compiler then evaluates + `X.segment(q)` exactly once in its original position. + + `_code_point_at`'s `k` is **segment-relative and segment-bounded** — `k` past + the segment's end is `undefined` even though the input continues — and decodes + from the cursor's byte offset, so `k = 0` is O(1) rather than a walk from + index 0. + + `_regexp_test` matches against a **bounded haystack whose bounds are the + string's ends**, so `^`, `$` and lookbehind are segment-local; it is + three-valued and returns `undefined` ("I decline, materialise and call the + normal path") for a global or sticky regex, whose `test` is stateful in + `lastIndex`, and for a patched `RegExp.prototype.test`. + + Affected files: + + - `crates/perry-runtime/src/intl/segments_view.rs` — the entry points. + - `crates/perry-runtime/src/regex.rs` — `regexp_test_str_bounded`, the + bounded-haystack primitive. + - `crates/perry-runtime/src/object/regex_proto_thunks.rs` — + `regexp_prototype_test_is_canonical`, the allocation-free proof that + `RegExp.prototype.test` is still the builtin. + + Measured: the loop this exists for is 60-85 % of claude-code's active + main-thread CPU and allocates ~420,000 times per 400-character reply. The + falsifier is a unit counter — 200 `next` + `code_point_at` steps move + `arena_in_use_bytes` by **zero**, with the minor-cycle count pinned so a + collection cannot manufacture the zero. diff --git a/crates/perry-runtime/src/intl.rs b/crates/perry-runtime/src/intl.rs index 88084c0db4..7e195ee4ec 100644 --- a/crates/perry-runtime/src/intl.rs +++ b/crates/perry-runtime/src/intl.rs @@ -64,6 +64,7 @@ mod numbering_system; use numbering_system::{is_well_formed_numbering_system, resolve_numbering_system}; mod canon_aliases; pub(crate) mod segmenter; +pub mod segments_view; use canon_aliases::canonicalize_unicode_extension_types; pub(crate) use date_collator::{ diff --git a/crates/perry-runtime/src/intl/segments_view.rs b/crates/perry-runtime/src/intl/segments_view.rs new file mode 100644 index 0000000000..f5bac48926 --- /dev/null +++ b/crates/perry-runtime/src/intl/segments_view.rs @@ -0,0 +1,746 @@ +//! `Intl.Segmenter` **view mode** — the runtime half of +//! `INTERFACE_segments_view.md` §9, agreed with the keystroke lane. +//! +//! The compiler proves that a `for (let {segment: O} of X.segment(q))` loop +//! never lets the record or `O` escape, and then drives this cursor instead of +//! building either. Nothing here constructs a `Segments`: `open` takes the +//! segmenter and the input, which is why the view mode never depended on the +//! lazy-`Segments` work (measured and refuted separately). +//! +//! ## The rooting contract, which is the reason this file is small +//! +//! The cursor is an **ordinary GC object** whose slot 0 holds the input string +//! as a **traced value**, so the collector marks and rewrites it like any other +//! field — no registered root, no side table, no new scanner. Every entry point +//! re-derives its `&str` from that slot on entry and drops it before returning. +//! **No address derived from the input outlives a single entry point, and the +//! only thing that crosses the loop body is a GC pointer the collector +//! maintains.** `_next` and `_code_point_at` allocate nothing at all, so inside +//! them the question cannot even arise; `open` and `_segment` allocate and +//! carry the obligation explicitly. + +use crate::object::ObjectHeader; +use crate::string::StringHeader; +use crate::value::JSValue; +use std::sync::atomic::{AtomicU64, Ordering}; + +/// Class id for the view cursor. It is the brand: a load, where a +/// `get_string_field(obj, "__brand")` check would allocate a key string on a +/// path that runs per loop entry. +pub const SEGMENTS_CURSOR_CLASS_ID: u32 = 0xFFFF_000E; + +const F_INPUT: u32 = 0; +const F_BYTE_START: u32 = 1; +const F_UTF16_START: u32 = 2; +const F_BYTE_END: u32 = 3; +const F_UTF16_LEN: u32 = 4; +const CURSOR_FIELDS: u32 = 5; + +// --- counters (PERRY_SEGVIEW_DIAG=1) --------------------------------------- + +static OPENS: AtomicU64 = AtomicU64::new(0); +static DECLINE_NOT_SEGMENTER: AtomicU64 = AtomicU64::new(0); +static DECLINE_NOT_GRAPHEME: AtomicU64 = AtomicU64::new(0); +static DECLINE_SEGMENT_PATCHED: AtomicU64 = AtomicU64::new(0); +static DECLINE_NOT_STRING: AtomicU64 = AtomicU64::new(0); +static DECLINE_NOT_UTF8: AtomicU64 = AtomicU64::new(0); +static DECLINE_EMPTY: AtomicU64 = AtomicU64::new(0); +static NEXTS: AtomicU64 = AtomicU64::new(0); +static CODE_POINT_ATS: AtomicU64 = AtomicU64::new(0); +static MATERIALISE_SEGMENT: AtomicU64 = AtomicU64::new(0); +static REGEXP_TEST_ACCEPTED: AtomicU64 = AtomicU64::new(0); +static REGEXP_TEST_DECLINED: AtomicU64 = AtomicU64::new(0); + +fn diag_on() -> bool { + static ON: std::sync::OnceLock = std::sync::OnceLock::new(); + *ON.get_or_init(|| std::env::var("PERRY_SEGVIEW_DIAG").is_ok()) +} + +#[inline(always)] +fn bump(c: &AtomicU64) { + if diag_on() { + c.fetch_add(1, Ordering::Relaxed); + } +} + +/// One line, on demand. A decline that names its own reason is the difference +/// between "the tier did not fire" and "the tier fired and found nothing". +pub fn report_segview_counters() { + if !diag_on() { + return; + } + eprintln!( + "[segview] opens={} declines: not_segmenter={} not_grapheme={} segment_patched={} \ + not_string={} not_utf8={} empty={} | nexts={} code_point_at={} materialise_segment={} \ + regexp_test: accepted={} declined={}", + OPENS.load(Ordering::Relaxed), + DECLINE_NOT_SEGMENTER.load(Ordering::Relaxed), + DECLINE_NOT_GRAPHEME.load(Ordering::Relaxed), + DECLINE_SEGMENT_PATCHED.load(Ordering::Relaxed), + DECLINE_NOT_STRING.load(Ordering::Relaxed), + DECLINE_NOT_UTF8.load(Ordering::Relaxed), + DECLINE_EMPTY.load(Ordering::Relaxed), + NEXTS.load(Ordering::Relaxed), + CODE_POINT_ATS.load(Ordering::Relaxed), + MATERIALISE_SEGMENT.load(Ordering::Relaxed), + REGEXP_TEST_ACCEPTED.load(Ordering::Relaxed), + REGEXP_TEST_DECLINED.load(Ordering::Relaxed), + ); +} + +// --- cursor plumbing -------------------------------------------------------- + +#[inline(always)] +fn cursor_ptr(value: f64) -> Option<*mut ObjectHeader> { + let obj = unsafe { crate::object::object_ptr_from_value(value) }? as *mut ObjectHeader; + if unsafe { (*obj).class_id } != SEGMENTS_CURSOR_CLASS_ID { + return None; + } + Some(obj) +} + +#[inline(always)] +fn num_field(obj: *mut ObjectHeader, index: u32) -> usize { + let bits = crate::object::js_object_get_field(obj, index); + let n = JSValue::from_bits(bits.bits()).to_number(); + if n.is_finite() && n >= 0.0 { + n as usize + } else { + 0 + } +} + +#[inline(always)] +fn set_num_field(obj: *mut ObjectHeader, index: u32, value: usize) { + crate::object::js_object_set_field(obj, index, JSValue::number(value as f64)); +} + +/// Run `f` with the cursor's input as a `&str`. The borrow is derived here and +/// dropped at the end of the call; a short (SSO) string is decoded into the +/// caller's stack buffer, so neither case allocates and neither case leaks an +/// address. +#[inline] +fn with_input(cursor: *mut ObjectHeader, f: impl FnOnce(&str) -> R) -> Option { + let value = crate::object::js_object_get_field(cursor, F_INPUT); + let mut sso = [0u8; crate::value::SHORT_STRING_MAX_LEN]; + let bytes = unsafe { crate::string::js_string_key_bytes(JSValue::from_bits(value.bits()), &mut sso) }?; + let text = std::str::from_utf8(bytes).ok()?; + Some(f(text)) +} + +#[cfg(feature = "intl-segmenter")] +fn next_boundary(text: &str, from: usize) -> Option { + if from >= text.len() { + return None; + } + let mut c = unicode_segmentation::GraphemeCursor::new(from, text.len(), true); + match c.next_boundary(text, 0) { + Ok(Some(next)) if next > from => Some(next), + _ => None, + } +} + +#[cfg(not(feature = "intl-segmenter"))] +fn next_boundary(text: &str, from: usize) -> Option { + text[from..].chars().next().map(|c| from + c.len_utf8()) +} + +// --- entry points ----------------------------------------------------------- + +/// `open(segmenter, input)` — a cursor positioned BEFORE the first segment, or +/// `0.0` to mean "take the spec path you already emit". +/// +/// **The decline path has no observable effect of any kind.** The compiler +/// emits `open(X, q)` first and, on a decline, evaluates `X.segment(q)` exactly +/// once in its original position — so a decline must not coerce, allocate, +/// advance or throw. In particular `input` must ALREADY be a string primitive: +/// `build_segments` coerces with `js_jsvalue_to_string`, which runs user +/// `toString`/`valueOf` and **throws on a Symbol**, and doing that here would +/// either run user code twice or move the TypeError out of the spec path. +/// Nothing before the final step allocates. +#[no_mangle] +pub extern "C" fn js_segments_view_open(segmenter: f64, input: f64) -> f64 { + // 1. a pristine Intl.Segmenter whose `segment` is still the builtin. + let Some(obj) = (unsafe { crate::object::object_ptr_from_value(segmenter) }) else { + bump(&DECLINE_NOT_SEGMENTER); + return 0.0; + }; + let obj = obj as *mut ObjectHeader; + if !intl_kind_is_segmenter(obj) { + bump(&DECLINE_NOT_SEGMENTER); + return 0.0; + } + if !segment_method_is_canonical(obj) { + bump(&DECLINE_SEGMENT_PATCHED); + return 0.0; + } + // 2. grapheme only (§4): a resumable word cursor is not equivalent to + // segmenting the whole string, and nothing measured needs one. + if !granularity_is_grapheme(obj) { + bump(&DECLINE_NOT_GRAPHEME); + return 0.0; + } + // 3. an ALREADY-string input, checked before any coercion could happen. + let jv = JSValue::from_bits(input.to_bits()); + if !jv.is_string() { + bump(&DECLINE_NOT_STRING); + return 0.0; + } + // 4. valid UTF-8 and non-empty. A WTF-8 lone surrogate is repaired by + // `segmenter_input_text` on the spec path by COPYING, which a borrowing + // cursor cannot do. + let mut sso = [0u8; crate::value::SHORT_STRING_MAX_LEN]; + let Some(bytes) = (unsafe { crate::string::js_string_key_bytes(jv, &mut sso) }) else { + bump(&DECLINE_NOT_STRING); + return 0.0; + }; + if bytes.is_empty() { + bump(&DECLINE_EMPTY); + return 0.0; + } + if std::str::from_utf8(bytes).is_err() { + bump(&DECLINE_NOT_UTF8); + return 0.0; + } + // 5. only now allocate. The input is held in a rooted handle ACROSS the + // cursor allocation and re-read from it afterwards: this is the + // #9539/#9445 shape in its simplest form — allocate, then store a value + // that predates the allocation. + let scope = crate::gc::RuntimeHandleScope::new(); + let input_h = scope.root_nanbox_f64(input); + let cursor = crate::object::js_object_alloc(SEGMENTS_CURSOR_CLASS_ID, CURSOR_FIELDS); + if cursor.is_null() { + return 0.0; + } + crate::object::js_object_set_field( + cursor, + F_INPUT, + JSValue::from_bits(input_h.get_nanbox_f64().to_bits()), + ); + set_num_field(cursor, F_BYTE_START, 0); + set_num_field(cursor, F_UTF16_START, 0); + set_num_field(cursor, F_BYTE_END, 0); + set_num_field(cursor, F_UTF16_LEN, 0); + bump(&OPENS); + crate::value::js_nanbox_pointer(cursor as i64) +} + +/// Advance to the next grapheme boundary. `1.0` if a segment is now current, +/// `0.0` at the end. **Allocation-free by contract**: three integer field +/// writes and a UAX #29 boundary scan, no arena allocation, no owned `String`, +/// no descriptor insert — which is also why it cannot collect. +#[no_mangle] +pub extern "C" fn js_segments_view_next(cursor: f64) -> f64 { + let Some(c) = cursor_ptr(cursor) else { + return 0.0; + }; + let from = num_field(c, F_BYTE_END); + let utf16_start = num_field(c, F_UTF16_START) + num_field(c, F_UTF16_LEN); + let step = with_input(c, |text| { + next_boundary(text, from).map(|next| (next, super::segmenter::utf16_len(&text[from..next]))) + }) + .flatten(); + let Some((next, seg_u16)) = step else { + return 0.0; + }; + set_num_field(c, F_BYTE_START, from); + set_num_field(c, F_UTF16_START, utf16_start); + set_num_field(c, F_BYTE_END, next); + set_num_field(c, F_UTF16_LEN, seg_u16 as usize); + bump(&NEXTS); + 1.0 +} + +/// `segment.codePointAt(k)` for the CURRENT segment, without materialising it. +/// +/// `k` is a UTF-16 offset **relative to the segment start** and is bounded by +/// the **segment**, not the input: `k` at or past the segment's UTF-16 length +/// is `undefined` even though the input has more code units there. Decoding +/// starts from the cursor's BYTE offset, so `k = 0` is O(1) — calling +/// `js_string_code_point_at` on the input instead would walk from index 0 on +/// any non-ASCII string and make the loop quadratic. +#[no_mangle] +pub extern "C" fn js_segments_view_code_point_at(cursor: f64, k: f64) -> f64 { + let undef = f64::from_bits(crate::value::TAG_UNDEFINED); + let Some(c) = cursor_ptr(cursor) else { + return undef; + }; + if !k.is_finite() || k < 0.0 || k.fract() != 0.0 { + return undef; + } + let k = k as usize; + if k >= num_field(c, F_UTF16_LEN) { + return undef; + } + let start = num_field(c, F_BYTE_START); + let end = num_field(c, F_BYTE_END); + bump(&CODE_POINT_ATS); + with_input(c, |text| { + let seg = &text[start..end]; + let mut utf16_pos = 0usize; + for ch in seg.chars() { + let units = ch.len_utf16(); + if utf16_pos + units > k { + if units == 1 || utf16_pos == k { + // A BMP code point, or the START of a surrogate pair, + // which per spec is the whole code point. + return u32::from(ch) as f64; + } + // `k` lands on the low surrogate half: return the bare unit, + // exactly as `js_string_code_point_at` does. + let v = u32::from(ch) - 0x10000; + return (0xDC00 + (v & 0x3FF)) as f64; + } + utf16_pos += units; + } + undef + }) + .unwrap_or(undef) +} + +/// Materialise the current segment. The compiler emits this for a use it +/// cannot answer from the view — the per-use materialise-on-miss. +#[no_mangle] +pub extern "C" fn js_segments_view_segment(cursor: f64) -> f64 { + let undef = f64::from_bits(crate::value::TAG_UNDEFINED); + let Some(c) = cursor_ptr(cursor) else { + return undef; + }; + let start = num_field(c, F_BYTE_START); + let end = num_field(c, F_BYTE_END); + bump(&MATERIALISE_SEGMENT); + // The allocation happens INSIDE the borrow, so the borrow must not outlive + // it: take the bytes out first, then allocate from a copy on the stack path + // `js_string_from_bytes` performs. Nothing derived from the input survives + // this call. + let made = with_input(c, |text| { + let seg = &text[start..end]; + crate::string::js_string_from_bytes(seg.as_ptr(), seg.len() as u32) + }); + match made { + Some(ptr) if !ptr.is_null() => { + f64::from_bits(JSValue::string_ptr(ptr as *mut StringHeader).bits()) + } + _ => undef, + } +} + +/// `regex.test(segment)` without materialising the segment. **Three-valued**: +/// `true` / `false` / **`undefined` = "I decline"**, on which the compiler +/// materialises and calls the ordinary path. +/// +/// It declines for a global or sticky regex, because `test` is then stateful +/// (`lastIndex` must be consulted and advanced) and that bookkeeping is written +/// against a `StringHeader`. It declines for a patched `RegExp.prototype.test` +/// or an own `test`, because `is RegExp` at the call site does not rule those +/// out and a view-mode test would silently bypass user code. +/// +/// When it accepts, the haystack is a **slice whose bounds are the string's +/// ends**, so `^`, `$` and lookbehind are segment-local — the same answer the +/// materialised call would give, not "a match starting at an offset". +#[no_mangle] +pub extern "C" fn js_segments_view_regexp_test(cursor: f64, regex: f64) -> f64 { + let undef = f64::from_bits(crate::value::TAG_UNDEFINED); + let Some(c) = cursor_ptr(cursor) else { + return undef; + }; + let jv = JSValue::from_bits(regex.to_bits()); + if !jv.is_pointer() { + bump(®EXP_TEST_DECLINED); + return undef; + } + let re = jv.as_pointer::(); + if !crate::regex::is_valid_regex_ptr(re) { + bump(®EXP_TEST_DECLINED); + return undef; + } + // `is RegExp` at the call site does not rule out a patched + // `RegExp.prototype.test`, so the runtime re-checks and declines. + if !crate::object::regex_proto_thunks::regexp_prototype_test_is_canonical(regex) { + bump(®EXP_TEST_DECLINED); + return undef; + } + let start = num_field(c, F_BYTE_START); + let end = num_field(c, F_BYTE_END); + let verdict = with_input(c, |text| { + crate::regex::regexp_test_str_bounded(re, &text[start..end]) + }) + .flatten(); + match verdict { + Some(v) => { + bump(®EXP_TEST_ACCEPTED); + f64::from_bits(JSValue::bool(v).bits()) + } + None => { + bump(®EXP_TEST_DECLINED); + undef + } + } +} + +// Keepalive anchors. The compiler emits calls to these only when the view tier +// fires, so without a reference the bundle link's stub localization can drop +// them before the lowering that needs them is ever compiled — the same reason +// `js_for_of_next` carries `KEEP_JS_FOR_OF_NEXT`. +#[cfg(feature = "keepalive-anchors")] +#[used] +static KEEP_JS_SEGMENTS_VIEW_OPEN: extern "C" fn(f64, f64) -> f64 = js_segments_view_open; +#[cfg(feature = "keepalive-anchors")] +#[used] +static KEEP_JS_SEGMENTS_VIEW_NEXT: extern "C" fn(f64) -> f64 = js_segments_view_next; +#[cfg(feature = "keepalive-anchors")] +#[used] +static KEEP_JS_SEGMENTS_VIEW_CODE_POINT_AT: extern "C" fn(f64, f64) -> f64 = + js_segments_view_code_point_at; +#[cfg(feature = "keepalive-anchors")] +#[used] +static KEEP_JS_SEGMENTS_VIEW_SEGMENT: extern "C" fn(f64) -> f64 = js_segments_view_segment; +#[cfg(feature = "keepalive-anchors")] +#[used] +static KEEP_JS_SEGMENTS_VIEW_REGEXP_TEST: extern "C" fn(f64, f64) -> f64 = + js_segments_view_regexp_test; + +// --- the `open` predicates, all non-allocating after the first intern ------- + +fn interned(name: &[u8]) -> *const StringHeader { + crate::string::intern_ascii_literal(name) +} + +fn string_field_is(obj: *mut ObjectHeader, key: &[u8], expected: &[u8]) -> bool { + let k = interned(key); + if k.is_null() { + return false; + } + let value = crate::object::js_object_get_field_by_name(obj, k); + let mut sso = [0u8; crate::value::SHORT_STRING_MAX_LEN]; + match unsafe { crate::string::js_string_key_bytes(value, &mut sso) } { + Some(bytes) => bytes == expected, + None => false, + } +} + +fn intl_kind_is_segmenter(obj: *mut ObjectHeader) -> bool { + string_field_is(obj, b"__intlKind", b"Segmenter") +} + +fn granularity_is_grapheme(obj: *mut ObjectHeader) -> bool { + string_field_is(obj, b"__intlGranularity", b"grapheme") +} + +/// Is `obj.segment` still the builtin? One inherited lookup catches BOTH an own +/// shadow on the instance and a replaced `Intl.Segmenter.prototype.segment`, +/// because the lookup resolves whatever the call would have resolved. +fn segment_method_is_canonical(obj: *mut ObjectHeader) -> bool { + let k = interned(b"segment"); + if k.is_null() { + return false; + } + let value = crate::object::js_object_get_field_by_name(obj, k); + let jv = JSValue::from_bits(value.bits()); + if !jv.is_pointer() { + return false; + } + let closure = jv.as_pointer::(); + if closure.is_null() { + return false; + } + let entry = crate::closure::get_valid_func_ptr(closure); + entry == super::segmenter::segmenter_segment_thunk as *const u8 + || entry == super::segmenter::segmenter_bound_segment_thunk as *const u8 +} + +#[cfg(test)] +mod view_mode_tests { + use super::*; + + fn js_string(s: &str) -> f64 { + let ptr = crate::string::js_string_from_bytes(s.as_ptr(), s.len() as u32); + f64::from_bits(JSValue::string_ptr(ptr as *mut StringHeader).bits()) + } + + /// A real `Intl.Segmenter` instance, built by the runtime's own + /// constructor path so the test cannot pass against a hand-made object the + /// production code would reject. + fn grapheme_segmenter() -> f64 { + let options = crate::object::js_object_alloc(0, 1); + let key = crate::string::js_string_from_bytes(b"granularity".as_ptr(), 11); + crate::object::js_object_set_field_by_name(options, key, js_string("grapheme")); + // The runtime's OWN constructor path, so the instance carries the same + // internal fields and the same own bound `segment` a real + // `new Intl.Segmenter(...)` produces. A hand-made object would test the + // predicates against something production never sees. + super::super::make_instance( + std::ptr::null(), + super::super::KIND_SEGMENTER, + js_string("en"), + crate::value::js_nanbox_pointer(options as i64), + ) + } + + fn is_undefined(v: f64) -> bool { + JSValue::from_bits(v.to_bits()).is_undefined() + } + + /// Walk the cursor and collect (index, code point at 0, segment string). + fn walk(cursor: f64) -> Vec<(usize, u32, String)> { + let mut out = Vec::new(); + while js_segments_view_next(cursor) == 1.0 { + let c = cursor_ptr(cursor).expect("cursor"); + let cp = js_segments_view_code_point_at(cursor, 0.0); + let seg = js_segments_view_segment(cursor); + let mut sso = [0u8; crate::value::SHORT_STRING_MAX_LEN]; + let bytes = unsafe { + crate::string::js_string_key_bytes(JSValue::from_bits(seg.to_bits()), &mut sso) + } + .expect("segment string"); + out.push(( + num_field(c, F_UTF16_START), + cp as u32, + String::from_utf8_lossy(bytes).into_owned(), + )); + } + out + } + + /// The view must agree with `graphemes(true)` — the same segmentation the + /// spec path uses — on the shapes cc actually renders. + #[test] + fn view_walk_matches_the_spec_segmentation() { + let input = "a\u{301}b\u{1f469}\u{200d}\u{1f4bb}\u{1f1fa}\u{1f1f8}c"; + let cursor = js_segments_view_open(grapheme_segmenter(), js_string(input)); + assert!(cursor != 0.0, "open must accept a pristine grapheme segmenter"); + let got = walk(cursor); + + #[cfg(feature = "intl-segmenter")] + { + use unicode_segmentation::UnicodeSegmentation; + let mut want = Vec::new(); + let mut idx = 0usize; + for g in input.graphemes(true) { + want.push(( + idx, + g.chars().next().map(u32::from).unwrap_or(0), + g.to_string(), + )); + idx += super::super::segmenter::utf16_len(g) as usize; + } + assert_eq!(got, want, "view segmentation must equal graphemes(true)"); + } + assert!(!got.is_empty()); + } + + /// THE FALSIFIER. The two in-loop entry points must move the arena by + /// ZERO, with the minor count pinned so a collection cannot manufacture the + /// zero. This is the whole point of the view mode. + #[test] + fn next_and_code_point_at_allocate_nothing() { + let input = "a\u{301}b\u{1f469}\u{200d}\u{1f4bb}c d e f g h i j k l m n o p"; + let scope = crate::gc::RuntimeHandleScope::new(); + let cursor_h = scope.root_nanbox_f64(js_segments_view_open( + grapheme_segmenter(), + js_string(input), + )); + assert!(cursor_h.get_nanbox_f64() != 0.0); + // Warm: the first call may lazily build anything it builds. + js_segments_view_next(cursor_h.get_nanbox_f64()); + js_segments_view_code_point_at(cursor_h.get_nanbox_f64(), 0.0); + + let minors_before = crate::gc::instruments::copying_minor_cycles(); + let bytes_before = crate::arena::arena_in_use_bytes(); + let mut steps = 0usize; + for _ in 0..200 { + if js_segments_view_next(cursor_h.get_nanbox_f64()) != 1.0 { + // Re-open rather than stop: a short input would otherwise make + // this test pass by doing nothing. + break; + } + let cp = js_segments_view_code_point_at(cursor_h.get_nanbox_f64(), 0.0); + assert!(!is_undefined(cp), "every segment has a code point at 0"); + steps += 1; + } + let bytes_after = crate::arena::arena_in_use_bytes(); + assert!(steps > 5, "the walk must actually have stepped (got {steps})"); + assert_eq!( + crate::gc::instruments::copying_minor_cycles(), + minors_before, + "a collection inside the window would make a zero delta prove nothing" + ); + assert_eq!( + bytes_after.saturating_sub(bytes_before), + 0, + "next + code_point_at allocated {} bytes over {steps} steps", + bytes_after.saturating_sub(bytes_before) + ); + } + + /// The rooting obligation of §9e, exercised rather than asserted: `open` + /// allocates the cursor while holding the input, so a collection landing in + /// that window must not leave a dead value in the traced slot. Force a + /// collection immediately before each `open` and then READ the input back + /// through the cursor: with the handle removed (`PERRY_SABOTAGE_SEGVIEW= + /// norooting`) this is the test that fails. + #[test] + fn open_survives_a_collection_between_its_two_allocations() { + for round in 0..40 { + let input = format!("a\u{301}b{round}\u{1f600}c"); + let s = js_string(&input); + // Churn, so the cursor allocation below is likely to be the one + // that trips the collector, and collect explicitly as well. + let scope = crate::gc::RuntimeHandleScope::new(); + let s_h = scope.root_nanbox_f64(s); + for _ in 0..64 { + let _ = crate::object::js_object_alloc(0, 4); + } + crate::gc::js_gc_collect(); + let cursor = js_segments_view_open(grapheme_segmenter(), s_h.get_nanbox_f64()); + assert!(cursor != 0.0, "open must accept round {round}"); + let mut seen = String::new(); + while js_segments_view_next(cursor) == 1.0 { + let seg = js_segments_view_segment(cursor); + let mut sso = [0u8; crate::value::SHORT_STRING_MAX_LEN]; + let bytes = unsafe { + crate::string::js_string_key_bytes( + JSValue::from_bits(seg.to_bits()), + &mut sso, + ) + } + .expect("segment string"); + seen.push_str(&String::from_utf8_lossy(bytes)); + } + assert_eq!( + seen, input, + "the cursor's input slot must survive a collection inside open (round {round})" + ); + } + } + + /// `k` is bounded by the SEGMENT, not the input: reading past the end of a + /// one-unit segment must be `undefined` even though the input continues. + /// A view that clamped to the input would answer the NEXT grapheme. + #[test] + fn code_point_at_is_bounded_by_the_segment() { + let cursor = js_segments_view_open(grapheme_segmenter(), js_string("ab")); + assert!(cursor != 0.0); + assert_eq!(js_segments_view_next(cursor), 1.0); + assert_eq!(js_segments_view_code_point_at(cursor, 0.0), 'a' as u32 as f64); + assert!( + is_undefined(js_segments_view_code_point_at(cursor, 1.0)), + "k past the segment end must be undefined, not the next grapheme" + ); + assert!(is_undefined(js_segments_view_code_point_at(cursor, -1.0))); + assert!(is_undefined(js_segments_view_code_point_at(cursor, 0.5))); + } + + /// A surrogate pair is ONE grapheme and `codePointAt(0)` is the whole code + /// point; `k = 1` is the bare low surrogate, exactly as + /// `js_string_code_point_at` answers on the materialised substring. + #[test] + fn code_point_at_matches_the_materialised_answer_on_a_surrogate_pair() { + let cursor = js_segments_view_open(grapheme_segmenter(), js_string("\u{1f600}x")); + assert_eq!(js_segments_view_next(cursor), 1.0); + assert_eq!(js_segments_view_code_point_at(cursor, 0.0), 0x1f600 as f64); + assert_eq!(js_segments_view_code_point_at(cursor, 1.0), 0xDE00 as f64); + let seg = js_segments_view_segment(cursor); + let ptr = JSValue::from_bits(seg.to_bits()).as_string_ptr(); + assert_eq!( + crate::string::js_string_code_point_at(ptr, 0), + js_segments_view_code_point_at(cursor, 0.0), + "the view must answer exactly what the materialised segment does" + ); + assert_eq!( + crate::string::js_string_code_point_at(ptr, 1), + js_segments_view_code_point_at(cursor, 1.0) + ); + } + + /// Every decline in §9f, and the one that matters most: a non-string input + /// must be refused BEFORE any coercion, and `open` must never throw — the + /// compiler evaluates `X.segment(q)` itself on a decline. + #[test] + fn open_declines_without_side_effects() { + let seg = grapheme_segmenter(); + assert_eq!(js_segments_view_open(seg, js_string("")), 0.0, "empty"); + assert_eq!( + js_segments_view_open(seg, f64::from_bits(crate::value::TAG_UNDEFINED)), + 0.0, + "undefined input must decline, not coerce to \"undefined\"" + ); + assert_eq!(js_segments_view_open(seg, 42.0), 0.0, "number input"); + let obj = crate::object::js_object_alloc(0, 0); + assert_eq!( + js_segments_view_open(seg, crate::value::js_nanbox_pointer(obj as i64)), + 0.0, + "an object input must decline before running toString" + ); + assert_eq!( + js_segments_view_open(crate::value::js_nanbox_pointer(obj as i64), js_string("a")), + 0.0, + "a non-Segmenter receiver must decline" + ); + // A lone surrogate is WTF-8: the spec path repairs it by copying, a + // borrowing cursor cannot, so it declines. + let wtf8 = crate::string::js_string_from_bytes(b"a\xED\xA0\x80b".as_ptr(), 5); + assert_eq!( + js_segments_view_open( + seg, + f64::from_bits(JSValue::string_ptr(wtf8 as *mut StringHeader).bits()) + ), + 0.0, + "invalid UTF-8 must decline" + ); + } + + /// `_regexp_test` answers the same as the materialised call for a plain + /// regex, and DECLINES (three-valued `undefined`) for a global one, whose + /// `test` is stateful in `lastIndex`. + #[cfg(feature = "regex-engine")] + #[test] + fn regexp_test_matches_the_materialised_call_and_declines_when_stateful() { + let cursor = js_segments_view_open(grapheme_segmenter(), js_string("a1")); + assert_eq!(js_segments_view_next(cursor), 1.0); + + let plain = crate::regex::js_regexp_construct(js_string("^[a-z]$"), js_string("")); + let plain_v = f64::from_bits(JSValue::pointer(plain as *const u8).bits()); + let seg = js_segments_view_segment(cursor); + let seg_ptr = JSValue::from_bits(seg.to_bits()).as_string_ptr(); + let materialised = crate::regex::js_regexp_test(plain, seg_ptr) != 0; + let viewed = js_segments_view_regexp_test(cursor, plain_v); + assert!(!is_undefined(viewed), "a plain regex must be accepted"); + assert_eq!( + crate::value::js_is_truthy(viewed) != 0, + materialised, + "the view answer must equal the materialised answer" + ); + + let global = crate::regex::js_regexp_construct(js_string("[a-z]"), js_string("g")); + let global_v = f64::from_bits(JSValue::pointer(global as *const u8).bits()); + assert!( + is_undefined(js_segments_view_regexp_test(cursor, global_v)), + "a global regex is stateful in lastIndex and must DECLINE" + ); + } + + /// The anchors are segment-local: `^`/`$` must bind to the segment's ends, + /// not the input's. A start-offset match instead of a bounded haystack + /// would make this pass for the first segment and fail for the second. + #[cfg(feature = "regex-engine")] + #[test] + fn regexp_test_anchors_are_segment_local() { + let cursor = js_segments_view_open(grapheme_segmenter(), js_string("ab")); + let anchored = crate::regex::js_regexp_construct(js_string("^b$"), js_string("")); + let v = f64::from_bits(JSValue::pointer(anchored as *const u8).bits()); + assert_eq!(js_segments_view_next(cursor), 1.0); // "a" + assert_eq!( + crate::value::js_is_truthy(js_segments_view_regexp_test(cursor, v)), + 0, + "^b$ must not match the segment \"a\"" + ); + assert_eq!(js_segments_view_next(cursor), 1.0); // "b" + assert_ne!( + crate::value::js_is_truthy(js_segments_view_regexp_test(cursor, v)), + 0, + "^b$ MUST match the segment \"b\" — the haystack's bounds are the \ + segment's ends, so the anchors are segment-local" + ); + } +} diff --git a/crates/perry-runtime/src/object/iterator_prototypes.rs b/crates/perry-runtime/src/object/iterator_prototypes.rs index 98e53d1fbe..1865aa9fed 100644 --- a/crates/perry-runtime/src/object/iterator_prototypes.rs +++ b/crates/perry-runtime/src/object/iterator_prototypes.rs @@ -494,6 +494,31 @@ pub(crate) unsafe fn call_overridden_iterator_next( if proto.with_const_ptr::(|proto| proto.is_null()) { return None; } + // The null-tower proof above is dead on any program that has allocated + // one iterator: `attach_iterator_prototype` materializes the tower at the + // FIRST iterator allocation, so every builtin advance after that reached + // the by-name lookup below and minted a fresh "next" key string just to + // learn nothing was patched — one 24-byte string per `for…of` step, on + // every array / Map / Set / string iterator in the program (~137,000 per + // 400-character claude-code reply; the second 32-byte site of the + // 2026-09-06 allocation census). + // + // Allocation-free proof of "not overridden": the prototype's OWN `next` + // slot still holds a closure whose native entry is the canonical thunk, + // AND no accessor descriptor is recorded for "next" on it. The own read + // is the certified non-allocating leaf (#9480); the accessor check is + // the per-key Bloom bit `set_accessor_descriptor` sets BEFORE inserting + // (#6759 C2), needed because `defineProperty(proto, "next", {get})` on + // an existing data property leaves the old closure in the slot and puts + // the accessor in the side table. Anything else — replaced, deleted, + // accessor, a bound copy — takes the by-name path, unchanged. + // The closure body is NOT covered by the enclosing `unsafe fn`'s implicit + // unsafe block, so the call is spelled out. + if proto.with_const_ptr::(|proto| unsafe { + prototype_next_is_canonical(proto, canonical) + }) { + return None; + } let key = scope.root_raw_const_ptr(crate::string::js_string_from_bytes(b"next".as_ptr(), 4)); let method = proto.with_const_ptr::(|proto| { key.with_const_ptr::(|key| { @@ -517,3 +542,221 @@ pub(crate) unsafe fn call_overridden_iterator_next( Err(error) => crate::exception::js_throw(error), } } + +/// Does `proto`'s OWN `next` data slot hold a closure whose native entry is +/// `canonical`, with no accessor descriptor recorded for `"next"`? A `true` +/// proves the prototype's `next` is the builtin (a user restoring the +/// original closure object after a patch matches too, by entry rather than +/// by object identity); a `false` proves nothing and the caller must run the +/// full by-name lookup. Reads only: no allocation, no collection point. +#[inline] +unsafe fn prototype_next_is_canonical(proto: *const ObjectHeader, canonical: *const u8) -> bool { + let own = super::js_object_get_own_field_or_undef( + crate::value::js_nanbox_pointer(proto as i64), + b"next".as_ptr(), + 4, + ); + if !JSValue::from_bits(own.to_bits()).is_pointer() { + return false; + } + let own_ptr = crate::value::js_nanbox_get_pointer(own) as *const crate::closure::ClosureHeader; + if own_ptr.is_null() || crate::closure::get_valid_func_ptr(own_ptr) != canonical { + return false; + } + !super::descriptor_state::may_have_descriptor_entry(proto as usize, "next", true) +} + +/// The prototype-override probe must be free on the path every real program +/// takes: tower materialized (any iterator allocation does that), nothing +/// patched. Before this module's `prototype_next_is_canonical`, that path +/// allocated a "next" key string per call — the second-largest 32-byte +/// allocation site of a claude-code reply (2026-09-06 census, ~137,000 per +/// 400 characters), mislabelled there as a substring copy. +#[cfg(test)] +mod override_probe_allocation_tests { + use super::*; + use crate::closure::ClosureHeader; + use crate::value::{js_nanbox_get_pointer, js_nanbox_pointer, TAG_UNDEFINED}; + + const PATCHED_SENTINEL: f64 = 4242.0; + + extern "C" fn patched_next_thunk(_closure: *const ClosureHeader) -> f64 { + PATCHED_SENTINEL + } + + extern "C" fn accessor_getter_thunk(_closure: *const ClosureHeader) -> f64 { + f64::from_bits(TAG_UNDEFINED) + } + + /// One array iterator, rooted; materializes the tower as a side effect. + unsafe fn rooted_array_iterator( + scope: &crate::gc::RuntimeHandleScope, + ) -> crate::gc::RuntimeHandle<'_> { + let arr = crate::array::js_array_alloc(1); + crate::array::js_array_push_f64(arr, 1.0); + let iter = crate::array::array_values_iter(js_nanbox_pointer(arr as i64)); + assert!( + iterator_prototypes_materialized(), + "premise: allocating an iterator materializes the tower" + ); + scope.root_nanbox_f64(iter) + } + + unsafe fn array_proto() -> *mut ObjectHeader { + ARRAY_ITERATOR_PROTOTYPE_PTR.load(Ordering::Acquire) as *mut ObjectHeader + } + + unsafe fn set_proto_next(value: f64) { + let key = crate::string::js_string_from_bytes(b"next".as_ptr(), 4); + super::super::js_object_set_field_by_name(array_proto(), key, value); + } + + unsafe fn own_next(proto: *const ObjectHeader) -> f64 { + super::super::js_object_get_own_field_or_undef( + js_nanbox_pointer(proto as i64), + b"next".as_ptr(), + 4, + ) + } + + /// The counter, and the falsifier for the fix: N probes on an unpatched + /// iterator with the tower up must bump the arena by ZERO bytes. Before + /// the fix every probe minted a 24-byte "next" string (32 B rounded), so + /// this read N × 32 — the number the census reported per grapheme. + #[test] + fn probe_on_an_unpatched_iterator_allocates_nothing() { + unsafe { + let scope = crate::gc::RuntimeHandleScope::new(); + let iter_h = rooted_array_iterator(&scope); + let iter_obj = || js_nanbox_get_pointer(iter_h.get_nanbox_f64()) as *mut ObjectHeader; + + // Warm once: a first call may lazily build anything it builds. + assert!( + call_overridden_iterator_next(iter_obj(), crate::array::ARRAY_ITERATOR_CLASS_ID) + .is_none() + ); + const N: usize = 1000; + let minors_before = crate::gc::instruments::copying_minor_cycles(); + let bytes_before = crate::arena::arena_in_use_bytes(); + for _ in 0..N { + assert!( + call_overridden_iterator_next( + iter_obj(), + crate::array::ARRAY_ITERATOR_CLASS_ID + ) + .is_none(), + "nothing is patched, so the probe must decline" + ); + } + let bytes_after = crate::arena::arena_in_use_bytes(); + assert_eq!( + crate::gc::instruments::copying_minor_cycles(), + minors_before, + "a collection inside the window would make a zero delta prove nothing" + ); + assert_eq!( + bytes_after.saturating_sub(bytes_before), + 0, + "the override probe allocated {} bytes over {N} calls on an unpatched \ + iterator with the tower materialized (it minted a \"next\" key string per call)", + bytes_after.saturating_sub(bytes_before) + ); + } + } + + /// The fast path must not be too eager: a replaced prototype `next` is + /// still honoured, and restoring the ORIGINAL closure object (what + /// `test_gap_array_iterator_manual_next.ts` (7) does) returns the probe + /// to its allocation-free decline — by native entry, not by identity. + #[test] + fn probe_honours_a_replaced_prototype_next_and_a_restored_one() { + unsafe { + let scope = crate::gc::RuntimeHandleScope::new(); + let iter_h = rooted_array_iterator(&scope); + let iter_obj = || js_nanbox_get_pointer(iter_h.get_nanbox_f64()) as *mut ObjectHeader; + + let original = scope.root_nanbox_f64(own_next(array_proto())); + assert!( + JSValue::from_bits(original.get_nanbox_f64().to_bits()).is_pointer(), + "premise: the prototype carries an own `next` closure" + ); + + let patched = crate::closure::js_closure_alloc(patched_next_thunk as *const u8, 0); + crate::closure::js_register_closure_arity(patched_next_thunk as *const u8, 0); + let patched_h = scope.root_nanbox_f64(js_nanbox_pointer(patched as i64)); + set_proto_next(patched_h.get_nanbox_f64()); + assert!( + !prototype_next_is_canonical(array_proto(), array_iterator_next_thunk as *const u8), + "a replaced prototype `next` must defeat the allocation-free proof" + ); + assert_eq!( + call_overridden_iterator_next(iter_obj(), crate::array::ARRAY_ITERATOR_CLASS_ID), + Some(PATCHED_SENTINEL), + "the replacement installed on the prototype must be the one called" + ); + + set_proto_next(original.get_nanbox_f64()); + assert!( + prototype_next_is_canonical(array_proto(), array_iterator_next_thunk as *const u8), + "restoring the original closure must re-enable the allocation-free proof" + ); + assert!( + call_overridden_iterator_next(iter_obj(), crate::array::ARRAY_ITERATOR_CLASS_ID) + .is_none(), + "after the restore the builtin advance is back" + ); + } + } + + /// `Object.defineProperty(proto, "next", { get })` records the accessor in + /// the descriptor side table and leaves the old data slot behind, so the + /// own-slot read alone would still see the canonical closure. The per-key + /// accessor bit is what makes the proof decline; without it the getter + /// would be silently bypassed. + #[test] + fn probe_declines_when_an_accessor_next_is_defined_on_the_prototype() { + unsafe { + let scope = crate::gc::RuntimeHandleScope::new(); + let _iter_h = rooted_array_iterator(&scope); + let original = scope.root_nanbox_f64(own_next(array_proto())); + assert!( + prototype_next_is_canonical(array_proto(), array_iterator_next_thunk as *const u8), + "premise: unpatched prototype passes the proof" + ); + + let getter = crate::closure::js_closure_alloc(accessor_getter_thunk as *const u8, 0); + crate::closure::js_register_closure_arity(accessor_getter_thunk as *const u8, 0); + let getter_h = scope.root_nanbox_f64(js_nanbox_pointer(getter as i64)); + let key = scope.root_string_ptr(crate::string::js_string_from_bytes(b"next".as_ptr(), 4)); + super::super::js_object_define_accessor( + js_nanbox_pointer(array_proto() as i64), + key.with_const_ptr::(|k| { + f64::from_bits(JSValue::string_ptr(k as *mut crate::StringHeader).bits()) + }), + getter_h.get_nanbox_f64(), + f64::from_bits(TAG_UNDEFINED), + ); + assert!( + !prototype_next_is_canonical(array_proto(), array_iterator_next_thunk as *const u8), + "an accessor `next` on the prototype must defeat the allocation-free proof \ + even though the data slot may still hold the canonical closure" + ); + + // Delete the accessor and put the data property back. The Bloom + // bit is sticky (zeroed only at meta creation), so the PROOF stays + // declined on this prototype for good — conservative: the by-name + // path runs, exactly as before the fix. Only the semantics are + // pinned here: the builtin advance is back. + key.with_const_ptr::(|k| { + super::super::js_object_delete_field(array_proto(), k); + }); + set_proto_next(original.get_nanbox_f64()); + let iter_obj = js_nanbox_get_pointer(_iter_h.get_nanbox_f64()) as *mut ObjectHeader; + assert!( + call_overridden_iterator_next(iter_obj, crate::array::ARRAY_ITERATOR_CLASS_ID) + .is_none(), + "after delete + restore the builtin advance must be back" + ); + } + } +} diff --git a/crates/perry-runtime/src/object/mod.rs b/crates/perry-runtime/src/object/mod.rs index c7b48fc87d..66b691f43c 100644 --- a/crates/perry-runtime/src/object/mod.rs +++ b/crates/perry-runtime/src/object/mod.rs @@ -162,7 +162,7 @@ mod prototype_helpers; mod reflect_support; mod reserved_floor; pub(crate) use reserved_floor::{ensure_reserved_floor_keys, reserved_slot_floor_for_class_id}; -mod regex_proto_thunks; +pub(crate) mod regex_proto_thunks; // #6812 object-owned overflow storage + the legacy thread-local side table. // Split out of this file to stay under the 2000-line CI cap; the sibling // `object::*` modules reach these through `use super::*`, so re-export the diff --git a/crates/perry-runtime/src/object/regex_proto_thunks.rs b/crates/perry-runtime/src/object/regex_proto_thunks.rs index 0c073bb321..db5dba4de6 100644 --- a/crates/perry-runtime/src/object/regex_proto_thunks.rs +++ b/crates/perry-runtime/src/object/regex_proto_thunks.rs @@ -316,6 +316,41 @@ fn regex_instance_or_throw(method: &str) -> *const crate::regex::RegExpHeader { )) } +/// Is `RegExp.prototype.test` still the builtin, for the regex `value`? +/// +/// The `Intl.Segmenter` view mode answers `regex.test(segment)` without +/// materialising the segment, so it must not silently bypass a user +/// replacement. Same allocation-free proof as +/// `iterator_prototypes::prototype_next_is_canonical`: the prototype's OWN +/// `test` slot still holds a closure whose native entry is this module's +/// thunk, AND no accessor descriptor is recorded for `"test"` (a +/// `defineProperty(proto, "test", {get})` leaves the old closure in the data +/// slot). Any other state returns `false` and the caller declines. +#[cfg(feature = "regex-engine")] +pub(crate) fn regexp_prototype_test_is_canonical(value: f64) -> bool { + let proto = super::js_object_get_prototype_of(value); + let jv = crate::value::JSValue::from_bits(proto.to_bits()); + if !jv.is_pointer() { + return false; + } + let proto_obj = jv.as_pointer::() as *mut ObjectHeader; + if proto_obj.is_null() { + return false; + } + let own = super::js_object_get_own_field_or_undef(proto, b"test".as_ptr(), 4); + let own_jv = crate::value::JSValue::from_bits(own.to_bits()); + if !own_jv.is_pointer() { + return false; + } + let closure = own_jv.as_pointer::(); + if closure.is_null() + || crate::closure::get_valid_func_ptr(closure) != regex_proto_test_thunk as *const u8 + { + return false; + } + !super::descriptor_state::may_have_descriptor_entry(proto_obj as usize, "test", true) +} + /// Install the real (brand-checking) `exec`/`test`/`toString`/`compile` /// prototype methods. `compile` is only installed here when the `regex-engine` /// feature is on; the fallback no-op (for builds without an engine) is installed diff --git a/crates/perry-runtime/src/regex.rs b/crates/perry-runtime/src/regex.rs index b6787e3c51..ba9da2366f 100644 --- a/crates/perry-runtime/src/regex.rs +++ b/crates/perry-runtime/src/regex.rs @@ -1249,6 +1249,45 @@ fn regexp_pattern_is_regexp_like(pattern: f64) -> bool { /// regex.test(string) -> boolean #[cfg(feature = "regex-engine")] #[no_mangle] +/// `regex.test(haystack)` where `haystack` is a **bounded slice whose bounds +/// ARE the string's ends** — the primitive the `Intl.Segmenter` view mode needs +/// so a segment can be tested without being materialised. +/// +/// Passing a sub-slice rather than a start offset is the whole point: `^` +/// anchors at the slice start, `$` at its end, and a lookbehind cannot see the +/// preceding grapheme, which is exactly what `test` on the materialised +/// substring means. A start-offset match would be silently wrong for an +/// anchored pattern, which is what claude-code's `oR_` is. +/// +/// Returns `None` — "I decline, materialise and call the normal path" — for a +/// **global or sticky** regex, because `test` is then stateful: it must consult +/// and advance `lastIndex`, and that bookkeeping (`regexp_find_advancing`) is +/// written against a `StringHeader`, not a slice. Answering it from a slice +/// would either lose the update or invent one. +pub(crate) fn regexp_test_str_bounded(re: *const RegExpHeader, hay: &str) -> Option { + if !is_valid_regex_ptr(re) { + return None; + } + unsafe { + if (*re).global || (*re).sticky { + return None; + } + if crate::hot_diag::regex_on() { + diag_note_op(re, crate::hot_diag::RegexOp::Test); + } + if let Some(repeat_matcher) = lookup_repeat_matcher(re) { + return Some(repeat_matcher.regex.find(hay).is_some()); + } + if let Some(fre) = lookup_fancy_regex(re) { + return match fre.is_match(hay) { + Ok(v) => Some(v), + Err(_) => None, + }; + } + Some(lazy::header_std_regex(re).is_match(hay)) + } +} + pub extern "C" fn js_regexp_test(re: *const RegExpHeader, s: *const StringHeader) -> i32 { if !is_valid_regex_ptr(re) || !is_valid_ptr(s) { return 0; diff --git a/crates/perry/tests/issue_9846_iterator_prototype_next_patch.rs b/crates/perry/tests/issue_9846_iterator_prototype_next_patch.rs new file mode 100644 index 0000000000..18bcdae86e --- /dev/null +++ b/crates/perry/tests/issue_9846_iterator_prototype_next_patch.rs @@ -0,0 +1,111 @@ +//! Regression coverage for the allocation-free "is `next` still the builtin?" +//! proof in `object/iterator_prototypes.rs`. +//! +//! The probe that lets a user replacement of `%ArrayIteratorPrototype%.next` +//! (and the Map / Set / String family prototypes) drive `for…of`, spread, +//! `Array.from` and manual `.next()` used to end in a by-name prototype lookup +//! that minted a fresh `"next"` key string on EVERY built-in iterator step. +//! The proof that removed it reads the prototype's own `next` slot and the +//! per-key accessor Bloom bit instead — so every way of *defeating* that proof +//! has to keep working, and every way of *restoring* it has to hand iteration +//! back to the builtin. +//! +//! The expected output below is `node v26.5.1` running the same source +//! (`test-files/test_gap_iterator_prototype_next_patch.ts`), captured +//! 2026-09-06. The discriminating lines are: +//! +//! * `F-bound-copy 100,200` — `orig.bind(other)` has the SAME native entry as +//! the builtin thunk but a different `this`. A proof that compared by native +//! entry alone, without first reading the prototype's own slot, would call +//! the builtin and print `1,2`. +//! * `G-accessor … true` — an accessor `next` installed by `defineProperty` +//! leaves the old closure in the data slot, so the own-slot read alone still +//! sees the canonical closure. Only the accessor Bloom bit makes the proof +//! decline; without it the getter is silently bypassed and `gets` stays 0. +//! * `H true` — a deleted `next` must throw a TypeError, not fall through to +//! the builtin advance. +//! * `I true` x5 — a non-callable prototype `next` (a number, a string, +//! `undefined`, `null`, a plain object) must throw a TypeError. The proof +//! reads the own slot as a RAW value, so each of these has to defeat it: the +//! number and the string never reach `is_pointer`/`get_valid_func_ptr` as a +//! closure, and the plain object passes `is_pointer` but fails the +//! CLOSURE_MAGIC probe inside `get_valid_func_ptr`. + +use std::path::PathBuf; +use std::process::Command; + +const SOURCE: &str = include_str!("../../../test-files/test_gap_iterator_prototype_next_patch.ts"); + +const EXPECTED: &str = "A-forof 2,4,6\n\ +A-spread 8,10\n\ +A-from 12\n\ +A-manual 14 16 true\n\ +B-forof 1,2,3\n\ +B-spread 4,5\n\ +B-manual 7 8 true\n\ +C-forof-empty 0\n\ +C-restored 9\n\ +D-map a=101,b=102\n\ +D-map-restored a,1\n\ +D-set s1,s2\n\ +D-set-restored 3\n\ +E-string A,B\n\ +E-string-restored c,d\n\ +F-same-object 1,2\n\ +F-bound-copy 100,200\n\ +F-restored 3\n\ +G-accessor 1,2 true\n\ +G-restored 4\n\ +H true\n\ +H-restored 5,6\n\ +I number true\n\ +I string true\n\ +I undefined true\n\ +I object true\n\ +I object true\n\ +I-restored 7,8\n"; + +fn perry_bin() -> PathBuf { + PathBuf::from(env!("CARGO_BIN_EXE_perry")) +} + +#[test] +fn patched_iterator_prototype_next_drives_every_iteration_form() { + let dir = tempfile::tempdir().expect("tempdir"); + let entry = dir.path().join("iterator_next_patch.ts"); + let output = dir.path().join("iterator_next_patch_bin"); + std::fs::write(&entry, SOURCE).expect("write entry"); + + let compile = Command::new(perry_bin()) + .current_dir(dir.path()) + .arg("compile") + .arg(&entry) + .arg("-o") + .arg(&output) + .arg("--no-cache") + .output() + .expect("run perry compile"); + assert!( + compile.status.success(), + "perry compile failed\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&compile.stdout), + String::from_utf8_lossy(&compile.stderr) + ); + + let run = Command::new(&output) + .current_dir(dir.path()) + .output() + .expect("run compiled binary"); + assert!( + run.status.success(), + "compiled binary failed\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&run.stdout), + String::from_utf8_lossy(&run.stderr) + ); + assert_eq!( + String::from_utf8_lossy(&run.stdout), + EXPECTED, + "output must match node v26.5.1\nstderr:\n{}", + String::from_utf8_lossy(&run.stderr) + ); +} diff --git a/test-files/test_gap_iterator_prototype_next_patch.ts b/test-files/test_gap_iterator_prototype_next_patch.ts new file mode 100644 index 0000000000..7f66bdac27 --- /dev/null +++ b/test-files/test_gap_iterator_prototype_next_patch.ts @@ -0,0 +1,192 @@ +// A replaced `%ArrayIteratorPrototype%.next` (and the Map / Set / String +// family prototypes) must drive for-of, spread, Array.from and manual calls; +// restoring the ORIGINAL closure must hand iteration back to the builtin. +// The runtime proves "not patched" allocation-free by comparing the +// prototype's own `next` against the builtin thunk, so both directions of a +// replace / restore cycle are exercised on every family, plus an accessor +// `next` on the prototype, which that proof must decline. + +const arrayProto: any = Object.getPrototypeOf([][Symbol.iterator]()); +const mapProto: any = Object.getPrototypeOf(new Map().entries()); +const setProto: any = Object.getPrototypeOf(new Set().values()); +const stringProto: any = Object.getPrototypeOf(""[Symbol.iterator]()); + +function withPatched(proto: any, patch: (orig: any) => any, body: () => void) { + const orig = proto.next; + proto.next = patch(orig); + try { + body(); + } finally { + proto.next = orig; + } +} + +// A: array family, every driver, doubled values through the patch. +withPatched( + arrayProto, + (orig) => + function (this: any) { + const r = orig.call(this); + if (!r.done) r.value = (r.value as number) * 2; + return r; + }, + () => { + const got: number[] = []; + for (const v of [1, 2, 3]) got.push(v); + console.log("A-forof", got.join(",")); + console.log("A-spread", [...[4, 5]].join(",")); + console.log("A-from", Array.from([6].values()).join(",")); + const it = [7, 8].values(); + console.log("A-manual", it.next().value, it.next().value, it.next().done); + }, +); + +// B: after the restore the builtin is back, in every driver. +{ + const got: number[] = []; + for (const v of [1, 2, 3]) got.push(v); + console.log("B-forof", got.join(",")); + console.log("B-spread", [...[4, 5]].join(",")); + const it = [7, 8].values(); + console.log("B-manual", it.next().value, it.next().value, it.next().done); +} + +// C: a second replace after the restore is honoured again (the proof is a +// per-call read, not a one-shot latch). +withPatched( + arrayProto, + () => + function () { + return { done: true, value: undefined }; + }, + () => { + const got: number[] = []; + for (const v of [1, 2]) got.push(v); + console.log("C-forof-empty", got.length); + }, +); +console.log("C-restored", [...[9]].join(",")); + +// D: Map and Set family prototypes, patched and restored. +withPatched( + mapProto, + (orig) => + function (this: any) { + const r = orig.call(this); + if (!r.done) r.value = [r.value[0], (r.value[1] as number) + 100]; + return r; + }, + () => { + const got: string[] = []; + for (const [k, v] of new Map([["a", 1], ["b", 2]])) got.push(k + "=" + v); + console.log("D-map", got.join(",")); + }, +); +console.log("D-map-restored", [...new Map([["a", 1]])].join(",")); +withPatched( + setProto, + (orig) => + function (this: any) { + const r = orig.call(this); + if (!r.done) r.value = "s" + r.value; + return r; + }, + () => { + console.log("D-set", [...new Set([1, 2])].join(",")); + }, +); +console.log("D-set-restored", [...new Set([3])].join(",")); + +// E: String family prototype. +withPatched( + stringProto, + (orig) => + function (this: any) { + const r = orig.call(this); + if (!r.done) r.value = (r.value as string).toUpperCase(); + return r; + }, + () => { + console.log("E-string", [..."ab"].join(",")); + }, +); +console.log("E-string-restored", [..."cd"].join(",")); + +// F: restoring by assigning the very same closure object, then a patch that +// is a bound copy of the original (same algorithm, different function +// object) — the proof compares by builtin entry, so the bound copy must NOT +// be mistaken for the builtin: its `this` is fixed to a different iterator. +{ + const orig = arrayProto.next; + arrayProto.next = orig; + console.log("F-same-object", [...[1, 2]].join(",")); + const other = [100, 200].values(); + arrayProto.next = orig.bind(other); + try { + console.log("F-bound-copy", [...[1, 2]].join(",")); + } finally { + arrayProto.next = orig; + } + console.log("F-restored", [...[3]].join(",")); +} + +// G: an accessor `next` on the prototype is consulted on every step. +{ + const orig = arrayProto.next; + let gets = 0; + Object.defineProperty(arrayProto, "next", { + configurable: true, + get() { + gets++; + return orig; + }, + }); + try { + console.log("G-accessor", [...[1, 2]].join(","), gets > 0); + } finally { + Object.defineProperty(arrayProto, "next", { + value: orig, + writable: true, + enumerable: false, + configurable: true, + }); + } + console.log("G-restored", [...[4]].join(",")); +} + +// H: a deleted prototype `next` makes for-of throw a TypeError; restoring +// it by plain assignment brings the builtin back. +{ + const orig = arrayProto.next; + delete arrayProto.next; + try { + for (const _v of [1]) { + console.log("H-unexpected"); + } + console.log("H", "no-throw"); + } catch (e: any) { + console.log("H", e instanceof TypeError); + } finally { + arrayProto.next = orig; + } + console.log("H-restored", [...[5, 6]].join(",")); +} + +// I: a NON-CALLABLE prototype `next` must throw a TypeError, not be mistaken +// for a pointer. The allocation-free proof reads the own slot as a raw value +// first, so a number, a string and `undefined` each have to defeat it. +for (const bad of [42, "not a function", undefined, null, {}]) { + const orig = arrayProto.next; + arrayProto.next = bad; + try { + for (const _v of [1]) { + console.log("I-unexpected"); + } + console.log("I", typeof bad, "no-throw"); + } catch (e: any) { + console.log("I", typeof bad, e instanceof TypeError); + } finally { + arrayProto.next = orig; + } +} +console.log("I-restored", [...[7, 8]].join(","));