diff --git a/BRIEFING-cpython-core.md b/BRIEFING-cpython-core.md new file mode 100644 index 0000000..2184189 --- /dev/null +++ b/BRIEFING-cpython-core.md @@ -0,0 +1,103 @@ +# Briefing: `cpython-core` compat work (CPython 3.12 test suite) + +_Self-contained handoff doc. Generated 2026-07-17 from the latest committed run._ + +## Context + +This repo (`elide-dev` conformance testsuite) runs upstream conformance suites +against pinned [Elide](https://elide.dev) builds and publishes versioned +reports. `cpython-core` runs CPython 3.12's top-level `Lib/test/test_*` +modules against Elide's Python runtime (GraalPy-based). Coverage is 438 +modules (see `PYTHON.md` and `manifests/cpython-core.toml`): language/object +model, containers, numbers, text/codecs, data formats, import machinery, and +an exploratory full-suite remainder. + +The suite is at **95.9%** with a *pure long tail* — 72 failing cases spread +across 70 distinct root-cause signatures. There is no single big win here; +this is classification and burn-down work, well suited to parallel sub-agents +working one test module at a time. + +## Current state (Elide `1.4.1+20260716.b8b6531`, digest `3d3ea83ed640`) + +| pass | fail | error | skip | total | pass rate (excl. skips) | +|---:|---:|---:|---:|---:|---:| +| 1,690 | 57 | 15 | 103 | 1,865 | **95.9%** | + +Counts are unittest cases/classes as reported by the CPython runner. Latest +reports: + +- `reports/1.4.1+20260716.b8b6531/3d3ea83ed640/cpython-core/impact.md` +- `expectations/cpython-core.ratchet.toml` — known-failure baseline + +## How to run + +```bash +bun run setup # once: deps + submodules +# smoke: +bun run testsuite --elide nightly --suite cpython-core --include 'test_json' --log +# one module under investigation: +bun run testsuite --elide nightly --suite cpython-core --include 'test_builtin' --log +# full suite (~90s): +bun run testsuite --elide nightly --suite cpython-core +# after fixes land in an Elide nightly, re-baseline: +bun run testsuite --elide nightly --suite cpython-core --ratchet +``` + +## Failure analysis (72 failing cases) + +Rollup by test module (from impact.json example ids): + +| failing | module | flavor of failures | +|---:|---|---| +| 17 | `test_builtin` | exec/eval builtins mapping, iterator misbehavior edge cases, shutdown cleanup, subprocess-based cases | +| 6 | `test_descr` | descriptor protocol edges (e.g. Carlo Verre hackcheck multi-inherit) | +| 6 | `test_dict` | dict internals/ordering edge cases | +| 5 | `test_ast` | `ast.AST._fields` NULL-check / attribute deletion behavior | +| 5 | `test_re` | regex engine divergences | +| 4 | `test_exceptions` | exception context/`__context__` details (e.g. test_3114) | +| 4 | `test_weakref` | finalize-at-shutdown, `test_atexit` (subprocess-based) | +| 4 | `test_fstring` | f-string parse/format edges | +| 4 | `test_list` | MemoryError on huge ranges, free-after-iterating | +| 3 each | `test_tuple`, `test_class` | attribute-error message formats, free-after-iterating | +| ~11 | singletons | `test_json.test_tool` (subprocess), `test_math`, `test_extcall`, `test_genexps`, `test_functools`, `test_patma`, … | + +Recurring cross-cutting themes worth knowing before diving in: + +1. **Subprocess-dependent cases** (`script_helper.assert_python_ok`, + `test_json.test_tool` piping, `test_weakref.test_atexit`): blocked on + Elide's `child_process`/subprocess support — the same gap tracked in the + `node-api` briefing. These are *platform limitations* today; consider + classifying them as skips with a comment instead of failures. +2. **Attribute-error message format tests** (`test_class`, `test_descr`): + CPython 3.12 asserts on suggestion-rich error messages ("Did you mean: + ..."). Cheap wins if Elide's error messages can match; otherwise skip- + classify as implementation detail. +3. **GC/refcount-semantics tests** (`free_after_iterating`, weakref + finalization order, MemoryError on `list(range(2**63))`): often + fundamentally different on a JVM runtime. Classify rather than chase. +4. **Genuine runtime defects** (`test_re`, `test_fstring`, `test_ast`, + `test_builtin` non-subprocess cases): real fixes, one module at a time. + +## Recommended workflow (parallelizable) + +This suite is ideal for a fan-out: one agent per test module, each tasked to + +1. run `--include '' --log` and reproduce, +2. classify each failure: **(a)** Elide defect / **(b)** platform limitation + (subprocess, GC semantics, native details) / **(c)** harness gap, +3. for (a): produce a minimal Python repro + pointer into the Elide runtime; + for (b): propose a `[skip]` entry with a one-line justification; + for (c): file against this repo. + +Start with `test_builtin` (17), `test_descr`/`test_dict` (12), then +`test_ast`/`test_re` (10). + +## Notes for the implementer + +- Runtime fixes happen in the Elide repo (GraalPy-based Python engine); this + repo pins nightlies and measures. +- 103 skips are already classified in `expectations/cpython-core.toml` and + excluded from the denominator; keep skip entries commented. +- The pass rate is already 95.9% — the value here is *classification honesty* + (separating defects from platform limits) more than raw number movement. +- After a fix lands: full run → check `changes.md` → `--ratchet` → commit. diff --git a/BRIEFING-javac-jtreg.md b/BRIEFING-javac-jtreg.md new file mode 100644 index 0000000..04b7005 --- /dev/null +++ b/BRIEFING-javac-jtreg.md @@ -0,0 +1,93 @@ +# Briefing: `javac-jtreg` compat work (OpenJDK langtools javac suite) + +_Self-contained handoff doc. Generated 2026-07-17 from the latest committed run._ + +## Context + +This repo (`elide-dev` conformance testsuite) runs upstream conformance suites +against pinned [Elide](https://elide.dev) builds and publishes versioned +reports. `javac-jtreg` runs the full OpenJDK langtools `tools/javac` tree +(5,554 jtreg `.java` test files, 100% coverage — see `JAVAC.md` and +`manifests/javac-langtools.toml`) through jtreg, compiling with **Elide's +embedded javac** and running generated programs with a regular JDK. + +The suite is at **98.7%** with only 47 failures. This is precision work: +most failures sit in one opaque bucket that needs classification before +fixing, so step one is diagnosis, not patching. + +## Current state (Elide `1.4.1+20260716.b8b6531`, digest `3d3ea83ed640`) + +| pass | fail | error | skip | total | pass rate (excl. skips) | +|---:|---:|---:|---:|---:|---:| +| 3,655 | 45 | 2 | 232 | 3,934 | **98.7%** | + +Counts are jtreg test executions. Latest reports: + +- `reports/1.4.1+20260716.b8b6531/3d3ea83ed640/javac-jtreg/impact.md` +- `expectations/javac-jtreg.ratchet.toml` — known-failure baseline +- Note: `tools/javac/processing` failures are muted/skipped per earlier triage + (annotation processing), and 232 skips are classified in + `expectations/javac-jtreg.toml`. + +## How to run + +```bash +bun run setup # once: deps + submodules + sparse checkout +# smoke: +bun run testsuite --elide nightly --suite javac-jtreg --include 'tools/javac/IllDefinedOrderOfInit.java' --log +# one failing test with full output: +bun run testsuite --elide nightly --suite javac-jtreg --include 'tools/javac/T6411379.java' --log --show-failure-output +# full suite (~12 min): +bun run testsuite --elide nightly --suite javac-jtreg +# after fixes land in an Elide nightly, re-baseline: +bun run testsuite --elide nightly --suite javac-jtreg --ratchet +``` + +## Failure analysis (47 failing executions, 11 signatures) + +| count | signature | examples | +|---:|---|---| +| 36 | `Compilation failed: Compilation failed` (opaque — needs classification) | `tools/javac/MethodParameters/ClassReaderTest/ClassReaderTest.java`, `T6411379`, `T6423583`, `T6458749`, `T6855236` | +| 2 | test timed out | `tools/javac/file/zip/T6836682.java` | +| 6 | output does not match reference `.out` file | `T8181464/LambdaInAnnotationsCausesNPETest{1,2,3}`, `T8171332/Processor`, `7129225/TestImportStar`, `annotations/neg/8171322/TypeVariableAsAnnotationTest`, `typeAnnotations/failures/TypeVariableMissingTA` | +| 1 | wrong compiler output line count | `tools/javac/newlines/NewLineTest.java` | +| 1 | toolbox javac task rc != 0 | `tools/javac/platform/PlatformProviderTest.java` | +| 1 | AssertionError in test main | (see impact.md) | + +Reading of the buckets: + +1. **The 36-test "Compilation failed" bucket is heterogeneous** — the summary + line carries no diagnostic. First task: re-run each with + `--show-failure-output --log` and bucket the actual javac diagnostics. + Expect a mix of: missing `-parameters`/ClassReader metadata handling + (`MethodParameters/*`), old regression tests exercising unusual compiler + options, and genuine Elide-javac divergences from stock javac. +2. **Reference-file diffs (6)** are usually *diagnostic formatting* + divergences (error message wording/position), clustered around annotation + processing and lambda-in-annotation error recovery. Compare Elide's output + with the `.out` golden files; these can be quick fixes if the embedded + javac is just a version step behind the pinned OpenJDK checkout. +3. **Timeouts (2)** are zip-file-handling tests (`file/zip/T6836682`) — check + whether they hang in Elide's filesystem layer or just exceed the limit. + +## Recommended workflow + +1. **Classification pass first** (an agent can do this mechanically): for each + of the 36 opaque failures, run with `--show-failure-output`, capture the + real diagnostic, and produce a table: test → actual error → suspected + cause (Elide javac defect / test needs specific JDK feature / harness gap). +2. Group the classified failures and hand coherent groups to the compiler + team: `MethodParameters/*` is likely one fix; annotation-related reference + diffs likely another. +3. Skip-classify anything that depends on javac internals Elide intentionally + diverges from (document in `expectations/javac-jtreg.toml`). + +## Notes for the implementer + +- Compiler fixes happen in Elide's embedded javac (Elide repo); this repo + pins nightlies and measures. +- At 98.7% with 47 failures, expectations-honesty matters more than raw rate: + every remaining failure should end up either fixed or annotated-skip with a + reason. +- After a fix lands: full run → check `changes.md` (new passes, no + regressions) → `--ratchet` → commit reports + expectations together. diff --git a/BRIEFING-node-api.md b/BRIEFING-node-api.md new file mode 100644 index 0000000..fc2dcc8 --- /dev/null +++ b/BRIEFING-node-api.md @@ -0,0 +1,129 @@ +# Briefing: `node-api` compat work (Node.js core test suite) + +_Self-contained handoff doc. Generated 2026-07-17 from the latest committed run._ + +## Context + +This repo (`elide-dev` conformance testsuite) runs upstream conformance suites +against pinned [Elide](https://elide.dev) builds and publishes versioned +reports. The `node-api` suite runs a sparse slice of Node.js core's +`test/parallel/` JS tests (assert, async_hooks, buffer, console, dns, events, +fs, module/require, path, process, stream, timers, vm, worker groups — see +`manifests/node-api.toml`). Tests run against a checked-in `common/` overlay +(`node-api-overlay`), so `require('../common')` works without full Node +internals. + +This is the lowest-scoring suite (25.4%) and was, until today, hidden from the +README summary table. It is now visible; the goal is to make the number honest +and then move it. + +## Current state (Elide `1.4.1+20260716.b8b6531`, digest `3d3ea83ed640`) + +| pass | fail | error | skip | total | pass rate (excl. skips) | +|---:|---:|---:|---:|---:|---:| +| 309 | 875 | 34 | 345 | 1,563 | **25.4%** | + +Counts are whole test *files* (one Node test file = one result). Latest reports: + +- `reports/1.4.1+20260716.b8b6531/3d3ea83ed640/node-api/impact.md` — 481 + root-cause signatures, largest first (4,800 lines; read the top 100) +- `.../node-api/impact.json` — machine-readable (`bySignature`) +- `expectations/node-api.ratchet.toml` — known-failure baseline + +## How to run + +```bash +bun run setup # once: deps + submodules + sparse checkouts +# smoke: +bun run testsuite --elide nightly --suite node-api --include 'test/parallel/test-path.js' --log +# a target group: +bun run testsuite --elide nightly --suite node-api --include 'test/parallel/test-buffer*.js' --log +# full suite (~6 min): +bun run testsuite --elide nightly --suite node-api +# after fixes land in an Elide nightly, re-baseline: +bun run testsuite --elide nightly --suite node-api --ratchet +``` + +## Failure analysis (909 failing files) + +### Systemic root causes (fix once, unblock many) + +1. **`__filename` / `process.argv[1]` is the literal string `"unknown"`** — + **~49 files directly**, likely more indirectly. Symptoms across several + signatures: + - `The worker script or module filename ... Received "unknown"` (31+2+1+1) + - `ENOENT: no such file or directory, open 'unknown'` (6+3+1+1) — tests + that re-open `__filename` + Elide isn't propagating the entry-script path to `process.argv`/module + metadata when launched by the harness. Almost certainly ONE runtime fix. + **Best effort/reward ratio in the entire cross-suite triage.** +2. **`node:child_process` unimplemented** — **~39 files direct**: + `spawnSync()` (14), `spawn()` (10+3), `exec()` (5), `fork()` (4), + `execFileSync()` (3). Also implicated in some of the 34 timeouts and in + cpython-core failures (subprocess-based tests) — cross-suite leverage. + High complexity; treat as a project, not a quick win. +3. **`node:http` (+ `_http_common`) missing** — 64 files fail at + `Cannot load module: 'http'`. Very large surface; park unless http is + already on Elide's roadmap. Alternatively add these to `[skip]` with a + comment to make the pass rate reflect reachable tests. +4. **Timeouts** — 34 files (`Node API test timed out`), e.g. + `test-fs-promises-watch.js`. Mostly fs.watch / event-loop-keepalive + related; diagnose a couple before assuming one cause. + +### Module-level clusters (each a coherent mini-project) + +- **buffer** (~80 files): assorted semantic gaps — `buffer.constants` + undefined, `Buffer.from` on SharedArrayBuffer/objects with `valueOf` + (`unsupported input type`, 3), `buf.copy` return value undefined (8), + inspect truncation format (20-bucket), negative-allocation / range checks + missing (`Missing expected exception`, part of 36+7+4). +- **events / EventEmitter** (~25): symbol event names + (`Cannot convert a Symbol value to a string`, 4), `receiver is not an + EventEmitter` when using prototype-less receivers (4), + `listeners side-effects` (`Cannot convert undefined or null to object`, 3), + listener-count semantics. +- **streams** (~90): `Readable.toWeb` missing (4), `Stream is not a + constructor` from `require('stream')` shape (3), many + `mustCall` count mismatches (13+10+9+6...) pointing at async scheduling / + destroy/finish event-ordering divergences. +- **process** (~40): `process.chdir()` (5) and `process.getuid()` (3) throw + `UnsupportedOperationException`, `execve` missing (3+2), + `hasUncaughtExceptionCaptureCallback` missing (3), + `getactiveresources` tracking (2+2+...), env delete semantics. +- **vm** (~15): internal `Symbol(elide.vm.context)` leaks into + `Object.getOwnPropertyNames`/`ownKeys` of contextified objects (3 — + trivial fix: filter the internal symbol), `defineProperty called on + non-object` on context args (2), property forwarding. +- **timers** (~10): `timeout.refresh()` missing (3), timer cancellation + warnings (3), active-resource tracking. +- **worker_threads** (~10 beyond the argv issue): structured-clone + `DOMException` on SharedArrayBuffer/MessagePort transfer (4). + +## Ranked work items + +1. **argv/`__filename` "unknown" fix** — ~49+ files, complexity **low**. Do + this first; re-run to see the true shape of worker/fs failures behind it. +2. **vm internal-symbol filter** — 3 files, complexity **trivial**. Good + warm-up in the same session as (1). +3. **buffer semantics sweep** — ~40–80 files, complexity **medium**, highly + parallelizable (each signature is independent). Run + `--include 'test/parallel/test-buffer*.js' --log` and burn down. +4. **process misc (`chdir`, `getuid`, capture-callback, refresh)** — ~15 + files, complexity **low-medium** each. +5. **events/EventEmitter conformance** — ~25 files, complexity **medium**. +6. **child_process `spawn`/`spawnSync`** — ~39 files + cross-suite + (cpython-core) leverage, complexity **high**. Schedule as its own project. +7. **streams event-ordering** — large but diffuse; recommend AFTER (1) and + (3), since many stream tests also touch buffer/process gaps. +8. **`node:http`** — 64 files; park or skip-classify for now. + +## Notes for the implementer + +- Fixes happen in the Elide runtime repo; this repo pins nightlies and + measures (`--elide nightly` picks up a fresh build). +- One file = one result here, so "N files" = N report-visible wins. +- The `common/` overlay lives in the harness work dir + (`.harness/work/node-api/node-api-overlay/test/common/index.js`); `mustCall` + mismatch stack traces point there, not at a runtime bug in itself. +- After a fix lands: full run → check `changes.md` (new passes, no + regressions) → `--ratchet` → commit reports + expectations together. diff --git a/BRIEFING-test262.md b/BRIEFING-test262.md new file mode 100644 index 0000000..08c7b79 --- /dev/null +++ b/BRIEFING-test262.md @@ -0,0 +1,104 @@ +# Briefing: `test262` compat work (TC39 JavaScript conformance) + +_Self-contained handoff doc. Generated 2026-07-17 from the latest committed run._ + +## Context + +This repo (`elide-dev` conformance testsuite) runs upstream conformance suites +against pinned [Elide](https://elide.dev) builds and publishes versioned +reports. `test262` is TC39's official JavaScript conformance suite, run at +100% file coverage (53,658 files: language, built-ins, intl402, annexB, +staging, harness self-tests — see `JAVASCRIPT.md`). It runs through the stock +`test262-harness` + `eshost` with a custom eshost `elide` host. + +The suite is at **99.8%** — this is polish work. The remaining failures are +few but concentrated, so a single feature closes over a third of them. + +## Current state (Elide `1.4.1+20260716.b8b6531`, digest `3d3ea83ed640`) + +| pass | fail | error | skip | total | pass rate (excl. skips) | +|---:|---:|---:|---:|---:|---:| +| 92,589 | 14 | 171 | 10,587 | 103,361 | **99.8%** | + +Counts are test executions (a file can run in both default and strict mode). +185 failing executions total. Latest reports: + +- `reports/1.4.1+20260716.b8b6531/3d3ea83ed640/test262/impact.md` — failures + by root-cause signature; `impact.json` also has a `byFeature` rollup +- `expectations/test262.ratchet.toml` — known-failure baseline + +## How to run + +```bash +bun run setup # once: deps + submodules +# quick slice (seconds): +bun run testsuite --elide nightly --suite test262 --include 'test/built-ins/Error/prototype/stack/**/*.js' --log +# full suite (~20 min): +bun run testsuite --elide nightly --suite test262 +# after fixes land in an Elide nightly, re-baseline: +bun run testsuite --elide nightly --suite test262 --ratchet +``` + +## Failure analysis (185 failing executions, by feature) + +From `impact.json` `byFeature`: + +| failing | feature | notes | +|---:|---|---| +| 70 | `error-stack-accessor` | **the** big bucket — see below | +| 18 | `dynamic-import` | assorted `import()` semantics | +| 17 | `TypedArray` | mostly immutable-ArrayBuffer-adjacent (proposal-stage) | +| 15 | `align-detached-buffer-semantics-with-web-reality` | detached-buffer edge cases | +| 11 | `BigInt` | overlaps TypedArray internals (`Set/BigInt/*-tobigint`) | +| 10 | `Proxy` | trap-order/invariant edges | +| 8 | `let` | TDZ-related, overlaps module bindings below | +| ~45 | module instantiation / TDZ | see below | + +Signature view of the top clusters: + +1. **`Error.prototype.stack` accessor (error-stack proposal) — 70 tests.** + `Expected no error, got TypeError: Cannot read property ... of undefined` + in `test/built-ins/Error/prototype/stack/*` (getter-cross-realm, + getter-data-property-shadows, getter-error-as-prototype, + instance-not-enumerable, …). Spec: `stack` should be an **accessor pair on + `Error.prototype`**, not an own data property on instances. One contained + feature; well-specified by the proposal's tests. **Do this first.** +2. **Module named-binding instantiation / TDZ — ~42 tests.** + `test/language/module-code/instn-*` (instn-named-bndng-*, instn-iee-*, + namespace delete-exported-uninit, …). Expected `ReferenceError` for + accesses to uninitialized (TDZ) module bindings is not thrown; indirect + export bindings resolve too early. Engine-level (GraalJS/Elide module + semantics) — **medium-hard**, but it's the second-biggest bucket. +3. **Global declaration instantiation errors — ~13 tests.** + `test/language/global-code/script-decl-*`: missing `TypeError` when + declaring over a non-configurable global / non-extensible globalThis, and + missing `SyntaxError` for lexical/var collisions across scripts. +4. **Source-phase imports — 4 tests** (`Module not found` for + `source-phase-import/*`) and **ShadowRealm globalThis configurability — 2**. +5. Long tail (~50): Proxy invariants, detached-buffer semantics, BigInt + ToBigInt coercion order, dynamic-import edge cases. Individually small. + +## Ranked work items + +1. **error-stack-accessor** — 70 tests, complexity **medium-low**, one + feature. Verify with + `--include 'test/built-ins/Error/prototype/stack/**/*.js'`. +2. **Module TDZ / binding instantiation** — ~42 tests, complexity + **medium-hard**. Worth scoping in the engine before committing; the tests + are all in `test/language/module-code/`. +3. **Global-code declaration instantiation** — ~13 tests, complexity + **medium**. `CanDeclareGlobalFunction`/`CanDeclareGlobalVar` error paths. +4. **Long tail** — pick off Proxy/TypedArray/BigInt singletons + opportunistically; not worth a dedicated project. + +## Notes for the implementer + +- Fixes happen in the Elide runtime (GraalJS-based); this repo pins nightlies + and measures. `--elide nightly` picks up a fresh build. +- 10,587 skips are intentional (see `expectations/test262.toml`) and excluded + from the pass-rate denominator. +- Some failing features are proposal-stage (immutable ArrayBuffer, source- + phase imports). If Elide won't ship them yet, classify as `[skip]` with a + comment rather than leaving them as permanent failures. +- After a fix lands: full run → check `changes.md` (new passes, no + regressions) → `--ratchet` → commit reports + expectations together. diff --git a/BRIEFING-wpt-wintertc.md b/BRIEFING-wpt-wintertc.md new file mode 100644 index 0000000..d5ab603 --- /dev/null +++ b/BRIEFING-wpt-wintertc.md @@ -0,0 +1,122 @@ +# Briefing: `wpt-wintertc` compat work (WPT / WinterTC suite) + +_Self-contained handoff doc. Generated 2026-07-17 from the latest committed run._ + +## Context + +This repo (`elide-dev` conformance testsuite) runs upstream conformance suites +against pinned [Elide](https://elide.dev) builds and publishes versioned +reports. The `wpt-wintertc` suite is a sparse Web Platform Tests slice covering +WinterTC / ECMA-429 JavaScript-facing APIs: **encoding** (40 files), **url** +(28 files), and **fetch** (245 files). The manifest is +`manifests/wintertc-wpt-2025.toml`; the WPT checkout is a sparse submodule +under `suites/wpt`. + +**This suite has the single largest and most concentrated compat wins across +all five suites in the repo.** + +## Current state (Elide `1.4.1+20260716.b8b6531`, digest `3d3ea83ed640`) + +| pass | fail | error | skip | total | pass rate (excl. skips) | +|---:|---:|---:|---:|---:|---:| +| 10,480 | 3,422 | 12 | 661 | 14,575 | **75.3%** | + +Counts are WPT *subtests* (`file :: subtest name`). Latest reports: + +- `reports/1.4.1+20260716.b8b6531/3d3ea83ed640/wpt-wintertc/impact.md` — failures clustered by root-cause signature, largest first +- `.../wpt-wintertc/impact.json` — same data, machine-readable (`bySignature`) +- `.../wpt-wintertc/wpt-wintertc.md` — full run report +- `expectations/wpt-wintertc.ratchet.toml` — the known-failure baseline (3,438 lines) + +## How to run + +```bash +bun run setup # once: deps + submodules + sparse checkouts +# smoke (seconds): +bun run testsuite --elide nightly --suite wpt-wintertc --include 'url/urlsearchparams-constructor.any.js' --log +# a target area: +bun run testsuite --elide nightly --suite wpt-wintertc --include 'encoding/textdecoder-fatal-single-byte.any.js' --log +# full suite (~12s of harness runtime inside Docker): +bun run testsuite --elide nightly --suite wpt-wintertc --threads 8 +# after fixes land in an Elide nightly, re-baseline: +bun run testsuite --elide nightly --suite wpt-wintertc --ratchet +``` + +Green = zero regressions vs the expectations baseline; new passes show up in +`changes.md` and are locked in by `--ratchet`. + +## Failure analysis (3,434 failing subtests, top-down) + +Failures are extremely concentrated. Directory rollup: + +| failing | where | root cause | +|---:|---|---| +| 1,158 | `fetch/api/**` | no WPT server: relative URLs fail (`Failed to parse URL: ../resources/...`), promise_test unhandled rejections | +| 1,092 | `encoding/textdecoder-fatal-single-byte.any.js` | missing WHATWG single-byte encodings + wrong fatal-mode behavior | +| 342 | `encoding/legacy-mb-schinese/**` | gbk/gb18030 decoder tables wrong/missing | +| 136 | `fetch/fetch-later/**` | `fetchLater` not defined (browser-only API; also needs window/iframe) | +| 136 | `fetch/http-cache/**` | needs WPT server | +| 105 | `url/url-setters-stripping.any.js` | URL setters don't strip U+0000/tab/newline per spec | +| 66 | `encoding/encodeInto.any.js` | SharedArrayBuffer ctor missing; detached-buffer handling | +| 51 | `encoding/textdecoder-mistakes.any.js` | decoder error-recovery divergences | +| ~350 | rest of `encoding/*`, `url/*`, `fetch/*` | mixed, mostly encoding-related | + +Signature-level highlights (see `impact.md` for the full list): + +- `TextDecoder: encoding ISO-8859-10 / -14 not available on this JVM` — 524 +- `TextDecoder: encoding macintosh not available on this JVM` — 262 +- `TextDecoder: encoding ISO-8859-8-I not available on this JVM` — 226 +- `decoding error (fatal mode): UNMAPPABLE[n]` (should NOT throw) — 68 +- throws `RangeError` where spec wants `TypeError` (fatal-flag tests) — 47 +- `sabConstructor is not a constructor` (no SharedArrayBuffer) — 71 +- URL: `assert_equals: property expected "https:" but got "http:"` etc. — 80; `Invalid URL port` — 24 + +**Root cause for the encoding mass:** Elide's `TextDecoder` delegates to JVM +`Charset`. The JVM lacks several WHATWG-required encodings (ISO-8859-10, +ISO-8859-14, ISO-8859-8-I, macintosh, x-mac-cyrillic, x-user-defined; label +aliases like `unicode11utf8`) and diverges from the Encoding Standard on +error handling (fatal mode, replacement-character emission, ISO-2022-JP state +machine). + +## Ranked work items + +1. **WHATWG single-byte encodings (Elide runtime)** — unlocks **~1,400 + subtests**, complexity **low-medium**. Implement the Encoding Standard's + single-byte decoders directly from its index tables (128 entries each; + https://encoding.spec.whatwg.org/indexes.json) instead of JVM `Charset`. + Include the full label→encoding table and correct fatal/replacement + behavior. Verify with `encoding/textdecoder-fatal-single-byte.any.js` + (1,092 failing), `textdecoder-labels.any.js`, `single-byte-decoder.window.js`. +2. **URL setter stripping (Elide runtime)** — unlocks **~130 subtests**, + complexity **low-medium**. URL component setters must strip tab/newline and + handle U+0000 per the URL Standard; port setter must ignore invalid input + rather than throw. Verify with `url/url-setters-stripping.any.js`. +3. **Legacy multi-byte decoders: gbk/gb18030, ISO-2022-JP (Elide runtime)** — + unlocks **~450 subtests**, complexity **medium** (stateful decoders, larger + index tables). Natural follow-on to item 1 once the index-table + infrastructure exists. +4. **WPT server support in the harness (THIS repo, not Elide)** — un-gates up + to **~1,300 subtests** (`fetch/api`, `fetch/http-cache`, …), complexity + **medium-high**. The runner executes `.any.js` files without a wptserve + instance or base URL, so anything fetching a relative resource dies in + setup. Standing up wptserve (or a minimal substitute honoring WPT's + `{{host}}`/`{{ports}}` substitution) inside the Docker run converts these + from permanently-failing into a real fetch-conformance signal. Expect real + fetch bugs to surface after; that's the point. +5. **Park / skip candidates**: `fetch/fetch-later` (136 — browser-only, + needs window/iframe/BFCache), `.window.js` tests requiring `document`/ + `XMLHttpRequest` (~90). Consider adding these to `[skip]` in + `expectations/wpt-wintertc.toml` with a comment, so the pass-rate + denominator reflects reachable tests. + +Items 1–3 alone should move the suite from 75.3% to roughly **88–90%**. + +## Notes for the implementer + +- Elide fixes happen in the Elide runtime repo; this repo only pins nightlies + and measures. Use `--elide nightly` to pick up a fresh build. +- Failure text like `not available on this JVM` comes from Elide's own error + messages — grep the Elide codebase for it to find the TextDecoder impl fast. +- The suite runs in ~12s, so iterate with `--include 'encoding/**' --log`. +- After a fix lands: run full suite → check `changes.md` for new passes and no + regressions → `--ratchet` → commit reports + expectations together. diff --git a/README.md b/README.md index b2ab83d..bfc3fc6 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,9 @@ Last updated: 2026-07-17T21:06:56.786Z |---|---|---|---:|---:| | cpython-core | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 95.9% | 0 | | javac-jtreg | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 98.7% | 0 | +| node-api | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 25.4% | 0 | | test262 | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 99.8% | 0 | +| wpt-wintertc | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 75.3% | 0 | diff --git a/expectations/cpython-core.ratchet.toml b/expectations/cpython-core.ratchet.toml index 9913b49..d646adf 100644 --- a/expectations/cpython-core.ratchet.toml +++ b/expectations/cpython-core.ratchet.toml @@ -1,12 +1,13 @@ # AUTO-GENERATED by `run --ratchet`. Do not edit by hand. -# Generated for elide 1.4.1+20260716.b8b6531 (sha 3d3ea83ed640) on 2026-07-17T20:47:59.909Z. +# Generated for elide 1.4.1+20260718.acbafa6 (sha unknown) on 2026-07-18T04:03:48.964Z +# from the compliance CI run (linux-amd64) of 2026-07-18; the 20260718 nightly +# expanded the runnable cpython universe from 1,865 to 5,334 cases. [fail] +"test_aifc" = "" "test_ast.test_ast.ASTHelpers_Test.test_literal_eval_str_int_limit" = "" -"test_ast.test_ast.AST_Tests.test_AST_fields_NULL_check" = "" "test_ast.test_ast.AST_Tests.test_exception_groups_feature_version" = "" "test_ast.test_ast.EndPositionTests.test_source_segment_padded" = "" -"test_builtin.BuiltinTest.test_bytearray_join_with_misbehaving_iterator" = "" "test_builtin.BuiltinTest.test_compile" = "" "test_builtin.BuiltinTest.test_compile_top_level_await" = "" "test_builtin.BuiltinTest.test_compile_top_level_await_invalid_cases" = "" @@ -18,58 +19,271 @@ "test_builtin.BuiltinTest.test_exec_globals_dict_subclass" = "" "test_builtin.BuiltinTest.test_exec_globals_error_on_get" = "" "test_builtin.BuiltinTest.test_exec_globals_frozen" = "" -"test_builtin.BuiltinTest.test_hash" = "" "test_builtin.BuiltinTest.test_max" = "" "test_builtin.BuiltinTest.test_min" = "" "test_builtin.BuiltinTest.test_warning_notimplemented" = "" -"test_builtin.ShutdownTest.test_cleanup" = "" -"test_class.ClassTests.testDel" = "" -"test_class.ClassTests.testObjectAttributeAccessErrorMessages" = "" -"test_class.ClassTests.testTypeAttributeAccessErrorMessages" = "" +"test_call" = "" +"test_cgitb.TestCgitb.test_syshook_no_logdir_default_format" = "" +"test_cgitb.TestCgitb.test_syshook_no_logdir_text_format" = "" +"test_codeccallbacks.CodecCallbackTest.test_callbacks" = "" +"test_codeccallbacks.CodecCallbackTest.test_decodehelper" = "" +"test_codeccallbacks.CodecCallbackTest.test_mutating_decode_handler" = "" +"test_codeccallbacks.CodecCallbackTest.test_mutating_decode_handler_unicode_escape" = "" +"test_codeccallbacks.CodecCallbackTest.test_nameescape" = "" +"test_codeccallbacks.CodecCallbackTest.test_unicodedecodeerror" = "" +"test_codeccallbacks.CodecCallbackTest.test_unicodeencodeerror" = "" +"test_codeccallbacks.CodecCallbackTest.test_unicodetranslateerror" = "" +"test_codeccallbacks.CodecCallbackTest.test_uninamereplace" = "" +"test_collections.TestCollectionABCs.test_Buffer" = "" +"test_collections.TestCollectionABCs.test_Set_hash_matches_frozenset" = "" +"test_collections.TestNamedTuple.test_new_builtins_issue_43102" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_compiles_as_much_as_possible" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_d_compile_error" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_d_runtime_error" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_force" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_ignore_symlink_destination" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_include_bad_file" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_include_file_no_arg" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_include_file_with_arg" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_include_on_stdin" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_multiple_dirs" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_multiple_optimization_levels" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_multiple_runs" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_no_args_compiles_path" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_no_args_respects_force_flag" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_no_args_respects_quiet_flag" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_pep3147_paths_doubleoptimize" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_pep3147_paths_normal" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_pep3147_paths_optimize" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_pyc_invalidation_mode" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_quiet" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_recursion_control" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_recursion_limit" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_regexp" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_silent" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_strip_and_prepend" = "" +"test_compileall.CommandLineTestsNoSourceEpoch.test_symlink_loop" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_compiles_as_much_as_possible" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_d_compile_error" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_d_runtime_error" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_force" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_ignore_symlink_destination" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_include_bad_file" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_include_file_no_arg" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_include_file_with_arg" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_include_on_stdin" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_multiple_dirs" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_multiple_optimization_levels" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_multiple_runs" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_no_args_compiles_path" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_no_args_respects_force_flag" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_no_args_respects_quiet_flag" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_pep3147_paths_doubleoptimize" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_pep3147_paths_normal" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_pep3147_paths_optimize" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_pyc_invalidation_mode" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_quiet" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_recursion_control" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_recursion_limit" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_regexp" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_silent" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_strip_and_prepend" = "" +"test_compileall.CommandLineTestsWithSourceEpoch.test_symlink_loop" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_dir_maxlevels" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_dir_pathlike" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_dir_pathlike_prependdir" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_dir_pathlike_stripdir" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_file_encoding_fallback" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_file_pathlike" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_file_pathlike_ddir" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_file_pathlike_prependdir" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_file_pathlike_stripdir" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_files" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_compile_path" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_ddir_empty_only_one_worker" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_ddir_only_one_worker" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_ignore_symlink_destination" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_larger_than_32_bit_times" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_multiple_optimization_levels" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_optimize" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_prepend_only" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_strip_and_prepend" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_strip_only" = "" +"test_compileall.CompileallTestsWithSourceEpoch.test_year_2038_mtime_compilation" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_dir_maxlevels" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_dir_pathlike" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_dir_pathlike_prependdir" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_dir_pathlike_stripdir" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_file_encoding_fallback" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_file_pathlike" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_file_pathlike_ddir" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_file_pathlike_prependdir" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_file_pathlike_stripdir" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_files" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_compile_path" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_ddir_empty_only_one_worker" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_ddir_only_one_worker" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_ignore_symlink_destination" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_larger_than_32_bit_times" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_magic_number" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_mtime" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_multiple_optimization_levels" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_optimize" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_prepend_only" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_strip_and_prepend" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_strip_only" = "" +"test_compileall.CompileallTestsWithoutSourceEpoch.test_year_2038_mtime_compilation" = "" +"test_compileall.EncodingTest.test_error" = "" +"test_compileall.HardlinkDedupTestsNoSourceEpoch.test_duplicated_levels" = "" +"test_compileall.HardlinkDedupTestsNoSourceEpoch.test_import" = "" +"test_compileall.HardlinkDedupTestsNoSourceEpoch.test_recompilation" = "" +"test_compileall.HardlinkDedupTestsWithSourceEpoch.test_duplicated_levels" = "" +"test_compileall.HardlinkDedupTestsWithSourceEpoch.test_import" = "" +"test_compileall.HardlinkDedupTestsWithSourceEpoch.test_recompilation" = "" +"test_contextlib_async.TestAsyncExitStack.test_exit_exception_traceback" = "" "test_defaultdict.TestDefaultDict.test_recursive_repr" = "" -"test_deque.TestSubclass.test_weakref" = "" "test_descr.ClassPropertiesAndMethods.test_carloverre_multi_inherit_invalid" = "" -"test_descr.ClassPropertiesAndMethods.test_cycle_through_dict" = "" -"test_descr.ClassPropertiesAndMethods.test_delete_hook" = "" "test_descr.ClassPropertiesAndMethods.test_descrdoc" = "" "test_descr.ClassPropertiesAndMethods.test_qualname" = "" -"test_descr.ClassPropertiesAndMethods.test_slots" = "" -"test_dict.DictTest.test_free_after_iterating" = "" -"test_dict.DictTest.test_mutating_iteration_delete" = "" -"test_dict.DictTest.test_mutating_iteration_delete_over_items" = "" -"test_dict.DictTest.test_mutating_iteration_delete_over_values" = "" -"test_dict.DictTest.test_oob_indexing_dictiter_iternextitem" = "" "test_dict.DictTest.test_setdefault_atomic" = "" -"test_exceptions.ExceptionTests.testExceptionCleanupState" = "" -"test_exceptions.ExceptionTests.test_3114" = "" "test_exceptions.ExceptionTests.test_invalid_delattr" = "" -"test_exceptions.ExceptionTests.test_memory_error_in_subinterp" = "" "test_extcall" = "" +"test_float.GeneralFloatCases.test_hash_nan" = "" +"test_float.GeneralFloatCases.test_keywords_in_subclass" = "" "test_fstring.TestCase.test_debug_expressions_are_raw_strings" = "" -"test_fstring.TestCase.test_filename_in_syntaxerror" = "" "test_fstring.TestCase.test_gh129093" = "" -"test_fstring.TestCase.test_syntax_warning_infinite_recursion_in_file" = "" -"test_functools.TestCmpToKeyC.test_cmp_to_signature" = "" -"test_genexps.__test__.doctests" = "" -"test_json.test_scanstring.TestCScanstring.test_surrogates" = "" -"test_json.test_tool.TestTool.test_broken_pipe_error" = "" -"test_list.ListTest.test_basic" = "" -"test_list.ListTest.test_free_after_iterating" = "" +"test_funcattrs.FunctionPropertiesTest.test___builtins__" = "" +"test_future_stmt.test_future.AnnotationsFutureTestCase.test_annotations" = "" +"test_future_stmt.test_future.AnnotationsFutureTestCase.test_infinity_numbers" = "" +"test_future_stmt.test_future.FutureTest.test_badfuture3" = "" +"test_future_stmt.test_future.FutureTest.test_badfuture8" = "" +"test_future_stmt.test_future.FutureTest.test_badfuture9" = "" +"test_hmac.CompareDigestTestCase.test_operator_compare_digest" = "" +"test_htmlparser.AttributesTestCase.test_attr_syntax" = "" +"test_htmlparser.AttributesTestCase.test_attr_values" = "" +"test_htmlparser.HTMLParserTestCase.test_broken_invalid_end_tag" = "" +"test_htmlparser.HTMLParserTestCase.test_cdata_section" = "" +"test_htmlparser.HTMLParserTestCase.test_comments" = "" +"test_htmlparser.HTMLParserTestCase.test_eof_in_comments" = "" +"test_htmlparser.HTMLParserTestCase.test_eof_in_declarations" = "" +"test_htmlparser.HTMLParserTestCase.test_eof_no_quadratic_complexity" = "" +"test_int.IntStrDigitLimitsTests.test_int_max_str_digits_is_per_interpreter" = "" +"test_int.IntSubclassStrDigitLimitsTests.test_int_max_str_digits_is_per_interpreter" = "" +"test_ipaddress.AddressTestCase_v6.test_bad_address_split_v6_too_long" = "" +"test_ipaddress.IpaddrUnitTest.testCompressIPv6Address" = "" +"test_ipaddress.IpaddrUnitTest.testIPv6IPv4MappedStringRepresentation" = "" +"test_ipaddress.IpaddrUnitTest.testNetworkV4HashCollisions" = "" +"test_ipaddress.IpaddrUnitTest.testNetworkV6HashCollisions" = "" "test_list.ListTest.test_keywords_in_subclass" = "" "test_list.ListTest.test_list_index_modifing_operand" = "" +"test_list.ListTest.test_list_resize_overflow" = "" "test_math.MathTests.testCbrt" = "" "test_math.MathTests.testExp2" = "" -"test_patma.TestSourceLocations.test_jump_threading" = "" +"test_math.MathTests.test_sumprod_stress" = "" +"test_named_expressions.NamedExpressionInvalidTest.test_named_expression_invalid_mangled_class_variables" = "" +"test_pickle.CDumpPickle_LoadPickle.test_attribute_name_interning" = "" +"test_pickle.CPicklerTests.test_attribute_name_interning" = "" +"test_pickle.CPicklingErrorTests.test_bad_getattr" = "" +"test_pickle.CUnpicklerTests.test_custom_find_class" = "" +"test_pickle.CUnpicklerTests.test_find_class" = "" +"test_pickle.CUnpicklerTests.test_load_global" = "" +"test_pickle.CUnpicklerTests.test_load_stack_global" = "" +"test_positional_only_arg.PositionalOnlyTestCase.test_annotations_constant_fold" = "" +"test_posixpath" = "" +"test_property.PropertySubclassTests.test_docstring_copy2" = "" +"test_property.PropertySubclassTests.test_issue41287" = "" +"test_property.PropertySubclassTests.test_prefer_explicit_doc" = "" +"test_property.PropertySubclassTests.test_property_no_doc_on_getter" = "" +"test_property.PropertySubclassTests.test_property_with_slots_and_doc_slot_docstring_present" = "" +"test_property.PropertySubclassTests.test_property_with_slots_no_docstring" = "" +"test_property.PropertyTests.test_property_set_name_incorrect_args" = "" +"test_pyclbr" = "" "test_range.RangeTest.test_range_constructor_error_messages" = "" "test_re.ReTests.test_bug_40736" = "" -"test_re.ReTests.test_keep_buffer" = "" -"test_re.ReTests.test_locale_caching" = "" -"test_re.ReTests.test_locale_compiled" = "" "test_re.ReTests.test_possible_set_operations" = "" -"test_tuple.TupleTest.test_free_after_iterating" = "" -"test_tuple.TupleTest.test_hash_exact" = "" +"test_reprlib.LongReprTest.test_class" = "" +"test_reprlib.LongReprTest.test_instance" = "" +"test_reprlib.LongReprTest.test_method" = "" +"test_reprlib.LongReprTest.test_module" = "" +"test_reprlib.LongReprTest.test_type" = "" +"test_reprlib.TestRecursiveRepr.test_assigned_attributes" = "" +"test_runpy.RunModuleTestCase.test_run_module" = "" +"test_runpy.RunModuleTestCase.test_run_module_alter_sys" = "" +"test_runpy.RunModuleTestCase.test_run_module_in_namespace_package" = "" +"test_runpy.RunModuleTestCase.test_run_name" = "" +"test_runpy.RunModuleTestCase.test_run_namespace_package" = "" +"test_runpy.RunModuleTestCase.test_run_namespace_package_in_namespace_package" = "" +"test_runpy.RunModuleTestCase.test_run_package" = "" +"test_runpy.RunModuleTestCase.test_run_package_alter_sys" = "" +"test_runpy.RunModuleTestCase.test_run_package_in_namespace_package" = "" +"test_runpy.RunPathTestCase.test_directory_compiled" = "" +"test_runpy.RunPathTestCase.test_script_compiled" = "" +"test_runpy.RunPathTestCase.test_zipfile_compiled" = "" +"test_runpy.TestExit.test_pymain_run_command" = "" +"test_runpy.TestExit.test_pymain_run_command_run_module" = "" +"test_runpy.TestExit.test_pymain_run_file" = "" +"test_runpy.TestExit.test_pymain_run_file_runpy_run_module" = "" +"test_runpy.TestExit.test_pymain_run_file_runpy_run_module_as_main" = "" +"test_runpy.TestExit.test_pymain_run_module" = "" +"test_runpy.TestExit.test_pymain_run_stdin" = "" +"test_sax.BytesXmlgenTest.test_xmlgen_encoding" = "" +"test_sax.BytesXmlgenTest.test_xmlgen_encoding_bytes" = "" +"test_sax.CDATAHandlerTest.test_handlers" = "" +"test_sax.ExpatReaderTest.test_expat_entityresolver_enabled" = "" +"test_sax.ExpatReaderTest.test_expat_external_dtd_enabled" = "" +"test_sax.ExpatReaderTest.test_expat_inpsource_character_stream" = "" +"test_sax.ExpatReaderTest.test_expat_text_file" = "" +"test_sax.ExpatReaderTest.test_flush_reparse_deferral_disabled" = "" +"test_sax.ExpatReaderTest.test_flush_reparse_deferral_enabled" = "" +"test_sax.LexicalHandlerTest.test_handlers" = "" +"test_sax.ParseTest.test_parseString_bytes" = "" +"test_sax.ParseTest.test_parseString_text" = "" +"test_sax.ParseTest.test_parse_bytes" = "" +"test_sax.ParseTest.test_parse_text" = "" +"test_sax.WriterXmlgenTest.test_xmlgen_encoding" = "" +"test_sax.WriterXmlgenTest.test_xmlgen_encoding_bytes" = "" +"test_scope.ScopeTests.testLeaks" = "" +"test_set.TestFrozenSet.test_free_after_iterating" = "" +"test_set.TestFrozenSetSubclass.test_free_after_iterating" = "" +"test_set.TestFrozenSetSubclass.test_keywords_in_subclass" = "" +"test_set.TestSet.test_free_after_iterating" = "" +"test_set.TestSetSubclass.test_free_after_iterating" = "" +"test_stringprep.StringprepTests.test" = "" +"test_structseq.StructSeqTest.test_match_args" = "" +"test_structseq.StructSeqTest.test_match_args_with_unnamed_fields" = "" +"test_structseq.StructSeqTest.test_reference_cycle" = "" +"test_syntax.SyntaxTestCase.test_error_parenthesis" = "" +"test_syntax.SyntaxTestCase.test_invalid_line_continuation_error_position" = "" +"test_syntax.SyntaxTestCase.test_invisible_characters" = "" +"test_syntax.SyntaxTestCase.test_multiline_compiler_error_points_to_the_end" = "" +"test_threading.AtexitTests.test_atexit_after_shutdown" = "" +"test_tokenize.TestRoundtrip.test_random_files" = "" "test_tuple.TupleTest.test_keywords_in_subclass" = "" -"test_weakref.FinalizeTestCase.test_all_freed" = "" -"test_weakref.FinalizeTestCase.test_atexit" = "" -"test_weakref.FinalizeTestCase.test_finalize" = "" -"test_weakref.FinalizeTestCase.test_order" = "" +"test_type_params.TypeParamsManglingTest.test_no_mangling_in_nested_scopes" = "" +"test_unicode_identifiers.PEP3131Test.test_valid" = "" +"test_weakset.TestWeakSet.test_add" = "" +"test_weakset.TestWeakSet.test_intersection" = "" +"test_weakset.TestWeakSet.test_len" = "" +"test_weakset.TestWeakSet.test_len_cycles" = "" +"test_weakset.TestWeakSet.test_len_race" = "" +"test_weakset.TestWeakSet.test_symmetric_difference" = "" +"test_weakset.TestWeakSet.test_union" = "" +"test_weakset.TestWeakSet.test_weak_destroy_and_mutate_while_iterating" = "" +"test_weakset.TestWeakSet.test_weak_destroy_while_iterating" = "" +"test_xml_etree.BasicElementTest.test_weakref" = "" +"test_xml_etree.ElementSlicingTest.test_setslice_negative_steps" = "" +"test_xml_etree.ElementSlicingTest.test_setslice_steps" = "" +"test_xml_etree.ElementTreeTest.test_custom_builder_only_end_ns" = "" +"test_xml_etree.ElementTreeTest.test_encoding" = "" +"test_xml_etree.ElementTreeTest.test_entity" = "" +"test_xml_etree.ElementTreeTest.test_iterparse" = "" +"test_xml_etree.ElementTreeTest.test_simpleops" = "" +"test_xml_etree.IOTest.test_encoding" = "" +"test_xml_etree.IOTest.test_write_to_binary_file_with_bom" = "" +"test_xml_etree.IOTest.test_write_to_user_binary_writer_with_bom" = "" +"test_xml_etree.ParseErrorTest.test_error_position" = "" +"test_xml_etree.TreeBuilderTest.test_doctype" = "" +"test_xml_etree.XMLParserTest.test_parse_string" = "" +"test_xml_etree.XMLParserTest.test_subclass_doctype" = "" +"test_xml_etree.XMLPullParserTest.test_flush_reparse_deferral_disabled" = "" +"test_xml_etree.XMLPullParserTest.test_flush_reparse_deferral_enabled" = "" diff --git a/expectations/cpython-core.toml b/expectations/cpython-core.toml index 09ac4c3..1a8d3c0 100644 --- a/expectations/cpython-core.toml +++ b/expectations/cpython-core.toml @@ -38,4 +38,49 @@ "test_pathlib.WalkTests.*" = "Directory walking deferred until sandboxed file I/O" "test_pathlib.WindowsPathTest.*" = "Concrete pathlib filesystem behavior deferred until sandboxed file I/O" +# --- Platform-limitation classifications (cpython-core triage 2026-07-17) --- +# Subprocess / script_helper.assert_python_ok: Elide has no child_process/subprocess. +"test_builtin.ShutdownTest.test_cleanup" = "Spawns a child interpreter via assert_python_ok; subprocess + interpreter-shutdown finalization unsupported" +"test_exceptions.ExceptionTests.test_memory_error_in_subinterp" = "script_helper.assert_python_ok subprocess + _testcapi subinterpreters unsupported" +"test_weakref.FinalizeTestCase.test_atexit" = "script_helper.assert_python_ok subprocess unsupported" +"test_fstring.TestCase.test_filename_in_syntaxerror" = "assert_python_failure subprocess unsupported" +"test_fstring.TestCase.test_syntax_warning_infinite_recursion_in_file" = "assert_python_ok subprocess + stderr line-count assertion unsupported" +"test_json.test_tool.TestTool.test_broken_pipe_error" = "Spawns json.tool subprocess and asserts EPIPE returncode; subprocess/pipe unsupported" +# GC / refcount / __del__ finalization timing: JVM GC is non-deterministic, no refcount-0 finalization. +"test_descr.ClassPropertiesAndMethods.test_delete_hook" = "__del__ finalization timing; not run synchronously after del+gc.collect on GraalPy" +"test_descr.ClassPropertiesAndMethods.test_slots" = "instance __del__ counting depends on refcount/GC finalization timing" +"test_dict.DictTest.test_free_after_iterating" = "free-after-iterating requires __del__ to run immediately after iteration (refcount finalization)" +"test_list.ListTest.test_free_after_iterating" = "free-after-iterating depends on refcount __del__; JVM GC non-deterministic" +"test_list.ListTest.test_basic" = "MemoryError not raised on huge list(range(maxsize//2)) allocation under the JVM" +"test_tuple.TupleTest.test_free_after_iterating" = "free-after-iterating depends on refcount __del__; JVM GC non-deterministic" +"test_class.ClassTests.testDel" = "__del__ not run deterministically under JVM GC" +"test_exceptions.ExceptionTests.test_3114" = "JVM finalization: object __del__ not run at refcount 0, sys.exception() sentinel never cleared" +"test_exceptions.ExceptionTests.testExceptionCleanupState" = "weakref/refcount: exception-local object not deterministically freed under JVM GC" +"test_weakref.FinalizeTestCase.test_all_freed" = "finalize/weakref timing: del+gc.collect does not free object under JVM GC" +"test_weakref.FinalizeTestCase.test_finalize" = "finalize timing: del does not run finalizer under JVM GC" +"test_weakref.FinalizeTestCase.test_order" = "finalizer ordering/timing at JVM GC differs from CPython refcount-0 ordering" +# dict mutation-during-iteration guard: GraalPy detects size change only, not same-size structural mutation (no version/keys tag). +"test_dict.DictTest.test_mutating_iteration_delete" = "same-size dict mutation during iteration needs version/keys-tag internals GraalPy lacks (size-change is detected)" +"test_dict.DictTest.test_mutating_iteration_delete_over_values" = "same-size dict mutation during iteration needs version/keys-tag internals GraalPy lacks" +"test_dict.DictTest.test_mutating_iteration_delete_over_items" = "same-size dict mutation during iteration needs version/keys-tag internals GraalPy lacks" +"test_dict.DictTest.test_oob_indexing_dictiter_iternextitem" = "mutation is driven by X.__del__ which never fires during iteration under GraalPy (no refcounting)" +# Buffer-protocol export/resize locking not modeled on the JVM (no BufferError on resize-while-exported). +"test_builtin.BuiltinTest.test_bytearray_join_with_misbehaving_iterator" = "Requires buffer-protocol export/resize locking (BufferError on resize-while-exported), not implemented" +"test_re.ReTests.test_keep_buffer" = "JVM does not track buffer-protocol exports; resizing a bytearray held by a live regex iterator does not raise BufferError" +# Native / locale / immutable-builtin / hash-impl details. +"test_builtin.BuiltinTest.test_hash" = "hash(str)==hash(bytes) is a CPython siphash implementation detail; GraalPy hashes str and bytes differently" +"test_tuple.TupleTest.test_hash_exact" = "tuple hash algorithm differs from CPython's exact hash values" +"test_re.ReTests.test_locale_caching" = "Locale-dependent re.L matching needs native LC_CTYPE support; setlocale is a no-op under Elide" +"test_re.ReTests.test_locale_compiled" = "Locale-dependent re.L compiled patterns need native LC_CTYPE support; setlocale is a no-op under Elide" +"test_ast.test_ast.AST_Tests.test_AST_fields_NULL_check" = "ast.AST is an immutable builtin type on the JVM; del/set ast.AST._fields raises TypeError instead of succeeding" +"test_descr.ClassPropertiesAndMethods.test_cycle_through_dict" = "gc.get_objects not present in Elide's GraalPy fork" +"test_patma.TestSourceLocations.test_jump_threading" = "dis.get_instructions raises NotImplementedError: dis module is not supported on GraalPy" +"test_deque.TestSubclass.test_weakref" = "weakref.proxy of a builtin collections.deque raises TypeError (subclasses are referenceable)" +"test_functools.TestCmpToKeyC.test_cmp_to_signature" = "C cmp_to_key signature is '(mycmp, /)' (positional-only marker) vs CPython docstring-derived '(mycmp)'" +"test_json.test_scanstring.TestCScanstring.test_surrogates" = "GraalPy UTF-16 strings combine adjacent lone surrogates into one astral code point; CPython keeps them separate" +"test_genexps" = "Doctest relies on CPython tuple-object identity reuse across genexp iterations (max-min tupleids==0); GraalPy allocates fresh tuples" +# Attribute-error message wording (behavior correct, phrasing differs; low value to chase). +"test_class.ClassTests.testObjectAttributeAccessErrorMessages" = "AttributeError message wording differs (del/setattr/read-only phrasing)" +"test_class.ClassTests.testTypeAttributeAccessErrorMessages" = "AttributeError says \"'A' object\" instead of \"type object 'A'\"" + [fail] diff --git a/expectations/javac-jtreg.toml b/expectations/javac-jtreg.toml index 2ea7f63..e5bd3ed 100644 --- a/expectations/javac-jtreg.toml +++ b/expectations/javac-jtreg.toml @@ -10,4 +10,58 @@ # 150/192 errors in the 1.3.6 run live here. See WHIPLASH #1174. "tools/javac/processing/**" = "annotation processing unsupported by Elide embedded javac (jtreg exit code 3); tracked in WHIPLASH #1174" +# The following tests live outside tools/javac/processing/ but share the same +# root cause (WHIPLASH #1174): each compiles an annotation-processor class +# during the test and passes it via -processor/-processorpath. Native-image +# javac cannot define user bytecode at runtime, so javac reports +# "Annotation processor 'X' not found". This includes six negative tests +# whose .out golden files expect processor-driven diagnostics. +"tools/javac/7129225/TestImportStar.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/MethodParameters/ClassReaderTest/ClassReaderTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T6411379.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T6423583.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T6458749.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T6855236.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T8151191/ErrorRunningJavadocOnInnerClasses.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T8170667/ParameterProcessor.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T8171332/Processor.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T8181464/LambdaInAnnotationsCausesNPETest1.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T8181464/LambdaInAnnotationsCausesNPETest2.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/T8181464/LambdaInAnnotationsCausesNPETest3.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/TypeToString.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/annotations/neg/8171322/TypeVariableAsAnnotationTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/annotations/repeatingAnnotations/generatedInRepeating/GeneratedInRepeating.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/annotations/typeAnnotations/failures/TypeVariableMissingTA.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/annotations/typeAnnotations/packageanno/PackageProcessor.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/annotations/typeAnnotations/position/TypeAnnotationPositionTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/api/TestGetTree.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/api/TestOperators.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/cast/intersection/model/Model01.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/classreader/8215407/BrokenEnclosingClass.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/defaultMethods/private/PrivateInterfaceMethodProcessorTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/doctree/ReferenceTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/doctree/positions/TestPosition.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/enum/6350057/T6350057.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/enum/6424358/T6424358.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/modules/T8160454/NPEGetDirectivesTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/modules/T8161501/UnnamedModuleUnnamedPackageTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/multicatch/model/ModelChecker.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/patterns/Annotations.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/patterns/PatternMatchPosTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/platform/NonPublicAnnotations.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/platform/ReleaseModulesAndTypeElement.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/tree/ArrayTypeToString.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/tree/MakeTypeTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/tree/NoPrivateTypesExported.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/tree/TreePosRoundsTest.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/treeannotests/AnnoTreeTests.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +"tools/javac/util/NewlineOnlyDiagnostic.java" = "runtime annotation-processor loading unsupported in native image (loads a -processor class compiled during the test); same root cause as tools/javac/processing/**; tracked in WHIPLASH #1174" +# Custom PlatformProvider via ToolProvider SPI: same runtime class-discovery +# limit as above ("release version name not supported"). +"tools/javac/platform/PlatformProviderTest.java" = "custom --release PlatformProvider SPI requires runtime service discovery, unsupported in native image; see WHIPLASH #1174" +# Elide is a native binary, not a JVM launcher: the jtreg wrapper strips -J* +# flags, so -J-Dline.separator cannot take effect and the single-line usage +# assertion fails. Environmental, not a compiler defect. +"tools/javac/newlines/NewLineTest.java" = "-J JVM flags cannot apply to a native binary; jtreg wrapper strips them by design" + [fail] diff --git a/harness/src/eshost-elide/runtime.elide.js b/harness/src/eshost-elide/runtime.elide.js index 5ef39e6..b462239 100644 --- a/harness/src/eshost-elide/runtime.elide.js +++ b/harness/src/eshost-elide/runtime.elide.js @@ -14,13 +14,14 @@ var elideNative262 = globalThis["\x24262"]; var $262 = { global: globalThis, destroy: function () {}, + // Per test262 INTERPRETING.md, $262.evalScript must evaluate `code` as a + // script and return its completion value, propagating any error. The prior + // implementation caught the error and returned a `{type, value}` record, so + // `assert.throws(..., () => $262.evalScript(...))` saw no exception — breaking + // the language/global-code script-declaration tests (SyntaxError/TypeError on + // redeclaration over restricted/non-configurable/non-extensible globals). evalScript: function (code) { - try { - elideNative262.evalScript(code); - return { type: "normal", value: undefined }; - } catch (e) { - return { type: "throw", value: e }; - } + return elideNative262.evalScript(code); }, getGlobal: function (name) { return this.global[name]; diff --git a/harness/src/report/index.test.ts b/harness/src/report/index.test.ts index 433b112..916fa5b 100644 --- a/harness/src/report/index.test.ts +++ b/harness/src/report/index.test.ts @@ -98,7 +98,7 @@ test("latestRunSummariesFromIndex excludes skipped tests from the pass-rate deno expect(latest[0]!.passRate).toBeCloseTo(70 / 90, 5); }); -test("top-level summaries omit muted workloads (node-api, wpt-wintertc)", () => { +test("top-level summaries include all workloads (mute set is empty)", () => { const mk = (workload: string, reportDir: string) => ({ workload, semver: "1.3.6", digest: "3d3ea83ed640", pass: 1, total: 10, skip: 0, regressions: 0, @@ -107,6 +107,5 @@ test("top-level summaries omit muted workloads (node-api, wpt-wintertc)", () => const latest = latestRunSummariesFromIndex({ runs: [mk("test262", "a"), mk("node-api", "b"), mk("wpt-wintertc", "c"), mk("javac-jtreg", "d")], }); - // node-api and wpt-wintertc are hidden from the rolled-up summaries. - expect(latest.map((r) => r.workload)).toEqual(["javac-jtreg", "test262"]); + expect(latest.map((r) => r.workload)).toEqual(["javac-jtreg", "node-api", "test262", "wpt-wintertc"]); }); diff --git a/harness/src/report/index.ts b/harness/src/report/index.ts index 7565e36..676ed59 100644 --- a/harness/src/report/index.ts +++ b/harness/src/report/index.ts @@ -77,12 +77,11 @@ export async function buildIndexJson(reportsDir: string): Promise<{ runs: IndexE } // Workloads hidden from the rolled-up, top-level summaries (README compat table, -// reports/index.md, reports/pass-rate.svg) for now. Their per-run reports and the -// raw reports/index.json entries are unaffected — this only suppresses them from -// the headline views while their coverage is still being shaped: -// - node-api: broad slice still dominated by unimplemented core modules. -// - wpt-wintertc: fetch surface enabled but failing, not yet representative. -export const SUMMARY_MUTED_WORKLOADS = new Set(["node-api", "wpt-wintertc"]); +// reports/index.md, reports/pass-rate.svg). Their per-run reports and the raw +// reports/index.json entries are unaffected — this only suppresses them from +// the headline views while their coverage is still being shaped. Currently empty: +// all suites (including node-api and wpt-wintertc) report in the headline views. +export const SUMMARY_MUTED_WORKLOADS = new Set([]); export function latestRunSummariesFromIndex(index: { runs: IndexEntry[] }): RunSummary[] { const latest = new Map(); diff --git a/registry.toml b/registry.toml index 713bda0..707419d 100644 --- a/registry.toml +++ b/registry.toml @@ -31,8 +31,11 @@ path = "suites/openjdk" [workload.settings] manifest = "manifests/javac-langtools.toml" timeoutMs = 3600000 - # jtreg's default action timeout is 120s; this maps to -timeoutFactor:2.5. - jtregCaseTimeoutSeconds = 300 + # jtreg's default action timeout is 120s; this maps to -timeoutFactor:7.5. + # 300s starved slow-but-passing tests (combo.ComboTestHelper suites, file/zip) + # under full-suite concurrency: they pass in isolation but hit the wall when + # thousands of concurrent per-compile launches contend for cores. + jtregCaseTimeoutSeconds = 900 jtregPath = "jtreg" # Optional explicit real JDK home for the jtreg wrapper layout. # If omitted, the adapter derives java.home from javaRunner and only falls back to JAVA_HOME for compatibility. diff --git a/reports/index.json b/reports/index.json index 51d0c9b..be1fe1b 100644 --- a/reports/index.json +++ b/reports/index.json @@ -1,5 +1,16 @@ { "runs": [ + { + "workload": "javac-jtreg", + "semver": "1.4.1+20260717.acbafa6", + "digest": "3d3ea83ed640", + "pass": 2, + "total": 2, + "skip": 0, + "regressions": 0, + "finishedAt": "2026-07-18T00:48:06.442Z", + "reportDir": "1.4.1+20260717.acbafa6/3d3ea83ed640/javac-jtreg" + }, { "workload": "test262", "semver": "1.4.1+20260716.b8b6531", diff --git a/reports/index.md b/reports/index.md index d9bb0fb..bb3d59b 100644 --- a/reports/index.md +++ b/reports/index.md @@ -5,5 +5,7 @@ | Suite | Version | Digest | Pass rate | Status | |---|---|---|---:|:--:| | cpython-core | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 95.9% | ✅ | -| javac-jtreg | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 98.7% | ✅ | +| javac-jtreg | `1.4.1+20260717.acbafa6` | `3d3ea83ed640` | 100.0% | ✅ | +| node-api | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 25.4% | ✅ | | test262 | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 99.8% | ✅ | +| wpt-wintertc | `1.4.1+20260716.b8b6531` | `3d3ea83ed640` | 75.3% | ✅ | diff --git a/reports/pass-rate.svg b/reports/pass-rate.svg index 57a791a..25d2517 100644 --- a/reports/pass-rate.svg +++ b/reports/pass-rate.svg @@ -1,7 +1,7 @@ - + Elide compatibility pass rates Latest pass rate by compliance suite. - + Latest compatibility cpython-core @@ -9,10 +9,18 @@ 95.9% javac-jtreg - - 98.7% - test262 + + 100.0% + node-api - - 99.8% + + 25.4% + test262 + + + 99.8% + wpt-wintertc + + + 75.3%