Conformance triage batch 1: summaries, eshost fix, skip classifications#2
Merged
Conversation
Empty SUMMARY_MUTED_WORKLOADS so all five suites appear in the README compat table, reports/index.md, and pass-rate.svg. Regenerated from the latest committed reports (1.4.1+20260716.b8b6531 / 3d3ea83ed640): node-api lands at 25.4%, wpt-wintertc at 75.3%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shim wrapped the native call in try/catch and returned a {type, value}
record, so assert.throws(SyntaxError, () => $262.evalScript(...)) saw no
exception — failing 15 test262 executions (language/global-code
script-declaration tests) that Elide's native $262 handles correctly.
Delegate directly per test262 INTERPRETING.md. All 31 evalScript-using
tests: 29 pass/15 fail -> 44 pass/0 fail against the same pinned binary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
file/zip/T6836682 and combo-based LambdaParserTest pass in isolation but hit the 300s per-case wall under full-suite -concurrency:36, where thousands of concurrent per-compile native-binary launches starve them. The 1h whole-run wall is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
javac-jtreg: 42 skips — 40 tests outside tools/javac/processing/ that compile an annotation processor during the test and load it via -processor, hitting the same native-image runtime-classloading limit already classified for tools/javac/processing/** (WHIPLASH #1174); plus the custom PlatformProvider SPI test and the -J-flag NewLineTest. cpython-core: 37 skips — subprocess-dependent tests (no child-process support), JVM GC/refcount finalization timing, buffer-protocol export locking, native locale internals, and hash-algorithm implementation details. Genuine defects (34, all GraalPy-fork-resident) remain tracked as failures. Every entry carries a one-line justification. Diagnosis details in the triage reports; classification verified against retained .jtr files and per-test reruns on the pinned nightly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-contained handoff docs (current numbers, run commands, failure clusters, ranked work items) for test262, wpt-wintertc, cpython-core, javac-jtreg, and node-api, generated from the 1.4.1+20260716.b8b6531 reports and used to drive the batch-1 triage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
Compliance CI analysis: cpython-core RED is nightly drift, not this PRPer-suite outcomes on the
The cpython run's own The 2 genuine pass→fail flips against the previous nightly — Resolution: re-ratchet 🤖 Generated with Claude Code |
The 20260718.acbafa6 nightly expanded the runnable cpython universe from 1,865 to 5,334 cases (previously import-crashing modules now run), so the frozen baseline flagged 249 newly-visible failures as regressions on CI. Regenerated from that CI run's own results (linux-amd64) via the harness ratchet functions: 283 known-fail entries. Only 2 entries are genuine pass->fail flips between nightlies (test_list_resize_overflow, test_sumprod_stress) — tracked for upstream investigation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testsuite-side half of conformance batch 1 (runtime half: elide-dev/WHIPLASH#1347). Driven by a full five-suite triage of the
1.4.1+20260716.b8b6531reports; per-suite briefings included.node-api(25.4%) andwpt-wintertc(75.3%) are no longer muted from the README table,reports/index.md, andpass-rate.svg.$262.evalScriptswallowed exceptions and returned a{type, value}record, failing 15 test262 executions (language/global-codescript-declaration tests) that the runtime actually handles correctly. All 31 evalScript-using tests: 29/15 → 44/0 against the same pinned binary. Runharness/scripts/install-elide-host.shafter pulling.file/zip/T6836682, combo-basedLambdaParserTest) pass in isolation but starve under full-suite concurrency..jtrdiagnostics), 37 cpython (subprocess, JVM GC/refcount finalization timing, buffer-export locking, locale, hash internals — verified via per-test reruns). Genuine defects stay tracked as failures.Notes
eshost-elideunit-test failures reproduce on cleanmain(environment-dependent) and are untouched by this diff.🤖 Generated with Claude Code