test262 full in ci - #121
Open
toshok wants to merge 6 commits into
Open
Conversation
The full sharded suite now runs on every push to main and every PR (alongside the nightly + workflow_dispatch), on Linux runners rather than macOS: JS semantics don't vary by platform, so one Linux arch is enough, and Linux bills at 1x — the shards+2 fan-out stays cheap. The curated lane stays the macOS gate in bootstrap.yml. The build and shard jobs move to ubuntu-24.04 with the Linux toolchain (apt.llvm.org llvm-22, build-essential, libuv1-dev, libunwind-dev); assemble-workroot.sh forwards --config llvm.prefix=/usr/lib/llvm-22 to buck2. Concurrency cancels superseded PR runs. Report-only: the shards run without --expectations, so the check surfaces the pass-rate summary on every PR/commit but does not block merges. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…of it
The runner selected language/ + built-ins/ + harness/ and ran every
selected test, so tests no ahead-of-time engine can pass — they need a
compiler at run time, or a host hook with no AOT meaning — each cost a
compile+link to reach a foregone failure and then dragged down the
pass rate. They are now classified `skip-unsupported` before
compiling, with a `needs` tag on the row: tagged cross-realm /
ShadowRealm / dynamic-import, living under an eval-code tree, calling
eval(...) or Function(...) in the body, reaching $262.agent, or
including a harness file that does. That last set is derived from the
suite rather than listed, so it tracks SHA bumps — it is what catches
fnGlobalObject.js (`Function("return this;")()`) and the 437 tests
that include it. Unimplemented features stay in the denominator:
Temporal and Atomics are missing work, not impossible work.
annexB/ was excluded by omission rather than decision — the selection
policy named intl402/ and staging/ and never mentioned it. Annex B is
normative for web-facing engines, so annexB/language now rides the
language stride and annexB/built-ins the built-ins cap. Of 48.7k
tests, 3.3k are out of scope and 45.5k evaluate.
report grows the headline it was missing (pass over evaluated, plus
the skip breakdown), tests failing-set membership through a Set rather
than an array scan that goes quadratic at full-suite volume, and
learns --baseline: the full run's ratchet, {evaluated, pass,
tolerance}. Per-test expectations are the lane's contract and do not
scale to 45k rows, so the full run holds two numbers instead —
coverage must not shrink, conformance must not go backwards.
expectations.txt regenerated on macos-arm64 for the new selection:
1544 -> 1318 entries (110 annexB failures in, 336 now-skipped out),
lane pass 77.4%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The collect job concatenated whatever shard results happened to download. With fail-fast disabled, one shard timing out or losing a race with apt.llvm.org produced a plausible pass rate over a fraction of the suite, and the only red mark was the shard job itself — easy to miss on a report-only workflow. It now asserts every shard reported before reporting anything, and checks the result against full-baseline.json, so a regression or a shrunk denominator marks the run red. The job also needs build (for the shard count) and no longer runs when the shards never ran, where it used to die on a glob. Setup was repeated on the build job and every shard: an ~1.5GB apt.llvm.org install, a dozen chances per run for an external mirror to fail, plus a test262 fetch each. The new setup-llvm composite action restores the toolchain from the actions cache and falls back to apt when the entry is cold or does not run; the build job populates the cache and now fetches the suite once, shipping it with the workroot (~25MB packed). Sizing from a measured 0.61 CPU-s per test: 45.5k evaluated tests over 12 shards is ~3.8k each, ~20-35 min on a 4-vCPU runner, so the shard timeout goes to 120 min for headroom. Twelve shards plus the bootstrap matrix stays under the 20-job concurrency cap. Artifacts stop being re-zipped (both are already gzipped) and expire in days rather than the 90-day default, and doc-only pushes no longer spend a matrix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eir was squash-merged as "whole new world" (#116) and main has moved on through v0.2.0 and P8; the branch has not been touched since. The push trigger outlived it, and test262-full.yml already only watches main, so the two workflows now agree on what a mainline push is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sharded suite was its own workflow, so every push ran two independent workflows that each built the compiler — the bootstrap matrix built it three times over, and test262-full built it a fourth just to have something to test. It is now a branch in bootstrap.yml at the point where conformance runs: Linux x86_64 uploads the stage1 workroot it already built (plus the suite checkout, fetched once) and the test262-shard matrix extracts that archive, so no shard compiles anything. Everywhere else runs the curated lane in-job, as before. ci.yml is the only root now. It gains the workflow_dispatch input for regenerating full-baseline.json, which bootstrap.yml takes as a workflow_call input; the nightly cron is gone, since the suite runs on every push and PR. release.yml calls the same matrix, so a conformance regression against a committed baseline holds a release — worth knowing before committing the first baseline, which is what arms the check at all. The shard count moves to SHARDS at the top of bootstrap.yml, next to the matrix list it has to agree with; disagreement is caught by the collect job's completeness check or by --shard rejecting an out-of-range slice. The Linux toolchain install becomes the setup-llvm action the shards already used, so it is defined once and the build seeds the cache the shards restore. Comments in the files touched here drop their phase ids (release-P1/P2/P3, runtime-P3, language-P1/P3/P4): they named the phase that added a step, which says nothing about what the step does now. The pointer that read "release-P3 points both at hosted release assets" names release.yml instead, and release.yml's summary of the bootstrap call picks up the test262 run it now includes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bootstrap.yml owned the platform list, and its two Linux arches were legs of one matrix job. That put the list in a file callers can't see and made "the Linux build" a single node in the graph: anything downstream of x86_64 waited on arm64 too, and the arches needed fail-fast: false to keep one from cancelling the other. It is now the build for ONE platform, parameterized by runner plus the handful of things that actually vary — a buck-config string, and booleans for the typecheck and the two test262 modes — with brew-vs-apt and the homebrew smoke keyed off runner.os, which is a real OS difference rather than a policy choice. ci.yml and release.yml each call it three times, so every platform is its own job under its own name, and separate jobs never cancel each other. The sharded suite becomes a reusable workflow of its own rather than jobs wired into the build: no triggers, called after the platform jobs by both callers, downloading the workroot artifact from the run it is part of. It waits on all three platforms — a red matrix is not worth spending a dozen runners to measure conformance on — while release's smoke jobs now wait only on the platform whose tarball they install, which the old single bootstrap node couldn't express. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.