feat(ci): promote mutation testing and the web e2e suite to required checks - #1210
Merged
Conversation
vite.config.ts declares environment: "jsdom" inside the "unit" project
entry, but vitest.mutation.config.ts replaces the whole test block for
Stryker's vitest-runner, so the key never survived the replacement and
the DOM-rendering tests executed in plain node under mutation runs --
the first one ("document is not defined") failed Stryker's initial dry
run, which is why the web package has never produced a mutation score.
Restating it on the replacement block keeps the mutation run identical
to the plain unit suite: dry run verified green locally (47 tests).
… test Stryker's initial dry run runs the whole unit suite through mutant instrumentation, which multiplies per-call cost by around two orders of magnitude: math-stretch.test.ts's whole-Unicode-range glyphId enumeration measures ~28s instrumented on a fast local machine (well under 200ms plain) and exceeded its own 90s per-test ceiling on a GitHub mutation runner, failing the dry run before a single mutant was tested. Raise the test's ceiling to 300s (headroom above the worst instrumented case, not a match for it) and give pdf-codec's stryker config a 15-minute dry-run budget through a new packageStrykerConfig option, since the enumeration test plus the rest of the instrumented suite leaves the default 5-minute budget too little room on a loaded runner.
Stryker runs were already reporting real per-package scores from main-branch shard runs without ever gating on them. Derive each package's thresholds.break from its first measured baseline through a new packageStrykerConfig option: floor the measured score and subtract that package's own Timeout-classified share of valid mutants (rounded up, minimum one point), because Timeout is the one mutant classification that legitimately flaps between runs -- runner load alone decides whether the same mutant times out or survives -- so the margin absorbs every timeout re-classifying while anything beyond it is a real regression. A package whose mutation run has never completed (documents.js, pdf-codec, web) passes no break yet and stays ungated until one does; the derivation rule and its rationale live once on PackageStrykerOptions.breakThreshold.
…eck shape The e2e suite has run unattended on every pull request and main push since it landed, and its only observed red run tracked a regression on that pull request's own branch and cleared when the branch was fixed, so it meets the bar its job comment set for gating merges: requiredness is recorded in the comment and README, and test-e2e joins deploy-site's needs so a main-branch e2e failure blocks deploying the site the suite drives a browser through. Mutation testing gains a mutation-result aggregation job: the shard matrix's width varies with the affected package set, so no individual shard context can be required by name without stranding a pull request that produced fewer shards -- the always-running aggregate is the one stable context for the ruleset, once every package completes a green run behind its new thresholds.
Mearman
force-pushed
the
chore/promote-required-checks
branch
from
September 11, 2026 03:20
796f172 to
d391a61
Compare
Mearman
marked this pull request as ready for review
September 11, 2026 03:52
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Contributor
|
🎉 This PR is included in version 4.9.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1194.
The promotion path the issue named, measured rather than assumed:
fix(pdf-codec): budget the mutation dry run for the instrumented STIX test -- the live failure PR feat(ooxml.js): associate a docx Caption paragraph with the figure it describes #1201 hit: Stryker's initial dry run timed out at 90s on the heavy STIX Two Math font test under instrumentation. The mutation timeout is now budgeted for the instrumented reality of that suite, following Stryker's own guidance that the budget should be a multiple of the slowest test.feat(ci): gate each measured package on its first CI mutation baseline -- Stryker shard runs on main were already reporting real per-package scores without ever gating on them. Each package'sthresholds.breakis derived from its first measured baseline (floored score minus that package's own Timeout-classified share, rounded up, minimum one point -- Timeout is the one classification that legitimately flaps with runner load, so the margin absorbs every timeout re-classifying while anything beyond it is a real regression). Packages whose mutation runs have never completed stay ungated until one does; the derivation rule lives once onPackageStrykerOptions.breakThreshold.