fix(test-optimization): defer Vitest EFD faultiness detection - #9700
fix(test-optimization): defer Vitest EFD faultiness detection#9700juan-fernandez wants to merge 4 commits into
Conversation
Overall package sizeSelf size: 7.87 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 00df06e | Docs | Datadog PR Page | Give us feedback! |
Bits found no code fix to apply🟢 Investigated · ⚪ No code fix needed
View in Datadog | Reviewed commit dd2487b · Any feedback? Reach out in #deveng-pr-agent |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #9700 +/- ##
==========================================
- Coverage 98.52% 98.51% -0.01%
==========================================
Files 957 963 +6
Lines 135131 136089 +958
Branches 11681 11934 +253
==========================================
+ Hits 133132 134072 +940
- Misses 1999 2017 +18
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@codex review |
BenchmarksBenchmark execution time: 2026-08-05 21:23:08 Comparing candidate commit 00df06e in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2320 metrics, 38 unstable metrics.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b710f8c77f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
![datadog-official[bot]](https://avatars.githubusercontent.com/in/1358509?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
Runtime suite admission preserves the legacy absolute-and-percentage faultiness boundary while counting only runnable new tests, so empty selected Vitest suites no longer suppress EFD. The worker, no-worker, and Browser Mode paths consistently retain admitted retries and mark later suites as faulty after the configured threshold.
📊 Validated against 4 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit dd2487b · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dd2487be7b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00df06e454
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -1,5 +1,7 @@ | |||
| import { afterEach, beforeAll, beforeEach, inject } from 'vitest' | |||
|
|
|||
| import { requestEfdSuiteAdmission } from './vitest-efd-suite-admission.mjs' | |||
There was a problem hiding this comment.
In Browser Mode the setup file is served by Vite from outside the user's project, and addVitestBrowserSetupFileAccess() only adds vitest-no-worker-init-setup.mjs itself to server.fs.allow. This new static sibling import makes the browser request ci/vitest-efd-suite-admission.mjs too; because that file is not added to the allow list, Vite rejects the module load before any tests run, so Browser Mode Test Optimization setup breaks even when EFD admission is not used.
Useful? React with 👍 / 👎.
1 participant
What does this PR do?
Changes Vitest 4 Early Flake Detection faultiness evaluation from a pre-run comparison of selected suites against known tests to runtime suite admission.
For Vitest 4, the tracer uses the existing worker transports to count unique suites that actually collect at least one runnable new test. EFD retries continue until that count exceeds the larger of the configured absolute and percentage-derived suite limits. Suites admitted before the limit is crossed finish their retries, while the crossing suite and subsequent suites do not retry. New tests continue to report
test.is_newafter EFD is marked faulty.This applies to standard Vitest workers, no-worker init, and Browser Mode. Vitest 1–3 retain the existing pre-run faultiness check.
Motivation
Vitest and Storybook can select suite specifications that collect no tests. Those suites never appear in the known-tests response, so the previous pre-run suite comparison treated them as new and could disable EFD before any tests ran.
Basing the numerator on suites containing runnable new tests avoids false faulty-session detection without backend changes. Limiting the new protocol to Vitest 4 keeps the implementation small and covers the affected Vitest 4.1.10 session.
Additional Notes
Validation:
git diff --check