feat(runner): report a test that only passed on retry as flaky - #1040
Merged
Conversation
--retry hid flakiness instead of surfacing it. A test that failed and then passed was recorded as a plain pass; the only trace was a console suffix, absent from the counters, from JUnit, TAP, JSON and HTML, and from the GitHub Actions annotations. A CI job that retried a flaky test looked exactly like a clean run. Flaky is now a facet of passed, not a seventh outcome: the test is still counted in the pass total, so the exit code is unchanged unless --fail-on-flaky is given, and the flaky count is deliberately left out of the total to keep it from exceeding the tests actually run. The first attempt's failure message is the whole diagnostic value and was being overwritten by the next attempt, so it is captured before the retry loop discards it and carried into every report. The retry count lives in the runner shell, not in the test subshell that builds the payload, so it is appended to the payload after the loop. That is what lets it cross the fork: the parallel aggregator re-derives every status from the .result file and would otherwise see a plain pass. The field is read and defaulted on its own there, because a payload without it would leave the strip a no-op and hand the numeric guard arbitrary text, marking every test failed. Closes #1012
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤔 Background
Related #1012
--retryhid flakiness instead of surfacing it. A test that failed and then passed was recorded as a plain pass, with the only trace a console suffix that never reached the counters or any report. A CI job that retried a flaky test was indistinguishable from a clean run, so flakiness never got triaged.💡 Changes
Flaky: Nin the summary.--fail-on-flakyturns such a run red, mirroring--fail-on-risky.<flakyFailure>), TAP, JSON, HTML and the GitHub Actions annotations along with the retry count.--parallel: the retry count crosses the fork in the per-test payload, which the aggregator reads and defaults on its own.