feat: add --max-suite-retries option to cap total reruns across suite#332
Open
Borda wants to merge 9 commits into
Open
feat: add --max-suite-retries option to cap total reruns across suite#332Borda wants to merge 9 commits into
Borda wants to merge 9 commits into
Conversation
- New CLI option `--max-suite-retries` (int, default None = no limit); once the suite-wide rerun count hits the cap, failing tests are logged as final failures without further retry - `StatusDB` gains thread-safe `increment_suite_reruns()` / `get_suite_reruns()` via `threading.Lock` for the single-process path - `ServerStatusDB` overrides use the existing `rerunfailures_db` dict with a `"__suite__"` key and an atomic socket `inc` operation in `run_connection` - `ClientStatusDB` overrides route increment through the socket to the master - Five new tests covering: cap enforcement, pass-through when under cap, zero disables all reruns, passing tests don't consume the budget, and standalone option no-op - CHANGES.rst and README.rst updated Fixes pytest-dev#298 --- Co-authored-by: Claude Code <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a new CLI option to cap total reruns across an entire pytest session, preventing excessive reruns in large/flaky suites.
Changes:
- Introduce
--max-suite-retriesoption and enforce it during rerun scheduling. - Add test coverage for suite-wide rerun caps (including 0 and “no effect” scenarios).
- Document the new option and note it in the changelog (Fixes #298).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/pytest_rerunfailures.py |
Adds the new CLI option and suite-wide rerun counting (including xdist socket DB support). |
tests/test_pytest_rerunfailures.py |
Adds functional tests validating suite-wide rerun cap behavior. |
README.rst |
Documents how to use --max-suite-retries and its intent. |
CHANGES.rst |
Records the new feature in the unreleased changelog. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
[resolve pytest-dev#3] Review by @Copilot (PR pytest-dev#332): "This increments the suite-wide counter even when the cap is already exhausted, which can permanently overshoot the configured limit..." Challenge: evidence=VALID suggestion=VALID resolution=as-suggested - Add `try_increment_suite_reruns(max_cap)` to StatusDB, ServerStatusDB, ClientStatusDB - ServerStatusDB adds `try_inc` socket protocol (atomic check-then-increment) - Caller uses bool return instead of post-increment comparison - Counter now reflects reruns actually performed, not attempts --- Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: OpenAI Codex <codex@openai.com>
[resolve pytest-dev#1] Review by @Copilot (PR pytest-dev#332): "group._addoption is a private pytest API and can break across pytest versions." Challenge: evidence=VALID suggestion=VALID resolution=as-suggested --- Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: OpenAI Codex <codex@openai.com>
--- Co-authored-by: Claude Code <noreply@anthropic.com>
[resolve pytest-dev#5] Review by @Copilot (PR pytest-dev#332): "--max-suite-retries currently accepts negative integers (because `type=int`)..." Challenge: evidence=VALID suggestion=VALID resolution=as-suggested --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>
[resolve pytest-dev#6] Review by @Copilot (PR pytest-dev#332): "get_suite_reruns() reads _suite_rerun_count without the lock..." Challenge: evidence=VALID suggestion=VALID resolution=as-suggested --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>
[resolve pytest-dev#7] Review by @Copilot (PR pytest-dev#332): "This test manually inspects parseoutcomes() while the surrounding..." Challenge: evidence=VALID suggestion=REJECT resolution=self-resolved --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>
--- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Comment on lines
144
to
+151
| if config.option.reruns != 0: | ||
| if config.option.usepdb: # a core option | ||
| raise pytest.UsageError("--reruns incompatible with --pdb") | ||
| if ( | ||
| config.option.max_suite_retries is not None | ||
| and config.option.max_suite_retries < 0 | ||
| ): | ||
| raise pytest.UsageError("--max-suite-retries must be >= 0") |
Comment on lines
+124
to
+132
| group.addoption( | ||
| "--max-suite-retries", | ||
| action="store", | ||
| dest="max_suite_retries", | ||
| type=int, | ||
| default=None, | ||
| help="Maximum total number of reruns across the entire test suite. " | ||
| "Once this limit is reached, no further reruns will occur.", | ||
| ) |
Comment on lines
+726
to
+731
| max_suite_reruns = item.session.config.option.max_suite_retries | ||
| if max_suite_reruns is not None: | ||
| if not db.try_increment_suite_reruns(max_suite_reruns): | ||
| # suite-wide limit exhausted — log as final failure | ||
| item.ihook.pytest_runtest_logreport(report=report) | ||
| continue |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new feature to cap the total number of test reruns across the entire suite using the
--max-suite-retriesoption. This allows users to limit overall reruns, regardless of per-test retry settings, to better control resource usage in large or flaky test suites. The implementation includes command-line interface changes, core logic updates, and comprehensive tests.New Feature: Suite-wide rerun cap
--max-suite-retriesoption to the CLI, allowing users to specify a maximum total number of reruns across the entire test suite. Once this cap is reached, no further reruns are performed, regardless of individual test retry settings. [1] [2] [3]Implementation: Core logic and state management
StatusDBand related classes to track and increment a suite-wide rerun counter in a thread-safe manner, supporting both in-memory and socket-based backends. [1] [2] [3] [4]Testing: Validation of new behavior
--max-suite-retries(int, default None = no limit); once the suite-wide rerun count hits the cap, failing tests are logged as final failures without further retryStatusDBgains thread-safeincrement_suite_reruns()/get_suite_reruns()viathreading.Lockfor the single-process pathServerStatusDBoverrides use the existingrerunfailures_dbdict with a"__suite__"key and an atomic socketincoperation inrun_connectionClientStatusDBoverrides route increment through the socket to the masterFixes #298