Pull test_prv_006/007 out of the default smoke gate (unresolved concurrent-load failures) - #100
Merged
Merged
Conversation
Both fail reproducibly under concurrent (-n 3) provider-smoke CI runs - confirmed on the post-PR#99-merge CI run, 3/3 attempts including auto-reruns, same failures each time (test_prv_006: TimeoutException on a wait already bumped 30s->60s for this same reason; test_prv_007: AssertionError, Summary mismatch, Found: ''). The org-collision theory that explained test_prv_011's flakiness does not hold here: TEST_EMAIL_3 was moved off the shared org specifically to test this, and both tests still failed identically afterward. Root cause is still open - not safe to keep gating default PR/push smoke runs on two tests that fail this consistently under load. Removed @pytest.mark.smoke, kept @pytest.mark.functional, so both still run on workflow_dispatch (-m "smoke or functional") and aren't lost from coverage entirely - just out of the default gate until root-caused.
saqibmanan
marked this pull request as ready for review
September 18, 2026 06:43
This was referenced Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Both tests fail reproducibly under concurrent (`-n 3`) provider-smoke CI runs — confirmed on the post-#99-merge run on `CI`, 3/3 attempts including auto-reruns:
The org-collision theory that correctly explained (and fixed) `test_prv_011`'s flakiness in #99 does not hold here — `TEST_EMAIL_3` was moved off the org it shared with `TEST_EMAIL_2` specifically to test this, and both tests still failed identically on the next real run. Root cause is still open.
Given they're not safe to trust as a merge gate right now, pulling them out of the default `-m smoke` selection rather than leaving PR/push CI red (or silently green via more reruns masking a real, unexplained, consistent failure).
What changed
Removed `@pytest.mark.smoke` from both, kept `@pytest.mark.functional`. They still run on `workflow_dispatch` (`-m "smoke or functional"`) — not lost from coverage, just out of the default gate until root-caused.
Verification
Collection diff, confirmed directly: