Skip to content

Pull test_prv_006/007 out of the default smoke gate (unresolved concurrent-load failures) - #100

Merged
saqibmanan merged 1 commit into
CIfrom
pull-006-007-from-default-smoke
Sep 18, 2026
Merged

saqibmanan merged 1 commit into
CIfrom
pull-006-007-from-default-smoke

Conversation

@saqibmanan

Copy link
Copy Markdown
Contributor

Why

Both tests fail reproducibly under concurrent (`-n 3`) provider-smoke CI runs — confirmed on the post-#99-merge run on `CI`, 3/3 attempts including auto-reruns:

  • `test_prv_006_org_create_dataset`: `TimeoutException: Timed out waiting for Metadata tab to appear after creating dataset` — a wait already bumped 30s→60s for this exact reason (2026-09-15), still failing.
  • `test_prv_007_org_create_usecase`: `AssertionError: Summary mismatch. Expected: '...', Found: ''`

The org-collision theory that correctly explained (and fixed) `test_prv_011`'s flakiness in #99 does not hold here — `TEST_EMAIL_3` was moved off the org it shared with `TEST_EMAIL_2` specifically to test this, and both tests still failed identically on the next real run. Root cause is still open.

Given they're not safe to trust as a merge gate right now, pulling them out of the default `-m smoke` selection rather than leaving PR/push CI red (or silently green via more reruns masking a real, unexplained, consistent failure).

What changed

Removed `@pytest.mark.smoke` from both, kept `@pytest.mark.functional`. They still run on `workflow_dispatch` (`-m "smoke or functional"`) — not lost from coverage, just out of the default gate until root-caused.

Verification

Collection diff, confirmed directly:

  • `-m smoke`: 9/18 provider/smoke tests collected (was 11/18) — 006 and 007 gone, 006b (unaffected, different test) still present.
  • `-m "smoke or functional"`: 18/18 — both still collected and runnable on dispatch.

Both fail reproducibly under concurrent (-n 3) provider-smoke CI runs -
confirmed on the post-PR#99-merge CI run, 3/3 attempts including
auto-reruns, same failures each time (test_prv_006: TimeoutException on
a wait already bumped 30s->60s for this same reason; test_prv_007:
AssertionError, Summary mismatch, Found: '').

The org-collision theory that explained test_prv_011's flakiness does
not hold here: TEST_EMAIL_3 was moved off the shared org specifically
to test this, and both tests still failed identically afterward. Root
cause is still open - not safe to keep gating default PR/push smoke
runs on two tests that fail this consistently under load.

Removed @pytest.mark.smoke, kept @pytest.mark.functional, so both still
run on workflow_dispatch (-m "smoke or functional") and aren't lost
from coverage entirely - just out of the default gate until root-caused.
@saqibmanan
saqibmanan marked this pull request as ready for review September 18, 2026 06:43
@saqibmanan
saqibmanan merged commit 00a9f8a into CI Sep 18, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant