Bump tight waits on test_prv_011's collaborative flow (concurrent-load flakiness) - #99
Merged
Merged
Conversation
Under concurrent (-n 3) provider-smoke load, test_prv_011_org_create_collaborative failed reproducibly (2/2 runs) with TimeoutException, always at this same heavy flow, while lighter tests in the same run passed. Diagnosed before fixing: backend GraphQL latency stayed flat under load (213ms avg vs 229ms idle baseline, p99 only ~1s), and local CPU never saturated (68% avg idle, load avg peaked 6.18/8 cores) - ruling out both dev backend capacity and raw CPU contention. A third concurrent run then got OOM-killed by the OS on this dev machine (which also runs normal desktop Chrome/VSCode alongside the tests) - the real constraint is local RAM headroom, not the platform. Bumped the 5 wait_with_timeout(10) calls that sit on this specific test's call path (collaborative name edit, SDG goals toggle/option, publish tab, and the org-dashboard Collaboratives nav link) to 15s, matching the class's own default timeout used elsewhere in these files. Left every other provider/smoke test's waits untouched - this is the one flow demonstrated to be timing-sensitive under load. Verified: solo run passes (3:33). Concurrent re-verification skipped this round - machine was already under real memory pressure (428MB unused, 2.1GB in the compressor, swap in use) from the prior OOM-killed run, and forcing another 3-Chrome run risked repeating it.
TEST_EMAIL_2 and TEST_EMAIL_3 were both provisioned with admin/canAdd on 'my test agency'. OrganizationsPage.select_org() hardcodes a preference for that specific org (falling back to 'first available card' only when it is not clickable for the account), so both accounts landed on the same org row whenever their org-write tests ran concurrently - real, reproducible contention that showed up as test_prv_006/007 failing on PR #99's own CI run (gw2/TEST_EMAIL_3) while test_prv_006b/009 on gw1/TEST_EMAIL_2 passed, both in the same job. TEST_EMAIL_3 has now been moved to its own org (test org name) in Keycloak, verified live via the userPermissions query - no longer holds any role on 'my test agency' at all, so select_org()'s hardcoded preference can no longer put both accounts on the same org. No code change required; correcting the docstring's org-role snapshot and its stale claim that org selection is fully dynamic.
saqibmanan
marked this pull request as ready for review
September 18, 2026 06:03
This was referenced Sep 18, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
test_prv_011_org_create_collaborativewas failing reproducibly under concurrent (-n 3) provider-smoke runs — 2/2 attempts on this branch — while passing every time run alone. Diagnosed before touching anything (per the earlier discussion on the load-issue thread):top/load average throughout the same run: CPU idle averaged 68% (range 18-85%), load average peaked at 6.18 on an 8-core machine.So this isn't a dev-backend capacity problem, and it isn't a logic bug in the test — it's that this one flow (the longest, most form-heavy provider test: multi-tab wizard, file uploads, several dropdowns) has less timing slack than the rest of the suite, and a handful of its waits were tighter (10s) than the page-object class's own default (15s) used everywhere else in the same file.
What changed
Bumped exactly the 5
wait_with_timeout(10)calls that sit ontest_prv_011's actual call path to15(matchingBasePage's existing default), and nowhere else:pages/provider/create_collaborative_page.py: collaborative-name-edit input, SDG-goals toggle, SDG-goals option, Publish-tab clickpages/provider/organizations_page.py:click_collaboratives_card's nav-link waitEvery other provider/smoke test's waits are untouched — this is the one flow demonstrated to be timing-sensitive under concurrency.
Verification
test_prv_011_org_create_collaborativepasses cleanly (3:33).-n 3) re-verification was skipped this round: right after the OOM-killed run above, this machine was still under real memory pressure (428MB unused, 2.1GB in the memory compressor, swap in use) — forcing another 3-Chrome run risked repeating the OOM kill rather than proving anything. Relying on the solo pass plus the pre-fix evidence (this exact test failed 2/2 concurrent runs at this exact flow, with backend/CPU both ruled out) to justify the fix; flagging this gap explicitly rather than claiming untested concurrent coverage.No workflow/CI changes — this only touches page-object wait timeouts.