test: bring more of consumer/smoke and provider/smoke under the default smoke run - #98
Merged
Merged
Conversation
test_con_003/005/007 and test_publishers_flow were functional-only, so they never ran under the default -m smoke path despite sitting in tests/consumer/smoke/ alongside tests that do. Additive -- functional stays, so the full/functional suite selections are unaffected. Closes out consumer/smoke entirely: all 61 tests in the directory now run under -m smoke. Verified: 61 passed, -n 4, 4m18s against dev.
Adds smoke (kept functional) to the org-scoped create/edit flows -- 006, 006b, 007, 009, 011, 012 -- plus a new dashboard_components.py (4 read-only visibility checks, already smoke-only, no writes). Deliberately NOT included: the individual-scoped duplicates (002, 002b, 003), the two xfail chart tests (004, 008 -- feature not fully built), and 005 (edit profile) and 010 (individual collaborative) -- selection is user's, not derived from a blanket rule. Verified against dev, -n 3, the 8 files now selected under -m smoke: 9 passed, 2 failed (test_prv_006, test_prv_007). Both confirmed load-contention, not real bugs -- each passes cleanly alone (099s and 145s respectively). Same class of failure already documented on the consumer-account-parallelism PR: concurrent provider testing against the shared dev backend is measurably flakier than serial, independent of which specific files are running. Not fixed here -- a CI-config or backend-capacity question, not a marker one.
saqibmanan
marked this pull request as ready for review
September 17, 2026 19:45
This was referenced Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Option 1 from our discussion: fix the marker mismatch rather than fold everything into the default gate. Directory placement and the
smokemarker had drifted apart significantly — this closes that gap for a specific, chosen set, not blanket-wide.consumer/smoke/— all of it, nowtest_con_003/005/007andtest_publishers_flow(download-verification tests) werefunctional-only, so they sat intests/consumer/smoke/but never ran under the default-m smokepath. Addedsmokealongside the existingfunctional— additive, sofull/functionalsuite selections are unaffected.Verified: 61/61 passed,
-n 4, 4m18s against dev — every test in the directory now runs under-m smoke.provider/smoke/— 8 specific files, chosen deliberatelyAdded
smoke(keptfunctional) to:test_prv_002_dashboard_components(new, 4 read-only visibility checks, no writes),006,006b,007,009,011,012— the org-scoped create/edit flows, plus the pre-existing001login test.Deliberately not included: the individual-scoped duplicates (
002,002b,003), the twoxfailchart tests (004,008— feature not fully built),005(edit profile — separately flagged, hardcodes a name assuming one specific account) and010(individual collaborative). This is a chosen subset, not derived from a rule.Verified against dev,
-n 3, the 8 files now selected: 9 passed, 2 failed (test_prv_006,test_prv_007). Both confirmed load-contention, not real bugs — each passes cleanly alone (99s and 145s respectively). This is the same class of flakiness already documented on the account-parallelism PR (#97): concurrent provider testing against the shared dev backend is measurably less reliable than serial, independent of which specific files are running that round — a different failing pair each time. Not something this PR fixes; a CI-concurrency or backend-capacity question, not a marker one. Flagging it here rather than silently shipping around it.No workflow YAML change
run-smoke.yml's case-statement already scans the whole domain tree by marker (p="tests/consumer"/p="tests/provider",m="smoke") for every suite selection — thesmoke/folder name has never actually scoped anything on its own. So adding the marker to these specific tests is sufficient by itself; nothing else to wire up.