From 51dd974878da6a87347b4d3f18e9c31678fbd0ad Mon Sep 17 00:00:00 2001 From: Saqib Date: Thu, 17 Sep 2026 19:30:10 +0530 Subject: [PATCH 1/2] Add a 3rd test account slot TEST_EMAIL_3/PASSWORD_3, matching the existing User 1/User 2 pattern. --- .env.example | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.env.example b/.env.example index 7e19bde..44a6c4f 100644 --- a/.env.example +++ b/.env.example @@ -20,6 +20,10 @@ TEST_PASSWORD_1=your-password-here TEST_EMAIL_2=test-user-2@example.com TEST_PASSWORD_2=your-password-here +# User 3 +TEST_EMAIL_3=test-user-3@example.com +TEST_PASSWORD_3=your-password-here + # Test Configuration TEST_USER_INDEX=1 From 014fc575518aab22ff38c2db6f3ee678b287ed61 Mon Sep 17 00:00:00 2001 From: Saqib Date: Thu, 17 Sep 2026 19:30:10 +0530 Subject: [PATCH 2/2] ci: 3-way test-account parallelism + consumer-smoke gets -n 4 Two independent changes, landing together since both touch the same worker/credential system. ## consumer-smoke: -n 4 Was fully serial (~5-6 min of the suite's runtime). Nothing in tests/consumer/ needs a real login (no test_credentials/auth_token usage anywhere in it), so unlike api-smoke/provider-smoke this was never bounded by account count -- just never given -n at all. Verified locally: -n 4, 55 passed in ~2 min, down from serial. ## 3rd test account (test_1@civicdatalab.in / TEST_EMAIL_3) Account already created and verified live (real Keycloak token, HTTP 200) before this PR -- wired in, not provisioned here. - workflow_call secrets schema: TEST_EMAIL_3/PASSWORD_3, optional, matching TEST_EMAIL_2's existing declaration. - api-smoke: added TEST_EMAIL_3/PASSWORD_3, and separately TEST_EMAIL_2/ PASSWORD_2 -- found while doing this that api-smoke has been running -n 2 without ever forwarding TEST_EMAIL_2 at all. gw1 has been silently falling back to TEST_EMAIL_1 this whole time (test_credentials does that for any worker slot with no matching credentials, rather than failing). Fixed alongside adding _3, not filed separately -- same category of gap, cheap to fix while already in this exact code. - provider-smoke: added TEST_EMAIL_3/PASSWORD_3 alongside the existing _1/_2 forwarding. - Both bumped -n 2 -> -n 3. ## org_add_permission docstring corrected, not just extended It was already stale before this PR: claimed TEST_EMAIL_2 was 'a lone auditor on CivicDataLab' with canAdd=false everywhere. User changed TEST_EMAIL_2's role to admin on a different org ('my test agency') partway through this work. Verified live via the exact GraphQL query the fixture itself uses, for all three accounts: - TEST_EMAIL_1: admin/canAdd on 11 orgs (CivicDataLab and 10 others) - TEST_EMAIL_2: admin/canAdd on 'my test agency' only; still auditor/canAdd=false on CivicDataLab - TEST_EMAIL_3: admin/canAdd on 'my test agency', same as TEST_EMAIL_2 -- already write-capable from creation No test code changes needed -- org_add_permission already resolves this dynamically from the live permissions query, nothing hardcodes an org name. Rewrote the docstring to state the verified facts and warn that it will go stale again the next time a role changes. ## What this does NOT change The org-create flows this unblocks (test_prv_006 through 011) are functional-marked, not smoke -- confirmed via collection, only test_prv_001_login_smoke carries the smoke marker in tests/provider/ smoke/. So this doesn't add real-data-creation load to the default push/PR path; those flows already only ran on workflow_dispatch/ workflow_call, unchanged by this PR. ## Verified - actionlint: zero findings. - consumer-smoke -n 4: 55 passed, ~2 min (real run against dev). - api-smoke -n 3, all three TEST_EMAIL_* forwarded: 26 passed, 7 skipped (all pre-existing/expected for a local run), no new failures. - provider-smoke test_prv_001_login_smoke -n 3: 1 passed (only file at this marker, so only gw0 got scheduled work -- but all three accounts independently confirmed able to obtain a real Keycloak token before this PR, which is the actual risk surface for the credential change). --- .github/workflows/run-smoke.yml | 37 +++++++++++++++++++++++++++++---- conftest.py | 25 +++++++++++++++++++--- 2 files changed, 55 insertions(+), 7 deletions(-) diff --git a/.github/workflows/run-smoke.yml b/.github/workflows/run-smoke.yml index e158e4d..80eb1e3 100644 --- a/.github/workflows/run-smoke.yml +++ b/.github/workflows/run-smoke.yml @@ -61,6 +61,10 @@ on: required: false TEST_PASSWORD_2: required: false + TEST_EMAIL_3: + required: false + TEST_PASSWORD_3: + required: false # Required in practice by api-smoke, declared optional so the existing # callers (DataSpaceBackend, DataSpaceFrontend) keep parsing until they # pass it. api-smoke preflights it and fails with a readable message @@ -165,7 +169,13 @@ jobs: - name: Run consumer smoke tests run: | - pytest "${{ steps.markers.outputs.path }}" -m "${{ steps.markers.outputs.value }}" -v --json-report + # -n 4: unlike api-smoke/provider-smoke, nothing here needs a real + # login (no test_credentials/auth_token usage in tests/consumer/), + # so this isn't bounded by account count the way those two are. + # Picked against file count (7 files under tests/consumer/smoke/ + # today, --dist loadfile keeps each on one worker) and typical + # GH-hosted runner core counts -- not account availability. + pytest "${{ steps.markers.outputs.path }}" -m "${{ steps.markers.outputs.value }}" -v --json-report -n 4 - name: Upload screenshots if: always() @@ -212,6 +222,15 @@ jobs: HOME_URL_DEV: ${{ secrets.HOME_URL_DEV }} TEST_EMAIL_1: ${{ secrets.TEST_EMAIL_1 }} TEST_PASSWORD_1: ${{ secrets.TEST_PASSWORD_1 }} + # TEST_EMAIL_2 was never forwarded here despite this job already running + # -n 2 -- gw1 has been silently falling back to TEST_EMAIL_1 this whole + # time (conftest.py's test_credentials does that when a worker's slot has + # no matching credentials, rather than failing). Adding _2 fixes that + # existing gap; adding _3 is for the new worker below. + TEST_EMAIL_2: ${{ secrets.TEST_EMAIL_2 }} + TEST_PASSWORD_2: ${{ secrets.TEST_PASSWORD_2 }} + TEST_EMAIL_3: ${{ secrets.TEST_EMAIL_3 }} + TEST_PASSWORD_3: ${{ secrets.TEST_PASSWORD_3 }} # Falls back to the dev backend so push/PR runs actually exercise the # API tests. Without a default, `inputs` is empty on a push event, every # api-smoke test skipped, and the job reported success having asserted @@ -349,7 +368,11 @@ jobs: - name: Run API smoke tests run: | - pytest "${{ steps.markers.outputs.path }}" -m "${{ steps.markers.outputs.value }}" -v --json-report -n 2 + # -n 3: matches the 3 configured test accounts (test_credentials maps + # gw0/gw1/gw2 -> TEST_EMAIL_1/2/3). Don't raise this without adding a + # 4th account first -- an extra worker with no matching credentials + # falls back to TEST_EMAIL_1 silently, not an error. + pytest "${{ steps.markers.outputs.path }}" -m "${{ steps.markers.outputs.value }}" -v --json-report -n 3 - name: Enforce minimum passed count # Now that API_BASE_URL always has a value, this runs on every event, @@ -421,9 +444,11 @@ jobs: TEST_PASSWORD_1: ${{ secrets.TEST_PASSWORD_1 }} TEST_EMAIL_2: ${{ secrets.TEST_EMAIL_2 }} TEST_PASSWORD_2: ${{ secrets.TEST_PASSWORD_2 }} + TEST_EMAIL_3: ${{ secrets.TEST_EMAIL_3 }} + TEST_PASSWORD_3: ${{ secrets.TEST_PASSWORD_3 }} # org_add_permission (conftest) resolves which orgs this worker's account # can create content in. Without these it skips every org-create test on - # both workers -- including the account that has canAdd. + # any of the three workers -- including accounts that have canAdd. API_BASE_URL: ${{ inputs.api_base_url || vars.API_BASE_URL || 'https://dev.api.civicdataspace.in' }} KEYCLOAK_URL: ${{ vars.KEYCLOAK_URL || 'https://auth.civicdatalab.in' }} KEYCLOAK_REALM: ${{ vars.KEYCLOAK_REALM || 'DataSpace' }} @@ -514,7 +539,11 @@ jobs: - name: Run provider smoke tests run: | - pytest "${{ steps.markers.outputs.path }}" -m "${{ steps.markers.outputs.value }}" -v --json-report -n 2 + # -n 3: matches the 3 configured test accounts, same reasoning as + # api-smoke above. --dist loadfile (pytest.ini) keeps the numbered + # provider flows in file order per worker, so each of the 3 accounts + # runs its own files start to finish rather than interleaving. + pytest "${{ steps.markers.outputs.path }}" -m "${{ steps.markers.outputs.value }}" -v --json-report -n 3 - name: Upload screenshots if: always() diff --git a/conftest.py b/conftest.py index a3183d9..9ef2710 100644 --- a/conftest.py +++ b/conftest.py @@ -246,14 +246,33 @@ def org_add_permission(test_credentials): The org-scoped provider flows (create dataset / prompt dataset / usecase / collaborative under an org) all need `canAdd` on some organization. A Keycloak account whose org role is `auditor` has canAdd=false and cannot pass - those flows no matter how the UI is driven — TEST_EMAIL_2 is exactly that: a - lone `auditor` on CivicDataLab, which is why every org-create test failed on - gw1 while the read-only ones passed. + those flows no matter how the UI is driven. That is an account-provisioning gap, not a product defect and not a test bug, so the flows skip with a precise reason instead of failing. Grant the account an `admin` (or otherwise canAdd) role on an org and they run again with no code change. + + Current roles (verified live against dev 2026-09-17 via this exact query — + re-check here before trusting this comment, don't just read it; it already + went stale once when TEST_EMAIL_2's role changed and nothing here was + updated to match): + + - TEST_EMAIL_1 — admin/canAdd=true on 11 orgs (CivicDataLab, Open Budgets + India, JusticeHub, "my test agency", "test org name", ASDMA, HPSDMA, The + Rockefeller Foundation, Patrick J. McGovern Foundation, BMA, Gates + Foundation). Broadest account by far. + - TEST_EMAIL_2 — admin/canAdd=true on "my test agency" only; still + auditor/canAdd=false on CivicDataLab. Org-create flows for gw1 now run + (they used to skip), but land in "my test agency" specifically. + - TEST_EMAIL_3 — admin/canAdd=true on "my test agency" only, same as + TEST_EMAIL_2. Added 2026-09-17 for a 3rd xdist worker; already + write-capable from creation, nothing further to grant. + + None of this suite's code hardcodes an org name — `writable` is whatever + the live permissions query returns for the account, and provider tests + pick from it dynamically. Don't add an assertion that assumes a specific + worker always lands on a specific org; these roles will change again. """ api = os.getenv("API_BASE_URL") kc, realm = os.getenv("KEYCLOAK_URL"), os.getenv("KEYCLOAK_REALM")