E2E: utility - cleanup stores

[phyllis-sy-wu]

WHY are these changes introduced?

E2E tests create dev stores that can accumulate when tests fail mid-run, CI times out, or teardown fails. This script automates bulk-clean for leftover stores.

WHAT is this pull request doing?

cleanup-stores.ts

Standalone cleanup script that finds leftover E2E dev stores, uninstalls their apps, and deletes them via browser automation.

pnpm --filter e2e exec tsx scripts/cleanup-stores.ts              # Full: uninstall apps + delete stores
pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --list        # List stores with app counts
pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --delete      # Delete only stores with 0 apps installed
pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --headed      # Show browser window
pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --pattern X   # Match stores containing "X" (default: "e2e-w")

Logic

Discovery phase:

1. Log in via completeLogin helper
2. Navigate to dev.shopify.com/dashboard/{orgId}/stores
3. Recover from transient 500/502 via refreshIfPageError
4. Wait for #stores-tbody tr to render, then scroll-to-bottom in a loop until row count stabilizes (lazy-loaded table)
5. Extract store FQDNs via regex on full page HTML — catches slugs in hrefs/attributes, not just visible text
6. Filter by name pattern (default: e2e-w), deduplicate via Set

--list mode (per store):

1. Navigate to admin.shopify.com/store/{slug}/settings/apps
2. Wait for page readiness via Promise.race — either empty state or first app menu button
3. Count menu buttons across all pages (paginate via button#nextURL)
4. Print store name + app count

Default mode (per store):

1. Navigate to admin.shopify.com/store/{slug}/settings/apps
2. Dismiss Dev Console if visible
3. Wait for page readiness via Promise.race — either empty state or first app menu button
4. Check for "Add apps to your store" empty state — only trust positive confirmation as proof of zero apps
5. If apps present: uninstall all apps (see uninstall logic below)
6. After uninstall: verify empty state is now visible — if not, skip store deletion and log warning
7. If safe to delete: call deleteStore() (from setup/store.ts) with up to 3 retries

--delete mode: same as default but skips stores that have apps installed (step 4 → skip if not empty).

Uninstall logic (uninstallAllAppsFromStore):

1. Check isStoreAppsEmpty — if true, done (primary termination)
2. Find button[aria-label="More actions"] at position consecutiveSkips
3. Extract app name from div[role="listitem"] → a span
4. Click menu button → click "Uninstall" → click confirm
5. If "Uninstall" option not in menu: press Escape, increment consecutiveSkips, try next button
6. If confirm never appears: increment consecutiveSkips (prevents infinite loop)
7. Reload page after each uninstall to refresh the list
8. When no more menu buttons visible: check for button#nextURL pagination → continue on next page

Features:

• Empty state safety: never deletes a store unless "Add apps to your store" is positively confirmed
• Page readiness wait via Promise.race before checking empty state (avoids false negatives from slow renders)
• Scroll-based store discovery handles lazy-loaded tables (stabilizes after 3 idle rounds)
• Per-store timing in output
• Exports cleanupStores() for use from other scripts

How is this different from per-test teardown?

• Per-test teardown (setup/teardown.ts) — knows the specific app name and store FQDN, uses direct URLs, no discovery. Runs automatically in test finally blocks.
• cleanup-stores.ts (bulk, manual) — discovers all matching stores via scroll-based lazy loading. Safety net for orphaned stores from failed/interrupted test runs.

How to test your changes?

1. Create leftover stores by skipping cleanup:

   E2E_SKIP_TEARDOWN=1 DEBUG=1 pnpm --filter e2e exec playwright test app

2. List them:

   pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --list

3. Clean up:

   pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --headed

Example: pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --headed

cleanup-stores.mov

Expand for complete log

cli % pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --headed

[cleanup-stores] Mode:    Uninstall apps + Delete stores
[cleanup-stores] Org:     161686155
[cleanup-stores] Pattern: "e2e-w"

[cleanup-stores] Logging in...
[cleanup-stores] Logged in successfully.
[cleanup-stores] Navigating to stores page...
[cleanup-stores]   ...loaded 20 stores
[cleanup-stores]   ...loaded 30 stores
[cleanup-stores]   ...loaded 40 stores
[cleanup-stores]   ...loaded 43 stores
[cleanup-stores] Found 41 store(s) matching pattern "e2e-w"

[cleanup-stores] [1/41] e2e-w8-1776939053026
  No apps installed (empty state confirmed)
  Deleting store...
  Deleted
  (29.2s)

[cleanup-stores] [2/41] e2e-w5-1776939053547
  No apps installed (empty state confirmed)
  Deleting store...
  Deleted
  (30.5s)

...

Post-release steps

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

[phyllis-sy-wu]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• E2E - QA: init, deploy, versions list, config link, 2nd deploy #7367
• E2E: utility - cleanup all #7357
• E2E: utility - cleanup apps #7356
• E2E: utility - cleanup stores #7355 👈 (View in Graphite)
• E2E: automatic teardown polish #7358
• E2E: improve CLI log format #7364
• E2E: remove dead code #7360
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Copilot]

Pull request overview

Adds a standalone E2E utility script to bulk-discover and clean up leftover dev stores created by failing/aborted E2E runs, using Playwright browser automation against the Dev Dashboard + Store Admin.

Changes:

• Introduces packages/e2e/scripts/cleanup-stores.ts with --list, --delete, and default “full” cleanup modes.
• Implements store discovery via Dev Dashboard pagination and regex extraction of store FQDNs from page HTML.
• Automates app uninstall (with pagination) and store deletion flows with retries and safety checks (empty-state confirmation).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

isDirectRun compares process.argv[1] to fileURLToPath(import.meta.url) using strict string equality. When running via npx tsx packages/e2e/scripts/cleanup-stores.ts, process.argv[1] is typically a relative path, so this check can be false and main() won’t run at all. Consider normalizing both sides (e.g., path.resolve(process.argv[1])) or comparing URLs via pathToFileURL(path.resolve(process.argv[1])).href === import.meta.url.

[Copilot]

This script defines a local deleteStore() helper that largely overlaps with the existing deleteStore browser helper in packages/e2e/setup/store.ts. Having two different implementations with the same name risks divergence and makes future maintenance/debugging harder. Consider reusing the shared helper (and extending it if needed) or renaming this one to avoid confusion (e.g., deleteStoreViaPlanPage).