E2E: utility - cleanup apps

[phyllis-sy-wu]

WHY are these changes introduced?

E2E tests create apps that can accumulate when tests fail mid-run, CI times out, or teardown fails. This script automates bulk-clean for leftover apps.

WHAT is this pull request doing?

cleanup-apps.ts

Standalone cleanup script that finds leftover E2E test apps on the Dev Dashboard, uninstalls them from all stores, and deletes them.

pnpm --filter e2e exec tsx scripts/cleanup-apps.ts              # Full cleanup: uninstall + delete
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --list        # List matching apps with install counts
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --uninstall   # Uninstall from all stores only (no delete)
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --delete      # Delete only apps with 0 installs
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed      # Show browser window
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --pattern X   # Match apps containing "X" (default: "E2E-")

Logic

The per-app mechanics delegate to the shared setup/ building blocks (uninstallAppFromStore, deleteAppFromDevDashboard, refreshIfPageError) — same primitives used by per-test teardown. The script adds bulk discovery, pagination, and per-app retry on top.

Discovery phase:

1. Log in via completeLogin helper
2. Navigate to Dev Dashboard, then loop refreshIfPageError up to 3× as extra resilience on 500/502 (hard-fails after 3 attempts)
3. Find app cards via a[href*=\"/apps/\"] selectors
4. Extract app name from card text (split on install count pattern), install count via regex, and URL from href
5. Filter by name pattern (default: E2E-)
6. Paginate via a[href*=\"next_cursor\"]; refreshIfPageError runs at the top of every iteration so error-page returns throw instead of silently yielding 0 apps

Uninstall (per app):

1. Navigate to {appUrl}/installs
2. Collect store slugs via FQDN regex on full page HTML (page.content())
3. If no FQDNs found on this page: fall back to table row text (store name = slug)
4. Paginate via button#nextURL and repeat 2–3 on each page
5. For each store slug: call uninstallAppFromStore(page, slug, appName) — the shared setup helper navigates to the store's /settings/apps, clicks the ⋯ menu → Uninstall → confirm, then reloads and verifies the app is gone. Returns true if gone (or already absent), false if still listed.
6. If any store returns false or throws: mark allUninstalled = false and log the store slug + error
7. Final verification: navigate back to {appUrl}/installs, scan every row across all pages — any non-empty row → return false
8. Return allUninstalled

Delete (per app):

1. Call deleteAppFromDevDashboard(page, appUrl) — the shared setup helper navigates to {appUrl}/settings, clicks Delete app (scroll + reload-once fallback for the button's propagation lag), types "DELETE" if the confirm input is present, clicks confirm, then reloads and returns true on 404 (deleted) or false otherwise. Throws STILL_HAS_INSTALLS if the Delete button stays disabled after reload (fail-fast signal — retries won't help).
2. Treat false as "deletion could not be verified" and retry via the outer loop.

Per-app retry wrapper:

1. Each app gets up to 3 attempts
2. On failure: log the error ((N/3) failed: ...), wait, re-navigate to dashboard, retry the full uninstall + delete flow
3. STILL_HAS_INSTALLS short-circuits the retry loop → record as skipped
4. Per-app elapsed time printed ((Xs)); summary printed at end: X succeeded, Y skipped, Z failed (Xs total)

--list mode: runs discovery only, prints app names and install counts.

--uninstall mode: runs uninstall only, skips apps with 0 installs.

--delete mode: runs delete only, skips apps with installs > 0.

Features:

• Reuses setup/ building blocks — single source of truth for per-store uninstall and per-app delete semantics (shared with per-test teardown)
• Dashboard error handling via shared refreshIfPageError helper — hard-fails after 3 consecutive 500/502 responses
• Store slug extraction via full HTML regex with per-page table text fallback
• Paginated final installs verification across all pages
• Error page differentiation: 404 = already deleted (success), 500/502 = server error (retry), disabled Delete button = STILL_HAS_INSTALLS (fail-fast skip)
• Exports cleanupAllApps() for use as a Playwright globalTeardown or from other scripts

How is this different from per-test teardown?

• Per-test teardown (setup/teardown.ts) — knows the specific app name and store FQDN, uses direct URLs, no discovery. Runs automatically in test `finally` blocks.
• `cleanup-apps.ts` (bulk, manual) — discovers all matching apps via dashboard pagination, discovers stores via installs page. Safety net for orphaned apps from failed/interrupted test runs.
• Both share the same underlying primitives in `setup/` (`uninstallAppFromStore`, `deleteAppFromDevDashboard`, `refreshIfPageError`), so browser automation semantics stay consistent.

How to test your changes?

1. Create leftover apps by skipping cleanup:

   E2E_SKIP_TEARDOWN=1 DEBUG=1 pnpm --filter e2e exec playwright test app

2. List them:

   pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --list

3. Clean up:

   pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed

Example

pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed

cleanup-apps.mov

Expand for complete log

cli % pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed

[cleanup-apps] Mode:    Uninstall + Delete
[cleanup-apps] Org:     161686155
[cleanup-apps] Pattern: "E2E-"

[cleanup-apps] Logging in...
[cleanup-apps] Logged in successfully.
[cleanup-apps] Navigating to dashboard...
[cleanup-apps] Dashboard loaded.
[cleanup-apps] Finding matching apps...
[cleanup-apps]   ...loaded 20 apps
[cleanup-apps]   ...loaded 40 apps
[cleanup-apps]   ...loaded 60 apps
[cleanup-apps]   ...loaded 73 apps
[cleanup-apps] Found 41 app(s) matching pattern "E2E-"

  1. E2E-dev-basic-1776959380611 (1 install)
  2. E2E-dev-basic-1776957041976 (0 installs)
  3. E2E-dev-basic-1776956913189 (0 installs)
  4. E2E-hot-create-1776956797812 (1 install)
  5. E2E-hot-reload-1776956796321 (1 install)
  6. E2E-dev-1776956791662 (1 install)
  7. E2E-multi-cfg-1776956790856 (1 install)
  8. E2E-dev-basic-1776955811113 (0 installs)
  9. E2E-hot-create-1776947144539 (1 install)
  10. E2E-dev-1776947140553 (1 install)
  11. E2E-hot-delete-1776947140856 (1 install)
  12. E2E-multi-cfg-1776947136959 (1 install)
  13. E2E-hot-reload-1776947140231 (1 install)
  14. E2E-multi-cfg-1776947138343 (0 installs)
  15. E2E-dev-1776947134834 (0 installs)
  16. E2E-hot-create-1776947136636 (0 installs)
  17. E2E-toml-dev-1776947144639 (1 install)
  18. E2E-multi-cfg-1776946913357 (0 installs)
  19. E2E-hot-create-1776943244407 (1 install)
  20. E2E-hot-reload-1776943242934 (1 install)
  21. E2E-multi-cfg-1776943243151 (0 installs)
  22. E2E-dev-1776942633118 (0 installs)
  23. E2E-multi-cfg-1776942630633 (0 installs)
  24. E2E-hot-create-1776942600305 (0 installs)
  25. E2E-hot-reload-1776942598644 (0 installs)
  26. E2E-dev-1776942598380 (1 install)
  27. E2E-hot-reload-1776942554353 (0 installs)
  28. E2E-dev-1776942552276 (1 install)
  29. E2E-hot-delete-1776942549780 (1 install)
  30. E2E-toml-deploy-1776942540447 (0 installs)
  31. E2E-hot-create-1776942343056 (0 installs)
  32. E2E-hot-reload-1776942340258 (1 install)
  33. E2E-multi-cfg-1776942333644 (1 install)
  34. E2E-scaffold-1776942269970 (0 installs)
  35. E2E-hot-reload-1776942269254 (0 installs)
  36. E2E-hot-delete-1776942264353 (1 install)
  37. E2E-dev-1776942284449 (0 installs)
  38. E2E-deploy-1776942269971 (0 installs)
  39. E2E-toml-dev-1776942268415 (0 installs)
  40. E2E-toml-deploy-1776942247534 (0 installs)
  41. E2E-ext-only-1776942269968 (0 installs)

[cleanup-apps] [1/41] E2E-dev-basic-1776959380611
  Uninstalling...
  Uninstalled
  Deleting...
  Deleted
  (15.0s)

[cleanup-apps] [2/41] E2E-dev-basic-1776957041976
  Not installed
  Deleting...
  Deleted
  (11.4s)

[cleanup-apps] [3/41] E2E-dev-basic-1776956913189
  Not installed
  Deleting...
  Deleted
  (11.3s)

...

Post-release steps

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing — I've identified the correct bump type (`patch` for bug fixes · `minor` for new features · `major` for breaking changes) and added a changeset with `pnpm changeset add`

[phyllis-sy-wu]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• E2E - QA: init, deploy, versions list, config link, 2nd deploy #7367
• E2E: utility - cleanup all #7357
• E2E: utility - cleanup apps #7356 👈 (View in Graphite)
• E2E: utility - cleanup stores #7355
• E2E: automatic teardown polish #7358
• E2E: improve CLI log format #7364
• E2E: remove dead code #7360
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.