Skip to content

Commit bdd0a48

Browse files
BillLeoutsakosvl346Bill Leoutsakos
andauthored
Add reusable Playwright E2E foundation (#5792)
* test(e2e): add reusable Playwright foundation Establish a hermetic full-stack browser harness so settings and future surfaces can exercise real authentication, APIs, and isolated database state safely. * test(e2e): harden runner lifecycle and artifacts Fail closed on stale processes, preserve cleanup during interruption, and keep credentials and personal fixture data out of uploaded diagnostics. * test(e2e): tighten isolation and shutdown guards Make destructive database operations and service bindings strictly local while ensuring interrupted runs fully stop managed children before cleanup. * test(e2e): finalize harness safety boundaries Close the remaining lifecycle, credential-isolation, and CLI escape hatches so future persona and settings suites can build on a fail-closed foundation. * test(e2e): harden interrupted-run cleanup Use a detached cleanup supervisor so repeated signals cannot interrupt process-group shutdown or forced removal of the guarded run database. * test(e2e): guard signal cleanup edge cases Prevent empty process-group targets and verify forced-drop retries against final database state so interrupted runs cannot self-signal or report false cleanup failures. * test(e2e): require exact host and race-safe signals Reject dual-stack host mappings that cannot reach IPv4-only services and tolerate child exit races during process-group signal fallback. * test(e2e): force IPv4 for hosted test origin Accept safe dual-stack loopback resolution while pinning Chromium and Node traffic to 127.0.0.1 so CI reaches IPv4-only services reliably. * test(e2e): settle readiness exit race Mark successful readiness before the managed process completion branch can reject, preventing later normal shutdown from surfacing an abandoned promise failure. * test(e2e): propagate IPv4 preference to probes Keep Node-based Playwright requests on the same IPv4 path as Chromium and use direct loopback for orchestrator probes under dual-stack CI DNS. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>
1 parent 954b395 commit bdd0a48

39 files changed

Lines changed: 2817 additions & 12 deletions

.github/workflows/test-build.yml

Lines changed: 94 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -245,4 +245,97 @@ jobs:
245245
AWS_REGION: 'us-west-2'
246246
ENCRYPTION_KEY: '7cf672e460e430c1fba707575c2b0e2ad5a99dddf9b7b7e3b5646e630861db1c' # dummy key for CI only
247247
TURBO_CACHE_DIR: .turbo
248-
run: bunx turbo run build --filter=sim
248+
run: bunx turbo run build --filter=sim
249+
250+
settings-e2e:
251+
name: Settings E2E (informational)
252+
runs-on: blacksmith-8vcpu-ubuntu-2404
253+
timeout-minutes: 45
254+
continue-on-error: true
255+
256+
services:
257+
postgres:
258+
image: pgvector/pgvector:pg17
259+
env:
260+
POSTGRES_USER: postgres
261+
POSTGRES_PASSWORD: postgres
262+
POSTGRES_DB: postgres
263+
ports:
264+
- 5432:5432
265+
options: >-
266+
--health-cmd "pg_isready -U postgres"
267+
--health-interval 5s
268+
--health-timeout 5s
269+
--health-retries 10
270+
271+
steps:
272+
- name: Checkout code
273+
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
274+
275+
- name: Setup Bun
276+
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
277+
with:
278+
bun-version: 1.3.13
279+
280+
- name: Setup Node
281+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
282+
with:
283+
node-version: 22
284+
285+
- name: Mount Bun cache (Sticky Disk)
286+
uses: useblacksmith/stickydisk@4c034ba57b706cf0e3b4b0ce098c2a3b1071580c # v1
287+
with:
288+
key: ${{ github.repository }}-bun-cache-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}
289+
path: ~/.bun/install/cache
290+
291+
- name: Mount node_modules (Sticky Disk)
292+
uses: useblacksmith/stickydisk@4c034ba57b706cf0e3b4b0ce098c2a3b1071580c # v1
293+
with:
294+
key: ${{ github.repository }}-node-modules-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}
295+
path: ./node_modules
296+
297+
- name: Mount Playwright browsers (Sticky Disk)
298+
uses: useblacksmith/stickydisk@4c034ba57b706cf0e3b4b0ce098c2a3b1071580c # v1
299+
with:
300+
key: ${{ github.repository }}-playwright-browsers-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}
301+
path: ~/.cache/ms-playwright
302+
303+
- name: Restore E2E Next.js build cache
304+
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
305+
with:
306+
path: ./apps/sim/.next/cache
307+
key: ${{ runner.os }}-nextjs-e2e-hosted-billing-${{ hashFiles('bun.lock') }}-${{ github.sha }}
308+
restore-keys: |
309+
${{ runner.os }}-nextjs-e2e-hosted-billing-${{ hashFiles('bun.lock') }}-
310+
311+
- name: Install dependencies
312+
run: bun install --frozen-lockfile
313+
314+
- name: Configure E2E hostname
315+
run: echo "127.0.0.1 e2e.sim.ai" | sudo tee -a /etc/hosts
316+
317+
- name: Install Chromium
318+
working-directory: apps/sim
319+
run: bun run test:e2e:install-browsers -- --with-deps
320+
321+
- name: Run settings E2E foundation
322+
timeout-minutes: 40
323+
working-directory: apps/sim
324+
env:
325+
CI: 'true'
326+
E2E_PG_ADMIN_URL: 'postgresql://postgres:postgres@127.0.0.1:5432/postgres'
327+
run: bun run test:e2e
328+
329+
- name: Upload E2E diagnostics
330+
if: failure() || cancelled()
331+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
332+
with:
333+
name: settings-e2e-${{ github.run_id }}
334+
path: |
335+
apps/sim/playwright-report/
336+
apps/sim/test-results/
337+
apps/sim/e2e/.runs/
338+
!apps/sim/e2e/.runs/**/auth/**
339+
if-no-files-found: ignore
340+
include-hidden-files: true
341+
retention-days: 7

.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@ package-lock.json
1818
# testing
1919
/coverage
2020
/apps/**/coverage
21+
/apps/**/playwright-report/
22+
/apps/**/test-results/
23+
/apps/**/e2e/.runs/
24+
/apps/**/e2e/.auth/
2125

2226
# next.js
2327
/.next/
@@ -31,6 +35,7 @@ package-lock.json
3135
**/dist/
3236
**/standalone/
3337
sim-standalone.tar.gz
38+
/.artifacts/
3439

3540
# redis
3641
dump.rdb

apps/realtime/src/env.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ const EnvSchema = z.object({
1414
INTERNAL_API_SECRET: z.string().min(32),
1515
NEXT_PUBLIC_APP_URL: z.string().url(),
1616
ALLOWED_ORIGINS: z.string().optional(),
17+
REALTIME_HOST: z.string().min(1).default('0.0.0.0'),
1718
PORT: z.coerce.number().int().positive().default(3002),
1819
SIM_DB_ROLE: z.enum(['web', 'trigger', 'realtime']).optional(),
1920
DISABLE_AUTH: z

apps/realtime/src/index.ts

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ async function createRoomManager(io: SocketIOServer): Promise<IRoomManager> {
3131
async function main() {
3232
const httpServer = createServer()
3333
const PORT = env.PORT
34+
const HOST = env.REALTIME_HOST
3435

3536
logger.info('Starting Socket.IO server...', {
3637
port: PORT,
@@ -96,9 +97,9 @@ async function main() {
9697

9798
await assertSchemaCompatibility()
9899

99-
httpServer.listen(PORT, '0.0.0.0', () => {
100-
logger.info(`Socket.IO server running on port ${PORT}`)
101-
logger.info(`Health check available at: http://localhost:${PORT}/health`)
100+
httpServer.listen(PORT, HOST, () => {
101+
logger.info(`Socket.IO server running on ${HOST}:${PORT}`)
102+
logger.info(`Health check available at: http://${HOST}:${PORT}/health`)
102103
})
103104

104105
const shutdown = async () => {

apps/realtime/src/routes/http.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ export function createHttpHandler(roomManager: IRoomManager, logger: Logger) {
6969
status: 'ok',
7070
timestamp: new Date().toISOString(),
7171
connections,
72+
...(process.env.E2E_RUN_ID ? { runId: process.env.E2E_RUN_ID } : {}),
7273
})
7374
)
7475
} catch (error) {

apps/sim/app/api/health/route.test.ts

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
/**
22
* @vitest-environment node
33
*/
4-
import { describe, expect, it } from 'vitest'
4+
import { describe, expect, it, vi } from 'vitest'
55
import { GET } from '@/app/api/health/route'
66

77
describe('GET /api/health', () => {
@@ -14,4 +14,17 @@ describe('GET /api/health', () => {
1414
timestamp: expect.any(String),
1515
})
1616
})
17+
18+
it('returns the E2E run identity when configured', async () => {
19+
vi.stubEnv('E2E_RUN_ID', 'run-health-check')
20+
try {
21+
const response = await GET()
22+
await expect(response.json()).resolves.toMatchObject({
23+
status: 'ok',
24+
runId: 'run-health-check',
25+
})
26+
} finally {
27+
vi.unstubAllEnvs()
28+
}
29+
})
1730
})

apps/sim/app/api/health/route.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
11
/**
22
* Health check endpoint for deployment platforms and container probes.
33
*/
4+
export const dynamic = 'force-dynamic'
5+
46
export async function GET(): Promise<Response> {
57
return Response.json(
68
{
79
status: 'ok',
810
timestamp: new Date().toISOString(),
11+
...(process.env.E2E_RUN_ID ? { runId: process.env.E2E_RUN_ID } : {}),
912
},
1013
{ status: 200 }
1114
)

apps/sim/e2e/README.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Sim browser E2E
2+
3+
This directory contains the reusable full-stack Playwright harness. It runs the
4+
production Next.js app, realtime, deterministic external fakes, and a migrated
5+
per-run pgvector database.
6+
7+
## One-time setup
8+
9+
0. Install Node 22 and Bun. Playwright workers require Node 22; set
10+
`E2E_NODE_BINARY` to an alternate Node 22 executable when `node` on `PATH`
11+
points elsewhere.
12+
13+
1. Map the hosted E2E origin to loopback:
14+
15+
```bash
16+
echo "127.0.0.1 e2e.sim.ai" | sudo tee -a /etc/hosts
17+
```
18+
19+
The runner refuses to start unless every resolved address is loopback and an
20+
IPv4 `127.0.0.1` result is present. Chromium and Node are configured to prefer
21+
that IPv4 mapping, so CI environments that also synthesize `::1` remain safe.
22+
23+
2. Start a local pgvector/Postgres admin instance:
24+
25+
```bash
26+
docker run --rm --name sim-e2e-postgres \
27+
-e POSTGRES_USER=postgres \
28+
-e POSTGRES_PASSWORD=postgres \
29+
-e POSTGRES_DB=postgres \
30+
-p 5432:5432 \
31+
pgvector/pgvector:pg17
32+
```
33+
34+
Cleanup uses `DROP DATABASE ... WITH (FORCE)`, which requires PostgreSQL 13
35+
or newer and is supported by the pinned pgvector/PostgreSQL 17 image.
36+
37+
3. Install Chromium from `apps/sim`:
38+
39+
```bash
40+
bun run test:e2e:install-browsers
41+
```
42+
43+
## Run the foundation
44+
45+
From `apps/sim`:
46+
47+
```bash
48+
E2E_PG_ADMIN_URL=postgresql://postgres:postgres@127.0.0.1:5432/postgres \
49+
bun run test:e2e
50+
```
51+
52+
The runner creates a unique `sim_e2e_<runId>` database, migrates it, starts the
53+
Stripe fake and realtime, builds and starts Next.js, runs Playwright on Node 22,
54+
then stops services and drops only that guarded database.
55+
56+
On interruption, the runner launches a detached cleanup supervisor before
57+
exiting. It terminates managed process groups, force-drops the guarded database,
58+
and removes temporary auth/cloud-config directories even if another Ctrl-C
59+
terminates the foreground package runner.
60+
61+
Pass Playwright arguments after `--`:
62+
63+
```bash
64+
bun run test:e2e -- --project=hosted-billing-chromium-navigation
65+
bun run test:e2e -- --grep "unauthenticated"
66+
```
67+
68+
Do not invoke `playwright test` directly. Raw Playwright bypasses environment,
69+
database, process, sharding, and teardown guards; the config rejects runs that
70+
were not launched by the orchestrator. Report and trace viewer commands remain
71+
safe because they do not execute tests.
72+
73+
Sharding is supported only for the navigation project. The runner rejects
74+
`--shard` for `hosted-billing-chromium-workflows`.
75+
76+
## Diagnostics
77+
78+
- HTML report: `playwright-report/`
79+
- Traces and screenshots: `test-results/`
80+
- App, realtime, migration, and fake logs: `e2e/.runs/<runId>/logs/`
81+
82+
Open the report:
83+
84+
```bash
85+
node ../../node_modules/@playwright/test/cli.js show-report playwright-report
86+
```
87+
88+
Open a trace:
89+
90+
```bash
91+
node ../../node_modules/@playwright/test/cli.js show-trace test-results/<test>/trace.zip
92+
```
93+
94+
The runner starts every child process from a fresh environment. It allowlists
95+
only deterministic E2E values and shadows keys found in local `.env*` files, so
96+
developer credentials are not used as test state or written to reports.
97+
98+
Provider log scans are diagnostic tripwires, not proof of zero egress. The
99+
primary boundaries are the default-deny child environment, provider disabling,
100+
loopback-only service bindings, and guarded Stripe transport.

0 commit comments

Comments
 (0)