Skip to content

Commit f7d10c7

Browse files
Bill Leoutsakoscursoragent
authored andcommitted
Seed and verify E2E settings personas
Provision the validated world through real product boundaries, capture isolated sessions through the login UI, and assert persona contracts plus two-worker cross-world isolation without exposing credentials to Playwright. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 48c2d45 commit f7d10c7

14 files changed

Lines changed: 953 additions & 16 deletions

apps/sim/e2e/README.md

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,10 @@ E2E_PG_ADMIN_URL=postgresql://postgres:postgres@127.0.0.1:5432/postgres \
5050
```
5151

5252
The runner creates a unique `sim_e2e_<runId>` database, migrates it, starts the
53-
Stripe fake and realtime, builds and starts Next.js, runs Playwright on Node 22,
54-
then stops services and drops only that guarded database.
53+
Stripe fake and realtime, builds and starts Next.js, seeds the validated persona
54+
world through production APIs plus narrow trusted arrangements, captures each
55+
persona through the real login UI, runs Playwright on Node 22, then stops
56+
services and drops only that guarded database.
5557

5658
On interruption, the runner launches a detached cleanup supervisor before
5759
exiting. It terminates managed process groups, force-drops the guarded database,
@@ -89,13 +91,17 @@ were not launched by the orchestrator. Report and trace viewer commands remain
8991
safe because they do not execute tests.
9092

9193
Sharding is supported only for the navigation project. The runner rejects
92-
`--shard` for `hosted-billing-chromium-workflows`.
94+
`--shard` for workflows, persona contracts, and the dedicated two-worker
95+
cross-world isolation project.
9396

9497
## Diagnostics
9598

9699
- HTML report: `playwright-report/`
97100
- Traces and screenshots: `test-results/`
98-
- App, realtime, migration, and fake logs: `e2e/.runs/<runId>/logs/`
101+
- App, realtime, migration, seed, auth-capture, and fake logs:
102+
`e2e/.runs/<runId>/logs/`
103+
- Non-secret persona manifest and post-login auth-capture screenshots:
104+
`e2e/.runs/<runId>/`
99105

100106
Open the report:
101107

@@ -111,11 +117,14 @@ node ../../node_modules/@playwright/test/cli.js show-trace test-results/<test>/t
111117

112118
The runner starts every child process from a fresh, purpose-specific
113119
environment. Next build receives deterministic sentinels instead of the run
114-
database/fake endpoint; app, realtime, migrations, future seeding/auth capture,
115-
and Playwright each receive only their required values. Next build/start shadow
120+
database/fake endpoint; app, realtime, migrations, seeding, auth capture, and
121+
Playwright each receive only their required values. Playwright receives the
122+
non-secret manifest and storage-state directory, never passwords, the admin API
123+
key, or the database URL. Next build/start shadow
116124
keys found in local `.env*` files, while children that cannot load those files
117125
omit denied keys entirely. Developer credentials are not used as test state or
118-
written to reports.
126+
written to reports. Synthetic persona passwords live in a private run-home file
127+
and are removed with the auth/cloud-config directory during teardown.
119128

120129
E2E builds verify the reviewed sandbox-bundle source/dependency/output
121130
fingerprint and never regenerate committed `.cjs` files. Regeneration remains an

apps/sim/e2e/fakes/stripe/server.ts

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ export const STRIPE_FAKE_ENDPOINTS = {
66
health: '/health',
77
requestLog: '/__control/requests',
88
reset: '/__control/reset',
9+
telemetry: '/v1/traces',
910
} as const
1011

1112
const DEFAULT_MAX_BODY_BYTES = 64 * 1024
@@ -63,6 +64,7 @@ function isExpectedStripeRequest(method: string, path: string): boolean {
6364
((method === 'GET' || method === 'POST') && path === '/v1/customers/search') ||
6465
(method === 'GET' && path === '/v1/customers') ||
6566
(method === 'POST' && path === '/v1/customers') ||
67+
(method === 'POST' && path === STRIPE_FAKE_ENDPOINTS.telemetry) ||
6668
(method === 'GET' && /^\/v1\/customers\/cus_e2e_[a-f0-9]+$/.test(path))
6769
)
6870
}
@@ -326,6 +328,11 @@ export function createStripeFakeServer(options: StripeFakeServerOptions): Stripe
326328
unexpected: !expected,
327329
})
328330

331+
if (method === 'POST' && url.pathname === STRIPE_FAKE_ENDPOINTS.telemetry) {
332+
sendJson(response, 200, { partialSuccess: {} }, requestId)
333+
return
334+
}
335+
329336
if (!secureEqual(request.headers.authorization, expectedAuthorization)) {
330337
sendStripeError(
331338
response,
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
import path from 'node:path'
2+
import { test as base } from '@playwright/test'
3+
import {
4+
type PersonaManifestEntry,
5+
readScenarioManifest,
6+
type ScenarioManifest,
7+
} from './e2e-world'
8+
9+
interface PersonaFixtures {
10+
personaManifest: ScenarioManifest
11+
contextForPersona: (
12+
personaKey: string
13+
) => Promise<import('@playwright/test').BrowserContext>
14+
}
15+
16+
export const test = base.extend<PersonaFixtures>({
17+
personaManifest: [
18+
async ({}, use) => {
19+
const manifest = readScenarioManifest(requiredEnv('E2E_MANIFEST_PATH'))
20+
if (!manifest.authCaptureComplete) {
21+
throw new Error('Persona storage states were not captured successfully')
22+
}
23+
await use(manifest)
24+
},
25+
{ scope: 'test' },
26+
],
27+
contextForPersona: async ({ browser, personaManifest }, use) => {
28+
const contexts = new Set<import('@playwright/test').BrowserContext>()
29+
await use(async (personaKey) => {
30+
const persona = requirePersona(personaManifest, personaKey)
31+
const context = await browser.newContext({
32+
storageState: path.join(requiredEnv('E2E_STORAGE_STATE_DIR'), persona.storageStatePath),
33+
})
34+
contexts.add(context)
35+
return context
36+
})
37+
await Promise.all([...contexts].map((context) => context.close()))
38+
},
39+
})
40+
41+
export { expect } from '@playwright/test'
42+
43+
export function requirePersona(
44+
manifest: ScenarioManifest,
45+
personaKey: string
46+
): PersonaManifestEntry {
47+
const persona = manifest.personas[personaKey]
48+
if (!persona) throw new Error(`Unknown persona: ${personaKey}`)
49+
return persona
50+
}
51+
52+
function requiredEnv(key: string): string {
53+
const value = process.env[key]
54+
if (!value) throw new Error(`Missing persona fixture environment value: ${key}`)
55+
return value
56+
}

apps/sim/e2e/foundation/safety.spec.ts

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,8 @@ test.describe('foundation safety guards', () => {
7979
expect(build.env.DATABASE_URL).toContain('/sim_e2e_build_sentinel')
8080
expect(build.env.DATABASE_URL).not.toBe(app.env.DATABASE_URL)
8181
expect(build.env.STRIPE_API_BASE_URL).toBe('http://127.0.0.1:1')
82+
expect(build.env.TELEMETRY_ENDPOINT).toBe('http://127.0.0.1:1/v1/traces')
83+
expect(app.env.TELEMETRY_ENDPOINT).toMatch(/^http:\/\/127\.0\.0\.1:\d+\/v1\/traces$/)
8284
expect(build.env.E2E_RUN_ID).toBe('build_sentinel')
8385

8486
for (const key of Object.keys(app.env).filter((key) => key.startsWith('NEXT_PUBLIC_'))) {
@@ -134,7 +136,13 @@ test.describe('foundation safety guards', () => {
134136
).not.toThrow()
135137
expect(() =>
136138
parseRunOptions(['--project=hosted-billing-chromium-workflows', '--shard=1/2'])
137-
).toThrow(/coupled workflows must remain unsharded/)
139+
).toThrow(/coupled E2E projects must remain unsharded/)
140+
expect(() =>
141+
parseRunOptions(['--project=hosted-billing-chromium-personas'])
142+
).not.toThrow()
143+
expect(() =>
144+
parseRunOptions(['--project=hosted-billing-chromium-persona-isolation', '--shard=1/2'])
145+
).toThrow(/coupled E2E projects must remain unsharded/)
138146
})
139147

140148
test('Playwright CLI arguments cannot override orchestration invariants', () => {

apps/sim/e2e/foundation/stripe-fake.spec.ts

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,19 @@ test('Stripe fake records allowlisted calls and rejects unknown routes', async (
3636
)
3737
expect(search.status).toBe(200)
3838

39+
const telemetry = await fetch(`${baseUrl}/v1/traces`, {
40+
method: 'POST',
41+
headers: { 'content-type': 'application/json' },
42+
body: JSON.stringify({ resourceSpans: [] }),
43+
})
44+
expect(telemetry.status).toBe(200)
45+
expect(
46+
fake.requestLog.some(
47+
({ method, path, unexpected }) =>
48+
method === 'POST' && path === '/v1/traces' && !unexpected
49+
)
50+
).toBe(true)
51+
3952
const unsupportedSearch = await fetch(
4053
`${baseUrl}/v1/customers/search?${new URLSearchParams({
4154
query: 'name:"Foundation"',
Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
import { mkdirSync } from 'node:fs'
2+
import path from 'node:path'
3+
import { chromium, expect } from '@playwright/test'
4+
import {
5+
readPersonaCredentials,
6+
readScenarioManifest,
7+
scenarioManifestSchema,
8+
writeJsonAtomic,
9+
} from '../fixtures/e2e-world'
10+
11+
const baseUrl = requiredEnv('E2E_BASE_URL')
12+
const manifestPath = requiredEnv('E2E_MANIFEST_PATH')
13+
const credentialsPath = requiredEnv('E2E_CREDENTIALS_PATH')
14+
const storageStateDirectory = requiredEnv('E2E_STORAGE_STATE_DIR')
15+
const screenshotsDirectory = requiredEnv('E2E_AUTH_SCREENSHOT_DIR')
16+
const UI_RETRY_DELAYS_MS = [1_000, 2_000, 4_000, 8_000, 16_000, 30_000] as const
17+
18+
async function main(): Promise<void> {
19+
const manifest = readScenarioManifest(manifestPath)
20+
const credentials = readPersonaCredentials(credentialsPath)
21+
if (manifest.runId !== credentials.runId || manifest.runId !== requiredEnv('E2E_RUN_ID')) {
22+
throw new Error('Persona auth inputs belong to different E2E runs')
23+
}
24+
mkdirSync(storageStateDirectory, { recursive: true })
25+
mkdirSync(screenshotsDirectory, { recursive: true })
26+
27+
const browser = await chromium.launch({
28+
args: ['--host-resolver-rules=MAP e2e.sim.ai 127.0.0.1'],
29+
})
30+
try {
31+
for (const [personaKey, persona] of Object.entries(manifest.personas)) {
32+
const login = credentials.personas[personaKey]
33+
if (!login) throw new Error(`Missing private login for persona: ${personaKey}`)
34+
const context = await browser.newContext({ baseURL: baseUrl })
35+
try {
36+
const page = await context.newPage()
37+
await page.goto('/login')
38+
await expect(page.getByRole('heading', { name: 'Sign in' })).toBeVisible()
39+
await page.getByLabel('Email').fill(login.email)
40+
await page.getByRole('textbox', { name: 'Password' }).fill(login.password)
41+
await signInThroughUi(page, personaKey)
42+
43+
const sessionResponse = await context.request.get('/api/auth/get-session')
44+
if (!sessionResponse.ok()) {
45+
throw new Error(`Session probe failed for ${personaKey}: ${sessionResponse.status()}`)
46+
}
47+
const session = (await sessionResponse.json()) as {
48+
user?: { id?: string; email?: string }
49+
session?: { activeOrganizationId?: string | null }
50+
}
51+
if (session.user?.id !== persona.userId || session.user.email !== persona.email) {
52+
throw new Error(`Session identity mismatch for persona: ${personaKey}`)
53+
}
54+
if (
55+
(session.session?.activeOrganizationId ?? null) !== persona.expectedActiveOrganizationId
56+
) {
57+
throw new Error(`Active organization mismatch for persona: ${personaKey}`)
58+
}
59+
60+
await page.goto(persona.canonicalRoute)
61+
await expect(page).toHaveURL(/\/settings\/general$/)
62+
await expect(page.getByRole('heading', { name: 'General', level: 1 })).toBeVisible()
63+
await page.screenshot({
64+
path: path.join(screenshotsDirectory, `${personaKey}.png`),
65+
fullPage: false,
66+
})
67+
const storageStatePath = path.join(storageStateDirectory, persona.storageStatePath)
68+
await context.storageState({ path: storageStatePath })
69+
} finally {
70+
await context.close()
71+
}
72+
}
73+
74+
manifest.authCaptureComplete = true
75+
writeJsonAtomic(manifestPath, scenarioManifestSchema.parse(manifest))
76+
} finally {
77+
await browser.close()
78+
}
79+
}
80+
81+
function requiredEnv(key: string): string {
82+
const value = process.env[key]
83+
if (!value) throw new Error(`Missing auth capture environment value: ${key}`)
84+
return value
85+
}
86+
87+
async function signInThroughUi(
88+
page: import('@playwright/test').Page,
89+
personaKey: string
90+
): Promise<void> {
91+
for (let attempt = 0; attempt <= UI_RETRY_DELAYS_MS.length; attempt += 1) {
92+
const responsePromise = page.waitForResponse((response) => {
93+
const url = new URL(response.url())
94+
return url.pathname === '/api/auth/sign-in/email'
95+
})
96+
await page.getByRole('button', { name: 'Sign in' }).click()
97+
const response = await responsePromise
98+
if (response.status() === 200) {
99+
try {
100+
await page.waitForURL(/\/workspace(?:\/|$)/, { timeout: 30_000 })
101+
} catch {
102+
throw new Error(`UI sign-in did not redirect for persona: ${personaKey}`)
103+
}
104+
return
105+
}
106+
if (response.status() !== 429 || attempt === UI_RETRY_DELAYS_MS.length) {
107+
throw new Error(
108+
`UI sign-in failed for persona ${personaKey} with status ${response.status()}; response body redacted`
109+
)
110+
}
111+
await new Promise((resolve) => setTimeout(resolve, UI_RETRY_DELAYS_MS[attempt]))
112+
}
113+
}
114+
115+
main().catch((error) => {
116+
const raw = error instanceof Error ? (error.stack ?? error.message) : String(error)
117+
let redacted = raw
118+
try {
119+
const credentials = readPersonaCredentials(credentialsPath)
120+
for (const { password } of Object.values(credentials.personas)) {
121+
redacted = redacted.replaceAll(password, '[REDACTED]')
122+
}
123+
} catch {
124+
redacted = 'Auth capture failed; credentials unavailable for safe diagnostic formatting'
125+
}
126+
console.error(redacted)
127+
process.exitCode = 1
128+
})

apps/sim/e2e/scripts/options.ts

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
11
const NAVIGATION_PROJECT = 'hosted-billing-chromium-navigation'
22
const WORKFLOWS_PROJECT = 'hosted-billing-chromium-workflows'
3+
const PERSONAS_PROJECT = 'hosted-billing-chromium-personas'
4+
const PERSONA_ISOLATION_PROJECT = 'hosted-billing-chromium-persona-isolation'
5+
const E2E_PROJECTS = new Set([
6+
NAVIGATION_PROJECT,
7+
WORKFLOWS_PROJECT,
8+
PERSONAS_PROJECT,
9+
PERSONA_ISOLATION_PROJECT,
10+
])
311
const FORBIDDEN_OPTIONS = [
412
'--config',
513
'-c',
@@ -51,9 +59,7 @@ export function parseRunOptions(
5159
}
5260
const playwrightArgs = normalizedArgs.filter((arg) => arg !== '--reuse-build')
5361
const projects = getEqualsOptionValues(playwrightArgs, '--project')
54-
const unknownProject = projects.find(
55-
(project) => project !== NAVIGATION_PROJECT && project !== WORKFLOWS_PROJECT
56-
)
62+
const unknownProject = projects.find((project) => !E2E_PROJECTS.has(project))
5763
if (unknownProject) throw new Error(`Unknown E2E Playwright project: ${unknownProject}`)
5864
const hasShard = hasOption(playwrightArgs, '--shard')
5965

@@ -64,7 +70,7 @@ export function parseRunOptions(
6470
projects.some((project) => project !== NAVIGATION_PROJECT))
6571
) {
6672
throw new Error(
67-
`--shard is supported only with --project=${NAVIGATION_PROJECT}; coupled workflows must remain unsharded`
73+
`--shard is supported only with --project=${NAVIGATION_PROJECT}; coupled E2E projects must remain unsharded`
6874
)
6975
}
7076

0 commit comments

Comments
 (0)