From 6a5cc0dac5a31eb286a3e16b21f7cc47eeda8f5c Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Wed, 7 Oct 2026 10:54:05 -0700 Subject: [PATCH 1/4] ci: shard integration and unit tests, run e2e groups in parallel, add a ci gate The PR critical path was the PostgreSQL integration suite (~15 min), run in full on an 8 vCPU runner once per provisioning path. Its files run one at a time, so the runner sat mostly idle. - integration: each provisioning path (push, migrate) is split into 4 Vitest shards on 4 vCPU runners. Both paths keep the full suite: migrations add triggers, checks and NOT VALID constraints that db:push does not, so the schemas differ. - e2e: the four next-dev groups (scim, cli, stop-after, desktop-inbox) run as a matrix, each on its own database. The boot/wait/stop shell lives once in http-e2e.sh. - lint: lint, audits, type-check and schema sync split off from the tests. - test: apps/sim unit tests sharded 2 ways; shard 1 also runs root scripts and the other workspaces. Each shard has its own Turbo cache disk. - ci: one aggregate job that fails unless every check passed (skipped is allowed), so a ruleset can require a single stable check. Deploy gating is unchanged: migrate still requires the whole Test and Build workflow. --- .agents/skills/ship/SKILL.md | 2 +- .claude/rules/sim-testing.md | 6 +- .cursor/rules/sim-testing.mdc | 6 +- .github/scripts/http-e2e.sh | 157 ++++++++++++ .github/workflows/test-build.yml | 411 +++++++++---------------------- 5 files changed, 276 insertions(+), 306 deletions(-) create mode 100755 .github/scripts/http-e2e.sh diff --git a/.agents/skills/ship/SKILL.md b/.agents/skills/ship/SKILL.md index fdd7e52ad4f..375887fd1b8 100644 --- a/.agents/skills/ship/SKILL.md +++ b/.agents/skills/ship/SKILL.md @@ -47,7 +47,7 @@ When the user runs `/ship`: - Run `/db-migrate` to review the migration for zero-downtime safety (expand/contract phasing, backward-compatibility with the deployed app version). - `(cd packages/db && bunx drizzle-kit generate && git status --porcelain ./migrations)` must print nothing (CI's schema/migration sync step). - `bun run check:migrations origin/staging` must pass (staging is the PR base). Do not silence a flagged statement with a `-- migration-safe:` annotation unless `/db-migrate` confirmed the old code no longer depends on it; otherwise split the destructive change into a later deploy. -6. **Run pre-ship checks** from the repo root before staging. This has two phases: first **regenerate** every committed artifact so generated files never drift into a CI failure (this is what catches things like `agent-stream-docs` going stale after a `models.ts` edit), then run the **full audit suite** CI's `Lint and Test` job enforces. Both phases parallelize — but only across commands that write **disjoint** outputs — and a bare `wait` swallows child exit codes, so both phases below explicitly collect each job's status and abort ship if any failed. +6. **Run pre-ship checks** from the repo root before staging. This has two phases: first **regenerate** every committed artifact so generated files never drift into a CI failure (this is what catches things like `agent-stream-docs` going stale after a `models.ts` edit), then run the **full audit suite** CI's `lint` job enforces. Both phases parallelize — but only across commands that write **disjoint** outputs — and a bare `wait` swallows child exit codes, so both phases below explicitly collect each job's status and abort ship if any failed. **Phase A — regenerate the always-in-repo committed artifacts (parallel), then let step 7 stage whatever changed.** Regenerate only the generators whose inputs live entirely in this repo and that any ordinary code change can drift — `agent-stream-docs:generate` (derives from the provider model registry), `docs-manifest:generate` (derives from docs page paths), and `skills:sync` (derives from `.agents/skills/**`). They write disjoint outputs (`apps/docs/…/agent.mdx`, `apps/sim/lib/mothership/generated/docs-manifest.ts`, and `.claude/skills` links), so they parallelize safely, and each is idempotent (a no-op when already in sync): ```bash diff --git a/.claude/rules/sim-testing.md b/.claude/rules/sim-testing.md index dbbb381daac..95881561a31 100644 --- a/.claude/rules/sim-testing.md +++ b/.claude/rules/sim-testing.md @@ -32,11 +32,11 @@ contracts, and demonstrated regressions. | Suffix | Needs | Run with | In CI | |--------|-------|----------|-------| -| `*.test.ts(x)` | nothing; global mocks from `vitest.setup.ts` | `vitest run` | Lint and Test job | -| `*.integration.ts` | real PostgreSQL (`TEST_DATABASE_URL`), optionally Redis (`TEST_REDIS_URL`) | `vitest run --mode integration` | `PostgreSQL integration` job, by glob | +| `*.test.ts(x)` | nothing; global mocks from `vitest.setup.ts` | `vitest run` | `test` jobs (sharded) | +| `*.integration.ts` | real PostgreSQL (`TEST_DATABASE_URL`), optionally Redis (`TEST_REDIS_URL`) | `vitest run --mode integration` | `integration` jobs, by glob (sharded per provisioning path) | | `*.live.test.ts` | provider APIs, hosted sandboxes, local runtimes, or sibling checkouts | `vitest run --mode live ` (apps/sim) | never | | `apps/desktop/e2e/*.spec.ts` | the packaged Electron app | Playwright | desktop E2E workflow | -| `apps/sim/scripts/test-*-e2e.ts` | a running app over HTTP | its `package.json` script when one exists (`bun run test:scim:e2e`; `test:workflow-version-compare:e2e` adds `--no-env-file`), else `bun scripts/test--e2e.ts` from apps/sim | End-to-end over real HTTP job | +| `apps/sim/scripts/test-*-e2e.ts` | a running app over HTTP | its `package.json` script when one exists (`bun run test:scim:e2e`; `test:workflow-version-compare:e2e` adds `--no-env-file`), else `bun scripts/test--e2e.ts` from apps/sim | `e2e` jobs (`.github/scripts/http-e2e.sh`) | - A unit test lives next to its source: `feature.ts` → `feature.test.ts`. No network, no database, no real timers. diff --git a/.cursor/rules/sim-testing.mdc b/.cursor/rules/sim-testing.mdc index e4cc023d9f4..75197fc8a0e 100644 --- a/.cursor/rules/sim-testing.mdc +++ b/.cursor/rules/sim-testing.mdc @@ -30,11 +30,11 @@ contracts, and demonstrated regressions. | Suffix | Needs | Run with | In CI | |--------|-------|----------|-------| -| `*.test.ts(x)` | nothing; global mocks from `vitest.setup.ts` | `vitest run` | Lint and Test job | -| `*.integration.ts` | real PostgreSQL (`TEST_DATABASE_URL`), optionally Redis (`TEST_REDIS_URL`) | `vitest run --mode integration` | `PostgreSQL integration` job, by glob | +| `*.test.ts(x)` | nothing; global mocks from `vitest.setup.ts` | `vitest run` | `test` jobs (sharded) | +| `*.integration.ts` | real PostgreSQL (`TEST_DATABASE_URL`), optionally Redis (`TEST_REDIS_URL`) | `vitest run --mode integration` | `integration` jobs, by glob (sharded per provisioning path) | | `*.live.test.ts` | provider APIs, hosted sandboxes, local runtimes, or sibling checkouts | `vitest run --mode live ` (apps/sim) | never | | `apps/desktop/e2e/*.spec.ts` | the packaged Electron app | Playwright | desktop E2E workflow | -| `apps/sim/scripts/test-*-e2e.ts` | a running app over HTTP | its `package.json` script when one exists (`bun run test:scim:e2e`; `test:workflow-version-compare:e2e` adds `--no-env-file`), else `bun scripts/test--e2e.ts` from apps/sim | End-to-end over real HTTP job | +| `apps/sim/scripts/test-*-e2e.ts` | a running app over HTTP | its `package.json` script when one exists (`bun run test:scim:e2e`; `test:workflow-version-compare:e2e` adds `--no-env-file`), else `bun scripts/test--e2e.ts` from apps/sim | `e2e` jobs (`.github/scripts/http-e2e.sh`) | - A unit test lives next to its source: `feature.ts` → `feature.test.ts`. No network, no database, no real timers. diff --git a/.github/scripts/http-e2e.sh b/.github/scripts/http-e2e.sh new file mode 100755 index 00000000000..202a9a2f2a2 --- /dev/null +++ b/.github/scripts/http-e2e.sh @@ -0,0 +1,157 @@ +#!/usr/bin/env bash +# Runs one end-to-end suite group over real HTTP, each against its own `next dev` app. +# +# Usage: http-e2e.sh (run from apps/sim) +# +# The job provides DATABASE_URL, BETTER_AUTH_SECRET and ENCRYPTION_KEY; each group sets the rest of +# its app's environment here. Reports and server logs land in $RUNNER_TEMP/e2e. +# +# The first request cold-compiles the app under Turbopack, which takes 42-150s on CI runners, so +# the readiness deadline only has to catch a hung boot: an exited server fails immediately, and +# either way the server log tail lands in the job log. +# +# Each app starts from an empty Turbopack dev cache: a cache written under other NEXT_PUBLIC_* +# values, by a server that `next dev` SIGKILLs 100ms after SIGTERM, can panic Turbopack or wedge a +# route compile on restore. It runs in its own session under an E2E_APP tag, and stop-session.sh +# returns only once every process in that session or carrying that tag has exited (Next's +# telemetry flush runs detached and still writes .next/dev). +set -euo pipefail + +group=${1:?usage: http-e2e.sh } +report_dir="$RUNNER_TEMP/e2e" +ready_timeout_seconds=300 +mkdir -p "$report_dir" + +server_pid='' +app_tag='' +server_log='' +status_log='' + +finish() { + local status=$? + if [ -n "$server_pid" ]; then + bash "$GITHUB_WORKSPACE/.github/scripts/stop-session.sh" "$server_pid" "$app_tag" || status=1 + wait "$server_pid" 2>/dev/null || true + if [ -n "$status_log" ]; then + awk '/^ (GET|POST|PUT|PATCH|DELETE|HEAD) \/api\// { print }' "$server_log" > "$status_log" + fi + if [ "$status" -ne 0 ]; then + tail -n 200 "$server_log" + fi + fi + exit "$status" +} +trap finish EXIT + +# start_app