Skip to content

ci: verify Endform against the product Playwright baseline - #2

Open
ostenbom wants to merge 3 commits into
codex/playwright-baselinefrom
codex/endform-product-suite
Open

ostenbom wants to merge 3 commits into
codex/playwright-baselinefrom
codex/endform-product-suite

Conversation

@ostenbom

@ostenbom ostenbom commented Oct 8, 2026 •

Copy link
Copy Markdown

What does this PR do?

Run the existing main Chromium product suite on Endform for both Node and Cloudflare, using the same application and assertions as the passing native baseline in #1. This PR is stacked on codex/playwright-baseline; leave #1 open for reproducible evidence. Retarget to main only after reviewing the comparison; neither PR is merged by this task.

Baseline: commit 40152078f550ba50e012812450f175799d88db43, passing CI run. One measured sample: workflow 19m 58s; product test-stage wall-clock span 16m 49s; Node 328 passed / 34 skipped, Cloudflare 315 passed / 47 skipped. Both select the existing 362 tests; skips follow existing gates.

Endform 0.81.3 is a dev dependency; CI invokes npx endform@latest test --organization-id 2G1ZCj7X. Its existing GitHub connection is confirmed, with job-scoped contents:read and id-token:write for OIDC. The eight existing fixture partitions remain isolated. Each job generates its native partition with Playwright’s list command, then gives that exact test list to Endform for Node followed by Cloudflare. The lists are disjoint and cover all 362 cases; no shard runs the full suite. Endform distributes within each partition, with concurrency capped at one to preserve the shared fixture state. This suite cannot remove its existing shard orchestration without changing test isolation. Supplemental browser, table, playground, and package jobs remain native and intact. Browser installation is removed only from the selected Endform path.

The managed webServer reuses the existing fixture setup and teardown, including SQLite or D1/R2, registry mock, migrations, and seeds. A loopback-only readiness/state service supplies ephemeral fixture state to remote workers and accepts the refreshed PAT after existing setup resets. No production credentials or application services are added. Only the runtime-read test image is explicitly transferred.

Endform enables fullyParallel and retain-on-failure tracing while preserving existing retries (CI 1, local 0). It limits each fixture to one concurrent test because the suite shares database and authentication state. The CRUD group is serial because read tests depend on the original seeded title before a later test renames it; assertions and test bodies are unchanged. Existing setup groups remain serial. The admin warm-up is shared with the native setup and runs on Endform’s supplied test page; it also waits for the page heading before tests begin. Endform’s job limit is 90 minutes to allow a completed run with this conservative concurrency; that limit is not a timing result.

Local execution remains unverified: the unmodified setup kept its server process alive, HTTP-proxied development modules exceeded the login hydration assertion, and an attempted TCP tunnel hit a protocol authentication failure. The TCP workaround was removed. Diagnostic runs are smaller suites and are excluded from the benchmark. Example diagnostic dashboard. Full Endform CI evidence and the comparison are pending; this configuration is not yet claimed to work.

Type of change

  • Bug fix
  • Feature (requires maintainer-approved Discussion)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Checklist

  • I have read CONTRIBUTING.md
  • pnpm typecheck passes
  • pnpm lint passes
  • pnpm test passes (or targeted tests for my change)
  • pnpm format has been run
  • I have added/updated tests for my changes (if applicable)
  • User-visible strings in the admin UI are wrapped for translation (if applicable). Do not include messages.po changes except in translation PRs — a workflow extracts catalogs on merge to main.
  • I have added and reviewed the user-facing changeset (if this PR changes a published package)
  • New features link to an approved Discussion: https://github.com/emdash-cms/emdash/discussions/...
  • I have included screenshots below if this PR changes the UI

AI-generated code disclosure

  • This PR includes AI-generated code — model/tool: GPT-6 / Codex

Screenshots / test output

Not applicable: no published-package, user interface, localization, or product-feature changes, so changeset, screenshots, translation, and prior Discussion are unnecessary. Verification uses the existing behavior tests rather than adding config-pin tests.

Verified locally: root pnpm build, pnpm format, pnpm lint:quick, type-aware lint with zero diagnostics, pnpm typecheck, and native targeted tests (login accessibility: 1 passed; CRUD: 10 passed). Full Endform CI is the remaining integration check.

Verified selection: native partitions contain 50, 43, 55, 36, 53, 48, 35, and 42 tests; their union is exactly the full 362-case suite with no overlaps. The first list also round-trips through Playwright’s test-list filter without changing selection. The earlier two-target pilot is superseded and excluded from timing because it reduced fixture isolation. Current full CI: https://github.com/endformdev/emdash/actions/runs/37712253469, commit 32992da58a1624b773a5ff3fe7c16f9b0bcb89b8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant