From ef4abe592bd6eaac7deeeeec82b29f56e2638ed2 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 10:12:09 +0200 Subject: [PATCH 01/38] docs: add micro Supabase stacks design spec High-density Postgres + minimal stack design: pod architecture, micro.conf profile, CoW templates, fleet daemon with suspend-on-idle. Co-Authored-By: Claude Fable 5 --- ...2026-07-07-micro-supabase-stacks-design.md | 210 ++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 docs/specs/2026-07-07-micro-supabase-stacks-design.md diff --git a/docs/specs/2026-07-07-micro-supabase-stacks-design.md b/docs/specs/2026-07-07-micro-supabase-stacks-design.md new file mode 100644 index 0000000000..dcb690ff19 --- /dev/null +++ b/docs/specs/2026-07-07-micro-supabase-stacks-design.md @@ -0,0 +1,210 @@ +# Micro Supabase Stacks — High-Density Local & Free-Tier Postgres + +- **Date:** 2026-07-07 +- **Status:** Draft for review +- **Scope:** Postgres first; architecture sized for the full minimal stack (Postgres, PostgREST, Auth, Realtime, Edge Functions) + +## Goal + +Run 100+ Supabase-compatible Postgres instances in parallel on small machines (8–16GB), for agents working in parallel git worktrees, local development, and very-low-resource free plans. "Compatible" means real upstream Postgres with the complete extension set of `supabase/postgres 17.6.1.143` — no PGlite-style reimplementation, no SQLite backend, no dropped extensions. + +## Requirements (agreed) + +- **Target environments:** both local dev machines (macOS/Linux) and Linux micro-VMs/containers for cloud free tier, from one shared design. +- **Compatibility bar:** every Supabase extension installable and behaviorally identical via `CREATE EXTENSION`; `shared_preload_libraries` trimmed to near-empty by default (heavy libraries load on demand). +- **Suspend-on-idle is acceptable:** first connection after idle may pay a wake-up latency of tens to hundreds of milliseconds. +- **Scope now:** Postgres and its orchestration. Sidecar services (PostgREST, Auth, Realtime, Edge Functions) are designed for but implemented later; their needs shape Postgres decisions today. +- **Durability:** data is disposable/re-cloneable everywhere (`fsync=off` profile). Recovery from host-crash corruption is "re-clone from template." +- **Density target:** 100+ registered instances on an 8–16GB host, most idle at any moment. + +## Non-goals + +- Production/paid-tier deployments. +- Postgres major-version in-place upgrades (pods are disposable; reset onto a new base template). +- Windows native support (Docker fallback only). +- Slimming the sidecar services themselves (tracked as follow-up; see Risks). + +## Approach + +**Approach A — stock Postgres + aggressive configuration + a host-level orchestrator. No Postgres source patches.** + +Rejected alternatives: + +- **B — one cluster, database-per-worktree:** cheapest possible, but roles and `ALTER SYSTEM` are cluster-wide, `pg_cron` binds to one database, restart isolation is impossible, and it cannot model free-tier isolation. Kept in the back pocket for pure local agent swarms only. +- **C — patched "micro-mode" Postgres:** a fork of a fork (Supabase already patches Postgres) with every extension becoming a compatibility risk, for gains profiling suggests are small once suspend-on-idle and preload trimming are in place. Revisit only if Approach A hits a measured wall. + +Postgres's heavy reputation comes from default configs and Supabase's preload stack, not the engine. Tuned honestly, an idle instance costs ~15–25MB unique memory; suspended, it costs zero. + +## Architecture + +The deployment unit is a **pod**: one Postgres per worktree/project, with sidecars slotting in beside it later. A host is composed of: + +1. **Shared read-only artifacts (per host, not per pod):** + - One Postgres 17.6 install tree with the full Supabase extension set, built from the `supabase/postgres` Nix package definitions pinned to tag `17.6.1.143` (extension parity guaranteed, Supabase's own patches included, none of ours added). One artifact per platform: `aarch64-darwin`, `x86_64-linux`, `aarch64-linux`. All pods share the code pages. + - A **template store** (see Templates below). +2. **Per-pod state (cheap):** a copy-on-write clone of a template plus a generated conf overlay and allocated ports. +3. **One fleet daemon per host** — the orchestrator (see Fleet daemon below). + +**Suspend/resume operates on the pod as a unit, never on Postgres alone.** Realtime holds a logical-replication connection and PostgREST holds a pool; Postgres napping independently would look like an outage to them. Idle detection watches external traffic at the proxy edge and ignores intra-pod connections. + +### How the full-stack goal shapes Postgres decisions now + +- `wal_level=logical` with replication slots budgeted (Realtime), not `minimal`. +- `max_connections=40`: ~15 reserved for sidecar pools, ~25 for the user. +- Template pre-seeded with Supabase roles/schemas/baseline migrations so sidecars boot against a fresh clone without setup. +- `max_slot_wal_keep_size` capped so an undrained slot can never eat the disk. Because Realtime stops and starts with the pod, slots never dangle while WAL accumulates. +- Realtime (Elixir/BEAM, ~150–300MB) will dominate a full pod's memory. It is multi-tenant by design; the likely future is one shared Realtime serving all pods. Nothing here blocks either choice — slots are per-instance regardless. + +### Per-service lazy start + +Scale-to-zero applies at two levels: the pod, and each service within a warm pod. + +- The pod manifest declares which services are **enabled** (default: all, for cloud parity). Enabled means "may be started on demand," not "running." +- The gateway routes per path (`/rest/v1`, `/auth/v1`, `/realtime/v1`, `/functions/v1`), so the first request to a service starts exactly that service, health-gates, then forwards. Until then the pod runs without it. Postgres is the only member that always starts on pod wake. +- A request to a never-used service therefore still succeeds — indistinguishable from cloud except for first-request latency (PostgREST/GoTrue fast; Realtime seconds). +- **DB-originated traffic wakes services:** `pg_cron`/trigger-driven webhooks via `pg_net` arrive at the proxy edge and legitimately start/keep-alive services. Consequence (documented, intended): a pod with a scheduled workload never suspends. +- **Max capacity is a testing concern, not a runtime default:** CI verifies the all-services-warm envelope; runtime optimizes for the typical case. + +## Postgres build + +- Source of truth: `supabase/postgres` Nix package set at `17.6.1.143`, built unchanged, producing a relocatable install tree per platform. +- No size-oriented compile flags or feature stripping (ICU, SSL stay — compat-relevant). Binary size is a per-host cost. +- Locale/collation setup mirrors the Supabase image exactly (provider and `C.UTF-8` defaults) to preserve dump/restore fidelity. +- **First implementation spike:** validate the Nix build of the full extension set on `aarch64-darwin`. Documented fallback: run Linux pods in a lightweight VM on macOS. + +## The `micro.conf` profile + +Everything not listed stays at PG defaults. + +**Memory:** + +| Setting | Value | Rationale | +|---|---|---| +| `shared_buffers` | `16MB` | Dev datasets are small; the OS page cache (shared across pods) backs reads; only touched pages cost RSS. | +| `work_mem` | `4MB` | Allocated only while sorting; safe ceiling. | +| `maintenance_work_mem` | `32MB` | Allocated only during vacuum/index build. | +| `jit` | `off` | Never loads LLVM into backends; the single biggest per-backend saving, useless on dev-sized data. | +| `huge_pages` | `off` | Irrelevant at this scale. | +| `max_connections` | `40` | Slots are cheap; only spawned backends (~2–5MB) cost real memory. | + +**Background CPU:** `autovacuum_naptime=5min`, `autovacuum_max_workers=1`, `bgwriter_lru_maxpages=0`, `walwriter_delay=10s`, `checkpoint_timeout=30min`, `max_parallel_workers=0`, `max_parallel_workers_per_gather=0`, `max_worker_processes=4` (headroom for on-demand `pg_cron`/`pg_net`/timescale workers), `track_io_timing=off`, minimal logging to a per-pod file. + +**Durability (disposable profile):** `fsync=off`, `synchronous_commit=off`, `full_page_writes=off`. Also makes startup/shutdown fast, which suspend/resume relies on. Host-crash corruption recovery: `reset` (re-clone). + +**Replication:** `wal_level=logical`, `max_wal_senders=5`, `max_replication_slots=5`, `max_slot_wal_keep_size=256MB`, `wal_keep_size=0`. + +### Preload policy: preload-on-enable + +`shared_preload_libraries` starts **empty**. Extensions that require preload (`pg_cron`, `pg_net`, `timescaledb`, `pg_stat_statements`, `auto_explain`, `pgaudit`, `plan_filter`, `supautils`, legacy `pgsodium`) are handled by an orchestrator API `enable-extension `: append the library to the pod's conf overlay, restart that pod's Postgres (sub-second with `fsync=off`), then `CREATE EXTENSION` works. All other extensions (`pgvector`, `pg_graphql`, `postgis`, `pgjwt`, `vault`, …) are plain `CREATE EXTENSION`, zero cost until used. `supautils` (platform role guardrails) is a profile flag: off for local dev, on for free-tier fidelity. + +### Config layering + +Per-pod `postgresql.conf` = `include micro.conf` (shared, read-only) + generated per-pod overlay (port, socket dir, preloads). User `ALTER SYSTEM` lands in `postgresql.auto.conf` inside the pod's data dir, surviving suspend/resume and matching real-project semantics. Auth via scram with the standard local-dev password, matching Supabase CLI conventions. + +## Templates & provisioning + +### Pod manifest + +Small declarative file stored with the pod (versions illustrative; defaults come from the stack package's version manifest): + +```yaml +id: worktree-nifty-dhawan +versions: { postgres: 17.6.1.143, auth: 2.177.0, realtime: 2.34.47, postgrest: 13.0.4, functions: 1.67.0 } +services: { rest: enabled, auth: enabled, realtime: enabled, functions: enabled } +flags: { supautils: off } +``` + +The `versions` tuple picks binaries, keys the warm-template cache, and tells the runtime what to run. + +### Template store (per host) + +Each service owns and applies its own migrations at boot (GoTrue → `auth`, Realtime → `realtime`, Storage → `storage`); the Postgres image ships only the baseline. Hence two layers: + +- **Base template**, tagged by Postgres image version only (`pg-17.6.1.143`): `initdb` with image-matching locale settings, boot, apply the `supabase/postgres` baseline migrations (roles: `anon`, `authenticated`, `service_role`, `supabase_admin`, …; schemas: `auth`, `realtime`, `storage`, `extensions`; default extensions), clean shutdown, mark read-only. This alone guarantees any combination of service versions can boot against a fresh clone — each service self-migrates exactly as it would against a cloud project. +- **Warm templates**, keyed by the full version-tuple hash, built lazily: on first provision of a new tuple, clone base → boot the full stack once → services self-migrate → shutdown → freeze as a cached template. Subsequent pods with the same tuple skip migration time entirely (first boot drops from seconds to clone cost). Pure optimization — cache miss falls back to the base path. LRU garbage-collected. + +**Service upgrades on an existing pod need no template machinery:** bump the manifest tuple; the new service version self-migrates the pod's live data dir on next boot, as in production. Templates matter only at provisioning time; a pod is never re-cloned under a live data directory. + +### Provisioning (fast path) + +1. Clone template → `pods//data` via `clonefile` (APFS) / `cp --reflink=auto` (btrfs/xfs) / plain copy fallback (template ~40–50MB, so ~a second worst case). +2. Write the per-pod conf overlay; register pod + ports with the fleet daemon. +3. Done — Postgres doesn't start until the first connection. Provision cost: milliseconds and near-zero bytes. + +### Lifecycle operations + +- `create` / `destroy` (destroy releases ports, reclaims disk). +- `reset` — re-clone from template; doubles as corruption recovery. +- `fork ` — CoW-clone an existing pod's data dir (after a brief clean suspend of the source) into a new pod. Agents branching a worktree get a byte-identical, independently-diverging database branch in milliseconds. +- `upgrade` — edit the manifest tuple; services self-migrate on next boot. + +On-disk layout follows the existing `~/.supabase` convention (where the binary cache already lives): `~/.supabase/{templates, pods//{data, conf, logs, run}}` — a single inspectable, deletable tree. + +### Addressing + +- **Postgres:** one TCP port per pod from an orchestrator-allocated range (the pgwire protocol has no pre-TLS host routing to exploit). +- **HTTP services:** one shared gateway port, host/path-routed (`.localhost:/rest/v1/…`) — also where lazy start hooks in. On cloud free tier, where pods have their own network identity, the same proxy binds per-pod addresses. +- Connection strings printed by the CLI match Supabase-CLI conventions. + +## Orchestrator: evolve `@supabase/stack` + new fleet daemon + +`@supabase/stack` (in the Bun CLI monorepo, `packages/stack`) is viable and is kept. It already provides: native-binary resolution with Docker fallback (shared cache at `~/.supabase/bin` — which *is* the shared install tree), dependency-ordered supervision via `@supabase/process-compose`, health-gated readiness, the key-translating API proxy, per-service `startService`/`stopService` (the substrate lazy start needs), port allocation, daemon/connect modes, and parallel-stack E2E tests. + +### Division of responsibilities + +- **`@supabase/stack` = pod runtime.** "Given a prepared data dir and version manifest, run this stack": ServiceDefs, supervision, health checks, API proxy, per-service start/stop, log streaming. +- **Fleet daemon = new thin host-level layer** owning everything that must exist when pods don't: pod registry and manifests, template store + CoW provisioning + `fork`, persistent port registry, the always-listening network edge, idle timers, wake/suspend policy. Hosts warm pods as in-process `StackHandle`s inside one Bun process. Programmatic TypeScript API (`fleet.create()`, `fleet.wake()`, `fleet.fork()`, `fleet.suspend()`); `supabase start/stop` become thin calls over it. + +This replaces the current daemon-per-stack fork model (100 pods must not mean 100 daemons, and a suspended pod must cost zero processes — only a shared daemon holding its port delivers that) and replaces `readReservedPorts()`'s per-create filesystem scan with an owned registry. + +### Suspend/resume mechanics + +- **Edge:** the fleet daemon permanently binds every pod's Postgres TCP port and the shared HTTP gateway. Postgres traffic is spliced TCP (per-port; negligible overhead for dev). HTTP routes by host/path as ApiProxy does today. +- **Wake:** connection to a suspended pod's port → start Postgres (~100–300ms with `fsync=off`) → forward. First HTTP request to a not-yet-running service on a warm pod → `stack.startService(name)` → health-gate → forward. +- **Idle:** suspend when a pod has no open external connections **and** no bytes flowing for T (per-profile: 5min local dev, 15min free tier). Suspend = `stack.stop()` (graceful, dependency-ordered, already implemented). Live Realtime websockets correctly keep a pod warm; `pg_cron`-driven pods correctly never sleep. +- **Crash handling:** pods run as detached process groups with pidfiles; the fleet daemon **adopts** running pods on restart via liveness checks (the pattern `StateManager` already uses, promoted to fleet level). Daemon crash ≠ database outage. Postgres crash → process-compose restart policy; unrecoverable data dir → `reset`. + +### Changes inside `@supabase/stack` + +1. **Provision via template clone, not per-boot init:** `postgres-init` reduces to a no-op check; the fleet daemon hands `createStack` a pre-cloned data dir. (Per-boot init can never support `fork`; templates can.) +2. **Micro config profile:** the Postgres service gains the conf-overlay mechanism and an `enableExtension(name)` API implementing preload-on-enable. +3. **Version bump `17.6.1.081 → 17.6.1.143`; native-first hardening.** Roadmap flag (not this project): Edge Runtime is currently Docker-forced; native bundles for remaining services are the ask to service teams — Docker-only services undercut the density story on tiny machines. + +Untouched: Effect-based internals, BinaryResolver, health checks, key-translation proxy, log streaming. + +## Memory budget, measurement & testing + +### Methodology + +- **Memory = PSS** (not RSS), summed over the pod's process group: `/proc//smaps_rollup` on Linux, `phys_footprint` on macOS. RSS double-counts shared binary pages ~100× across the fleet; PSS attributes them fractionally, which is what actually fills the host. +- **CPU = idle wakeups + %core over 60s** per warm pod (`powermetrics` / `pidstat`). Idle CPU is what melts a laptop at 20 warm pods; Section "Background CPU" settings exist for this number. + +### Budget guardrails (per pod, to be validated by the harness) + +| State | Memory (PSS) | CPU idle | +|---|---|---| +| Suspended | 0 (manifest on disk only) | 0 | +| Warm, Postgres only | ≤ 30MB | < 0.5% core | +| Typical warm (PG + PostgREST + Auth) | ≤ 120MB | < 1% core | +| Max capacity (all services incl. Realtime + Functions) | ≤ 450MB | measured, documented | + +Host math: an 8GB machine holds 100+ registered pods with ~15–20 typical-warm or ~8–10 max-warm concurrent — past the density target. Realtime dominates the full-pod number (see Risks). + +### Benchmark harness + +A script in the CLI repo: provision N pods, wake a subset, drive synthetic traffic, sample PSS/CPU, assert the table above. Runs in CI so a version bump or config change that blows the envelope fails a build. + +### Tests + +- **Unit:** port registry, template cache keys (tuple hashing), idle-timer state machine, manifest round-trips. +- **Integration:** base/warm template builds; clone and `fork` divergence (write to fork, source untouched); preload-on-enable end-to-end; suspend/resume preserves committed data; wake-latency assertions. +- **Compatibility suite (the "no compromise on extensions" proof):** `CREATE EXTENSION` for every extension in `supabase/postgres 17.6.1.143` plus a smoke query each; real CLI migration/seed flows; round-trip fidelity (`pg_dump` from pod → restore into stock image, and vice versa); logical-replication smoke test (create slot, stream changes) standing in for Realtime until sidecars land. +- **Density E2E:** extend `parallelStacks.e2e` — 100 registered pods, wake 10 at random, assert envelope, port uniqueness, clean suspend. +- **Chaos:** `kill -9` fleet daemon → adoption on restart; kill Postgres → supervised restart; simulated host crash → verify documented `reset` recovery. + +## Risks & open questions + +1. **Nix darwin builds** of some extensions in the `supabase/postgres` set may not compile on `aarch64-darwin`. First implementation spike; fallback is Linux pods in a lightweight VM on macOS. +2. **Realtime memory per pod** (~150–300MB BEAM VM) dominates full pods. Mitigated now by lazy start; the structural fix is one shared multi-tenant Realtime per host — future project, not blocked by this design. +3. **Docker-only services** (Edge Runtime today) undercut density on tiny machines; native bundles are a roadmap ask to service teams. +4. **Windows:** Docker fallback only; explicitly out of the density story. From cc6a848f8eeed521ce3cbc8b8ff15708b2dc0253 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 10:16:17 +0200 Subject: [PATCH 02/38] docs: Windows supports supabase start via Docker path Co-Authored-By: Claude Fable 5 --- docs/specs/2026-07-07-micro-supabase-stacks-design.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/specs/2026-07-07-micro-supabase-stacks-design.md b/docs/specs/2026-07-07-micro-supabase-stacks-design.md index dcb690ff19..cb5d27aa4c 100644 --- a/docs/specs/2026-07-07-micro-supabase-stacks-design.md +++ b/docs/specs/2026-07-07-micro-supabase-stacks-design.md @@ -16,12 +16,13 @@ Run 100+ Supabase-compatible Postgres instances in parallel on small machines (8 - **Scope now:** Postgres and its orchestration. Sidecar services (PostgREST, Auth, Realtime, Edge Functions) are designed for but implemented later; their needs shape Postgres decisions today. - **Durability:** data is disposable/re-cloneable everywhere (`fsync=off` profile). Recovery from host-crash corruption is "re-clone from template." - **Density target:** 100+ registered instances on an 8–16GB host, most idle at any moment. +- **Windows:** `supabase start` must work, always via the Docker path (no native Windows service binaries). Windows gets functional parity, not the density optimizations — templates, suspend-on-idle, and the budget guardrails apply to macOS/Linux native pods. ## Non-goals - Production/paid-tier deployments. - Postgres major-version in-place upgrades (pods are disposable; reset onto a new base template). -- Windows native support (Docker fallback only). +- Windows *native* binaries and Windows density optimizations (`supabase start` still works on Windows via Docker; see Requirements). - Slimming the sidecar services themselves (tracked as follow-up; see Risks). ## Approach @@ -207,4 +208,4 @@ A script in the CLI repo: provision N pods, wake a subset, drive synthetic traff 1. **Nix darwin builds** of some extensions in the `supabase/postgres` set may not compile on `aarch64-darwin`. First implementation spike; fallback is Linux pods in a lightweight VM on macOS. 2. **Realtime memory per pod** (~150–300MB BEAM VM) dominates full pods. Mitigated now by lazy start; the structural fix is one shared multi-tenant Realtime per host — future project, not blocked by this design. 3. **Docker-only services** (Edge Runtime today) undercut density on tiny machines; native bundles are a roadmap ask to service teams. -4. **Windows:** Docker fallback only; explicitly out of the density story. +4. **Windows:** served by the Docker path for functional parity (`supabase start` works); explicitly out of the density story. The fleet daemon must degrade gracefully there — pods run as Docker containers without templates or suspend-on-idle. From c15afd469fef227625e56e79e74e99a2c41bc84b Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 10:59:28 +0200 Subject: [PATCH 03/38] docs: phase 1 implementation plan (stack changes + fleet daemon) Co-Authored-By: Claude Fable 5 --- ...-07-micro-stacks-phase1-stack-and-fleet.md | 2265 +++++++++++++++++ 1 file changed, 2265 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-07-micro-stacks-phase1-stack-and-fleet.md diff --git a/docs/superpowers/plans/2026-07-07-micro-stacks-phase1-stack-and-fleet.md b/docs/superpowers/plans/2026-07-07-micro-stacks-phase1-stack-and-fleet.md new file mode 100644 index 0000000000..67fc79ae70 --- /dev/null +++ b/docs/superpowers/plans/2026-07-07-micro-stacks-phase1-stack-and-fleet.md @@ -0,0 +1,2265 @@ +# Micro Stacks Phase 1 — `@supabase/stack` Changes + `@supabase/fleet` Daemon + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `@supabase/stack` pods template-provisioned, micro-tuned, and lazily started, and add a new `@supabase/fleet` package: a host-level daemon giving 100+ registered pods with suspend-on-idle, wake-on-connect, and CoW fork/reset. + +**Architecture:** `@supabase/stack` stays the pod runtime (supervise processes for one stack); `@supabase/fleet` is a new package owning what exists when pods don't: pod/port registries, template store, CoW provisioning, a TCP wake-proxy edge, and idle timers. Fleet hosts warm pods as in-process `StackHandle`s. Spec: `docs/specs/2026-07-07-micro-supabase-stacks-design.md` (in this repo). + +**Tech Stack:** TypeScript on Bun, Effect 4 (beta, via pnpm `catalog:`), vitest, nx, `node:net` for TCP splicing (works on Bun and Node), `node:crypto` for hashing. + +## Global Constraints + +- Work in `/Users/jgoux/Code/supabase/cli/.claude/worktrees/micro-supabase-stacks-spec`, branch `claude/nifty-dhawan-86e0c6`'s sibling `claude/micro-supabase-stacks-spec`. Never commit to `develop`. +- `@supabase/stack` is unpublished — breaking API changes are allowed. +- All new deps via `catalog:` in `pnpm-workspace.yaml`; run `pnpm install` after editing any package.json. +- Test naming: `*.unit.test.ts` (no I/O), `*.integration.test.ts` (may spawn postgres), `*.e2e.test.ts` (full stacks, serial). Runner: vitest via nx (`pnpm vitest run ` works directly inside a package). +- Effect style: `Context.Service` classes, `Layer`, `Effect.gen`; public APIs are Promise-based wrappers via `ManagedRuntime` (mirror `createStack.ts:694-763`). +- Postgres micro profile values are normative from the spec — copy exactly (note the real GUC name is `wal_writer_delay`). +- On-disk layout: `~/.supabase/templates//data`, `~/.supabase/pods//{data,pod.json,logs}`, `~/.supabase/fleet-state.json` (tests must override the root via options — never touch the real `~/.supabase` in tests; use `mkdtemp`). +- Commit after every task with a conventional-commit message. +- Windows: fleet features are macOS/Linux native-mode only in this phase; `mode: "docker"` paths must keep compiling and existing tests must keep passing. + +--- + +### Task 1: Scaffold `@supabase/fleet` package + +**Files:** +- Create: `packages/fleet/package.json` +- Create: `packages/fleet/tsconfig.json` +- Create: `packages/fleet/src/index.ts` +- Test: `packages/fleet/src/index.unit.test.ts` + +**Interfaces:** +- Produces: an empty package `@supabase/fleet` that later tasks fill; exports nothing yet but `FLEET_PACKAGE` marker for the scaffold test. + +- [ ] **Step 1: Write package.json** (mirror `packages/cli-test-helpers/package.json` conventions) + +```json +{ + "name": "@supabase/fleet", + "version": "0.1.0", + "private": true, + "type": "module", + "exports": { + ".": "./src/index.ts" + }, + "scripts": { + "test": "nx run-many -t test:core --projects=$npm_package_name", + "check:all": "nx run-many -t types:check lint:check fmt:check knip:check --projects=$npm_package_name", + "fix:all": "nx run-many -t lint:fix fmt:fix knip:fix --projects=$npm_package_name" + }, + "dependencies": { + "@supabase/process-compose": "workspace:*", + "@supabase/stack": "workspace:*", + "effect": "catalog:" + }, + "devDependencies": { + "@tsconfig/bun": "catalog:", + "@types/bun": "catalog:", + "@vitest/coverage-istanbul": "catalog:", + "knip": "catalog:", + "oxfmt": "catalog:", + "oxlint": "catalog:", + "vitest": "catalog:" + }, + "knip": { + "entry": ["src/**/*.test.ts"], + "ignoreDependencies": ["@typescript/native-preview", "oxfmt", "oxlint", "oxlint-tsgolint"], + "ignoreBinaries": ["nx"] + } +} +``` + +- [ ] **Step 2: Write tsconfig.json** + +```json +{ + "extends": "@tsconfig/bun/tsconfig.json", + "compilerOptions": { + "lib": ["ESNext", "DOM"], + "types": ["bun"] + } +} +``` + +- [ ] **Step 3: Write src/index.ts and a smoke test** + +```typescript +// src/index.ts +export const FLEET_PACKAGE = "@supabase/fleet"; +``` + +```typescript +// src/index.unit.test.ts +import { describe, expect, it } from "vitest"; +import { FLEET_PACKAGE } from "./index.ts"; + +describe("fleet scaffold", () => { + it("exports the package marker", () => { + expect(FLEET_PACKAGE).toBe("@supabase/fleet"); + }); +}); +``` + +- [ ] **Step 4: Install and run** + +Run: `pnpm install && cd packages/fleet && pnpm vitest run src/index.unit.test.ts` +Expected: 1 passed. + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet pnpm-lock.yaml +git commit -m "feat(fleet): scaffold @supabase/fleet package" +``` + +--- + +### Task 2: Micro profile module in `@supabase/stack` + +**Files:** +- Create: `packages/stack/src/micro.ts` +- Test: `packages/stack/src/micro.unit.test.ts` + +**Interfaces:** +- Produces: + - `MICRO_POSTGRES_SETTINGS: ReadonlyArray` + - `buildMicroConf(): string` — full `micro.conf` file content + - `PRELOAD_REQUIRED_EXTENSIONS: ReadonlySet` + - `buildPodConf(preloadLibraries: ReadonlyArray): string` + +- [ ] **Step 1: Write the failing test** + +```typescript +// src/micro.unit.test.ts +import { describe, expect, it } from "vitest"; +import { + buildMicroConf, + buildPodConf, + MICRO_POSTGRES_SETTINGS, + PRELOAD_REQUIRED_EXTENSIONS, +} from "./micro.ts"; + +describe("micro profile", () => { + it("contains the normative spec settings", () => { + const map = new Map(MICRO_POSTGRES_SETTINGS); + expect(map.get("shared_buffers")).toBe("16MB"); + expect(map.get("jit")).toBe("off"); + expect(map.get("fsync")).toBe("off"); + expect(map.get("wal_level")).toBe("logical"); + expect(map.get("max_slot_wal_keep_size")).toBe("256MB"); + expect(map.get("wal_writer_delay")).toBe("10s"); + }); + + it("renders micro.conf as key = 'value' lines", () => { + const conf = buildMicroConf(); + expect(conf).toContain("shared_buffers = '16MB'"); + expect(conf).toContain("max_connections = '40'"); + expect(conf.endsWith("\n")).toBe(true); + }); + + it("knows which extensions need preload", () => { + expect(PRELOAD_REQUIRED_EXTENSIONS.has("pg_cron")).toBe(true); + expect(PRELOAD_REQUIRED_EXTENSIONS.has("pgvector")).toBe(false); + }); + + it("renders pod.conf with shared_preload_libraries", () => { + expect(buildPodConf(["pg_cron", "pg_net"])).toBe( + "shared_preload_libraries = 'pg_cron,pg_net'\n", + ); + expect(buildPodConf([])).toBe("shared_preload_libraries = ''\n"); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/stack && pnpm vitest run src/micro.unit.test.ts` +Expected: FAIL — cannot resolve `./micro.ts`. + +- [ ] **Step 3: Implement** + +```typescript +// src/micro.ts +/** + * Micro profile: normative values from + * docs/specs/2026-07-07-micro-supabase-stacks-design.md ("The micro.conf profile"). + */ +export const MICRO_POSTGRES_SETTINGS: ReadonlyArray = [ + // memory + ["shared_buffers", "16MB"], + ["work_mem", "4MB"], + ["maintenance_work_mem", "32MB"], + ["jit", "off"], + ["huge_pages", "off"], + ["max_connections", "40"], + // background CPU + ["autovacuum_naptime", "5min"], + ["autovacuum_max_workers", "1"], + ["bgwriter_lru_maxpages", "0"], + ["wal_writer_delay", "10s"], + ["checkpoint_timeout", "30min"], + ["max_parallel_workers", "0"], + ["max_parallel_workers_per_gather", "0"], + ["max_worker_processes", "4"], + ["track_io_timing", "off"], + // durability (disposable profile) + ["fsync", "off"], + ["synchronous_commit", "off"], + ["full_page_writes", "off"], + // replication reservations + ["wal_level", "logical"], + ["max_wal_senders", "5"], + ["max_replication_slots", "5"], + ["max_slot_wal_keep_size", "256MB"], + ["wal_keep_size", "0"], +]; + +export const buildMicroConf = (): string => + `${MICRO_POSTGRES_SETTINGS.map(([k, v]) => `${k} = '${v}'`).join("\n")}\n`; + +/** Extensions that only work when named in shared_preload_libraries. */ +export const PRELOAD_REQUIRED_EXTENSIONS: ReadonlySet = new Set([ + "pg_cron", + "pg_net", + "timescaledb", + "pg_stat_statements", + "auto_explain", + "pgaudit", + "plan_filter", + "supautils", + "pgsodium", +]); + +export const buildPodConf = (preloadLibraries: ReadonlyArray): string => + `shared_preload_libraries = '${preloadLibraries.join(",")}'\n`; +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/stack && pnpm vitest run src/micro.unit.test.ts` +Expected: 4 passed. + +- [ ] **Step 5: Commit** + +```bash +git add packages/stack/src/micro.ts packages/stack/src/micro.unit.test.ts +git commit -m "feat(stack): add micro postgres profile and preload-required registry" +``` + +--- + +### Task 3: PGDATA conf-layering utilities + +**Files:** +- Create: `packages/stack/src/pgconf.ts` +- Test: `packages/stack/src/pgconf.unit.test.ts` + +**Interfaces:** +- Consumes: `buildMicroConf`, `buildPodConf` from Task 2. +- Produces (all plain async, `node:fs/promises` — used by both stack and fleet): + - `installMicroProfile(pgdata: string): Promise` — writes `micro.conf` + empty `pod.conf` into PGDATA and appends include lines to `postgresql.conf` once (idempotent) + - `readPreloadLibraries(pgdata: string): Promise` + - `writePreloadLibraries(pgdata: string, libs: ReadonlyArray): Promise` + +- [ ] **Step 1: Write the failing test** + +```typescript +// src/pgconf.unit.test.ts +import { mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { + installMicroProfile, + readPreloadLibraries, + writePreloadLibraries, +} from "./pgconf.ts"; + +async function fakePgdata(): Promise { + const dir = await mkdtemp(join(tmpdir(), "pgconf-test-")); + await writeFile(join(dir, "postgresql.conf"), "# stock conf\nport = 5432\n"); + return dir; +} + +describe("pgconf", () => { + it("installs micro.conf, pod.conf, and include lines idempotently", async () => { + const pgdata = await fakePgdata(); + await installMicroProfile(pgdata); + await installMicroProfile(pgdata); // idempotent + const main = await readFile(join(pgdata, "postgresql.conf"), "utf8"); + expect(main.match(/include_if_exists = 'micro\.conf'/g)).toHaveLength(1); + expect(main.match(/include_if_exists = 'pod\.conf'/g)).toHaveLength(1); + // pod.conf must be included AFTER micro.conf so pod overrides micro + expect(main.indexOf("micro.conf")).toBeLessThan(main.indexOf("pod.conf")); + const micro = await readFile(join(pgdata, "micro.conf"), "utf8"); + expect(micro).toContain("shared_buffers = '16MB'"); + }); + + it("round-trips preload libraries via pod.conf", async () => { + const pgdata = await fakePgdata(); + await installMicroProfile(pgdata); + expect(await readPreloadLibraries(pgdata)).toEqual([]); + await writePreloadLibraries(pgdata, ["pg_cron"]); + expect(await readPreloadLibraries(pgdata)).toEqual(["pg_cron"]); + await writePreloadLibraries(pgdata, ["pg_cron", "pg_net"]); + expect(await readPreloadLibraries(pgdata)).toEqual(["pg_cron", "pg_net"]); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/stack && pnpm vitest run src/pgconf.unit.test.ts` +Expected: FAIL — cannot resolve `./pgconf.ts`. + +- [ ] **Step 3: Implement** + +```typescript +// src/pgconf.ts +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { buildMicroConf, buildPodConf } from "./micro.ts"; + +const INCLUDE_BLOCK = [ + "", + "# --- supabase micro profile (managed; do not edit below) ---", + "include_if_exists = 'micro.conf'", + "include_if_exists = 'pod.conf'", + "", +].join("\n"); + +export async function installMicroProfile(pgdata: string): Promise { + await writeFile(join(pgdata, "micro.conf"), buildMicroConf()); + const podConf = join(pgdata, "pod.conf"); + const existing = await readFile(podConf, "utf8").catch(() => undefined); + if (existing === undefined) { + await writeFile(podConf, buildPodConf([])); + } + const mainPath = join(pgdata, "postgresql.conf"); + const main = await readFile(mainPath, "utf8"); + if (!main.includes("include_if_exists = 'micro.conf'")) { + await writeFile(mainPath, main + INCLUDE_BLOCK); + } +} + +export async function readPreloadLibraries(pgdata: string): Promise { + const content = await readFile(join(pgdata, "pod.conf"), "utf8").catch(() => ""); + const match = content.match(/^shared_preload_libraries = '([^']*)'/m); + if (!match || match[1] === "") return []; + return match[1].split(","); +} + +export async function writePreloadLibraries( + pgdata: string, + libs: ReadonlyArray, +): Promise { + await writeFile(join(pgdata, "pod.conf"), buildPodConf(libs)); +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/stack && pnpm vitest run src/pgconf.unit.test.ts` +Expected: 2 passed. + +- [ ] **Step 5: Commit** + +```bash +git add packages/stack/src/pgconf.ts packages/stack/src/pgconf.unit.test.ts +git commit -m "feat(stack): PGDATA conf layering (micro.conf + pod.conf includes)" +``` + +--- + +### Task 4: `provisioned` + `profile` on PostgresConfig; skip init; version bump + +**Files:** +- Modify: `packages/stack/src/StackBuilder.ts` (PostgresConfig ~line 42-169; service assembly ~line 556-870) +- Modify: `packages/stack/src/services/postgres.ts` (NATIVE_POSTGRES_RUNTIME_ARGS, lines 44-51) +- Modify: `packages/stack/src/versions.ts` (DEFAULT_VERSIONS, line ~49) +- Test: `packages/stack/src/StackBuilder.provisioned.unit.test.ts` + +**Interfaces:** +- Consumes: nothing new. +- Produces: + - `PostgresConfig` gains `readonly provisioned?: boolean` (data dir is a pre-initialized template clone → **exclude the `postgres-init` service entirely**) and `readonly profile?: "default" | "micro"`. + - When `profile: "micro"`, the native postgres ServiceDef gets NO `-c` runtime args (micro.conf in PGDATA carries them; command-line `-c` would override users' `ALTER SYSTEM`). When `profile` is absent/default, current behavior is unchanged. + - `DEFAULT_VERSIONS.postgres` becomes `"17.6.1.143"`. + +- [ ] **Step 1: Write the failing test** + +Find the existing StackBuilder unit tests for reference on constructing a `ResolvedStackConfig`/builder in isolation (see `packages/stack/tests/` and any `StackBuilder*.test.ts`; follow the same fixture helpers). The test asserts on the built ServiceDef list: + +```typescript +// src/StackBuilder.provisioned.unit.test.ts +// NOTE: reuse the existing StackBuilder test fixture pattern in this package for +// constructing a builder; the assertions below are the contract. +import { describe, expect, it } from "vitest"; +import { buildServicesForTest } from "../tests/helpers/buildServices.ts"; // create if absent, wrapping StackBuilder.build() with a minimal ResolvedStackConfig fixture + +describe("provisioned postgres", () => { + it("excludes postgres-init when postgres.provisioned is true", async () => { + const services = await buildServicesForTest({ postgres: { provisioned: true } }); + expect(services.map((s) => s.name)).not.toContain("postgres-init"); + }); + + it("includes postgres-init by default", async () => { + const services = await buildServicesForTest({}); + expect(services.map((s) => s.name)).toContain("postgres-init"); + }); + + it("drops -c runtime args when profile is micro", async () => { + const services = await buildServicesForTest({ + postgres: { provisioned: true, profile: "micro" }, + }); + const pg = services.find((s) => s.name === "postgres"); + expect(pg?.args?.join(" ")).not.toContain("wal_level=logical"); + }); + + it("keeps -c runtime args on the default profile", async () => { + const services = await buildServicesForTest({}); + const pg = services.find((s) => s.name === "postgres"); + expect(pg?.args?.join(" ")).toContain("wal_level=logical"); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/stack && pnpm vitest run src/StackBuilder.provisioned.unit.test.ts` +Expected: FAIL — `provisioned` not a known property / init service always present. + +- [ ] **Step 3: Implement** + +In `StackBuilder.ts`, extend the config interface: + +```typescript +export interface PostgresConfig { + readonly port?: number; + readonly dataDir?: string; + readonly version?: string; + readonly autoExposeNewTables?: boolean; + /** Data dir is a pre-initialized template clone; skip the postgres-init service. */ + readonly provisioned?: boolean; + /** "micro": settings come from micro.conf/pod.conf inside PGDATA, not -c args. */ + readonly profile?: "default" | "micro"; +} +``` + +In the service-assembly section, wrap the `postgres-init` push: + +```typescript +if (config.postgres?.provisioned !== true) { + services.push({ ...makePostgresInitService(postgresInitOpts), enabled: true }); +} +``` + +In `services/postgres.ts`, thread the profile through `NativePostgresOptions` and select args: + +```typescript +export interface NativePostgresOptions { + // ...existing fields... + readonly profile?: "default" | "micro"; +} + +const runtimeArgs = (profile?: "default" | "micro"): readonly string[] => + profile === "micro" ? [] : NATIVE_POSTGRES_RUNTIME_ARGS; + +// in makePostgresService: +args: [initScript, "-p", String(opts.port), ...runtimeArgs(opts.profile)], +``` + +In `versions.ts`: `postgres: "17.6.1.143",`. + +- [ ] **Step 4: Run to verify pass, plus existing suites** + +Run: `cd packages/stack && pnpm vitest run src/StackBuilder.provisioned.unit.test.ts && pnpm vitest run --exclude '**/*.e2e.test.ts'` +Expected: new tests pass; no regressions in unit/integration suites. + +- [ ] **Step 5: Commit** + +```bash +git add packages/stack/src packages/stack/tests +git commit -m "feat(stack): provisioned data dirs, micro profile wiring, postgres 17.6.1.143" +``` + +--- + +### Task 5: `enableExtension` on the coordinator and StackHandle + +**Files:** +- Modify: `packages/stack/src/StackLifecycleCoordinator.ts` (service methods, ~lines 131-171 and 518-564) +- Modify: `packages/stack/src/createStack.ts` (StackHandle interface ~line 90-111 and Promise wiring ~line 694-763) +- Modify: `packages/stack/src/index.ts` (export new types) +- Test: `packages/stack/src/enableExtension.unit.test.ts` + +**Interfaces:** +- Consumes: `PRELOAD_REQUIRED_EXTENSIONS` (Task 2), `readPreloadLibraries`/`writePreloadLibraries` (Task 3), coordinator `restartService` (existing). +- Produces: + - Coordinator: `readonly enableExtension: (name: string) => Effect.Effect` + - `StackHandle.enableExtension(name: string): Promise` + - Behavior: if `name` is not in `PRELOAD_REQUIRED_EXTENSIONS` → no-op (plain `CREATE EXTENSION` works). If already in pod.conf → no-op. Else append to pod.conf and `restartService("postgres")`. + +- [ ] **Step 1: Write the failing test** + +Test the pure decision logic separately from the Effect plumbing so it runs without postgres: + +```typescript +// src/enableExtension.unit.test.ts +import { describe, expect, it } from "vitest"; +import { planEnableExtension } from "./enableExtension.ts"; + +describe("planEnableExtension", () => { + it("no-ops for extensions that do not need preload", () => { + expect(planEnableExtension("pgvector", [])).toEqual({ action: "none" }); + }); + it("no-ops when already preloaded", () => { + expect(planEnableExtension("pg_cron", ["pg_cron"])).toEqual({ action: "none" }); + }); + it("appends and restarts otherwise", () => { + expect(planEnableExtension("pg_cron", ["pg_net"])).toEqual({ + action: "restart", + libraries: ["pg_net", "pg_cron"], + }); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/stack && pnpm vitest run src/enableExtension.unit.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +```typescript +// src/enableExtension.ts +import { PRELOAD_REQUIRED_EXTENSIONS } from "./micro.ts"; + +export type EnableExtensionPlan = + | { readonly action: "none" } + | { readonly action: "restart"; readonly libraries: ReadonlyArray }; + +export const planEnableExtension = ( + name: string, + currentLibraries: ReadonlyArray, +): EnableExtensionPlan => { + if (!PRELOAD_REQUIRED_EXTENSIONS.has(name)) return { action: "none" }; + if (currentLibraries.includes(name)) return { action: "none" }; + return { action: "restart", libraries: [...currentLibraries, name] }; +}; +``` + +In `StackLifecycleCoordinator.ts`, add to the service interface and implementation (the coordinator already knows the resolved postgres `dataDir` from config): + +```typescript +enableExtension: (name: string) => + Effect.gen(function* () { + const libs = yield* Effect.promise(() => readPreloadLibraries(pgDataDir)); + const plan = planEnableExtension(name, libs); + if (plan.action === "none") return; + yield* Effect.promise(() => writePreloadLibraries(pgDataDir, plan.libraries)); + yield* restartServiceImpl("postgres"); + }), +``` + +In `createStack.ts`, add to `StackHandle` and the wiring: + +```typescript +enableExtension(name: string): Promise; +// ... +enableExtension: (name) => run(localStack.enableExtension(name)), +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/stack && pnpm vitest run src/enableExtension.unit.test.ts && pnpm vitest run --exclude '**/*.e2e.test.ts'` +Expected: PASS, no regressions. + +- [ ] **Step 5: Commit** + +```bash +git add packages/stack/src +git commit -m "feat(stack): enableExtension with preload-on-enable restart" +``` + +--- + +### Task 6: Lazy service start behind the ApiProxy + +**Files:** +- Modify: `packages/stack/src/StackBuilder.ts` (StackConfig + service `enabled` flags) +- Modify: `packages/stack/src/ApiProxy.ts` (ProxyHandlerOptions + makeProxyHandler, lines ~13-202; route table ~218-365) +- Test: `packages/stack/src/lazyServices.unit.test.ts` + +**Interfaces:** +- Consumes: coordinator `startService(name)` (existing), process-compose `ServiceDef.enabled` (existing). +- Produces: + - `StackConfig` gains `readonly lazyServices?: boolean` (default false → existing behavior). + - When true: every service except `postgres` (and `postgres-init` when present) is built with `enabled: false`; ApiProxy's `ProxyConfig` gains `readonly ensureService?: (name: ServiceName) => Promise`; each route entry declares its owning `service: ServiceName`; the handler awaits `ensureService(service)` before forwarding (idempotent, resolves immediately if running). + - `createStack` wires `ensureService` to `startService` + `serviceReady`, memoizing in-flight starts so concurrent first requests trigger one start. + +- [ ] **Step 1: Write the failing test** (route-level behavior with a fake ensureService) + +```typescript +// src/lazyServices.unit.test.ts +import { describe, expect, it } from "vitest"; +import { makeEnsureServiceMemo } from "./lazyServices.ts"; + +describe("makeEnsureServiceMemo", () => { + it("starts a service once across concurrent calls", async () => { + let starts = 0; + const ensure = makeEnsureServiceMemo(async (name) => { + starts += 1; + await new Promise((r) => setTimeout(r, 10)); + }); + await Promise.all([ensure("realtime"), ensure("realtime"), ensure("realtime")]); + expect(starts).toBe(1); + }); + + it("retries after a failed start", async () => { + let attempt = 0; + const ensure = makeEnsureServiceMemo(async () => { + attempt += 1; + if (attempt === 1) throw new Error("boom"); + }); + await expect(ensure("auth")).rejects.toThrow("boom"); + await ensure("auth"); // second attempt allowed + expect(attempt).toBe(2); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/stack && pnpm vitest run src/lazyServices.unit.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +```typescript +// src/lazyServices.ts +import type { ServiceName } from "./versions.ts"; + +/** Memoize in-flight starts; clear on failure so the next request retries. */ +export const makeEnsureServiceMemo = ( + start: (name: ServiceName) => Promise, +): ((name: ServiceName) => Promise) => { + const inFlight = new Map>(); + const done = new Set(); + return (name) => { + if (done.has(name)) return Promise.resolve(); + const existing = inFlight.get(name); + if (existing) return existing; + const p = start(name).then( + () => { + done.add(name); + inFlight.delete(name); + }, + (err: unknown) => { + inFlight.delete(name); + throw err; + }, + ); + inFlight.set(name, p); + return p; + }; +}; +``` + +In `StackBuilder.ts`: add `readonly lazyServices?: boolean` to `StackConfig`; where each non-postgres service is pushed, compute `enabled: config.lazyServices === true ? false : `. + +In `ApiProxy.ts`: add to `ProxyConfig` `readonly ensureService?: (name: ServiceName) => Promise`; extend `ProxyHandlerOptions` with `readonly service: ServiceName`; annotate every route in the table with its service (`/auth/v1` → `"auth"`, `/rest/v1` → `"postgrest"`, `/functions/v1` → `"edge-runtime"`, `/realtime/v1` → `"realtime"`, `/storage/v1` → `"storage"`, `/pg` → `"pgmeta"`, etc. — follow the routing table); at the top of `makeProxyHandler`'s returned effect: + +```typescript +if (config.ensureService) { + yield* Effect.promise(() => config.ensureService!(opts.service)); +} +``` + +In `createStack.ts`, when `resolved.lazyServices`, build the memo over `startService` + `waitReady` and pass it into the ApiProxy layer's `ProxyConfig`. + +- [ ] **Step 4: Run to verify pass + no regressions** + +Run: `cd packages/stack && pnpm vitest run src/lazyServices.unit.test.ts && pnpm vitest run --exclude '**/*.e2e.test.ts'` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add packages/stack/src +git commit -m "feat(stack): lazy per-service start behind the API proxy" +``` + +--- + +### Task 7: Fleet manifest types + template cache key + +**Files:** +- Create: `packages/fleet/src/PodManifest.ts` +- Test: `packages/fleet/src/PodManifest.unit.test.ts` + +**Interfaces:** +- Consumes: `VersionManifest`, `ServiceName` types from `@supabase/stack`. +- Produces: + - `interface PodManifest { readonly id: string; readonly versions: Partial; readonly services: Partial>; readonly flags: { readonly supautils: boolean }; readonly ports: { readonly dbPort: number; readonly apiPort: number }; readonly createdAt: string; }` + - `templateKey(versions: Partial): string` — stable sha256-based key: same versions in any order → same key; different versions → different key. + - `baseTemplateKey(postgresVersion: string): string` → `"pg-" + postgresVersion`. + +- [ ] **Step 1: Write the failing test** + +```typescript +// src/PodManifest.unit.test.ts +import { describe, expect, it } from "vitest"; +import { baseTemplateKey, templateKey } from "./PodManifest.ts"; + +describe("templateKey", () => { + it("is stable across key order", () => { + expect(templateKey({ postgres: "17.6.1.143", auth: "2.192.0" })).toBe( + templateKey({ auth: "2.192.0", postgres: "17.6.1.143" }), + ); + }); + it("changes when any version changes", () => { + expect(templateKey({ postgres: "17.6.1.143", auth: "2.192.0" })).not.toBe( + templateKey({ postgres: "17.6.1.143", auth: "2.192.1" }), + ); + }); + it("base key is human-readable", () => { + expect(baseTemplateKey("17.6.1.143")).toBe("pg-17.6.1.143"); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/fleet && pnpm vitest run src/PodManifest.unit.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +```typescript +// src/PodManifest.ts +import { createHash } from "node:crypto"; +import type { ServiceName, VersionManifest } from "@supabase/stack"; + +export interface PodManifest { + readonly id: string; + readonly versions: Partial; + readonly services: Partial>; + readonly flags: { readonly supautils: boolean }; + readonly ports: { readonly dbPort: number; readonly apiPort: number }; + readonly createdAt: string; +} + +export const baseTemplateKey = (postgresVersion: string): string => + `pg-${postgresVersion}`; + +export const templateKey = (versions: Partial): string => { + const canonical = JSON.stringify( + Object.fromEntries(Object.entries(versions).sort(([a], [b]) => a.localeCompare(b))), + ); + return `tuple-${createHash("sha256").update(canonical).digest("hex").slice(0, 16)}`; +}; +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/fleet && pnpm vitest run src/PodManifest.unit.test.ts` +Expected: 3 passed. + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet/src +git commit -m "feat(fleet): pod manifest types and template cache keys" +``` + +--- + +### Task 8: Copy-on-write directory clone + +**Files:** +- Create: `packages/fleet/src/cowClone.ts` +- Test: `packages/fleet/src/cowClone.integration.test.ts` (touches the filesystem) + +**Interfaces:** +- Produces: `cloneDir(src: string, dest: string): Promise` — APFS `cp -Rc` on darwin, `cp -R --reflink=auto` on linux, recursive copy fallback; `dest` must not exist; preserves file modes (postgres requires `0700` on PGDATA). + +- [ ] **Step 1: Write the failing test** + +```typescript +// src/cowClone.integration.test.ts +import { mkdtemp, mkdir, readFile, stat, writeFile, chmod } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { cloneDir } from "./cowClone.ts"; + +describe("cloneDir", () => { + it("clones a tree with content and modes; clones diverge", async () => { + const root = await mkdtemp(join(tmpdir(), "cow-test-")); + const src = join(root, "src"); + await mkdir(join(src, "sub"), { recursive: true }); + await writeFile(join(src, "sub", "a.txt"), "hello"); + await chmod(src, 0o700); + + const dest = join(root, "dest"); + await cloneDir(src, dest); + + expect(await readFile(join(dest, "sub", "a.txt"), "utf8")).toBe("hello"); + expect(((await stat(dest)).mode & 0o777)).toBe(0o700); + + await writeFile(join(dest, "sub", "a.txt"), "changed"); + expect(await readFile(join(src, "sub", "a.txt"), "utf8")).toBe("hello"); + }); + + it("refuses to clone onto an existing destination", async () => { + const root = await mkdtemp(join(tmpdir(), "cow-test-")); + const src = join(root, "src"); + await mkdir(src); + const dest = join(root, "dest"); + await mkdir(dest); + await expect(cloneDir(src, dest)).rejects.toThrow(/exists/); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/fleet && pnpm vitest run src/cowClone.integration.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +```typescript +// src/cowClone.ts +import { spawn } from "node:child_process"; +import { cp, stat } from "node:fs/promises"; + +const run = (cmd: string, args: string[]): Promise => + new Promise((resolve, reject) => { + const child = spawn(cmd, args, { stdio: "ignore" }); + child.on("error", reject); + child.on("exit", (code) => resolve(code ?? 1)); + }); + +/** Copy-on-write directory clone: APFS clonefile on macOS, reflink on Linux, plain copy fallback. */ +export async function cloneDir(src: string, dest: string): Promise { + const exists = await stat(dest).then(() => true, () => false); + if (exists) throw new Error(`cloneDir: destination already exists: ${dest}`); + + if (process.platform === "darwin") { + if ((await run("cp", ["-Rc", src, dest])) === 0) return; + } else if (process.platform === "linux") { + if ((await run("cp", ["-R", "--reflink=auto", src, dest])) === 0) return; + } + // Fallback (non-CoW filesystems, other platforms): plain recursive copy. + await cp(src, dest, { recursive: true, force: false, errorOnExist: true }); +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/fleet && pnpm vitest run src/cowClone.integration.test.ts` +Expected: 2 passed. + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet/src +git commit -m "feat(fleet): copy-on-write directory clone (clonefile/reflink/copy)" +``` + +--- + +### Task 9: Persistent PortRegistry + +**Files:** +- Create: `packages/fleet/src/PortRegistry.ts` +- Test: `packages/fleet/src/PortRegistry.unit.test.ts` + +**Interfaces:** +- Produces a class (plain TS, JSON persistence, no Effect needed at this layer): + - `PortRegistry.load(stateFile: string): Promise` + - `allocate(podId: string): Promise<{ dbPort: number; apiPort: number }>` — deterministic scan from a base (default 55000), skipping assigned ports; persists after each allocation (atomic write: tmp file + rename) + - `release(podId: string): Promise` + - `get(podId: string): { dbPort: number; apiPort: number } | undefined` +- This replaces `readReservedPorts()`'s cross-stack filesystem scan for fleet-managed pods: the registry is the single owner of the port space. + +- [ ] **Step 1: Write the failing test** + +```typescript +// src/PortRegistry.unit.test.ts +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { PortRegistry } from "./PortRegistry.ts"; + +describe("PortRegistry", () => { + it("allocates unique port pairs and persists them", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + const a = await reg.allocate("pod-a"); + const b = await reg.allocate("pod-b"); + expect(new Set([a.dbPort, a.apiPort, b.dbPort, b.apiPort]).size).toBe(4); + + const reloaded = await PortRegistry.load(file); + expect(reloaded.get("pod-a")).toEqual(a); + expect(reloaded.get("pod-b")).toEqual(b); + }); + + it("is idempotent per pod and reuses released ports", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + const a1 = await reg.allocate("pod-a"); + const a2 = await reg.allocate("pod-a"); + expect(a2).toEqual(a1); + await reg.release("pod-a"); + expect(reg.get("pod-a")).toBeUndefined(); + const c = await reg.allocate("pod-c"); + expect(c.dbPort).toBe(a1.dbPort); // freed ports are reusable + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/fleet && pnpm vitest run src/PortRegistry.unit.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +```typescript +// src/PortRegistry.ts +import { mkdir, readFile, rename, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; + +export interface PodPorts { + readonly dbPort: number; + readonly apiPort: number; +} + +interface PortState { + readonly basePort: number; + readonly pods: Record; +} + +const DEFAULT_BASE_PORT = 55000; + +export class PortRegistry { + private constructor( + private readonly stateFile: string, + private state: PortState, + ) {} + + static async load(stateFile: string): Promise { + const raw = await readFile(stateFile, "utf8").catch(() => undefined); + const state: PortState = + raw !== undefined + ? (JSON.parse(raw) as PortState) + : { basePort: DEFAULT_BASE_PORT, pods: {} }; + return new PortRegistry(stateFile, state); + } + + get(podId: string): PodPorts | undefined { + return this.state.pods[podId]; + } + + async allocate(podId: string): Promise { + const existing = this.state.pods[podId]; + if (existing) return existing; + const used = new Set( + Object.values(this.state.pods).flatMap((p) => [p.dbPort, p.apiPort]), + ); + let candidate = this.state.basePort; + const next = (): number => { + while (used.has(candidate)) candidate += 1; + used.add(candidate); + return candidate; + }; + const ports: PodPorts = { dbPort: next(), apiPort: next() }; + this.state = { ...this.state, pods: { ...this.state.pods, [podId]: ports } }; + await this.persist(); + return ports; + } + + async release(podId: string): Promise { + const { [podId]: _, ...rest } = this.state.pods; + this.state = { ...this.state, pods: rest }; + await this.persist(); + } + + private async persist(): Promise { + await mkdir(dirname(this.stateFile), { recursive: true }); + const tmp = `${this.stateFile}.tmp`; + await writeFile(tmp, JSON.stringify(this.state, null, 2)); + await rename(tmp, this.stateFile); + } +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/fleet && pnpm vitest run src/PortRegistry.unit.test.ts` +Expected: 2 passed. + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet/src +git commit -m "feat(fleet): persistent port registry" +``` + +--- + +### Task 10: TemplateStore — base and warm templates built via the stack itself + +**Files:** +- Create: `packages/fleet/src/TemplateStore.ts` +- Test: `packages/fleet/src/TemplateStore.integration.test.ts` (spawns real postgres — needs the binary cache; skip in CI environments without it via `describe.skipIf(!process.env.FLEET_PG_TESTS)`) + +**Interfaces:** +- Consumes: `createStack` from `@supabase/stack` (bun entry), `installMicroProfile` from `@supabase/stack` (export it from stack's index in this task), `cloneDir` (Task 8), `baseTemplateKey`/`templateKey` (Task 7). +- Produces: + - `class TemplateStore` with: + - `constructor(root: string)` — `root` = templates dir (e.g. `~/.supabase/templates`) + - `ensureBaseTemplate(postgresVersion: string): Promise` — returns template data-dir path. If absent: run a one-shot stack (`postgres` only, non-provisioned, so `postgres-init` applies baseline migrations), stop it, run `installMicroProfile(dataDir)`, move into `root/pg-/data`, write `template.json` marker. + - `ensureWarmTemplate(versions: Partial, enabledServices: ReadonlyArray): Promise` — clone base, boot the listed services once (services self-migrate), stop, freeze under `root//data`. Falls back to base when `enabledServices` is empty. + - `has(key: string): Promise` +- Build must be concurrency-safe on one host: take a lockfile (`root/.lock` created with `wx` flag; poll-wait if held; stale after 10min). + +- [ ] **Step 1: Write the failing test** + +```typescript +// src/TemplateStore.integration.test.ts +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { TemplateStore } from "./TemplateStore.ts"; + +// Requires postgres binaries in the local cache; opt-in via env. +describe.skipIf(!process.env.FLEET_PG_TESTS)("TemplateStore", () => { + it("builds a base template once and reuses it", async () => { + const root = await mkdtemp(join(tmpdir(), "templates-")); + const store = new TemplateStore(root); + const first = await store.ensureBaseTemplate("17.6.1.143"); + expect(first).toContain("pg-17.6.1.143"); + // PGDATA got the micro profile + const { readFile } = await import("node:fs/promises"); + const conf = await readFile(join(first, "postgresql.conf"), "utf8"); + expect(conf).toContain("include_if_exists = 'micro.conf'"); + + const started = Date.now(); + const second = await store.ensureBaseTemplate("17.6.1.143"); + expect(second).toBe(first); + expect(Date.now() - started).toBeLessThan(1000); // cache hit, no rebuild + }, 300_000); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/fleet && FLEET_PG_TESTS=1 pnpm vitest run src/TemplateStore.integration.test.ts` +Expected: FAIL — module not found. (Without the env var it must SKIP, verify that too.) + +- [ ] **Step 3: Implement** + +```typescript +// src/TemplateStore.ts +import { mkdir, open, rename, rm, stat, writeFile, unlink } from "node:fs/promises"; +import { join } from "node:path"; +import { createStack, installMicroProfile } from "@supabase/stack/bun"; +import type { ServiceName, VersionManifest } from "@supabase/stack"; +import { cloneDir } from "./cowClone.ts"; +import { baseTemplateKey, templateKey } from "./PodManifest.ts"; + +const LOCK_STALE_MS = 10 * 60 * 1000; + +export class TemplateStore { + constructor(private readonly root: string) {} + + private dataDir(key: string): string { + return join(this.root, key, "data"); + } + + async has(key: string): Promise { + return stat(join(this.root, key, "template.json")).then(() => true, () => false); + } + + async ensureBaseTemplate(postgresVersion: string): Promise { + const key = baseTemplateKey(postgresVersion); + if (await this.has(key)) return this.dataDir(key); + return this.withLock(key, async () => { + if (await this.has(key)) return this.dataDir(key); + const buildDir = join(this.root, `${key}.build`); + await rm(buildDir, { recursive: true, force: true }); + await mkdir(buildDir, { recursive: true }); + // One-shot stack: postgres only, non-provisioned → postgres-init applies + // roles/schemas/baseline migrations exactly as today. + const stack = await createStack({ + postgres: { version: postgresVersion, dataDir: join(buildDir, "data") }, + postgrest: false, auth: false, edgeRuntime: false, realtime: false, + storage: false, imgproxy: false, mailpit: false, pgmeta: false, + studio: false, analytics: false, vector: false, pooler: false, + functions: false, + }); + await stack.start(); + await stack.ready(); + await stack.dispose(); // clean shutdown + await installMicroProfile(join(buildDir, "data")); + await mkdir(join(this.root, key), { recursive: true }); + await rename(join(buildDir, "data"), this.dataDir(key)); + await writeFile( + join(this.root, key, "template.json"), + JSON.stringify({ key, postgresVersion, builtAt: new Date().toISOString() }), + ); + await rm(buildDir, { recursive: true, force: true }); + return this.dataDir(key); + }); + } + + async ensureWarmTemplate( + versions: Partial, + enabledServices: ReadonlyArray, + ): Promise { + const pgVersion = versions.postgres; + if (pgVersion === undefined) throw new Error("versions.postgres is required"); + const base = await this.ensureBaseTemplate(pgVersion); + if (enabledServices.length === 0) return base; + const key = templateKey(versions); + if (await this.has(key)) return this.dataDir(key); + return this.withLock(key, async () => { + if (await this.has(key)) return this.dataDir(key); + const buildDir = join(this.root, `${key}.build`); + await rm(buildDir, { recursive: true, force: true }); + await mkdir(buildDir, { recursive: true }); + await cloneDir(base, join(buildDir, "data")); + const stack = await createStack({ + postgres: { version: pgVersion, dataDir: join(buildDir, "data"), provisioned: true, profile: "micro" }, + // enable exactly the listed services so each self-migrates once: + postgrest: enabledServices.includes("postgrest") ? {} : false, + auth: enabledServices.includes("auth") ? {} : false, + realtime: enabledServices.includes("realtime") ? {} : false, + edgeRuntime: enabledServices.includes("edge-runtime") ? {} : false, + storage: false, imgproxy: false, mailpit: false, pgmeta: false, + studio: false, analytics: false, vector: false, pooler: false, + functions: false, + }); + await stack.start(); + await stack.ready(); + await stack.dispose(); + await mkdir(join(this.root, key), { recursive: true }); + await rename(join(buildDir, "data"), this.dataDir(key)); + await writeFile( + join(this.root, key, "template.json"), + JSON.stringify({ key, versions, enabledServices, builtAt: new Date().toISOString() }), + ); + await rm(buildDir, { recursive: true, force: true }); + return this.dataDir(key); + }); + } + + private async withLock(key: string, body: () => Promise): Promise { + const lockPath = join(this.root, `${key}.lock`); + await mkdir(this.root, { recursive: true }); + for (;;) { + try { + const handle = await open(lockPath, "wx"); + await handle.close(); + break; + } catch { + const s = await stat(lockPath).catch(() => undefined); + if (s && Date.now() - s.mtimeMs > LOCK_STALE_MS) { + await unlink(lockPath).catch(() => {}); + continue; + } + await new Promise((r) => setTimeout(r, 250)); + } + } + try { + return await body(); + } finally { + await unlink(lockPath).catch(() => {}); + } + } +} +``` + +Also in `packages/stack/src/index.ts`, export the pgconf helpers so fleet can use them: + +```typescript +export { installMicroProfile, readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/fleet && FLEET_PG_TESTS=1 pnpm vitest run src/TemplateStore.integration.test.ts` +Expected: PASS (first run downloads/boots postgres — allow minutes). Then without env: `pnpm vitest run src/TemplateStore.integration.test.ts` → skipped. + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet/src packages/stack/src/index.ts +git commit -m "feat(fleet): template store with base and warm templates" +``` + +--- + +### Task 11: PodRegistry + Provisioner (create / reset / fork / destroy) + +**Files:** +- Create: `packages/fleet/src/PodRegistry.ts` +- Create: `packages/fleet/src/Provisioner.ts` +- Test: `packages/fleet/src/Provisioner.integration.test.ts` + +**Interfaces:** +- Consumes: `PodManifest`/`templateKey` (Task 7), `cloneDir` (Task 8), `PortRegistry` (Task 9), `TemplateStore` (Task 10). +- Produces: + - `class PodRegistry { constructor(podsRoot: string); read(id): Promise; write(manifest): Promise; list(): Promise; remove(id): Promise; podDir(id): string; dataDir(id): string; }` — manifest at `podsRoot//pod.json`. + - `class Provisioner { constructor(opts: { templates: TemplateStore; pods: PodRegistry; ports: PortRegistry }); create(opts: { id: string; versions: Partial; services?: Partial>; flags?: { supautils?: boolean }; warm?: boolean }): Promise; reset(id: string): Promise; fork(sourceId: string, newId: string): Promise; destroy(id: string): Promise; }` + - `create`: allocate ports → ensure template (warm if `warm: true`, else base) → `cloneDir(template, dataDir)` → write manifest. Rejects on duplicate id. + - `reset`: delete `data`, re-clone from the same template. `fork`: requires source **suspended** (caller's responsibility at this layer; Fleet enforces in Task 14) → clone source's data dir + fresh ports + new manifest. `destroy`: remove pod dir, release ports. + +- [ ] **Step 1: Write the failing test** (uses base template; postgres-dependent → gate like Task 10) + +```typescript +// src/Provisioner.integration.test.ts +import { mkdtemp, stat, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { PodRegistry } from "./PodRegistry.ts"; +import { PortRegistry } from "./PortRegistry.ts"; +import { Provisioner } from "./Provisioner.ts"; +import { TemplateStore } from "./TemplateStore.ts"; + +const PG_VERSION = "17.6.1.143"; + +describe.skipIf(!process.env.FLEET_PG_TESTS)("Provisioner", () => { + async function makeProvisioner() { + const root = await mkdtemp(join(tmpdir(), "fleet-")); + const templates = new TemplateStore(join(root, "templates")); + const pods = new PodRegistry(join(root, "pods")); + const ports = await PortRegistry.load(join(root, "fleet-state.json")); + return { p: new Provisioner({ templates, pods, ports }), pods }; + } + + it("creates, forks, resets, destroys", async () => { + const { p, pods } = await makeProvisioner(); + const a = await p.create({ id: "a", versions: { postgres: PG_VERSION } }); + expect(a.ports.dbPort).toBeGreaterThan(0); + expect(await stat(join(pods.dataDir("a"), "PG_VERSION")).then(() => true)).toBe(true); + + // fork: divergence + await writeFile(join(pods.dataDir("a"), "marker.txt"), "from-a"); + const b = await p.fork("a", "b"); + expect(b.ports.dbPort).not.toBe(a.ports.dbPort); + await writeFile(join(pods.dataDir("b"), "marker.txt"), "from-b"); + expect(await Bun.file(join(pods.dataDir("a"), "marker.txt")).text()).toBe("from-a"); + + // reset: marker disappears (re-cloned from template) + await p.reset("a"); + expect(await stat(join(pods.dataDir("a"), "marker.txt")).then(() => true, () => false)).toBe(false); + + await p.destroy("a"); + await p.destroy("b"); + expect(await pods.list()).toEqual([]); + }, 300_000); + + it("rejects duplicate ids", async () => { + const { p } = await makeProvisioner(); + await p.create({ id: "dup", versions: { postgres: PG_VERSION } }); + await expect(p.create({ id: "dup", versions: { postgres: PG_VERSION } })).rejects.toThrow(/exists/); + }, 300_000); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/fleet && FLEET_PG_TESTS=1 pnpm vitest run src/Provisioner.integration.test.ts` +Expected: FAIL — modules not found. + +- [ ] **Step 3: Implement** + +```typescript +// src/PodRegistry.ts +import { mkdir, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import type { PodManifest } from "./PodManifest.ts"; + +export class PodRegistry { + constructor(private readonly podsRoot: string) {} + + podDir(id: string): string { + return join(this.podsRoot, id); + } + dataDir(id: string): string { + return join(this.podsRoot, id, "data"); + } + + async read(id: string): Promise { + const raw = await readFile(join(this.podDir(id), "pod.json"), "utf8").catch(() => undefined); + return raw === undefined ? undefined : (JSON.parse(raw) as PodManifest); + } + + async write(manifest: PodManifest): Promise { + await mkdir(this.podDir(manifest.id), { recursive: true }); + await writeFile(join(this.podDir(manifest.id), "pod.json"), JSON.stringify(manifest, null, 2)); + } + + async list(): Promise { + const entries = await readdir(this.podsRoot).catch(() => [] as string[]); + const manifests = await Promise.all(entries.map((id) => this.read(id))); + return manifests.filter((m): m is PodManifest => m !== undefined); + } + + async remove(id: string): Promise { + await rm(this.podDir(id), { recursive: true, force: true }); + } +} +``` + +```typescript +// src/Provisioner.ts +import { rm } from "node:fs/promises"; +import type { ServiceName, VersionManifest } from "@supabase/stack"; +import { cloneDir } from "./cowClone.ts"; +import type { PodManifest } from "./PodManifest.ts"; +import type { PodRegistry } from "./PodRegistry.ts"; +import type { PortRegistry } from "./PortRegistry.ts"; +import type { TemplateStore } from "./TemplateStore.ts"; + +export interface CreatePodOptions { + readonly id: string; + readonly versions: Partial; + readonly services?: Partial>; + readonly flags?: { readonly supautils?: boolean }; + /** Build/use a warm template (services pre-migrated). Default: base template. */ + readonly warm?: boolean; +} + +export class Provisioner { + constructor( + private readonly deps: { + readonly templates: TemplateStore; + readonly pods: PodRegistry; + readonly ports: PortRegistry; + }, + ) {} + + async create(opts: CreatePodOptions): Promise { + const { templates, pods, ports } = this.deps; + if ((await pods.read(opts.id)) !== undefined) { + throw new Error(`pod already exists: ${opts.id}`); + } + const pgVersion = opts.versions.postgres; + if (pgVersion === undefined) throw new Error("versions.postgres is required"); + const enabled = Object.entries(opts.services ?? {}) + .filter(([, on]) => on === true) + .map(([name]) => name as ServiceName); + const template = + opts.warm === true + ? await templates.ensureWarmTemplate(opts.versions, enabled) + : await templates.ensureBaseTemplate(pgVersion); + const allocated = await ports.allocate(opts.id); + await cloneDir(template, pods.dataDir(opts.id)); + const manifest: PodManifest = { + id: opts.id, + versions: opts.versions, + services: opts.services ?? {}, + flags: { supautils: opts.flags?.supautils ?? false }, + ports: allocated, + createdAt: new Date().toISOString(), + }; + await pods.write(manifest); + return manifest; + } + + async reset(id: string): Promise { + const { templates, pods } = this.deps; + const manifest = await pods.read(id); + if (manifest === undefined) throw new Error(`unknown pod: ${id}`); + const pgVersion = manifest.versions.postgres; + if (pgVersion === undefined) throw new Error(`pod ${id} has no postgres version`); + const template = await templates.ensureBaseTemplate(pgVersion); + await rm(pods.dataDir(id), { recursive: true, force: true }); + await cloneDir(template, pods.dataDir(id)); + } + + /** Caller must ensure the source pod is stopped/suspended first. */ + async fork(sourceId: string, newId: string): Promise { + const { pods, ports } = this.deps; + const source = await pods.read(sourceId); + if (source === undefined) throw new Error(`unknown pod: ${sourceId}`); + if ((await pods.read(newId)) !== undefined) { + throw new Error(`pod already exists: ${newId}`); + } + const allocated = await ports.allocate(newId); + await cloneDir(pods.dataDir(sourceId), pods.dataDir(newId)); + const manifest: PodManifest = { + ...source, + id: newId, + ports: allocated, + createdAt: new Date().toISOString(), + }; + await pods.write(manifest); + return manifest; + } + + async destroy(id: string): Promise { + const { pods, ports } = this.deps; + await pods.remove(id); + await ports.release(id); + } +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/fleet && FLEET_PG_TESTS=1 pnpm vitest run src/Provisioner.integration.test.ts` +Expected: 2 passed. + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet/src +git commit -m "feat(fleet): pod registry and provisioner (create/reset/fork/destroy)" +``` + +--- + +### Task 12: EdgeProxy — TCP wake-proxy with traffic events + +**Files:** +- Create: `packages/fleet/src/EdgeProxy.ts` +- Test: `packages/fleet/src/EdgeProxy.unit.test.ts` (uses plain TCP echo servers, no postgres) + +**Interfaces:** +- Produces: + +```typescript +export interface EdgeProxyEvents { + /** Fired on connect/disconnect/bytes; IdleMonitor consumes these. */ + onActivity: (podId: string, event: "connect" | "data" | "disconnect", openConnections: number) => void; +} +export interface PodUpstream { readonly host: string; readonly port: number } +export class EdgeProxy { + constructor(events?: Partial); + /** Bind listenPort now and forever; `wake` is awaited per-connection to get the upstream. */ + register(podId: string, listenPort: number, wake: () => Promise): Promise; + unregister(podId: string): Promise; + openConnections(podId: string): number; + close(): Promise; +} +``` + +- Semantics: listener accepts immediately; each accepted socket pauses, awaits `wake()`, then splices to the upstream with `socket.pipe(upstream)` both ways. `wake` failures destroy the client socket. Activity events fire on every data chunk in either direction. + +- [ ] **Step 1: Write the failing test** + +```typescript +// src/EdgeProxy.unit.test.ts +import { createServer, connect, type AddressInfo, type Server } from "node:net"; +import { afterEach, describe, expect, it } from "vitest"; +import { EdgeProxy } from "./EdgeProxy.ts"; + +function echoServer(): Promise<{ server: Server; port: number }> { + return new Promise((resolve) => { + const server = createServer((sock) => sock.pipe(sock)); + server.listen(0, "127.0.0.1", () => + resolve({ server, port: (server.address() as AddressInfo).port }), + ); + }); +} + +function freePort(): Promise { + return new Promise((resolve) => { + const s = createServer(); + s.listen(0, "127.0.0.1", () => { + const port = (s.address() as AddressInfo).port; + s.close(() => resolve(port)); + }); + }); +} + +describe("EdgeProxy", () => { + const proxies: EdgeProxy[] = []; + afterEach(async () => { + for (const p of proxies.splice(0)) await p.close(); + }); + + it("wakes on first connection and splices bytes both ways", async () => { + const { server, port: upstreamPort } = await echoServer(); + let wakes = 0; + const proxy = new EdgeProxy(); + proxies.push(proxy); + const listenPort = await freePort(); + await proxy.register("pod-a", listenPort, async () => { + wakes += 1; + return { host: "127.0.0.1", port: upstreamPort }; + }); + + const reply = await new Promise((resolve, reject) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write("ping")); + sock.on("data", (d) => { resolve(d.toString()); sock.end(); }); + sock.on("error", reject); + }); + expect(reply).toBe("ping"); + expect(wakes).toBe(1); + server.close(); + }); + + it("tracks open connections and reports activity", async () => { + const { server, port: upstreamPort } = await echoServer(); + const events: string[] = []; + const proxy = new EdgeProxy({ + onActivity: (id, ev) => events.push(`${id}:${ev}`), + }); + proxies.push(proxy); + const listenPort = await freePort(); + await proxy.register("pod-b", listenPort, async () => ({ host: "127.0.0.1", port: upstreamPort })); + + await new Promise((resolve) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write("x")); + sock.on("data", () => sock.end()); + sock.on("close", () => resolve()); + }); + await new Promise((r) => setTimeout(r, 50)); + expect(events).toContain("pod-b:connect"); + expect(events).toContain("pod-b:data"); + expect(events).toContain("pod-b:disconnect"); + expect(proxy.openConnections("pod-b")).toBe(0); + server.close(); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/fleet && pnpm vitest run src/EdgeProxy.unit.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +```typescript +// src/EdgeProxy.ts +import { connect, createServer, type Server, type Socket } from "node:net"; + +export interface PodUpstream { + readonly host: string; + readonly port: number; +} + +export interface EdgeProxyEvents { + onActivity: ( + podId: string, + event: "connect" | "data" | "disconnect", + openConnections: number, + ) => void; +} + +interface Registration { + readonly server: Server; + readonly sockets: Set; +} + +export class EdgeProxy { + private readonly registrations = new Map(); + + constructor(private readonly events: Partial = {}) {} + + openConnections(podId: string): number { + return this.registrations.get(podId)?.sockets.size ?? 0; + } + + register( + podId: string, + listenPort: number, + wake: () => Promise, + ): Promise { + const sockets = new Set(); + const emit = (event: "connect" | "data" | "disconnect") => + this.events.onActivity?.(podId, event, sockets.size); + + const server = createServer((client) => { + sockets.add(client); + client.pause(); + emit("connect"); + const cleanup = () => { + if (sockets.delete(client)) emit("disconnect"); + }; + client.on("close", cleanup); + client.on("error", cleanup); + wake().then( + (upstream) => { + const backend = connect(upstream.port, upstream.host); + backend.on("error", () => client.destroy()); + client.on("close", () => backend.destroy()); + backend.on("close", () => client.destroy()); + client.on("data", () => emit("data")); + backend.on("data", () => emit("data")); + backend.on("connect", () => { + client.pipe(backend); + backend.pipe(client); + client.resume(); + }); + }, + () => client.destroy(), + ); + }); + + this.registrations.set(podId, { server, sockets }); + return new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(listenPort, "127.0.0.1", () => resolve()); + }); + } + + async unregister(podId: string): Promise { + const reg = this.registrations.get(podId); + if (!reg) return; + this.registrations.delete(podId); + for (const sock of reg.sockets) sock.destroy(); + await new Promise((resolve) => reg.server.close(() => resolve())); + } + + async close(): Promise { + await Promise.all([...this.registrations.keys()].map((id) => this.unregister(id))); + } +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/fleet && pnpm vitest run src/EdgeProxy.unit.test.ts` +Expected: 2 passed. + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet/src +git commit -m "feat(fleet): TCP wake-proxy edge with activity events" +``` + +--- + +### Task 13: IdleMonitor + +**Files:** +- Create: `packages/fleet/src/IdleMonitor.ts` +- Test: `packages/fleet/src/IdleMonitor.unit.test.ts` (fake timers) + +**Interfaces:** +- Consumes: activity events shape from Task 12. +- Produces: + +```typescript +export class IdleMonitor { + constructor(opts: { + idleMs: number; + onIdle: (podId: string) => void; + now?: () => number; // injectable clock for tests + schedule?: typeof setTimeout; // injectable timer for tests + }); + /** Wire to EdgeProxy events: any activity resets the timer; open connections hold it. */ + recordActivity(podId: string, openConnections: number): void; + /** Start tracking a pod (e.g. on wake). */ + track(podId: string): void; + /** Stop tracking (on suspend/destroy). */ + untrack(podId: string): void; +} +``` + +- Semantics (spec: "no open external connections AND no bytes for T"): while `openConnections > 0` a pod never goes idle; when the last connection closes, the countdown starts from that moment; any activity resets it; `onIdle` fires at most once per warm period (re-armed by the next `track`). + +- [ ] **Step 1: Write the failing test** + +```typescript +// src/IdleMonitor.unit.test.ts +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { IdleMonitor } from "./IdleMonitor.ts"; + +describe("IdleMonitor", () => { + beforeEach(() => vi.useFakeTimers()); + afterEach(() => vi.useRealTimers()); + + it("fires onIdle after idleMs with no connections and no activity", () => { + const idled: string[] = []; + const mon = new IdleMonitor({ idleMs: 1000, onIdle: (id) => idled.push(id) }); + mon.track("a"); + mon.recordActivity("a", 0); + vi.advanceTimersByTime(999); + expect(idled).toEqual([]); + vi.advanceTimersByTime(2); + expect(idled).toEqual(["a"]); + }); + + it("open connections hold the pod warm indefinitely", () => { + const idled: string[] = []; + const mon = new IdleMonitor({ idleMs: 1000, onIdle: (id) => idled.push(id) }); + mon.track("a"); + mon.recordActivity("a", 1); // one open connection + vi.advanceTimersByTime(10_000); + expect(idled).toEqual([]); + mon.recordActivity("a", 0); // last connection closed + vi.advanceTimersByTime(1001); + expect(idled).toEqual(["a"]); + }); + + it("activity resets the countdown; untrack cancels", () => { + const idled: string[] = []; + const mon = new IdleMonitor({ idleMs: 1000, onIdle: (id) => idled.push(id) }); + mon.track("a"); + mon.recordActivity("a", 0); + vi.advanceTimersByTime(900); + mon.recordActivity("a", 0); // reset + vi.advanceTimersByTime(900); + expect(idled).toEqual([]); + mon.untrack("a"); + vi.advanceTimersByTime(5000); + expect(idled).toEqual([]); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/fleet && pnpm vitest run src/IdleMonitor.unit.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +```typescript +// src/IdleMonitor.ts +interface Tracked { + timer: ReturnType | undefined; + openConnections: number; +} + +export class IdleMonitor { + private readonly tracked = new Map(); + + constructor( + private readonly opts: { + readonly idleMs: number; + readonly onIdle: (podId: string) => void; + }, + ) {} + + track(podId: string): void { + if (!this.tracked.has(podId)) { + this.tracked.set(podId, { timer: undefined, openConnections: 0 }); + this.arm(podId); + } + } + + untrack(podId: string): void { + const entry = this.tracked.get(podId); + if (entry?.timer) clearTimeout(entry.timer); + this.tracked.delete(podId); + } + + recordActivity(podId: string, openConnections: number): void { + const entry = this.tracked.get(podId); + if (!entry) return; + entry.openConnections = openConnections; + this.arm(podId); + } + + private arm(podId: string): void { + const entry = this.tracked.get(podId); + if (!entry) return; + if (entry.timer) clearTimeout(entry.timer); + entry.timer = undefined; + if (entry.openConnections > 0) return; // held warm by open connections + entry.timer = setTimeout(() => { + this.tracked.delete(podId); + this.opts.onIdle(podId); + }, this.opts.idleMs); + } +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/fleet && pnpm vitest run src/IdleMonitor.unit.test.ts` +Expected: 3 passed. + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet/src +git commit -m "feat(fleet): idle monitor with connection-aware countdown" +``` + +--- + +### Task 14: Fleet facade — wake/suspend lifecycle + startup reconciliation + +**Files:** +- Create: `packages/fleet/src/Fleet.ts` +- Modify: `packages/fleet/src/index.ts` (public exports) +- Test: `packages/fleet/src/Fleet.integration.test.ts` + +**Interfaces:** +- Consumes: everything from Tasks 7–13, `createStack` + `StackHandle` from `@supabase/stack/bun`. +- Produces the package's public API: + +```typescript +export interface FleetOptions { + readonly root?: string; // default: join(homedir(), ".supabase") + readonly idleMs?: number; // default: 5 * 60_000 +} +export interface PodStatus { + readonly manifest: PodManifest; + readonly state: "suspended" | "waking" | "warm" | "suspending"; + readonly dbUrl: string; // through the edge proxy port — stable across suspend cycles +} +export interface FleetHandle extends AsyncDisposable { + createPod(opts: CreatePodOptions): Promise; + destroyPod(id: string): Promise; + resetPod(id: string): Promise; + forkPod(sourceId: string, newId: string): Promise; + wake(id: string): Promise; + suspend(id: string): Promise; + enableExtension(id: string, extension: string): Promise; + listPods(): Promise>; + dispose(): Promise; +} +export function createFleet(opts?: FleetOptions): Promise; +``` + +- Key behaviors: + - `createPod` registers the pod's `dbPort` on the EdgeProxy immediately (suspended pods answer the port; first connection wakes them). The pod's internal postgres listens on an ephemeral port chosen at wake time (`apiPort`-style allocation is internal); the **external** `dbPort` never changes. + - Wake path: `wake(id)` (or first proxied connection) → `createStack({ postgres: { dataDir, provisioned: true, profile: "micro", port: }, lazyServices: true, ...services-from-manifest })` → `start()` → `serviceReady("postgres")` → IdleMonitor `track(id)`. Memoize concurrent wakes (reuse `makeEnsureServiceMemo` pattern with per-pod keys). + - Suspend path: IdleMonitor `onIdle` → `suspend(id)` → `stack.dispose()` → drop the StackHandle → state `suspended`. EdgeProxy registration stays. + - `forkPod`: `suspend(source)` first if warm, then `Provisioner.fork`. + - `enableExtension`: if warm → delegate to `StackHandle.enableExtension`; if suspended → edit pod.conf directly via `writePreloadLibraries(dataDir, ...)` (no restart needed — next wake picks it up). + - **Startup reconciliation:** on `createFleet`, scan `podsRoot/*/run.pid`; any live process groups from a previous daemon are terminated (SIGTERM, then SIGKILL after 5s) and their pods marked suspended. Phase 1 explicitly kills-then-wakes rather than adopting (documented deviation from the spec's adoption goal; acceptable because data is disposable and wake is ~fast — revisit in a later phase). Write `run.pid` on wake, remove on suspend. + - `dispose()` suspends all warm pods and closes the EdgeProxy. + +- [ ] **Step 1: Write the failing test** + +```typescript +// src/Fleet.integration.test.ts +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createFleet } from "./Fleet.ts"; + +const PG_VERSION = "17.6.1.143"; + +async function query(dbUrl: string, sql: string): Promise { + // Minimal client via Bun's built-in postgres support. If this suite runs under + // Node-based vitest instead of Bun, swap for the `postgres` npm package (dev dep). + const { SQL } = await import("bun"); + const db = new SQL(dbUrl); + const rows = await db.unsafe(sql); + await db.close(); + return JSON.stringify(rows); +} + +describe.skipIf(!process.env.FLEET_PG_TESTS)("Fleet", () => { + it("wake-on-connect, suspend-on-idle, fork", async () => { + const root = await mkdtemp(join(tmpdir(), "fleet-e2e-")); + await using fleet = await createFleet({ root, idleMs: 2000 }); + + const a = await fleet.createPod({ id: "a", versions: { postgres: PG_VERSION } }); + expect(a.state).toBe("suspended"); + + // First connection wakes the pod transparently. + await query(a.dbUrl, "create table t(x int); insert into t values (1)"); + const warm = (await fleet.listPods()).find((p) => p.manifest.id === "a"); + expect(warm?.state).toBe("warm"); + + // Idle out (no connections) → suspended. + await new Promise((r) => setTimeout(r, 4000)); + const idle = (await fleet.listPods()).find((p) => p.manifest.id === "a"); + expect(idle?.state).toBe("suspended"); + + // Wake again on the SAME dbUrl; data survived suspend. + expect(await query(a.dbUrl, "select x from t")).toContain("1"); + + // Fork inherits data, diverges independently. + const b = await fleet.forkPod("a", "b"); + await query(b.dbUrl, "insert into t values (2)"); + expect(await query(a.dbUrl, "select count(*)::int as n from t")).toContain("1"); + expect(await query(b.dbUrl, "select count(*)::int as n from t")).toContain("2"); + }, 600_000); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd packages/fleet && FLEET_PG_TESTS=1 pnpm vitest run src/Fleet.integration.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement `Fleet.ts`** + +```typescript +// src/Fleet.ts +import { homedir } from "node:os"; +import { join } from "node:path"; +import { readFile, rm, writeFile } from "node:fs/promises"; +import { + createStack, + writePreloadLibraries, + type StackHandle, +} from "@supabase/stack/bun"; +import { EdgeProxy, type PodUpstream } from "./EdgeProxy.ts"; +import { IdleMonitor } from "./IdleMonitor.ts"; +import { PodRegistry } from "./PodRegistry.ts"; +import { PortRegistry } from "./PortRegistry.ts"; +import { Provisioner, type CreatePodOptions } from "./Provisioner.ts"; +import { TemplateStore } from "./TemplateStore.ts"; +import type { PodManifest } from "./PodManifest.ts"; + +export interface FleetOptions { + readonly root?: string; + readonly idleMs?: number; +} + +export type PodState = "suspended" | "waking" | "warm" | "suspending"; + +export interface PodStatus { + readonly manifest: PodManifest; + readonly state: PodState; + readonly dbUrl: string; +} + +export interface FleetHandle extends AsyncDisposable { + createPod(opts: CreatePodOptions): Promise; + destroyPod(id: string): Promise; + resetPod(id: string): Promise; + forkPod(sourceId: string, newId: string): Promise; + wake(id: string): Promise; + suspend(id: string): Promise; + enableExtension(id: string, extension: string): Promise; + listPods(): Promise>; + dispose(): Promise; +} + +interface WarmPod { + readonly stack: StackHandle; + readonly internalDbPort: number; +} + +const DB_PASSWORD = "postgres"; // matches supabase CLI local-dev convention + +export async function createFleet(opts: FleetOptions = {}): Promise { + const root = opts.root ?? join(homedir(), ".supabase"); + const idleMs = opts.idleMs ?? 5 * 60_000; + + const templates = new TemplateStore(join(root, "templates")); + const pods = new PodRegistry(join(root, "pods")); + const ports = await PortRegistry.load(join(root, "fleet-state.json")); + const provisioner = new Provisioner({ templates, pods, ports }); + + const states = new Map(); + const warm = new Map(); + const wakesInFlight = new Map>(); + + const monitor = new IdleMonitor({ + idleMs, + onIdle: (podId) => { + void suspend(podId).catch(() => {}); + }, + }); + + const proxy = new EdgeProxy({ + onActivity: (podId, _event, openConnections) => { + monitor.recordActivity(podId, openConnections); + }, + }); + + const dbUrl = (manifest: PodManifest): string => + `postgresql://postgres:${DB_PASSWORD}@127.0.0.1:${manifest.ports.dbPort}/postgres`; + + async function wakeUpstream(id: string): Promise { + const existing = warm.get(id); + if (existing) return { host: "127.0.0.1", port: existing.internalDbPort }; + const inFlight = wakesInFlight.get(id); + if (inFlight) return inFlight; + const p = (async (): Promise => { + const manifest = await pods.read(id); + if (manifest === undefined) throw new Error(`unknown pod: ${id}`); + states.set(id, "waking"); + // Internal port: derive deterministically from external dbPort to stay clash-free + // within the registry-owned range (external ports are even offsets; internal = +10000). + const internalDbPort = manifest.ports.dbPort + 10_000; + const stack = await createStack({ + stackRoot: join(pods.podDir(id), "stack"), + port: manifest.ports.apiPort + 10_000, + lazyServices: true, + postgres: { + dataDir: pods.dataDir(id), + version: manifest.versions.postgres, + port: internalDbPort, + provisioned: true, + profile: "micro", + }, + postgrest: manifest.services.postgrest === true ? {} : false, + auth: manifest.services.auth === true ? {} : false, + realtime: manifest.services.realtime === true ? {} : false, + edgeRuntime: manifest.services["edge-runtime"] === true ? {} : false, + storage: false, imgproxy: false, mailpit: false, pgmeta: false, + studio: false, analytics: false, vector: false, pooler: false, + functions: false, + }); + await stack.start(); + await stack.serviceReady("postgres"); + warm.set(id, { stack, internalDbPort }); + states.set(id, "warm"); + monitor.track(id); + monitor.recordActivity(id, proxy.openConnections(id)); + await writeFile(join(pods.podDir(id), "run.pid"), String(process.pid)); + wakesInFlight.delete(id); + return { host: "127.0.0.1", port: internalDbPort }; + })().catch((err: unknown) => { + wakesInFlight.delete(id); + states.set(id, "suspended"); + throw err; + }); + wakesInFlight.set(id, p); + return p; + } + + async function registerEdge(manifest: PodManifest): Promise { + states.set(manifest.id, "suspended"); + await proxy.register(manifest.id, manifest.ports.dbPort, () => wakeUpstream(manifest.id)); + } + + async function suspend(id: string): Promise { + const pod = warm.get(id); + if (!pod) return; + states.set(id, "suspending"); + monitor.untrack(id); + warm.delete(id); + await pod.stack.dispose(); + await rm(join(pods.podDir(id), "run.pid"), { force: true }); + states.set(id, "suspended"); + } + + async function status(manifest: PodManifest): Promise { + return { + manifest, + state: states.get(manifest.id) ?? "suspended", + dbUrl: dbUrl(manifest), + }; + } + + // Startup reconciliation: phase 1 policy is kill-then-suspend, not adoption. + // (Spec notes adoption as the goal; deferred — data is disposable and wake is fast.) + for (const manifest of await pods.list()) { + const pidRaw = await readFile(join(pods.podDir(manifest.id), "run.pid"), "utf8") + .catch(() => undefined); + if (pidRaw !== undefined) { + const pid = Number(pidRaw); + if (Number.isFinite(pid) && pid > 0 && pid !== process.pid) { + try { process.kill(-pid, "SIGTERM"); } catch { /* group gone */ } + try { process.kill(pid, "SIGTERM"); } catch { /* gone */ } + } + await rm(join(pods.podDir(manifest.id), "run.pid"), { force: true }); + } + await registerEdge(manifest); + } + + const handle: FleetHandle = { + async createPod(opts) { + const manifest = await provisioner.create(opts); + await registerEdge(manifest); + return status(manifest); + }, + async destroyPod(id) { + await suspend(id); + await proxy.unregister(id); + states.delete(id); + await provisioner.destroy(id); + }, + async resetPod(id) { + await suspend(id); + await provisioner.reset(id); + }, + async forkPod(sourceId, newId) { + await suspend(sourceId); + const manifest = await provisioner.fork(sourceId, newId); + await registerEdge(manifest); + return status(manifest); + }, + async wake(id) { + await wakeUpstream(id); + }, + suspend, + async enableExtension(id, extension) { + const pod = warm.get(id); + if (pod) { + await pod.stack.enableExtension(extension); + return; + } + const manifest = await pods.read(id); + if (manifest === undefined) throw new Error(`unknown pod: ${id}`); + const { readPreloadLibraries } = await import("@supabase/stack/bun"); + const libs = await readPreloadLibraries(pods.dataDir(id)); + if (!libs.includes(extension)) { + await writePreloadLibraries(pods.dataDir(id), [...libs, extension]); + } + }, + async listPods() { + const manifests = await pods.list(); + return Promise.all(manifests.map((m) => status(m))); + }, + async dispose() { + for (const id of [...warm.keys()]) await suspend(id); + await proxy.close(); + }, + async [Symbol.asyncDispose]() { + await handle.dispose(); + }, + }; + return handle; +} +``` + +And `src/index.ts`: + +```typescript +export { createFleet } from "./Fleet.ts"; +export type { FleetHandle, FleetOptions, PodState, PodStatus } from "./Fleet.ts"; +export type { CreatePodOptions } from "./Provisioner.ts"; +export type { PodManifest } from "./PodManifest.ts"; +export { templateKey, baseTemplateKey } from "./PodManifest.ts"; +``` + +Note for the implementer: `enableExtension` filtering of non-preload extensions happens inside the stack for warm pods; for suspended pods, add the same `PRELOAD_REQUIRED_EXTENSIONS` guard by importing it from `@supabase/stack` (export it from stack's index alongside the pgconf helpers). + +- [ ] **Step 4: Run to verify pass** + +Run: `cd packages/fleet && FLEET_PG_TESTS=1 pnpm vitest run src/Fleet.integration.test.ts` +Expected: PASS (allow several minutes on first run for binary download + template build). + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet/src +git commit -m "feat(fleet): fleet facade with wake-on-connect and suspend-on-idle" +``` + +--- + +### Task 15: Density + lifecycle E2E and package README + +**Files:** +- Create: `packages/fleet/tests/fleetDensity.e2e.test.ts` +- Create: `packages/fleet/README.md` + +**Interfaces:** +- Consumes: `createFleet` (Task 14). +- Produces: the guardrail test from the spec ("Density E2E: 100 registered pods, wake a subset") scaled to CI reality, plus user-facing docs. + +- [ ] **Step 1: Write the E2E test** + +```typescript +// tests/fleetDensity.e2e.test.ts +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createFleet } from "../src/Fleet.ts"; + +const PG_VERSION = "17.6.1.143"; +const REGISTERED = Number(process.env.FLEET_E2E_PODS ?? 20); // 100+ locally, 20 in CI +const WARM = 3; + +describe.skipIf(!process.env.FLEET_PG_TESTS)("fleet density", () => { + it(`registers ${REGISTERED} pods, wakes ${WARM}, suspends cleanly`, async () => { + const root = await mkdtemp(join(tmpdir(), "fleet-density-")); + await using fleet = await createFleet({ root, idleMs: 60_000 }); + + // Registration is cheap: template built once, then CoW clones. + for (let i = 0; i < REGISTERED; i += 1) { + await fleet.createPod({ id: `pod-${i}`, versions: { postgres: PG_VERSION } }); + } + const all = await fleet.listPods(); + expect(all).toHaveLength(REGISTERED); + expect(all.every((p) => p.state === "suspended")).toBe(true); + + // Distinct external ports across the whole fleet. + const portSet = new Set(all.map((p) => p.manifest.ports.dbPort)); + expect(portSet.size).toBe(REGISTERED); + + // Wake a subset; the rest stay suspended (zero processes). + for (let i = 0; i < WARM; i += 1) await fleet.wake(`pod-${i}`); + const after = await fleet.listPods(); + expect(after.filter((p) => p.state === "warm")).toHaveLength(WARM); + expect(after.filter((p) => p.state === "suspended")).toHaveLength(REGISTERED - WARM); + + // Explicit suspend brings a pod back to zero. + await fleet.suspend("pod-0"); + const final = await fleet.listPods(); + expect(final.find((p) => p.manifest.id === "pod-0")?.state).toBe("suspended"); + }, 900_000); +}); +``` + +- [ ] **Step 2: Run it** + +Run: `cd packages/fleet && FLEET_PG_TESTS=1 pnpm vitest run tests/fleetDensity.e2e.test.ts` +Expected: PASS. Locally, re-run once with `FLEET_E2E_PODS=100` and record wall-clock + `ps` observations in the PR description (PSS harness is a later phase; this is the smoke-level guardrail). + +- [ ] **Step 3: Write README.md** + +```markdown +# @supabase/fleet + +Host-level daemon for running many lightweight Supabase pods in parallel: +CoW template provisioning, wake-on-connect, suspend-on-idle, instant fork. + +## Quick start + +​```ts +import { createFleet } from "@supabase/fleet"; + +const fleet = await createFleet(); +const pod = await fleet.createPod({ id: "my-worktree", versions: { postgres: "17.6.1.143" } }); +// pod.dbUrl is live immediately — the first connection wakes postgres (~200ms). +// After 5 idle minutes the pod suspends to zero processes; the port keeps listening. +const branch = await fleet.forkPod("my-worktree", "my-worktree-experiment"); +​``` + +Design: `docs/specs/2026-07-07-micro-supabase-stacks-design.md`. +Phase 1 limitations: native mode (macOS/Linux) only; daemon restart kills-then-resuspends +running pods instead of adopting them; HTTP service lazy-start requires `lazyServices: true`. +``` + +(Remove the zero-width escapes around the code fences when writing the real file.) + +- [ ] **Step 4: Full check** + +Run: `cd packages/fleet && pnpm vitest run --exclude '**/*.e2e.test.ts' && cd ../stack && pnpm vitest run --exclude '**/*.e2e.test.ts'` +Expected: everything green. Also run `pnpm check:all` in both packages and fix lint/format findings. + +- [ ] **Step 5: Commit** + +```bash +git add packages/fleet +git commit -m "test(fleet): density e2e and package README" +``` + +--- + +## Deferred to later phases (explicitly out of Phase 1) + +- Binary/Docker image optimizations (user direction: after this phase). +- CLI wiring of `supabase start/stop` onto `createFleet` (thin layer; do once fleet API settles). +- HTTP gateway host-based routing across pods + edge wake for the api port (Phase 1 wakes via the db port and per-service lazy start inside a warm pod). +- PSS/CPU benchmark harness with budget assertions in CI. +- True pod adoption across fleet-daemon restarts (Phase 1: kill-then-resuspend). +- Warm-template LRU garbage collection. +- `supautils` profile flag enforcement (manifest field exists; config plumbing later). +- Compatibility suite (`CREATE EXTENSION` sweep, dump/restore round-trip). +``` From 267ca875f7083f942ad491d3a0ec9f4cde0420a5 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 11:19:05 +0200 Subject: [PATCH 04/38] feat(fleet): scaffold @supabase/fleet package Co-Authored-By: Claude Fable 5 --- packages/fleet/package.json | 33 ++++++++++++++++++++++++++ packages/fleet/src/index.ts | 1 + packages/fleet/src/index.unit.test.ts | 8 +++++++ packages/fleet/tsconfig.json | 7 ++++++ packages/fleet/vitest.config.ts | 23 ++++++++++++++++++ pnpm-lock.yaml | 34 +++++++++++++++++++++++++++ 6 files changed, 106 insertions(+) create mode 100644 packages/fleet/package.json create mode 100644 packages/fleet/src/index.ts create mode 100644 packages/fleet/src/index.unit.test.ts create mode 100644 packages/fleet/tsconfig.json create mode 100644 packages/fleet/vitest.config.ts diff --git a/packages/fleet/package.json b/packages/fleet/package.json new file mode 100644 index 0000000000..2e31dcb79d --- /dev/null +++ b/packages/fleet/package.json @@ -0,0 +1,33 @@ +{ + "name": "@supabase/fleet", + "version": "0.1.0", + "private": true, + "type": "module", + "exports": { + ".": "./src/index.ts" + }, + "scripts": { + "test": "nx run-many -t test:unit --projects=$npm_package_name", + "check:all": "nx run-many -t types:check lint:check fmt:check knip:check --projects=$npm_package_name", + "fix:all": "nx run-many -t lint:fix fmt:fix knip:fix --projects=$npm_package_name" + }, + "dependencies": { + "@supabase/process-compose": "workspace:*", + "@supabase/stack": "workspace:*", + "effect": "catalog:" + }, + "devDependencies": { + "@tsconfig/bun": "catalog:", + "@types/bun": "catalog:", + "@vitest/coverage-istanbul": "catalog:", + "knip": "catalog:", + "oxfmt": "catalog:", + "oxlint": "catalog:", + "vitest": "catalog:" + }, + "knip": { + "entry": ["src/**/*.test.ts"], + "ignoreDependencies": ["@typescript/native-preview", "oxfmt", "oxlint", "oxlint-tsgolint"], + "ignoreBinaries": ["nx"] + } +} diff --git a/packages/fleet/src/index.ts b/packages/fleet/src/index.ts new file mode 100644 index 0000000000..2ec70b447c --- /dev/null +++ b/packages/fleet/src/index.ts @@ -0,0 +1 @@ +export const FLEET_PACKAGE = "@supabase/fleet"; diff --git a/packages/fleet/src/index.unit.test.ts b/packages/fleet/src/index.unit.test.ts new file mode 100644 index 0000000000..a9db3c5fd9 --- /dev/null +++ b/packages/fleet/src/index.unit.test.ts @@ -0,0 +1,8 @@ +import { describe, expect, it } from "vitest"; +import { FLEET_PACKAGE } from "./index.ts"; + +describe("fleet scaffold", () => { + it("exports the package marker", () => { + expect(FLEET_PACKAGE).toBe("@supabase/fleet"); + }); +}); diff --git a/packages/fleet/tsconfig.json b/packages/fleet/tsconfig.json new file mode 100644 index 0000000000..eef2f2a863 --- /dev/null +++ b/packages/fleet/tsconfig.json @@ -0,0 +1,7 @@ +{ + "extends": "@tsconfig/bun/tsconfig.json", + "compilerOptions": { + "lib": ["ESNext", "DOM"], + "types": ["bun"] + } +} diff --git a/packages/fleet/vitest.config.ts b/packages/fleet/vitest.config.ts new file mode 100644 index 0000000000..7a35ffa8d7 --- /dev/null +++ b/packages/fleet/vitest.config.ts @@ -0,0 +1,23 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + passWithNoTests: true, + coverage: { + enabled: false, + provider: "istanbul", + clean: false, + include: ["src/**/*.ts"], + reporter: ["text", "lcov"], + reportsDirectory: "coverage", + }, + projects: [ + { + test: { + name: "unit", + include: ["**/*.unit.test.ts"], + }, + }, + ], + }, +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index ae8e53cb34..22c0cce372 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -451,6 +451,40 @@ importers: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) + packages/fleet: + dependencies: + '@supabase/process-compose': + specifier: workspace:* + version: link:../process-compose + '@supabase/stack': + specifier: workspace:* + version: link:../stack + effect: + specifier: 'catalog:' + version: 4.0.0-beta.93 + devDependencies: + '@tsconfig/bun': + specifier: 'catalog:' + version: 1.0.10 + '@types/bun': + specifier: 'catalog:' + version: 1.3.14 + '@vitest/coverage-istanbul': + specifier: 'catalog:' + version: 4.1.9(vitest@4.1.9) + knip: + specifier: 'catalog:' + version: 6.23.0 + oxfmt: + specifier: 'catalog:' + version: 0.56.0 + oxlint: + specifier: 'catalog:' + version: 1.71.0(oxlint-tsgolint@0.23.0) + vitest: + specifier: 'catalog:' + version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) + packages/process-compose: dependencies: '@effect/platform-bun': From 37537aac4a5d57dbb73f3ea459de21e4f2f28f67 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 11:24:42 +0200 Subject: [PATCH 05/38] feat(stack): add micro postgres profile and preload-required registry --- packages/stack/src/micro.ts | 52 +++++++++++++++++++++++++++ packages/stack/src/micro.unit.test.ts | 38 ++++++++++++++++++++ 2 files changed, 90 insertions(+) create mode 100644 packages/stack/src/micro.ts create mode 100644 packages/stack/src/micro.unit.test.ts diff --git a/packages/stack/src/micro.ts b/packages/stack/src/micro.ts new file mode 100644 index 0000000000..c22a891ea9 --- /dev/null +++ b/packages/stack/src/micro.ts @@ -0,0 +1,52 @@ +/** + * Micro profile: normative values from + * docs/specs/2026-07-07-micro-supabase-stacks-design.md ("The micro.conf profile"). + */ +export const MICRO_POSTGRES_SETTINGS: ReadonlyArray = [ + // memory + ["shared_buffers", "16MB"], + ["work_mem", "4MB"], + ["maintenance_work_mem", "32MB"], + ["jit", "off"], + ["huge_pages", "off"], + ["max_connections", "40"], + // background CPU + ["autovacuum_naptime", "5min"], + ["autovacuum_max_workers", "1"], + ["bgwriter_lru_maxpages", "0"], + ["wal_writer_delay", "10s"], + ["checkpoint_timeout", "30min"], + ["max_parallel_workers", "0"], + ["max_parallel_workers_per_gather", "0"], + ["max_worker_processes", "4"], + ["track_io_timing", "off"], + // durability (disposable profile) + ["fsync", "off"], + ["synchronous_commit", "off"], + ["full_page_writes", "off"], + // replication reservations + ["wal_level", "logical"], + ["max_wal_senders", "5"], + ["max_replication_slots", "5"], + ["max_slot_wal_keep_size", "256MB"], + ["wal_keep_size", "0"], +]; + +export const buildMicroConf = (): string => + `${MICRO_POSTGRES_SETTINGS.map(([k, v]) => `${k} = '${v}'`).join("\n")}\n`; + +/** Extensions that only work when named in shared_preload_libraries. */ +export const PRELOAD_REQUIRED_EXTENSIONS: ReadonlySet = new Set([ + "pg_cron", + "pg_net", + "timescaledb", + "pg_stat_statements", + "auto_explain", + "pgaudit", + "plan_filter", + "supautils", + "pgsodium", +]); + +export const buildPodConf = (preloadLibraries: ReadonlyArray): string => + `shared_preload_libraries = '${preloadLibraries.join(",")}'\n`; diff --git a/packages/stack/src/micro.unit.test.ts b/packages/stack/src/micro.unit.test.ts new file mode 100644 index 0000000000..8bb928e636 --- /dev/null +++ b/packages/stack/src/micro.unit.test.ts @@ -0,0 +1,38 @@ +import { describe, expect, it } from "vitest"; +import { + buildMicroConf, + buildPodConf, + MICRO_POSTGRES_SETTINGS, + PRELOAD_REQUIRED_EXTENSIONS, +} from "./micro.ts"; + +describe("micro profile", () => { + it("contains the normative spec settings", () => { + const map = new Map(MICRO_POSTGRES_SETTINGS); + expect(map.get("shared_buffers")).toBe("16MB"); + expect(map.get("jit")).toBe("off"); + expect(map.get("fsync")).toBe("off"); + expect(map.get("wal_level")).toBe("logical"); + expect(map.get("max_slot_wal_keep_size")).toBe("256MB"); + expect(map.get("wal_writer_delay")).toBe("10s"); + }); + + it("renders micro.conf as key = 'value' lines", () => { + const conf = buildMicroConf(); + expect(conf).toContain("shared_buffers = '16MB'"); + expect(conf).toContain("max_connections = '40'"); + expect(conf.endsWith("\n")).toBe(true); + }); + + it("knows which extensions need preload", () => { + expect(PRELOAD_REQUIRED_EXTENSIONS.has("pg_cron")).toBe(true); + expect(PRELOAD_REQUIRED_EXTENSIONS.has("pgvector")).toBe(false); + }); + + it("renders pod.conf with shared_preload_libraries", () => { + expect(buildPodConf(["pg_cron", "pg_net"])).toBe( + "shared_preload_libraries = 'pg_cron,pg_net'\n", + ); + expect(buildPodConf([])).toBe("shared_preload_libraries = ''\n"); + }); +}); From a799f85d877c2fed0910e1856123cef05234ebfd Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 11:28:10 +0200 Subject: [PATCH 06/38] feat(stack): PGDATA conf layering (micro.conf + pod.conf includes) Adds installMicroProfile/readPreloadLibraries/writePreloadLibraries to layer the micro postgres profile onto an existing PGDATA directory via include_if_exists lines in postgresql.conf, idempotently. --- packages/stack/src/pgconf.ts | 39 +++++++++++++++++++++++++ packages/stack/src/pgconf.unit.test.ts | 40 ++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 packages/stack/src/pgconf.ts create mode 100644 packages/stack/src/pgconf.unit.test.ts diff --git a/packages/stack/src/pgconf.ts b/packages/stack/src/pgconf.ts new file mode 100644 index 0000000000..f1027debc9 --- /dev/null +++ b/packages/stack/src/pgconf.ts @@ -0,0 +1,39 @@ +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { buildMicroConf, buildPodConf } from "./micro.ts"; + +const INCLUDE_BLOCK = [ + "", + "# --- supabase micro profile (managed; do not edit below) ---", + "include_if_exists = 'micro.conf'", + "include_if_exists = 'pod.conf'", + "", +].join("\n"); + +export async function installMicroProfile(pgdata: string): Promise { + await writeFile(join(pgdata, "micro.conf"), buildMicroConf()); + const podConf = join(pgdata, "pod.conf"); + const existing = await readFile(podConf, "utf8").catch(() => undefined); + if (existing === undefined) { + await writeFile(podConf, buildPodConf([])); + } + const mainPath = join(pgdata, "postgresql.conf"); + const main = await readFile(mainPath, "utf8"); + if (!main.includes("include_if_exists = 'micro.conf'")) { + await writeFile(mainPath, main + INCLUDE_BLOCK); + } +} + +export async function readPreloadLibraries(pgdata: string): Promise { + const content = await readFile(join(pgdata, "pod.conf"), "utf8").catch(() => ""); + const match = content.match(/^shared_preload_libraries = '([^']*)'/m); + if (!match || match[1] === "") return []; + return match[1].split(","); +} + +export async function writePreloadLibraries( + pgdata: string, + libs: ReadonlyArray, +): Promise { + await writeFile(join(pgdata, "pod.conf"), buildPodConf(libs)); +} diff --git a/packages/stack/src/pgconf.unit.test.ts b/packages/stack/src/pgconf.unit.test.ts new file mode 100644 index 0000000000..9e20db89c9 --- /dev/null +++ b/packages/stack/src/pgconf.unit.test.ts @@ -0,0 +1,40 @@ +import { mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { + installMicroProfile, + readPreloadLibraries, + writePreloadLibraries, +} from "./pgconf.ts"; + +async function fakePgdata(): Promise { + const dir = await mkdtemp(join(tmpdir(), "pgconf-test-")); + await writeFile(join(dir, "postgresql.conf"), "# stock conf\nport = 5432\n"); + return dir; +} + +describe("pgconf", () => { + it("installs micro.conf, pod.conf, and include lines idempotently", async () => { + const pgdata = await fakePgdata(); + await installMicroProfile(pgdata); + await installMicroProfile(pgdata); // idempotent + const main = await readFile(join(pgdata, "postgresql.conf"), "utf8"); + expect(main.match(/include_if_exists = 'micro\.conf'/g)).toHaveLength(1); + expect(main.match(/include_if_exists = 'pod\.conf'/g)).toHaveLength(1); + // pod.conf must be included AFTER micro.conf so pod overrides micro + expect(main.indexOf("micro.conf")).toBeLessThan(main.indexOf("pod.conf")); + const micro = await readFile(join(pgdata, "micro.conf"), "utf8"); + expect(micro).toContain("shared_buffers = '16MB'"); + }); + + it("round-trips preload libraries via pod.conf", async () => { + const pgdata = await fakePgdata(); + await installMicroProfile(pgdata); + expect(await readPreloadLibraries(pgdata)).toEqual([]); + await writePreloadLibraries(pgdata, ["pg_cron"]); + expect(await readPreloadLibraries(pgdata)).toEqual(["pg_cron"]); + await writePreloadLibraries(pgdata, ["pg_cron", "pg_net"]); + expect(await readPreloadLibraries(pgdata)).toEqual(["pg_cron", "pg_net"]); + }); +}); From ad76b322096bc5a1229024da0ecaf9f978b4d439 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 11:35:55 +0200 Subject: [PATCH 07/38] fix(stack): harden pgconf include detection, GUC parsing, and pod.conf writes Address code review findings on the PGDATA conf-layering utilities: - fix TS2532 undefined-narrowing in readPreloadLibraries - match include_if_exists only when active (not commented out) so a disabled include no longer silently short-circuits profile install - accept flexible spacing/quoting in shared_preload_libraries parsing and trim comma-separated entries - writePreloadLibraries now updates/appends the preload line in place instead of clobbering the rest of pod.conf --- packages/stack/src/pgconf.ts | 29 +++++++++++--- packages/stack/src/pgconf.unit.test.ts | 54 +++++++++++++++++++++++--- 2 files changed, 73 insertions(+), 10 deletions(-) diff --git a/packages/stack/src/pgconf.ts b/packages/stack/src/pgconf.ts index f1027debc9..77e5cb1333 100644 --- a/packages/stack/src/pgconf.ts +++ b/packages/stack/src/pgconf.ts @@ -10,6 +10,12 @@ const INCLUDE_BLOCK = [ "", ].join("\n"); +// Line-anchored, active (non-commented) include directive only — a commented-out +// line like `#include_if_exists = 'micro.conf'` must NOT count as installed. +const ACTIVE_INCLUDE_RE = /^\s*include_if_exists = 'micro\.conf'/m; + +const PRELOAD_LIBRARIES_RE = /^\s*shared_preload_libraries\s*=\s*['"]([^'"]*)['"]/m; + export async function installMicroProfile(pgdata: string): Promise { await writeFile(join(pgdata, "micro.conf"), buildMicroConf()); const podConf = join(pgdata, "pod.conf"); @@ -19,21 +25,34 @@ export async function installMicroProfile(pgdata: string): Promise { } const mainPath = join(pgdata, "postgresql.conf"); const main = await readFile(mainPath, "utf8"); - if (!main.includes("include_if_exists = 'micro.conf'")) { + if (!ACTIVE_INCLUDE_RE.test(main)) { await writeFile(mainPath, main + INCLUDE_BLOCK); } } export async function readPreloadLibraries(pgdata: string): Promise { const content = await readFile(join(pgdata, "pod.conf"), "utf8").catch(() => ""); - const match = content.match(/^shared_preload_libraries = '([^']*)'/m); - if (!match || match[1] === "") return []; - return match[1].split(","); + const match = content.match(PRELOAD_LIBRARIES_RE); + if (!match || match[1] === undefined || match[1] === "") return []; + return match[1].split(",").map((lib) => lib.trim()); } export async function writePreloadLibraries( pgdata: string, libs: ReadonlyArray, ): Promise { - await writeFile(join(pgdata, "pod.conf"), buildPodConf(libs)); + const podConf = join(pgdata, "pod.conf"); + const existing = await readFile(podConf, "utf8").catch(() => undefined); + const line = `shared_preload_libraries = '${libs.join(",")}'`; + if (existing === undefined) { + await writeFile(podConf, `${line}\n`); + return; + } + if (PRELOAD_LIBRARIES_RE.test(existing)) { + const updated = existing.replace(PRELOAD_LIBRARIES_RE, line); + await writeFile(podConf, updated); + return; + } + const separator = existing.endsWith("\n") || existing === "" ? "" : "\n"; + await writeFile(podConf, `${existing}${separator}${line}\n`); } diff --git a/packages/stack/src/pgconf.unit.test.ts b/packages/stack/src/pgconf.unit.test.ts index 9e20db89c9..73d12e0740 100644 --- a/packages/stack/src/pgconf.unit.test.ts +++ b/packages/stack/src/pgconf.unit.test.ts @@ -2,11 +2,7 @@ import { mkdtemp, readFile, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; -import { - installMicroProfile, - readPreloadLibraries, - writePreloadLibraries, -} from "./pgconf.ts"; +import { installMicroProfile, readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; async function fakePgdata(): Promise { const dir = await mkdtemp(join(tmpdir(), "pgconf-test-")); @@ -37,4 +33,52 @@ describe("pgconf", () => { await writePreloadLibraries(pgdata, ["pg_cron", "pg_net"]); expect(await readPreloadLibraries(pgdata)).toEqual(["pg_cron", "pg_net"]); }); + + it("appends an active include block when only a commented-out include line exists", async () => { + const pgdata = await fakePgdata(); + const mainPath = join(pgdata, "postgresql.conf"); + await writeFile(mainPath, "# stock conf\nport = 5432\n#include_if_exists = 'micro.conf'\n"); + await installMicroProfile(pgdata); + const main = await readFile(mainPath, "utf8"); + expect(main.match(/^include_if_exists = 'micro\.conf'$/m)).toHaveLength(1); + expect(main.match(/^include_if_exists = 'pod\.conf'$/m)).toHaveLength(1); + // the commented-out line must remain untouched + expect(main).toContain("#include_if_exists = 'micro.conf'"); + }); + + it("parses shared_preload_libraries with flexible spacing and quoting", async () => { + const pgdata = await fakePgdata(); + await writeFile(join(pgdata, "pod.conf"), "shared_preload_libraries='pg_cron, pg_net'\n"); + expect(await readPreloadLibraries(pgdata)).toEqual(["pg_cron", "pg_net"]); + + await writeFile(join(pgdata, "pod.conf"), ' shared_preload_libraries = "pg_cron,pg_net"\n'); + expect(await readPreloadLibraries(pgdata)).toEqual(["pg_cron", "pg_net"]); + }); + + it("writePreloadLibraries preserves other settings already in pod.conf", async () => { + const pgdata = await fakePgdata(); + const podPath = join(pgdata, "pod.conf"); + await writeFile(podPath, "other_setting = 'foo'\nshared_preload_libraries = 'pg_cron'\n"); + await writePreloadLibraries(pgdata, ["pg_cron", "pg_net"]); + const pod = await readFile(podPath, "utf8"); + expect(pod).toContain("other_setting = 'foo'"); + expect(pod).toMatch(/^shared_preload_libraries = 'pg_cron,pg_net'$/m); + expect(await readPreloadLibraries(pgdata)).toEqual(["pg_cron", "pg_net"]); + }); + + it("writePreloadLibraries appends the line when pod.conf has no existing preload line", async () => { + const pgdata = await fakePgdata(); + const podPath = join(pgdata, "pod.conf"); + await writeFile(podPath, "other_setting = 'foo'\n"); + await writePreloadLibraries(pgdata, ["pg_net"]); + const pod = await readFile(podPath, "utf8"); + expect(pod).toContain("other_setting = 'foo'"); + expect(pod).toMatch(/^shared_preload_libraries = 'pg_net'$/m); + }); + + it("writePreloadLibraries creates pod.conf when missing", async () => { + const pgdata = await fakePgdata(); + await writePreloadLibraries(pgdata, ["pg_cron"]); + expect(await readPreloadLibraries(pgdata)).toEqual(["pg_cron"]); + }); }); From 6f2edb551d549f50af772f8d251c53501d9f4000 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 11:45:08 +0200 Subject: [PATCH 08/38] feat(stack): provisioned data dirs, micro profile wiring, postgres 17.6.1.143 Wire postgres.provisioned (skip postgres-init for pre-initialized template clones) and postgres.profile (skip -c runtime args on "micro", since those settings live in micro.conf/pod.conf inside PGDATA) through StackBuilder and the native postgres service. Bump DEFAULT_VERSIONS.postgres to 17.6.1.143. --- .../src/StackBuilder.provisioned.unit.test.ts | 28 +++ packages/stack/src/StackBuilder.ts | 17 +- packages/stack/src/createStack.ts | 2 + packages/stack/src/services/postgres.ts | 18 +- packages/stack/src/versions.ts | 2 +- packages/stack/tests/helpers/buildServices.ts | 162 ++++++++++++++++++ 6 files changed, 225 insertions(+), 4 deletions(-) create mode 100644 packages/stack/src/StackBuilder.provisioned.unit.test.ts create mode 100644 packages/stack/tests/helpers/buildServices.ts diff --git a/packages/stack/src/StackBuilder.provisioned.unit.test.ts b/packages/stack/src/StackBuilder.provisioned.unit.test.ts new file mode 100644 index 0000000000..e118cb1622 --- /dev/null +++ b/packages/stack/src/StackBuilder.provisioned.unit.test.ts @@ -0,0 +1,28 @@ +import { describe, expect, it } from "vitest"; +import { buildServicesForTest } from "../tests/helpers/buildServices.ts"; + +describe("provisioned postgres", () => { + it("excludes postgres-init when postgres.provisioned is true", async () => { + const services = await buildServicesForTest({ postgres: { provisioned: true } }); + expect(services.map((s) => s.name)).not.toContain("postgres-init"); + }); + + it("includes postgres-init by default", async () => { + const services = await buildServicesForTest({}); + expect(services.map((s) => s.name)).toContain("postgres-init"); + }); + + it("drops -c runtime args when profile is micro", async () => { + const services = await buildServicesForTest({ + postgres: { provisioned: true, profile: "micro" }, + }); + const pg = services.find((s) => s.name === "postgres"); + expect(pg?.args?.join(" ")).not.toContain("wal_level=logical"); + }); + + it("keeps -c runtime args on the default profile", async () => { + const services = await buildServicesForTest({}); + const pg = services.find((s) => s.name === "postgres"); + expect(pg?.args?.join(" ")).toContain("wal_level=logical"); + }); +}); diff --git a/packages/stack/src/StackBuilder.ts b/packages/stack/src/StackBuilder.ts index 9975ca4fb0..0f69f4a853 100644 --- a/packages/stack/src/StackBuilder.ts +++ b/packages/stack/src/StackBuilder.ts @@ -51,6 +51,17 @@ export interface PostgresConfig { * through the Data API. */ readonly autoExposeNewTables?: boolean; + /** + * When true, `dataDir` is a pre-initialized template clone (already migrated and + * configured), so the `postgres-init` service is skipped entirely. + */ + readonly provisioned?: boolean; + /** + * "micro": settings live in `micro.conf`/`pod.conf` inside PGDATA rather than being passed as + * `-c` runtime args, so ALTER SYSTEM changes made by users aren't overridden on every boot. + * Absent or "default": current behavior (settings passed via `-c` args) is unchanged. + */ + readonly profile?: "default" | "micro"; } export interface PostgrestConfig { @@ -173,6 +184,8 @@ export interface ResolvedPostgresConfig { readonly dataDir: string; readonly version: string; readonly autoExposeNewTables: boolean; + readonly provisioned?: boolean; + readonly profile?: "default" | "micro"; } export interface ResolvedPostgrestConfig { @@ -549,7 +562,8 @@ export class StackBuilder extends Context.Service< postgresResolution, dockerServicesEnabled, ); - const hasPostgresInit = postgresResolution.type === "binary"; + const hasPostgresInit = + postgresResolution.type === "binary" && config.postgres.provisioned !== true; const postgresDeps = dependsOnPostgres(hasPostgresInit); const jwtJwks = generateJwks(config.jwtSecret); @@ -562,6 +576,7 @@ export class StackBuilder extends Context.Service< port: config.dbPort, dockerAccessible: needsDockerAccess, cleanupDataDirOnExit: hasAutoManagedPath(config, config.postgres.dataDir), + profile: config.postgres.profile, }) : makePostgresServiceDocker({ image: postgresResolution.image, diff --git a/packages/stack/src/createStack.ts b/packages/stack/src/createStack.ts index 0684ac867b..20e460a7fe 100644 --- a/packages/stack/src/createStack.ts +++ b/packages/stack/src/createStack.ts @@ -560,6 +560,8 @@ export async function resolveConfig( dataDir: postgresDataDir, version: postgresInput.version ?? DEFAULT_VERSIONS.postgres, autoExposeNewTables: postgresInput.autoExposeNewTables ?? true, + provisioned: postgresInput.provisioned, + profile: postgresInput.profile, }, postgrest: resolvePostgrestConfig(postgrestInput, config.postgrest, ports), auth: resolveAuthConfig(authInput, config.auth, ports, ports.apiPort), diff --git a/packages/stack/src/services/postgres.ts b/packages/stack/src/services/postgres.ts index bb3b5953f7..3a21cb3280 100644 --- a/packages/stack/src/services/postgres.ts +++ b/packages/stack/src/services/postgres.ts @@ -16,6 +16,12 @@ interface NativePostgresOptions extends PostgresServiceOptions { readonly binPath: string; /** When true, patches postgres to listen on all interfaces so Docker containers can connect. */ readonly dockerAccessible?: boolean; + /** + * "micro": settings live in micro.conf/pod.conf inside PGDATA, so no `-c` runtime args are + * passed (command-line `-c` would override users' `ALTER SYSTEM` changes on every boot). + * Absent or "default": current `-c` args are passed unchanged. + */ + readonly profile?: "default" | "micro"; } interface DockerPostgresOptions extends PostgresServiceOptions { @@ -50,6 +56,14 @@ const NATIVE_POSTGRES_RUNTIME_ARGS = [ "max_replication_slots=5", ] as const; +/** + * On the "micro" profile, these settings live in micro.conf/pod.conf inside PGDATA instead, + * so no `-c` runtime args are emitted; passing them on the command line would take precedence + * over the conf files and override users' `ALTER SYSTEM` changes on every restart. + */ +const runtimeArgsForProfile = (profile?: "default" | "micro"): readonly string[] => + profile === "micro" ? [] : NATIVE_POSTGRES_RUNTIME_ARGS; + const orphanCleanup = (opts: PostgresServiceOptions) => opts.cleanupDataDirOnExit ? removePathOnOrphanCleanup(opts.dataDir) : []; @@ -141,7 +155,7 @@ export const makePostgresService = (opts: NativePostgresOptions): ServiceDef => initScript, "-p", String(opts.port), - ...NATIVE_POSTGRES_RUNTIME_ARGS, + ...runtimeArgsForProfile(opts.profile), "-c", "listen_addresses=*", "-c", @@ -163,7 +177,7 @@ export const makePostgresService = (opts: NativePostgresOptions): ServiceDef => return { name: "postgres", command: "bash", - args: [initScript, "-p", String(opts.port), ...NATIVE_POSTGRES_RUNTIME_ARGS], + args: [initScript, "-p", String(opts.port), ...runtimeArgsForProfile(opts.profile)], env: postgresEnv(opts), healthCheck: postgresHealthCheck(opts.binPath, opts.port), shutdown: { signal: "SIGTERM", timeoutSeconds: 10 }, diff --git a/packages/stack/src/versions.ts b/packages/stack/src/versions.ts index 14666bfcc9..2313ba7b8c 100644 --- a/packages/stack/src/versions.ts +++ b/packages/stack/src/versions.ts @@ -46,7 +46,7 @@ export interface VersionManifest { } export const DEFAULT_VERSIONS: VersionManifest = { - postgres: "17.6.1.142", + postgres: "17.6.1.143", postgrest: "14.14", auth: "2.192.0", "edge-runtime": "1.74.2", diff --git a/packages/stack/tests/helpers/buildServices.ts b/packages/stack/tests/helpers/buildServices.ts new file mode 100644 index 0000000000..188db80cd8 --- /dev/null +++ b/packages/stack/tests/helpers/buildServices.ts @@ -0,0 +1,162 @@ +import type { ServiceDef } from "@supabase/process-compose"; +import { Deferred, Effect, Layer, Sink, Stream } from "effect"; +import { ChildProcessSpawner } from "effect/unstable/process"; +import { defaultPublishableKey, defaultSecretKey, generateJwt } from "../../src/JwtGenerator.ts"; +import type { ResolvedStackConfig, StackConfig } from "../../src/StackBuilder.ts"; +import { + enabledServicesForConfig, + StackBuilder, + versionsForConfig, +} from "../../src/StackBuilder.ts"; +import type { StackPreparationInput } from "../../src/StackPreparation.ts"; +import { StackPreparation } from "../../src/StackPreparation.ts"; +import { DEFAULT_VERSIONS } from "../../src/versions.ts"; +import { mockBinaryResolver } from "./mocks.ts"; + +const encoder = new TextEncoder(); + +/** + * Mirrors the local helper of the same name in `src/StackBuilder.unit.test.ts` / + * `src/prefetch.unit.test.ts`. Not centralized in `mocks.ts` to avoid touching those + * pre-existing test files for this change; only the exit code matters here since the + * fixtures below never enable any docker-only service (no registry pulls happen). + */ +function mockSequenceSpawner( + results: ReadonlyArray<{ readonly exitCode: number; readonly stderr?: string[] }>, +) { + let index = 0; + return Layer.succeed( + ChildProcessSpawner.ChildProcessSpawner, + ChildProcessSpawner.make((_command) => + Effect.gen(function* () { + const result = results[index] ?? { exitCode: 0 }; + index += 1; + const exitDeferred = yield* Deferred.make(); + yield* Deferred.succeed(exitDeferred, ChildProcessSpawner.ExitCode(result.exitCode)); + + return ChildProcessSpawner.makeHandle({ + pid: ChildProcessSpawner.ProcessId(3000 + index), + stdout: Stream.empty, + stderr: Stream.fromIterable( + (result.stderr ?? []).map((line) => encoder.encode(`${line}\n`)), + ), + all: Stream.empty, + exitCode: Deferred.await(exitDeferred), + isRunning: Effect.succeed(true), + stdin: Sink.drain, + kill: () => Effect.void, + unref: Effect.succeed(Effect.void), + getInputFd: () => Sink.drain, + getOutputFd: () => Stream.empty, + }); + }), + ), + ); +} + +const testJwtSecret = "super-secret-jwt-token-with-at-least-32-characters"; + +const basePorts = { + apiPort: 3000, + dbPort: 5432, + authPort: 9999, + postgrestPort: 3001, + postgrestAdminPort: 3002, + edgeRuntimePort: 3003, + edgeRuntimeInspectorPort: 3004, + realtimePort: 3010, + storagePort: 3011, + imgproxyPort: 3012, + mailpitPort: 3013, + mailpitSmtpPort: 3014, + mailpitPop3Port: 3015, + pgmetaPort: 3016, + studioPort: 3017, + analyticsPort: 3018, + poolerPort: 3019, + poolerApiPort: 3020, +}; + +/** + * Minimal ResolvedStackConfig fixture with every optional service disabled. Mirrors + * `baseConfig` in `src/StackBuilder.unit.test.ts`; kept in sync manually since the two + * currently have no shared export. + */ +const baseConfig: ResolvedStackConfig = { + cacheRoot: "/tmp/supabase-cache", + stackRoot: "/tmp/supabase-stack", + runtimeRoot: "/tmp/supabase-runtime", + projectDir: "/tmp/supabase-project", + mode: "auto", + jwtSecret: testJwtSecret, + ports: basePorts, + apiPort: 3000, + dbPort: 5432, + publishableKey: defaultPublishableKey, + secretKey: defaultSecretKey, + functions: false, + autoManagedPaths: [], + anonJwt: generateJwt(testJwtSecret, "anon"), + serviceRoleJwt: generateJwt(testJwtSecret, "service_role"), + postgres: { + port: 5432, + dataDir: "/tmp/pg-data", + version: DEFAULT_VERSIONS.postgres, + autoExposeNewTables: true, + }, + postgrest: false, + auth: false, + edgeRuntime: false, + realtime: false, + storage: false, + imgproxy: false, + mailpit: false, + pgmeta: false, + studio: false, + analytics: false, + vector: false, + pooler: false, +}; + +/** + * Builds the ServiceDef list produced by `StackBuilder.build()` for a partial + * `StackConfig`, with every service other than postgres disabled by default and no real + * process spawning (BinaryResolver and ChildProcessSpawner are both mocked). Only the + * `postgres` sub-config is honored from the provided partial config; this stays intentionally + * narrow because it currently only backs the `provisioned`/`profile` wiring tests. + */ +export async function buildServicesForTest( + partial: Pick, +): Promise> { + const config: ResolvedStackConfig = { + ...baseConfig, + postgres: { + ...baseConfig.postgres, + ...partial.postgres, + }, + }; + + const resolver = mockBinaryResolver(); + const layer = Layer.mergeAll( + StackBuilder.layer, + StackPreparation.layer.pipe( + Layer.provide(resolver.layer), + Layer.provide(mockSequenceSpawner([{ exitCode: 0 }])), + ), + ); + + const program = Effect.gen(function* () { + const builder = yield* StackBuilder; + const preparation = yield* StackPreparation; + const input: StackPreparationInput = { + mode: config.mode, + services: enabledServicesForConfig(config), + versions: versionsForConfig(config), + }; + const prepared = yield* preparation.prepare(input); + const { graph } = yield* builder.build(config, prepared); + return graph.startOrder; + }).pipe(Effect.provide(layer)); + + return Effect.runPromise(program as Effect.Effect, unknown>); +} From 46f5d43eb57350793d53e5762fdacfe1f4f95b66 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 11:54:47 +0200 Subject: [PATCH 09/38] feat(stack): enableExtension with preload-on-enable restart Adds enableExtension(name) across the coordinator, Stack service, RemoteStack/DaemonServer HTTP surface, and public StackHandle. When an extension requires shared_preload_libraries (per PRELOAD_REQUIRED_EXTENSIONS) and isn't already preloaded, it appends to pod.conf and restarts postgres; otherwise it's a no-op so plain CREATE EXTENSION keeps working. Co-Authored-By: Claude Fable 5 --- .../src/DaemonServer.integration.test.ts | 6 +++++ packages/stack/src/DaemonServer.ts | 22 +++++++++++++++++++ .../stack/src/RemoteStack.integration.test.ts | 13 +++++++++++ packages/stack/src/RemoteStack.ts | 19 ++++++++++++++++ packages/stack/src/Stack.ts | 4 ++++ .../stack/src/StackLifecycleCoordinator.ts | 22 +++++++++++++++++++ .../src/UnixSocketSse.integration.test.ts | 2 ++ packages/stack/src/createStack.ts | 2 ++ packages/stack/src/enableExtension.ts | 14 ++++++++++++ .../stack/src/enableExtension.unit.test.ts | 17 ++++++++++++++ 10 files changed, 121 insertions(+) create mode 100644 packages/stack/src/enableExtension.ts create mode 100644 packages/stack/src/enableExtension.unit.test.ts diff --git a/packages/stack/src/DaemonServer.integration.test.ts b/packages/stack/src/DaemonServer.integration.test.ts index 5e949f64f6..1f2ea9e431 100644 --- a/packages/stack/src/DaemonServer.integration.test.ts +++ b/packages/stack/src/DaemonServer.integration.test.ts @@ -76,6 +76,12 @@ function mockStack() { : Effect.sync(() => { serviceCalls.push(`restart:${name}`); }), + enableExtension: (name: string) => + name === "unknown" + ? Effect.fail(new ServiceNotFoundError({ name })) + : Effect.sync(() => { + serviceCalls.push(`enable-extension:${name}`); + }), reloadFunctions: () => Effect.sync(() => { serviceCalls.push("reload-functions"); diff --git a/packages/stack/src/DaemonServer.ts b/packages/stack/src/DaemonServer.ts index 4dca711302..a9ec89d821 100644 --- a/packages/stack/src/DaemonServer.ts +++ b/packages/stack/src/DaemonServer.ts @@ -207,6 +207,28 @@ export class DaemonServer extends Context.Service< ), ), + HttpRouter.route( + "POST", + "/extensions/:name/enable", + Effect.gen(function* () { + const routeParams = yield* HttpRouter.params; + yield* stack.enableExtension(routeParams.name!); + return HttpServerResponse.jsonUnsafe({ ok: true }); + }).pipe( + Effect.catchTag("ServiceNotFoundError", (e) => + Effect.succeed( + HttpServerResponse.jsonUnsafe( + { error: `Service not found: ${e.name}` }, + { status: 404 }, + ), + ), + ), + Effect.catchTag("ServiceReadyError", (e) => + Effect.succeed(HttpServerResponse.jsonUnsafe({ error: e.reason }, { status: 500 })), + ), + ), + ), + HttpRouter.route( "POST", "/functions/reload", diff --git a/packages/stack/src/RemoteStack.integration.test.ts b/packages/stack/src/RemoteStack.integration.test.ts index f7581f902d..e96b41dfc5 100644 --- a/packages/stack/src/RemoteStack.integration.test.ts +++ b/packages/stack/src/RemoteStack.integration.test.ts @@ -86,6 +86,12 @@ function mockStack() { : Effect.sync(() => { serviceCalls.push(`restart:${name}`); }), + enableExtension: (name: string) => + name === "unknown" + ? Effect.fail(new ServiceNotFoundError({ name })) + : Effect.sync(() => { + serviceCalls.push(`enable-extension:${name}`); + }), reloadFunctions: () => Effect.sync(() => { serviceCalls.push("reload-functions"); @@ -228,6 +234,13 @@ describe("RemoteStack integration", () => { ); if (res.status === 404) return yield* new ServiceNotFoundError({ name }); }), + enableExtension: (name: string) => + Effect.gen(function* () { + const res = yield* Effect.promise(() => + fetch(`${url}/extensions/${name}/enable`, { method: "POST" }), + ); + if (res.status === 404) return yield* new ServiceNotFoundError({ name }); + }), reloadFunctions: () => Effect.gen(function* () { yield* Effect.promise(() => fetch(`${url}/functions/reload`, { method: "POST" })); diff --git a/packages/stack/src/RemoteStack.ts b/packages/stack/src/RemoteStack.ts index 00803d7444..61132817e5 100644 --- a/packages/stack/src/RemoteStack.ts +++ b/packages/stack/src/RemoteStack.ts @@ -278,6 +278,25 @@ export const RemoteStack = { }), ), + enableExtension: (name: string) => + withUnixHttpClient( + Effect.gen(function* () { + const response = yield* unixResponse(socketPath, `/extensions/${name}/enable`, { + method: "POST", + }); + if (response.status === 404) { + return yield* new ServiceNotFoundError({ name }); + } + if (response.status === 500) { + const body = yield* HttpClientResponse.schemaBodyJson(ServiceErrorResponseSchema)( + response, + ).pipe(Effect.orDie); + return yield* new ServiceReadyError({ name, reason: body.error }); + } + yield* HttpClientResponse.filterStatusOk(response).pipe(Effect.orDie); + }), + ), + reloadFunctions: (opts) => withUnixHttpClient( Effect.gen(function* () { diff --git a/packages/stack/src/Stack.ts b/packages/stack/src/Stack.ts index 27b53f7c14..ef0cdf72ad 100644 --- a/packages/stack/src/Stack.ts +++ b/packages/stack/src/Stack.ts @@ -67,6 +67,9 @@ export class Stack extends Context.Service< readonly restartService: ( name: string, ) => Effect.Effect; + readonly enableExtension: ( + name: string, + ) => Effect.Effect; readonly reloadFunctions: ( opts?: FunctionsConfig, ) => Effect.Effect; @@ -107,6 +110,7 @@ export class Stack extends Context.Service< startService: coordinator.startService, stopService: coordinator.stopService, restartService: coordinator.restartService, + enableExtension: coordinator.enableExtension, reloadFunctions: coordinator.reloadFunctions, reloadEdgeRuntime: coordinator.reloadEdgeRuntime, getState: coordinator.getState, diff --git a/packages/stack/src/StackLifecycleCoordinator.ts b/packages/stack/src/StackLifecycleCoordinator.ts index 0a8ba5bee1..a287684ef4 100644 --- a/packages/stack/src/StackLifecycleCoordinator.ts +++ b/packages/stack/src/StackLifecycleCoordinator.ts @@ -15,9 +15,11 @@ import { import { ChildProcessSpawner } from "effect/unstable/process"; import type { CleanupTargets } from "./CleanupTargets.ts"; import { cleanupLocalStackResources } from "./cleanup.ts"; +import { planEnableExtension } from "./enableExtension.ts"; import { StackBuildError } from "./errors.ts"; import { configureFunctionsRuntime, type FunctionsConfig } from "./functions.ts"; import { detectPlatform, dockerHostAddress } from "./Platform.ts"; +import { readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; import { StackMetadataPersistence } from "./StackMetadataPersistence.ts"; import { StackPreparation } from "./StackPreparation.ts"; import type { PreparedStackArtifacts } from "./StackPreparation.ts"; @@ -145,6 +147,9 @@ export class StackLifecycleCoordinator extends Context.Service< readonly restartService: ( name: string, ) => Effect.Effect; + readonly enableExtension: ( + name: string, + ) => Effect.Effect; readonly reloadFunctions: ( opts?: FunctionsConfig, ) => Effect.Effect; @@ -554,6 +559,23 @@ export class StackLifecycleCoordinator extends Context.Service< const runtime = yield* ensureRuntime; yield* runtime.orchestrator.restartService(name); }), + enableExtension: (name) => + Effect.gen(function* () { + const currentLibraries = yield* Effect.promise(() => + readPreloadLibraries(config.postgres.dataDir), + ); + const plan = planEnableExtension(name, currentLibraries); + if (plan.action === "none") { + return; + } + yield* Effect.promise(() => + writePreloadLibraries(config.postgres.dataDir, plan.libraries), + ); + yield* requireKnownService("postgres"); + const runtime = yield* ensureRuntime; + yield* runtime.orchestrator.restartService("postgres"); + yield* runtime.orchestrator.waitReady("postgres"); + }), reloadFunctions: (opts) => Effect.gen(function* () { yield* requireKnownService("edge-runtime"); diff --git a/packages/stack/src/UnixSocketSse.integration.test.ts b/packages/stack/src/UnixSocketSse.integration.test.ts index e6694814da..16875deb0e 100644 --- a/packages/stack/src/UnixSocketSse.integration.test.ts +++ b/packages/stack/src/UnixSocketSse.integration.test.ts @@ -67,6 +67,8 @@ function makeStackLayer(opts: { name === "postgres" ? Effect.void : Effect.fail(new ServiceNotFoundError({ name })), restartService: (name: string) => name === "postgres" ? Effect.void : Effect.fail(new ServiceNotFoundError({ name })), + enableExtension: (name: string) => + name === "postgres" ? Effect.void : Effect.fail(new ServiceNotFoundError({ name })), reloadFunctions: () => Effect.void, reloadEdgeRuntime: () => Effect.void, getState: (name: string) => diff --git a/packages/stack/src/createStack.ts b/packages/stack/src/createStack.ts index 20e460a7fe..ad12d51585 100644 --- a/packages/stack/src/createStack.ts +++ b/packages/stack/src/createStack.ts @@ -98,6 +98,7 @@ export interface StackHandle extends AsyncDisposable { startService(name: string): Promise; stopService(name: string): Promise; restartService(name: string): Promise; + enableExtension(name: string): Promise; reloadFunctions(opts?: FunctionsConfig): Promise; reloadEdgeRuntime(opts: EdgeRuntimeReloadConfig): Promise; ready(opts?: ReadyOptions): Promise; @@ -730,6 +731,7 @@ export async function createStack( startService: (name) => run(localStack.startService(name)), stopService: (name) => run(localStack.stopService(name)), restartService: (name) => run(localStack.restartService(name)), + enableExtension: (name) => run(localStack.enableExtension(name)), reloadFunctions: (opts) => run(localStack.reloadFunctions(opts)), reloadEdgeRuntime: (opts) => run(localStack.reloadEdgeRuntime(opts)), ready: (opts) => { diff --git a/packages/stack/src/enableExtension.ts b/packages/stack/src/enableExtension.ts new file mode 100644 index 0000000000..fc0fec99b3 --- /dev/null +++ b/packages/stack/src/enableExtension.ts @@ -0,0 +1,14 @@ +import { PRELOAD_REQUIRED_EXTENSIONS } from "./micro.ts"; + +export type EnableExtensionPlan = + | { readonly action: "none" } + | { readonly action: "restart"; readonly libraries: ReadonlyArray }; + +export const planEnableExtension = ( + name: string, + currentLibraries: ReadonlyArray, +): EnableExtensionPlan => { + if (!PRELOAD_REQUIRED_EXTENSIONS.has(name)) return { action: "none" }; + if (currentLibraries.includes(name)) return { action: "none" }; + return { action: "restart", libraries: [...currentLibraries, name] }; +}; diff --git a/packages/stack/src/enableExtension.unit.test.ts b/packages/stack/src/enableExtension.unit.test.ts new file mode 100644 index 0000000000..f231dffa9b --- /dev/null +++ b/packages/stack/src/enableExtension.unit.test.ts @@ -0,0 +1,17 @@ +import { describe, expect, it } from "vitest"; +import { planEnableExtension } from "./enableExtension.ts"; + +describe("planEnableExtension", () => { + it("no-ops for extensions that do not need preload", () => { + expect(planEnableExtension("pgvector", [])).toEqual({ action: "none" }); + }); + it("no-ops when already preloaded", () => { + expect(planEnableExtension("pg_cron", ["pg_cron"])).toEqual({ action: "none" }); + }); + it("appends and restarts otherwise", () => { + expect(planEnableExtension("pg_cron", ["pg_net"])).toEqual({ + action: "restart", + libraries: ["pg_net", "pg_cron"], + }); + }); +}); From 99816b778c554d6007c3d057d9c87b12bf9c73b2 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 12:12:03 +0200 Subject: [PATCH 10/38] fix(stack): serialize enableExtension to prevent pod.conf write races Two concurrent enableExtension calls could both read the same shared_preload_libraries list, then write independently, silently dropping one extension when the second write clobbered the first. Concurrent restarts of the same "postgres" service also deadlocked the orchestrator. Guard the whole enableExtension body with a per-coordinator Effect Semaphore(1) so calls are serialized. Add a regression test that fires two enableExtension calls concurrently through the real StackLifecycleCoordinator (mocked binary resolver/spawner, temp PGDATA) and asserts both extensions land in pod.conf; without the fix the test times out instead of completing. --- .../stack/src/StackLifecycleCoordinator.ts | 36 +++-- .../StackLifecycleCoordinator.unit.test.ts | 141 ++++++++++++++++++ 2 files changed, 161 insertions(+), 16 deletions(-) create mode 100644 packages/stack/src/StackLifecycleCoordinator.unit.test.ts diff --git a/packages/stack/src/StackLifecycleCoordinator.ts b/packages/stack/src/StackLifecycleCoordinator.ts index a287684ef4..2515c05139 100644 --- a/packages/stack/src/StackLifecycleCoordinator.ts +++ b/packages/stack/src/StackLifecycleCoordinator.ts @@ -9,6 +9,7 @@ import { Path, Ref, Context, + Semaphore, Stream, SubscriptionRef, } from "effect"; @@ -201,6 +202,7 @@ export class StackLifecycleCoordinator extends Context.Service< const info = stackInfoFor(config); const stateRef = yield* SubscriptionRef.make(initialPublicStates(config)); const phaseRef = yield* Ref.make("idle"); + const enableExtensionLock = yield* Semaphore.make(1); const logBufferServices = yield* Layer.buildWithScope(LogBuffer.layer, scope); const logBuffer = Context.get(logBufferServices, LogBuffer); @@ -560,22 +562,24 @@ export class StackLifecycleCoordinator extends Context.Service< yield* runtime.orchestrator.restartService(name); }), enableExtension: (name) => - Effect.gen(function* () { - const currentLibraries = yield* Effect.promise(() => - readPreloadLibraries(config.postgres.dataDir), - ); - const plan = planEnableExtension(name, currentLibraries); - if (plan.action === "none") { - return; - } - yield* Effect.promise(() => - writePreloadLibraries(config.postgres.dataDir, plan.libraries), - ); - yield* requireKnownService("postgres"); - const runtime = yield* ensureRuntime; - yield* runtime.orchestrator.restartService("postgres"); - yield* runtime.orchestrator.waitReady("postgres"); - }), + enableExtensionLock.withPermits(1)( + Effect.gen(function* () { + const currentLibraries = yield* Effect.promise(() => + readPreloadLibraries(config.postgres.dataDir), + ); + const plan = planEnableExtension(name, currentLibraries); + if (plan.action === "none") { + return; + } + yield* Effect.promise(() => + writePreloadLibraries(config.postgres.dataDir, plan.libraries), + ); + yield* requireKnownService("postgres"); + const runtime = yield* ensureRuntime; + yield* runtime.orchestrator.restartService("postgres"); + yield* runtime.orchestrator.waitReady("postgres"); + }), + ), reloadFunctions: (opts) => Effect.gen(function* () { yield* requireKnownService("edge-runtime"); diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts new file mode 100644 index 0000000000..429c3094e3 --- /dev/null +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -0,0 +1,141 @@ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "@effect/vitest"; +import { NodeServices } from "@effect/platform-node"; +import { Effect, Layer } from "effect"; +import { mockChildProcessSpawner } from "../../process-compose/tests/helpers/mocks.ts"; +import { mockBinaryResolver } from "../tests/helpers/mocks.ts"; +import { defaultPublishableKey, defaultSecretKey, generateJwt } from "./JwtGenerator.ts"; +import { readPreloadLibraries } from "./pgconf.ts"; +import type { AllocatedPorts } from "./PortAllocator.ts"; +import { Stack } from "./Stack.ts"; +import { StackLifecycleCoordinator } from "./StackLifecycleCoordinator.ts"; +import { StackMetadataPersistence } from "./StackMetadataPersistence.ts"; +import { StackPreparation } from "./StackPreparation.ts"; +import { StackBuilder } from "./StackBuilder.ts"; +import type { ResolvedStackConfig } from "./StackBuilder.ts"; +import { DEFAULT_VERSIONS } from "./versions.ts"; + +const testJwtSecret = "super-secret-jwt-token-with-at-least-32-characters-long"; + +const defaultPorts: AllocatedPorts = { + apiPort: 54321, + dbPort: 54322, + authPort: 9999, + postgrestPort: 54323, + postgrestAdminPort: 54324, + edgeRuntimePort: 54325, + edgeRuntimeInspectorPort: 54326, + realtimePort: 54330, + storagePort: 54331, + imgproxyPort: 54332, + mailpitPort: 54333, + mailpitSmtpPort: 54334, + mailpitPop3Port: 54335, + pgmetaPort: 54336, + studioPort: 54337, + analyticsPort: 54338, + poolerPort: 54339, + poolerApiPort: 54340, +}; + +function makeConfig(dataDir: string): ResolvedStackConfig { + return { + cacheRoot: "/tmp/supabase-cache", + stackRoot: "/tmp/supabase-stack", + runtimeRoot: "/tmp/supabase-runtime", + projectDir: "/tmp/supabase-project", + mode: "native", + jwtSecret: testJwtSecret, + ports: defaultPorts, + apiPort: 54321, + dbPort: 54322, + publishableKey: defaultPublishableKey, + secretKey: defaultSecretKey, + functions: false, + autoManagedPaths: [], + anonJwt: generateJwt(testJwtSecret, "anon"), + serviceRoleJwt: generateJwt(testJwtSecret, "service_role"), + postgres: { + port: 54322, + dataDir, + version: DEFAULT_VERSIONS.postgres, + autoExposeNewTables: true, + }, + // postgrest/auth are disabled: their health checks are real HTTP probes + // (unlike postgres's Exec-based pg_isready probe, which the mocked + // ChildProcessSpawner satisfies), so nothing in this mock setup would ever + // answer them and waitAllReady would hang/fail. Keeping only postgres + // enabled is sufficient to exercise the enableExtension race. + postgrest: false, + auth: false, + edgeRuntime: false, + realtime: false, + storage: false, + imgproxy: false, + mailpit: false, + pgmeta: false, + studio: false, + analytics: false, + vector: false, + pooler: false, + }; +} + +function setupLayer(config: ResolvedStackConfig) { + const resolver = mockBinaryResolver(); + const spawner = mockChildProcessSpawner(); + const stackPreparationLayer = StackPreparation.layer.pipe(Layer.provide(resolver.layer)); + const coordinatorLayer = StackLifecycleCoordinator.layer(config).pipe( + Layer.provide(StackBuilder.layer), + Layer.provide(stackPreparationLayer), + Layer.provide(StackMetadataPersistence.noop), + ); + + const layer = Stack.layer(config).pipe( + Layer.provide(coordinatorLayer), + Layer.provide(spawner.layer), + Layer.provide(NodeServices.layer), + ); + + return { layer }; +} + +describe("StackLifecycleCoordinator enableExtension", () => { + // Regression test: two concurrent enableExtension calls used to race on + // pod.conf — both read the same preload list, both wrote independently, and + // the second write clobbered the first, silently dropping one extension + // from shared_preload_libraries. Racing the two restarts of "postgres" also + // deadlocks the orchestrator (concurrent FiberMap.run + stopForRestart calls + // on the same service name step on each other), so without the fix this + // test times out rather than merely asserting the wrong libraries. + // enableExtension is now serialized per coordinator instance with an Effect + // Semaphore, which fixes both the lost write and the restart deadlock. + // + // it.live is required (not it.effect/TestClock): the mock ChildProcessSpawner + // resolves exit codes and the postgres health-check probe via a real + // `Effect.sleep("10 millis")`, which needs the real clock to progress. + it.live("serializes concurrent enableExtension calls so no write is lost", () => { + const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-test-")); + const config = makeConfig(dataDir); + const { layer } = setupLayer(config); + + return Effect.gen(function* () { + const stack = yield* Stack; + yield* stack.start(); + + yield* Effect.all([stack.enableExtension("pg_cron"), stack.enableExtension("pg_net")], { + concurrency: "unbounded", + }); + + const libraries = yield* Effect.promise(() => readPreloadLibraries(dataDir)); + expect(libraries).toContain("pg_cron"); + expect(libraries).toContain("pg_net"); + expect(libraries).toHaveLength(2); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), + ); + }); +}); From 20d89da3ee654e4b9453f186c0ff5f66ab07085f Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 12:44:50 +0200 Subject: [PATCH 11/38] feat(stack): lazy per-service start behind the API proxy Adds StackConfig.lazyServices: when true, StackLifecycleCoordinator.start() only eagerly starts postgres (and postgres-init); every other service starts on demand the first time the ApiProxy routes a request to it, via a new ProxyConfig.ensureService hook memoized by lazyServices.ts so concurrent first requests trigger a single start. Deviates from the original sketch's `enabled: false` mechanic: process-compose's buildGraph excludes disabled ServiceDefs from the dependency graph entirely, so a lazily-disabled service can never be resurrected via startService/ updateServiceDefinition once the orchestrator is built. Services stay enabled in the graph; laziness is enforced by skipping the eager start in the coordinator instead. --- packages/stack/src/ApiProxy.ts | 37 +++++++ packages/stack/src/ApiProxy.unit.test.ts | 64 ++++++++++++ packages/stack/src/Stack.unit.test.ts | 1 + packages/stack/src/StackBuilder.ts | 17 ++++ packages/stack/src/StackBuilder.unit.test.ts | 27 ++++++ .../stack/src/StackLifecycleCoordinator.ts | 57 ++++++++++- .../StackLifecycleCoordinator.unit.test.ts | 93 +++++++++++++++++- packages/stack/src/createStack.ts | 1 + packages/stack/src/layers.ts | 97 ++++++++++++------- packages/stack/src/lazyServices.ts | 33 +++++++ packages/stack/src/lazyServices.unit.test.ts | 45 +++++++++ packages/stack/tests/helpers/buildServices.ts | 1 + 12 files changed, 432 insertions(+), 41 deletions(-) create mode 100644 packages/stack/src/lazyServices.ts create mode 100644 packages/stack/src/lazyServices.unit.test.ts diff --git a/packages/stack/src/ApiProxy.ts b/packages/stack/src/ApiProxy.ts index 12f258235e..837f826789 100644 --- a/packages/stack/src/ApiProxy.ts +++ b/packages/stack/src/ApiProxy.ts @@ -9,6 +9,7 @@ import { HttpServerRequest, HttpServerResponse, } from "effect/unstable/http"; +import type { ServiceName } from "./versions.ts"; export interface ProxyConfig { readonly listenPort: number; @@ -26,6 +27,12 @@ export interface ProxyConfig { readonly secretKey: string; readonly anonJwt: string; readonly serviceRoleJwt: string; + /** + * When set (lazyServices mode), invoked with a route's owning service before the request is + * forwarded. Expected to be idempotent — it should start the service on first call and resolve + * immediately on later calls once the service is ready. + */ + readonly ensureService?: (name: ServiceName) => Promise; } function transformAuthorization( @@ -114,6 +121,7 @@ const COLD_START_RETRY_SCHEDULE = Schedule.spaced("250 millis").pipe(Schedule.ta interface ProxyHandlerOptions { readonly backendPort: number; + readonly service: ServiceName; readonly stripPrefix?: string; readonly backendPath?: string; readonly transformAuth?: boolean; @@ -132,6 +140,18 @@ function makeProxyHandler( ) { return (req: HttpServerRequest.HttpServerRequest) => Effect.gen(function* () { + if (config.ensureService) { + const ensured = yield* Effect.result( + Effect.tryPromise(() => config.ensureService!(opts.service)), + ); + if (Result.isFailure(ensured)) { + return HttpServerResponse.text( + `Bad gateway: failed to start ${opts.service}: ${String(ensured.failure)}`, + { status: 502 }, + ); + } + } + let backendPath = opts.backendPath; if (backendPath === undefined) { @@ -222,6 +242,7 @@ export class ApiProxy extends Context.Service< "/.well-known/oauth-authorization-server", makeProxyHandler(client, config, { backendPort: config.gotruePort, + service: "auth", backendPath: "/.well-known/oauth-authorization-server", }), ), @@ -230,6 +251,7 @@ export class ApiProxy extends Context.Service< "/auth/v1/verify", makeProxyHandler(client, config, { backendPort: config.gotruePort, + service: "auth", stripPrefix: "/auth/v1", }), ), @@ -238,6 +260,7 @@ export class ApiProxy extends Context.Service< "/auth/v1/callback", makeProxyHandler(client, config, { backendPort: config.gotruePort, + service: "auth", stripPrefix: "/auth/v1", }), ), @@ -246,6 +269,7 @@ export class ApiProxy extends Context.Service< "/auth/v1/authorize", makeProxyHandler(client, config, { backendPort: config.gotruePort, + service: "auth", stripPrefix: "/auth/v1", }), ), @@ -254,6 +278,7 @@ export class ApiProxy extends Context.Service< "/auth/v1/*", makeProxyHandler(client, config, { backendPort: config.gotruePort, + service: "auth", stripPrefix: "/auth/v1", transformAuth: true, }), @@ -263,6 +288,7 @@ export class ApiProxy extends Context.Service< "/rest/v1/*", makeProxyHandler(client, config, { backendPort: config.postgrestPort, + service: "postgrest", stripPrefix: "/rest/v1", transformAuth: true, }), @@ -272,6 +298,7 @@ export class ApiProxy extends Context.Service< "/rest-admin/v1/*", makeProxyHandler(client, config, { backendPort: config.postgrestAdminPort, + service: "postgrest", stripPrefix: "/rest-admin/v1", }), ), @@ -280,6 +307,7 @@ export class ApiProxy extends Context.Service< "/graphql/v1", makeProxyHandler(client, config, { backendPort: config.postgrestPort, + service: "postgrest", backendPath: "/rpc/graphql", transformAuth: true, extraHeaders: { "content-profile": "graphql_public" }, @@ -290,6 +318,7 @@ export class ApiProxy extends Context.Service< "/functions/v1/*", makeProxyHandler(client, config, { backendPort: config.edgeRuntimePort, + service: "edge-runtime", stripPrefix: "/functions/v1", transformAuth: true, transformAuthCustomHeader: true, @@ -301,6 +330,7 @@ export class ApiProxy extends Context.Service< "/realtime/v1/api/*", makeProxyHandler(client, config, { backendPort: config.realtimePort, + service: "realtime", stripPrefix: "/realtime/v1", transformAuth: true, }), @@ -310,6 +340,7 @@ export class ApiProxy extends Context.Service< "/realtime/v1/*", makeProxyHandler(client, config, { backendPort: config.realtimePort, + service: "realtime", stripPrefix: "/realtime/v1", }), ), @@ -318,6 +349,7 @@ export class ApiProxy extends Context.Service< "/storage/v1/s3/*", makeProxyHandler(client, config, { backendPort: config.storagePort, + service: "storage", stripPrefix: "/storage/v1", }), ), @@ -326,6 +358,7 @@ export class ApiProxy extends Context.Service< "/storage/v1/*", makeProxyHandler(client, config, { backendPort: config.storagePort, + service: "storage", stripPrefix: "/storage/v1", transformAuth: true, }), @@ -335,6 +368,7 @@ export class ApiProxy extends Context.Service< "/pg/*", makeProxyHandler(client, config, { backendPort: config.pgmetaPort, + service: "pgmeta", stripPrefix: "/pg", }), ), @@ -343,6 +377,7 @@ export class ApiProxy extends Context.Service< "/analytics/v1/*", makeProxyHandler(client, config, { backendPort: config.analyticsPort, + service: "analytics", stripPrefix: "/analytics/v1", }), ), @@ -351,6 +386,7 @@ export class ApiProxy extends Context.Service< "/pooler/v2/*", makeProxyHandler(client, config, { backendPort: config.poolerPort, + service: "pooler", stripPrefix: "/pooler", }), ), @@ -359,6 +395,7 @@ export class ApiProxy extends Context.Service< "/mcp", makeProxyHandler(client, config, { backendPort: config.studioPort, + service: "studio", backendPath: "/api/mcp", }), ), diff --git a/packages/stack/src/ApiProxy.unit.test.ts b/packages/stack/src/ApiProxy.unit.test.ts index d2b3a00bbc..d5f8873d18 100644 --- a/packages/stack/src/ApiProxy.unit.test.ts +++ b/packages/stack/src/ApiProxy.unit.test.ts @@ -488,4 +488,68 @@ describe("ApiProxy", () => { await echoBody.stop(); } }); + + // --------------------------------------------------------------------------- + // Lazy service start — ensureService + // --------------------------------------------------------------------------- + + describe("ensureService", () => { + test("is invoked with the route's owning service before forwarding", async () => { + const backend = await startEchoBackend(); + const calls: string[] = []; + const config: ProxyConfig = { + ...configForPort(backend.port), + ensureService: async (name) => { + calls.push(name); + }, + }; + const proxy = await startProxy(config); + try { + await fetch(`${proxy.url}/rest/v1/users`); + await fetch(`${proxy.url}/auth/v1/token`, { headers: { apikey: PUBLISHABLE_KEY } }); + expect(calls).toEqual(["postgrest", "auth"]); + } finally { + await proxy.dispose(); + await backend.stop(); + } + }); + + test("is not invoked for /health", async () => { + const backend = await startEchoBackend(); + const calls: string[] = []; + const config: ProxyConfig = { + ...configForPort(backend.port), + ensureService: async (name) => { + calls.push(name); + }, + }; + const proxy = await startProxy(config); + try { + const res = await fetch(`${proxy.url}/health`); + expect(res.status).toBe(200); + expect(calls).toEqual([]); + } finally { + await proxy.dispose(); + await backend.stop(); + } + }); + + test("returns 502 when ensureService rejects", async () => { + const backend = await startEchoBackend(); + const config: ProxyConfig = { + ...configForPort(backend.port), + ensureService: async () => { + throw new Error("failed to start"); + }, + }; + const proxy = await startProxy(config); + try { + const res = await fetch(`${proxy.url}/rest/v1/users`); + expect(res.status).toBe(502); + } finally { + await proxy.dispose(); + await backend.stop(); + } + }); + }); }); diff --git a/packages/stack/src/Stack.unit.test.ts b/packages/stack/src/Stack.unit.test.ts index ea98797efe..2fb2570294 100644 --- a/packages/stack/src/Stack.unit.test.ts +++ b/packages/stack/src/Stack.unit.test.ts @@ -44,6 +44,7 @@ const defaultConfig: ResolvedStackConfig = { projectDir: "/tmp/supabase-project", mode: "native", jwtSecret: testJwtSecret, + lazyServices: false, ports: defaultPorts, apiPort: 54321, dbPort: 54322, diff --git a/packages/stack/src/StackBuilder.ts b/packages/stack/src/StackBuilder.ts index 0f69f4a853..fad3e38147 100644 --- a/packages/stack/src/StackBuilder.ts +++ b/packages/stack/src/StackBuilder.ts @@ -161,6 +161,12 @@ export interface StackConfig { readonly mode?: "native" | "auto" | "docker"; readonly jwtSecret?: string; readonly port?: number; + /** + * When true, `start()` only eagerly starts `postgres` (and `postgres-init` when present); every + * other service is started on demand by the ApiProxy on the first request to its route, instead + * of starting everything up front. Default false: existing eager-start behavior is unchanged. + */ + readonly lazyServices?: boolean; readonly publishableKey?: string; readonly secretKey?: string; readonly functions?: FunctionsConfig | false; @@ -286,6 +292,7 @@ export interface ResolvedStackConfig { readonly projectDir: string; readonly mode: "native" | "auto" | "docker"; readonly jwtSecret: string; + readonly lazyServices: boolean; readonly ports: AllocatedPorts; readonly apiPort: number; readonly dbPort: number; @@ -567,6 +574,16 @@ export class StackBuilder extends Context.Service< const postgresDeps = dependsOnPostgres(hasPostgresInit); const jwtJwks = generateJwks(config.jwtSecret); + // Every service def stays enabled in the process-compose graph, including under + // lazyServices: process-compose's dependency graph excludes `enabled: false` defs + // entirely (they're never added as nodes), so a service disabled at build time can never + // later be started via `startService` — there's no supported "enable" path once the + // orchestrator is built. Instead, laziness is enforced one layer up: when + // config.lazyServices is on, StackLifecycleCoordinator.start() only eagerly starts + // postgres (and postgres-init); every other service is started on demand by the ApiProxy + // via `ensureService`, which calls the ordinary `startService` + `waitReady` coordinator + // methods against these same (enabled) defs. + const defs: Array = [ { ...(postgresResolution.type === "binary" diff --git a/packages/stack/src/StackBuilder.unit.test.ts b/packages/stack/src/StackBuilder.unit.test.ts index 261ccd277c..afe6a496e6 100644 --- a/packages/stack/src/StackBuilder.unit.test.ts +++ b/packages/stack/src/StackBuilder.unit.test.ts @@ -43,6 +43,7 @@ const baseConfig: ResolvedStackConfig = { projectDir: "/tmp/supabase-project", mode: "auto", jwtSecret: testJwtSecret, + lazyServices: false, ports: basePorts, apiPort: 3000, dbPort: 5432, @@ -298,6 +299,32 @@ describe("StackBuilder", () => { }).pipe(Effect.provide(layer)); }); + it.effect("lazyServices still builds every service enabled in the graph", () => { + // process-compose's dependency graph excludes `enabled: false` defs entirely (they're never + // added as nodes), so a service disabled at build time could never later be started via + // `startService`. lazyServices is enforced one layer up instead: StackLifecycleCoordinator + // only eagerly starts postgres/postgres-init, and the ApiProxy starts everything else on + // demand via the ordinary startService + waitReady coordinator methods, which require every + // def to remain a node in the graph. + const resolver = mockBinaryResolver(); + const layer = builderLayer(resolver); + + return Effect.gen(function* () { + const builder = yield* StackBuilder; + const preparation = yield* StackPreparation; + const { graph } = yield* prepareAndBuild(builder, preparation, { + ...baseConfig, + lazyServices: true, + }); + + const byName = new Map(graph.startOrder.map((def) => [def.name, def] as const)); + expect(byName.get("postgres")?.enabled).toBe(true); + expect(byName.get("postgres-init")?.enabled).toBe(true); + expect(byName.get("postgrest")?.enabled).toBe(true); + expect(byName.get("auth")?.enabled).toBe(true); + }).pipe(Effect.provide(layer)); + }); + it.effect("docker mode produces Docker service defs for all services", () => { const resolver = mockBinaryResolver(); const layer = builderLayer(resolver); diff --git a/packages/stack/src/StackLifecycleCoordinator.ts b/packages/stack/src/StackLifecycleCoordinator.ts index 2515c05139..8ef6525bbe 100644 --- a/packages/stack/src/StackLifecycleCoordinator.ts +++ b/packages/stack/src/StackLifecycleCoordinator.ts @@ -1,6 +1,6 @@ import { LogBuffer, Orchestrator } from "@supabase/process-compose"; import { ServiceNotFoundError } from "@supabase/process-compose"; -import type { LogEntry, ServiceReadyError } from "@supabase/process-compose"; +import type { LogEntry, ResolvedGraph, ServiceReadyError } from "@supabase/process-compose"; import { Deferred, Effect, @@ -47,8 +47,40 @@ type LifecyclePhase = interface RuntimeState { readonly orchestrator: Orchestrator["Service"]; readonly cleanupTargets: CleanupTargets; + readonly graph: ResolvedGraph; } +// postgres/postgres-init are always nodes in the graph (StackBuilder never disables them), so +// ServiceNotFoundError can't actually occur when lazyServices eager-starts them — map it to +// StackBuildError only to satisfy start()'s declared error type. +const eagerStartService = ( + orchestrator: Orchestrator["Service"], + name: string, +): Effect.Effect => + orchestrator.startService(name).pipe( + Effect.catchTag( + "ServiceNotFoundError", + (error) => + new StackBuildError({ + detail: `lazyServices eager-start: unexpected missing service "${error.name}"`, + }), + ), + ); + +const eagerWaitReady = ( + orchestrator: Orchestrator["Service"], + name: string, +): Effect.Effect => + orchestrator.waitReady(name).pipe( + Effect.catchTag( + "ServiceNotFoundError", + (error) => + new StackBuildError({ + detail: `lazyServices eager-start: unexpected missing service "${error.name}"`, + }), + ), + ); + const sameState = (a: StackServiceState | undefined, b: StackServiceState): boolean => a?.name === b.name && a.status === b.status && @@ -405,6 +437,7 @@ export class StackLifecycleCoordinator extends Context.Service< return { orchestrator, cleanupTargets, + graph, } satisfies RuntimeState; }).pipe( Effect.tap((value) => @@ -527,8 +560,26 @@ export class StackLifecycleCoordinator extends Context.Service< yield* Ref.set(phaseRef, "starting"); const runtime = yield* ensureRuntime; yield* configureFunctions(config); - yield* runtime.orchestrator.start(); - yield* runtime.orchestrator.waitAllReady(); + if (config.lazyServices === true) { + // Only bring up postgres (and postgres-init, which depends on postgres and + // bootstraps the schema once). Every other service stays Pending until the + // ApiProxy starts it on demand via startService + waitReady (see ensureService in + // ApiProxy.ts / lazyServices.ts). + const eagerServices = runtime.graph.startOrder + .map((def) => def.name) + .filter((name) => name === "postgres" || name === "postgres-init"); + for (const name of eagerServices) { + yield* eagerStartService(runtime.orchestrator, name); + } + yield* Effect.forEach( + eagerServices, + (name) => eagerWaitReady(runtime.orchestrator, name), + { concurrency: "unbounded" }, + ); + } else { + yield* runtime.orchestrator.start(); + yield* runtime.orchestrator.waitAllReady(); + } yield* Ref.set(phaseRef, "running"); }), stop: () => diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts index 429c3094e3..17ed964b14 100644 --- a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -1,4 +1,5 @@ import { mkdtempSync, rmSync } from "node:fs"; +import * as http from "node:http"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "@effect/vitest"; @@ -48,6 +49,7 @@ function makeConfig(dataDir: string): ResolvedStackConfig { projectDir: "/tmp/supabase-project", mode: "native", jwtSecret: testJwtSecret, + lazyServices: false, ports: defaultPorts, apiPort: 54321, dbPort: 54322, @@ -99,7 +101,40 @@ function setupLayer(config: ResolvedStackConfig) { Layer.provide(NodeServices.layer), ); - return { layer }; + return { layer, spawner }; +} + +function makeLazyConfig(dataDir: string): ResolvedStackConfig { + return { + ...makeConfig(dataDir), + lazyServices: true, + // postgrest's health check is a real HTTP probe against 127.0.0.1:postgrestPort; the test + // below binds a fake listener there so waitReady can actually resolve once startService + // spawns it (spawning itself is satisfied generically by the mocked ChildProcessSpawner). + postgrest: { + port: defaultPorts.postgrestPort, + adminPort: defaultPorts.postgrestAdminPort, + schemas: ["public"], + extraSearchPath: ["public"], + maxRows: 1000, + version: DEFAULT_VERSIONS.postgrest, + }, + }; +} + +function startFakeHealthyServer(port: number): Promise<() => Promise> { + return new Promise((resolve, reject) => { + const server = http.createServer((_req, res) => { + res.writeHead(200); + res.end("OK"); + }); + server.listen(port, "127.0.0.1", () => { + resolve( + () => new Promise((res, rej) => server.close((err) => (err ? rej(err) : res()))), + ); + }); + server.on("error", reject); + }); } describe("StackLifecycleCoordinator enableExtension", () => { @@ -139,3 +174,59 @@ describe("StackLifecycleCoordinator enableExtension", () => { ); }); }); + +describe("StackLifecycleCoordinator lazyServices", () => { + // it.live: the mocked ChildProcessSpawner and postgres's health-check probe both resolve via a + // real `Effect.sleep`, which needs the real clock to progress (same reason as the + // enableExtension test above). + it.live("start() only eager-starts postgres/postgres-init; postgrest starts on demand", () => { + const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-lazy-test-")); + const config = makeLazyConfig(dataDir); + const { layer, spawner } = setupLayer(config); + + return Effect.gen(function* () { + const stopFakeServer = yield* Effect.promise(() => + startFakeHealthyServer(defaultPorts.postgrestPort), + ); + + const isPostgrestPayload = (s: { args: ReadonlyArray }) => { + const encoded = s.args.at(-1); + if (encoded === undefined) return false; + try { + const decoded = JSON.parse(Buffer.from(encoded, "base64url").toString("utf8")) as { + command?: string; + }; + return decoded.command?.endsWith("/postgrest") ?? false; + } catch { + return false; + } + }; + + const stack = yield* Stack; + yield* stack.start(); + + // postgrest must not have been spawned by start() itself. + expect(spawner.spawned.some(isPostgrestPayload)).toBe(false); + + const postgrestState = yield* stack.getState("postgrest"); + expect(postgrestState.status).toBe("Pending"); + + // The ApiProxy's ensureService would call startService + waitReady on first request; + // simulate that here directly against the coordinator. + yield* stack.startService("postgrest"); + yield* stack.waitReady("postgrest"); + + expect(spawner.spawned.some(isPostgrestPayload)).toBe(true); + // waitReady already proved the service reached a ready state (Running or Healthy — health + // checks poll on an interval, so the coordinator's projected state may briefly lag the + // orchestrator's internal readiness signal that waitReady resolves on). + const readyState = yield* stack.getState("postgrest"); + expect(["Running", "Healthy"]).toContain(readyState.status); + + yield* Effect.promise(() => stopFakeServer()); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), + ); + }); +}); diff --git a/packages/stack/src/createStack.ts b/packages/stack/src/createStack.ts index ad12d51585..a4538e17b9 100644 --- a/packages/stack/src/createStack.ts +++ b/packages/stack/src/createStack.ts @@ -547,6 +547,7 @@ export async function resolveConfig( projectDir, mode: resolvedMode, jwtSecret, + lazyServices: config.lazyServices === true, ports, apiPort: ports.apiPort, dbPort: ports.dbPort, diff --git a/packages/stack/src/layers.ts b/packages/stack/src/layers.ts index 7cc36fb504..c1337fdf55 100644 --- a/packages/stack/src/layers.ts +++ b/packages/stack/src/layers.ts @@ -1,11 +1,12 @@ import { fork, type ChildProcess } from "node:child_process"; import { Data, Effect, Layer, Option } from "effect"; import { FileSystem, Path } from "effect"; -import { FetchHttpClient } from "effect/unstable/http"; +import { FetchHttpClient, HttpClient, HttpServer } from "effect/unstable/http"; import { ApiProxy, type ProxyConfig } from "./ApiProxy.ts"; import { BinaryResolver } from "./BinaryResolver.ts"; import type { PlatformFactory } from "./createStack.ts"; import type { DaemonMessage, DaemonStartMessage } from "./daemon.ts"; +import { makeEnsureServiceMemo } from "./lazyServices.ts"; import { RemoteStack } from "./RemoteStack.ts"; import { Stack } from "./Stack.ts"; import { StackLifecycleCoordinator } from "./StackLifecycleCoordinator.ts"; @@ -20,10 +21,59 @@ import { type StateManagerService, } from "./StateManager.ts"; import { StackBuilder, type ResolvedStackConfig } from "./StackBuilder.ts"; +import type { ServiceName } from "./versions.ts"; import { UnixHttpClient } from "./UnixHttpClient.ts"; import { resolveManagedStack } from "./managed-stack.ts"; import { terminateChildProcess } from "./terminateChild.ts"; +const baseProxyConfig = (config: ResolvedStackConfig): Omit => ({ + listenPort: config.apiPort, + gotruePort: config.auth !== false ? config.auth.port : 0, + postgrestPort: config.postgrest !== false ? config.postgrest.port : 0, + postgrestAdminPort: config.postgrest !== false ? config.postgrest.adminPort : 0, + edgeRuntimePort: config.edgeRuntime !== false ? config.edgeRuntime.port : 0, + realtimePort: config.realtime !== false ? config.realtime.port : 0, + storagePort: config.storage !== false ? config.storage.port : 0, + pgmetaPort: config.pgmeta !== false ? config.pgmeta.port : 0, + analyticsPort: config.analytics !== false ? config.analytics.port : 0, + poolerPort: config.pooler !== false ? config.pooler.apiPort : 0, + studioPort: config.studio !== false ? config.studio.port : 0, + publishableKey: config.publishableKey, + secretKey: config.secretKey, + anonJwt: config.anonJwt, + serviceRoleJwt: config.serviceRoleJwt, +}); + +/** + * Build the ApiProxy layer for a stack. + * + * When `config.lazyServices` is on, the layer depends on `Stack` so it can wire + * `ProxyConfig.ensureService` to `Stack.startService` + `Stack.waitReady`, memoized so concurrent + * first requests to a route trigger a single start. Otherwise it's the plain config-only layer. + */ +const makeApiProxyLayer = ( + config: ResolvedStackConfig, +): Layer.Layer => { + if (!config.lazyServices) { + return ApiProxy.layer(baseProxyConfig(config)); + } + + return Layer.unwrap( + Effect.gen(function* () { + const stack = yield* Stack; + const ensureService = makeEnsureServiceMemo((name: ServiceName) => + Effect.runPromise( + Effect.gen(function* () { + yield* stack.startService(name); + yield* stack.waitReady(name); + }), + ), + ); + return ApiProxy.layer({ ...baseProxyConfig(config), ensureService }); + }), + ); +}; + /** * Build a foreground layer that runs the stack in-process. * @@ -47,24 +97,10 @@ export const foregroundLayer = ( ); const stackLayer = Stack.layer(config).pipe(Layer.provide(coordinatorLayer)); - const proxyConfig: ProxyConfig = { - listenPort: config.apiPort, - gotruePort: config.auth !== false ? config.auth.port : 0, - postgrestPort: config.postgrest !== false ? config.postgrest.port : 0, - postgrestAdminPort: config.postgrest !== false ? config.postgrest.adminPort : 0, - edgeRuntimePort: config.edgeRuntime !== false ? config.edgeRuntime.port : 0, - realtimePort: config.realtime !== false ? config.realtime.port : 0, - storagePort: config.storage !== false ? config.storage.port : 0, - pgmetaPort: config.pgmeta !== false ? config.pgmeta.port : 0, - analyticsPort: config.analytics !== false ? config.analytics.port : 0, - poolerPort: config.pooler !== false ? config.pooler.apiPort : 0, - studioPort: config.studio !== false ? config.studio.port : 0, - publishableKey: config.publishableKey, - secretKey: config.secretKey, - anonJwt: config.anonJwt, - serviceRoleJwt: config.serviceRoleJwt, - }; - const apiProxyLayer = ApiProxy.layer(proxyConfig).pipe(Layer.provide(FetchHttpClient.layer)); + const apiProxyLayer = makeApiProxyLayer(config).pipe( + Layer.provide(FetchHttpClient.layer), + Layer.provide(stackLayer), + ); return Layer.mergeAll(stackLayer, apiProxyLayer).pipe(Layer.provide(platform), Layer.orDie); }; @@ -95,24 +131,6 @@ export const foregroundDaemonLayer = ( const binaryResolverLayer = BinaryResolver.make(config.cacheRoot).pipe( Layer.provide(FetchHttpClient.layer), ); - const proxyConfig: ProxyConfig = { - listenPort: config.apiPort, - gotruePort: config.auth !== false ? config.auth.port : 0, - postgrestPort: config.postgrest !== false ? config.postgrest.port : 0, - postgrestAdminPort: config.postgrest !== false ? config.postgrest.adminPort : 0, - edgeRuntimePort: config.edgeRuntime !== false ? config.edgeRuntime.port : 0, - realtimePort: config.realtime !== false ? config.realtime.port : 0, - storagePort: config.storage !== false ? config.storage.port : 0, - pgmetaPort: config.pgmeta !== false ? config.pgmeta.port : 0, - analyticsPort: config.analytics !== false ? config.analytics.port : 0, - poolerPort: config.pooler !== false ? config.pooler.apiPort : 0, - studioPort: config.studio !== false ? config.studio.port : 0, - publishableKey: config.publishableKey, - secretKey: config.secretKey, - anonJwt: config.anonJwt, - serviceRoleJwt: config.serviceRoleJwt, - }; - const apiProxyLayer = ApiProxy.layer(proxyConfig).pipe(Layer.provide(FetchHttpClient.layer)); const stateManagerLayer = StateManager.make( singleStackStateManagerPaths(config.stackRoot, config.runtimeRoot, config.name), ); @@ -127,6 +145,11 @@ export const foregroundDaemonLayer = ( ); const stackLayer = Stack.layer(config).pipe(Layer.provide(coordinatorLayer)); + const apiProxyLayer = makeApiProxyLayer(config).pipe( + Layer.provide(FetchHttpClient.layer), + Layer.provide(stackLayer), + ); + return Layer.mergeAll(stackLayer, apiProxyLayer, stateManagerLayer).pipe( Layer.provide(platform), Layer.orDie, diff --git a/packages/stack/src/lazyServices.ts b/packages/stack/src/lazyServices.ts new file mode 100644 index 0000000000..00a6c983de --- /dev/null +++ b/packages/stack/src/lazyServices.ts @@ -0,0 +1,33 @@ +import type { ServiceName } from "./versions.ts"; + +/** + * Wrap a service-start function so concurrent callers for the same service + * share a single in-flight start, and a completed start resolves immediately + * on subsequent calls without re-invoking `start`. + * + * On failure, the in-flight entry is cleared (not marked done) so the next + * call retries from scratch. + */ +export const makeEnsureServiceMemo = ( + start: (name: ServiceName) => Promise, +): ((name: ServiceName) => Promise) => { + const inFlight = new Map>(); + const done = new Set(); + return (name) => { + if (done.has(name)) return Promise.resolve(); + const existing = inFlight.get(name); + if (existing) return existing; + const p = start(name).then( + () => { + done.add(name); + inFlight.delete(name); + }, + (err: unknown) => { + inFlight.delete(name); + throw err; + }, + ); + inFlight.set(name, p); + return p; + }; +}; diff --git a/packages/stack/src/lazyServices.unit.test.ts b/packages/stack/src/lazyServices.unit.test.ts new file mode 100644 index 0000000000..f42d669d1f --- /dev/null +++ b/packages/stack/src/lazyServices.unit.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from "vitest"; +import { makeEnsureServiceMemo } from "./lazyServices.ts"; + +describe("makeEnsureServiceMemo", () => { + it("starts a service once across concurrent calls", async () => { + let starts = 0; + const ensure = makeEnsureServiceMemo(async (_name) => { + starts += 1; + await new Promise((r) => setTimeout(r, 10)); + }); + await Promise.all([ensure("realtime"), ensure("realtime"), ensure("realtime")]); + expect(starts).toBe(1); + }); + + it("retries after a failed start", async () => { + let attempt = 0; + const ensure = makeEnsureServiceMemo(async () => { + attempt += 1; + if (attempt === 1) throw new Error("boom"); + }); + await expect(ensure("auth")).rejects.toThrow("boom"); + await ensure("auth"); // second attempt allowed + expect(attempt).toBe(2); + }); + + it("resolves immediately for a service already marked done", async () => { + let starts = 0; + const ensure = makeEnsureServiceMemo(async (_name) => { + starts += 1; + }); + await ensure("storage"); + await ensure("storage"); + await ensure("storage"); + expect(starts).toBe(1); + }); + + it("tracks each service name independently", async () => { + const seen: string[] = []; + const ensure = makeEnsureServiceMemo(async (name) => { + seen.push(name); + }); + await Promise.all([ensure("realtime"), ensure("storage")]); + expect(seen.sort()).toEqual(["realtime", "storage"]); + }); +}); diff --git a/packages/stack/tests/helpers/buildServices.ts b/packages/stack/tests/helpers/buildServices.ts index 188db80cd8..b8993c5446 100644 --- a/packages/stack/tests/helpers/buildServices.ts +++ b/packages/stack/tests/helpers/buildServices.ts @@ -89,6 +89,7 @@ const baseConfig: ResolvedStackConfig = { projectDir: "/tmp/supabase-project", mode: "auto", jwtSecret: testJwtSecret, + lazyServices: false, ports: basePorts, apiPort: 3000, dbPort: 5432, From 7296801c18036fef365f9b06e209d54026dc0bfc Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 12:58:34 +0200 Subject: [PATCH 12/38] fix(stack): scope waitAllReady() to started services under lazyServices StackLifecycleCoordinator.waitAllReady() unconditionally delegated to orchestrator.waitAllReady() over the full graph startOrder. Under lazyServices, services that are never started on demand (e.g. postgrest) never resolve their healthy deferred and never emit a Failed state, so ready() hung forever. Track the set of services that have actually been started (eager set from start(), plus startService/restartService/enableExtension/reload* calls). Under lazyServices, waitAllReady() now waits per-service on that snapshot instead of the full graph. Non-lazy behavior is unchanged: it still calls orchestrator.waitAllReady() over the whole graph. --- .../stack/src/StackLifecycleCoordinator.ts | 50 ++++++++++++++-- .../StackLifecycleCoordinator.unit.test.ts | 60 ++++++++++++++++++- 2 files changed, 105 insertions(+), 5 deletions(-) diff --git a/packages/stack/src/StackLifecycleCoordinator.ts b/packages/stack/src/StackLifecycleCoordinator.ts index 8ef6525bbe..8dcd01fb93 100644 --- a/packages/stack/src/StackLifecycleCoordinator.ts +++ b/packages/stack/src/StackLifecycleCoordinator.ts @@ -67,7 +67,12 @@ const eagerStartService = ( ), ); -const eagerWaitReady = ( +// Used both for the eager-start set in start() and for waitAllReady()'s started-service set +// under lazyServices. In both cases the name comes from either the graph's own startOrder or +// from a name the coordinator already validated via requireKnownService/startService, so +// ServiceNotFoundError can't actually occur — map it to StackBuildError only to satisfy the +// declared error types of the callers. +const waitReadyKnownService = ( orchestrator: Orchestrator["Service"], name: string, ): Effect.Effect => @@ -76,7 +81,7 @@ const eagerWaitReady = ( "ServiceNotFoundError", (error) => new StackBuildError({ - detail: `lazyServices eager-start: unexpected missing service "${error.name}"`, + detail: `waitReady: unexpected missing service "${error.name}"`, }), ), ); @@ -235,6 +240,14 @@ export class StackLifecycleCoordinator extends Context.Service< const stateRef = yield* SubscriptionRef.make(initialPublicStates(config)); const phaseRef = yield* Ref.make("idle"); const enableExtensionLock = yield* Semaphore.make(1); + // Tracks every service that has actually been asked to start: the eager set from + // start() under lazyServices, plus anything started later via startService (the + // ApiProxy's ensureService on-demand path) or restartService. Under lazyServices, + // waitAllReady() only waits on this set, since never-started ServiceDefs never resolve + // their `healthy` deferred and never emit a Failed state either (see waitAllReady below). + const startedServicesRef = yield* Ref.make>(new Set()); + const markStarted = (names: Iterable) => + Ref.update(startedServicesRef, (current) => new Set([...current, ...names])); const logBufferServices = yield* Layer.buildWithScope(LogBuffer.layer, scope); const logBuffer = Context.get(logBufferServices, LogBuffer); @@ -571,13 +584,15 @@ export class StackLifecycleCoordinator extends Context.Service< for (const name of eagerServices) { yield* eagerStartService(runtime.orchestrator, name); } + yield* markStarted(eagerServices); yield* Effect.forEach( eagerServices, - (name) => eagerWaitReady(runtime.orchestrator, name), + (name) => waitReadyKnownService(runtime.orchestrator, name), { concurrency: "unbounded" }, ); } else { yield* runtime.orchestrator.start(); + yield* markStarted(runtime.graph.startOrder.map((def) => def.name)); yield* runtime.orchestrator.waitAllReady(); } yield* Ref.set(phaseRef, "running"); @@ -598,6 +613,7 @@ export class StackLifecycleCoordinator extends Context.Service< yield* requireKnownService(name); const runtime = yield* ensureRuntime; yield* runtime.orchestrator.startService(name); + yield* markStarted([name]); yield* runtime.orchestrator.waitReady(name); }), stopService: (name) => @@ -611,6 +627,10 @@ export class StackLifecycleCoordinator extends Context.Service< yield* requireKnownService(name); const runtime = yield* ensureRuntime; yield* runtime.orchestrator.restartService(name); + // Idempotent: the service was necessarily already in the started set (you can't + // restart something that was never started), but marking again is harmless and + // keeps this call site self-contained if that invariant ever changes. + yield* markStarted([name]); }), enableExtension: (name) => enableExtensionLock.withPermits(1)( @@ -628,6 +648,7 @@ export class StackLifecycleCoordinator extends Context.Service< yield* requireKnownService("postgres"); const runtime = yield* ensureRuntime; yield* runtime.orchestrator.restartService("postgres"); + yield* markStarted(["postgres"]); yield* runtime.orchestrator.waitReady("postgres"); }), ), @@ -637,6 +658,7 @@ export class StackLifecycleCoordinator extends Context.Service< const runtime = yield* ensureRuntime; yield* configureFunctions(configWithFunctionOptions(opts)); yield* runtime.orchestrator.restartService("edge-runtime"); + yield* markStarted(["edge-runtime"]); yield* runtime.orchestrator.waitReady("edge-runtime"); }), reloadEdgeRuntime: (opts) => @@ -667,6 +689,7 @@ export class StackLifecycleCoordinator extends Context.Service< ), ); yield* runtime.orchestrator.restartService("edge-runtime"); + yield* markStarted(["edge-runtime"]); yield* runtime.orchestrator.waitReady("edge-runtime"); }), getState: (name) => @@ -694,7 +717,26 @@ export class StackLifecycleCoordinator extends Context.Service< waitAllReady: () => Effect.gen(function* () { const runtime = yield* ensureRuntime; - yield* runtime.orchestrator.waitAllReady(); + if (config.lazyServices !== true) { + // Non-lazy: identical to pre-lazyServices behavior — every ServiceDef gets + // started by start(), so waiting on the full graph is correct and byte-identical + // to today. + yield* runtime.orchestrator.waitAllReady(); + return; + } + // Lazy: "ready" means "every service that has been started is ready". Services + // that were never started (e.g. postgrest, still Pending until the ApiProxy's + // ensureService calls startService on demand) never resolve their `healthy` + // deferred and never emit a Failed state either, so waiting on them would hang + // forever. Snapshot the started set at call time — if a service starts + // concurrently between this read and the wait below, it's acceptable to not wait + // on it; callers that need to observe it can call waitReady/waitAllReady again. + const startedServices = yield* Ref.get(startedServicesRef); + yield* Effect.forEach( + startedServices, + (name) => waitReadyKnownService(runtime.orchestrator, name), + { concurrency: "unbounded" }, + ); }), subscribeLogs: (name) => logBuffer.subscribe(name), subscribeAllLogs: (services) => diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts index 17ed964b14..9c0c986a45 100644 --- a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -4,7 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "@effect/vitest"; import { NodeServices } from "@effect/platform-node"; -import { Effect, Layer } from "effect"; +import { Duration, Effect, Layer } from "effect"; import { mockChildProcessSpawner } from "../../process-compose/tests/helpers/mocks.ts"; import { mockBinaryResolver } from "../tests/helpers/mocks.ts"; import { defaultPublishableKey, defaultSecretKey, generateJwt } from "./JwtGenerator.ts"; @@ -229,4 +229,62 @@ describe("StackLifecycleCoordinator lazyServices", () => { Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), ); }); + + // Regression test: waitAllReady() used to unconditionally delegate to + // orchestrator.waitAllReady() over the FULL graph startOrder. Under lazyServices, services + // that were never started (e.g. postgrest, which stays "Pending" until the ApiProxy's + // ensureService calls startService on first request) never resolve their `healthy` deferred + // and never emit a Failed state either, so the old implementation hung forever. This is the + // red test against the pre-fix code: bounded with a short timeout so it fails fast (as a + // timeout) rather than hanging the suite. + it.live("waitAllReady() resolves promptly when only postgres was eager-started", () => { + const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-lazy-ready-test-")); + const config = makeLazyConfig(dataDir); + const { layer } = setupLayer(config); + + return Effect.gen(function* () { + const stack = yield* Stack; + yield* stack.start(); + + // postgrest was never started; under the old behavior this would hang because + // orchestrator.waitAllReady() waits on the full graph, including never-started services. + yield* stack.waitAllReady().pipe(Effect.timeout(Duration.seconds(5))); + + const postgrestState = yield* stack.getState("postgrest"); + expect(postgrestState.status).toBe("Pending"); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), + ); + }); + + it.live("waitAllReady() covers a service started on demand after start()", () => { + const dataDir = mkdtempSync( + join(tmpdir(), "stack-lifecycle-coordinator-lazy-ready-started-test-"), + ); + const config = makeLazyConfig(dataDir); + const { layer } = setupLayer(config); + + return Effect.gen(function* () { + const stopFakeServer = yield* Effect.promise(() => + startFakeHealthyServer(defaultPorts.postgrestPort), + ); + + const stack = yield* Stack; + yield* stack.start(); + + // Simulate the ApiProxy's ensureService on-demand path. + yield* stack.startService("postgrest"); + + yield* stack.waitAllReady().pipe(Effect.timeout(Duration.seconds(5))); + + const postgrestState = yield* stack.getState("postgrest"); + expect(["Running", "Healthy"]).toContain(postgrestState.status); + + yield* Effect.promise(() => stopFakeServer()); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), + ); + }); }); From 6eb46213e3b56f1c3501c2d2c6054828af698c5c Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 13:08:27 +0200 Subject: [PATCH 13/38] feat(fleet): pod manifest types and template cache keys Adds PodManifest interface plus templateKey/baseTemplateKey helpers that derive stable sha256-based cache keys from a partial VersionManifest, independent of key order. --- packages/fleet/src/PodManifest.ts | 20 ++++++++++++++++++++ packages/fleet/src/PodManifest.unit.test.ts | 18 ++++++++++++++++++ packages/fleet/src/index.ts | 3 +++ 3 files changed, 41 insertions(+) create mode 100644 packages/fleet/src/PodManifest.ts create mode 100644 packages/fleet/src/PodManifest.unit.test.ts diff --git a/packages/fleet/src/PodManifest.ts b/packages/fleet/src/PodManifest.ts new file mode 100644 index 0000000000..b406f0bd9a --- /dev/null +++ b/packages/fleet/src/PodManifest.ts @@ -0,0 +1,20 @@ +import { createHash } from "node:crypto"; +import type { ServiceName, VersionManifest } from "@supabase/stack"; + +export interface PodManifest { + readonly id: string; + readonly versions: Partial; + readonly services: Partial>; + readonly flags: { readonly supautils: boolean }; + readonly ports: { readonly dbPort: number; readonly apiPort: number }; + readonly createdAt: string; +} + +export const baseTemplateKey = (postgresVersion: string): string => `pg-${postgresVersion}`; + +export const templateKey = (versions: Partial): string => { + const canonical = JSON.stringify( + Object.fromEntries(Object.entries(versions).sort(([a], [b]) => a.localeCompare(b))), + ); + return `tuple-${createHash("sha256").update(canonical).digest("hex").slice(0, 16)}`; +}; diff --git a/packages/fleet/src/PodManifest.unit.test.ts b/packages/fleet/src/PodManifest.unit.test.ts new file mode 100644 index 0000000000..95224d0009 --- /dev/null +++ b/packages/fleet/src/PodManifest.unit.test.ts @@ -0,0 +1,18 @@ +import { describe, expect, it } from "vitest"; +import { baseTemplateKey, templateKey } from "./PodManifest.ts"; + +describe("templateKey", () => { + it("is stable across key order", () => { + expect(templateKey({ postgres: "17.6.1.143", auth: "2.192.0" })).toBe( + templateKey({ auth: "2.192.0", postgres: "17.6.1.143" }), + ); + }); + it("changes when any version changes", () => { + expect(templateKey({ postgres: "17.6.1.143", auth: "2.192.0" })).not.toBe( + templateKey({ postgres: "17.6.1.143", auth: "2.192.1" }), + ); + }); + it("base key is human-readable", () => { + expect(baseTemplateKey("17.6.1.143")).toBe("pg-17.6.1.143"); + }); +}); diff --git a/packages/fleet/src/index.ts b/packages/fleet/src/index.ts index 2ec70b447c..f07d2085ed 100644 --- a/packages/fleet/src/index.ts +++ b/packages/fleet/src/index.ts @@ -1 +1,4 @@ export const FLEET_PACKAGE = "@supabase/fleet"; + +export type { PodManifest } from "./PodManifest.ts"; +export { baseTemplateKey, templateKey } from "./PodManifest.ts"; From 69ae47e386d7a875485dab2e29d4ce21e5335ef4 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 13:16:15 +0200 Subject: [PATCH 14/38] feat(fleet): copy-on-write directory clone (clonefile/reflink/copy) --- .../fleet/src/cowClone.integration.test.ts | 33 +++++++++++++++++++ packages/fleet/src/cowClone.ts | 26 +++++++++++++++ packages/fleet/src/index.ts | 1 + packages/fleet/vitest.config.ts | 6 ++++ 4 files changed, 66 insertions(+) create mode 100644 packages/fleet/src/cowClone.integration.test.ts create mode 100644 packages/fleet/src/cowClone.ts diff --git a/packages/fleet/src/cowClone.integration.test.ts b/packages/fleet/src/cowClone.integration.test.ts new file mode 100644 index 0000000000..8a52af52fd --- /dev/null +++ b/packages/fleet/src/cowClone.integration.test.ts @@ -0,0 +1,33 @@ +import { mkdtemp, mkdir, readFile, stat, writeFile, chmod } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { cloneDir } from "./cowClone.ts"; + +describe("cloneDir", () => { + it("clones a tree with content and modes; clones diverge", async () => { + const root = await mkdtemp(join(tmpdir(), "cow-test-")); + const src = join(root, "src"); + await mkdir(join(src, "sub"), { recursive: true }); + await writeFile(join(src, "sub", "a.txt"), "hello"); + await chmod(src, 0o700); + + const dest = join(root, "dest"); + await cloneDir(src, dest); + + expect(await readFile(join(dest, "sub", "a.txt"), "utf8")).toBe("hello"); + expect((await stat(dest)).mode & 0o777).toBe(0o700); + + await writeFile(join(dest, "sub", "a.txt"), "changed"); + expect(await readFile(join(src, "sub", "a.txt"), "utf8")).toBe("hello"); + }); + + it("refuses to clone onto an existing destination", async () => { + const root = await mkdtemp(join(tmpdir(), "cow-test-")); + const src = join(root, "src"); + await mkdir(src); + const dest = join(root, "dest"); + await mkdir(dest); + await expect(cloneDir(src, dest)).rejects.toThrow(/exists/); + }); +}); diff --git a/packages/fleet/src/cowClone.ts b/packages/fleet/src/cowClone.ts new file mode 100644 index 0000000000..8ad081c48d --- /dev/null +++ b/packages/fleet/src/cowClone.ts @@ -0,0 +1,26 @@ +import { spawn } from "node:child_process"; +import { cp, stat } from "node:fs/promises"; + +const run = (cmd: string, args: string[]): Promise => + new Promise((resolve, reject) => { + const child = spawn(cmd, args, { stdio: "ignore" }); + child.on("error", reject); + child.on("exit", (code) => resolve(code ?? 1)); + }); + +/** Copy-on-write directory clone: APFS clonefile on macOS, reflink on Linux, plain copy fallback. */ +export async function cloneDir(src: string, dest: string): Promise { + const exists = await stat(dest).then( + () => true, + () => false, + ); + if (exists) throw new Error(`cloneDir: destination already exists: ${dest}`); + + if (process.platform === "darwin") { + if ((await run("cp", ["-Rc", src, dest])) === 0) return; + } else if (process.platform === "linux") { + if ((await run("cp", ["-R", "--reflink=auto", src, dest])) === 0) return; + } + // Fallback (non-CoW filesystems, other platforms): plain recursive copy. + await cp(src, dest, { recursive: true, force: false, errorOnExist: true }); +} diff --git a/packages/fleet/src/index.ts b/packages/fleet/src/index.ts index f07d2085ed..92e2574259 100644 --- a/packages/fleet/src/index.ts +++ b/packages/fleet/src/index.ts @@ -1,4 +1,5 @@ export const FLEET_PACKAGE = "@supabase/fleet"; +export { cloneDir } from "./cowClone.ts"; export type { PodManifest } from "./PodManifest.ts"; export { baseTemplateKey, templateKey } from "./PodManifest.ts"; diff --git a/packages/fleet/vitest.config.ts b/packages/fleet/vitest.config.ts index 7a35ffa8d7..e10a870951 100644 --- a/packages/fleet/vitest.config.ts +++ b/packages/fleet/vitest.config.ts @@ -18,6 +18,12 @@ export default defineConfig({ include: ["**/*.unit.test.ts"], }, }, + { + test: { + name: "integration", + include: ["**/*.integration.test.ts"], + }, + }, ], }, }); From b22587b9af9db248af13615f248a1451a297b7a4 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 13:26:31 +0200 Subject: [PATCH 15/38] fix(fleet): clean up partial dest before CoW fallback, preserve symlinks cloneDir() now rm -rf's dest before falling back to fs.cp when the platform-specific cp -Rc/--reflink step exits non-zero, so a partially written dest from a failed CoW attempt can't silently mix with fresh fallback output. The fs.cp fallback also now passes verbatimSymlinks: true so relative symlinks in src aren't rewritten to absolute paths that point back into the source tree. Adds an internal, documented `cowCommand` test seam (CloneDirOptions) to force the CoW step to fail deterministically in tests, exercising the fallback-cleanup and symlink-preservation paths without depending on filesystem-specific CoW failure conditions. --- .../fleet/src/cowClone.integration.test.ts | 56 ++++++++++++++++++- packages/fleet/src/cowClone.ts | 52 +++++++++++++++-- 2 files changed, 102 insertions(+), 6 deletions(-) diff --git a/packages/fleet/src/cowClone.integration.test.ts b/packages/fleet/src/cowClone.integration.test.ts index 8a52af52fd..096a9a461d 100644 --- a/packages/fleet/src/cowClone.integration.test.ts +++ b/packages/fleet/src/cowClone.integration.test.ts @@ -1,9 +1,21 @@ -import { mkdtemp, mkdir, readFile, stat, writeFile, chmod } from "node:fs/promises"; +import { + mkdtemp, + mkdir, + readFile, + readlink, + stat, + symlink, + writeFile, + chmod, +} from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; import { cloneDir } from "./cowClone.ts"; +/** Forces the CoW `cp` step to fail deterministically, so tests can exercise the fallback path. */ +const FORCE_FALLBACK = { cowCommand: "false" }; + describe("cloneDir", () => { it("clones a tree with content and modes; clones diverge", async () => { const root = await mkdtemp(join(tmpdir(), "cow-test-")); @@ -30,4 +42,46 @@ describe("cloneDir", () => { await mkdir(dest); await expect(cloneDir(src, dest)).rejects.toThrow(/exists/); }); + + it("does not silently mix stale CoW leftovers with fallback output when the CoW command fails after partially writing dest", async () => { + const root = await mkdtemp(join(tmpdir(), "cow-test-")); + const src = join(root, "src"); + await mkdir(src, { recursive: true }); + await writeFile(join(src, "fresh.txt"), "fresh-from-src"); + + const dest = join(root, "dest"); + + // Use a "CoW command" that partially writes dest (mkdir + one file) and then fails, + // to model a real clonefile/reflink command that dies partway through. cloneDir + // invokes it as ` `, so `dest` is always the + // last argument regardless of platform-specific flags. + const partialWriteThenFail = join(root, "partial-write-then-fail.sh"); + await writeFile( + partialWriteThenFail, + `#!/bin/sh\nlast=""\nfor arg in "$@"; do last="$arg"; done\nmkdir -p "$last"\necho "leftover" > "$last/leftover.txt"\nexit 1\n`, + { mode: 0o755 }, + ); + + await cloneDir(src, dest, { cowCommand: partialWriteThenFail }); + + // The fallback must have run to completion (fresh.txt present)... + expect(await readFile(join(dest, "fresh.txt"), "utf8")).toBe("fresh-from-src"); + // ...and the stale leftover from the failed CoW attempt must be gone, not silently + // preserved alongside the fallback's output. + await expect(stat(join(dest, "leftover.txt"))).rejects.toThrow(); + }); + + it("preserves relative symlink targets through the fallback copy (verbatimSymlinks)", async () => { + const root = await mkdtemp(join(tmpdir(), "cow-test-")); + const src = join(root, "src"); + await mkdir(join(src, "sub"), { recursive: true }); + await writeFile(join(src, "sub", "target.txt"), "hi"); + await symlink("target.txt", join(src, "sub", "link.txt")); + + const dest = join(root, "dest"); + await cloneDir(src, dest, FORCE_FALLBACK); + + expect(await readlink(join(dest, "sub", "link.txt"))).toBe("target.txt"); + expect(await readFile(join(dest, "sub", "link.txt"), "utf8")).toBe("hi"); + }); }); diff --git a/packages/fleet/src/cowClone.ts b/packages/fleet/src/cowClone.ts index 8ad081c48d..48882abe4e 100644 --- a/packages/fleet/src/cowClone.ts +++ b/packages/fleet/src/cowClone.ts @@ -1,5 +1,5 @@ import { spawn } from "node:child_process"; -import { cp, stat } from "node:fs/promises"; +import { cp, rm, stat } from "node:fs/promises"; const run = (cmd: string, args: string[]): Promise => new Promise((resolve, reject) => { @@ -8,19 +8,61 @@ const run = (cmd: string, args: string[]): Promise => child.on("exit", (code) => resolve(code ?? 1)); }); +export interface CloneDirOptions { + /** + * Overrides the executable used for the platform-specific copy-on-write attempt + * (`cp` by default). Test-only seam: lets tests force the CoW step to fail + * deterministically (e.g. `"false"`, or a stub script) so the fallback path — + * including its pre-fallback cleanup of any partially-written `dest` — can be + * exercised without depending on filesystem-specific CoW failure conditions. + */ + cowCommand?: string; +} + /** Copy-on-write directory clone: APFS clonefile on macOS, reflink on Linux, plain copy fallback. */ -export async function cloneDir(src: string, dest: string): Promise { +export async function cloneDir( + src: string, + dest: string, + options?: CloneDirOptions, +): Promise { const exists = await stat(dest).then( () => true, () => false, ); if (exists) throw new Error(`cloneDir: destination already exists: ${dest}`); + const cowCommand = options?.cowCommand ?? "cp"; + let attemptedCow = false; + if (process.platform === "darwin") { - if ((await run("cp", ["-Rc", src, dest])) === 0) return; + attemptedCow = true; + if ((await run(cowCommand, ["-Rc", src, dest])) === 0) return; } else if (process.platform === "linux") { - if ((await run("cp", ["-R", "--reflink=auto", src, dest])) === 0) return; + attemptedCow = true; + if ((await run(cowCommand, ["-R", "--reflink=auto", src, dest])) === 0) return; + } else if (options?.cowCommand) { + // No platform-specific CoW branch would normally run, but tests may still force + // the seam to exercise the fallback-cleanup path uniformly across platforms. + attemptedCow = true; + if ((await run(cowCommand, ["-R", src, dest])) === 0) return; } + + // The CoW attempt failed after possibly writing a partial dest (e.g. a clonefile/reflink + // command that dies partway through a large tree). Remove any such leftovers before + // falling back, so dest never ends up a silent mix of truncated CoW output and fresh + // fallback files. + if (attemptedCow) { + await rm(dest, { recursive: true, force: true }); + } + // Fallback (non-CoW filesystems, other platforms): plain recursive copy. - await cp(src, dest, { recursive: true, force: false, errorOnExist: true }); + // `verbatimSymlinks: true` preserves relative symlink targets as-is; the default + // (false) rewrites them to absolute paths pointing back into `src`, making the + // clone silently depend on its source tree. + await cp(src, dest, { + recursive: true, + force: false, + errorOnExist: true, + verbatimSymlinks: true, + }); } From ec88d293f5bd37d4f40f1a6d67aa515d450931a9 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 13:34:35 +0200 Subject: [PATCH 16/38] feat(fleet): add persistent port registry Introduces PortRegistry as the single owner of the port space for all pods on a host, replacing the plan for a cross-stack filesystem scan (readReservedPorts()). Persists pod->port allocations to a JSON state file, is idempotent per pod, and reuses freed ports on release. --- packages/fleet/src/PortRegistry.ts | 64 ++++++++++++++++++++ packages/fleet/src/PortRegistry.unit.test.ts | 31 ++++++++++ packages/fleet/src/index.ts | 2 + 3 files changed, 97 insertions(+) create mode 100644 packages/fleet/src/PortRegistry.ts create mode 100644 packages/fleet/src/PortRegistry.unit.test.ts diff --git a/packages/fleet/src/PortRegistry.ts b/packages/fleet/src/PortRegistry.ts new file mode 100644 index 0000000000..00d778c145 --- /dev/null +++ b/packages/fleet/src/PortRegistry.ts @@ -0,0 +1,64 @@ +import { mkdir, readFile, rename, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; + +export interface PodPorts { + readonly dbPort: number; + readonly apiPort: number; +} + +interface PortState { + readonly basePort: number; + readonly pods: Record; +} + +const DEFAULT_BASE_PORT = 55000; + +export class PortRegistry { + private constructor( + private readonly stateFile: string, + private state: PortState, + ) {} + + static async load(stateFile: string): Promise { + const raw = await readFile(stateFile, "utf8").catch(() => undefined); + const state: PortState = + raw !== undefined + ? (JSON.parse(raw) as PortState) + : { basePort: DEFAULT_BASE_PORT, pods: {} }; + return new PortRegistry(stateFile, state); + } + + get(podId: string): PodPorts | undefined { + return this.state.pods[podId]; + } + + async allocate(podId: string): Promise { + const existing = this.state.pods[podId]; + if (existing) return existing; + const used = new Set(Object.values(this.state.pods).flatMap((p) => [p.dbPort, p.apiPort])); + let candidate = this.state.basePort; + const next = (): number => { + while (used.has(candidate)) candidate += 1; + used.add(candidate); + return candidate; + }; + const ports: PodPorts = { dbPort: next(), apiPort: next() }; + this.state = { ...this.state, pods: { ...this.state.pods, [podId]: ports } }; + await this.persist(); + return ports; + } + + async release(podId: string): Promise { + const rest = { ...this.state.pods }; + delete rest[podId]; + this.state = { ...this.state, pods: rest }; + await this.persist(); + } + + private async persist(): Promise { + await mkdir(dirname(this.stateFile), { recursive: true }); + const tmp = `${this.stateFile}.tmp`; + await writeFile(tmp, JSON.stringify(this.state, null, 2)); + await rename(tmp, this.stateFile); + } +} diff --git a/packages/fleet/src/PortRegistry.unit.test.ts b/packages/fleet/src/PortRegistry.unit.test.ts new file mode 100644 index 0000000000..0e43ce7de9 --- /dev/null +++ b/packages/fleet/src/PortRegistry.unit.test.ts @@ -0,0 +1,31 @@ +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { PortRegistry } from "./PortRegistry.ts"; + +describe("PortRegistry", () => { + it("allocates unique port pairs and persists them", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + const a = await reg.allocate("pod-a"); + const b = await reg.allocate("pod-b"); + expect(new Set([a.dbPort, a.apiPort, b.dbPort, b.apiPort]).size).toBe(4); + + const reloaded = await PortRegistry.load(file); + expect(reloaded.get("pod-a")).toEqual(a); + expect(reloaded.get("pod-b")).toEqual(b); + }); + + it("is idempotent per pod and reuses released ports", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + const a1 = await reg.allocate("pod-a"); + const a2 = await reg.allocate("pod-a"); + expect(a2).toEqual(a1); + await reg.release("pod-a"); + expect(reg.get("pod-a")).toBeUndefined(); + const c = await reg.allocate("pod-c"); + expect(c.dbPort).toBe(a1.dbPort); // freed ports are reusable + }); +}); diff --git a/packages/fleet/src/index.ts b/packages/fleet/src/index.ts index 92e2574259..26a91b0a3e 100644 --- a/packages/fleet/src/index.ts +++ b/packages/fleet/src/index.ts @@ -3,3 +3,5 @@ export const FLEET_PACKAGE = "@supabase/fleet"; export { cloneDir } from "./cowClone.ts"; export type { PodManifest } from "./PodManifest.ts"; export { baseTemplateKey, templateKey } from "./PodManifest.ts"; +export type { PodPorts } from "./PortRegistry.ts"; +export { PortRegistry } from "./PortRegistry.ts"; From 84e64ffbfc618070cc01f3064a7bd02b439ed4a1 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 13:42:01 +0200 Subject: [PATCH 17/38] fix(fleet): harden PortRegistry against corrupt state, add restore() - load() no longer crashes on invalid JSON or structurally invalid state (missing/NaN basePort, non-object pods); it quarantines the bad file to `${stateFile}.corrupt` (overwriting any prior quarantine) and starts from fresh empty state instead. - Add restore(podId, ports) so the fleet daemon can re-seed known allocations from each pod's manifest after a quarantine/reset, without a full port scan. Idempotent for identical ports; throws on conflicting ports for the same pod or ports already held by another pod. - Document the single-owner / no-port-probing / quarantine-and-restore design assumptions on the class. --- packages/fleet/src/PortRegistry.ts | 93 ++++++++++++++++++- packages/fleet/src/PortRegistry.unit.test.ts | 97 +++++++++++++++++++- 2 files changed, 184 insertions(+), 6 deletions(-) diff --git a/packages/fleet/src/PortRegistry.ts b/packages/fleet/src/PortRegistry.ts index 00d778c145..e54cec0000 100644 --- a/packages/fleet/src/PortRegistry.ts +++ b/packages/fleet/src/PortRegistry.ts @@ -13,6 +13,40 @@ interface PortState { const DEFAULT_BASE_PORT = 55000; +function freshState(): PortState { + return { basePort: DEFAULT_BASE_PORT, pods: {} }; +} + +function isValidState(value: unknown): value is PortState { + if (typeof value !== "object" || value === null || Array.isArray(value)) return false; + const candidate = value as Record; + const { basePort, pods } = candidate; + if (typeof basePort !== "number" || Number.isNaN(basePort)) return false; + if (typeof pods !== "object" || pods === null || Array.isArray(pods)) return false; + return true; +} + +/** + * Persistent registry mapping pod IDs to their allocated `{ dbPort, apiPort }` + * pair, backed by a single JSON state file on disk. + * + * Design assumptions: + * - **Single owner process.** Exactly one `PortRegistry` instance (the fleet + * daemon) is expected to read and write the state file at a time. There is + * no file locking or cross-process coordination, so concurrent writers will + * silently lose updates (last write wins). + * - **No host-level port probing.** The registry never checks whether a port + * is actually free on the host; it only tracks what it has handed out + * itself. The daemon owns the 55000+ range by convention, so nothing else + * on the host is expected to bind those ports. + * - **Corrupt-state recovery.** If the state file is missing, unreadable as + * JSON, or structurally invalid, `load()` never throws. Instead it + * quarantines the bad file (renaming it to `.corrupt`, replacing + * any previous quarantine) and starts from fresh empty state. Since pod + * ports are also duplicated in each pod's own manifest (`pod.json`), the + * daemon is expected to re-seed the registry after such a reset by calling + * `restore()` for each known pod. + */ export class PortRegistry { private constructor( private readonly stateFile: string, @@ -21,11 +55,28 @@ export class PortRegistry { static async load(stateFile: string): Promise { const raw = await readFile(stateFile, "utf8").catch(() => undefined); - const state: PortState = - raw !== undefined - ? (JSON.parse(raw) as PortState) - : { basePort: DEFAULT_BASE_PORT, pods: {} }; - return new PortRegistry(stateFile, state); + if (raw === undefined) { + return new PortRegistry(stateFile, freshState()); + } + + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + await PortRegistry.quarantine(stateFile); + return new PortRegistry(stateFile, freshState()); + } + + if (!isValidState(parsed)) { + await PortRegistry.quarantine(stateFile); + return new PortRegistry(stateFile, freshState()); + } + + return new PortRegistry(stateFile, parsed); + } + + private static async quarantine(stateFile: string): Promise { + await rename(stateFile, `${stateFile}.corrupt`); } get(podId: string): PodPorts | undefined { @@ -48,6 +99,38 @@ export class PortRegistry { return ports; } + /** + * Records a known allocation (typically read back from a pod's own + * manifest) without scanning for free ports. Idempotent when the pod + * already holds exactly these ports. Throws if the pod already holds + * different ports, or if either port is already assigned to a different + * pod. + */ + async restore(podId: string, ports: PodPorts): Promise { + const existing = this.state.pods[podId]; + if (existing) { + if (existing.dbPort === ports.dbPort && existing.apiPort === ports.apiPort) { + return; + } + throw new Error( + `PortRegistry: cannot restore pod "${podId}" with ports ${JSON.stringify(ports)}; ` + + `it is already recorded with different ports ${JSON.stringify(existing)}`, + ); + } + + for (const [otherPodId, otherPorts] of Object.entries(this.state.pods)) { + if (otherPorts.dbPort === ports.dbPort || otherPorts.apiPort === ports.apiPort) { + throw new Error( + `PortRegistry: cannot restore pod "${podId}" with ports ${JSON.stringify(ports)}; ` + + `port already assigned to pod "${otherPodId}"`, + ); + } + } + + this.state = { ...this.state, pods: { ...this.state.pods, [podId]: ports } }; + await this.persist(); + } + async release(podId: string): Promise { const rest = { ...this.state.pods }; delete rest[podId]; diff --git a/packages/fleet/src/PortRegistry.unit.test.ts b/packages/fleet/src/PortRegistry.unit.test.ts index 0e43ce7de9..539859022b 100644 --- a/packages/fleet/src/PortRegistry.unit.test.ts +++ b/packages/fleet/src/PortRegistry.unit.test.ts @@ -1,4 +1,4 @@ -import { mkdtemp } from "node:fs/promises"; +import { mkdtemp, readFile, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; @@ -28,4 +28,99 @@ describe("PortRegistry", () => { const c = await reg.allocate("pod-c"); expect(c.dbPort).toBe(a1.dbPort); // freed ports are reusable }); + + it("quarantines a corrupt state file and loads with fresh empty state", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const garbage = "{not valid json at all"; + await writeFile(file, garbage); + + const reg = await PortRegistry.load(file); + + // Loads successfully with fresh empty state. + expect(reg.get("anything")).toBeUndefined(); + const allocated = await reg.allocate("pod-a"); + expect(allocated.dbPort).toBeGreaterThanOrEqual(55000); + + // Original bad bytes are preserved in the quarantine file. + const quarantined = await readFile(`${file}.corrupt`, "utf8"); + expect(quarantined).toBe(garbage); + }); + + it("quarantines a structurally invalid state file (bad basePort/pods)", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const badStructure = JSON.stringify({ basePort: "not-a-number", pods: [] }); + await writeFile(file, badStructure); + + const reg = await PortRegistry.load(file); + expect(reg.get("anything")).toBeUndefined(); + const allocated = await reg.allocate("pod-a"); + expect(allocated.dbPort).toBeGreaterThanOrEqual(55000); + + const quarantined = await readFile(`${file}.corrupt`, "utf8"); + expect(quarantined).toBe(badStructure); + }); + + it("overwrites any previous quarantine file", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + await writeFile(`${file}.corrupt`, "old-quarantine"); + const garbage = "{ still bad"; + await writeFile(file, garbage); + + await PortRegistry.load(file); + + const quarantined = await readFile(`${file}.corrupt`, "utf8"); + expect(quarantined).toBe(garbage); + }); + + describe("restore", () => { + it("records known allocations from pod manifests without scanning", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + + await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); + await reg.restore("pod-b", { dbPort: 55012, apiPort: 55013 }); + + expect(reg.get("pod-a")).toEqual({ dbPort: 55010, apiPort: 55011 }); + expect(reg.get("pod-b")).toEqual({ dbPort: 55012, apiPort: 55013 }); + + // New allocations must skip the restored ports. + const next = await reg.allocate("new-pod"); + expect([next.dbPort, next.apiPort]).not.toContain(55010); + expect([next.dbPort, next.apiPort]).not.toContain(55011); + expect([next.dbPort, next.apiPort]).not.toContain(55012); + expect([next.dbPort, next.apiPort]).not.toContain(55013); + + // Persisted, so a reload sees the restored allocation. + const reloaded = await PortRegistry.load(file); + expect(reloaded.get("pod-a")).toEqual({ dbPort: 55010, apiPort: 55011 }); + }); + + it("is idempotent when restoring identical ports for the same pod", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + + await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); + await expect( + reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }), + ).resolves.toBeUndefined(); + expect(reg.get("pod-a")).toEqual({ dbPort: 55010, apiPort: 55011 }); + }); + + it("throws if the pod already has different ports recorded", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + + await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); + await expect(reg.restore("pod-a", { dbPort: 55010, apiPort: 55099 })).rejects.toThrow(); + }); + + it("throws if a port is already assigned to a different pod", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + + await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); + await expect(reg.restore("pod-b", { dbPort: 55010, apiPort: 55012 })).rejects.toThrow(); + await expect(reg.restore("pod-b", { dbPort: 55020, apiPort: 55011 })).rejects.toThrow(); + }); + }); }); From 494f0bcdaa11a0147045202f7d146b80e714e90a Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 14:15:04 +0200 Subject: [PATCH 18/38] feat(fleet): template store with base and warm templates Adds TemplateStore, which builds golden Postgres data-dir templates by running @supabase/stack one-shot: base = postgres-only boot so postgres-init applies baseline migrations, then installMicroProfile freezes the result; warm = clone of base + enabled services booted once to self-migrate. Builds are concurrency-safe via a per-key wx-flag lockfile with a 10min stale threshold. Also exports installMicroProfile/readPreloadLibraries/writePreloadLibraries from stack's index so fleet can consume them. --- .../src/TemplateStore.integration.test.ts | 34 ++++ packages/fleet/src/TemplateStore.ts | 168 ++++++++++++++++++ packages/fleet/src/index.ts | 1 + packages/stack/src/index.ts | 1 + 4 files changed, 204 insertions(+) create mode 100644 packages/fleet/src/TemplateStore.integration.test.ts create mode 100644 packages/fleet/src/TemplateStore.ts diff --git a/packages/fleet/src/TemplateStore.integration.test.ts b/packages/fleet/src/TemplateStore.integration.test.ts new file mode 100644 index 0000000000..c33260aee4 --- /dev/null +++ b/packages/fleet/src/TemplateStore.integration.test.ts @@ -0,0 +1,34 @@ +import { mkdtemp, readFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { TemplateStore } from "./TemplateStore.ts"; + +// Requires postgres binaries in the local cache; opt-in via env, since the first +// run downloads a real postgres release (~minutes) and boots it. +const POSTGRES_VERSION = "17.6.1.143"; + +describe.skipIf(!process.env.FLEET_PG_TESTS)("TemplateStore", () => { + it("builds a base template once and reuses it", async () => { + const root = await mkdtemp(join(tmpdir(), "templates-")); + const store = new TemplateStore(root); + const first = await store.ensureBaseTemplate(POSTGRES_VERSION); + expect(first).toContain(`pg-${POSTGRES_VERSION}`); + // PGDATA got the micro profile + const conf = await readFile(join(first, "postgresql.conf"), "utf8"); + expect(conf).toContain("include_if_exists = 'micro.conf'"); + + const started = Date.now(); + const second = await store.ensureBaseTemplate(POSTGRES_VERSION); + expect(second).toBe(first); + expect(Date.now() - started).toBeLessThan(1000); // cache hit, no rebuild + }, 600_000); + + it("falls back to the base template when no services are enabled", async () => { + const root = await mkdtemp(join(tmpdir(), "templates-")); + const store = new TemplateStore(root); + const base = await store.ensureBaseTemplate(POSTGRES_VERSION); + const warm = await store.ensureWarmTemplate({ postgres: POSTGRES_VERSION }, []); + expect(warm).toBe(base); + }, 600_000); +}); diff --git a/packages/fleet/src/TemplateStore.ts b/packages/fleet/src/TemplateStore.ts new file mode 100644 index 0000000000..639cdf1701 --- /dev/null +++ b/packages/fleet/src/TemplateStore.ts @@ -0,0 +1,168 @@ +import { mkdir, open, rename, rm, stat, unlink, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { createStack, installMicroProfile } from "@supabase/stack"; +import type { ServiceName, VersionManifest } from "@supabase/stack"; +import { cloneDir } from "./cowClone.ts"; +import { baseTemplateKey, templateKey } from "./PodManifest.ts"; + +const LOCK_STALE_MS = 10 * 60 * 1000; +const LOCK_POLL_MS = 250; + +/** + * Builds and caches golden Postgres template data directories that pods are later + * CoW-cloned from. + * + * - "base" templates are a one-shot `postgres`-only stack boot: `postgres-init` + * applies the baseline roles/schemas/migrations exactly as it does for a normal + * stack, then the micro profile (`micro.conf`/`pod.conf` includes) is installed + * before the data dir is frozen under `root/pg-/data`. + * - "warm" templates clone a base template and boot the requested services once + * so each self-migrates against a `provisioned` (post-init) data dir, then + * freeze the result under `root//data`. An empty service list has + * nothing to warm, so it resolves to the base template directly. + * + * Builds are concurrency-safe on one host via a per-key lockfile created with the + * `wx` flag; a stale lock (holder crashed) is reclaimed after `LOCK_STALE_MS`. + */ +export class TemplateStore { + constructor(private readonly root: string) {} + + private dataDir(key: string): string { + return join(this.root, key, "data"); + } + + async has(key: string): Promise { + return stat(join(this.root, key, "template.json")).then( + () => true, + () => false, + ); + } + + async ensureBaseTemplate(postgresVersion: string): Promise { + const key = baseTemplateKey(postgresVersion); + if (await this.has(key)) return this.dataDir(key); + return this.withLock(key, async () => { + if (await this.has(key)) return this.dataDir(key); + const buildDir = join(this.root, `${key}.build`); + await rm(buildDir, { recursive: true, force: true }); + await mkdir(buildDir, { recursive: true }); + const buildDataDir = join(buildDir, "data"); + // One-shot stack: postgres only, non-provisioned -> postgres-init applies + // roles/schemas/baseline migrations exactly as it does for a normal stack. + const stack = await createStack({ + postgres: { version: postgresVersion, dataDir: buildDataDir }, + postgrest: false, + auth: false, + edgeRuntime: false, + realtime: false, + storage: false, + imgproxy: false, + mailpit: false, + pgmeta: false, + studio: false, + analytics: false, + vector: false, + pooler: false, + functions: false, + }); + try { + await stack.start(); + await stack.ready(); + } finally { + await stack.dispose(); + } + await installMicroProfile(buildDataDir); + await this.freeze(buildDir, key, { key, postgresVersion, builtAt: new Date().toISOString() }); + return this.dataDir(key); + }); + } + + async ensureWarmTemplate( + versions: Partial, + enabledServices: ReadonlyArray, + ): Promise { + const pgVersion = versions.postgres; + if (pgVersion === undefined) throw new Error("versions.postgres is required"); + const base = await this.ensureBaseTemplate(pgVersion); + if (enabledServices.length === 0) return base; + + const key = templateKey(versions); + if (await this.has(key)) return this.dataDir(key); + return this.withLock(key, async () => { + if (await this.has(key)) return this.dataDir(key); + const buildDir = join(this.root, `${key}.build`); + await rm(buildDir, { recursive: true, force: true }); + await mkdir(buildDir, { recursive: true }); + const buildDataDir = join(buildDir, "data"); + await cloneDir(base, buildDataDir); + + // The clone is already post-init, so postgres-init is skipped (`provisioned: true`); + // each enabled service boots once and self-migrates against it. + const stack = await createStack({ + postgres: { + version: pgVersion, + dataDir: buildDataDir, + provisioned: true, + profile: "micro", + }, + postgrest: enabledServices.includes("postgrest") ? {} : false, + auth: enabledServices.includes("auth") ? {} : false, + edgeRuntime: enabledServices.includes("edge-runtime") ? {} : false, + realtime: enabledServices.includes("realtime") ? {} : false, + storage: enabledServices.includes("storage") ? {} : false, + imgproxy: enabledServices.includes("imgproxy") ? {} : false, + mailpit: enabledServices.includes("mailpit") ? {} : false, + pgmeta: enabledServices.includes("pgmeta") ? {} : false, + studio: enabledServices.includes("studio") ? {} : false, + analytics: enabledServices.includes("analytics") ? {} : false, + vector: enabledServices.includes("vector") ? {} : false, + pooler: enabledServices.includes("pooler") ? {} : false, + functions: false, + }); + try { + await stack.start(); + await stack.ready(); + } finally { + await stack.dispose(); + } + await this.freeze(buildDir, key, { + key, + versions, + enabledServices, + builtAt: new Date().toISOString(), + }); + return this.dataDir(key); + }); + } + + private async freeze(buildDir: string, key: string, marker: unknown): Promise { + await mkdir(join(this.root, key), { recursive: true }); + await rename(join(buildDir, "data"), this.dataDir(key)); + await writeFile(join(this.root, key, "template.json"), JSON.stringify(marker)); + await rm(buildDir, { recursive: true, force: true }); + } + + private async withLock(key: string, body: () => Promise): Promise { + const lockPath = join(this.root, `${key}.lock`); + await mkdir(this.root, { recursive: true }); + for (;;) { + try { + const handle = await open(lockPath, "wx"); + await handle.close(); + break; + } catch { + const s = await stat(lockPath).catch(() => undefined); + if (s && Date.now() - s.mtimeMs > LOCK_STALE_MS) { + await unlink(lockPath).catch(() => {}); + continue; + } + await new Promise((r) => setTimeout(r, LOCK_POLL_MS)); + } + } + try { + return await body(); + } finally { + await unlink(lockPath).catch(() => {}); + } + } +} diff --git a/packages/fleet/src/index.ts b/packages/fleet/src/index.ts index 26a91b0a3e..f022543a4e 100644 --- a/packages/fleet/src/index.ts +++ b/packages/fleet/src/index.ts @@ -5,3 +5,4 @@ export type { PodManifest } from "./PodManifest.ts"; export { baseTemplateKey, templateKey } from "./PodManifest.ts"; export type { PodPorts } from "./PortRegistry.ts"; export { PortRegistry } from "./PortRegistry.ts"; +export { TemplateStore } from "./TemplateStore.ts"; diff --git a/packages/stack/src/index.ts b/packages/stack/src/index.ts index 087b40503e..59750abf23 100644 --- a/packages/stack/src/index.ts +++ b/packages/stack/src/index.ts @@ -26,3 +26,4 @@ export type { PrefetchOptions, PrefetchResult } from "./prefetch.ts"; export type { ReadyOptions, StackHandle } from "./createStack.ts"; export type { FunctionsConfig, FunctionsRuntimeConfig } from "./functions.ts"; export { UnixHttpClient, UnixHttpClientError } from "./UnixHttpClient.ts"; +export { installMicroProfile, readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; From db0e27d72c42c7206fda90e7f185c89b1badf824 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 14:25:16 +0200 Subject: [PATCH 19/38] feat(fleet): pod registry and provisioner (create/reset/fork/destroy) PodRegistry persists pod manifests at podsRoot//pod.json. Provisioner creates pods by allocating ports and CoW-cloning a base or warm template into the pod's data dir, resets pods by re-cloning from the base template, forks a pod's data dir under a fresh id and ports, and destroys pods by removing their directory and releasing ports. --- packages/fleet/src/PodRegistry.ts | 39 ++++++++ .../fleet/src/Provisioner.integration.test.ts | 55 +++++++++++ packages/fleet/src/Provisioner.ts | 97 +++++++++++++++++++ packages/fleet/src/index.ts | 3 + 4 files changed, 194 insertions(+) create mode 100644 packages/fleet/src/PodRegistry.ts create mode 100644 packages/fleet/src/Provisioner.integration.test.ts create mode 100644 packages/fleet/src/Provisioner.ts diff --git a/packages/fleet/src/PodRegistry.ts b/packages/fleet/src/PodRegistry.ts new file mode 100644 index 0000000000..5909f3fbc7 --- /dev/null +++ b/packages/fleet/src/PodRegistry.ts @@ -0,0 +1,39 @@ +import { mkdir, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import type { PodManifest } from "./PodManifest.ts"; + +/** + * Persists pod manifests on disk, one per pod directory: `podsRoot//pod.json`. + * The pod's data directory lives alongside it at `podsRoot//data`. + */ +export class PodRegistry { + constructor(private readonly podsRoot: string) {} + + podDir(id: string): string { + return join(this.podsRoot, id); + } + + dataDir(id: string): string { + return join(this.podsRoot, id, "data"); + } + + async read(id: string): Promise { + const raw = await readFile(join(this.podDir(id), "pod.json"), "utf8").catch(() => undefined); + return raw === undefined ? undefined : (JSON.parse(raw) as PodManifest); + } + + async write(manifest: PodManifest): Promise { + await mkdir(this.podDir(manifest.id), { recursive: true }); + await writeFile(join(this.podDir(manifest.id), "pod.json"), JSON.stringify(manifest, null, 2)); + } + + async list(): Promise { + const entries = await readdir(this.podsRoot).catch(() => [] as string[]); + const manifests = await Promise.all(entries.map((id) => this.read(id))); + return manifests.filter((m): m is PodManifest => m !== undefined); + } + + async remove(id: string): Promise { + await rm(this.podDir(id), { recursive: true, force: true }); + } +} diff --git a/packages/fleet/src/Provisioner.integration.test.ts b/packages/fleet/src/Provisioner.integration.test.ts new file mode 100644 index 0000000000..792f28506f --- /dev/null +++ b/packages/fleet/src/Provisioner.integration.test.ts @@ -0,0 +1,55 @@ +import { mkdtemp, readFile, stat, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { PodRegistry } from "./PodRegistry.ts"; +import { PortRegistry } from "./PortRegistry.ts"; +import { Provisioner } from "./Provisioner.ts"; +import { TemplateStore } from "./TemplateStore.ts"; + +const PG_VERSION = "17.6.1.143"; + +describe.skipIf(!process.env.FLEET_PG_TESTS)("Provisioner", () => { + async function makeProvisioner() { + const root = await mkdtemp(join(tmpdir(), "fleet-")); + const templates = new TemplateStore(join(root, "templates")); + const pods = new PodRegistry(join(root, "pods")); + const ports = await PortRegistry.load(join(root, "fleet-state.json")); + return { p: new Provisioner({ templates, pods, ports }), pods }; + } + + it("creates, forks, resets, destroys", async () => { + const { p, pods } = await makeProvisioner(); + const a = await p.create({ id: "a", versions: { postgres: PG_VERSION } }); + expect(a.ports.dbPort).toBeGreaterThan(0); + expect(await stat(join(pods.dataDir("a"), "PG_VERSION")).then(() => true)).toBe(true); + + // fork: divergence + await writeFile(join(pods.dataDir("a"), "marker.txt"), "from-a"); + const b = await p.fork("a", "b"); + expect(b.ports.dbPort).not.toBe(a.ports.dbPort); + await writeFile(join(pods.dataDir("b"), "marker.txt"), "from-b"); + expect(await readFile(join(pods.dataDir("a"), "marker.txt"), "utf8")).toBe("from-a"); + + // reset: marker disappears (re-cloned from template) + await p.reset("a"); + expect( + await stat(join(pods.dataDir("a"), "marker.txt")).then( + () => true, + () => false, + ), + ).toBe(false); + + await p.destroy("a"); + await p.destroy("b"); + expect(await pods.list()).toEqual([]); + }, 300_000); + + it("rejects duplicate ids", async () => { + const { p } = await makeProvisioner(); + await p.create({ id: "dup", versions: { postgres: PG_VERSION } }); + await expect(p.create({ id: "dup", versions: { postgres: PG_VERSION } })).rejects.toThrow( + /exists/, + ); + }, 300_000); +}); diff --git a/packages/fleet/src/Provisioner.ts b/packages/fleet/src/Provisioner.ts new file mode 100644 index 0000000000..30efa0e95f --- /dev/null +++ b/packages/fleet/src/Provisioner.ts @@ -0,0 +1,97 @@ +import { rm } from "node:fs/promises"; +import type { ServiceName, VersionManifest } from "@supabase/stack"; +import { cloneDir } from "./cowClone.ts"; +import type { PodManifest } from "./PodManifest.ts"; +import type { PodRegistry } from "./PodRegistry.ts"; +import type { PortRegistry } from "./PortRegistry.ts"; +import type { TemplateStore } from "./TemplateStore.ts"; + +export interface CreatePodOptions { + readonly id: string; + readonly versions: Partial; + readonly services?: Partial>; + readonly flags?: { readonly supautils?: boolean }; + /** Build/use a warm template (services pre-migrated). Default: base template. */ + readonly warm?: boolean; +} + +/** + * Creates, resets, forks, and destroys pods by CoW-cloning template data + * directories (via `TemplateStore`) into per-pod data directories tracked by + * `PodRegistry`, and allocating/releasing ports via `PortRegistry`. + */ +export class Provisioner { + constructor( + private readonly deps: { + readonly templates: TemplateStore; + readonly pods: PodRegistry; + readonly ports: PortRegistry; + }, + ) {} + + async create(opts: CreatePodOptions): Promise { + const { templates, pods, ports } = this.deps; + if ((await pods.read(opts.id)) !== undefined) { + throw new Error(`pod already exists: ${opts.id}`); + } + const pgVersion = opts.versions.postgres; + if (pgVersion === undefined) throw new Error("versions.postgres is required"); + const enabled = Object.entries(opts.services ?? {}) + .filter(([, on]) => on === true) + .map(([name]) => name as ServiceName); + const template = + opts.warm === true + ? await templates.ensureWarmTemplate(opts.versions, enabled) + : await templates.ensureBaseTemplate(pgVersion); + const allocated = await ports.allocate(opts.id); + await cloneDir(template, pods.dataDir(opts.id)); + const manifest: PodManifest = { + id: opts.id, + versions: opts.versions, + services: opts.services ?? {}, + flags: { supautils: opts.flags?.supautils ?? false }, + ports: allocated, + createdAt: new Date().toISOString(), + }; + await pods.write(manifest); + return manifest; + } + + /** Re-clones the pod's data dir from the base template of its postgres version. */ + async reset(id: string): Promise { + const { templates, pods } = this.deps; + const manifest = await pods.read(id); + if (manifest === undefined) throw new Error(`unknown pod: ${id}`); + const pgVersion = manifest.versions.postgres; + if (pgVersion === undefined) throw new Error(`pod ${id} has no postgres version`); + const template = await templates.ensureBaseTemplate(pgVersion); + await rm(pods.dataDir(id), { recursive: true, force: true }); + await cloneDir(template, pods.dataDir(id)); + } + + /** Caller must ensure the source pod is stopped/suspended first. */ + async fork(sourceId: string, newId: string): Promise { + const { pods, ports } = this.deps; + const source = await pods.read(sourceId); + if (source === undefined) throw new Error(`unknown pod: ${sourceId}`); + if ((await pods.read(newId)) !== undefined) { + throw new Error(`pod already exists: ${newId}`); + } + const allocated = await ports.allocate(newId); + await cloneDir(pods.dataDir(sourceId), pods.dataDir(newId)); + const manifest: PodManifest = { + ...source, + id: newId, + ports: allocated, + createdAt: new Date().toISOString(), + }; + await pods.write(manifest); + return manifest; + } + + async destroy(id: string): Promise { + const { pods, ports } = this.deps; + await pods.remove(id); + await ports.release(id); + } +} diff --git a/packages/fleet/src/index.ts b/packages/fleet/src/index.ts index f022543a4e..c710fed7e5 100644 --- a/packages/fleet/src/index.ts +++ b/packages/fleet/src/index.ts @@ -3,6 +3,9 @@ export const FLEET_PACKAGE = "@supabase/fleet"; export { cloneDir } from "./cowClone.ts"; export type { PodManifest } from "./PodManifest.ts"; export { baseTemplateKey, templateKey } from "./PodManifest.ts"; +export { PodRegistry } from "./PodRegistry.ts"; export type { PodPorts } from "./PortRegistry.ts"; export { PortRegistry } from "./PortRegistry.ts"; +export type { CreatePodOptions } from "./Provisioner.ts"; +export { Provisioner } from "./Provisioner.ts"; export { TemplateStore } from "./TemplateStore.ts"; From 4a0cf4d904c89d8d6b7b8bb8d9d3ddcf0b5b15c9 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 14:44:43 +0200 Subject: [PATCH 20/38] fix(fleet): release ports on failed create/fork before manifest write Provisioner.create() and fork() allocated ports before cloning the data directory and writing the pod manifest. If cloneDir or pods.write threw, the allocated port pair stayed recorded in the PortRegistry state file with no pod ever created to release it. Wrap the post-allocation steps in try/catch and release the ports (plus best-effort cleanup of any partially-written data/pod dir) before rethrowing. Pre-checks (duplicate id, unknown source) still run before allocation so failure cleanup can never release a pre-existing pod's ports. Co-Authored-By: Claude Fable 5 --- packages/fleet/src/Provisioner.ts | 54 +++++--- packages/fleet/src/Provisioner.unit.test.ts | 140 ++++++++++++++++++++ 2 files changed, 174 insertions(+), 20 deletions(-) create mode 100644 packages/fleet/src/Provisioner.unit.test.ts diff --git a/packages/fleet/src/Provisioner.ts b/packages/fleet/src/Provisioner.ts index 30efa0e95f..fac901c0a7 100644 --- a/packages/fleet/src/Provisioner.ts +++ b/packages/fleet/src/Provisioner.ts @@ -44,17 +44,24 @@ export class Provisioner { ? await templates.ensureWarmTemplate(opts.versions, enabled) : await templates.ensureBaseTemplate(pgVersion); const allocated = await ports.allocate(opts.id); - await cloneDir(template, pods.dataDir(opts.id)); - const manifest: PodManifest = { - id: opts.id, - versions: opts.versions, - services: opts.services ?? {}, - flags: { supautils: opts.flags?.supautils ?? false }, - ports: allocated, - createdAt: new Date().toISOString(), - }; - await pods.write(manifest); - return manifest; + try { + await cloneDir(template, pods.dataDir(opts.id)); + const manifest: PodManifest = { + id: opts.id, + versions: opts.versions, + services: opts.services ?? {}, + flags: { supautils: opts.flags?.supautils ?? false }, + ports: allocated, + createdAt: new Date().toISOString(), + }; + await pods.write(manifest); + return manifest; + } catch (err) { + await ports.release(opts.id).catch(() => {}); + await rm(pods.dataDir(opts.id), { recursive: true, force: true }).catch(() => {}); + await rm(pods.podDir(opts.id), { recursive: true, force: true }).catch(() => {}); + throw err; + } } /** Re-clones the pod's data dir from the base template of its postgres version. */ @@ -78,15 +85,22 @@ export class Provisioner { throw new Error(`pod already exists: ${newId}`); } const allocated = await ports.allocate(newId); - await cloneDir(pods.dataDir(sourceId), pods.dataDir(newId)); - const manifest: PodManifest = { - ...source, - id: newId, - ports: allocated, - createdAt: new Date().toISOString(), - }; - await pods.write(manifest); - return manifest; + try { + await cloneDir(pods.dataDir(sourceId), pods.dataDir(newId)); + const manifest: PodManifest = { + ...source, + id: newId, + ports: allocated, + createdAt: new Date().toISOString(), + }; + await pods.write(manifest); + return manifest; + } catch (err) { + await ports.release(newId).catch(() => {}); + await rm(pods.dataDir(newId), { recursive: true, force: true }).catch(() => {}); + await rm(pods.podDir(newId), { recursive: true, force: true }).catch(() => {}); + throw err; + } } async destroy(id: string): Promise { diff --git a/packages/fleet/src/Provisioner.unit.test.ts b/packages/fleet/src/Provisioner.unit.test.ts new file mode 100644 index 0000000000..9a37b23bd0 --- /dev/null +++ b/packages/fleet/src/Provisioner.unit.test.ts @@ -0,0 +1,140 @@ +import { mkdir, mkdtemp, readdir, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { ServiceName, VersionManifest } from "@supabase/stack"; +import { describe, expect, it } from "vitest"; +import { PodRegistry } from "./PodRegistry.ts"; +import { PortRegistry } from "./PortRegistry.ts"; +import { Provisioner } from "./Provisioner.ts"; +import type { TemplateStore } from "./TemplateStore.ts"; + +const PG_VERSION = "17.6.1.143"; + +/** + * A minimal stand-in for `TemplateStore` that skips booting a real Postgres + * stack: `ensureBaseTemplate`/`ensureWarmTemplate` just hand back a + * pre-populated (or, for the failure tests, deliberately missing) directory. + * Cast through `unknown` because `TemplateStore` has private members that a + * structural object literal can never satisfy. + */ +function fakeTemplateStore(templateDir: string): TemplateStore { + return { + async ensureBaseTemplate(_postgresVersion: string): Promise { + return templateDir; + }, + async ensureWarmTemplate( + _versions: Partial, + _enabledServices: ReadonlyArray, + ): Promise { + return templateDir; + }, + } as unknown as TemplateStore; +} + +async function makeHarness(templateDir: string) { + const root = await mkdtemp(join(tmpdir(), "fleet-unit-")); + const podsRoot = join(root, "pods"); + const pods = new PodRegistry(podsRoot); + const ports = await PortRegistry.load(join(root, "fleet-state.json")); + const templates = fakeTemplateStore(templateDir); + return { p: new Provisioner({ templates, pods, ports }), pods, ports, podsRoot }; +} + +async function podsRootEntries(podsRoot: string): Promise { + return readdir(podsRoot).catch(() => [] as string[]); +} + +describe("Provisioner (unit, fake deps)", () => { + describe("create", () => { + it("provisions a pod from a valid template", async () => { + const templateDir = await mkdtemp(join(tmpdir(), "fleet-template-")); + await writeFile(join(templateDir, "PG_VERSION"), PG_VERSION); + const { p, pods, ports } = await makeHarness(templateDir); + + const manifest = await p.create({ id: "x", versions: { postgres: PG_VERSION } }); + + expect(manifest.id).toBe("x"); + expect(ports.get("x")).toEqual(manifest.ports); + expect(await pods.read("x")).toEqual(manifest); + }); + + it("releases the allocated ports and cleans up when cloning fails", async () => { + // Non-existent template dir => cloneDir's src stat/copy fails, throwing + // before the manifest is ever written. + const missingTemplateDir = join(await mkdtemp(join(tmpdir(), "fleet-template-")), "missing"); + const { p, ports, podsRoot } = await makeHarness(missingTemplateDir); + + await expect(p.create({ id: "x", versions: { postgres: PG_VERSION } })).rejects.toThrow(); + + expect(ports.get("x")).toBeUndefined(); + expect(await podsRootEntries(podsRoot)).not.toContain("x"); + }); + + it("does not release ports belonging to a pre-existing pod on the duplicate-id path", async () => { + const templateDir = await mkdtemp(join(tmpdir(), "fleet-template-")); + await writeFile(join(templateDir, "PG_VERSION"), PG_VERSION); + const { p, ports } = await makeHarness(templateDir); + + const first = await p.create({ id: "dup", versions: { postgres: PG_VERSION } }); + await expect(p.create({ id: "dup", versions: { postgres: PG_VERSION } })).rejects.toThrow( + /exists/, + ); + + // The duplicate-id pre-check throws before any (re-)allocation, so the + // original pod's ports must still be intact. + expect(ports.get("dup")).toEqual(first.ports); + }); + }); + + describe("fork", () => { + it("clones an existing pod into a new one", async () => { + const templateDir = await mkdtemp(join(tmpdir(), "fleet-template-")); + await writeFile(join(templateDir, "PG_VERSION"), PG_VERSION); + const { p, pods, ports } = await makeHarness(templateDir); + + const source = await p.create({ id: "src", versions: { postgres: PG_VERSION } }); + const forked = await p.fork("src", "dst"); + + expect(forked.id).toBe("dst"); + expect(forked.ports).not.toEqual(source.ports); + expect(ports.get("dst")).toEqual(forked.ports); + expect(await pods.read("dst")).toEqual(forked); + }); + + it("releases the allocated ports and cleans up when the clone fails", async () => { + const templateDir = await mkdtemp(join(tmpdir(), "fleet-template-")); + await writeFile(join(templateDir, "PG_VERSION"), PG_VERSION); + const { p, ports, podsRoot } = await makeHarness(templateDir); + + await p.create({ id: "src", versions: { postgres: PG_VERSION } }); + + // Force the fork's clone step to fail deterministically: cloneDir + // refuses to clone into a destination that already exists. + await mkdir(join(podsRoot, "dst", "data"), { recursive: true }); + + await expect(p.fork("src", "dst")).rejects.toThrow(); + + expect(ports.get("dst")).toBeUndefined(); + // The pre-created dest dir is removed as part of failure cleanup too. + expect(await podsRootEntries(podsRoot)).toContain("src"); + const dstDataEntries = await readdir(join(podsRoot, "dst", "data")).catch(() => undefined); + expect(dstDataEntries).toBeUndefined(); + }); + + it("does not release the source pod's ports when the new id already exists", async () => { + const templateDir = await mkdtemp(join(tmpdir(), "fleet-template-")); + await writeFile(join(templateDir, "PG_VERSION"), PG_VERSION); + const { p, ports } = await makeHarness(templateDir); + + const source = await p.create({ id: "src", versions: { postgres: PG_VERSION } }); + const existing = await p.create({ id: "dst", versions: { postgres: PG_VERSION } }); + + await expect(p.fork("src", "dst")).rejects.toThrow(/exists/); + + // The pre-check for an already-existing target throws before + // (re-)allocating, so neither pod's ports should be disturbed. + expect(ports.get("src")).toEqual(source.ports); + expect(ports.get("dst")).toEqual(existing.ports); + }); + }); +}); From b7fdb5d3b458661f976cb3bfc3b6224835938e66 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 14:56:14 +0200 Subject: [PATCH 21/38] feat(fleet): TCP wake-proxy edge with activity events Adds EdgeProxy: binds a pod's external port once and keeps it bound for the pod's lifetime, awaiting wake() per accepted connection to get the live upstream before splicing bytes both ways. Emits connect/data/disconnect activity per pod for the idle monitor to consume. wake() rejection destroys the client without unhandled rejections; concurrent connections to the same suspended pod both succeed regardless of how many times wake() runs (dedup is the fleet layer's job). Buffers client bytes behind a real 'data' listener (not just pause()) before replaying them to the backend, since some runtimes start sockets flowing before user code attaches a listener and would otherwise silently drop data received while wake() is in flight. --- packages/fleet/src/EdgeProxy.ts | 133 ++++++++++++++++ packages/fleet/src/EdgeProxy.unit.test.ts | 186 ++++++++++++++++++++++ packages/fleet/src/index.ts | 2 + 3 files changed, 321 insertions(+) create mode 100644 packages/fleet/src/EdgeProxy.ts create mode 100644 packages/fleet/src/EdgeProxy.unit.test.ts diff --git a/packages/fleet/src/EdgeProxy.ts b/packages/fleet/src/EdgeProxy.ts new file mode 100644 index 0000000000..f2a80ed639 --- /dev/null +++ b/packages/fleet/src/EdgeProxy.ts @@ -0,0 +1,133 @@ +import { connect, createServer, type Server, type Socket } from "node:net"; + +export interface PodUpstream { + readonly host: string; + readonly port: number; +} + +export interface EdgeProxyEvents { + /** Fired on connect/disconnect/bytes; IdleMonitor consumes these. */ + onActivity: ( + podId: string, + event: "connect" | "data" | "disconnect", + openConnections: number, + ) => void; +} + +interface Registration { + readonly server: Server; + readonly sockets: Set; +} + +/** + * TCP wake-proxy: binds a pod's external port once and keeps it bound for + * the pod's lifetime. Every accepted client connection is paused, then + * `wake()` is awaited to obtain the pod's live upstream address before + * splicing bytes both ways. Activity (connect/data/disconnect) is reported + * per pod so the idle monitor can track usage without touching sockets + * itself. + * + * Design assumptions: + * - **wake() is per-connection, not deduped here.** Two concurrent + * connections to a suspended pod may each call `wake()`; dedup (e.g. + * collapsing concurrent wakes into a single in-flight promise) is the + * fleet layer's responsibility, not EdgeProxy's. Both connections must + * still succeed regardless of how many times `wake()` runs. + * - **wake() rejection destroys the client.** If `wake()` rejects, the + * accepted client socket is destroyed and no upstream connection is + * attempted. The rejection is handled inline (never left as a dangling + * promise) so it can never surface as an unhandled rejection. + */ +export class EdgeProxy { + private readonly registrations = new Map(); + + constructor(private readonly events: Partial = {}) {} + + openConnections(podId: string): number { + return this.registrations.get(podId)?.sockets.size ?? 0; + } + + register(podId: string, listenPort: number, wake: () => Promise): Promise { + const sockets = new Set(); + const emit = (event: "connect" | "data" | "disconnect") => + this.events.onActivity?.(podId, event, sockets.size); + + const server = createServer((client) => { + sockets.add(client); + emit("connect"); + + let disconnected = false; + const cleanup = () => { + if (disconnected) return; + disconnected = true; + sockets.delete(client); + emit("disconnect"); + }; + client.on("close", cleanup); + client.on("error", cleanup); + + // Hold any bytes the client sends while `wake()` is in flight. Merely + // calling `client.pause()` is not sufficient: some runtimes start + // sockets flowing before user code gets a chance to react, silently + // dropping data received before a listener is attached. Attaching a + // real `data` listener up front guarantees nothing is lost; buffered + // chunks are replayed to the backend once it's connected, then the + // socket is handed off to `pipe()` for the rest of the stream. + const buffered: Buffer[] = []; + const bufferChunk = (chunk: Buffer) => buffered.push(chunk); + client.on("data", bufferChunk); + client.pause(); + + wake().then( + (upstream) => { + // The client may have already disconnected while wake() was + // in flight; don't bother connecting an upstream nobody needs. + if (disconnected) return; + + const backend = connect(upstream.port, upstream.host); + backend.on("error", () => { + client.destroy(); + backend.destroy(); + }); + client.on("close", () => backend.destroy()); + backend.on("close", () => client.destroy()); + backend.on("data", () => emit("data")); + backend.on("connect", () => { + client.off("data", bufferChunk); + client.on("data", () => emit("data")); + for (const chunk of buffered) { + backend.write(chunk); + emit("data"); + } + client.pipe(backend); + backend.pipe(client); + client.resume(); + }); + }, + () => { + // wake() failed: destroy the client without letting the + // rejection escape as an unhandled rejection. + client.destroy(); + }, + ); + }); + + this.registrations.set(podId, { server, sockets }); + return new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(listenPort, "127.0.0.1", () => resolve()); + }); + } + + async unregister(podId: string): Promise { + const reg = this.registrations.get(podId); + if (!reg) return; + this.registrations.delete(podId); + for (const sock of reg.sockets) sock.destroy(); + await new Promise((resolve) => reg.server.close(() => resolve())); + } + + async close(): Promise { + await Promise.all([...this.registrations.keys()].map((id) => this.unregister(id))); + } +} diff --git a/packages/fleet/src/EdgeProxy.unit.test.ts b/packages/fleet/src/EdgeProxy.unit.test.ts new file mode 100644 index 0000000000..5438d88bc7 --- /dev/null +++ b/packages/fleet/src/EdgeProxy.unit.test.ts @@ -0,0 +1,186 @@ +import { type AddressInfo, connect, createServer, type Server } from "node:net"; +import { afterEach, describe, expect, it } from "vitest"; +import { EdgeProxy } from "./EdgeProxy.ts"; + +function echoServer(): Promise<{ server: Server; port: number }> { + return new Promise((resolve) => { + const server = createServer((sock) => sock.pipe(sock)); + server.listen(0, "127.0.0.1", () => + resolve({ server, port: (server.address() as AddressInfo).port }), + ); + }); +} + +function freePort(): Promise { + return new Promise((resolve) => { + const s = createServer(); + s.listen(0, "127.0.0.1", () => { + const port = (s.address() as AddressInfo).port; + s.close(() => resolve(port)); + }); + }); +} + +describe("EdgeProxy", () => { + const proxies: EdgeProxy[] = []; + afterEach(async () => { + for (const p of proxies.splice(0)) await p.close(); + }); + + it("wakes on first connection and splices bytes both ways", async () => { + const { server, port: upstreamPort } = await echoServer(); + let wakes = 0; + const proxy = new EdgeProxy(); + proxies.push(proxy); + const listenPort = await freePort(); + await proxy.register("pod-a", listenPort, async () => { + wakes += 1; + return { host: "127.0.0.1", port: upstreamPort }; + }); + + const reply = await new Promise((resolve, reject) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write("ping")); + sock.on("data", (d) => { + resolve(d.toString()); + sock.end(); + }); + sock.on("error", reject); + }); + expect(reply).toBe("ping"); + expect(wakes).toBe(1); + server.close(); + }); + + it("tracks open connections and reports activity", async () => { + const { server, port: upstreamPort } = await echoServer(); + const events: string[] = []; + const proxy = new EdgeProxy({ + onActivity: (id, ev) => events.push(`${id}:${ev}`), + }); + proxies.push(proxy); + const listenPort = await freePort(); + await proxy.register("pod-b", listenPort, async () => ({ + host: "127.0.0.1", + port: upstreamPort, + })); + + await new Promise((resolve) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write("x")); + sock.on("data", () => sock.end()); + sock.on("close", () => resolve()); + }); + await new Promise((r) => setTimeout(r, 50)); + expect(events).toContain("pod-b:connect"); + expect(events).toContain("pod-b:data"); + expect(events).toContain("pod-b:disconnect"); + expect(proxy.openConnections("pod-b")).toBe(0); + server.close(); + }); + + it("destroys the client socket when wake() rejects, and stays usable afterward", async () => { + const proxy = new EdgeProxy(); + proxies.push(proxy); + const listenPort = await freePort(); + let attempts = 0; + await proxy.register("pod-c", listenPort, async () => { + attempts += 1; + if (attempts === 1) throw new Error("wake failed"); + return { host: "127.0.0.1", port: await freePort() }; + }); + + // First connection: wake() rejects, socket should close/error without + // taking down the process (no unhandled rejection) and without wedging + // the listener. + await new Promise((resolve) => { + const sock = connect(listenPort, "127.0.0.1"); + let settled = false; + const finish = () => { + if (!settled) { + settled = true; + resolve(); + } + }; + sock.on("error", finish); + sock.on("close", finish); + }); + + // Second connection: proxy must still be usable. Use a real echo server + // for the second wake so we can prove the pipe still works. + const { server, port: upstreamPort } = await echoServer(); + await proxy.unregister("pod-c"); + await proxy.register("pod-c", listenPort, async () => ({ + host: "127.0.0.1", + port: upstreamPort, + })); + + const reply = await new Promise((resolve, reject) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write("hello")); + sock.on("data", (d) => { + resolve(d.toString()); + sock.end(); + }); + sock.on("error", reject); + }); + expect(reply).toBe("hello"); + server.close(); + }); + + it("handles two concurrent connections to the same suspended pod, both succeed", async () => { + const { server, port: upstreamPort } = await echoServer(); + let wakeCalls = 0; + const proxy = new EdgeProxy(); + proxies.push(proxy); + const listenPort = await freePort(); + await proxy.register("pod-d", listenPort, async () => { + wakeCalls += 1; + // Simulate a suspended pod that takes a bit to wake. + await new Promise((r) => setTimeout(r, 20)); + return { host: "127.0.0.1", port: upstreamPort }; + }); + + const connectAndEcho = (payload: string) => + new Promise((resolve, reject) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write(payload)); + sock.on("data", (d) => { + resolve(d.toString()); + sock.end(); + }); + sock.on("error", reject); + }); + + const [replyA, replyB] = await Promise.all([connectAndEcho("aaa"), connectAndEcho("bbb")]); + expect(replyA).toBe("aaa"); + expect(replyB).toBe("bbb"); + // Contract: wake() may be called once or twice; dedup happens at the + // fleet layer, not in EdgeProxy. + expect(wakeCalls).toBeGreaterThanOrEqual(1); + expect(wakeCalls).toBeLessThanOrEqual(2); + server.close(); + }); + + it("does not connect a dangling backend when unregistered while wake() is in flight", async () => { + const proxy = new EdgeProxy(); + proxies.push(proxy); + const listenPort = await freePort(); + const upstreamPort = await freePort(); // nothing listening here + + await proxy.register("pod-e", listenPort, async () => { + await new Promise((r) => setTimeout(r, 100)); + return { host: "127.0.0.1", port: upstreamPort }; + }); + + await new Promise((resolve) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write("x")); + sock.on("error", () => resolve()); + sock.on("close", () => resolve()); + setTimeout(() => { + proxy.unregister("pod-e").catch(() => {}); + }, 20); + }); + + // Wait past wake()'s resolution point; if EdgeProxy tried to connect a + // backend for the now-destroyed client, it would leak a socket/listener. + await new Promise((r) => setTimeout(r, 150)); + expect(proxy.openConnections("pod-e")).toBe(0); + }); +}); diff --git a/packages/fleet/src/index.ts b/packages/fleet/src/index.ts index c710fed7e5..bd556cf8df 100644 --- a/packages/fleet/src/index.ts +++ b/packages/fleet/src/index.ts @@ -1,6 +1,8 @@ export const FLEET_PACKAGE = "@supabase/fleet"; export { cloneDir } from "./cowClone.ts"; +export type { EdgeProxyEvents, PodUpstream } from "./EdgeProxy.ts"; +export { EdgeProxy } from "./EdgeProxy.ts"; export type { PodManifest } from "./PodManifest.ts"; export { baseTemplateKey, templateKey } from "./PodManifest.ts"; export { PodRegistry } from "./PodRegistry.ts"; From 62bcf18df65f9e01f35fc82df96d4ff81393029d Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 15:33:00 +0200 Subject: [PATCH 22/38] fix(fleet): cap pre-wake buffer and catch throws from wake() resolution Two review findings in EdgeProxy: (1) a client streaming data at a suspended pod during a slow wake() buffered without limit; add MAX_PREWAKE_BUFFER_BYTES (1 MiB) and destroy the client socket if it's exceeded before the backend is reachable. Fixing this exposed that the prior data+manual-array buffering never actually delivered data events while the socket was paused (confirmed under both Bun and Node), so the buffering mechanism was switched to readable+read(), which fires while paused and makes byte counting/capping actually work. (2) wake() resolving with a malformed upstream (e.g. an invalid port) could throw synchronously inside the .then() resolve handler, surfacing as an unhandled rejection; wrap that branch in try/catch routed to the same client-destroy path, and correct the doc comment's overstated guarantee. Also drop the redundant client.resume() after pipe() and note that the activity data listener is independent of pipe()'s internal listener. Co-Authored-By: Claude Fable 5 --- packages/fleet/src/EdgeProxy.ts | 113 ++++++++++++++++------ packages/fleet/src/EdgeProxy.unit.test.ts | 110 ++++++++++++++++++++- 2 files changed, 192 insertions(+), 31 deletions(-) diff --git a/packages/fleet/src/EdgeProxy.ts b/packages/fleet/src/EdgeProxy.ts index f2a80ed639..e9c12280fc 100644 --- a/packages/fleet/src/EdgeProxy.ts +++ b/packages/fleet/src/EdgeProxy.ts @@ -5,6 +5,16 @@ export interface PodUpstream { readonly port: number; } +/** + * Cap on bytes buffered per client while `wake()` is in flight. Postgres + * startup packets and HTTP request heads are tiny (low kilobytes at most); + * 1 MiB is generous headroom for either protocol while still bounding memory + * if a client streams data at a suspended pod during a slow wake. A client + * that exceeds this cap before the backend is reachable is treated as + * misbehaving (or attacking) and has its socket destroyed. + */ +export const MAX_PREWAKE_BUFFER_BYTES = 1024 * 1024; // 1 MiB + export interface EdgeProxyEvents { /** Fired on connect/disconnect/bytes; IdleMonitor consumes these. */ onActivity: ( @@ -35,8 +45,17 @@ interface Registration { * still succeed regardless of how many times `wake()` runs. * - **wake() rejection destroys the client.** If `wake()` rejects, the * accepted client socket is destroyed and no upstream connection is - * attempted. The rejection is handled inline (never left as a dangling - * promise) so it can never surface as an unhandled rejection. + * attempted. + * - **Pre-wake buffering is capped.** Bytes received from the client while + * `wake()` is in flight are buffered (see above) up to + * `MAX_PREWAKE_BUFFER_BYTES`. A client that exceeds the cap before the + * backend is reachable has its socket destroyed; the usual disconnect + * activity event still fires exactly once. + * - **Nothing here can surface as an unhandled rejection.** Both the + * rejection path and any synchronous throw from the resolution path + * (e.g. `wake()` resolving with a malformed upstream address that makes + * `net.connect()` throw) are caught and routed to the same client-destroy + * path. */ export class EdgeProxy { private readonly registrations = new Map(); @@ -66,43 +85,77 @@ export class EdgeProxy { client.on("close", cleanup); client.on("error", cleanup); - // Hold any bytes the client sends while `wake()` is in flight. Merely - // calling `client.pause()` is not sufficient: some runtimes start - // sockets flowing before user code gets a chance to react, silently - // dropping data received before a listener is attached. Attaching a - // real `data` listener up front guarantees nothing is lost; buffered - // chunks are replayed to the backend once it's connected, then the - // socket is handed off to `pipe()` for the rest of the stream. + // Hold any bytes the client sends while `wake()` is in flight. A plain + // `data` listener does not fire while the socket is paused (Node only + // delivers `data` once flowing, i.e. after `resume()`/`pipe()`), so we + // use `readable` + `read()` instead: those fire even in paused mode, + // draining the socket's internal buffer as bytes arrive and letting us + // count them without ever switching the stream into flowing mode. + // Buffered chunks are replayed to the backend once it's connected, + // then the socket is handed off to `pipe()` for the rest of the + // stream. + // + // Buffering is capped at MAX_PREWAKE_BUFFER_BYTES: a client that + // floods the socket while the backend isn't reachable yet (e.g. a + // slow wake()) is destroyed rather than allowed to grow this array + // without bound. const buffered: Buffer[] = []; - const bufferChunk = (chunk: Buffer) => buffered.push(chunk); - client.on("data", bufferChunk); + let bufferedBytes = 0; + let overflowed = false; + const onReadable = () => { + if (overflowed) return; + for (;;) { + const chunk = client.read() as Buffer | null; + if (chunk === null) return; + bufferedBytes += chunk.length; + if (bufferedBytes > MAX_PREWAKE_BUFFER_BYTES) { + overflowed = true; + buffered.length = 0; + client.destroy(); + return; + } + buffered.push(chunk); + } + }; + client.on("readable", onReadable); client.pause(); wake().then( (upstream) => { // The client may have already disconnected while wake() was - // in flight; don't bother connecting an upstream nobody needs. + // in flight (including due to a pre-wake buffer overflow); don't + // bother connecting an upstream nobody needs. if (disconnected) return; - const backend = connect(upstream.port, upstream.host); - backend.on("error", () => { + try { + const backend = connect(upstream.port, upstream.host); + backend.on("error", () => { + client.destroy(); + backend.destroy(); + }); + client.on("close", () => backend.destroy()); + backend.on("close", () => client.destroy()); + backend.on("data", () => emit("data")); + backend.on("connect", () => { + client.off("readable", onReadable); + client.on("data", () => emit("data")); + for (const chunk of buffered) { + backend.write(chunk); + emit("data"); + } + client.pipe(backend); + backend.pipe(client); + // The activity `data` listener above is independent of the + // listener `pipe()` attaches internally; both observe the + // same events, this isn't double-consumption of the stream. + }); + } catch { + // wake() resolved, but the upstream address it handed back was + // unusable (e.g. an invalid port that makes net.connect() + // throw synchronously). Destroy the client instead of letting + // the throw escape as an unhandled rejection. client.destroy(); - backend.destroy(); - }); - client.on("close", () => backend.destroy()); - backend.on("close", () => client.destroy()); - backend.on("data", () => emit("data")); - backend.on("connect", () => { - client.off("data", bufferChunk); - client.on("data", () => emit("data")); - for (const chunk of buffered) { - backend.write(chunk); - emit("data"); - } - client.pipe(backend); - backend.pipe(client); - client.resume(); - }); + } }, () => { // wake() failed: destroy the client without letting the diff --git a/packages/fleet/src/EdgeProxy.unit.test.ts b/packages/fleet/src/EdgeProxy.unit.test.ts index 5438d88bc7..36f4964eeb 100644 --- a/packages/fleet/src/EdgeProxy.unit.test.ts +++ b/packages/fleet/src/EdgeProxy.unit.test.ts @@ -1,6 +1,6 @@ import { type AddressInfo, connect, createServer, type Server } from "node:net"; import { afterEach, describe, expect, it } from "vitest"; -import { EdgeProxy } from "./EdgeProxy.ts"; +import { EdgeProxy, MAX_PREWAKE_BUFFER_BYTES } from "./EdgeProxy.ts"; function echoServer(): Promise<{ server: Server; port: number }> { return new Promise((resolve) => { @@ -183,4 +183,112 @@ describe("EdgeProxy", () => { await new Promise((r) => setTimeout(r, 150)); expect(proxy.openConnections("pod-e")).toBe(0); }); + + it("destroys the client if it floods the pre-wake buffer past the cap, and stays usable afterward", async () => { + const proxy = new EdgeProxy(); + proxies.push(proxy); + const listenPort = await freePort(); + + let unhandledRejection: unknown; + const onUnhandledRejection = (reason: unknown) => { + unhandledRejection = reason; + }; + process.on("unhandledRejection", onUnhandledRejection); + + try { + await proxy.register("pod-f", listenPort, async () => { + // Slow wake gives the flooding client plenty of time to exceed the cap. + await new Promise((r) => setTimeout(r, 200)); + return { host: "127.0.0.1", port: await freePort() }; + }); + + const clientDestroyed = await new Promise((resolve) => { + const sock = connect(listenPort, "127.0.0.1", () => { + // A single write just over the cap: this floods immediately and + // reflects that the cap must trigger regardless of how the + // client chooses to chunk its writes. + sock.write(Buffer.alloc(MAX_PREWAKE_BUFFER_BYTES + 64 * 1024, "a")); + }); + sock.on("error", () => resolve(true)); + sock.on("close", () => resolve(sock.destroyed)); + }); + expect(clientDestroyed).toBe(true); + + // Give any lingering wake() resolution a chance to run before we assert + // there was no unhandled rejection. + await new Promise((r) => setTimeout(r, 250)); + expect(unhandledRejection).toBeUndefined(); + + // Proxy must still be usable for a subsequent normal connection. + const { server, port: upstreamPort } = await echoServer(); + await proxy.unregister("pod-f"); + await proxy.register("pod-f", listenPort, async () => ({ + host: "127.0.0.1", + port: upstreamPort, + })); + + const reply = await new Promise((resolve, reject) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write("hello")); + sock.on("data", (d) => { + resolve(d.toString()); + sock.end(); + }); + sock.on("error", reject); + }); + expect(reply).toBe("hello"); + server.close(); + } finally { + process.off("unhandledRejection", onUnhandledRejection); + } + }); + + it("destroys the client without an unhandled rejection when wake() resolves with a garbage upstream", async () => { + const proxy = new EdgeProxy(); + proxies.push(proxy); + const listenPort = await freePort(); + + let unhandledRejection: unknown; + const onUnhandledRejection = (reason: unknown) => { + unhandledRejection = reason; + }; + process.on("unhandledRejection", onUnhandledRejection); + + try { + await proxy.register("pod-g", listenPort, async () => ({ + host: "127.0.0.1", + port: -1, + })); + + const clientDestroyed = await new Promise((resolve) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write("x")); + sock.on("error", () => resolve(true)); + sock.on("close", () => resolve(sock.destroyed)); + }); + expect(clientDestroyed).toBe(true); + + await new Promise((r) => setTimeout(r, 50)); + expect(unhandledRejection).toBeUndefined(); + + // Proxy must still be usable for a subsequent normal connection. + const { server, port: upstreamPort } = await echoServer(); + await proxy.unregister("pod-g"); + await proxy.register("pod-g", listenPort, async () => ({ + host: "127.0.0.1", + port: upstreamPort, + })); + + const reply = await new Promise((resolve, reject) => { + const sock = connect(listenPort, "127.0.0.1", () => sock.write("hello")); + sock.on("data", (d) => { + resolve(d.toString()); + sock.end(); + }); + sock.on("error", reject); + }); + expect(reply).toBe("hello"); + server.close(); + } finally { + process.off("unhandledRejection", onUnhandledRejection); + } + }); }); From 9f274068faa4c9e62be1a47bcd5ee7c52762eab8 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 15:46:38 +0200 Subject: [PATCH 23/38] feat(fleet): idle monitor with connection-aware countdown Adds IdleMonitor: fires onIdle(podId) once a tracked pod has no open connections and no activity for idleMs. Open connections hold the pod warm indefinitely; any recordActivity call (re)arms the countdown; onIdle fires at most once per warm period and untracks the pod. --- packages/fleet/src/IdleMonitor.ts | 68 +++++++++++++++++ packages/fleet/src/IdleMonitor.unit.test.ts | 82 +++++++++++++++++++++ packages/fleet/src/index.ts | 1 + 3 files changed, 151 insertions(+) create mode 100644 packages/fleet/src/IdleMonitor.ts create mode 100644 packages/fleet/src/IdleMonitor.unit.test.ts diff --git a/packages/fleet/src/IdleMonitor.ts b/packages/fleet/src/IdleMonitor.ts new file mode 100644 index 0000000000..90ec09ab90 --- /dev/null +++ b/packages/fleet/src/IdleMonitor.ts @@ -0,0 +1,68 @@ +interface Tracked { + timer: ReturnType | undefined; + openConnections: number; +} + +/** + * Connection-aware idle countdown: fires `onIdle(podId)` once a tracked pod + * has no open connections and no activity for `idleMs`. + * + * Design assumptions: + * - **Open connections hold the pod warm indefinitely.** While + * `openConnections > 0` (as last reported via `recordActivity`) the + * countdown never starts, no matter how much time passes. + * - **The countdown restarts on the last connection closing or on any + * activity.** Both are reported via `recordActivity`; each call re-arms + * the timer from `idleMs`. + * - **`onIdle` fires at most once per warm period.** When the timer fires, + * the pod is untracked before `onIdle` runs; a subsequent `track()` is + * required to re-arm it. + * - **`track()` on an already-tracked pod is a no-op.** It does not reset + * an in-flight countdown; callers that want a reset should use + * `recordActivity` instead. + */ +export class IdleMonitor { + private readonly tracked = new Map(); + + constructor( + private readonly opts: { + readonly idleMs: number; + readonly onIdle: (podId: string) => void; + }, + ) {} + + /** Start tracking a pod (e.g. on wake). No-op if already tracked. */ + track(podId: string): void { + if (!this.tracked.has(podId)) { + this.tracked.set(podId, { timer: undefined, openConnections: 0 }); + this.arm(podId); + } + } + + /** Stop tracking (on suspend/destroy). */ + untrack(podId: string): void { + const entry = this.tracked.get(podId); + if (entry?.timer) clearTimeout(entry.timer); + this.tracked.delete(podId); + } + + /** Wire to EdgeProxy events: any activity resets the timer; open connections hold it. */ + recordActivity(podId: string, openConnections: number): void { + const entry = this.tracked.get(podId); + if (!entry) return; + entry.openConnections = openConnections; + this.arm(podId); + } + + private arm(podId: string): void { + const entry = this.tracked.get(podId); + if (!entry) return; + if (entry.timer) clearTimeout(entry.timer); + entry.timer = undefined; + if (entry.openConnections > 0) return; // held warm by open connections + entry.timer = setTimeout(() => { + this.tracked.delete(podId); + this.opts.onIdle(podId); + }, this.opts.idleMs); + } +} diff --git a/packages/fleet/src/IdleMonitor.unit.test.ts b/packages/fleet/src/IdleMonitor.unit.test.ts new file mode 100644 index 0000000000..6916091e80 --- /dev/null +++ b/packages/fleet/src/IdleMonitor.unit.test.ts @@ -0,0 +1,82 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { IdleMonitor } from "./IdleMonitor.ts"; + +describe("IdleMonitor", () => { + beforeEach(() => vi.useFakeTimers()); + afterEach(() => vi.useRealTimers()); + + it("fires onIdle after idleMs with no connections and no activity", () => { + const idled: string[] = []; + const mon = new IdleMonitor({ idleMs: 1000, onIdle: (id) => idled.push(id) }); + mon.track("a"); + mon.recordActivity("a", 0); + vi.advanceTimersByTime(999); + expect(idled).toEqual([]); + vi.advanceTimersByTime(2); + expect(idled).toEqual(["a"]); + }); + + it("open connections hold the pod warm indefinitely", () => { + const idled: string[] = []; + const mon = new IdleMonitor({ idleMs: 1000, onIdle: (id) => idled.push(id) }); + mon.track("a"); + mon.recordActivity("a", 1); // one open connection + vi.advanceTimersByTime(10_000); + expect(idled).toEqual([]); + mon.recordActivity("a", 0); // last connection closed + vi.advanceTimersByTime(1001); + expect(idled).toEqual(["a"]); + }); + + it("activity resets the countdown; untrack cancels", () => { + const idled: string[] = []; + const mon = new IdleMonitor({ idleMs: 1000, onIdle: (id) => idled.push(id) }); + mon.track("a"); + mon.recordActivity("a", 0); + vi.advanceTimersByTime(900); + mon.recordActivity("a", 0); // reset + vi.advanceTimersByTime(900); + expect(idled).toEqual([]); + mon.untrack("a"); + vi.advanceTimersByTime(5000); + expect(idled).toEqual([]); + }); + + it("onIdle fires at most once per warm period and untracks the pod", () => { + const idled: string[] = []; + const mon = new IdleMonitor({ idleMs: 1000, onIdle: (id) => idled.push(id) }); + mon.track("a"); + mon.recordActivity("a", 0); + vi.advanceTimersByTime(1000); + expect(idled).toEqual(["a"]); + // Timer already fired and pod was untracked; further advancing must not + // fire onIdle again. + vi.advanceTimersByTime(5000); + expect(idled).toEqual(["a"]); + }); + + it("track on an already-tracked pod does not reset an existing countdown", () => { + const idled: string[] = []; + const mon = new IdleMonitor({ idleMs: 1000, onIdle: (id) => idled.push(id) }); + mon.track("a"); + mon.recordActivity("a", 0); + vi.advanceTimersByTime(900); + mon.track("a"); // already tracked: must be a no-op, not a reset + vi.advanceTimersByTime(100); + expect(idled).toEqual(["a"]); + }); + + it("re-arms after onIdle fires and the pod is tracked again", () => { + const idled: string[] = []; + const mon = new IdleMonitor({ idleMs: 1000, onIdle: (id) => idled.push(id) }); + mon.track("a"); + mon.recordActivity("a", 0); + vi.advanceTimersByTime(1000); + expect(idled).toEqual(["a"]); + + mon.track("a"); + mon.recordActivity("a", 0); + vi.advanceTimersByTime(1000); + expect(idled).toEqual(["a", "a"]); + }); +}); diff --git a/packages/fleet/src/index.ts b/packages/fleet/src/index.ts index bd556cf8df..a9c00c8c8a 100644 --- a/packages/fleet/src/index.ts +++ b/packages/fleet/src/index.ts @@ -3,6 +3,7 @@ export const FLEET_PACKAGE = "@supabase/fleet"; export { cloneDir } from "./cowClone.ts"; export type { EdgeProxyEvents, PodUpstream } from "./EdgeProxy.ts"; export { EdgeProxy } from "./EdgeProxy.ts"; +export { IdleMonitor } from "./IdleMonitor.ts"; export type { PodManifest } from "./PodManifest.ts"; export { baseTemplateKey, templateKey } from "./PodManifest.ts"; export { PodRegistry } from "./PodRegistry.ts"; From c22c57c404b1abc4e55b6c59cd3211779fed25d9 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 16:00:24 +0200 Subject: [PATCH 24/38] feat(fleet): fleet facade with wake-on-connect and suspend-on-idle Ties together TemplateStore/PodRegistry/PortRegistry/Provisioner with EdgeProxy/IdleMonitor into createFleet(): pods stay suspended until their first proxied connection wakes an in-process @supabase/stack StackHandle, and idle warm pods are suspended again after idleMs with no open connections. Startup reconciliation kills stale run.pid processes from a previous daemon and re-seeds PortRegistry from each pod's manifest via restore(). Also exports PRELOAD_REQUIRED_EXTENSIONS from @supabase/stack's index so the suspended-pod enableExtension path can guard non-preload extensions without a restart. --- packages/fleet/src/Fleet.integration.test.ts | 45 +++ packages/fleet/src/Fleet.ts | 296 +++++++++++++++++++ packages/fleet/src/index.ts | 2 + packages/stack/src/index.ts | 1 + 4 files changed, 344 insertions(+) create mode 100644 packages/fleet/src/Fleet.integration.test.ts create mode 100644 packages/fleet/src/Fleet.ts diff --git a/packages/fleet/src/Fleet.integration.test.ts b/packages/fleet/src/Fleet.integration.test.ts new file mode 100644 index 0000000000..d0508e04fb --- /dev/null +++ b/packages/fleet/src/Fleet.integration.test.ts @@ -0,0 +1,45 @@ +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createFleet } from "./Fleet.ts"; + +const PG_VERSION = "17.6.1.143"; + +async function query(dbUrl: string, sql: string): Promise { + // Minimal client via Bun's built-in postgres support; this suite runs under Bun. + const { SQL } = await import("bun"); + const db = new SQL(dbUrl); + const rows = await db.unsafe(sql); + await db.close(); + return JSON.stringify(rows); +} + +describe.skipIf(!process.env.FLEET_PG_TESTS)("Fleet", () => { + it("wake-on-connect, suspend-on-idle, fork", async () => { + const root = await mkdtemp(join(tmpdir(), "fleet-e2e-")); + await using fleet = await createFleet({ root, idleMs: 2000 }); + + const a = await fleet.createPod({ id: "a", versions: { postgres: PG_VERSION } }); + expect(a.state).toBe("suspended"); + + // First connection wakes the pod transparently. + await query(a.dbUrl, "create table t(x int); insert into t values (1)"); + const warm = (await fleet.listPods()).find((p) => p.manifest.id === "a"); + expect(warm?.state).toBe("warm"); + + // Idle out (no connections) -> suspended. + await new Promise((r) => setTimeout(r, 4000)); + const idle = (await fleet.listPods()).find((p) => p.manifest.id === "a"); + expect(idle?.state).toBe("suspended"); + + // Wake again on the SAME dbUrl; data survived suspend. + expect(await query(a.dbUrl, "select x from t")).toContain("1"); + + // Fork inherits data, diverges independently. + const b = await fleet.forkPod("a", "b"); + await query(b.dbUrl, "insert into t values (2)"); + expect(await query(a.dbUrl, "select count(*)::int as n from t")).toContain("1"); + expect(await query(b.dbUrl, "select count(*)::int as n from t")).toContain("2"); + }, 600_000); +}); diff --git a/packages/fleet/src/Fleet.ts b/packages/fleet/src/Fleet.ts new file mode 100644 index 0000000000..bd39af501a --- /dev/null +++ b/packages/fleet/src/Fleet.ts @@ -0,0 +1,296 @@ +import { readFile, rm, writeFile } from "node:fs/promises"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { + createStack, + PRELOAD_REQUIRED_EXTENSIONS, + readPreloadLibraries, + writePreloadLibraries, + type StackHandle, +} from "@supabase/stack"; +import { EdgeProxy, type PodUpstream } from "./EdgeProxy.ts"; +import { IdleMonitor } from "./IdleMonitor.ts"; +import type { PodManifest } from "./PodManifest.ts"; +import { PodRegistry } from "./PodRegistry.ts"; +import { PortRegistry } from "./PortRegistry.ts"; +import { Provisioner, type CreatePodOptions } from "./Provisioner.ts"; +import { TemplateStore } from "./TemplateStore.ts"; + +export interface FleetOptions { + readonly root?: string; + readonly idleMs?: number; +} + +export type PodState = "suspended" | "waking" | "warm" | "suspending"; + +export interface PodStatus { + readonly manifest: PodManifest; + readonly state: PodState; + readonly dbUrl: string; +} + +export interface FleetHandle extends AsyncDisposable { + createPod(opts: CreatePodOptions): Promise; + destroyPod(id: string): Promise; + resetPod(id: string): Promise; + forkPod(sourceId: string, newId: string): Promise; + wake(id: string): Promise; + suspend(id: string): Promise; + enableExtension(id: string, extension: string): Promise; + listPods(): Promise>; + dispose(): Promise; +} + +interface WarmPod { + readonly stack: StackHandle; + readonly internalDbPort: number; +} + +// Matches the supabase CLI local-dev convention (see packages/stack +// services/postgres.ts POSTGRES_PASSWORD default) — a template-built +// postgres data dir only accepts this password. +const DB_PASSWORD = "postgres"; + +// Internal (in-process stack) ports are derived from the externally-visible, +// PortRegistry-owned port by a fixed +10_000 offset so the two ranges never +// collide. PortRegistry hands out ports starting at 55_000, so this scheme +// only stays valid while every allocated external port is < 55_536; beyond +// that the derived internal port would exceed the 16-bit port ceiling +// (65_535). Fine for the pod counts this phase targets; a later phase should +// allocate internal ports dynamically (e.g. port 0) if the fleet needs to +// scale past that. +const INTERNAL_PORT_OFFSET = 10_000; + +/** + * Public facade tying together TemplateStore/PodRegistry/PortRegistry/Provisioner + * (pod lifecycle + storage) with EdgeProxy/IdleMonitor (wake-on-connect, + * suspend-on-idle) to host warm pods as in-process `@supabase/stack` StackHandles. + * + * - Every pod's external `dbPort` is registered on the EdgeProxy the moment the + * pod is known (created, forked, or discovered at startup) and never changes + * again; suspended pods still "answer" that port because the proxy itself + * owns the listener and defers to `wake()` on first connection. + * - A pod's postgres process (and any other lazily-started stack services) + * only exists while the pod is "warm": `wakeUpstream` creates an in-process + * `StackHandle` on demand and tears it down again in `suspend`. + * - Concurrent wakes of the same pod are deduped via `wakesInFlight`, since + * EdgeProxy may invoke `wake()` once per connection. + * - `run.pid` files under each pod's directory record which daemon process + * currently owns a warm pod; startup reconciliation uses them to kill stale + * processes from a previous daemon run before treating every pod as + * suspended. This is a phase-1 kill-then-suspend policy, not adoption: the + * spec's long-term goal is to adopt still-live pods across daemon restarts, + * but that requires reattaching the in-process StackHandle to an externally + * running set of processes, which isn't supported yet. Killing and letting + * the next connection re-wake the pod is acceptable because pod data is + * disposable and wake is fast. + */ +export async function createFleet(opts: FleetOptions = {}): Promise { + const root = opts.root ?? join(homedir(), ".supabase"); + const idleMs = opts.idleMs ?? 5 * 60_000; + + const templates = new TemplateStore(join(root, "templates")); + const pods = new PodRegistry(join(root, "pods")); + const ports = await PortRegistry.load(join(root, "fleet-state.json")); + const provisioner = new Provisioner({ templates, pods, ports }); + + const states = new Map(); + const warm = new Map(); + const wakesInFlight = new Map>(); + + const monitor = new IdleMonitor({ + idleMs, + onIdle: (podId) => { + void suspend(podId).catch(() => {}); + }, + }); + + const proxy = new EdgeProxy({ + onActivity: (podId, _event, openConnections) => { + monitor.recordActivity(podId, openConnections); + }, + }); + + const dbUrl = (manifest: PodManifest): string => + `postgresql://postgres:${DB_PASSWORD}@127.0.0.1:${manifest.ports.dbPort}/postgres`; + + const runPidFile = (id: string): string => join(pods.podDir(id), "run.pid"); + + async function wakeUpstream(id: string): Promise { + const existing = warm.get(id); + if (existing) return { host: "127.0.0.1", port: existing.internalDbPort }; + const inFlight = wakesInFlight.get(id); + if (inFlight) return inFlight; + const p = (async (): Promise => { + const manifest = await pods.read(id); + if (manifest === undefined) throw new Error(`unknown pod: ${id}`); + states.set(id, "waking"); + const internalDbPort = manifest.ports.dbPort + INTERNAL_PORT_OFFSET; + const stack = await createStack({ + stackRoot: join(pods.podDir(id), "stack"), + port: manifest.ports.apiPort + INTERNAL_PORT_OFFSET, + lazyServices: true, + postgres: { + dataDir: pods.dataDir(id), + version: manifest.versions.postgres, + port: internalDbPort, + provisioned: true, + profile: "micro", + }, + postgrest: manifest.services.postgrest === true ? {} : false, + auth: manifest.services.auth === true ? {} : false, + realtime: manifest.services.realtime === true ? {} : false, + edgeRuntime: manifest.services["edge-runtime"] === true ? {} : false, + storage: manifest.services.storage === true ? {} : false, + imgproxy: manifest.services.imgproxy === true ? {} : false, + mailpit: manifest.services.mailpit === true ? {} : false, + pgmeta: manifest.services.pgmeta === true ? {} : false, + studio: manifest.services.studio === true ? {} : false, + analytics: manifest.services.analytics === true ? {} : false, + vector: manifest.services.vector === true ? {} : false, + pooler: manifest.services.pooler === true ? {} : false, + functions: false, + }); + try { + await stack.start(); + await stack.serviceReady("postgres"); + } catch (err) { + await stack.dispose().catch(() => {}); + throw err; + } + warm.set(id, { stack, internalDbPort }); + states.set(id, "warm"); + monitor.track(id); + monitor.recordActivity(id, proxy.openConnections(id)); + await writeFile(runPidFile(id), String(process.pid)); + return { host: "127.0.0.1", port: internalDbPort }; + })().catch((err: unknown) => { + states.set(id, "suspended"); + throw err; + }); + const tracked = p.finally(() => { + wakesInFlight.delete(id); + }); + wakesInFlight.set(id, tracked); + return tracked; + } + + async function registerEdge(manifest: PodManifest): Promise { + states.set(manifest.id, "suspended"); + await proxy.register(manifest.id, manifest.ports.dbPort, () => wakeUpstream(manifest.id)); + } + + async function suspend(id: string): Promise { + const pod = warm.get(id); + if (!pod) return; + states.set(id, "suspending"); + monitor.untrack(id); + warm.delete(id); + await pod.stack.dispose(); + await rm(runPidFile(id), { force: true }); + states.set(id, "suspended"); + } + + async function status(manifest: PodManifest): Promise { + return { + manifest, + state: states.get(manifest.id) ?? "suspended", + dbUrl: dbUrl(manifest), + }; + } + + // Startup reconciliation: phase 1 policy is kill-then-suspend, not adoption + // (see class doc above). Any pod with a run.pid file left over from a + // previous daemon has its process terminated before we treat it as + // suspended; the pod's ports are also re-seeded into the freshly loaded + // PortRegistry from its manifest, which is the mechanism `restore()` exists + // for (recovering from a quarantined/corrupt port-state file). + for (const manifest of await pods.list()) { + await ports.restore(manifest.id, manifest.ports); + + const pidRaw = await readFile(runPidFile(manifest.id), "utf8").catch(() => undefined); + if (pidRaw !== undefined) { + const pid = Number(pidRaw); + if (Number.isFinite(pid) && pid > 0 && pid !== process.pid) { + try { + process.kill(-pid, "SIGTERM"); + } catch { + /* not a process-group leader, or already gone */ + } + try { + process.kill(pid, "SIGTERM"); + } catch { + /* already gone */ + } + setTimeout(() => { + try { + process.kill(-pid, "SIGKILL"); + } catch { + /* already gone */ + } + try { + process.kill(pid, "SIGKILL"); + } catch { + /* already gone */ + } + }, 5000).unref(); + } + await rm(runPidFile(manifest.id), { force: true }); + } + await registerEdge(manifest); + } + + const handle: FleetHandle = { + async createPod(opts) { + const manifest = await provisioner.create(opts); + await registerEdge(manifest); + return status(manifest); + }, + async destroyPod(id) { + await suspend(id); + await proxy.unregister(id); + states.delete(id); + await provisioner.destroy(id); + }, + async resetPod(id) { + await suspend(id); + await provisioner.reset(id); + }, + async forkPod(sourceId, newId) { + await suspend(sourceId); + const manifest = await provisioner.fork(sourceId, newId); + await registerEdge(manifest); + return status(manifest); + }, + async wake(id) { + await wakeUpstream(id); + }, + suspend, + async enableExtension(id, extension) { + const pod = warm.get(id); + if (pod) { + await pod.stack.enableExtension(extension); + return; + } + const manifest = await pods.read(id); + if (manifest === undefined) throw new Error(`unknown pod: ${id}`); + if (!PRELOAD_REQUIRED_EXTENSIONS.has(extension)) return; + const libs = await readPreloadLibraries(pods.dataDir(id)); + if (!libs.includes(extension)) { + await writePreloadLibraries(pods.dataDir(id), [...libs, extension]); + } + }, + async listPods() { + const manifests = await pods.list(); + return Promise.all(manifests.map((m) => status(m))); + }, + async dispose() { + for (const id of warm.keys()) await suspend(id); + await proxy.close(); + }, + async [Symbol.asyncDispose]() { + await handle.dispose(); + }, + }; + return handle; +} diff --git a/packages/fleet/src/index.ts b/packages/fleet/src/index.ts index a9c00c8c8a..9c056b0462 100644 --- a/packages/fleet/src/index.ts +++ b/packages/fleet/src/index.ts @@ -3,6 +3,8 @@ export const FLEET_PACKAGE = "@supabase/fleet"; export { cloneDir } from "./cowClone.ts"; export type { EdgeProxyEvents, PodUpstream } from "./EdgeProxy.ts"; export { EdgeProxy } from "./EdgeProxy.ts"; +export type { FleetHandle, FleetOptions, PodState, PodStatus } from "./Fleet.ts"; +export { createFleet } from "./Fleet.ts"; export { IdleMonitor } from "./IdleMonitor.ts"; export type { PodManifest } from "./PodManifest.ts"; export { baseTemplateKey, templateKey } from "./PodManifest.ts"; diff --git a/packages/stack/src/index.ts b/packages/stack/src/index.ts index 59750abf23..98aa5e4d09 100644 --- a/packages/stack/src/index.ts +++ b/packages/stack/src/index.ts @@ -27,3 +27,4 @@ export type { ReadyOptions, StackHandle } from "./createStack.ts"; export type { FunctionsConfig, FunctionsRuntimeConfig } from "./functions.ts"; export { UnixHttpClient, UnixHttpClientError } from "./UnixHttpClient.ts"; export { installMicroProfile, readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; +export { PRELOAD_REQUIRED_EXTENSIONS } from "./micro.ts"; From 2d5e012d083257496f2831559df633a3400a8e6f Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 16:14:04 +0200 Subject: [PATCH 25/38] fix(fleet): serialize per-pod lifecycle ops and reap by postmaster.pid Two race/leak fixes surfaced in review of the Fleet facade: - suspend() deleted from `warm` before awaiting stack.dispose(), and wakeUpstream()/destroyPod/resetPod/forkPod had no way to see an in-flight wake (tracked only in wakesInFlight, not yet `warm`) before mutating the same pod's data dir. Added a small per-pod async lock (PodLock, src/podLock.ts) and routed the wake body, suspend's full body, and the lifecycle-affecting sections of destroyPod/resetPod/ forkPod through it, so these can never interleave against the same pod's process/data dir. Wake dedup via wakesInFlight stays outside the lock so concurrent connections still share one wake. - Startup reconciliation killed `-run.pid` (the daemon's own pid), but process-compose/createStack spawn postgres `detached: true` in its own process group, so that kill missed stale postmasters entirely. Reap now keys off postgres's own ground truth, `/postmaster.pid` (src/reapStalePostmaster.ts): its first line is the postmaster pid, which is always its own process-group leader, so `-pid` reliably reaches the whole tree. run.pid is kept as a "was running" hint but no longer drives the kill decision. Also documented the enableExtension asymmetry: a suspended pod can only durably record intent for preload-required extensions; a non-preload extension request against a suspended pod is a silent no-op until the pod is woken. Adds a gated Fleet integration stress test interleaving suspend/wake/ query calls, plus unit tests for both new modules. Co-Authored-By: Claude Fable 5 --- packages/fleet/src/Fleet.integration.test.ts | 38 +++ packages/fleet/src/Fleet.ts | 236 +++++++++++------- packages/fleet/src/podLock.ts | 57 +++++ packages/fleet/src/podLock.unit.test.ts | 85 +++++++ packages/fleet/src/reapStalePostmaster.ts | 61 +++++ .../src/reapStalePostmaster.unit.test.ts | 81 ++++++ 6 files changed, 462 insertions(+), 96 deletions(-) create mode 100644 packages/fleet/src/podLock.ts create mode 100644 packages/fleet/src/podLock.unit.test.ts create mode 100644 packages/fleet/src/reapStalePostmaster.ts create mode 100644 packages/fleet/src/reapStalePostmaster.unit.test.ts diff --git a/packages/fleet/src/Fleet.integration.test.ts b/packages/fleet/src/Fleet.integration.test.ts index d0508e04fb..16030f8f00 100644 --- a/packages/fleet/src/Fleet.integration.test.ts +++ b/packages/fleet/src/Fleet.integration.test.ts @@ -42,4 +42,42 @@ describe.skipIf(!process.env.FLEET_PG_TESTS)("Fleet", () => { expect(await query(a.dbUrl, "select count(*)::int as n from t")).toContain("1"); expect(await query(b.dbUrl, "select count(*)::int as n from t")).toContain("2"); }, 600_000); + + it("serializes concurrent suspend/wake without corrupting the pod", async () => { + const root = await mkdtemp(join(tmpdir(), "fleet-e2e-")); + await using fleet = await createFleet({ root, idleMs: 60_000 }); + + const a = await fleet.createPod({ id: "a", versions: { postgres: PG_VERSION } }); + await query(a.dbUrl, "create table t(x int); insert into t values (1)"); + + // Wake explicitly, then hammer it with interleaved suspend calls and + // queries racing against each other. Queries may transiently fail as + // connections drop out from under them mid-suspend — that's expected + // and NOT asserted against — but the pod must never crash the fleet + // process, and per-pod lifecycle ops must never interleave badly enough + // to corrupt the data dir or leave the pod stuck in a bad state. + await fleet.wake("a"); + + const attempts = await Promise.allSettled([ + fleet.suspend("a"), + query(a.dbUrl, "select x from t"), + fleet.suspend("a"), + query(a.dbUrl, "select x from t"), + fleet.wake("a"), + fleet.suspend("a"), + query(a.dbUrl, "select x from t"), + ]); + // No assertions on individual outcomes: the point is none of this threw + // an unhandled exception past allSettled (i.e. the fleet process didn't + // crash) and that whatever state we land in is still usable below. + expect(attempts.length).toBe(7); + + // Whatever transient state the interleaving left the pod in, a final + // query must succeed and see the original data intact (wake-on-connect + // recovers a suspended pod; a warm pod just answers directly). + expect(await query(a.dbUrl, "select x from t")).toContain("1"); + + const final = (await fleet.listPods()).find((p) => p.manifest.id === "a"); + expect(["warm", "suspended"]).toContain(final?.state); + }, 600_000); }); diff --git a/packages/fleet/src/Fleet.ts b/packages/fleet/src/Fleet.ts index bd39af501a..e3119f2685 100644 --- a/packages/fleet/src/Fleet.ts +++ b/packages/fleet/src/Fleet.ts @@ -1,4 +1,4 @@ -import { readFile, rm, writeFile } from "node:fs/promises"; +import { rm, writeFile } from "node:fs/promises"; import { homedir } from "node:os"; import { join } from "node:path"; import { @@ -11,9 +11,11 @@ import { import { EdgeProxy, type PodUpstream } from "./EdgeProxy.ts"; import { IdleMonitor } from "./IdleMonitor.ts"; import type { PodManifest } from "./PodManifest.ts"; +import { PodLock } from "./podLock.ts"; import { PodRegistry } from "./PodRegistry.ts"; import { PortRegistry } from "./PortRegistry.ts"; import { Provisioner, type CreatePodOptions } from "./Provisioner.ts"; +import { reapStalePostmaster } from "./reapStalePostmaster.ts"; import { TemplateStore } from "./TemplateStore.ts"; export interface FleetOptions { @@ -75,15 +77,33 @@ const INTERNAL_PORT_OFFSET = 10_000; * `StackHandle` on demand and tears it down again in `suspend`. * - Concurrent wakes of the same pod are deduped via `wakesInFlight`, since * EdgeProxy may invoke `wake()` once per connection. + * - Every operation that touches a pod's live processes or on-disk data dir + * (the body of a wake, the body of `suspend`, and the process/data-dir + * affecting parts of `destroyPod`/`resetPod`/`forkPod`) runs inside + * `podLocks.withLock(id, ...)`, a per-pod FIFO chain (see `podLock.ts`). + * This prevents e.g. a wake racing an in-flight suspend from calling + * `createStack` against a data dir whose postmaster is still shutting + * down, or `destroyPod`/`resetPod` deleting a data dir out from under a + * wake that's still in `wakesInFlight` (and thus not yet visible in + * `warm`). Wake dedup via `wakesInFlight` deliberately stays OUTSIDE the + * lock so concurrent connections still share a single wake; only the + * shared wake body itself acquires the lock. * - `run.pid` files under each pod's directory record which daemon process - * currently owns a warm pod; startup reconciliation uses them to kill stale - * processes from a previous daemon run before treating every pod as - * suspended. This is a phase-1 kill-then-suspend policy, not adoption: the - * spec's long-term goal is to adopt still-live pods across daemon restarts, - * but that requires reattaching the in-process StackHandle to an externally - * running set of processes, which isn't supported yet. Killing and letting - * the next connection re-wake the pod is acceptable because pod data is - * disposable and wake is fast. + * currently owns a warm pod; startup reconciliation uses them as a hint + * that the pod *may* have been running under a previous daemon. The + * actual kill decision, however, keys off postgres's own + * `/postmaster.pid` (see `reapStalePostmaster.ts`) since + * process-compose/`createStack` spawn postgres `detached: true` — its own + * process group, not the daemon's — so killing `-daemonPid` would miss it + * entirely. Phase 1 only ever runs postgres under a fleet pod (no HTTP + * edge yet), so reaping the postmaster's process group covers every + * process a stale pod could have left behind. This is a phase-1 + * kill-then-suspend policy, not adoption: the spec's long-term goal is to + * adopt still-live pods across daemon restarts, but that requires + * reattaching the in-process StackHandle to an externally running set of + * processes, which isn't supported yet. Killing and letting the next + * connection re-wake the pod is acceptable because pod data is disposable + * and wake is fast. */ export async function createFleet(opts: FleetOptions = {}): Promise { const root = opts.root ?? join(homedir(), ".supabase"); @@ -97,6 +117,7 @@ export async function createFleet(opts: FleetOptions = {}): Promise const states = new Map(); const warm = new Map(); const wakesInFlight = new Map>(); + const podLocks = new PodLock(); const monitor = new IdleMonitor({ idleMs, @@ -119,55 +140,60 @@ export async function createFleet(opts: FleetOptions = {}): Promise async function wakeUpstream(id: string): Promise { const existing = warm.get(id); if (existing) return { host: "127.0.0.1", port: existing.internalDbPort }; + // Dedup deliberately stays OUTSIDE podLocks: EdgeProxy may invoke wake() + // once per connection, and every such caller should share the SAME wake + // (and thus the same lock acquisition), not each queue up its own. const inFlight = wakesInFlight.get(id); if (inFlight) return inFlight; - const p = (async (): Promise => { - const manifest = await pods.read(id); - if (manifest === undefined) throw new Error(`unknown pod: ${id}`); - states.set(id, "waking"); - const internalDbPort = manifest.ports.dbPort + INTERNAL_PORT_OFFSET; - const stack = await createStack({ - stackRoot: join(pods.podDir(id), "stack"), - port: manifest.ports.apiPort + INTERNAL_PORT_OFFSET, - lazyServices: true, - postgres: { - dataDir: pods.dataDir(id), - version: manifest.versions.postgres, - port: internalDbPort, - provisioned: true, - profile: "micro", - }, - postgrest: manifest.services.postgrest === true ? {} : false, - auth: manifest.services.auth === true ? {} : false, - realtime: manifest.services.realtime === true ? {} : false, - edgeRuntime: manifest.services["edge-runtime"] === true ? {} : false, - storage: manifest.services.storage === true ? {} : false, - imgproxy: manifest.services.imgproxy === true ? {} : false, - mailpit: manifest.services.mailpit === true ? {} : false, - pgmeta: manifest.services.pgmeta === true ? {} : false, - studio: manifest.services.studio === true ? {} : false, - analytics: manifest.services.analytics === true ? {} : false, - vector: manifest.services.vector === true ? {} : false, - pooler: manifest.services.pooler === true ? {} : false, - functions: false, - }); - try { - await stack.start(); - await stack.serviceReady("postgres"); - } catch (err) { - await stack.dispose().catch(() => {}); + const p = podLocks + .withLock(id, async (): Promise => { + const manifest = await pods.read(id); + if (manifest === undefined) throw new Error(`unknown pod: ${id}`); + states.set(id, "waking"); + const internalDbPort = manifest.ports.dbPort + INTERNAL_PORT_OFFSET; + const stack = await createStack({ + stackRoot: join(pods.podDir(id), "stack"), + port: manifest.ports.apiPort + INTERNAL_PORT_OFFSET, + lazyServices: true, + postgres: { + dataDir: pods.dataDir(id), + version: manifest.versions.postgres, + port: internalDbPort, + provisioned: true, + profile: "micro", + }, + postgrest: manifest.services.postgrest === true ? {} : false, + auth: manifest.services.auth === true ? {} : false, + realtime: manifest.services.realtime === true ? {} : false, + edgeRuntime: manifest.services["edge-runtime"] === true ? {} : false, + storage: manifest.services.storage === true ? {} : false, + imgproxy: manifest.services.imgproxy === true ? {} : false, + mailpit: manifest.services.mailpit === true ? {} : false, + pgmeta: manifest.services.pgmeta === true ? {} : false, + studio: manifest.services.studio === true ? {} : false, + analytics: manifest.services.analytics === true ? {} : false, + vector: manifest.services.vector === true ? {} : false, + pooler: manifest.services.pooler === true ? {} : false, + functions: false, + }); + try { + await stack.start(); + await stack.serviceReady("postgres"); + } catch (err) { + await stack.dispose().catch(() => {}); + throw err; + } + warm.set(id, { stack, internalDbPort }); + states.set(id, "warm"); + monitor.track(id); + monitor.recordActivity(id, proxy.openConnections(id)); + await writeFile(runPidFile(id), String(process.pid)); + return { host: "127.0.0.1", port: internalDbPort }; + }) + .catch((err: unknown) => { + states.set(id, "suspended"); throw err; - } - warm.set(id, { stack, internalDbPort }); - states.set(id, "warm"); - monitor.track(id); - monitor.recordActivity(id, proxy.openConnections(id)); - await writeFile(runPidFile(id), String(process.pid)); - return { host: "127.0.0.1", port: internalDbPort }; - })().catch((err: unknown) => { - states.set(id, "suspended"); - throw err; - }); + }); const tracked = p.finally(() => { wakesInFlight.delete(id); }); @@ -181,6 +207,16 @@ export async function createFleet(opts: FleetOptions = {}): Promise } async function suspend(id: string): Promise { + // Full body runs inside the per-pod lock so a suspend can never + // interleave with a concurrent wake, destroy, reset, or fork touching + // the same pod's process/data dir. Note this is a fresh, non-re-entrant + // acquisition: callers that already hold the lock for `id` (forkPod's + // source-pod lock) call the LOCKED body directly instead of this + // function — see `suspendLocked` usage below. + return podLocks.withLock(id, () => suspendLocked(id)); + } + + async function suspendLocked(id: string): Promise { const pod = warm.get(id); if (!pod) return; states.set(id, "suspending"); @@ -200,43 +236,25 @@ export async function createFleet(opts: FleetOptions = {}): Promise } // Startup reconciliation: phase 1 policy is kill-then-suspend, not adoption - // (see class doc above). Any pod with a run.pid file left over from a - // previous daemon has its process terminated before we treat it as - // suspended; the pod's ports are also re-seeded into the freshly loaded - // PortRegistry from its manifest, which is the mechanism `restore()` exists - // for (recovering from a quarantined/corrupt port-state file). + // (see class doc above). `run.pid` (the daemon's own pid) is only a HINT + // that a pod may have been running under a previous daemon; it's not the + // kill target, since process-compose/createStack spawn postgres + // `detached: true` — its own process group, distinct from the daemon's — + // so `kill(-daemonPid, ...)` would miss postgres entirely and leave a + // stale postmaster running forever. The actual kill decision keys off + // postgres's own ground truth, `/postmaster.pid`, whose first + // line is the postmaster's pid; the postmaster is always its own + // process-group leader, so `reapStalePostmaster` can reliably signal the + // whole tree via `-pid`. Phase 1 only ever runs postgres under a fleet pod + // (postgres-only ready gate; no HTTP edge yet), so this fully covers what + // a stale pod could have left behind. The pod's ports are also re-seeded + // into the freshly loaded PortRegistry from its manifest, which is the + // mechanism `restore()` exists for (recovering from a quarantined/corrupt + // port-state file). for (const manifest of await pods.list()) { await ports.restore(manifest.id, manifest.ports); - - const pidRaw = await readFile(runPidFile(manifest.id), "utf8").catch(() => undefined); - if (pidRaw !== undefined) { - const pid = Number(pidRaw); - if (Number.isFinite(pid) && pid > 0 && pid !== process.pid) { - try { - process.kill(-pid, "SIGTERM"); - } catch { - /* not a process-group leader, or already gone */ - } - try { - process.kill(pid, "SIGTERM"); - } catch { - /* already gone */ - } - setTimeout(() => { - try { - process.kill(-pid, "SIGKILL"); - } catch { - /* already gone */ - } - try { - process.kill(pid, "SIGKILL"); - } catch { - /* already gone */ - } - }, 5000).unref(); - } - await rm(runPidFile(manifest.id), { force: true }); - } + await reapStalePostmaster(pods.dataDir(manifest.id)); + await rm(runPidFile(manifest.id), { force: true }); await registerEdge(manifest); } @@ -247,18 +265,32 @@ export async function createFleet(opts: FleetOptions = {}): Promise return status(manifest); }, async destroyPod(id) { - await suspend(id); + // suspend (tears down live processes) and provisioner.destroy (deletes + // the data dir) run as ONE lock acquisition so a wake that's still + // mid-flight (registered in wakesInFlight, not yet in `warm`) can't + // slip in between them and recreate a stack against a dir we're about + // to delete. + await podLocks.withLock(id, async () => { + await suspendLocked(id); + await provisioner.destroy(id); + }); await proxy.unregister(id); states.delete(id); - await provisioner.destroy(id); }, async resetPod(id) { - await suspend(id); - await provisioner.reset(id); + await podLocks.withLock(id, async () => { + await suspendLocked(id); + await provisioner.reset(id); + }); }, async forkPod(sourceId, newId) { - await suspend(sourceId); - const manifest = await provisioner.fork(sourceId, newId); + // Lock the SOURCE pod around suspend + the fork's clone of its data + // dir (provisioner.fork reads sourceId's data dir). The new pod needs + // no lock: it isn't live yet, nothing else can race it. + const manifest = await podLocks.withLock(sourceId, async () => { + await suspendLocked(sourceId); + return provisioner.fork(sourceId, newId); + }); await registerEdge(manifest); return status(manifest); }, @@ -266,6 +298,18 @@ export async function createFleet(opts: FleetOptions = {}): Promise await wakeUpstream(id); }, suspend, + // Preload-only semantics when suspended: if the pod is warm, this + // enables `extension` immediately via the live StackHandle (CREATE + // EXTENSION, possibly after a preload-triggered restart) regardless of + // which extension it is. If the pod is SUSPENDED, there is no live + // postgres to run CREATE EXTENSION against, so this can only durably + // record intent for extensions that need a preload library — those get + // appended to the data dir's preload-libraries config so the next wake + // picks them up. For a suspended pod and a NON-preload extension (i.e. + // not in `PRELOAD_REQUIRED_EXTENSIONS`), this method is a silent no-op: + // nothing is persisted, and the extension is NOT created on the next + // wake. Callers that need a non-preload extension enabled on a + // currently-suspended pod must `wake()` it first. async enableExtension(id, extension) { const pod = warm.get(id); if (pod) { diff --git a/packages/fleet/src/podLock.ts b/packages/fleet/src/podLock.ts new file mode 100644 index 0000000000..6c2017b721 --- /dev/null +++ b/packages/fleet/src/podLock.ts @@ -0,0 +1,57 @@ +/** + * Per-key async operation lock: serializes `withLock` calls that share the + * same `id` onto a single chain, while calls for different ids run fully + * concurrently. + * + * Used by `Fleet` to make sure a pod's lifecycle operations (wake, suspend, + * destroy, reset, fork) never interleave against the same pod's data + * directory / postgres process — e.g. a wake racing an in-flight suspend + * could otherwise `createStack` against a data dir whose postmaster is + * still shutting down. + */ +export class PodLock { + private readonly chains = new Map>(); + + /** Number of ids currently holding a live (unsettled) chain entry. */ + get size(): number { + return this.chains.size; + } + + /** + * Runs `fn` after every previously chained op for `id` has settled + * (resolved OR rejected), and returns `fn`'s result/rejection. + * + * A rejection from `fn` (or from an earlier op in the chain) never + * poisons subsequent calls for the same `id` — the chain always advances + * to a resolved "tail" internally, regardless of whether the caller-visible + * promise for a given link rejects. + */ + async withLock(id: string, fn: () => Promise): Promise { + const prior = this.chains.get(id) ?? Promise.resolve(); + + // The tail settles once `prior` has settled, regardless of outcome, so + // the next `withLock` call for this id always proceeds. + const tail = prior.then( + () => undefined, + () => undefined, + ); + const result = tail.then(fn); + + // Keep the chain alive (swallow rejection here too) so the *next* link + // waits for this one without itself becoming a rejected chain entry. + const nextTail = result.then( + () => undefined, + () => undefined, + ); + this.chains.set(id, nextTail); + + // Once this op's link is the most recent one recorded and it has + // settled, clear the entry so the map doesn't grow unboundedly for pods + // that are no longer being operated on. + void nextTail.then(() => { + if (this.chains.get(id) === nextTail) this.chains.delete(id); + }); + + return result; + } +} diff --git a/packages/fleet/src/podLock.unit.test.ts b/packages/fleet/src/podLock.unit.test.ts new file mode 100644 index 0000000000..f0df27ce65 --- /dev/null +++ b/packages/fleet/src/podLock.unit.test.ts @@ -0,0 +1,85 @@ +import { describe, expect, it } from "vitest"; +import { PodLock } from "./podLock.ts"; + +describe("PodLock", () => { + it("serializes operations for the same id in call order", async () => { + const lock = new PodLock(); + const order: string[] = []; + + const first = lock.withLock("a", async () => { + order.push("first-start"); + await new Promise((r) => setTimeout(r, 20)); + order.push("first-end"); + return 1; + }); + const second = lock.withLock("a", async () => { + order.push("second-start"); + await new Promise((r) => setTimeout(r, 5)); + order.push("second-end"); + return 2; + }); + + expect(await first).toBe(1); + expect(await second).toBe(2); + expect(order).toEqual(["first-start", "first-end", "second-start", "second-end"]); + }); + + it("does not let a rejected op poison the chain for subsequent ops", async () => { + const lock = new PodLock(); + + await expect( + lock.withLock("a", async () => { + throw new Error("boom"); + }), + ).rejects.toThrow("boom"); + + // A later op on the same id must still run (and run after the failure). + const order: string[] = []; + await lock.withLock("a", async () => { + order.push("ran"); + }); + expect(order).toEqual(["ran"]); + }); + + it("does not serialize operations across different ids", async () => { + const lock = new PodLock(); + const order: string[] = []; + + const a = lock.withLock("a", async () => { + order.push("a-start"); + await new Promise((r) => setTimeout(r, 30)); + order.push("a-end"); + }); + const b = lock.withLock("b", async () => { + order.push("b-start"); + await new Promise((r) => setTimeout(r, 5)); + order.push("b-end"); + }); + + await Promise.all([a, b]); + // b, on a different id, should complete before a (which sleeps longer), + // proving the two chains ran concurrently rather than serialized. + expect(order.indexOf("b-end")).toBeLessThan(order.indexOf("a-end")); + expect(order).toContain("a-start"); + expect(order).toContain("b-start"); + }); + + it("clears the internal chain entry once settled (no unbounded growth)", async () => { + const lock = new PodLock(); + await lock.withLock("a", async () => {}); + expect(lock.size).toBe(0); + }); + + it("preserves order across many interleaved ops on the same id", async () => { + const lock = new PodLock(); + const results: number[] = []; + const ops = Array.from({ length: 10 }, (_, i) => + lock.withLock("a", async () => { + await new Promise((r) => setTimeout(r, (10 - i) % 3)); + results.push(i); + }), + ); + await Promise.all(ops); + expect(results).toEqual(Array.from({ length: 10 }, (_, i) => i)); + }); +}); diff --git a/packages/fleet/src/reapStalePostmaster.ts b/packages/fleet/src/reapStalePostmaster.ts new file mode 100644 index 0000000000..0ca4f8e04d --- /dev/null +++ b/packages/fleet/src/reapStalePostmaster.ts @@ -0,0 +1,61 @@ +import { readFile } from "node:fs/promises"; +import { join } from "node:path"; + +/** + * Reaps a stale postmaster (and its backends) left running under `dataDir` + * by a previous, no-longer-live daemon process. + * + * Ground truth is postgres's own `/postmaster.pid`: its FIRST LINE + * is the postmaster's pid. Unlike our `run.pid` marker (which records the + * *daemon's* pid, not postgres's), the postmaster is always its own + * process-group leader — its pgid equals its pid, and every backend it forks + * shares that pgid. So `kill(-pid, ...)` reliably reaches the whole + * postgres process tree for this data dir, regardless of what spawned it or + * how (in phase 1, `@supabase/stack`'s `createStack`, which — like + * process-compose's `detached: true` children — does not run postgres as a + * child of the fleet daemon's own process group). + * + * Phase 1 only ever runs postgres under a fleet pod (postgres-only ready + * gate; no HTTP edge / other services yet), so postmaster.pid alone is a + * complete picture of "is anything from this pod still alive" — no need to + * separately reap other service processes. + */ +export async function reapStalePostmaster(dataDir: string): Promise { + const raw = await readFile(join(dataDir, "postmaster.pid"), "utf8").catch(() => undefined); + if (raw === undefined) return; + + const firstLine = raw.split("\n")[0]?.trim(); + const pid = Number(firstLine); + if (!Number.isFinite(pid) || pid <= 0 || pid === process.pid) return; + + let alive = false; + try { + process.kill(pid, 0); + alive = true; + } catch { + alive = false; + } + if (!alive) return; + + try { + process.kill(-pid, "SIGTERM"); + } catch { + try { + process.kill(pid, "SIGTERM"); + } catch { + /* already gone */ + } + } + + setTimeout(() => { + try { + process.kill(-pid, "SIGKILL"); + } catch { + try { + process.kill(pid, "SIGKILL"); + } catch { + /* already gone */ + } + } + }, 5000).unref(); +} diff --git a/packages/fleet/src/reapStalePostmaster.unit.test.ts b/packages/fleet/src/reapStalePostmaster.unit.test.ts new file mode 100644 index 0000000000..c48957c71a --- /dev/null +++ b/packages/fleet/src/reapStalePostmaster.unit.test.ts @@ -0,0 +1,81 @@ +import { spawn } from "node:child_process"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { reapStalePostmaster } from "./reapStalePostmaster.ts"; + +function isAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} + +async function waitUntil(predicate: () => boolean, timeoutMs = 5000): Promise { + const start = Date.now(); + while (!predicate()) { + if (Date.now() - start > timeoutMs) throw new Error("timed out waiting for condition"); + await new Promise((r) => setTimeout(r, 50)); + } +} + +describe("reapStalePostmaster", () => { + const dirs: string[] = []; + const pidsToCleanUp: number[] = []; + + afterEach(async () => { + for (const pid of pidsToCleanUp.splice(0)) { + try { + process.kill(pid, "SIGKILL"); + } catch { + /* already gone */ + } + } + for (const dir of dirs.splice(0)) { + await rm(dir, { recursive: true, force: true }); + } + }); + + it("kills a live detached process group referenced by postmaster.pid", async () => { + const dir = await mkdtemp(join(tmpdir(), "reap-test-")); + dirs.push(dir); + + const child = spawn("sleep", ["60"], { detached: true, stdio: "ignore" }); + child.unref(); + const pid = child.pid; + if (pid === undefined) throw new Error("failed to spawn dummy process"); + pidsToCleanUp.push(pid); + + await writeFile(join(dir, "postmaster.pid"), `${pid}\n/some/data/dir\n1234567\n5432\n`); + + expect(isAlive(pid)).toBe(true); + await reapStalePostmaster(dir); + await waitUntil(() => !isAlive(pid)); + expect(isAlive(pid)).toBe(false); + }, 10_000); + + it("is a no-op when postmaster.pid is missing", async () => { + const dir = await mkdtemp(join(tmpdir(), "reap-test-")); + dirs.push(dir); + await expect(reapStalePostmaster(dir)).resolves.toBeUndefined(); + }); + + it("is a no-op when postmaster.pid contains garbage", async () => { + const dir = await mkdtemp(join(tmpdir(), "reap-test-")); + dirs.push(dir); + await writeFile(join(dir, "postmaster.pid"), "not-a-pid\nnoise\n"); + await expect(reapStalePostmaster(dir)).resolves.toBeUndefined(); + }); + + it("refuses to kill when the pid equals the current process pid", async () => { + const dir = await mkdtemp(join(tmpdir(), "reap-test-")); + dirs.push(dir); + await writeFile(join(dir, "postmaster.pid"), `${process.pid}\n/some/data/dir\n`); + await expect(reapStalePostmaster(dir)).resolves.toBeUndefined(); + // Sanity: we're still alive (obviously true, but documents intent). + expect(isAlive(process.pid)).toBe(true); + }); +}); From 51b242bdd411ba730c207d18b7c27cebcae60fde Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 16:38:28 +0200 Subject: [PATCH 26/38] test(fleet): density e2e and package README Adds the density guardrail E2E (register N pods, wake a subset, confirm the rest stay at zero processes, explicit suspend) gated behind FLEET_PG_TESTS with pod count tunable via FLEET_E2E_PODS. Mirrors packages/stack's e2e vitest project (fileParallelism: false) so nx picks up a test:e2e target for @supabase/fleet, and extends the package's knip entry points to cover tests/**/*.ts. Also adds the package README covering the public API, wake/suspend/fork lifecycle, and phase-1 limitations (native-only, kill-then-resuspend reconciliation, HTTP/realtime lazy-start gaps). --- packages/fleet/README.md | 88 +++++++++++++++++++ packages/fleet/package.json | 2 +- packages/fleet/tests/fleetDensity.e2e.test.ts | 39 ++++++++ packages/fleet/vitest.config.ts | 7 ++ 4 files changed, 135 insertions(+), 1 deletion(-) create mode 100644 packages/fleet/README.md create mode 100644 packages/fleet/tests/fleetDensity.e2e.test.ts diff --git a/packages/fleet/README.md b/packages/fleet/README.md new file mode 100644 index 0000000000..f2245f2866 --- /dev/null +++ b/packages/fleet/README.md @@ -0,0 +1,88 @@ +# @supabase/fleet + +Host-level daemon for running many lightweight Supabase pods in parallel: +CoW template provisioning, wake-on-connect, suspend-on-idle, instant fork. + +## Quick start + +```ts +import { createFleet } from "@supabase/fleet"; + +await using fleet = await createFleet(); +const pod = await fleet.createPod({ id: "my-worktree", versions: { postgres: "17.6.1.143" } }); +// pod.dbUrl is live immediately -- the first connection wakes postgres (~200ms). +// After 5 idle minutes (default) the pod suspends to zero processes; the port keeps listening. +const branch = await fleet.forkPod("my-worktree", "my-worktree-experiment"); +``` + +## How it works + +- `createPod` clones a per-Postgres-version template data directory (copy-on-write) and + registers the pod's external `dbPort` on an in-process edge proxy. No Postgres process + exists yet -- the pod starts `"suspended"`. +- The first connection to `dbPort` transparently wakes the pod (`"waking"` -> `"warm"`): the + proxy spins up an in-process `@supabase/stack` handle against the pod's data directory and + forwards the connection once Postgres is ready. +- A warm pod with no open connections for `idleMs` (default 5 minutes) suspends itself back to + zero processes (`"warm"` -> `"suspending"` -> `"suspended"`); the port keeps listening for the + next wake. +- `forkPod` suspends the source pod (if warm) and CoW-clones its data directory into a new pod + with its own external port, so the two diverge independently from that point on. +- `wake` / `suspend` let a caller force the transition explicitly instead of waiting on + connection activity or the idle timer. + +## API + +```ts +interface FleetOptions { + readonly root?: string; // defaults to ~/.supabase + readonly idleMs?: number; // defaults to 5 minutes +} + +interface FleetHandle extends AsyncDisposable { + createPod(opts: CreatePodOptions): Promise; + destroyPod(id: string): Promise; + resetPod(id: string): Promise; + forkPod(sourceId: string, newId: string): Promise; + wake(id: string): Promise; + suspend(id: string): Promise; + enableExtension(id: string, extension: string): Promise; + listPods(): Promise>; + dispose(): Promise; +} +``` + +`PodStatus.state` is one of `"suspended" | "waking" | "warm" | "suspending"`. `PodStatus.dbUrl` is +stable across suspend/wake cycles -- the external port never changes for the life of the pod. + +## Phase 1 limitations + +- **Native macOS/Linux only.** No Docker-mode fallback for fleet pods yet. +- **Kill-then-resuspend reconciliation, not adoption.** If the daemon restarts while pods are + warm, it does not reattach to their running processes. Instead it reaps each pod's stale + `postmaster.pid` (killing the leftover Postgres process group) and leaves the pod suspended; + the next connection re-wakes it normally. Pod data is disposable, so this is safe, just not + zero-downtime. +- **HTTP gateway / additional services are not fleet-wired yet.** Wake-on-connect only listens + on the Postgres port. Other stack services (PostgREST, Auth, Realtime, etc.) can be requested + per-pod via `services` and lazily started inside an already-warm pod, but there's no edge + proxy in front of the API port the way there is for `dbPort` -- so only a warm pod's HTTP + services are reachable, and connecting to them does not itself wake a suspended pod. +- **Realtime's WebSocket lazy-start is not covered.** Even inside a warm pod, Realtime's + lazy-start behavior over a persistent WebSocket connection is a known gap versus the + request/response lazy-start story for other HTTP services. + +## Design + +See [`docs/specs/2026-07-07-micro-supabase-stacks-design.md`](../../docs/specs/2026-07-07-micro-supabase-stacks-design.md) for the full design. + +## Deferred to later phases + +- Binary/Docker image optimizations. +- CLI wiring of `supabase start`/`stop` onto `createFleet`. +- HTTP gateway host-based routing across pods, plus edge wake for the API port. +- PSS/CPU benchmark harness with budget assertions in CI. +- True pod adoption across fleet-daemon restarts. +- Warm-template LRU garbage collection. +- `supautils` profile flag enforcement (the manifest field exists; config plumbing comes later). +- Compatibility suite (`CREATE EXTENSION` sweep, dump/restore round-trip). diff --git a/packages/fleet/package.json b/packages/fleet/package.json index 2e31dcb79d..0caa355f84 100644 --- a/packages/fleet/package.json +++ b/packages/fleet/package.json @@ -26,7 +26,7 @@ "vitest": "catalog:" }, "knip": { - "entry": ["src/**/*.test.ts"], + "entry": ["src/**/*.test.ts", "tests/**/*.ts"], "ignoreDependencies": ["@typescript/native-preview", "oxfmt", "oxlint", "oxlint-tsgolint"], "ignoreBinaries": ["nx"] } diff --git a/packages/fleet/tests/fleetDensity.e2e.test.ts b/packages/fleet/tests/fleetDensity.e2e.test.ts new file mode 100644 index 0000000000..e817b6999c --- /dev/null +++ b/packages/fleet/tests/fleetDensity.e2e.test.ts @@ -0,0 +1,39 @@ +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createFleet } from "../src/Fleet.ts"; + +const PG_VERSION = "17.6.1.143"; +const REGISTERED = Number(process.env.FLEET_E2E_PODS ?? 20); // 100+ locally, 20 in CI +const WARM = 3; + +describe.skipIf(!process.env.FLEET_PG_TESTS)("fleet density", () => { + it(`registers ${REGISTERED} pods, wakes ${WARM}, suspends cleanly`, async () => { + const root = await mkdtemp(join(tmpdir(), "fleet-density-")); + await using fleet = await createFleet({ root, idleMs: 60_000 }); + + // Registration is cheap: template built once, then CoW clones. + for (let i = 0; i < REGISTERED; i += 1) { + await fleet.createPod({ id: `pod-${i}`, versions: { postgres: PG_VERSION } }); + } + const all = await fleet.listPods(); + expect(all).toHaveLength(REGISTERED); + expect(all.every((p) => p.state === "suspended")).toBe(true); + + // Distinct external ports across the whole fleet. + const portSet = new Set(all.map((p) => p.manifest.ports.dbPort)); + expect(portSet.size).toBe(REGISTERED); + + // Wake a subset; the rest stay suspended (zero processes). + for (let i = 0; i < WARM; i += 1) await fleet.wake(`pod-${i}`); + const after = await fleet.listPods(); + expect(after.filter((p) => p.state === "warm")).toHaveLength(WARM); + expect(after.filter((p) => p.state === "suspended")).toHaveLength(REGISTERED - WARM); + + // Explicit suspend brings a pod back to zero. + await fleet.suspend("pod-0"); + const final = await fleet.listPods(); + expect(final.find((p) => p.manifest.id === "pod-0")?.state).toBe("suspended"); + }, 900_000); +}); diff --git a/packages/fleet/vitest.config.ts b/packages/fleet/vitest.config.ts index e10a870951..6d1fd270fa 100644 --- a/packages/fleet/vitest.config.ts +++ b/packages/fleet/vitest.config.ts @@ -24,6 +24,13 @@ export default defineConfig({ include: ["**/*.integration.test.ts"], }, }, + { + test: { + name: "e2e", + include: ["**/*.e2e.test.ts"], + fileParallelism: false, + }, + }, ], }, }); From f561ded1ab7fa136077cd9e5786b6432b11bdc91 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 17:20:53 +0200 Subject: [PATCH 27/38] fix(stack,fleet,cli): address final-review findings for stacks spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add enableExtension stub to Stack mocks (mocks.ts, running-stack.ts) to match the Stack service contract added in this branch. - Fix StackLifecycleCoordinator.unit.test.ts flakiness under Bun: widen the projected-status assertion to tolerate the transient "Restarting" state, and bind the fake healthy postgrest server on an OS-assigned port (listen(0)) instead of a hard-coded port to avoid EADDRINUSE under parallel test runs. - Remove unused effect and @supabase/process-compose dependencies from packages/fleet/package.json (zero imports), reformat with oxfmt, and move @tsconfig/bun into knip's ignoreDependencies (false positive — used only via tsconfig extends). - Sync pnpm-lock.yaml after the dependency removal. --- apps/cli/tests/helpers/mocks.ts | 1 + apps/cli/tests/helpers/running-stack.ts | 4 + packages/fleet/package.json | 21 ++- .../StackLifecycleCoordinator.unit.test.ts | 150 ++++++++++-------- pnpm-lock.yaml | 6 - 5 files changed, 102 insertions(+), 80 deletions(-) diff --git a/apps/cli/tests/helpers/mocks.ts b/apps/cli/tests/helpers/mocks.ts index d9e1aca763..5ce114e052 100644 --- a/apps/cli/tests/helpers/mocks.ts +++ b/apps/cli/tests/helpers/mocks.ts @@ -677,6 +677,7 @@ export function mockStack( startService: () => Effect.void, stopService: () => Effect.void, restartService: () => Effect.void, + enableExtension: () => Effect.void, reloadFunctions: () => Effect.void, reloadEdgeRuntime: () => Effect.void, getState: () => diff --git a/apps/cli/tests/helpers/running-stack.ts b/apps/cli/tests/helpers/running-stack.ts index 261c512eee..5069dcd13c 100644 --- a/apps/cli/tests/helpers/running-stack.ts +++ b/apps/cli/tests/helpers/running-stack.ts @@ -152,6 +152,10 @@ function makeStackLayer(opts: { opts.states.some((state) => state.name === name) ? Effect.void : Effect.fail(new ServiceNotFoundError({ name })), + enableExtension: (name: string) => + opts.states.some((state) => state.name === name) + ? Effect.void + : Effect.fail(new ServiceNotFoundError({ name })), reloadFunctions: () => opts.states.some((state) => state.name === "edge-runtime") ? Effect.void diff --git a/packages/fleet/package.json b/packages/fleet/package.json index 0caa355f84..2238ec2824 100644 --- a/packages/fleet/package.json +++ b/packages/fleet/package.json @@ -12,9 +12,7 @@ "fix:all": "nx run-many -t lint:fix fmt:fix knip:fix --projects=$npm_package_name" }, "dependencies": { - "@supabase/process-compose": "workspace:*", - "@supabase/stack": "workspace:*", - "effect": "catalog:" + "@supabase/stack": "workspace:*" }, "devDependencies": { "@tsconfig/bun": "catalog:", @@ -26,8 +24,19 @@ "vitest": "catalog:" }, "knip": { - "entry": ["src/**/*.test.ts", "tests/**/*.ts"], - "ignoreDependencies": ["@typescript/native-preview", "oxfmt", "oxlint", "oxlint-tsgolint"], - "ignoreBinaries": ["nx"] + "entry": [ + "src/**/*.test.ts", + "tests/**/*.ts" + ], + "ignoreDependencies": [ + "@typescript/native-preview", + "oxfmt", + "oxlint", + "oxlint-tsgolint", + "@tsconfig/bun" + ], + "ignoreBinaries": [ + "nx" + ] } } diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts index 9c0c986a45..f7b08a0e7a 100644 --- a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -104,7 +104,7 @@ function setupLayer(config: ResolvedStackConfig) { return { layer, spawner }; } -function makeLazyConfig(dataDir: string): ResolvedStackConfig { +function makeLazyConfig(dataDir: string, postgrestPort: number): ResolvedStackConfig { return { ...makeConfig(dataDir), lazyServices: true, @@ -112,7 +112,7 @@ function makeLazyConfig(dataDir: string): ResolvedStackConfig { // below binds a fake listener there so waitReady can actually resolve once startService // spawns it (spawning itself is satisfied generically by the mocked ChildProcessSpawner). postgrest: { - port: defaultPorts.postgrestPort, + port: postgrestPort, adminPort: defaultPorts.postgrestAdminPort, schemas: ["public"], extraSearchPath: ["public"], @@ -122,16 +122,28 @@ function makeLazyConfig(dataDir: string): ResolvedStackConfig { }; } -function startFakeHealthyServer(port: number): Promise<() => Promise> { +interface FakeHealthyServer { + readonly port: number; + readonly stop: () => Promise; +} + +function startFakeHealthyServer(): Promise { return new Promise((resolve, reject) => { const server = http.createServer((_req, res) => { res.writeHead(200); res.end("OK"); }); - server.listen(port, "127.0.0.1", () => { - resolve( - () => new Promise((res, rej) => server.close((err) => (err ? rej(err) : res()))), - ); + server.listen(0, "127.0.0.1", () => { + const addr = server.address(); + if (!addr || typeof addr === "string") { + reject(new Error("Unexpected server address")); + return; + } + resolve({ + port: addr.port, + stop: () => + new Promise((res, rej) => server.close((err) => (err ? rej(err) : res()))), + }); }); server.on("error", reject); }); @@ -181,51 +193,52 @@ describe("StackLifecycleCoordinator lazyServices", () => { // enableExtension test above). it.live("start() only eager-starts postgres/postgres-init; postgrest starts on demand", () => { const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-lazy-test-")); - const config = makeLazyConfig(dataDir); - const { layer, spawner } = setupLayer(config); - - return Effect.gen(function* () { - const stopFakeServer = yield* Effect.promise(() => - startFakeHealthyServer(defaultPorts.postgrestPort), - ); - - const isPostgrestPayload = (s: { args: ReadonlyArray }) => { - const encoded = s.args.at(-1); - if (encoded === undefined) return false; - try { - const decoded = JSON.parse(Buffer.from(encoded, "base64url").toString("utf8")) as { - command?: string; - }; - return decoded.command?.endsWith("/postgrest") ?? false; - } catch { - return false; - } - }; - - const stack = yield* Stack; - yield* stack.start(); - - // postgrest must not have been spawned by start() itself. - expect(spawner.spawned.some(isPostgrestPayload)).toBe(false); - - const postgrestState = yield* stack.getState("postgrest"); - expect(postgrestState.status).toBe("Pending"); - // The ApiProxy's ensureService would call startService + waitReady on first request; - // simulate that here directly against the coordinator. - yield* stack.startService("postgrest"); - yield* stack.waitReady("postgrest"); - - expect(spawner.spawned.some(isPostgrestPayload)).toBe(true); - // waitReady already proved the service reached a ready state (Running or Healthy — health - // checks poll on an interval, so the coordinator's projected state may briefly lag the - // orchestrator's internal readiness signal that waitReady resolves on). - const readyState = yield* stack.getState("postgrest"); - expect(["Running", "Healthy"]).toContain(readyState.status); - - yield* Effect.promise(() => stopFakeServer()); - }).pipe( - Effect.provide(layer), + return Effect.promise(() => startFakeHealthyServer()).pipe( + Effect.flatMap((fakeServer) => { + const config = makeLazyConfig(dataDir, fakeServer.port); + const { layer, spawner } = setupLayer(config); + + const isPostgrestPayload = (s: { args: ReadonlyArray }) => { + const encoded = s.args.at(-1); + if (encoded === undefined) return false; + try { + const decoded = JSON.parse(Buffer.from(encoded, "base64url").toString("utf8")) as { + command?: string; + }; + return decoded.command?.endsWith("/postgrest") ?? false; + } catch { + return false; + } + }; + + return Effect.gen(function* () { + const stack = yield* Stack; + yield* stack.start(); + + // postgrest must not have been spawned by start() itself. + expect(spawner.spawned.some(isPostgrestPayload)).toBe(false); + + const postgrestState = yield* stack.getState("postgrest"); + expect(postgrestState.status).toBe("Pending"); + + // The ApiProxy's ensureService would call startService + waitReady on first request; + // simulate that here directly against the coordinator. + yield* stack.startService("postgrest"); + yield* stack.waitReady("postgrest"); + + expect(spawner.spawned.some(isPostgrestPayload)).toBe(true); + // waitReady already proved the service reached a ready state — that's the actual + // assertion above. The coordinator's projected status here is best-effort: health + // checks poll on an interval, so it may still lag behind (or even show a transient + // "Restarting", if a flaky health check briefly failed and triggered a restart) + // relative to the orchestrator's internal readiness signal that waitReady resolves on. + const readyState = yield* stack.getState("postgrest"); + expect(["Running", "Healthy", "Restarting"]).toContain(readyState.status); + + yield* Effect.promise(() => fakeServer.stop()); + }).pipe(Effect.provide(layer)); + }), Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), ); }); @@ -239,7 +252,9 @@ describe("StackLifecycleCoordinator lazyServices", () => { // timeout) rather than hanging the suite. it.live("waitAllReady() resolves promptly when only postgres was eager-started", () => { const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-lazy-ready-test-")); - const config = makeLazyConfig(dataDir); + // postgrest is never started in this test, so nothing binds to its configured port; any + // fixed port is safe here (no EADDRINUSE risk). + const config = makeLazyConfig(dataDir, defaultPorts.postgrestPort); const { layer } = setupLayer(config); return Effect.gen(function* () { @@ -262,28 +277,27 @@ describe("StackLifecycleCoordinator lazyServices", () => { const dataDir = mkdtempSync( join(tmpdir(), "stack-lifecycle-coordinator-lazy-ready-started-test-"), ); - const config = makeLazyConfig(dataDir); - const { layer } = setupLayer(config); - return Effect.gen(function* () { - const stopFakeServer = yield* Effect.promise(() => - startFakeHealthyServer(defaultPorts.postgrestPort), - ); + return Effect.promise(() => startFakeHealthyServer()).pipe( + Effect.flatMap((fakeServer) => { + const config = makeLazyConfig(dataDir, fakeServer.port); + const { layer } = setupLayer(config); - const stack = yield* Stack; - yield* stack.start(); + return Effect.gen(function* () { + const stack = yield* Stack; + yield* stack.start(); - // Simulate the ApiProxy's ensureService on-demand path. - yield* stack.startService("postgrest"); + // Simulate the ApiProxy's ensureService on-demand path. + yield* stack.startService("postgrest"); - yield* stack.waitAllReady().pipe(Effect.timeout(Duration.seconds(5))); + yield* stack.waitAllReady().pipe(Effect.timeout(Duration.seconds(5))); - const postgrestState = yield* stack.getState("postgrest"); - expect(["Running", "Healthy"]).toContain(postgrestState.status); + const postgrestState = yield* stack.getState("postgrest"); + expect(["Running", "Healthy"]).toContain(postgrestState.status); - yield* Effect.promise(() => stopFakeServer()); - }).pipe( - Effect.provide(layer), + yield* Effect.promise(() => fakeServer.stop()); + }).pipe(Effect.provide(layer)); + }), Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), ); }); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 22c0cce372..36638dd3cf 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -453,15 +453,9 @@ importers: packages/fleet: dependencies: - '@supabase/process-compose': - specifier: workspace:* - version: link:../process-compose '@supabase/stack': specifier: workspace:* version: link:../stack - effect: - specifier: 'catalog:' - version: 4.0.0-beta.93 devDependencies: '@tsconfig/bun': specifier: 'catalog:' From 5382e0f7a262d45f39804e4aab45addd5d451033 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 17:51:37 +0200 Subject: [PATCH 28/38] fix: regenerate pnpm-lock.yaml to repair broken fleet importer entries The packages/fleet importer referenced oxfmt@0.56.0/oxlint@1.71.0 while the lockfile package set and catalogs snapshot had moved on, breaking frozen installs in CI (ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY). Verified with a clean-clone 'pnpm install --frozen-lockfile' from scratch. Co-Authored-By: Claude Fable 5 --- pnpm-lock.yaml | 576 ++++++++++++++++++++++++------------------------- 1 file changed, 286 insertions(+), 290 deletions(-) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 36638dd3cf..e39941ba78 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -37,8 +37,8 @@ catalogs: specifier: ^1.3.14 version: 1.3.14 '@typescript/native-preview': - specifier: 7.0.0-dev.20260628.1 - version: 7.0.0-dev.20260628.1 + specifier: 7.0.0-dev.20260629.1 + version: 7.0.0-dev.20260629.1 '@vitest/coverage-istanbul': specifier: ^4.1.9 version: 4.1.9 @@ -52,11 +52,11 @@ catalogs: specifier: ^23.0.0 version: 23.0.1 oxfmt: - specifier: ^0.56.0 - version: 0.56.0 + specifier: ^0.57.0 + version: 0.57.0 oxlint: - specifier: ^1.70.0 - version: 1.71.0 + specifier: ^1.72.0 + version: 1.73.0 oxlint-tsgolint: specifier: ^0.23.0 version: 0.23.0 @@ -97,11 +97,11 @@ importers: version: 6.2.3 devDependencies: '@anthropic-ai/claude-agent-sdk': - specifier: ^0.3.195 - version: 0.3.195(@anthropic-ai/sdk@0.106.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.29.0(zod@4.4.3))(zod@4.4.3) + specifier: ^0.3.196 + version: 0.3.202(@anthropic-ai/sdk@0.107.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.29.0(zod@4.4.3))(zod@4.4.3) '@anthropic-ai/sdk': - specifier: ^0.106.0 - version: 0.106.0(zod@4.4.3) + specifier: ^0.107.0 + version: 0.107.0(zod@4.4.3) '@clack/prompts': specifier: ^1.6.0 version: 1.6.0 @@ -155,7 +155,7 @@ importers: version: 19.2.17 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260628.1 + version: 7.0.0-dev.20260629.1 '@vercel/detect-agent': specifier: ^1.2.3 version: 1.2.3 @@ -182,10 +182,10 @@ importers: version: 6.23.0 oxfmt: specifier: 'catalog:' - version: 0.56.0 + version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.71.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.23.0) oxlint-tsgolint: specifier: 'catalog:' version: 0.23.0 @@ -196,8 +196,8 @@ importers: specifier: ^7.0.0 version: 7.0.0 posthog-node: - specifier: ^5.38.6 - version: 5.38.6 + specifier: ^5.38.8 + version: 5.39.4 react: specifier: ^19.2.7 version: 19.2.7 @@ -259,7 +259,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260628.1 + version: 7.0.0-dev.20260629.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -268,10 +268,10 @@ importers: version: 6.23.0 oxfmt: specifier: 'catalog:' - version: 0.56.0 + version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.71.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.23.0) oxlint-tsgolint: specifier: 'catalog:' version: 0.23.0 @@ -282,14 +282,14 @@ importers: apps/docs: dependencies: fumadocs-core: - specifier: ^16.10.6 - version: 16.10.6(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3) + specifier: ^16.10.7 + version: 16.11.1(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3) fumadocs-mdx: specifier: ^15.0.13 - version: 15.0.13(@types/mdast@4.0.4)(@types/mdx@2.0.14)(@types/react@19.2.17)(fumadocs-core@16.10.6(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react@19.2.7)(rolldown@1.0.2)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) + version: 15.0.13(@types/mdast@4.0.4)(@types/mdx@2.0.14)(@types/react@19.2.17)(fumadocs-core@16.11.1(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react@19.2.7)(rolldown@1.0.2)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) fumadocs-ui: - specifier: ^16.10.6 - version: 16.10.6(@types/mdx@2.0.14)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(fumadocs-core@16.10.6(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7) + specifier: ^16.10.7 + version: 16.11.1(@types/mdx@2.0.14)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(fumadocs-core@16.11.1(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7) next: specifier: ^16.2.9 version: 16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7) @@ -339,7 +339,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260628.1 + version: 7.0.0-dev.20260629.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -348,10 +348,10 @@ importers: version: 6.23.0 oxfmt: specifier: 'catalog:' - version: 0.56.0 + version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.71.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.23.0) oxlint-tsgolint: specifier: 'catalog:' version: 0.23.0 @@ -381,7 +381,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260628.1 + version: 7.0.0-dev.20260629.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -390,10 +390,10 @@ importers: version: 6.23.0 oxfmt: specifier: 'catalog:' - version: 0.56.0 + version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.71.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.23.0) oxlint-tsgolint: specifier: 'catalog:' version: 0.23.0 @@ -431,7 +431,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260628.1 + version: 7.0.0-dev.20260629.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -440,10 +440,10 @@ importers: version: 6.23.0 oxfmt: specifier: 'catalog:' - version: 0.56.0 + version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.71.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.23.0) oxlint-tsgolint: specifier: 'catalog:' version: 0.23.0 @@ -471,10 +471,10 @@ importers: version: 6.23.0 oxfmt: specifier: 'catalog:' - version: 0.56.0 + version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.71.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.23.0) vitest: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) @@ -499,7 +499,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260628.1 + version: 7.0.0-dev.20260629.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -508,10 +508,10 @@ importers: version: 6.23.0 oxfmt: specifier: 'catalog:' - version: 0.56.0 + version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.71.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.23.0) oxlint-tsgolint: specifier: 'catalog:' version: 0.23.0 @@ -551,7 +551,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260628.1 + version: 7.0.0-dev.20260629.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -560,10 +560,10 @@ importers: version: 6.23.0 oxfmt: specifier: 'catalog:' - version: 0.56.0 + version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.71.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.23.0) oxlint-tsgolint: specifier: 'catalog:' version: 0.23.0 @@ -598,60 +598,60 @@ packages: resolution: {integrity: sha512-p+CMKJ93HFmLkjXKlXiVGlMQEuRb6H0MokBSwUsX+S6BRX8eV5naFZpQJFfJHjRZY0Hmnqy1/r6UWl3x+19zYA==} engines: {node: '>=18'} - '@anthropic-ai/claude-agent-sdk-darwin-arm64@0.3.195': - resolution: {integrity: sha512-WIMM/8HRCLsTDHFTIwQvvE8WCA/oaMJtdQxsP7iNyfzIGwXbuOyU95V8vYIhZfaO2yaSpbBRncunq4CtR5H4ng==} + '@anthropic-ai/claude-agent-sdk-darwin-arm64@0.3.202': + resolution: {integrity: sha512-ujR3zDthDPkZs+AxW95iHpqLT5cuwGImsS3mVxLt1DlDij4qeTnihLX8+EpQTK+oNW9jjvFA86yKwa84fa1KYA==} cpu: [arm64] os: [darwin] - '@anthropic-ai/claude-agent-sdk-darwin-x64@0.3.195': - resolution: {integrity: sha512-RY7DB+4LXosE0MJ+XELmakfPrDN1YX4lkk9CTDm28jGCVcESRz9kAEqbyaiC48dZcmN9V1NCLutzINGdcr1TBg==} + '@anthropic-ai/claude-agent-sdk-darwin-x64@0.3.202': + resolution: {integrity: sha512-s/RVSGgkVmIMfyt1ndR8braLLu82bARoijmt1kk8d4IptUZ0Sc+zNUWKoFXwR9XqDBu6rBbBF9RIzD02raT57w==} cpu: [x64] os: [darwin] - '@anthropic-ai/claude-agent-sdk-linux-arm64-musl@0.3.195': - resolution: {integrity: sha512-ZmyBA/AFzhgutcxb7dbhCm6GTjJytwNYXTxJoKE2B3A409WCYccjMqeji6vCMNxyyfylglGo5D8dVMIxW9aoug==} + '@anthropic-ai/claude-agent-sdk-linux-arm64-musl@0.3.202': + resolution: {integrity: sha512-abSb3Gah45kUNyOeKjmQ/dd1KZ4CaQz5JAr9YQxRDXoOwx8wJVx6huBIpDxjms9wyS9X5Rqxn0Lx7zFP+wV2zQ==} cpu: [arm64] os: [linux] libc: [musl] - '@anthropic-ai/claude-agent-sdk-linux-arm64@0.3.195': - resolution: {integrity: sha512-JuIq5Fnz/F1snl0aqi1gcuRZqPWoPNrL9dJ0DuievCxKkO8hnEz/Mmn5Zos7x1X8HE//ZnEvmQXoEQEZXonJew==} + '@anthropic-ai/claude-agent-sdk-linux-arm64@0.3.202': + resolution: {integrity: sha512-a4YtRkgGYt3ogePJDW8Ts6bNW690jb9LHyZaiWXsi+zT53xCNqJB2zKPyRc7hXWOqzIk4nCfwJpjmhLzMu3WIg==} cpu: [arm64] os: [linux] libc: [glibc] - '@anthropic-ai/claude-agent-sdk-linux-x64-musl@0.3.195': - resolution: {integrity: sha512-nf8Q/LauB+ZOC6QDjxNhbsvwUtYjKYnaWJLTYFwhkmsLujePnety1AtT/1ubaUoq5AM1j297DhMlYTasa79OUA==} + '@anthropic-ai/claude-agent-sdk-linux-x64-musl@0.3.202': + resolution: {integrity: sha512-fze5nAQL1ErcMCQNB10ILaWdM0QbJSaTQzBz8NVAy0FGW8ZL0t4Wf/VgFkfzXbfkaxmPuM1C27Dn5HiU7UDEHQ==} cpu: [x64] os: [linux] libc: [musl] - '@anthropic-ai/claude-agent-sdk-linux-x64@0.3.195': - resolution: {integrity: sha512-s1lNi1cL93luoqsItH+fNO4KpIhdkvnVhWGGQUQ/8ftwa2gfmcIQnOg1hG8Ks+KzeD3UUQ8L9YEVHVADnFI/9A==} + '@anthropic-ai/claude-agent-sdk-linux-x64@0.3.202': + resolution: {integrity: sha512-XIvhdCWAAT4OdOA82fOJII+WH0Tf8pFckckEbJMMmOgQBKOnHT+609Pd3Ehw6zGcA9iFrhG5mY8Ncuckeo1aMw==} cpu: [x64] os: [linux] libc: [glibc] - '@anthropic-ai/claude-agent-sdk-win32-arm64@0.3.195': - resolution: {integrity: sha512-hbkDE+xPIZzRWm+D+BKrH9uJH6USIZdDIlsyrIlGi3JFHoieYoA1vdUNyldSS9+F3ZqQtfPjr2Qy08IVB6akYA==} + '@anthropic-ai/claude-agent-sdk-win32-arm64@0.3.202': + resolution: {integrity: sha512-N1J0HRvC+8a69bqNY7+ENIYQzR0i7s+rOIGH5XtuLxvLqOnZO8LHxWEZOe8ezabGq5eZqphSCgL6vQnQQpNh+A==} cpu: [arm64] os: [win32] - '@anthropic-ai/claude-agent-sdk-win32-x64@0.3.195': - resolution: {integrity: sha512-av0piEB3X1Dzhpr8A+DqHVZ9y8s1jpn8enzwX0TKKUPBn5IqLTWC7wD6v66aoUgu4f+g4ThZirmDZA6shyPEZQ==} + '@anthropic-ai/claude-agent-sdk-win32-x64@0.3.202': + resolution: {integrity: sha512-ytLGEC1fjTSiVSoXukS+j9G+06Mi20NSzxxzlG6uE75SEB0+17tHdWUaHqd8PhH/6GPzcYx81czxWQl1MVbq4Q==} cpu: [x64] os: [win32] - '@anthropic-ai/claude-agent-sdk@0.3.195': - resolution: {integrity: sha512-FVmXu9pvOMbuBKWrF8YsYQdQ/upOpv5rS8lFAnFO5jbyXT/2hN7kEPd2vd2GJpaMvNcO/KptyQUK5AxjjTz3+w==} + '@anthropic-ai/claude-agent-sdk@0.3.202': + resolution: {integrity: sha512-LnaLxDtsZP7J6g++xRSnnpTX7CHNe4v+cvBRIlD2ar+N+xi0aqY2YDaCsxPsl+haVUB9kqlUMd0zosmwsfTGjQ==} engines: {node: '>=18.0.0'} peerDependencies: '@anthropic-ai/sdk': '>=0.93.0' '@modelcontextprotocol/sdk': ^1.29.0 zod: ^4.0.0 - '@anthropic-ai/sdk@0.106.0': - resolution: {integrity: sha512-ufwVvYNDBj2dzOGupBCTaNzBLxqcTnGOzI4z8Wouxlt+mT3J3HuOmatgCy1VmwCHOUueqZ41ERhm0O99OUcbWA==} + '@anthropic-ai/sdk@0.107.0': + resolution: {integrity: sha512-RWDWyvIeZnatUTzyX8+ayFzAqqLyoDHKnDEODFyW8H89zH+qEsh5h6XAmnbHY5DCoa58o3rjuNe3F3Hg851ayA==} hasBin: true peerDependencies: zod: ^3.25.0 || ^4.0.0 @@ -1002,14 +1002,11 @@ packages: '@types/react': optional: true - '@fumadocs/tailwind@0.0.5': - resolution: {integrity: sha512-ENKPWUDRmriccsrUDE4bDBq3FNr/ms3BP2rWlsAEMV1yP23pcCaan+ceGfeBUsAQjw7sj9Q3R4Kl3g/TCStPzQ==} + '@fumadocs/tailwind@0.1.0': + resolution: {integrity: sha512-nF/DCAwOR21HZ4AkjIOv3Iqwyqywzb6pdyeMcoa+aZzirXj5ntvNZbe3jJ0v3ehhtrRfYYeXBezvjn8ZmV+fuQ==} peerDependencies: - '@tailwindcss/oxide': ^4.0.0 tailwindcss: ^4.0.0 peerDependenciesMeta: - '@tailwindcss/oxide': - optional: true tailwindcss: optional: true @@ -1758,124 +1755,124 @@ packages: cpu: [x64] os: [win32] - '@oxfmt/binding-android-arm-eabi@0.56.0': - resolution: {integrity: sha512-CSCxi7ovYojgfdPOdUb9T508HKeAdDIKeRGg7x8IZwVJrWz9gVgX7MbUnFqtQAE4QvoNo07mj2JlwnOzJw4qqA==} + '@oxfmt/binding-android-arm-eabi@0.57.0': + resolution: {integrity: sha512-qVBsEO+KugOsCmUHcO8iqNnqc65p7PCKpCs8M66mPZ+Ri+CWbcpoQOEJBg2OTu03+0qu++NK1jj6IzvQVs0Sig==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [android] - '@oxfmt/binding-android-arm64@0.56.0': - resolution: {integrity: sha512-HYJFnd+PkDwf6S9ZPGzXXtjNqvRWFnnhdbWaouh4mi/SxU8wmDuzlMn3xo/wDTGnr4Q1VA7ZzOaE/D4biW0W6A==} + '@oxfmt/binding-android-arm64@0.57.0': + resolution: {integrity: sha512-mp6PibWbao3aizijcheOeHQaYEhcUAt8pwLniYbtLfHxL/psFF0BykAwCj+s3c6qIpa8yN8keZICWrqtZ70w8g==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [android] - '@oxfmt/binding-darwin-arm64@0.56.0': - resolution: {integrity: sha512-sftR/bEOr+t1gs+evwsHi/Xbq2FAPA2uU3VMr8n6ZU9PoK/IMSfnfu7+OEe/uy1+knhrFl4Wvy7Vkm3uo9mJ7g==} + '@oxfmt/binding-darwin-arm64@0.57.0': + resolution: {integrity: sha512-T+0stuCBqmUVY+aMIvrgXhzGhHO3sD5tNiiEcYqgSdPsnukskQqn2u5qOVD0sv1l7RLdFS5Z/f5Wi9Ktyjr3Eg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [darwin] - '@oxfmt/binding-darwin-x64@0.56.0': - resolution: {integrity: sha512-z66SdjLqa3MUPKvTp3Mbb5nSjKSbnYxJGeB+Wx987s8T5hPcIRiBMfnJ6zcPgYtQn3x5xjvdzNVkXrSeYH6ZFg==} + '@oxfmt/binding-darwin-x64@0.57.0': + resolution: {integrity: sha512-O+3JbqWs/mCI2oi4xfhRO2IVPFJNDDEBV8Odo+ZpmsUOeKJfjXoNH7nDmBEQcDgK7NfjDIyE7kRgYSZcTLDO0A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [darwin] - '@oxfmt/binding-freebsd-x64@0.56.0': - resolution: {integrity: sha512-t2tkrV1vtZyaItSQ71dTi2ZVKZEI39b/LqLT12V5KMfIeXK6N32TUC1jhOXKVQmhECq9j2ZXMQV3JeT1kh9Vmg==} + '@oxfmt/binding-freebsd-x64@0.57.0': + resolution: {integrity: sha512-pxwhxVC+JkLX9twOQ/8C/vbuOQcMZyKIDmiRDZfO7yITuVcIdZCiLRqqf4QOxb2+8FWrRXzQpm+1DBKcMpHSSQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [freebsd] - '@oxfmt/binding-linux-arm-gnueabihf@0.56.0': - resolution: {integrity: sha512-+gCy+Tp3RHeXQ9y/QrS76lXIpZkbziTyp6hIgjB2MssCwfMph3vG/GEfkhO34Rai1vhYIaUkvv8UT1BcDorJPw==} + '@oxfmt/binding-linux-arm-gnueabihf@0.57.0': + resolution: {integrity: sha512-pxBU4zH2imB/MDBfth2rOMeVxXUMjRQLCazagwLARIFH3hVlxZJBlM4nSnHXaIHJK4/qezoFCIORN6AY8Mra4A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxfmt/binding-linux-arm-musleabihf@0.56.0': - resolution: {integrity: sha512-0kKkVvQ2I+FJ2sxQyUu1zJ0yWP5kcWse/yVFnGQSFCXMwSSkfEaUGu0dW774O7nyy3jrcBGap7OSc8dZmU/CdA==} + '@oxfmt/binding-linux-arm-musleabihf@0.57.0': + resolution: {integrity: sha512-JAprOzt8tycYou36ZgEw14DlRHTiN8qdtKANdV3VZIRIvTI/lh/cX13c9pJ/EnDk2GT3FASH7KvCgQ2AufAifQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxfmt/binding-linux-arm64-gnu@0.56.0': - resolution: {integrity: sha512-npkA2siMbyWRh+wEhi1aTAx4RirukGcGNt8V4Ch86pG+xU9aurqS1MZOnKYMu03ISwat3rB6zkQx51SsB9obNw==} + '@oxfmt/binding-linux-arm64-gnu@0.57.0': + resolution: {integrity: sha512-ajtjaxSaj9xl4BW7REt+Cef/ttzbAq00Bq4z7JUDZEfgFXdwSjH8K9bF+IcIJzZB9lKqMfQ4eHuSFOvvlvtqOg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-arm64-musl@0.56.0': - resolution: {integrity: sha512-UekqOjGkV4/MkqreCV9SPIB2jlR3/HbXrmhV1rVXJZ9wfDXMyCMriLtq3tHqLY4PkbVWNtfcm1kMojJ26KLSJw==} + '@oxfmt/binding-linux-arm64-musl@0.57.0': + resolution: {integrity: sha512-p4Y/+RYk9Bk5WO+zHSUXAClRmZ2fbJCejMuCAsU2HhyME4jqf6Ftt/mJYEwIah1wGCBDYOB7wEGV1x5bCEZ6hA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [musl] - '@oxfmt/binding-linux-ppc64-gnu@0.56.0': - resolution: {integrity: sha512-XSzveSpeZMD5XJpew5lRFVtNnT04xd3rJxENXmk7wkZzN9oWzv2aFJyoNDhOtoz69BYaS/fg4SYl+CfEZRpB0Q==} + '@oxfmt/binding-linux-ppc64-gnu@0.57.0': + resolution: {integrity: sha512-By6tRALAZsno0F4zedmtG+wdMvJiJmJoXM4d3+A9zHE4HRXLqXITwRH8mgrlcXc5yJM2g2W3riRPwTYdgemZLQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-riscv64-gnu@0.56.0': - resolution: {integrity: sha512-EkQ0nJa7k7HDDIVuPF7WY+k4k+bzdclLYtyIXNt7/OqVghfNiMym6YGppFBgx1XRIHW6QylxBz5OogumPjPJbQ==} + '@oxfmt/binding-linux-riscv64-gnu@0.57.0': + resolution: {integrity: sha512-skYeG+RgvyzspqVEBsEprL90OYYZfoVNqB3HcCNR6QDJyXKOzfDRT3zncnHmUaFluIlBHuY23mU1b5WGgR98hA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-riscv64-musl@0.56.0': - resolution: {integrity: sha512-dyjAGW8jKRge0ik6U/dgvQG0nVpA3iBlRskQTz5qJLvQWIrySxX5jpqzPetLBNIIZ231KA82fDdi1nLTk8ENCw==} + '@oxfmt/binding-linux-riscv64-musl@0.57.0': + resolution: {integrity: sha512-FFgACrZOXAXUh5KQh2mt1CDOVOZmn+QzHP71wM9QobNwyQvoFfyAeefVUltW83g3sm7LTiH3yfFqLLVUpA5ZFQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [musl] - '@oxfmt/binding-linux-s390x-gnu@0.56.0': - resolution: {integrity: sha512-60ZGH3LtfqlW8X6vcLdSFY4lvCQYINurttYBKaALnHCDVAUCYJ1LsUgS6p1XOzVlzEDx3yNUZvDF1Lvt59zoZw==} + '@oxfmt/binding-linux-s390x-gnu@0.57.0': + resolution: {integrity: sha512-Nm/BAOfQeFiiKd502mZn/GAVKJwtd0RdCg17G3Wz/WSOIQmDi3+7/SZH4BHn1Ye5KvTVH3ua8WvfwLLycNIuvA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-x64-gnu@0.56.0': - resolution: {integrity: sha512-u1suj1tgJHK4ZqB7buCtdbNef2n8+d0lXTPJwLHNmtyK6p+DTpsaoDvmqhQrA56fgKYv4LuRxNtL8YooebKOew==} + '@oxfmt/binding-linux-x64-gnu@0.57.0': + resolution: {integrity: sha512-BiSy5Ku3mQqyxS6YIqAJgd403wEUWvI7kerfzPxc2l/txZVmZM0pSj7oDM+4bGBExowxOi7o73jEam1W0EDTZg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-x64-musl@0.56.0': - resolution: {integrity: sha512-aYGLvlQHt80y+qKEtfJY/Nm27G0125Lv+qyh9SJ4Cjc6lXnXjD+ndfhqQnbV24POpMi7rNRi0jvx/0d70FRpCQ==} + '@oxfmt/binding-linux-x64-musl@0.57.0': + resolution: {integrity: sha512-BCRkJiotz5s9afLYD2LuMvzAoDYx9H17E/YbDyu4xK7l4zHDPeny9ErSXL//i/nJyaOwRk08x4b8cgJC00+JDg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [musl] - '@oxfmt/binding-openharmony-arm64@0.56.0': - resolution: {integrity: sha512-H/re/gO+7ysVc+kywHNuzY3C33EN9sQcZhg0kp1ZwOZl7y998ZE5mhnBiuGR/nYI0pqLL5xQfrHVUOJ/cIJsCA==} + '@oxfmt/binding-openharmony-arm64@0.57.0': + resolution: {integrity: sha512-4Oaxe1qrGgXfpCJ1C/ERJ2iCtV2rN1R79ga9fsfyVHfSQRu/hVW780u2KDqZWFZ/iGTHODJji0JemxqFZ63eIQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [openharmony] - '@oxfmt/binding-win32-arm64-msvc@0.56.0': - resolution: {integrity: sha512-6qLNXfXmtAs8jXDvYMkxk6Wec5SUJoew+ZX1uOZmqaR7ks0EJFbAohuOCELDyJMWyVlxotVG8Xf8m74Bfq0O2w==} + '@oxfmt/binding-win32-arm64-msvc@0.57.0': + resolution: {integrity: sha512-MYLAsDnhdNsSGheLYhWgbk0vfIrlS84iQYun/y21fX6u0jj8iBtYtbpZMdiqYeuf8U12eVPUjVY2xE2NrCfJ0g==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [win32] - '@oxfmt/binding-win32-ia32-msvc@0.56.0': - resolution: {integrity: sha512-UXEXuKphAe15bsob4AswNMArCw38XSmUIs3wk1s6e6MX9OWGW/IRWU95s1hZDiVg09STy1jHgyN2qkqbu1FT0w==} + '@oxfmt/binding-win32-ia32-msvc@0.57.0': + resolution: {integrity: sha512-PBwdzZALJY/jcCx2E6is0yu+cuVXeySTDmwuseD+9j0mHqlRNxwlKgsyRTBed/woPeqfVfuXfWjoq4Cx2Zt3Eg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ia32] os: [win32] - '@oxfmt/binding-win32-x64-msvc@0.56.0': - resolution: {integrity: sha512-HPyNDjky+NIOuaMvHZflR+kst3YWdUOH2JUQYkf99grqZ5mEBTQM6h9iGy501Z8Xt5xMScrwHOuVCOlqDrktRw==} + '@oxfmt/binding-win32-x64-msvc@0.57.0': + resolution: {integrity: sha512-bQJdH9i4RRfw55jm7+8/xS7GzHLLTbHx4huhrrDxQJaJtbSDbsyOnODvP1ftT7EG0KFKAYO2S+q6AcioXODx8w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [win32] @@ -1910,124 +1907,124 @@ packages: cpu: [x64] os: [win32] - '@oxlint/binding-android-arm-eabi@1.71.0': - resolution: {integrity: sha512-ImGmd1njEg4FEJH03jhRnveEegtO3czCtfptvaHivKAZQIYATbVFBrrzbaYMYv0oJioTnxZAZVSyV+oL7W8S2g==} + '@oxlint/binding-android-arm-eabi@1.73.0': + resolution: {integrity: sha512-HZQRN/UMBu+Ut+/9MiAChkbP4qZqrNOWBcNI45vOT40GVhbGR0JgHB87L48D4iAqFQIdVmeQYtV9RF89AjTKkg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [android] - '@oxlint/binding-android-arm64@1.71.0': - resolution: {integrity: sha512-4A5BEexBrwY1YFF8Kiq/lp/wQPRG79G3BWIE1FuWaM5MvmpYSd+7ZySVcKkHdwo0UDzdQGddp6pD9mpctMqLnw==} + '@oxlint/binding-android-arm64@1.73.0': + resolution: {integrity: sha512-Gp+KJRylv2aW7thRpG5p1KTxZq4ZJFbWowrKzufNq9d3ssl3r3JviYV45/+p+7CN1Nv0zDd1e8Ex0b/HUDq4TQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [android] - '@oxlint/binding-darwin-arm64@1.71.0': - resolution: {integrity: sha512-9wJA9GJulLwS2usU3CEisI/ESDO1n1z9eyTCvApMDrAkbJ1ve0mORgTMjcWWsKxkzkeZ2N/Gpra5IQE7x8tYgQ==} + '@oxlint/binding-darwin-arm64@1.73.0': + resolution: {integrity: sha512-3de96NdtXhxERMjIz7wsp2HYMY6pMQycGxFWac2mFecAx6VeARF/IqFb1QIaqiCRIdfzBwzTed+pCTCoiS+CYA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [darwin] - '@oxlint/binding-darwin-x64@1.71.0': - resolution: {integrity: sha512-PlLCjS06V0PeJMAJwzjrExw1sYNW9Gch3JtNlcwwZDXGlTYDuwHNN89zYH8LTXFfgkVtsYvs2nv0FqrzyuFDzg==} + '@oxlint/binding-darwin-x64@1.73.0': + resolution: {integrity: sha512-5zx/uPW32TiaOeVY1dQ/H5iOf0K1HOdFKOJhLqGl4o63+i1fpzoqqu/mKtd7OFgFjNCdhlyTGgjVkQTZm1ELcg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [darwin] - '@oxlint/binding-freebsd-x64@1.71.0': - resolution: {integrity: sha512-Lhil7bWre0ncxbUoDoxfS0JzpTz17BRQKW7iwoAUY8GJ66+WwJEfYPCFJ1P0WgVZR5/O/b3Q2pENlHOjeXLOGQ==} + '@oxlint/binding-freebsd-x64@1.73.0': + resolution: {integrity: sha512-qNe4gKHaGnLuZJ8toUg90JAa0S2vTVvDw+0bRi3q1avXZXDT4u5mMeECf3nD4HYrbdn1O7dXqWut4onY/yx/Xg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [freebsd] - '@oxlint/binding-linux-arm-gnueabihf@1.71.0': - resolution: {integrity: sha512-Oo9/L58PYD3RC0x05d2upAPLllHytTjHQGsnC06P6Ynn7jKkp5mdImQxXdJ3+FnBaKspNpGogzgVsi6g872LiA==} + '@oxlint/binding-linux-arm-gnueabihf@1.73.0': + resolution: {integrity: sha512-cCehYh5hTbfShm/fxTD6wwrGUWIpvX+N5OxmAMhFhDeTGXvw+BeNj889tpxsFQ9ZLatQ6wImuY8tsKLZ+FMz7w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxlint/binding-linux-arm-musleabihf@1.71.0': - resolution: {integrity: sha512-mSHfyfgJrEbyIR29ejaeS50BdPk+GoNPlC1dckpDiUZbJAIel68sjSMdOt4WY0/gva+ECC7FNITQkxMJU+vSBw==} + '@oxlint/binding-linux-arm-musleabihf@1.73.0': + resolution: {integrity: sha512-d5j5GDU/2dMgjVhw7TQT9ITrsIr1Y02KEXKyVGIXUkD+KiaxE9TP65FS2ZdgTBemQvoRL+gSBdbrIm3cQIeacg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxlint/binding-linux-arm64-gnu@1.71.0': - resolution: {integrity: sha512-n9yY4M2tiy3aij4AqtlnspzpfdpeT5JQfK2/w2d8oyp5W0FRwOb1dIeX99nORNcxGr08iD9bH8N5XFz3I2iy8w==} + '@oxlint/binding-linux-arm64-gnu@1.73.0': + resolution: {integrity: sha512-Eyf1SrP3+yR1DI3OJgOY2Pvrr9dWP9TK37xPaDYycwTtlGlI45erJAVIfH5/m/xosDt6BupJYEFi47bvbTuuyw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [glibc] - '@oxlint/binding-linux-arm64-musl@1.71.0': - resolution: {integrity: sha512-fJZrs5sDZtTaPIOiemRQQmo82Ezy+vOGXemPc4Ok7iVVsYsFa7SlW6Z5XN819VfsqBHRm3NJ3rTdnR8+bJYJdQ==} + '@oxlint/binding-linux-arm64-musl@1.73.0': + resolution: {integrity: sha512-IlT/OJApEDKaMmCooHuncgJZbbCe7T5QIWmTZBEtYscWvzPQuuEinVcid6kwQRVQOUdb7PUCz4jQHnaYXdfJXw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [musl] - '@oxlint/binding-linux-ppc64-gnu@1.71.0': - resolution: {integrity: sha512-cwl7VKGERIy9p+G+AvZdfy/06q0aHXaTt/mMRReC751iuNYJgqKjB7NydXSS30nBT9vtr2tunciOtrR4fD6FUA==} + '@oxlint/binding-linux-ppc64-gnu@1.73.0': + resolution: {integrity: sha512-L+JYcb/vdg5fmcH08V6o0YYLU28cTH1SPNulwJdvK9NK49aXSkYy6oNpKBmddArVOXYqNepriDGiZ04G54kh1Q==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] libc: [glibc] - '@oxlint/binding-linux-riscv64-gnu@1.71.0': - resolution: {integrity: sha512-eZ8ieVXvzGi8jr7+ybQGPK2STw3mldfxZlgA2738iflfB/rzA69sE6m5rDRpQaxC7dpm745Enlh1Tod0QAk9Gg==} + '@oxlint/binding-linux-riscv64-gnu@1.73.0': + resolution: {integrity: sha512-Qtk0g3bKV6OwWjIm7R8kQN1uOZRKQt/MODK2a8QfkwhTpXBD53ozx5XLVWLGDQAVyp2otLW4D2wB98XfAfMPGA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [glibc] - '@oxlint/binding-linux-riscv64-musl@1.71.0': - resolution: {integrity: sha512-puMDbQYe6+NXwfMusojoA7CXGn2b3utukmd23PQqc1E3XhVCwyZ+FueSMzDYeNgDV2dUfIVXAAKZBcFDeCL6sA==} + '@oxlint/binding-linux-riscv64-musl@1.73.0': + resolution: {integrity: sha512-wX0NQKZVxltkAOVmzFcpOaMpdaUvsq1Eqpx9tkAfl71UdkTlSo1R4AdAnGccR1Fm2+TzFgZ22CyyGuZ41RDr/A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [musl] - '@oxlint/binding-linux-s390x-gnu@1.71.0': - resolution: {integrity: sha512-4NJLxBs1ujISCt3L/1FcywLs73PWtJuw+piD6feK2V6h6OS6P7xu9/sWt1DTRLibe6QCzmfZzmM/2HPORoV/Lg==} + '@oxlint/binding-linux-s390x-gnu@1.73.0': + resolution: {integrity: sha512-vPe7UGBMWyiLTtnqS4xxgMQFSFGmtQwhwCxuiw6lXygaO6bVt0D8dFVg8Xv05eaiN3ybC0HXXHUAohFMFvqoCQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] libc: [glibc] - '@oxlint/binding-linux-x64-gnu@1.71.0': - resolution: {integrity: sha512-cFDaiR8L3430qp88tfZnvFlt3KotFhR/DlbIL0nHOMMYiG/9Wy4l+6f7t8G8pTa9bd8Lt8+M0y/qjRQ/xcB74g==} + '@oxlint/binding-linux-x64-gnu@1.73.0': + resolution: {integrity: sha512-2CwIWr9cemFC/CbRBWZvuk5mffz6ObmfFkfcC/9rTQ7f+icNhYr2kOjf9Rt8lLvugvkdGDOmkoVoFFHh6ClCTw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [glibc] - '@oxlint/binding-linux-x64-musl@1.71.0': - resolution: {integrity: sha512-orfixdt76KlpNly9z0PkWBBNfwjKz+JFVLP/7wnVchlKNU9Dpt9InU/ZggeSej6fC7qwHmHNOGlhLnQXcYoGuA==} + '@oxlint/binding-linux-x64-musl@1.73.0': + resolution: {integrity: sha512-nDadfJgg7NBBxG0N560wOe7LLX5QiYp6qBaI7viuk5EUORFBktU/NfV0MbTqU3gTqQDCh4VyxKdo5VADxk9w8Q==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [musl] - '@oxlint/binding-openharmony-arm64@1.71.0': - resolution: {integrity: sha512-9emQu2lAp6yhPB3XuI+++vR+l/o6JR1X+EpxwcumPdQXBWXEPAsquPGL7l158EqU8SebQMXTUa/S5zN98juyHw==} + '@oxlint/binding-openharmony-arm64@1.73.0': + resolution: {integrity: sha512-wGjJC+NLH9xP+IKGn9RDW94ojJR/wPbg5WCnQjj/oReaOtCQthr8ws1zICe77JFmo4ouUdeTHHZL/ESGiF6Pmw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [openharmony] - '@oxlint/binding-win32-arm64-msvc@1.71.0': - resolution: {integrity: sha512-bd5kI8spYwTm3BILDtGhi73zoup5dw8MlPQNT8YB3BD5UIsjNe3K9/4ctrzQMX4SZMoK5HgzVLkLJzacEXB7fA==} + '@oxlint/binding-win32-arm64-msvc@1.73.0': + resolution: {integrity: sha512-I7X47GPGljw225YUQ5SbC/rb1Kkdrd0yQf0x+hYxeKS6DpfjMbo9ccQPQ6LNY6BoJQ1sHhgDUGuMn5Vg5gHT6w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [win32] - '@oxlint/binding-win32-ia32-msvc@1.71.0': - resolution: {integrity: sha512-W4HvOHGzVLHcrmFu+bMrJlho+/yrlX5ZNdJZqGe8MEldkQG+RHYhxxad9P4jvWAYFmIqUA5i9DQ8QsJqSU9GIw==} + '@oxlint/binding-win32-ia32-msvc@1.73.0': + resolution: {integrity: sha512-5lWj+3h+74Fm1jYOO9qkJA4xkAlZA099DkXppuXsk7UpnpZLttsefrZU469vChGaG6hcSqrkKXQOvMTZtbjeNg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ia32] os: [win32] - '@oxlint/binding-win32-x64-msvc@1.71.0': - resolution: {integrity: sha512-D2kyEIPHk/G/wiZLnwTVC/sVst+T/lKldVOjAFpgTIBUAOlry72e5OiapDbDBF4LfJLkN5ypJb/8Eu6yJzkveQ==} + '@oxlint/binding-win32-x64-msvc@1.73.0': + resolution: {integrity: sha512-WaNRvh4f6zY9CvUQk2YoA1O90ieWrIklI84+HXFr9Isjz9CSESrdqo/RtIYt4Dll/cAchqGDMehfaZd0vqEFZw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [win32] @@ -2883,50 +2880,50 @@ packages: '@types/ws@8.18.1': resolution: {integrity: sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg==} - '@typescript/native-preview-darwin-arm64@7.0.0-dev.20260628.1': - resolution: {integrity: sha512-eHHDHAZjbZ681sHyW87tg8mH4+xIs+Q7cHKIlrdafqeny1KYWorj3O9Qfnjvcl2Yd2Eq+IzJxffF6Tepy13L4Q==} + '@typescript/native-preview-darwin-arm64@7.0.0-dev.20260629.1': + resolution: {integrity: sha512-wXRExZJweYoTzE4atRR7T5HwKJYkl6/KHxON0eF0iy2fvgLXDlyq4AQqhmV8mMx10PQKc/4sNbfhD4kjWWvm8A==} engines: {node: '>=16.20.0'} cpu: [arm64] os: [darwin] - '@typescript/native-preview-darwin-x64@7.0.0-dev.20260628.1': - resolution: {integrity: sha512-xIsJSXa0Fsv0pPfQ0YYa7nUQJ/nGRF/r8p60e0Aa29SexxgOXMsu3YhOnUnJEdbvaPzqlKqa1GqNGpbGnLQcLA==} + '@typescript/native-preview-darwin-x64@7.0.0-dev.20260629.1': + resolution: {integrity: sha512-xkL/tETzUuDxfRkk2rD0/CjjjGLyOLHeE103T52rpgj0VNSXMnn7tTiw+TSdxnS61b8ImOrWgQJujhPFTp0jng==} engines: {node: '>=16.20.0'} cpu: [x64] os: [darwin] - '@typescript/native-preview-linux-arm64@7.0.0-dev.20260628.1': - resolution: {integrity: sha512-p3yj2a70vkaFB3OD+Vt4oNUaiE7I30fwiXs6LVNAW6k0GSHNc4ucYcWVlpcp8+cej9RBQgxMnMH5MSVYmNhUvw==} + '@typescript/native-preview-linux-arm64@7.0.0-dev.20260629.1': + resolution: {integrity: sha512-y4ey82krq4iLNHML4G1dUObBWY6gNAjVqaUHFDv7+LBu5bfAJy8VClBaLtAJce3siQQeB0/4SkN4XsAa0oZktQ==} engines: {node: '>=16.20.0'} cpu: [arm64] os: [linux] - '@typescript/native-preview-linux-arm@7.0.0-dev.20260628.1': - resolution: {integrity: sha512-BoclteE+MBOnfK0Qh21mQgrvYPy/v2k7CPTPufcNp1g1fsSvsF3Xv6K8I/grEjo3ZjNrgIvVxivoAqaQhSMlGA==} + '@typescript/native-preview-linux-arm@7.0.0-dev.20260629.1': + resolution: {integrity: sha512-QlTxO+O6yELc0NYZZYIbncZhkhw+K/vBoVg+mKipbEIK5b1E6cwW7ivWIrDp1FfssQ0Wu7zxincPappoci7KNw==} engines: {node: '>=16.20.0'} cpu: [arm] os: [linux] - '@typescript/native-preview-linux-x64@7.0.0-dev.20260628.1': - resolution: {integrity: sha512-LKNKDoTnM8aacpbt1u8kJR1feXpBuLlvKKbVt0RYBL4j1OA148TXKjLtbVu37I0lcVxjqERYuAybpvut2xq31w==} + '@typescript/native-preview-linux-x64@7.0.0-dev.20260629.1': + resolution: {integrity: sha512-ckb4HyugC5beDpOMPOVpEx1H3l2Z2RgsGPxFOLGMU6LLIHQwlCaSdRjSfH8Hq8yffPgKuotTQLdA89cP1IC8xQ==} engines: {node: '>=16.20.0'} cpu: [x64] os: [linux] - '@typescript/native-preview-win32-arm64@7.0.0-dev.20260628.1': - resolution: {integrity: sha512-XUGCYlDAfeA4PIm7ZSZtVHmvffVoMct0LhTA/CoALhSQFnFnJdipOfsZghSyU6TCpTuzBoOhWCjBufrQC23xOQ==} + '@typescript/native-preview-win32-arm64@7.0.0-dev.20260629.1': + resolution: {integrity: sha512-Ktgepu9p0blqrbiryzYrw11ddnbESXPdeGKURDG/wXESoHQETsMNxKWhqAo587ItNnWjKOBFxoEM22eP9OzKnw==} engines: {node: '>=16.20.0'} cpu: [arm64] os: [win32] - '@typescript/native-preview-win32-x64@7.0.0-dev.20260628.1': - resolution: {integrity: sha512-rJMZ+YaRv9XybOZBYAsJt7x/K2IWmX9bgRatHobl0wwkKmfKd3giNnRXcDwOqYeCaWzunCbUhAirUtuUprRcnQ==} + '@typescript/native-preview-win32-x64@7.0.0-dev.20260629.1': + resolution: {integrity: sha512-b2wmeIpFJs/peej3NG26320ya+iYQz21JzFYDrnvtsEeR/g66rB9uvp4Owo4J1AG/BcmRWC/nuwrBy3Jdb+UDA==} engines: {node: '>=16.20.0'} cpu: [x64] os: [win32] - '@typescript/native-preview@7.0.0-dev.20260628.1': - resolution: {integrity: sha512-359WmBk3vA/bJxfeWyLbFeeejmky7Wssc8HMu0Iabu490WJLj/FqkDC51V65yuDp+anMAEkgeKO43fj6pMb/ZA==} + '@typescript/native-preview@7.0.0-dev.20260629.1': + resolution: {integrity: sha512-KImWFkxGUa/V78bUEAgzeJQT+6wKQXDDYHHrOavof8wnbMCpj86KuPNyBIPQMtoHiz2If8aNTums2m50oE3oqw==} engines: {node: '>=16.20.0'} hasBin: true @@ -4036,8 +4033,8 @@ packages: engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} os: [darwin] - fumadocs-core@16.10.6: - resolution: {integrity: sha512-vd/hidAsC1d8ldrfCvr/vb/H46AC/iRm0Zp4SOWWWfa1iNoFw1mFIRGHwaNDB9ETkZPZ7xUwhi3QkhlXPrYbVw==} + fumadocs-core@16.11.1: + resolution: {integrity: sha512-tKuh1AKoVTb+f7IoAOM2cfz5djd3YhePeqA95q6mf422gEvDTeJms23OJ+icYRWZ6ryNQ5W/ZsgKEe87M5HVYg==} peerDependencies: '@mdx-js/mdx': '*' '@mixedbread/sdk': 0.x.x @@ -4126,13 +4123,13 @@ packages: vite: optional: true - fumadocs-ui@16.10.6: - resolution: {integrity: sha512-Sgp3r1FKNMnuG1AQlLUDsOcuE9kR3GU6RGuIUC5io3PFUNRc+IE4cjwGYv4i8vZi+VPV1Qqwm8oWztHCkwLvsg==} + fumadocs-ui@16.11.1: + resolution: {integrity: sha512-Dq819PFV4RGhAI9Wd4erSCiRlEDLVOZae+kgE5LeOKFH8mbKX49U8N17ldFOhdkC9EZpxMZdEKul77RDgFHQww==} peerDependencies: '@takumi-rs/image-response': '*' '@types/mdx': '*' '@types/react': '*' - fumadocs-core: 16.10.6 + fumadocs-core: 16.11.1 next: 16.x.x react: ^19.2.0 react-dom: ^19.2.0 @@ -5392,8 +5389,8 @@ packages: oxc-resolver@11.21.3: resolution: {integrity: sha512-2Mx3fKQz7+xgrBONjsxOgCGtMHOn38/HxMzW1I5efwXB5a4lRN0Vp40gYUJFBWJslcrvwoofTrqoTnLbwTd3pA==} - oxfmt@0.56.0: - resolution: {integrity: sha512-9Dv0wV3zKiyvhjD7bRKaInKmHQ1sPx3RGOjQkGFJbbdQ16576yf8qhMSO9Q9cvHcs+1NpBsRTkuDDYFFPTJ6gw==} + oxfmt@0.57.0: + resolution: {integrity: sha512-ZB7Bi+rGDSqmVIo9jwcLyFgjxXvQhDdU+jx+ZrVy6VRiVXK2+CHc4hO3J4dUQjHe7V0ymHB+MDuv5z+NhK07HA==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true peerDependencies: @@ -5409,12 +5406,12 @@ packages: resolution: {integrity: sha512-3mBv3CoPbh8dFbzfDGIWa2ytZjn2v+3EX4aKRXjIhsoGFzG8GCjfRirz3rwZf1wYbZzsNLTSgpw8VjQuWdp/jA==} hasBin: true - oxlint@1.71.0: - resolution: {integrity: sha512-U1m1X+C0vDj7DC1e13IoZULzEcPczE7UOMTs8VlZGHUEIUaSTZKo5qkPsQEfzpgnQ29Pea/w3Xntk62UCecxZw==} + oxlint@1.73.0: + resolution: {integrity: sha512-u91G9TJzU6yqKWNZUYprQB07W7YvntZXaRxQ6CkoytepYhLWUXWsr1M8zUJ34VatNPuUAr3Z8GH+O2A331CluQ==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true peerDependencies: - oxlint-tsgolint: '>=0.22.1' + oxlint-tsgolint: '>=0.24.0' vite-plus: '*' peerDependenciesMeta: oxlint-tsgolint: @@ -5686,8 +5683,8 @@ packages: postgres-range@1.1.4: resolution: {integrity: sha512-i/hbxIE9803Alj/6ytL7UHQxRvZkI9O4Sy+J3HGc4F4oo/2eQAjTSNJ0bfxyse3bH0nuVesCk+3IRLaMtG3H6w==} - posthog-node@5.38.6: - resolution: {integrity: sha512-Sm2mCAa9/lTTYppnKyy0AhQrriq8fOd8B2vwd3EE/9uihyIx9qkJ1xGYbvADxQJlF7HqM1/TIFCKsI0JvF8gjQ==} + posthog-node@5.39.4: + resolution: {integrity: sha512-+fCQ7htBFRQQFbIzl1T0TA7bDwYyaB9XP308ZFMCUoB5LzTzOFxBa6TYVrxdH/VQl43WXTp6sf0QsG2Z4XlNBg==} engines: {node: ^20.20.0 || >=22.22.0} peerDependencies: rxjs: ^7.0.0 @@ -6873,46 +6870,46 @@ snapshots: ansi-styles: 6.2.3 is-fullwidth-code-point: 5.1.0 - '@anthropic-ai/claude-agent-sdk-darwin-arm64@0.3.195': + '@anthropic-ai/claude-agent-sdk-darwin-arm64@0.3.202': optional: true - '@anthropic-ai/claude-agent-sdk-darwin-x64@0.3.195': + '@anthropic-ai/claude-agent-sdk-darwin-x64@0.3.202': optional: true - '@anthropic-ai/claude-agent-sdk-linux-arm64-musl@0.3.195': + '@anthropic-ai/claude-agent-sdk-linux-arm64-musl@0.3.202': optional: true - '@anthropic-ai/claude-agent-sdk-linux-arm64@0.3.195': + '@anthropic-ai/claude-agent-sdk-linux-arm64@0.3.202': optional: true - '@anthropic-ai/claude-agent-sdk-linux-x64-musl@0.3.195': + '@anthropic-ai/claude-agent-sdk-linux-x64-musl@0.3.202': optional: true - '@anthropic-ai/claude-agent-sdk-linux-x64@0.3.195': + '@anthropic-ai/claude-agent-sdk-linux-x64@0.3.202': optional: true - '@anthropic-ai/claude-agent-sdk-win32-arm64@0.3.195': + '@anthropic-ai/claude-agent-sdk-win32-arm64@0.3.202': optional: true - '@anthropic-ai/claude-agent-sdk-win32-x64@0.3.195': + '@anthropic-ai/claude-agent-sdk-win32-x64@0.3.202': optional: true - '@anthropic-ai/claude-agent-sdk@0.3.195(@anthropic-ai/sdk@0.106.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.29.0(zod@4.4.3))(zod@4.4.3)': + '@anthropic-ai/claude-agent-sdk@0.3.202(@anthropic-ai/sdk@0.107.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.29.0(zod@4.4.3))(zod@4.4.3)': dependencies: - '@anthropic-ai/sdk': 0.106.0(zod@4.4.3) + '@anthropic-ai/sdk': 0.107.0(zod@4.4.3) '@modelcontextprotocol/sdk': 1.29.0(zod@4.4.3) zod: 4.4.3 optionalDependencies: - '@anthropic-ai/claude-agent-sdk-darwin-arm64': 0.3.195 - '@anthropic-ai/claude-agent-sdk-darwin-x64': 0.3.195 - '@anthropic-ai/claude-agent-sdk-linux-arm64': 0.3.195 - '@anthropic-ai/claude-agent-sdk-linux-arm64-musl': 0.3.195 - '@anthropic-ai/claude-agent-sdk-linux-x64': 0.3.195 - '@anthropic-ai/claude-agent-sdk-linux-x64-musl': 0.3.195 - '@anthropic-ai/claude-agent-sdk-win32-arm64': 0.3.195 - '@anthropic-ai/claude-agent-sdk-win32-x64': 0.3.195 + '@anthropic-ai/claude-agent-sdk-darwin-arm64': 0.3.202 + '@anthropic-ai/claude-agent-sdk-darwin-x64': 0.3.202 + '@anthropic-ai/claude-agent-sdk-linux-arm64': 0.3.202 + '@anthropic-ai/claude-agent-sdk-linux-arm64-musl': 0.3.202 + '@anthropic-ai/claude-agent-sdk-linux-x64': 0.3.202 + '@anthropic-ai/claude-agent-sdk-linux-x64-musl': 0.3.202 + '@anthropic-ai/claude-agent-sdk-win32-arm64': 0.3.202 + '@anthropic-ai/claude-agent-sdk-win32-x64': 0.3.202 - '@anthropic-ai/sdk@0.106.0(zod@4.4.3)': + '@anthropic-ai/sdk@0.107.0(zod@4.4.3)': dependencies: json-schema-to-ts: 3.1.1 standardwebhooks: 1.0.0 @@ -7266,7 +7263,7 @@ snapshots: optionalDependencies: '@types/react': 19.2.17 - '@fumadocs/tailwind@0.0.5': {} + '@fumadocs/tailwind@0.1.0': {} '@hono/node-server@1.19.14(hono@4.12.21)': dependencies: @@ -7820,61 +7817,61 @@ snapshots: '@oxc-resolver/binding-win32-x64-msvc@11.21.3': optional: true - '@oxfmt/binding-android-arm-eabi@0.56.0': + '@oxfmt/binding-android-arm-eabi@0.57.0': optional: true - '@oxfmt/binding-android-arm64@0.56.0': + '@oxfmt/binding-android-arm64@0.57.0': optional: true - '@oxfmt/binding-darwin-arm64@0.56.0': + '@oxfmt/binding-darwin-arm64@0.57.0': optional: true - '@oxfmt/binding-darwin-x64@0.56.0': + '@oxfmt/binding-darwin-x64@0.57.0': optional: true - '@oxfmt/binding-freebsd-x64@0.56.0': + '@oxfmt/binding-freebsd-x64@0.57.0': optional: true - '@oxfmt/binding-linux-arm-gnueabihf@0.56.0': + '@oxfmt/binding-linux-arm-gnueabihf@0.57.0': optional: true - '@oxfmt/binding-linux-arm-musleabihf@0.56.0': + '@oxfmt/binding-linux-arm-musleabihf@0.57.0': optional: true - '@oxfmt/binding-linux-arm64-gnu@0.56.0': + '@oxfmt/binding-linux-arm64-gnu@0.57.0': optional: true - '@oxfmt/binding-linux-arm64-musl@0.56.0': + '@oxfmt/binding-linux-arm64-musl@0.57.0': optional: true - '@oxfmt/binding-linux-ppc64-gnu@0.56.0': + '@oxfmt/binding-linux-ppc64-gnu@0.57.0': optional: true - '@oxfmt/binding-linux-riscv64-gnu@0.56.0': + '@oxfmt/binding-linux-riscv64-gnu@0.57.0': optional: true - '@oxfmt/binding-linux-riscv64-musl@0.56.0': + '@oxfmt/binding-linux-riscv64-musl@0.57.0': optional: true - '@oxfmt/binding-linux-s390x-gnu@0.56.0': + '@oxfmt/binding-linux-s390x-gnu@0.57.0': optional: true - '@oxfmt/binding-linux-x64-gnu@0.56.0': + '@oxfmt/binding-linux-x64-gnu@0.57.0': optional: true - '@oxfmt/binding-linux-x64-musl@0.56.0': + '@oxfmt/binding-linux-x64-musl@0.57.0': optional: true - '@oxfmt/binding-openharmony-arm64@0.56.0': + '@oxfmt/binding-openharmony-arm64@0.57.0': optional: true - '@oxfmt/binding-win32-arm64-msvc@0.56.0': + '@oxfmt/binding-win32-arm64-msvc@0.57.0': optional: true - '@oxfmt/binding-win32-ia32-msvc@0.56.0': + '@oxfmt/binding-win32-ia32-msvc@0.57.0': optional: true - '@oxfmt/binding-win32-x64-msvc@0.56.0': + '@oxfmt/binding-win32-x64-msvc@0.57.0': optional: true '@oxlint-tsgolint/darwin-arm64@0.23.0': @@ -7895,61 +7892,61 @@ snapshots: '@oxlint-tsgolint/win32-x64@0.23.0': optional: true - '@oxlint/binding-android-arm-eabi@1.71.0': + '@oxlint/binding-android-arm-eabi@1.73.0': optional: true - '@oxlint/binding-android-arm64@1.71.0': + '@oxlint/binding-android-arm64@1.73.0': optional: true - '@oxlint/binding-darwin-arm64@1.71.0': + '@oxlint/binding-darwin-arm64@1.73.0': optional: true - '@oxlint/binding-darwin-x64@1.71.0': + '@oxlint/binding-darwin-x64@1.73.0': optional: true - '@oxlint/binding-freebsd-x64@1.71.0': + '@oxlint/binding-freebsd-x64@1.73.0': optional: true - '@oxlint/binding-linux-arm-gnueabihf@1.71.0': + '@oxlint/binding-linux-arm-gnueabihf@1.73.0': optional: true - '@oxlint/binding-linux-arm-musleabihf@1.71.0': + '@oxlint/binding-linux-arm-musleabihf@1.73.0': optional: true - '@oxlint/binding-linux-arm64-gnu@1.71.0': + '@oxlint/binding-linux-arm64-gnu@1.73.0': optional: true - '@oxlint/binding-linux-arm64-musl@1.71.0': + '@oxlint/binding-linux-arm64-musl@1.73.0': optional: true - '@oxlint/binding-linux-ppc64-gnu@1.71.0': + '@oxlint/binding-linux-ppc64-gnu@1.73.0': optional: true - '@oxlint/binding-linux-riscv64-gnu@1.71.0': + '@oxlint/binding-linux-riscv64-gnu@1.73.0': optional: true - '@oxlint/binding-linux-riscv64-musl@1.71.0': + '@oxlint/binding-linux-riscv64-musl@1.73.0': optional: true - '@oxlint/binding-linux-s390x-gnu@1.71.0': + '@oxlint/binding-linux-s390x-gnu@1.73.0': optional: true - '@oxlint/binding-linux-x64-gnu@1.71.0': + '@oxlint/binding-linux-x64-gnu@1.73.0': optional: true - '@oxlint/binding-linux-x64-musl@1.71.0': + '@oxlint/binding-linux-x64-musl@1.73.0': optional: true - '@oxlint/binding-openharmony-arm64@1.71.0': + '@oxlint/binding-openharmony-arm64@1.73.0': optional: true - '@oxlint/binding-win32-arm64-msvc@1.71.0': + '@oxlint/binding-win32-arm64-msvc@1.73.0': optional: true - '@oxlint/binding-win32-ia32-msvc@1.71.0': + '@oxlint/binding-win32-ia32-msvc@1.73.0': optional: true - '@oxlint/binding-win32-x64-msvc@1.71.0': + '@oxlint/binding-win32-x64-msvc@1.73.0': optional: true '@parcel/watcher-android-arm64@2.5.6': @@ -8759,36 +8756,36 @@ snapshots: dependencies: '@types/node': 26.0.1 - '@typescript/native-preview-darwin-arm64@7.0.0-dev.20260628.1': + '@typescript/native-preview-darwin-arm64@7.0.0-dev.20260629.1': optional: true - '@typescript/native-preview-darwin-x64@7.0.0-dev.20260628.1': + '@typescript/native-preview-darwin-x64@7.0.0-dev.20260629.1': optional: true - '@typescript/native-preview-linux-arm64@7.0.0-dev.20260628.1': + '@typescript/native-preview-linux-arm64@7.0.0-dev.20260629.1': optional: true - '@typescript/native-preview-linux-arm@7.0.0-dev.20260628.1': + '@typescript/native-preview-linux-arm@7.0.0-dev.20260629.1': optional: true - '@typescript/native-preview-linux-x64@7.0.0-dev.20260628.1': + '@typescript/native-preview-linux-x64@7.0.0-dev.20260629.1': optional: true - '@typescript/native-preview-win32-arm64@7.0.0-dev.20260628.1': + '@typescript/native-preview-win32-arm64@7.0.0-dev.20260629.1': optional: true - '@typescript/native-preview-win32-x64@7.0.0-dev.20260628.1': + '@typescript/native-preview-win32-x64@7.0.0-dev.20260629.1': optional: true - '@typescript/native-preview@7.0.0-dev.20260628.1': + '@typescript/native-preview@7.0.0-dev.20260629.1': optionalDependencies: - '@typescript/native-preview-darwin-arm64': 7.0.0-dev.20260628.1 - '@typescript/native-preview-darwin-x64': 7.0.0-dev.20260628.1 - '@typescript/native-preview-linux-arm': 7.0.0-dev.20260628.1 - '@typescript/native-preview-linux-arm64': 7.0.0-dev.20260628.1 - '@typescript/native-preview-linux-x64': 7.0.0-dev.20260628.1 - '@typescript/native-preview-win32-arm64': 7.0.0-dev.20260628.1 - '@typescript/native-preview-win32-x64': 7.0.0-dev.20260628.1 + '@typescript/native-preview-darwin-arm64': 7.0.0-dev.20260629.1 + '@typescript/native-preview-darwin-x64': 7.0.0-dev.20260629.1 + '@typescript/native-preview-linux-arm': 7.0.0-dev.20260629.1 + '@typescript/native-preview-linux-arm64': 7.0.0-dev.20260629.1 + '@typescript/native-preview-linux-x64': 7.0.0-dev.20260629.1 + '@typescript/native-preview-win32-arm64': 7.0.0-dev.20260629.1 + '@typescript/native-preview-win32-x64': 7.0.0-dev.20260629.1 '@ungap/structured-clone@1.3.2': {} @@ -10068,7 +10065,7 @@ snapshots: fsevents@2.3.3: optional: true - fumadocs-core@16.10.6(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3): + fumadocs-core@16.11.1(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3): dependencies: '@orama/orama': 3.1.18 estree-util-value-to-estree: 3.5.0 @@ -10101,14 +10098,14 @@ snapshots: transitivePeerDependencies: - supports-color - fumadocs-mdx@15.0.13(@types/mdast@4.0.4)(@types/mdx@2.0.14)(@types/react@19.2.17)(fumadocs-core@16.10.6(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react@19.2.7)(rolldown@1.0.2)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)): + fumadocs-mdx@15.0.13(@types/mdast@4.0.4)(@types/mdx@2.0.14)(@types/react@19.2.17)(fumadocs-core@16.11.1(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react@19.2.7)(rolldown@1.0.2)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)): dependencies: '@mdx-js/mdx': 3.1.1 '@standard-schema/spec': 1.1.0 chokidar: 5.0.0 esbuild: 0.28.1 estree-util-value-to-estree: 3.5.0 - fumadocs-core: 16.10.6(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3) + fumadocs-core: 16.11.1(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3) js-yaml: 5.2.1 mdast-util-mdx: 3.0.0 picocolors: 1.1.1 @@ -10131,10 +10128,10 @@ snapshots: transitivePeerDependencies: - supports-color - fumadocs-ui@16.10.6(@types/mdx@2.0.14)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(fumadocs-core@16.10.6(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7): + fumadocs-ui@16.11.1(@types/mdx@2.0.14)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(fumadocs-core@16.11.1(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7): dependencies: '@fuma-translate/react': 1.0.2(@types/react@19.2.17)(react-dom@19.2.7(react@19.2.7))(react@19.2.7) - '@fumadocs/tailwind': 0.0.5 + '@fumadocs/tailwind': 0.1.0 '@radix-ui/react-accordion': 1.2.15(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.7(react@19.2.7))(react@19.2.7) '@radix-ui/react-collapsible': 1.1.15(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.7(react@19.2.7))(react@19.2.7) '@radix-ui/react-dialog': 1.1.18(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.7(react@19.2.7))(react@19.2.7) @@ -10147,7 +10144,7 @@ snapshots: '@radix-ui/react-tabs': 1.1.16(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.7(react@19.2.7))(react@19.2.7) class-variance-authority: 0.7.1 cnfast: 0.0.8 - fumadocs-core: 16.10.6(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3) + fumadocs-core: 16.11.1(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.17)(lucide-react@1.23.0(react@19.2.7))(next@16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(react-dom@19.2.7(react@19.2.7))(react@19.2.7)(zod@4.4.3) lucide-react: 1.23.0(react@19.2.7) motion: 12.42.2(react-dom@19.2.7(react@19.2.7))(react@19.2.7) next-themes: 0.4.6(react-dom@19.2.7(react@19.2.7))(react@19.2.7) @@ -10164,7 +10161,6 @@ snapshots: next: 16.2.9(react-dom@19.2.7(react@19.2.7))(react@19.2.7) transitivePeerDependencies: - '@emotion/is-prop-valid' - - '@tailwindcss/oxide' - '@types/react-dom' - tailwindcss @@ -11786,29 +11782,29 @@ snapshots: '@oxc-resolver/binding-win32-arm64-msvc': 11.21.3 '@oxc-resolver/binding-win32-x64-msvc': 11.21.3 - oxfmt@0.56.0: + oxfmt@0.57.0: dependencies: tinypool: 2.1.0 optionalDependencies: - '@oxfmt/binding-android-arm-eabi': 0.56.0 - '@oxfmt/binding-android-arm64': 0.56.0 - '@oxfmt/binding-darwin-arm64': 0.56.0 - '@oxfmt/binding-darwin-x64': 0.56.0 - '@oxfmt/binding-freebsd-x64': 0.56.0 - '@oxfmt/binding-linux-arm-gnueabihf': 0.56.0 - '@oxfmt/binding-linux-arm-musleabihf': 0.56.0 - '@oxfmt/binding-linux-arm64-gnu': 0.56.0 - '@oxfmt/binding-linux-arm64-musl': 0.56.0 - '@oxfmt/binding-linux-ppc64-gnu': 0.56.0 - '@oxfmt/binding-linux-riscv64-gnu': 0.56.0 - '@oxfmt/binding-linux-riscv64-musl': 0.56.0 - '@oxfmt/binding-linux-s390x-gnu': 0.56.0 - '@oxfmt/binding-linux-x64-gnu': 0.56.0 - '@oxfmt/binding-linux-x64-musl': 0.56.0 - '@oxfmt/binding-openharmony-arm64': 0.56.0 - '@oxfmt/binding-win32-arm64-msvc': 0.56.0 - '@oxfmt/binding-win32-ia32-msvc': 0.56.0 - '@oxfmt/binding-win32-x64-msvc': 0.56.0 + '@oxfmt/binding-android-arm-eabi': 0.57.0 + '@oxfmt/binding-android-arm64': 0.57.0 + '@oxfmt/binding-darwin-arm64': 0.57.0 + '@oxfmt/binding-darwin-x64': 0.57.0 + '@oxfmt/binding-freebsd-x64': 0.57.0 + '@oxfmt/binding-linux-arm-gnueabihf': 0.57.0 + '@oxfmt/binding-linux-arm-musleabihf': 0.57.0 + '@oxfmt/binding-linux-arm64-gnu': 0.57.0 + '@oxfmt/binding-linux-arm64-musl': 0.57.0 + '@oxfmt/binding-linux-ppc64-gnu': 0.57.0 + '@oxfmt/binding-linux-riscv64-gnu': 0.57.0 + '@oxfmt/binding-linux-riscv64-musl': 0.57.0 + '@oxfmt/binding-linux-s390x-gnu': 0.57.0 + '@oxfmt/binding-linux-x64-gnu': 0.57.0 + '@oxfmt/binding-linux-x64-musl': 0.57.0 + '@oxfmt/binding-openharmony-arm64': 0.57.0 + '@oxfmt/binding-win32-arm64-msvc': 0.57.0 + '@oxfmt/binding-win32-ia32-msvc': 0.57.0 + '@oxfmt/binding-win32-x64-msvc': 0.57.0 oxlint-tsgolint@0.23.0: optionalDependencies: @@ -11819,27 +11815,27 @@ snapshots: '@oxlint-tsgolint/win32-arm64': 0.23.0 '@oxlint-tsgolint/win32-x64': 0.23.0 - oxlint@1.71.0(oxlint-tsgolint@0.23.0): + oxlint@1.73.0(oxlint-tsgolint@0.23.0): optionalDependencies: - '@oxlint/binding-android-arm-eabi': 1.71.0 - '@oxlint/binding-android-arm64': 1.71.0 - '@oxlint/binding-darwin-arm64': 1.71.0 - '@oxlint/binding-darwin-x64': 1.71.0 - '@oxlint/binding-freebsd-x64': 1.71.0 - '@oxlint/binding-linux-arm-gnueabihf': 1.71.0 - '@oxlint/binding-linux-arm-musleabihf': 1.71.0 - '@oxlint/binding-linux-arm64-gnu': 1.71.0 - '@oxlint/binding-linux-arm64-musl': 1.71.0 - '@oxlint/binding-linux-ppc64-gnu': 1.71.0 - '@oxlint/binding-linux-riscv64-gnu': 1.71.0 - '@oxlint/binding-linux-riscv64-musl': 1.71.0 - '@oxlint/binding-linux-s390x-gnu': 1.71.0 - '@oxlint/binding-linux-x64-gnu': 1.71.0 - '@oxlint/binding-linux-x64-musl': 1.71.0 - '@oxlint/binding-openharmony-arm64': 1.71.0 - '@oxlint/binding-win32-arm64-msvc': 1.71.0 - '@oxlint/binding-win32-ia32-msvc': 1.71.0 - '@oxlint/binding-win32-x64-msvc': 1.71.0 + '@oxlint/binding-android-arm-eabi': 1.73.0 + '@oxlint/binding-android-arm64': 1.73.0 + '@oxlint/binding-darwin-arm64': 1.73.0 + '@oxlint/binding-darwin-x64': 1.73.0 + '@oxlint/binding-freebsd-x64': 1.73.0 + '@oxlint/binding-linux-arm-gnueabihf': 1.73.0 + '@oxlint/binding-linux-arm-musleabihf': 1.73.0 + '@oxlint/binding-linux-arm64-gnu': 1.73.0 + '@oxlint/binding-linux-arm64-musl': 1.73.0 + '@oxlint/binding-linux-ppc64-gnu': 1.73.0 + '@oxlint/binding-linux-riscv64-gnu': 1.73.0 + '@oxlint/binding-linux-riscv64-musl': 1.73.0 + '@oxlint/binding-linux-s390x-gnu': 1.73.0 + '@oxlint/binding-linux-x64-gnu': 1.73.0 + '@oxlint/binding-linux-x64-musl': 1.73.0 + '@oxlint/binding-openharmony-arm64': 1.73.0 + '@oxlint/binding-win32-arm64-msvc': 1.73.0 + '@oxlint/binding-win32-ia32-msvc': 1.73.0 + '@oxlint/binding-win32-x64-msvc': 1.73.0 oxlint-tsgolint: 0.23.0 p-cancelable@2.1.1: {} @@ -12081,7 +12077,7 @@ snapshots: postgres-range@1.1.4: {} - posthog-node@5.38.6: + posthog-node@5.39.4: dependencies: '@posthog/core': 1.39.6 From f1c8854de54afd208e5fbc4007f4f50df6079aa5 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 18:23:19 +0200 Subject: [PATCH 29/38] fix(process-compose): model long-running daemons in mock spawner to fix flaky test The shared mock ChildProcessSpawner made every spawned process exit with code 0 after a fixed 10ms, including long-running daemons (postgres, postgrest). Under the orchestrator's `unless-stopped` restart policy this turned each daemon into a crash-loop: RestartTriggered -> exponential backoff -> respawn, repeating with growing backoff (1s, 2s, 4s...). The projected status never settled on Running/Healthy, so StackLifecycleCoordinator.unit.test.ts's poll-until-ready helper spun until the 5s test timeout (~30-40% of runs under Bun; never under Node, whose timer scheduling happened to keep the churn tighter). This is a mock-fidelity artifact, not a product bug: real daemons run until stopped, so the restart loop never engages in production. The Orchestrator's restart/waitReady/resetService logic is correct and unchanged. The mock now models a spawned process as a long-running daemon (exit resolves only on kill, reporting 143) when it is a supervised launch wrapping a real binary. Short-lived processes keep the 10ms auto-exit: health-check probes (pg_isready), one-shot init services launched via bash/sh (postgres-init, restart:"no"), and docker commands. This keeps it.live clocks progressing and one-shots completing. Also keep the pre-existing waitForReadyStatus test edits (same file, same flake theme): they replace two racy projected-status snapshot assertions with a poll-until-settled helper. Co-Authored-By: Claude Fable 5 --- .../process-compose/tests/helpers/mocks.ts | 78 ++++++++++++++++--- .../StackLifecycleCoordinator.unit.test.ts | 29 +++++-- 2 files changed, 88 insertions(+), 19 deletions(-) diff --git a/packages/process-compose/tests/helpers/mocks.ts b/packages/process-compose/tests/helpers/mocks.ts index 4d564db8af..ac771edfc5 100644 --- a/packages/process-compose/tests/helpers/mocks.ts +++ b/packages/process-compose/tests/helpers/mocks.ts @@ -8,6 +8,52 @@ interface SpawnRecord { const encoder = new TextEncoder(); +/** + * Decode the supervisor payload (base64url JSON in the last arg — see + * `makeSupervisedCommand`) to recover the underlying service command. Returns + * `undefined` for non-supervised spawns (health-check probes, docker commands, + * etc.) whose last arg is not a supervisor payload. + */ +function decodeSupervisedInnerCommand(args: ReadonlyArray): string | undefined { + const encoded = args.at(-1); + if (encoded === undefined) return undefined; + try { + const decoded = JSON.parse(Buffer.from(encoded, "base64url").toString("utf8")) as { + command?: unknown; + }; + return typeof decoded.command === "string" ? decoded.command : undefined; + } catch { + return undefined; + } +} + +/** + * Whether a spawn models a long-running daemon (postgres, postgrest, …) that + * stays alive until it is explicitly killed, versus a short-lived process + * (health-check probes like `pg_isready`, one-shot init scripts, docker + * commands) that exits on its own. + * + * This distinction is essential for fidelity to real processes. Real daemons do + * NOT exit ~immediately after spawning; if the mock had them exit after a fixed + * delay (as it once did), the orchestrator's `unless-stopped`/`always` restart + * loop would treat every daemon as a crash-looping process and churn through + * `RestartTriggered` → backoff → respawn cycles forever. That never happens in + * production (daemons run until stopped), so the projected status would never + * settle on Running/Healthy — an artifact that only the mock could produce. + * + * A supervised spawn (launched through `makeSupervisedCommand`, i.e. the + * supervisor runtime with a base64url payload) is treated as a long-running + * daemon UNLESS its inner command is a shell (`bash`/`sh`), which is how + * one-shot init services like `postgres-init` (restart: "no") are launched — + * those must still exit so their `completed` signal fires. + */ +function isLongRunningDaemon(args: ReadonlyArray): boolean { + const inner = decodeSupervisedInnerCommand(args); + if (inner === undefined) return false; + const base = inner.split("/").pop() ?? inner; + return base !== "bash" && base !== "sh"; +} + export function mockChildProcessSpawner( opts: { exitCode?: number; @@ -33,16 +79,23 @@ export function mockChildProcessSpawner( const exitDeferred = yield* Deferred.make(); let running = true; - yield* Effect.forkDetach( - Effect.gen(function* () { - yield* Effect.sleep("10 millis"); - running = false; - yield* Deferred.succeed( - exitDeferred, - ChildProcessSpawner.ExitCode(opts.exitCode ?? 0), - ); - }), - ); + const longRunning = isLongRunningDaemon(args); + + // Long-running daemons stay alive until killed (matching real processes); + // short-lived processes (probes, one-shot init scripts, docker commands) + // resolve their exit code after a real 10ms sleep so `it.live` clocks progress. + if (!longRunning) { + yield* Effect.forkDetach( + Effect.gen(function* () { + yield* Effect.sleep("10 millis"); + running = false; + yield* Deferred.succeed( + exitDeferred, + ChildProcessSpawner.ExitCode(opts.exitCode ?? 0), + ); + }), + ); + } const stdoutBytes = (opts.stdout ?? []).map((line) => encoder.encode(`${line}\n`)); const stderrBytes = (opts.stderr ?? []).map((line) => encoder.encode(`${line}\n`)); @@ -56,9 +109,12 @@ export function mockChildProcessSpawner( isRunning: Effect.sync(() => running), stdin: Sink.drain, kill: (killOpts) => - Effect.sync(() => { + Effect.gen(function* () { killed.push(killOpts?.killSignal ?? "SIGTERM"); running = false; + // Resolve the exit code so callers awaiting it (the orchestrator's + // restart/stop paths) unblock — a killed process reports 143 (128 + SIGTERM). + yield* Deferred.succeed(exitDeferred, ChildProcessSpawner.ExitCode(143)); }), unref: Effect.succeed(Effect.void), getInputFd: () => Sink.drain, diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts index f7b08a0e7a..06aebf4f4e 100644 --- a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -20,6 +20,22 @@ import { DEFAULT_VERSIONS } from "./versions.ts"; const testJwtSecret = "super-secret-jwt-token-with-at-least-32-characters-long"; +/** + * The coordinator's projected status is fed by an async stream, so it lags the + * orchestrator's internal readiness signal that waitReady/waitAllReady resolve on — + * on slow machines it can transiently read "Starting" (still catching up) or even + * "Restarting" (a flaky health probe briefly failed). Only its EVENTUAL value is + * contractual, so poll until it settles instead of asserting a snapshot. + */ +const waitForReadyStatus = (getState: Effect.Effect<{ readonly status: string }, E, R>) => + Effect.gen(function* () { + for (;;) { + const state = yield* getState; + if (state.status === "Running" || state.status === "Healthy") return; + yield* Effect.sleep(Duration.millis(25)); + } + }).pipe(Effect.timeout(Duration.seconds(5))); + const defaultPorts: AllocatedPorts = { apiPort: 54321, dbPort: 54322, @@ -229,12 +245,8 @@ describe("StackLifecycleCoordinator lazyServices", () => { expect(spawner.spawned.some(isPostgrestPayload)).toBe(true); // waitReady already proved the service reached a ready state — that's the actual - // assertion above. The coordinator's projected status here is best-effort: health - // checks poll on an interval, so it may still lag behind (or even show a transient - // "Restarting", if a flaky health check briefly failed and triggered a restart) - // relative to the orchestrator's internal readiness signal that waitReady resolves on. - const readyState = yield* stack.getState("postgrest"); - expect(["Running", "Healthy", "Restarting"]).toContain(readyState.status); + // assertion above. The projected status only settles eventually; see helper doc. + yield* waitForReadyStatus(stack.getState("postgrest")); yield* Effect.promise(() => fakeServer.stop()); }).pipe(Effect.provide(layer)); @@ -292,8 +304,9 @@ describe("StackLifecycleCoordinator lazyServices", () => { yield* stack.waitAllReady().pipe(Effect.timeout(Duration.seconds(5))); - const postgrestState = yield* stack.getState("postgrest"); - expect(["Running", "Healthy"]).toContain(postgrestState.status); + // waitAllReady covering the on-demand service means it reaches ready; the + // projected status only settles eventually; see helper doc. + yield* waitForReadyStatus(stack.getState("postgrest")); yield* Effect.promise(() => fakeServer.stop()); }).pipe(Effect.provide(layer)); From 9fc3f471e7b02410723645b641669093ddb17add Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 21:15:33 +0200 Subject: [PATCH 30/38] chore(cli): address micro stack review feedback --- packages/fleet/package.json | 7 +- packages/fleet/src/EdgeProxy.ts | 16 ++- packages/fleet/src/Fleet.ts | 130 ++++++++++++------ packages/fleet/src/IdleMonitor.ts | 1 + packages/fleet/src/PodManifest.ts | 20 ++- packages/fleet/src/PodManifest.unit.test.ts | 13 +- packages/fleet/src/PodRegistry.ts | 15 +- packages/fleet/src/PodRegistry.unit.test.ts | 15 ++ packages/fleet/src/PortRegistry.ts | 94 ++++++++----- packages/fleet/src/PortRegistry.unit.test.ts | 25 ++++ .../src/TemplateStore.integration.test.ts | 3 +- packages/fleet/src/TemplateStore.ts | 46 ++++--- packages/fleet/src/reapStalePostmaster.ts | 94 ++++++++++--- .../src/reapStalePostmaster.unit.test.ts | 23 +++- packages/fleet/tsconfig.json | 6 +- packages/process-compose/src/Orchestrator.ts | 13 ++ .../src/Orchestrator.unit.test.ts | 37 +++++ packages/stack/src/ApiProxy.ts | 28 +++- packages/stack/src/ApiProxy.unit.test.ts | 24 ++++ .../stack/src/StackLifecycleCoordinator.ts | 11 +- .../StackLifecycleCoordinator.unit.test.ts | 3 +- packages/stack/src/index.ts | 7 +- packages/stack/src/layers.ts | 1 + packages/stack/src/lazyServices.ts | 11 +- packages/stack/src/lazyServices.unit.test.ts | 4 +- packages/stack/src/pgconf.ts | 15 ++ packages/stack/src/pgconf.unit.test.ts | 18 ++- packages/stack/src/services/postgres.ts | 6 +- pnpm-lock.yaml | 6 + 29 files changed, 530 insertions(+), 162 deletions(-) create mode 100644 packages/fleet/src/PodRegistry.unit.test.ts diff --git a/packages/fleet/package.json b/packages/fleet/package.json index 2238ec2824..558faa47a8 100644 --- a/packages/fleet/package.json +++ b/packages/fleet/package.json @@ -7,7 +7,7 @@ ".": "./src/index.ts" }, "scripts": { - "test": "nx run-many -t test:unit --projects=$npm_package_name", + "test": "nx run-many -t test:unit test:integration --projects=$npm_package_name", "check:all": "nx run-many -t types:check lint:check fmt:check knip:check --projects=$npm_package_name", "fix:all": "nx run-many -t lint:fix fmt:fix knip:fix --projects=$npm_package_name" }, @@ -17,10 +17,12 @@ "devDependencies": { "@tsconfig/bun": "catalog:", "@types/bun": "catalog:", + "@typescript/native-preview": "catalog:", "@vitest/coverage-istanbul": "catalog:", "knip": "catalog:", "oxfmt": "catalog:", "oxlint": "catalog:", + "oxlint-tsgolint": "catalog:", "vitest": "catalog:" }, "knip": { @@ -32,8 +34,7 @@ "@typescript/native-preview", "oxfmt", "oxlint", - "oxlint-tsgolint", - "@tsconfig/bun" + "oxlint-tsgolint" ], "ignoreBinaries": [ "nx" diff --git a/packages/fleet/src/EdgeProxy.ts b/packages/fleet/src/EdgeProxy.ts index e9c12280fc..9e91ea6760 100644 --- a/packages/fleet/src/EdgeProxy.ts +++ b/packages/fleet/src/EdgeProxy.ts @@ -167,8 +167,20 @@ export class EdgeProxy { this.registrations.set(podId, { server, sockets }); return new Promise((resolve, reject) => { - server.once("error", reject); - server.listen(listenPort, "127.0.0.1", () => resolve()); + const onError = (error: Error) => { + this.registrations.delete(podId); + for (const sock of sockets) sock.destroy(); + try { + server.close(() => reject(error)); + } catch { + reject(error); + } + }; + server.once("error", onError); + server.listen(listenPort, "127.0.0.1", () => { + server.off("error", onError); + resolve(); + }); }); } diff --git a/packages/fleet/src/Fleet.ts b/packages/fleet/src/Fleet.ts index e3119f2685..f99d996ea5 100644 --- a/packages/fleet/src/Fleet.ts +++ b/packages/fleet/src/Fleet.ts @@ -48,10 +48,10 @@ interface WarmPod { readonly internalDbPort: number; } -// Matches the supabase CLI local-dev convention (see packages/stack -// services/postgres.ts POSTGRES_PASSWORD default) — a template-built -// postgres data dir only accepts this password. -const DB_PASSWORD = "postgres"; +// Must match packages/stack/src/services/postgres.ts: the template build stores +// this password in the data dir, and pod connection URLs need to use the same +// value when exposing the suspended/warm pod. +const DB_PASSWORD = process.env.POSTGRES_PASSWORD ?? "postgres"; // Internal (in-process stack) ports are derived from the externally-visible, // PortRegistry-owned port by a fixed +10_000 offset so the two ranges never @@ -118,6 +118,7 @@ export async function createFleet(opts: FleetOptions = {}): Promise const warm = new Map(); const wakesInFlight = new Map>(); const podLocks = new PodLock(); + let disposed = false; const monitor = new IdleMonitor({ idleMs, @@ -137,7 +138,23 @@ export async function createFleet(opts: FleetOptions = {}): Promise const runPidFile = (id: string): string => join(pods.podDir(id), "run.pid"); + async function withPodLocks(ids: ReadonlyArray, body: () => Promise): Promise { + const ordered = [...new Set(ids)].sort((a, b) => a.localeCompare(b)); + const acquire = (index: number): Promise => + index >= ordered.length + ? body() + : podLocks.withLock(ordered[index]!, () => acquire(index + 1)); + return acquire(0); + } + + async function rollbackProvisionedPod(id: string): Promise { + await proxy.unregister(id).catch(() => {}); + await provisioner.destroy(id).catch(() => {}); + states.delete(id); + } + async function wakeUpstream(id: string): Promise { + if (disposed) throw new Error("fleet is disposed"); const existing = warm.get(id); if (existing) return { host: "127.0.0.1", port: existing.internalDbPort }; // Dedup deliberately stays OUTSIDE podLocks: EdgeProxy may invoke wake() @@ -162,33 +179,42 @@ export async function createFleet(opts: FleetOptions = {}): Promise provisioned: true, profile: "micro", }, - postgrest: manifest.services.postgrest === true ? {} : false, - auth: manifest.services.auth === true ? {} : false, - realtime: manifest.services.realtime === true ? {} : false, - edgeRuntime: manifest.services["edge-runtime"] === true ? {} : false, - storage: manifest.services.storage === true ? {} : false, - imgproxy: manifest.services.imgproxy === true ? {} : false, - mailpit: manifest.services.mailpit === true ? {} : false, - pgmeta: manifest.services.pgmeta === true ? {} : false, - studio: manifest.services.studio === true ? {} : false, - analytics: manifest.services.analytics === true ? {} : false, - vector: manifest.services.vector === true ? {} : false, - pooler: manifest.services.pooler === true ? {} : false, + postgrest: + manifest.services.postgrest === true ? { version: manifest.versions.postgrest } : false, + auth: manifest.services.auth === true ? { version: manifest.versions.auth } : false, + realtime: + manifest.services.realtime === true ? { version: manifest.versions.realtime } : false, + edgeRuntime: + manifest.services["edge-runtime"] === true + ? { version: manifest.versions["edge-runtime"] } + : false, + storage: + manifest.services.storage === true ? { version: manifest.versions.storage } : false, + imgproxy: + manifest.services.imgproxy === true ? { version: manifest.versions.imgproxy } : false, + mailpit: + manifest.services.mailpit === true ? { version: manifest.versions.mailpit } : false, + pgmeta: manifest.services.pgmeta === true ? { version: manifest.versions.pgmeta } : false, + studio: manifest.services.studio === true ? { version: manifest.versions.studio } : false, + analytics: + manifest.services.analytics === true ? { version: manifest.versions.analytics } : false, + vector: manifest.services.vector === true ? { version: manifest.versions.vector } : false, + pooler: manifest.services.pooler === true ? { version: manifest.versions.pooler } : false, functions: false, }); try { await stack.start(); await stack.serviceReady("postgres"); + await writeFile(runPidFile(id), String(process.pid)); + warm.set(id, { stack, internalDbPort }); + states.set(id, "warm"); + monitor.track(id); + monitor.recordActivity(id, proxy.openConnections(id)); + return { host: "127.0.0.1", port: internalDbPort }; } catch (err) { await stack.dispose().catch(() => {}); throw err; } - warm.set(id, { stack, internalDbPort }); - states.set(id, "warm"); - monitor.track(id); - monitor.recordActivity(id, proxy.openConnections(id)); - await writeFile(runPidFile(id), String(process.pid)); - return { host: "127.0.0.1", port: internalDbPort }; }) .catch((err: unknown) => { states.set(id, "suspended"); @@ -260,9 +286,16 @@ export async function createFleet(opts: FleetOptions = {}): Promise const handle: FleetHandle = { async createPod(opts) { - const manifest = await provisioner.create(opts); - await registerEdge(manifest); - return status(manifest); + return podLocks.withLock(opts.id, async () => { + const manifest = await provisioner.create(opts); + try { + await registerEdge(manifest); + return status(manifest); + } catch (err) { + await rollbackProvisionedPod(manifest.id); + throw err; + } + }); }, async destroyPod(id) { // suspend (tears down live processes) and provisioner.destroy (deletes @@ -284,14 +317,21 @@ export async function createFleet(opts: FleetOptions = {}): Promise }); }, async forkPod(sourceId, newId) { + if (sourceId === newId) throw new Error(`pod already exists: ${newId}`); // Lock the SOURCE pod around suspend + the fork's clone of its data - // dir (provisioner.fork reads sourceId's data dir). The new pod needs - // no lock: it isn't live yet, nothing else can race it. - const manifest = await podLocks.withLock(sourceId, async () => { + // dir (provisioner.fork reads sourceId's data dir), and lock the TARGET + // id so concurrent creates/forks cannot delete each other's results. + const manifest = await withPodLocks([sourceId, newId], async () => { await suspendLocked(sourceId); - return provisioner.fork(sourceId, newId); + const forked = await provisioner.fork(sourceId, newId); + try { + await registerEdge(forked); + return forked; + } catch (err) { + await rollbackProvisionedPod(forked.id); + throw err; + } }); - await registerEdge(manifest); return status(manifest); }, async wake(id) { @@ -311,26 +351,30 @@ export async function createFleet(opts: FleetOptions = {}): Promise // wake. Callers that need a non-preload extension enabled on a // currently-suspended pod must `wake()` it first. async enableExtension(id, extension) { - const pod = warm.get(id); - if (pod) { - await pod.stack.enableExtension(extension); - return; - } - const manifest = await pods.read(id); - if (manifest === undefined) throw new Error(`unknown pod: ${id}`); - if (!PRELOAD_REQUIRED_EXTENSIONS.has(extension)) return; - const libs = await readPreloadLibraries(pods.dataDir(id)); - if (!libs.includes(extension)) { - await writePreloadLibraries(pods.dataDir(id), [...libs, extension]); - } + await podLocks.withLock(id, async () => { + const pod = warm.get(id); + if (pod) { + await pod.stack.enableExtension(extension); + return; + } + const manifest = await pods.read(id); + if (manifest === undefined) throw new Error(`unknown pod: ${id}`); + if (!PRELOAD_REQUIRED_EXTENSIONS.has(extension)) return; + const libs = await readPreloadLibraries(pods.dataDir(id)); + if (!libs.includes(extension)) { + await writePreloadLibraries(pods.dataDir(id), [...libs, extension]); + } + }); }, async listPods() { const manifests = await pods.list(); return Promise.all(manifests.map((m) => status(m))); }, async dispose() { - for (const id of warm.keys()) await suspend(id); + disposed = true; await proxy.close(); + await Promise.allSettled(wakesInFlight.values()); + for (const id of warm.keys()) await suspend(id); }, async [Symbol.asyncDispose]() { await handle.dispose(); diff --git a/packages/fleet/src/IdleMonitor.ts b/packages/fleet/src/IdleMonitor.ts index 90ec09ab90..bbd5b4ca45 100644 --- a/packages/fleet/src/IdleMonitor.ts +++ b/packages/fleet/src/IdleMonitor.ts @@ -64,5 +64,6 @@ export class IdleMonitor { this.tracked.delete(podId); this.opts.onIdle(podId); }, this.opts.idleMs); + entry.timer.unref?.(); } } diff --git a/packages/fleet/src/PodManifest.ts b/packages/fleet/src/PodManifest.ts index b406f0bd9a..bdb2c892af 100644 --- a/packages/fleet/src/PodManifest.ts +++ b/packages/fleet/src/PodManifest.ts @@ -10,11 +10,19 @@ export interface PodManifest { readonly createdAt: string; } -export const baseTemplateKey = (postgresVersion: string): string => `pg-${postgresVersion}`; +const keyHash = (value: string): string => + createHash("sha256").update(value).digest("hex").slice(0, 16); -export const templateKey = (versions: Partial): string => { - const canonical = JSON.stringify( - Object.fromEntries(Object.entries(versions).sort(([a], [b]) => a.localeCompare(b))), - ); - return `tuple-${createHash("sha256").update(canonical).digest("hex").slice(0, 16)}`; +export const baseTemplateKey = (postgresVersion: string): string => + `pg-${keyHash(postgresVersion)}`; + +export const templateKey = ( + versions: Partial, + enabledServices: ReadonlyArray = [], +): string => { + const canonical = JSON.stringify({ + versions: Object.fromEntries(Object.entries(versions).sort(([a], [b]) => a.localeCompare(b))), + enabledServices: [...new Set(enabledServices)].sort((a, b) => a.localeCompare(b)), + }); + return `tuple-${keyHash(canonical)}`; }; diff --git a/packages/fleet/src/PodManifest.unit.test.ts b/packages/fleet/src/PodManifest.unit.test.ts index 95224d0009..659a67db7d 100644 --- a/packages/fleet/src/PodManifest.unit.test.ts +++ b/packages/fleet/src/PodManifest.unit.test.ts @@ -12,7 +12,16 @@ describe("templateKey", () => { templateKey({ postgres: "17.6.1.143", auth: "2.192.1" }), ); }); - it("base key is human-readable", () => { - expect(baseTemplateKey("17.6.1.143")).toBe("pg-17.6.1.143"); + it("includes enabled services in canonical order", () => { + const versions = { postgres: "17.6.1.143", auth: "2.192.0" }; + expect(templateKey(versions, ["auth", "postgrest"])).toBe( + templateKey(versions, ["postgrest", "auth"]), + ); + expect(templateKey(versions, ["auth"])).not.toBe(templateKey(versions, ["auth", "postgrest"])); + }); + it("base key is stable and path-safe", () => { + expect(baseTemplateKey("17.6.1.143")).toBe(baseTemplateKey("17.6.1.143")); + expect(baseTemplateKey("../17.6.1.143")).toMatch(/^pg-[a-f0-9]{16}$/); + expect(baseTemplateKey("../17.6.1.143")).not.toContain(".."); }); }); diff --git a/packages/fleet/src/PodRegistry.ts b/packages/fleet/src/PodRegistry.ts index 5909f3fbc7..af6f766b4b 100644 --- a/packages/fleet/src/PodRegistry.ts +++ b/packages/fleet/src/PodRegistry.ts @@ -1,7 +1,16 @@ import { mkdir, readdir, readFile, rm, writeFile } from "node:fs/promises"; -import { join } from "node:path"; +import { basename, join } from "node:path"; import type { PodManifest } from "./PodManifest.ts"; +const POD_ID_RE = /^[A-Za-z0-9][A-Za-z0-9._-]*$/; + +function validatePodId(id: string): string { + if (!POD_ID_RE.test(id) || basename(id) !== id) { + throw new Error(`invalid pod id: ${id}`); + } + return id; +} + /** * Persists pod manifests on disk, one per pod directory: `podsRoot//pod.json`. * The pod's data directory lives alongside it at `podsRoot//data`. @@ -10,11 +19,11 @@ export class PodRegistry { constructor(private readonly podsRoot: string) {} podDir(id: string): string { - return join(this.podsRoot, id); + return join(this.podsRoot, validatePodId(id)); } dataDir(id: string): string { - return join(this.podsRoot, id, "data"); + return join(this.podDir(id), "data"); } async read(id: string): Promise { diff --git a/packages/fleet/src/PodRegistry.unit.test.ts b/packages/fleet/src/PodRegistry.unit.test.ts new file mode 100644 index 0000000000..ede01df1ea --- /dev/null +++ b/packages/fleet/src/PodRegistry.unit.test.ts @@ -0,0 +1,15 @@ +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { PodRegistry } from "./PodRegistry.ts"; + +describe("PodRegistry", () => { + it("rejects ids that could escape the pod root", async () => { + const pods = new PodRegistry(await mkdtemp(join(tmpdir(), "pods-"))); + + expect(() => pods.podDir("../templates")).toThrow(/invalid pod id/); + expect(() => pods.dataDir("nested/pod")).toThrow(/invalid pod id/); + expect(() => pods.podDir("pod-a")).not.toThrow(); + }); +}); diff --git a/packages/fleet/src/PortRegistry.ts b/packages/fleet/src/PortRegistry.ts index e54cec0000..bddea84f01 100644 --- a/packages/fleet/src/PortRegistry.ts +++ b/packages/fleet/src/PortRegistry.ts @@ -46,8 +46,13 @@ function isValidState(value: unknown): value is PortState { * ports are also duplicated in each pod's own manifest (`pod.json`), the * daemon is expected to re-seed the registry after such a reset by calling * `restore()` for each known pod. + * - **In-process serialization.** Mutating operations are queued so concurrent + * lifecycle calls in the owner process cannot interleave writes to the shared + * temporary state file. */ export class PortRegistry { + private mutationQueue: Promise = Promise.resolve(); + private constructor( private readonly stateFile: string, private state: PortState, @@ -84,19 +89,21 @@ export class PortRegistry { } async allocate(podId: string): Promise { - const existing = this.state.pods[podId]; - if (existing) return existing; - const used = new Set(Object.values(this.state.pods).flatMap((p) => [p.dbPort, p.apiPort])); - let candidate = this.state.basePort; - const next = (): number => { - while (used.has(candidate)) candidate += 1; - used.add(candidate); - return candidate; - }; - const ports: PodPorts = { dbPort: next(), apiPort: next() }; - this.state = { ...this.state, pods: { ...this.state.pods, [podId]: ports } }; - await this.persist(); - return ports; + return this.withMutation(async () => { + const existing = this.state.pods[podId]; + if (existing) return existing; + const used = new Set(Object.values(this.state.pods).flatMap((p) => [p.dbPort, p.apiPort])); + let candidate = this.state.basePort; + const next = (): number => { + while (used.has(candidate)) candidate += 1; + used.add(candidate); + return candidate; + }; + const ports: PodPorts = { dbPort: next(), apiPort: next() }; + this.state = { ...this.state, pods: { ...this.state.pods, [podId]: ports } }; + await this.persist(); + return ports; + }); } /** @@ -107,35 +114,54 @@ export class PortRegistry { * pod. */ async restore(podId: string, ports: PodPorts): Promise { - const existing = this.state.pods[podId]; - if (existing) { - if (existing.dbPort === ports.dbPort && existing.apiPort === ports.apiPort) { - return; - } - throw new Error( - `PortRegistry: cannot restore pod "${podId}" with ports ${JSON.stringify(ports)}; ` + - `it is already recorded with different ports ${JSON.stringify(existing)}`, - ); - } - - for (const [otherPodId, otherPorts] of Object.entries(this.state.pods)) { - if (otherPorts.dbPort === ports.dbPort || otherPorts.apiPort === ports.apiPort) { + await this.withMutation(async () => { + const existing = this.state.pods[podId]; + if (existing) { + if (existing.dbPort === ports.dbPort && existing.apiPort === ports.apiPort) { + return; + } throw new Error( `PortRegistry: cannot restore pod "${podId}" with ports ${JSON.stringify(ports)}; ` + - `port already assigned to pod "${otherPodId}"`, + `it is already recorded with different ports ${JSON.stringify(existing)}`, ); } - } - this.state = { ...this.state, pods: { ...this.state.pods, [podId]: ports } }; - await this.persist(); + const restoredPorts = new Set([ports.dbPort, ports.apiPort]); + for (const [otherPodId, otherPorts] of Object.entries(this.state.pods)) { + if (restoredPorts.has(otherPorts.dbPort) || restoredPorts.has(otherPorts.apiPort)) { + throw new Error( + `PortRegistry: cannot restore pod "${podId}" with ports ${JSON.stringify(ports)}; ` + + `port already assigned to pod "${otherPodId}"`, + ); + } + } + + this.state = { ...this.state, pods: { ...this.state.pods, [podId]: ports } }; + await this.persist(); + }); } async release(podId: string): Promise { - const rest = { ...this.state.pods }; - delete rest[podId]; - this.state = { ...this.state, pods: rest }; - await this.persist(); + await this.withMutation(async () => { + const rest = { ...this.state.pods }; + delete rest[podId]; + this.state = { ...this.state, pods: rest }; + await this.persist(); + }); + } + + private async withMutation(body: () => Promise): Promise { + const previous = this.mutationQueue; + let release: () => void = () => {}; + this.mutationQueue = new Promise((resolve) => { + release = resolve; + }); + await previous; + try { + return await body(); + } finally { + release(); + } } private async persist(): Promise { diff --git a/packages/fleet/src/PortRegistry.unit.test.ts b/packages/fleet/src/PortRegistry.unit.test.ts index 539859022b..ba70f8f23f 100644 --- a/packages/fleet/src/PortRegistry.unit.test.ts +++ b/packages/fleet/src/PortRegistry.unit.test.ts @@ -29,6 +29,22 @@ describe("PortRegistry", () => { expect(c.dbPort).toBe(a1.dbPort); // freed ports are reusable }); + it("serializes concurrent mutations before persisting", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + + const pods = Array.from({ length: 20 }, (_, index) => `pod-${index}`); + await Promise.all(pods.map((pod) => reg.allocate(pod))); + await Promise.all(pods.slice(0, 10).map((pod) => reg.release(pod))); + await Promise.all(pods.slice(0, 10).map((pod) => reg.allocate(`${pod}-new`))); + + const reloaded = await PortRegistry.load(file); + const restored = [...pods.slice(10), ...pods.slice(0, 10).map((pod) => `${pod}-new`)].map( + (pod) => reloaded.get(pod), + ); + expect(restored.every((ports) => ports !== undefined)).toBe(true); + }); + it("quarantines a corrupt state file and loads with fresh empty state", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const garbage = "{not valid json at all"; @@ -122,5 +138,14 @@ describe("PortRegistry", () => { await expect(reg.restore("pod-b", { dbPort: 55010, apiPort: 55012 })).rejects.toThrow(); await expect(reg.restore("pod-b", { dbPort: 55020, apiPort: 55011 })).rejects.toThrow(); }); + + it("throws if a restored db port collides with another pod's api port", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + + await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); + await expect(reg.restore("pod-b", { dbPort: 55011, apiPort: 55012 })).rejects.toThrow(); + await expect(reg.restore("pod-b", { dbPort: 55012, apiPort: 55010 })).rejects.toThrow(); + }); }); }); diff --git a/packages/fleet/src/TemplateStore.integration.test.ts b/packages/fleet/src/TemplateStore.integration.test.ts index c33260aee4..06f1c4172a 100644 --- a/packages/fleet/src/TemplateStore.integration.test.ts +++ b/packages/fleet/src/TemplateStore.integration.test.ts @@ -2,6 +2,7 @@ import { mkdtemp, readFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; +import { baseTemplateKey } from "./PodManifest.ts"; import { TemplateStore } from "./TemplateStore.ts"; // Requires postgres binaries in the local cache; opt-in via env, since the first @@ -13,7 +14,7 @@ describe.skipIf(!process.env.FLEET_PG_TESTS)("TemplateStore", () => { const root = await mkdtemp(join(tmpdir(), "templates-")); const store = new TemplateStore(root); const first = await store.ensureBaseTemplate(POSTGRES_VERSION); - expect(first).toContain(`pg-${POSTGRES_VERSION}`); + expect(first).toContain(baseTemplateKey(POSTGRES_VERSION)); // PGDATA got the micro profile const conf = await readFile(join(first, "postgresql.conf"), "utf8"); expect(conf).toContain("include_if_exists = 'micro.conf'"); diff --git a/packages/fleet/src/TemplateStore.ts b/packages/fleet/src/TemplateStore.ts index 639cdf1701..f150b9744a 100644 --- a/packages/fleet/src/TemplateStore.ts +++ b/packages/fleet/src/TemplateStore.ts @@ -8,6 +8,11 @@ import { baseTemplateKey, templateKey } from "./PodManifest.ts"; const LOCK_STALE_MS = 10 * 60 * 1000; const LOCK_POLL_MS = 250; +function errorCode(error: unknown): string | undefined { + if (typeof error !== "object" || error === null || !("code" in error)) return undefined; + return typeof error.code === "string" ? error.code : undefined; +} + /** * Builds and caches golden Postgres template data directories that pods are later * CoW-cloned from. @@ -15,7 +20,7 @@ const LOCK_POLL_MS = 250; * - "base" templates are a one-shot `postgres`-only stack boot: `postgres-init` * applies the baseline roles/schemas/migrations exactly as it does for a normal * stack, then the micro profile (`micro.conf`/`pod.conf` includes) is installed - * before the data dir is frozen under `root/pg-/data`. + * before the data dir is frozen under `root/pg-/data`. * - "warm" templates clone a base template and boot the requested services once * so each self-migrates against a `provisioned` (post-init) data dir, then * freeze the result under `root//data`. An empty service list has @@ -86,7 +91,7 @@ export class TemplateStore { const base = await this.ensureBaseTemplate(pgVersion); if (enabledServices.length === 0) return base; - const key = templateKey(versions); + const key = templateKey(versions, enabledServices); if (await this.has(key)) return this.dataDir(key); return this.withLock(key, async () => { if (await this.has(key)) return this.dataDir(key); @@ -105,18 +110,20 @@ export class TemplateStore { provisioned: true, profile: "micro", }, - postgrest: enabledServices.includes("postgrest") ? {} : false, - auth: enabledServices.includes("auth") ? {} : false, - edgeRuntime: enabledServices.includes("edge-runtime") ? {} : false, - realtime: enabledServices.includes("realtime") ? {} : false, - storage: enabledServices.includes("storage") ? {} : false, - imgproxy: enabledServices.includes("imgproxy") ? {} : false, - mailpit: enabledServices.includes("mailpit") ? {} : false, - pgmeta: enabledServices.includes("pgmeta") ? {} : false, - studio: enabledServices.includes("studio") ? {} : false, - analytics: enabledServices.includes("analytics") ? {} : false, - vector: enabledServices.includes("vector") ? {} : false, - pooler: enabledServices.includes("pooler") ? {} : false, + postgrest: enabledServices.includes("postgrest") ? { version: versions.postgrest } : false, + auth: enabledServices.includes("auth") ? { version: versions.auth } : false, + edgeRuntime: enabledServices.includes("edge-runtime") + ? { version: versions["edge-runtime"] } + : false, + realtime: enabledServices.includes("realtime") ? { version: versions.realtime } : false, + storage: enabledServices.includes("storage") ? { version: versions.storage } : false, + imgproxy: enabledServices.includes("imgproxy") ? { version: versions.imgproxy } : false, + mailpit: enabledServices.includes("mailpit") ? { version: versions.mailpit } : false, + pgmeta: enabledServices.includes("pgmeta") ? { version: versions.pgmeta } : false, + studio: enabledServices.includes("studio") ? { version: versions.studio } : false, + analytics: enabledServices.includes("analytics") ? { version: versions.analytics } : false, + vector: enabledServices.includes("vector") ? { version: versions.vector } : false, + pooler: enabledServices.includes("pooler") ? { version: versions.pooler } : false, functions: false, }); try { @@ -136,10 +143,10 @@ export class TemplateStore { } private async freeze(buildDir: string, key: string, marker: unknown): Promise { - await mkdir(join(this.root, key), { recursive: true }); - await rename(join(buildDir, "data"), this.dataDir(key)); - await writeFile(join(this.root, key, "template.json"), JSON.stringify(marker)); - await rm(buildDir, { recursive: true, force: true }); + const finalDir = join(this.root, key); + await writeFile(join(buildDir, "template.json"), JSON.stringify(marker)); + await rm(finalDir, { recursive: true, force: true }); + await rename(buildDir, finalDir); } private async withLock(key: string, body: () => Promise): Promise { @@ -150,7 +157,8 @@ export class TemplateStore { const handle = await open(lockPath, "wx"); await handle.close(); break; - } catch { + } catch (error) { + if (errorCode(error) !== "EEXIST") throw error; const s = await stat(lockPath).catch(() => undefined); if (s && Date.now() - s.mtimeMs > LOCK_STALE_MS) { await unlink(lockPath).catch(() => {}); diff --git a/packages/fleet/src/reapStalePostmaster.ts b/packages/fleet/src/reapStalePostmaster.ts index 0ca4f8e04d..22e081b7db 100644 --- a/packages/fleet/src/reapStalePostmaster.ts +++ b/packages/fleet/src/reapStalePostmaster.ts @@ -1,6 +1,72 @@ -import { readFile } from "node:fs/promises"; +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import { readFile, realpath } from "node:fs/promises"; import { join } from "node:path"; +const execFileAsync = promisify(execFile); +const TERM_TIMEOUT_MS = 5000; +const POLL_MS = 100; + +function isAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} + +async function commandForPid(pid: number): Promise { + const procCmdline = await readFile(`/proc/${pid}/cmdline`, "utf8") + .then((raw) => raw.split("\u0000").join(" ").trim()) + .catch(() => undefined); + if (procCmdline !== undefined && procCmdline !== "") return procCmdline; + + return execFileAsync("ps", ["-p", String(pid), "-o", "command="]) + .then(({ stdout }) => stdout.trim()) + .catch(() => undefined); +} + +async function isPostmasterForDataDir(pid: number, dataDir: string, raw: string): Promise { + const lines = raw.split("\n"); + const recordedDataDir = lines[1]?.trim(); + if (recordedDataDir === undefined || recordedDataDir === "") return false; + + const [actualDir, recordedDir] = await Promise.all([ + realpath(dataDir).catch(() => dataDir), + realpath(recordedDataDir).catch(() => recordedDataDir), + ]); + if (actualDir !== recordedDir) return false; + + const command = await commandForPid(pid); + return ( + command !== undefined && + command.includes("postgres") && + (command.includes(actualDir) || command.includes(recordedDataDir) || command.includes(dataDir)) + ); +} + +async function waitUntilExited(pid: number, timeoutMs: number): Promise { + const start = Date.now(); + while (isAlive(pid)) { + if (Date.now() - start >= timeoutMs) return false; + await new Promise((resolve) => setTimeout(resolve, POLL_MS)); + } + return true; +} + +function signalPostmaster(pid: number, signal: NodeJS.Signals): void { + try { + process.kill(-pid, signal); + } catch { + try { + process.kill(pid, signal); + } catch { + /* already gone */ + } + } +} + /** * Reaps a stale postmaster (and its backends) left running under `dataDir` * by a previous, no-longer-live daemon process. @@ -37,25 +103,11 @@ export async function reapStalePostmaster(dataDir: string): Promise { } if (!alive) return; - try { - process.kill(-pid, "SIGTERM"); - } catch { - try { - process.kill(pid, "SIGTERM"); - } catch { - /* already gone */ - } - } + if (!(await isPostmasterForDataDir(pid, dataDir, raw))) return; - setTimeout(() => { - try { - process.kill(-pid, "SIGKILL"); - } catch { - try { - process.kill(pid, "SIGKILL"); - } catch { - /* already gone */ - } - } - }, 5000).unref(); + signalPostmaster(pid, "SIGTERM"); + if (await waitUntilExited(pid, TERM_TIMEOUT_MS)) return; + + signalPostmaster(pid, "SIGKILL"); + await waitUntilExited(pid, TERM_TIMEOUT_MS); } diff --git a/packages/fleet/src/reapStalePostmaster.unit.test.ts b/packages/fleet/src/reapStalePostmaster.unit.test.ts index c48957c71a..793377050c 100644 --- a/packages/fleet/src/reapStalePostmaster.unit.test.ts +++ b/packages/fleet/src/reapStalePostmaster.unit.test.ts @@ -43,13 +43,16 @@ describe("reapStalePostmaster", () => { const dir = await mkdtemp(join(tmpdir(), "reap-test-")); dirs.push(dir); - const child = spawn("sleep", ["60"], { detached: true, stdio: "ignore" }); + const child = spawn("bash", ["-c", `exec -a "postgres -D ${dir}" sleep 60`], { + detached: true, + stdio: "ignore", + }); child.unref(); const pid = child.pid; if (pid === undefined) throw new Error("failed to spawn dummy process"); pidsToCleanUp.push(pid); - await writeFile(join(dir, "postmaster.pid"), `${pid}\n/some/data/dir\n1234567\n5432\n`); + await writeFile(join(dir, "postmaster.pid"), `${pid}\n${dir}\n1234567\n5432\n`); expect(isAlive(pid)).toBe(true); await reapStalePostmaster(dir); @@ -78,4 +81,20 @@ describe("reapStalePostmaster", () => { // Sanity: we're still alive (obviously true, but documents intent). expect(isAlive(process.pid)).toBe(true); }); + + it("refuses to kill a reused pid that is not this postmaster", async () => { + const dir = await mkdtemp(join(tmpdir(), "reap-test-")); + dirs.push(dir); + + const child = spawn("sleep", ["60"], { detached: true, stdio: "ignore" }); + child.unref(); + const pid = child.pid; + if (pid === undefined) throw new Error("failed to spawn dummy process"); + pidsToCleanUp.push(pid); + + await writeFile(join(dir, "postmaster.pid"), `${pid}\n${dir}\n1234567\n5432\n`); + + await reapStalePostmaster(dir); + expect(isAlive(pid)).toBe(true); + }); }); diff --git a/packages/fleet/tsconfig.json b/packages/fleet/tsconfig.json index eef2f2a863..ba396eb057 100644 --- a/packages/fleet/tsconfig.json +++ b/packages/fleet/tsconfig.json @@ -1,7 +1,3 @@ { - "extends": "@tsconfig/bun/tsconfig.json", - "compilerOptions": { - "lib": ["ESNext", "DOM"], - "types": ["bun"] - } + "extends": "@tsconfig/bun/tsconfig.json" } diff --git a/packages/process-compose/src/Orchestrator.ts b/packages/process-compose/src/Orchestrator.ts index ad24308ffa..3e0c698f60 100644 --- a/packages/process-compose/src/Orchestrator.ts +++ b/packages/process-compose/src/Orchestrator.ts @@ -530,9 +530,16 @@ export class Orchestrator extends Context.Service< const restartClosureFor = (name: string): ReadonlyArray => { const names = new Set([name]); + const shouldRestartDependent = (dependentName: string): boolean => { + const svc = services.get(dependentName); + if (svc === undefined) return false; + const status = SubscriptionRef.getUnsafe(svc.state).status; + return status !== "Pending" && status !== "Stopped"; + }; const collectDependents = (current: string): void => { for (const dependent of graph.dependentsOf(current)) { if (names.has(dependent.name)) continue; + if (!shouldRestartDependent(dependent.name)) continue; names.add(dependent.name); collectDependents(dependent.name); } @@ -613,6 +620,12 @@ export class Orchestrator extends Context.Service< } const order = graph.startOrderFor(name); for (const d of order) { + const svc = services.get(d.name); + const status = + svc === undefined ? undefined : SubscriptionRef.getUnsafe(svc.state).status; + if (status === "Stopped" || status === "Failed") { + yield* resetService(d.name); + } yield* FiberMap.run(fibers, d.name, runServiceSafe(d), { onlyIfMissing: true }); } }), diff --git a/packages/process-compose/src/Orchestrator.unit.test.ts b/packages/process-compose/src/Orchestrator.unit.test.ts index 22a45f266b..ddf55f40d6 100644 --- a/packages/process-compose/src/Orchestrator.unit.test.ts +++ b/packages/process-compose/src/Orchestrator.unit.test.ts @@ -511,6 +511,21 @@ describe("Orchestrator", () => { }).pipe(Effect.provide(layer), Effect.scoped); }); + it.live("startService restarts a service that was stopped", () => { + const { layer, proc } = setupOrchestrator([svc("a")], { + exitDelay: "5 seconds", + }); + return Effect.gen(function* () { + const orc = yield* Orchestrator; + yield* orc.startService("a"); + yield* proc.waitForSpawnCount(1); + yield* orc.stopService("a"); + yield* orc.startService("a"); + yield* proc.waitForSpawnCount(2); + expect(proc.spawned.map((s) => s.command)).toEqual(["a", "a"]); + }).pipe(Effect.provide(layer), Effect.scoped); + }); + it.live("restartService stops and restarts a service", () => { const { layer, proc } = setupOrchestrator([svc("a")], { exitDelay: "5 seconds", @@ -563,6 +578,28 @@ describe("Orchestrator", () => { }).pipe(Effect.provide(layer), Effect.scoped); }); + it.live("restartService skips transitive dependents that were never started", () => { + const { layer, proc } = setupOrchestrator( + [ + svc("db"), + svc("api", { + dependencies: [{ service: "db", condition: "started" }], + }), + ], + { exitDelay: "5 seconds" }, + ); + return Effect.gen(function* () { + const orc = yield* Orchestrator; + yield* orc.startService("db"); + yield* proc.waitForSpawnCount(1); + + yield* orc.restartService("db"); + yield* proc.waitForSpawnCount(2); + + expect(proc.spawned.map((s) => s.command)).toEqual(["db", "db"]); + }).pipe(Effect.provide(layer), Effect.scoped); + }); + it.live("updateServiceDefinition restarts with the updated definition", () => { const { layer, proc } = setupOrchestrator([svc("a")], { exitDelay: "5 seconds", diff --git a/packages/stack/src/ApiProxy.ts b/packages/stack/src/ApiProxy.ts index 837f826789..a91945c7b6 100644 --- a/packages/stack/src/ApiProxy.ts +++ b/packages/stack/src/ApiProxy.ts @@ -23,6 +23,7 @@ export interface ProxyConfig { readonly analyticsPort: number; readonly poolerPort: number; readonly studioPort: number; + readonly imgproxyEnabled?: boolean; readonly publishableKey: string; readonly secretKey: string; readonly anonJwt: string; @@ -122,6 +123,7 @@ const COLD_START_RETRY_SCHEDULE = Schedule.spaced("250 millis").pipe(Schedule.ta interface ProxyHandlerOptions { readonly backendPort: number; readonly service: ServiceName; + readonly additionalServices?: ReadonlyArray; readonly stripPrefix?: string; readonly backendPath?: string; readonly transformAuth?: boolean; @@ -141,14 +143,15 @@ function makeProxyHandler( return (req: HttpServerRequest.HttpServerRequest) => Effect.gen(function* () { if (config.ensureService) { - const ensured = yield* Effect.result( - Effect.tryPromise(() => config.ensureService!(opts.service)), - ); - if (Result.isFailure(ensured)) { - return HttpServerResponse.text( - `Bad gateway: failed to start ${opts.service}: ${String(ensured.failure)}`, - { status: 502 }, + for (const service of [opts.service, ...(opts.additionalServices ?? [])]) { + const ensured = yield* Effect.result( + Effect.tryPromise(() => config.ensureService!(service)), ); + if (Result.isFailure(ensured)) { + return HttpServerResponse.text(`Bad gateway: failed to start ${service}`, { + status: 502, + }); + } } } @@ -353,6 +356,17 @@ export class ApiProxy extends Context.Service< stripPrefix: "/storage/v1", }), ), + HttpRouter.route( + "*", + "/storage/v1/render/image/*", + makeProxyHandler(client, config, { + backendPort: config.storagePort, + service: "storage", + additionalServices: config.imgproxyEnabled === true ? ["imgproxy"] : [], + stripPrefix: "/storage/v1", + transformAuth: true, + }), + ), HttpRouter.route( "*", "/storage/v1/*", diff --git a/packages/stack/src/ApiProxy.unit.test.ts b/packages/stack/src/ApiProxy.unit.test.ts index d5f8873d18..b2dc875b44 100644 --- a/packages/stack/src/ApiProxy.unit.test.ts +++ b/packages/stack/src/ApiProxy.unit.test.ts @@ -546,6 +546,30 @@ describe("ApiProxy", () => { try { const res = await fetch(`${proxy.url}/rest/v1/users`); expect(res.status).toBe(502); + expect(await res.text()).toBe("Bad gateway: failed to start postgrest"); + } finally { + await proxy.dispose(); + await backend.stop(); + } + }); + + test("starts imgproxy before transformed storage requests when enabled", async () => { + const backend = await startEchoBackend(); + const calls: string[] = []; + const config: ProxyConfig = { + ...configForPort(backend.port), + imgproxyEnabled: true, + ensureService: async (name) => { + calls.push(name); + }, + }; + const proxy = await startProxy(config); + try { + const res = await fetch(`${proxy.url}/storage/v1/render/image/public/bucket/image.png`, { + headers: { apikey: PUBLISHABLE_KEY }, + }); + expect(res.status).toBe(200); + expect(calls).toEqual(["storage", "imgproxy"]); } finally { await proxy.dispose(); await backend.stop(); diff --git a/packages/stack/src/StackLifecycleCoordinator.ts b/packages/stack/src/StackLifecycleCoordinator.ts index 8dcd01fb93..4cd89ba27d 100644 --- a/packages/stack/src/StackLifecycleCoordinator.ts +++ b/packages/stack/src/StackLifecycleCoordinator.ts @@ -20,7 +20,7 @@ import { planEnableExtension } from "./enableExtension.ts"; import { StackBuildError } from "./errors.ts"; import { configureFunctionsRuntime, type FunctionsConfig } from "./functions.ts"; import { detectPlatform, dockerHostAddress } from "./Platform.ts"; -import { readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; +import { installPodConfOverlay, readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; import { StackMetadataPersistence } from "./StackMetadataPersistence.ts"; import { StackPreparation } from "./StackPreparation.ts"; import type { PreparedStackArtifacts } from "./StackPreparation.ts"; @@ -248,6 +248,12 @@ export class StackLifecycleCoordinator extends Context.Service< const startedServicesRef = yield* Ref.make>(new Set()); const markStarted = (names: Iterable) => Ref.update(startedServicesRef, (current) => new Set([...current, ...names])); + const markStopped = (names: Iterable) => + Ref.update(startedServicesRef, (current) => { + const next = new Set(current); + for (const name of names) next.delete(name); + return next; + }); const logBufferServices = yield* Layer.buildWithScope(LogBuffer.layer, scope); const logBuffer = Context.get(logBufferServices, LogBuffer); @@ -605,6 +611,7 @@ export class StackLifecycleCoordinator extends Context.Service< } yield* Ref.set(phaseRef, "stopping"); yield* runtimeState.orchestrator.stop(); + yield* Ref.set(startedServicesRef, new Set()); yield* Ref.set(phaseRef, "stopped"); }), dispose: disposeOnce, @@ -621,6 +628,7 @@ export class StackLifecycleCoordinator extends Context.Service< yield* requireKnownService(name); const runtime = yield* ensureRuntime; yield* runtime.orchestrator.stopService(name); + yield* markStopped([name]); }), restartService: (name) => Effect.gen(function* () { @@ -635,6 +643,7 @@ export class StackLifecycleCoordinator extends Context.Service< enableExtension: (name) => enableExtensionLock.withPermits(1)( Effect.gen(function* () { + yield* Effect.promise(() => installPodConfOverlay(config.postgres.dataDir)); const currentLibraries = yield* Effect.promise(() => readPreloadLibraries(config.postgres.dataDir), ); diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts index 06aebf4f4e..b4e51fa63e 100644 --- a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -1,4 +1,4 @@ -import { mkdtempSync, rmSync } from "node:fs"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; import * as http from "node:http"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -181,6 +181,7 @@ describe("StackLifecycleCoordinator enableExtension", () => { // `Effect.sleep("10 millis")`, which needs the real clock to progress. it.live("serializes concurrent enableExtension calls so no write is lost", () => { const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-test-")); + writeFileSync(join(dataDir, "postgresql.conf"), "# stock conf\n"); const config = makeConfig(dataDir); const { layer } = setupLayer(config); diff --git a/packages/stack/src/index.ts b/packages/stack/src/index.ts index 98aa5e4d09..53cb417850 100644 --- a/packages/stack/src/index.ts +++ b/packages/stack/src/index.ts @@ -26,5 +26,10 @@ export type { PrefetchOptions, PrefetchResult } from "./prefetch.ts"; export type { ReadyOptions, StackHandle } from "./createStack.ts"; export type { FunctionsConfig, FunctionsRuntimeConfig } from "./functions.ts"; export { UnixHttpClient, UnixHttpClientError } from "./UnixHttpClient.ts"; -export { installMicroProfile, readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; +export { + installMicroProfile, + installPodConfOverlay, + readPreloadLibraries, + writePreloadLibraries, +} from "./pgconf.ts"; export { PRELOAD_REQUIRED_EXTENSIONS } from "./micro.ts"; diff --git a/packages/stack/src/layers.ts b/packages/stack/src/layers.ts index c1337fdf55..899e77e39a 100644 --- a/packages/stack/src/layers.ts +++ b/packages/stack/src/layers.ts @@ -38,6 +38,7 @@ const baseProxyConfig = (config: ResolvedStackConfig): Omit Promise, ): ((name: ServiceName) => Promise) => { const inFlight = new Map>(); - const done = new Set(); return (name) => { - if (done.has(name)) return Promise.resolve(); const existing = inFlight.get(name); if (existing) return existing; const p = start(name).then( () => { - done.add(name); inFlight.delete(name); }, (err: unknown) => { diff --git a/packages/stack/src/lazyServices.unit.test.ts b/packages/stack/src/lazyServices.unit.test.ts index f42d669d1f..2f5d1723db 100644 --- a/packages/stack/src/lazyServices.unit.test.ts +++ b/packages/stack/src/lazyServices.unit.test.ts @@ -23,7 +23,7 @@ describe("makeEnsureServiceMemo", () => { expect(attempt).toBe(2); }); - it("resolves immediately for a service already marked done", async () => { + it("checks again after a completed start", async () => { let starts = 0; const ensure = makeEnsureServiceMemo(async (_name) => { starts += 1; @@ -31,7 +31,7 @@ describe("makeEnsureServiceMemo", () => { await ensure("storage"); await ensure("storage"); await ensure("storage"); - expect(starts).toBe(1); + expect(starts).toBe(3); }); it("tracks each service name independently", async () => { diff --git a/packages/stack/src/pgconf.ts b/packages/stack/src/pgconf.ts index 77e5cb1333..a005bd5f49 100644 --- a/packages/stack/src/pgconf.ts +++ b/packages/stack/src/pgconf.ts @@ -13,6 +13,7 @@ const INCLUDE_BLOCK = [ // Line-anchored, active (non-commented) include directive only — a commented-out // line like `#include_if_exists = 'micro.conf'` must NOT count as installed. const ACTIVE_INCLUDE_RE = /^\s*include_if_exists = 'micro\.conf'/m; +const ACTIVE_POD_INCLUDE_RE = /^\s*include_if_exists = 'pod\.conf'/m; const PRELOAD_LIBRARIES_RE = /^\s*shared_preload_libraries\s*=\s*['"]([^'"]*)['"]/m; @@ -30,6 +31,20 @@ export async function installMicroProfile(pgdata: string): Promise { } } +export async function installPodConfOverlay(pgdata: string): Promise { + const podConf = join(pgdata, "pod.conf"); + const existing = await readFile(podConf, "utf8").catch(() => undefined); + if (existing === undefined) { + await writeFile(podConf, buildPodConf([])); + } + const mainPath = join(pgdata, "postgresql.conf"); + const main = await readFile(mainPath, "utf8"); + if (!ACTIVE_POD_INCLUDE_RE.test(main)) { + const separator = main.endsWith("\n") || main === "" ? "" : "\n"; + await writeFile(mainPath, `${main}${separator}include_if_exists = 'pod.conf'\n`); + } +} + export async function readPreloadLibraries(pgdata: string): Promise { const content = await readFile(join(pgdata, "pod.conf"), "utf8").catch(() => ""); const match = content.match(PRELOAD_LIBRARIES_RE); diff --git a/packages/stack/src/pgconf.unit.test.ts b/packages/stack/src/pgconf.unit.test.ts index 73d12e0740..408f99d9b0 100644 --- a/packages/stack/src/pgconf.unit.test.ts +++ b/packages/stack/src/pgconf.unit.test.ts @@ -2,7 +2,12 @@ import { mkdtemp, readFile, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; -import { installMicroProfile, readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; +import { + installMicroProfile, + installPodConfOverlay, + readPreloadLibraries, + writePreloadLibraries, +} from "./pgconf.ts"; async function fakePgdata(): Promise { const dir = await mkdtemp(join(tmpdir(), "pgconf-test-")); @@ -24,6 +29,17 @@ describe("pgconf", () => { expect(micro).toContain("shared_buffers = '16MB'"); }); + it("installs only the pod.conf overlay for default stacks", async () => { + const pgdata = await fakePgdata(); + await installPodConfOverlay(pgdata); + await installPodConfOverlay(pgdata); + + const main = await readFile(join(pgdata, "postgresql.conf"), "utf8"); + expect(main).toContain("include_if_exists = 'pod.conf'"); + expect(main).not.toContain("include_if_exists = 'micro.conf'"); + expect(main.match(/include_if_exists = 'pod\.conf'/g)).toHaveLength(1); + }); + it("round-trips preload libraries via pod.conf", async () => { const pgdata = await fakePgdata(); await installMicroProfile(pgdata); diff --git a/packages/stack/src/services/postgres.ts b/packages/stack/src/services/postgres.ts index 3a21cb3280..66e7f9989d 100644 --- a/packages/stack/src/services/postgres.ts +++ b/packages/stack/src/services/postgres.ts @@ -33,16 +33,18 @@ interface DockerPostgresOptions extends PostgresServiceOptions { readonly cleanupDataDirOnExit?: boolean; } +const POSTGRES_PASSWORD = process.env.POSTGRES_PASSWORD ?? "postgres"; + const postgresEnv = (opts: NativePostgresOptions): Record => ({ PGDATA: opts.dataDir, - POSTGRES_PASSWORD: "postgres", + POSTGRES_PASSWORD, DYLD_LIBRARY_PATH: `${opts.binPath}/lib`, LD_LIBRARY_PATH: `${opts.binPath}/lib`, TZDIR: "/var/db/timezone/zoneinfo", }); const postgresDockerEnv = (opts: DockerPostgresOptions): Record => ({ - POSTGRES_PASSWORD: "postgres", + POSTGRES_PASSWORD, JWT_SECRET: opts.jwtSecret, JWT_EXP: String(opts.jwtExpiry), }); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a34b0499ca..d62418845d 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -463,6 +463,9 @@ importers: '@types/bun': specifier: 'catalog:' version: 1.3.14 + '@typescript/native-preview': + specifier: 'catalog:' + version: 7.0.0-dev.20260629.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -475,6 +478,9 @@ importers: oxlint: specifier: 'catalog:' version: 1.73.0(oxlint-tsgolint@0.23.0) + oxlint-tsgolint: + specifier: 'catalog:' + version: 0.23.0 vitest: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) From 69b4bc810e23170fb6eadcdc67caa12b8dd73850 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Tue, 7 Jul 2026 22:00:55 +0200 Subject: [PATCH 31/38] chore(cli): address stack review follow-ups --- packages/fleet/src/Fleet.ts | 12 +++- packages/fleet/src/PodManifest.ts | 18 +++++- packages/fleet/src/PodManifest.unit.test.ts | 10 +++- packages/fleet/src/PortRegistry.ts | 21 +++++-- packages/fleet/src/PortRegistry.unit.test.ts | 19 ++++++ packages/fleet/src/Provisioner.ts | 7 ++- packages/fleet/src/Provisioner.unit.test.ts | 20 ++++++- packages/fleet/src/TemplateStore.ts | 43 +++++++++----- .../process-compose/tests/helpers/mocks.ts | 40 ++++++++----- packages/stack/src/Stack.unit.test.ts | 1 + .../src/StackBuilder.provisioned.unit.test.ts | 21 +++++-- packages/stack/src/StackBuilder.ts | 15 +++++ packages/stack/src/StackBuilder.unit.test.ts | 1 + .../stack/src/StackLifecycleCoordinator.ts | 20 ++++++- .../StackLifecycleCoordinator.unit.test.ts | 24 ++++++++ packages/stack/src/createStack.ts | 2 + packages/stack/src/functions.ts | 9 ++- packages/stack/src/index.ts | 2 + packages/stack/src/postgresCredentials.ts | 14 +++++ packages/stack/src/services/analytics.ts | 12 +++- packages/stack/src/services/auth.ts | 10 +++- packages/stack/src/services/pgmeta.ts | 3 +- packages/stack/src/services/pooler.ts | 13 +++- packages/stack/src/services/postgres-init.ts | 52 ++++++++-------- packages/stack/src/services/postgres.ts | 35 ++++++++--- packages/stack/src/services/postgrest.ts | 10 +++- packages/stack/src/services/realtime.ts | 3 +- .../stack/src/services/services.unit.test.ts | 59 ++++++++++++++++++- packages/stack/src/services/storage.ts | 10 +++- packages/stack/src/services/studio.ts | 3 +- packages/stack/tests/helpers/buildServices.ts | 1 + 31 files changed, 414 insertions(+), 96 deletions(-) create mode 100644 packages/stack/src/postgresCredentials.ts diff --git a/packages/fleet/src/Fleet.ts b/packages/fleet/src/Fleet.ts index f99d996ea5..18867772d5 100644 --- a/packages/fleet/src/Fleet.ts +++ b/packages/fleet/src/Fleet.ts @@ -3,8 +3,10 @@ import { homedir } from "node:os"; import { join } from "node:path"; import { createStack, + postgresConnectionUrl, PRELOAD_REQUIRED_EXTENSIONS, readPreloadLibraries, + resolvePostgresPassword, writePreloadLibraries, type StackHandle, } from "@supabase/stack"; @@ -51,7 +53,7 @@ interface WarmPod { // Must match packages/stack/src/services/postgres.ts: the template build stores // this password in the data dir, and pod connection URLs need to use the same // value when exposing the suspended/warm pod. -const DB_PASSWORD = process.env.POSTGRES_PASSWORD ?? "postgres"; +const DB_PASSWORD = resolvePostgresPassword(); // Internal (in-process stack) ports are derived from the externally-visible, // PortRegistry-owned port by a fixed +10_000 offset so the two ranges never @@ -134,7 +136,13 @@ export async function createFleet(opts: FleetOptions = {}): Promise }); const dbUrl = (manifest: PodManifest): string => - `postgresql://postgres:${DB_PASSWORD}@127.0.0.1:${manifest.ports.dbPort}/postgres`; + postgresConnectionUrl({ + user: "postgres", + password: DB_PASSWORD, + host: "127.0.0.1", + port: manifest.ports.dbPort, + database: "postgres", + }); const runPidFile = (id: string): string => join(pods.podDir(id), "run.pid"); diff --git a/packages/fleet/src/PodManifest.ts b/packages/fleet/src/PodManifest.ts index bdb2c892af..d82c019afa 100644 --- a/packages/fleet/src/PodManifest.ts +++ b/packages/fleet/src/PodManifest.ts @@ -1,5 +1,9 @@ import { createHash } from "node:crypto"; -import type { ServiceName, VersionManifest } from "@supabase/stack"; +import { + fillServiceVersionManifest, + type ServiceName, + type VersionManifest, +} from "@supabase/stack"; export interface PodManifest { readonly id: string; @@ -26,3 +30,15 @@ export const templateKey = ( }); return `tuple-${keyHash(canonical)}`; }; + +export const resolveTemplateVersions = ( + versions: Partial, + enabledServices: ReadonlyArray, +): Partial => { + const full = fillServiceVersionManifest(versions); + const resolved: Partial> = { postgres: full.postgres }; + for (const service of new Set(enabledServices)) { + resolved[service] = full[service]; + } + return resolved; +}; diff --git a/packages/fleet/src/PodManifest.unit.test.ts b/packages/fleet/src/PodManifest.unit.test.ts index 659a67db7d..754c8bf834 100644 --- a/packages/fleet/src/PodManifest.unit.test.ts +++ b/packages/fleet/src/PodManifest.unit.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from "vitest"; -import { baseTemplateKey, templateKey } from "./PodManifest.ts"; +import { DEFAULT_VERSIONS } from "@supabase/stack"; +import { baseTemplateKey, resolveTemplateVersions, templateKey } from "./PodManifest.ts"; describe("templateKey", () => { it("is stable across key order", () => { @@ -24,4 +25,11 @@ describe("templateKey", () => { expect(baseTemplateKey("../17.6.1.143")).toMatch(/^pg-[a-f0-9]{16}$/); expect(baseTemplateKey("../17.6.1.143")).not.toContain(".."); }); + + it("resolves default versions only for postgres and enabled services", () => { + expect(resolveTemplateVersions({ postgres: "17.6.1.143" }, ["postgrest"])).toEqual({ + postgres: "17.6.1.143", + postgrest: DEFAULT_VERSIONS.postgrest, + }); + }); }); diff --git a/packages/fleet/src/PortRegistry.ts b/packages/fleet/src/PortRegistry.ts index bddea84f01..767699b853 100644 --- a/packages/fleet/src/PortRegistry.ts +++ b/packages/fleet/src/PortRegistry.ts @@ -17,12 +17,23 @@ function freshState(): PortState { return { basePort: DEFAULT_BASE_PORT, pods: {} }; } +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function isPositiveInteger(value: unknown): value is number { + return typeof value === "number" && Number.isInteger(value) && value > 0; +} + function isValidState(value: unknown): value is PortState { - if (typeof value !== "object" || value === null || Array.isArray(value)) return false; - const candidate = value as Record; - const { basePort, pods } = candidate; - if (typeof basePort !== "number" || Number.isNaN(basePort)) return false; - if (typeof pods !== "object" || pods === null || Array.isArray(pods)) return false; + if (!isRecord(value)) return false; + const { basePort, pods } = value; + if (!isPositiveInteger(basePort)) return false; + if (!isRecord(pods)) return false; + for (const ports of Object.values(pods)) { + if (!isRecord(ports)) return false; + if (!isPositiveInteger(ports.dbPort) || !isPositiveInteger(ports.apiPort)) return false; + } return true; } diff --git a/packages/fleet/src/PortRegistry.unit.test.ts b/packages/fleet/src/PortRegistry.unit.test.ts index ba70f8f23f..34715f26a8 100644 --- a/packages/fleet/src/PortRegistry.unit.test.ts +++ b/packages/fleet/src/PortRegistry.unit.test.ts @@ -76,6 +76,25 @@ describe("PortRegistry", () => { expect(quarantined).toBe(badStructure); }); + it("quarantines saved pod entries with invalid port records", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const badStructure = JSON.stringify({ + basePort: 55000, + pods: { + "pod-a": { dbPort: 55010, apiPort: "55011" }, + }, + }); + await writeFile(file, badStructure); + + const reg = await PortRegistry.load(file); + expect(reg.get("pod-a")).toBeUndefined(); + const allocated = await reg.allocate("pod-a"); + expect(allocated).toEqual({ dbPort: 55000, apiPort: 55001 }); + + const quarantined = await readFile(`${file}.corrupt`, "utf8"); + expect(quarantined).toBe(badStructure); + }); + it("overwrites any previous quarantine file", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); await writeFile(`${file}.corrupt`, "old-quarantine"); diff --git a/packages/fleet/src/Provisioner.ts b/packages/fleet/src/Provisioner.ts index fac901c0a7..45b8bd0697 100644 --- a/packages/fleet/src/Provisioner.ts +++ b/packages/fleet/src/Provisioner.ts @@ -1,7 +1,7 @@ import { rm } from "node:fs/promises"; import type { ServiceName, VersionManifest } from "@supabase/stack"; import { cloneDir } from "./cowClone.ts"; -import type { PodManifest } from "./PodManifest.ts"; +import { resolveTemplateVersions, type PodManifest } from "./PodManifest.ts"; import type { PodRegistry } from "./PodRegistry.ts"; import type { PortRegistry } from "./PortRegistry.ts"; import type { TemplateStore } from "./TemplateStore.ts"; @@ -39,16 +39,17 @@ export class Provisioner { const enabled = Object.entries(opts.services ?? {}) .filter(([, on]) => on === true) .map(([name]) => name as ServiceName); + const resolvedVersions = resolveTemplateVersions(opts.versions, enabled); const template = opts.warm === true - ? await templates.ensureWarmTemplate(opts.versions, enabled) + ? await templates.ensureWarmTemplate(resolvedVersions, enabled) : await templates.ensureBaseTemplate(pgVersion); const allocated = await ports.allocate(opts.id); try { await cloneDir(template, pods.dataDir(opts.id)); const manifest: PodManifest = { id: opts.id, - versions: opts.versions, + versions: resolvedVersions, services: opts.services ?? {}, flags: { supautils: opts.flags?.supautils ?? false }, ports: allocated, diff --git a/packages/fleet/src/Provisioner.unit.test.ts b/packages/fleet/src/Provisioner.unit.test.ts index 9a37b23bd0..ad965c18fc 100644 --- a/packages/fleet/src/Provisioner.unit.test.ts +++ b/packages/fleet/src/Provisioner.unit.test.ts @@ -1,7 +1,7 @@ import { mkdir, mkdtemp, readdir, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import type { ServiceName, VersionManifest } from "@supabase/stack"; +import { DEFAULT_VERSIONS, type ServiceName, type VersionManifest } from "@supabase/stack"; import { describe, expect, it } from "vitest"; import { PodRegistry } from "./PodRegistry.ts"; import { PortRegistry } from "./PortRegistry.ts"; @@ -84,6 +84,24 @@ describe("Provisioner (unit, fake deps)", () => { // original pod's ports must still be intact. expect(ports.get("dup")).toEqual(first.ports); }); + + it("records resolved default versions for enabled warm services", async () => { + const templateDir = await mkdtemp(join(tmpdir(), "fleet-template-")); + await writeFile(join(templateDir, "PG_VERSION"), PG_VERSION); + const { p } = await makeHarness(templateDir); + + const manifest = await p.create({ + id: "warm", + versions: { postgres: PG_VERSION }, + services: { postgrest: true }, + warm: true, + }); + + expect(manifest.versions).toEqual({ + postgres: PG_VERSION, + postgrest: DEFAULT_VERSIONS.postgrest, + }); + }); }); describe("fork", () => { diff --git a/packages/fleet/src/TemplateStore.ts b/packages/fleet/src/TemplateStore.ts index f150b9744a..f6c12ad5b7 100644 --- a/packages/fleet/src/TemplateStore.ts +++ b/packages/fleet/src/TemplateStore.ts @@ -3,7 +3,7 @@ import { join } from "node:path"; import { createStack, installMicroProfile } from "@supabase/stack"; import type { ServiceName, VersionManifest } from "@supabase/stack"; import { cloneDir } from "./cowClone.ts"; -import { baseTemplateKey, templateKey } from "./PodManifest.ts"; +import { baseTemplateKey, resolveTemplateVersions, templateKey } from "./PodManifest.ts"; const LOCK_STALE_MS = 10 * 60 * 1000; const LOCK_POLL_MS = 250; @@ -91,7 +91,8 @@ export class TemplateStore { const base = await this.ensureBaseTemplate(pgVersion); if (enabledServices.length === 0) return base; - const key = templateKey(versions, enabledServices); + const resolvedVersions = resolveTemplateVersions(versions, enabledServices); + const key = templateKey(resolvedVersions, enabledServices); if (await this.has(key)) return this.dataDir(key); return this.withLock(key, async () => { if (await this.has(key)) return this.dataDir(key); @@ -110,20 +111,32 @@ export class TemplateStore { provisioned: true, profile: "micro", }, - postgrest: enabledServices.includes("postgrest") ? { version: versions.postgrest } : false, - auth: enabledServices.includes("auth") ? { version: versions.auth } : false, + postgrest: enabledServices.includes("postgrest") + ? { version: resolvedVersions.postgrest } + : false, + auth: enabledServices.includes("auth") ? { version: resolvedVersions.auth } : false, edgeRuntime: enabledServices.includes("edge-runtime") - ? { version: versions["edge-runtime"] } + ? { version: resolvedVersions["edge-runtime"] } + : false, + realtime: enabledServices.includes("realtime") + ? { version: resolvedVersions.realtime } + : false, + storage: enabledServices.includes("storage") + ? { version: resolvedVersions.storage } + : false, + imgproxy: enabledServices.includes("imgproxy") + ? { version: resolvedVersions.imgproxy } + : false, + mailpit: enabledServices.includes("mailpit") + ? { version: resolvedVersions.mailpit } + : false, + pgmeta: enabledServices.includes("pgmeta") ? { version: resolvedVersions.pgmeta } : false, + studio: enabledServices.includes("studio") ? { version: resolvedVersions.studio } : false, + analytics: enabledServices.includes("analytics") + ? { version: resolvedVersions.analytics } : false, - realtime: enabledServices.includes("realtime") ? { version: versions.realtime } : false, - storage: enabledServices.includes("storage") ? { version: versions.storage } : false, - imgproxy: enabledServices.includes("imgproxy") ? { version: versions.imgproxy } : false, - mailpit: enabledServices.includes("mailpit") ? { version: versions.mailpit } : false, - pgmeta: enabledServices.includes("pgmeta") ? { version: versions.pgmeta } : false, - studio: enabledServices.includes("studio") ? { version: versions.studio } : false, - analytics: enabledServices.includes("analytics") ? { version: versions.analytics } : false, - vector: enabledServices.includes("vector") ? { version: versions.vector } : false, - pooler: enabledServices.includes("pooler") ? { version: versions.pooler } : false, + vector: enabledServices.includes("vector") ? { version: resolvedVersions.vector } : false, + pooler: enabledServices.includes("pooler") ? { version: resolvedVersions.pooler } : false, functions: false, }); try { @@ -134,7 +147,7 @@ export class TemplateStore { } await this.freeze(buildDir, key, { key, - versions, + versions: resolvedVersions, enabledServices, builtAt: new Date().toISOString(), }); diff --git a/packages/process-compose/tests/helpers/mocks.ts b/packages/process-compose/tests/helpers/mocks.ts index ac771edfc5..a4a8bcc046 100644 --- a/packages/process-compose/tests/helpers/mocks.ts +++ b/packages/process-compose/tests/helpers/mocks.ts @@ -6,22 +6,32 @@ interface SpawnRecord { args: ReadonlyArray; } +interface SupervisorPayload { + readonly command: string; + readonly args: ReadonlyArray; +} + const encoder = new TextEncoder(); /** - * Decode the supervisor payload (base64url JSON in the last arg — see - * `makeSupervisedCommand`) to recover the underlying service command. Returns + * Decode the supervisor payload (base64url JSON in the last arg - see + * `makeSupervisedCommand`) to recover the underlying service command and args. Returns * `undefined` for non-supervised spawns (health-check probes, docker commands, * etc.) whose last arg is not a supervisor payload. */ -function decodeSupervisedInnerCommand(args: ReadonlyArray): string | undefined { +function decodeSupervisedPayload(args: ReadonlyArray): SupervisorPayload | undefined { const encoded = args.at(-1); if (encoded === undefined) return undefined; try { - const decoded = JSON.parse(Buffer.from(encoded, "base64url").toString("utf8")) as { - command?: unknown; - }; - return typeof decoded.command === "string" ? decoded.command : undefined; + const decoded: unknown = JSON.parse(Buffer.from(encoded, "base64url").toString("utf8")); + if (typeof decoded !== "object" || decoded === null || Array.isArray(decoded)) { + return undefined; + } + const command = "command" in decoded ? decoded.command : undefined; + const innerArgs = "args" in decoded ? decoded.args : undefined; + if (typeof command !== "string" || !Array.isArray(innerArgs)) return undefined; + if (!innerArgs.every((arg) => typeof arg === "string")) return undefined; + return { command, args: innerArgs }; } catch { return undefined; } @@ -43,15 +53,17 @@ function decodeSupervisedInnerCommand(args: ReadonlyArray): string | und * * A supervised spawn (launched through `makeSupervisedCommand`, i.e. the * supervisor runtime with a base64url payload) is treated as a long-running - * daemon UNLESS its inner command is a shell (`bash`/`sh`), which is how - * one-shot init services like `postgres-init` (restart: "no") are launched — - * those must still exit so their `completed` signal fires. + * daemon. The explicit exception is one-shot shell scripts (`bash -c` / `sh -c`) + * such as `postgres-init`, which must exit so their `completed` signal fires. + * Native Postgres is also launched through `bash`, but its first arg is the + * bundled `supabase-postgres-init.sh` wrapper and it stays alive like Postgres + * does in production. */ function isLongRunningDaemon(args: ReadonlyArray): boolean { - const inner = decodeSupervisedInnerCommand(args); - if (inner === undefined) return false; - const base = inner.split("/").pop() ?? inner; - return base !== "bash" && base !== "sh"; + const payload = decodeSupervisedPayload(args); + if (payload === undefined) return false; + const base = payload.command.split("/").pop() ?? payload.command; + return !((base === "bash" || base === "sh") && payload.args[0] === "-c"); } export function mockChildProcessSpawner( diff --git a/packages/stack/src/Stack.unit.test.ts b/packages/stack/src/Stack.unit.test.ts index 2fb2570294..6a397235fb 100644 --- a/packages/stack/src/Stack.unit.test.ts +++ b/packages/stack/src/Stack.unit.test.ts @@ -58,6 +58,7 @@ const defaultConfig: ResolvedStackConfig = { port: 54322, dataDir: "/tmp/supabase/data", version: DEFAULT_VERSIONS.postgres, + password: "postgres", autoExposeNewTables: true, }, postgrest: { diff --git a/packages/stack/src/StackBuilder.provisioned.unit.test.ts b/packages/stack/src/StackBuilder.provisioned.unit.test.ts index e118cb1622..2f16d1f7c3 100644 --- a/packages/stack/src/StackBuilder.provisioned.unit.test.ts +++ b/packages/stack/src/StackBuilder.provisioned.unit.test.ts @@ -1,3 +1,6 @@ +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { describe, expect, it } from "vitest"; import { buildServicesForTest } from "../tests/helpers/buildServices.ts"; @@ -12,12 +15,18 @@ describe("provisioned postgres", () => { expect(services.map((s) => s.name)).toContain("postgres-init"); }); - it("drops -c runtime args when profile is micro", async () => { - const services = await buildServicesForTest({ - postgres: { provisioned: true, profile: "micro" }, - }); - const pg = services.find((s) => s.name === "postgres"); - expect(pg?.args?.join(" ")).not.toContain("wal_level=logical"); + it("drops -c runtime args after the micro profile config is installed", async () => { + const dataDir = mkdtempSync(join(tmpdir(), "stack-micro-profile-")); + try { + writeFileSync(join(dataDir, "postgresql.conf"), "include_if_exists = 'micro.conf'\n"); + const services = await buildServicesForTest({ + postgres: { dataDir, provisioned: true, profile: "micro" }, + }); + const pg = services.find((s) => s.name === "postgres"); + expect(pg?.args?.join(" ")).not.toContain("wal_level=logical"); + } finally { + rmSync(dataDir, { recursive: true, force: true }); + } }); it("keeps -c runtime args on the default profile", async () => { diff --git a/packages/stack/src/StackBuilder.ts b/packages/stack/src/StackBuilder.ts index fad3e38147..b84c134d29 100644 --- a/packages/stack/src/StackBuilder.ts +++ b/packages/stack/src/StackBuilder.ts @@ -43,6 +43,7 @@ export interface PostgresConfig { readonly port?: number; readonly dataDir?: string; readonly version?: string; + readonly password?: string; /** * When true (default), the bundled initial schema GRANTs that expose new tables, views, * sequences, and functions in `public` to the Data API roles (`anon`, `authenticated`, @@ -189,6 +190,7 @@ export interface ResolvedPostgresConfig { readonly port: number; readonly dataDir: string; readonly version: string; + readonly password: string; readonly autoExposeNewTables: boolean; readonly provisioned?: boolean; readonly profile?: "default" | "micro"; @@ -591,6 +593,7 @@ export class StackBuilder extends Context.Service< binPath: postgresResolution.path, dataDir: config.postgres.dataDir, port: config.dbPort, + password: config.postgres.password, dockerAccessible: needsDockerAccess, cleanupDataDirOnExit: hasAutoManagedPath(config, config.postgres.dataDir), profile: config.postgres.profile, @@ -599,6 +602,7 @@ export class StackBuilder extends Context.Service< image: postgresResolution.image, dataDir: config.postgres.dataDir, port: config.dbPort, + password: config.postgres.password, networkArgs: dockerNetworkArgs(platform.os, [config.dbPort]), jwtSecret: config.jwtSecret, jwtExpiry: config.auth !== false ? config.auth.jwtExpiry : 3600, @@ -614,6 +618,7 @@ export class StackBuilder extends Context.Service< ...makePostgresInitService({ postgresDir: postgresResolution.path, dbPort: config.dbPort, + dbPassword: config.postgres.password, autoExposeNewTables: config.postgres.autoExposeNewTables, }), enabled: true, @@ -626,6 +631,7 @@ export class StackBuilder extends Context.Service< ? makePostgrestService({ binPath: postgrestResolution.path, dbPort: config.dbPort, + dbPassword: config.postgres.password, port: config.postgrest.port, schemas: config.postgrest.schemas, extraSearchPath: config.postgrest.extraSearchPath, @@ -636,6 +642,7 @@ export class StackBuilder extends Context.Service< image: postgrestResolution.image, dbHost: serviceHost, dbPort: config.dbPort, + dbPassword: config.postgres.password, port: config.postgrest.port, adminPort: config.postgrest.adminPort, schemas: config.postgrest.schemas, @@ -663,6 +670,7 @@ export class StackBuilder extends Context.Service< ? makeAuthServiceNative({ binPath: authResolution.path, dbPort: config.dbPort, + dbPassword: config.postgres.password, authPort: config.auth.port, siteUrl: config.auth.siteUrl, jwtSecret: config.jwtSecret, @@ -678,6 +686,7 @@ export class StackBuilder extends Context.Service< image: authResolution.image, dbHost: serviceHost, dbPort: config.dbPort, + dbPassword: config.postgres.password, authPort: config.auth.port, siteUrl: config.auth.siteUrl, jwtSecret: config.jwtSecret, @@ -751,6 +760,7 @@ export class StackBuilder extends Context.Service< apiPort: config.apiPort, dbHost: serviceHost, dbPort: config.dbPort, + dbPassword: config.postgres.password, jwtSecret: config.jwtSecret, jwtJwks, tenantId: config.realtime.tenantId, @@ -773,6 +783,7 @@ export class StackBuilder extends Context.Service< apiPort: config.apiPort, dbHost: serviceHost, dbPort: config.dbPort, + dbPassword: config.postgres.password, dataDir: config.storage.dataDir, anonKey: config.publishableKey, serviceKey: config.secretKey, @@ -816,6 +827,7 @@ export class StackBuilder extends Context.Service< port: config.pgmeta.port, dbHost: serviceHost, dbPort: config.dbPort, + dbPassword: config.postgres.password, networkArgs: dockerNetworkArgs(platform.os, [config.pgmeta.port]), dependencies: postgresDeps, }), @@ -839,6 +851,7 @@ export class StackBuilder extends Context.Service< nodeHost: analyticsRuntimeNetwork.nodeHost, dbHost: serviceHost, dbPort: config.dbPort, + dbPassword: config.postgres.password, apiKey: config.analytics.apiKey, backend: config.analytics.backend, networkArgs: dockerPortMapArgs(platform.os, [ @@ -876,6 +889,7 @@ export class StackBuilder extends Context.Service< hostAdminPort: config.pooler.apiPort, dbHost: serviceHost, dbPort: config.dbPort, + dbPassword: config.postgres.password, poolMode: config.pooler.mode, defaultPoolSize: config.pooler.defaultPoolSize, maxClientConn: config.pooler.maxClientConn, @@ -913,6 +927,7 @@ export class StackBuilder extends Context.Service< apiUrl: config.studio.apiUrl, publicApiUrl: `http://127.0.0.1:${config.apiPort}`, pgmetaUrl: pgmetaConfig === false ? "" : `http://${serviceHost}:${pgmetaConfig.port}`, + dbPassword: config.postgres.password, publishableKey: config.publishableKey, secretKey: config.secretKey, s3ProtocolAccessKeyId: LOCAL_S3_PROTOCOL_ACCESS_KEY_ID, diff --git a/packages/stack/src/StackBuilder.unit.test.ts b/packages/stack/src/StackBuilder.unit.test.ts index afe6a496e6..cfda4eeece 100644 --- a/packages/stack/src/StackBuilder.unit.test.ts +++ b/packages/stack/src/StackBuilder.unit.test.ts @@ -57,6 +57,7 @@ const baseConfig: ResolvedStackConfig = { port: 5432, dataDir: "/tmp/pg-data", version: DEFAULT_VERSIONS.postgres, + password: "postgres", autoExposeNewTables: true, }, postgrest: { diff --git a/packages/stack/src/StackLifecycleCoordinator.ts b/packages/stack/src/StackLifecycleCoordinator.ts index 4cd89ba27d..4d015fae53 100644 --- a/packages/stack/src/StackLifecycleCoordinator.ts +++ b/packages/stack/src/StackLifecycleCoordinator.ts @@ -20,6 +20,7 @@ import { planEnableExtension } from "./enableExtension.ts"; import { StackBuildError } from "./errors.ts"; import { configureFunctionsRuntime, type FunctionsConfig } from "./functions.ts"; import { detectPlatform, dockerHostAddress } from "./Platform.ts"; +import { postgresConnectionUrl } from "./postgresCredentials.ts"; import { installPodConfOverlay, readPreloadLibraries, writePreloadLibraries } from "./pgconf.ts"; import { StackMetadataPersistence } from "./StackMetadataPersistence.ts"; import { StackPreparation } from "./StackPreparation.ts"; @@ -111,7 +112,13 @@ const initialPublicStates = (config: ResolvedStackConfig): ReadonlyArray ({ url: `http://127.0.0.1:${config.apiPort}`, - dbUrl: `postgresql://postgres:postgres@127.0.0.1:${config.dbPort}/postgres`, + dbUrl: postgresConnectionUrl({ + user: "postgres", + password: config.postgres.password, + host: "127.0.0.1", + port: config.dbPort, + database: "postgres", + }), publishableKey: config.publishableKey, secretKey: config.secretKey, anonJwt: config.anonJwt, @@ -150,7 +157,13 @@ const stackInfoFor = (config: ResolvedStackConfig): StackInfo => ({ ...(config.pooler === false ? {} : { - pooler: `postgresql://postgres:postgres@127.0.0.1:${config.pooler.port}/postgres`, + pooler: postgresConnectionUrl({ + user: "postgres", + password: config.postgres.password, + host: "127.0.0.1", + port: config.pooler.port, + database: "postgres", + }), pooler_admin: `http://127.0.0.1:${config.pooler.apiPort}`, }), }, @@ -654,6 +667,9 @@ export class StackLifecycleCoordinator extends Context.Service< yield* Effect.promise(() => writePreloadLibraries(config.postgres.dataDir, plan.libraries), ); + if ((yield* Ref.get(phaseRef)) !== "running") { + return; + } yield* requireKnownService("postgres"); const runtime = yield* ensureRuntime; yield* runtime.orchestrator.restartService("postgres"); diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts index b4e51fa63e..04eb5af6f8 100644 --- a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -79,6 +79,7 @@ function makeConfig(dataDir: string): ResolvedStackConfig { port: 54322, dataDir, version: DEFAULT_VERSIONS.postgres, + password: "postgres", autoExposeNewTables: true, }, // postgrest/auth are disabled: their health checks are real HTTP probes @@ -202,6 +203,29 @@ describe("StackLifecycleCoordinator enableExtension", () => { Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), ); }); + + it.live("records preload libraries without restarting postgres while stopped", () => { + const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-stopped-test-")); + writeFileSync(join(dataDir, "postgresql.conf"), "# stock conf\n"); + const config = makeConfig(dataDir); + const { layer, spawner } = setupLayer(config); + + return Effect.gen(function* () { + const stack = yield* Stack; + yield* stack.start(); + yield* stack.stop(); + const spawnedBefore = spawner.spawned.length; + + yield* stack.enableExtension("pg_cron"); + + expect(spawner.spawned).toHaveLength(spawnedBefore); + expect((yield* stack.getState("postgres")).status).toBe("Stopped"); + expect(yield* Effect.promise(() => readPreloadLibraries(dataDir))).toEqual(["pg_cron"]); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), + ); + }); }); describe("StackLifecycleCoordinator lazyServices", () => { diff --git a/packages/stack/src/createStack.ts b/packages/stack/src/createStack.ts index a4538e17b9..b55e18c376 100644 --- a/packages/stack/src/createStack.ts +++ b/packages/stack/src/createStack.ts @@ -16,6 +16,7 @@ import { defaultSecretKey, generateJwt, } from "./JwtGenerator.ts"; +import { resolvePostgresPassword } from "./postgresCredentials.ts"; import { daemonLayer, foregroundLayer, @@ -561,6 +562,7 @@ export async function resolveConfig( port: ports.dbPort, dataDir: postgresDataDir, version: postgresInput.version ?? DEFAULT_VERSIONS.postgres, + password: resolvePostgresPassword(postgresInput.password), autoExposeNewTables: postgresInput.autoExposeNewTables ?? true, provisioned: postgresInput.provisioned, profile: postgresInput.profile, diff --git a/packages/stack/src/functions.ts b/packages/stack/src/functions.ts index 527a163b65..caaa0d7f5e 100644 --- a/packages/stack/src/functions.ts +++ b/packages/stack/src/functions.ts @@ -8,6 +8,7 @@ import { type ResolvedFunctionConfig, } from "@supabase/config"; import { Effect, FileSystem, Path, Redacted } from "effect"; +import { postgresConnectionUrl } from "./postgresCredentials.ts"; import type { ResolvedStackConfig } from "./StackBuilder.ts"; export interface FunctionsConfig { @@ -194,7 +195,13 @@ export const resolveFunctionsRuntimeConfig = Effect.fnUntraced(function* ( return { functionsUrl: `http://127.0.0.1:${stackConfig.apiPort}/functions/v1`, supabaseUrl: `http://${runtimeHost.hostname}:${stackConfig.apiPort}`, - dbUrl: `postgresql://postgres:postgres@${runtimeHost.hostname}:${stackConfig.dbPort}/postgres`, + dbUrl: postgresConnectionUrl({ + user: "postgres", + password: stackConfig.postgres.password, + host: runtimeHost.hostname, + port: stackConfig.dbPort, + database: "postgres", + }), publishableKey: stackConfig.publishableKey, secretKey: stackConfig.secretKey, jwtSecret: stackConfig.jwtSecret, diff --git a/packages/stack/src/index.ts b/packages/stack/src/index.ts index 53cb417850..ffd916d368 100644 --- a/packages/stack/src/index.ts +++ b/packages/stack/src/index.ts @@ -21,6 +21,8 @@ export type { } from "./StackBuilder.ts"; export type { ServiceName, VersionManifest } from "./versions.ts"; +export { DEFAULT_VERSIONS, fillServiceVersionManifest } from "./versions.ts"; +export { postgresConnectionUrl, resolvePostgresPassword } from "./postgresCredentials.ts"; export type { ServiceResolution } from "./resolve.ts"; export type { PrefetchOptions, PrefetchResult } from "./prefetch.ts"; export type { ReadyOptions, StackHandle } from "./createStack.ts"; diff --git a/packages/stack/src/postgresCredentials.ts b/packages/stack/src/postgresCredentials.ts new file mode 100644 index 0000000000..43fca3d11b --- /dev/null +++ b/packages/stack/src/postgresCredentials.ts @@ -0,0 +1,14 @@ +const DEFAULT_POSTGRES_PASSWORD = "postgres"; + +export const resolvePostgresPassword = (password?: string): string => + password ?? process.env.POSTGRES_PASSWORD ?? DEFAULT_POSTGRES_PASSWORD; + +export const postgresConnectionUrl = (opts: { + readonly scheme?: "postgresql" | "ecto"; + readonly user: string; + readonly password: string; + readonly host: string; + readonly port: number; + readonly database: string; +}): string => + `${opts.scheme ?? "postgresql"}://${opts.user}:${encodeURIComponent(opts.password)}@${opts.host}:${opts.port}/${opts.database}`; diff --git a/packages/stack/src/services/analytics.ts b/packages/stack/src/services/analytics.ts index 78c62f895e..383a4817f0 100644 --- a/packages/stack/src/services/analytics.ts +++ b/packages/stack/src/services/analytics.ts @@ -1,4 +1,5 @@ import type { ServiceDef } from "@supabase/process-compose"; +import { postgresConnectionUrl } from "../postgresCredentials.ts"; import { dockerRunService, type ServiceDependency } from "./service-utils.ts"; interface DockerAnalyticsOptions { @@ -9,6 +10,7 @@ interface DockerAnalyticsOptions { readonly nodeHost: string; readonly dbHost: string; readonly dbPort: number; + readonly dbPassword: string; readonly apiKey: string; readonly backend: "postgres" | "bigquery"; readonly networkArgs: ReadonlyArray; @@ -48,7 +50,7 @@ export const makeAnalyticsServiceDocker = (opts: DockerAnalyticsOptions): Servic DB_PORT: String(opts.dbPort), DB_SCHEMA: "_analytics", DB_USERNAME: "postgres", - DB_PASSWORD: "postgres", + DB_PASSWORD: opts.dbPassword, LOGFLARE_MIN_CLUSTER_SIZE: "1", LOGFLARE_SINGLE_TENANT: "true", LOGFLARE_SUPABASE_MODE: "true", @@ -60,7 +62,13 @@ export const makeAnalyticsServiceDocker = (opts: DockerAnalyticsOptions): Servic }; if (opts.backend === "postgres") { - env.POSTGRES_BACKEND_URL = `postgresql://postgres:postgres@${opts.dbHost}:${opts.dbPort}/_supabase`; + env.POSTGRES_BACKEND_URL = postgresConnectionUrl({ + user: "postgres", + password: opts.dbPassword, + host: opts.dbHost, + port: opts.dbPort, + database: "_supabase", + }); env.POSTGRES_BACKEND_SCHEMA = "_analytics"; } else { env.GOOGLE_DATASET_ID_APPEND = "_prod"; diff --git a/packages/stack/src/services/auth.ts b/packages/stack/src/services/auth.ts index b3e0360c83..803c496462 100644 --- a/packages/stack/src/services/auth.ts +++ b/packages/stack/src/services/auth.ts @@ -1,8 +1,10 @@ import type { ServiceDef } from "@supabase/process-compose"; +import { postgresConnectionUrl } from "../postgresCredentials.ts"; import { dockerServiceCleanup, dockerServiceOrphanCleanup } from "./docker-cleanup.ts"; interface AuthServiceOptions { readonly dbPort: number; + readonly dbPassword: string; readonly authPort: number; readonly siteUrl: string; readonly jwtSecret: string; @@ -30,7 +32,13 @@ interface DockerAuthOptions extends AuthServiceOptions { } const authEnv = (opts: AuthServiceOptions, dbHost = "127.0.0.1"): Record => ({ - GOTRUE_DB_DATABASE_URL: `postgresql://supabase_auth_admin:postgres@${dbHost}:${opts.dbPort}/postgres`, + GOTRUE_DB_DATABASE_URL: postgresConnectionUrl({ + user: "supabase_auth_admin", + password: opts.dbPassword, + host: dbHost, + port: opts.dbPort, + database: "postgres", + }), GOTRUE_DB_DRIVER: "postgres", GOTRUE_SITE_URL: opts.siteUrl, GOTRUE_JWT_SECRET: opts.jwtSecret, diff --git a/packages/stack/src/services/pgmeta.ts b/packages/stack/src/services/pgmeta.ts index 38c4fd282e..40c961eb8a 100644 --- a/packages/stack/src/services/pgmeta.ts +++ b/packages/stack/src/services/pgmeta.ts @@ -7,6 +7,7 @@ interface DockerPgmetaOptions { readonly port: number; readonly dbHost: string; readonly dbPort: number; + readonly dbPassword: string; readonly networkArgs: ReadonlyArray; readonly dependencies: ReadonlyArray; } @@ -36,7 +37,7 @@ export const makePgmetaServiceDocker = (opts: DockerPgmetaOptions): ServiceDef = PG_META_DB_NAME: "postgres", PG_META_DB_USER: "postgres", PG_META_DB_PORT: String(opts.dbPort), - PG_META_DB_PASSWORD: "postgres", + PG_META_DB_PASSWORD: opts.dbPassword, }, dependsOn: opts.dependencies, healthCheck: pgmetaHealthCheck(opts.port), diff --git a/packages/stack/src/services/pooler.ts b/packages/stack/src/services/pooler.ts index be75a7397b..aaa2f2a26e 100644 --- a/packages/stack/src/services/pooler.ts +++ b/packages/stack/src/services/pooler.ts @@ -1,4 +1,5 @@ import type { ServiceDef } from "@supabase/process-compose"; +import { postgresConnectionUrl } from "../postgresCredentials.ts"; import { dockerRunService, type ServiceDependency } from "./service-utils.ts"; type PoolMode = "transaction" | "session"; @@ -9,6 +10,7 @@ interface DockerPoolerOptions { readonly hostAdminPort: number; readonly dbHost: string; readonly dbPort: number; + readonly dbPassword: string; readonly poolMode: PoolMode; readonly defaultPoolSize: number; readonly maxClientConn: number; @@ -54,7 +56,7 @@ params = %{ "default_parameter_status" => %{"server_version" => version}, "users" => [%{ "db_user" => "pgbouncer", - "db_password" => "postgres", + "db_password" => ${JSON.stringify(opts.dbPassword)}, "mode_type" => "${opts.poolMode}", "pool_size" => ${opts.defaultPoolSize}, "is_manager" => true @@ -76,7 +78,14 @@ export const makePoolerServiceDocker = (opts: DockerPoolerOptions): ServiceDef = PORT: String(poolerContainerPorts.admin), PROXY_PORT_SESSION: String(poolerContainerPorts.session), PROXY_PORT_TRANSACTION: String(poolerContainerPorts.transaction), - DATABASE_URL: `ecto://postgres:postgres@${opts.dbHost}:${opts.dbPort}/_supabase`, + DATABASE_URL: postgresConnectionUrl({ + scheme: "ecto", + user: "postgres", + password: opts.dbPassword, + host: opts.dbHost, + port: opts.dbPort, + database: "_supabase", + }), CLUSTER_POSTGRES: "true", SECRET_KEY_BASE: opts.secretKeyBase, VAULT_ENC_KEY: opts.encryptionKey, diff --git a/packages/stack/src/services/postgres-init.ts b/packages/stack/src/services/postgres-init.ts index 694951230d..3565570329 100644 --- a/packages/stack/src/services/postgres-init.ts +++ b/packages/stack/src/services/postgres-init.ts @@ -3,6 +3,7 @@ import type { ServiceDef } from "@supabase/process-compose"; interface PostgresInitOptions { readonly postgresDir: string; readonly dbPort: number; + readonly dbPassword: string; /** * When false, append the SQL that Studio runs at cloud project creation to revoke the default * Data API privileges on the `public` schema so newly-created entities require explicit GRANTs. @@ -25,13 +26,17 @@ alter default privileges for role postgres in schema public revoke execute on functions from anon, authenticated, service_role; `.trim(); +function shellQuote(value: string): string { + return `'${value.replaceAll("'", "'\"'\"'")}'`; +} + export const makePostgresInitService = (opts: PostgresInitOptions): ServiceDef => { const pgBinDir = `${opts.postgresDir}/bin`; const pgLibDir = `${opts.postgresDir}/lib`; const migrationsDir = `${opts.postgresDir}/share/supabase-cli/migrations`; const psql = `${pgBinDir}/psql -h 127.0.0.1 -p ${opts.dbPort}`; - const psqlOpts = `-v ON_ERROR_STOP=1 --no-password --no-psqlrc`; + const psqlOpts = `-v ON_ERROR_STOP=1 --no-password --no-psqlrc -v pgpass="$PGPASSWORD"`; const revokeStep = opts.autoExposeNewTables ? "" @@ -48,7 +53,7 @@ EOSQL // postgres-init time from ~5s to ~1s. const script = ` export PATH="${pgBinDir}:$PATH" -export PGPASSWORD=postgres +export PGPASSWORD=${shellQuote(opts.dbPassword)} db="${migrationsDir}" # Check if already migrated (authenticator role created by initial-schema.sql) @@ -59,13 +64,9 @@ else # Create postgres role if missing (as supabase_admin) ${psql} ${psqlOpts} -U supabase_admin -d postgres <<'EOSQL' -do $$ -begin - if not exists (select from pg_roles where rolname = 'postgres') then - create role postgres superuser login password 'postgres'; - alter database postgres owner to postgres; - end if; -end $$ +SELECT format('CREATE ROLE postgres SUPERUSER LOGIN PASSWORD %L', :'pgpass') +WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'postgres')\\gexec +ALTER DATABASE postgres OWNER TO postgres; EOSQL # Run all init-scripts in a single psql session (as postgres) @@ -78,7 +79,9 @@ EOSQL fi # Set supabase_admin password (as postgres) - ${psql} ${psqlOpts} -U postgres -d postgres -c "ALTER USER supabase_admin WITH PASSWORD 'postgres'" + ${psql} ${psqlOpts} -U postgres -d postgres <<'EOSQL' +SELECT format('ALTER ROLE supabase_admin WITH PASSWORD %L', :'pgpass')\\gexec +EOSQL # Run all migrations in a single psql session (as supabase_admin) migrate_flags="" @@ -111,20 +114,19 @@ ALTER SCHEMA _supavisor OWNER TO postgres; EOSQL # Always update role passwords (idempotent) -${psql} -U supabase_admin -d postgres -c " -DO \\$\\$ -DECLARE - roles text[] := ARRAY['authenticator','supabase_auth_admin','supabase_storage_admin','supabase_functions_admin','supabase_replication_admin','supabase_read_only_user','postgres']; - r text; -BEGIN - FOREACH r IN ARRAY roles LOOP - IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = r) THEN - EXECUTE format('ALTER ROLE %I WITH PASSWORD ''postgres''', r); - END IF; - END LOOP; -END -\\$\\$; -" +${psql} ${psqlOpts} -U supabase_admin -d postgres <<'EOSQL' +SELECT format('ALTER ROLE %I WITH PASSWORD %L', rolname, :'pgpass') +FROM pg_roles +WHERE rolname = ANY (ARRAY[ + 'authenticator', + 'supabase_auth_admin', + 'supabase_storage_admin', + 'supabase_functions_admin', + 'supabase_replication_admin', + 'supabase_read_only_user', + 'postgres' +])\\gexec +EOSQL `; return { @@ -134,7 +136,7 @@ END env: { DYLD_LIBRARY_PATH: pgLibDir, LD_LIBRARY_PATH: pgLibDir, - PGPASSWORD: "postgres", + PGPASSWORD: opts.dbPassword, }, dependencies: [{ service: "postgres", condition: "healthy" }], supervision: {}, diff --git a/packages/stack/src/services/postgres.ts b/packages/stack/src/services/postgres.ts index 66e7f9989d..122882c396 100644 --- a/packages/stack/src/services/postgres.ts +++ b/packages/stack/src/services/postgres.ts @@ -1,4 +1,4 @@ -import { mkdirSync, writeFileSync } from "node:fs"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; import type { ServiceDef } from "@supabase/process-compose"; import { dockerServiceCleanup, @@ -9,6 +9,7 @@ import { interface PostgresServiceOptions { readonly dataDir: string; readonly port: number; + readonly password: string; readonly cleanupDataDirOnExit?: boolean; } @@ -33,18 +34,16 @@ interface DockerPostgresOptions extends PostgresServiceOptions { readonly cleanupDataDirOnExit?: boolean; } -const POSTGRES_PASSWORD = process.env.POSTGRES_PASSWORD ?? "postgres"; - const postgresEnv = (opts: NativePostgresOptions): Record => ({ PGDATA: opts.dataDir, - POSTGRES_PASSWORD, + POSTGRES_PASSWORD: opts.password, DYLD_LIBRARY_PATH: `${opts.binPath}/lib`, LD_LIBRARY_PATH: `${opts.binPath}/lib`, TZDIR: "/var/db/timezone/zoneinfo", }); const postgresDockerEnv = (opts: DockerPostgresOptions): Record => ({ - POSTGRES_PASSWORD, + POSTGRES_PASSWORD: opts.password, JWT_SECRET: opts.jwtSecret, JWT_EXP: String(opts.jwtExpiry), }); @@ -58,13 +57,26 @@ const NATIVE_POSTGRES_RUNTIME_ARGS = [ "max_replication_slots=5", ] as const; +const ACTIVE_MICRO_INCLUDE_RE = /^\s*include_if_exists = 'micro\.conf'/m; + /** * On the "micro" profile, these settings live in micro.conf/pod.conf inside PGDATA instead, * so no `-c` runtime args are emitted; passing them on the command line would take precedence * over the conf files and override users' `ALTER SYSTEM` changes on every restart. */ -const runtimeArgsForProfile = (profile?: "default" | "micro"): readonly string[] => - profile === "micro" ? [] : NATIVE_POSTGRES_RUNTIME_ARGS; +const runtimeArgsForProfile = ( + profile: "default" | "micro" | undefined, + dataDir: string, +): readonly string[] => { + if (profile !== "micro") return NATIVE_POSTGRES_RUNTIME_ARGS; + try { + return ACTIVE_MICRO_INCLUDE_RE.test(readFileSync(`${dataDir}/postgresql.conf`, "utf8")) + ? [] + : NATIVE_POSTGRES_RUNTIME_ARGS; + } catch { + return NATIVE_POSTGRES_RUNTIME_ARGS; + } +}; const orphanCleanup = (opts: PostgresServiceOptions) => opts.cleanupDataDirOnExit ? removePathOnOrphanCleanup(opts.dataDir) : []; @@ -157,7 +169,7 @@ export const makePostgresService = (opts: NativePostgresOptions): ServiceDef => initScript, "-p", String(opts.port), - ...runtimeArgsForProfile(opts.profile), + ...runtimeArgsForProfile(opts.profile, opts.dataDir), "-c", "listen_addresses=*", "-c", @@ -179,7 +191,12 @@ export const makePostgresService = (opts: NativePostgresOptions): ServiceDef => return { name: "postgres", command: "bash", - args: [initScript, "-p", String(opts.port), ...runtimeArgsForProfile(opts.profile)], + args: [ + initScript, + "-p", + String(opts.port), + ...runtimeArgsForProfile(opts.profile, opts.dataDir), + ], env: postgresEnv(opts), healthCheck: postgresHealthCheck(opts.binPath, opts.port), shutdown: { signal: "SIGTERM", timeoutSeconds: 10 }, diff --git a/packages/stack/src/services/postgrest.ts b/packages/stack/src/services/postgrest.ts index fffd349457..f89fe4814f 100644 --- a/packages/stack/src/services/postgrest.ts +++ b/packages/stack/src/services/postgrest.ts @@ -1,8 +1,10 @@ import type { ServiceDef } from "@supabase/process-compose"; +import { postgresConnectionUrl } from "../postgresCredentials.ts"; import { dockerServiceCleanup, dockerServiceOrphanCleanup } from "./docker-cleanup.ts"; interface PostgrestServiceOptions { readonly dbPort: number; + readonly dbPassword: string; readonly port: number; readonly schemas: ReadonlyArray; readonly extraSearchPath: ReadonlyArray; @@ -26,7 +28,13 @@ const postgrestEnv = ( opts: PostgrestServiceOptions, dbHost = "127.0.0.1", ): Record => ({ - PGRST_DB_URI: `postgresql://authenticator:postgres@${dbHost}:${opts.dbPort}/postgres`, + PGRST_DB_URI: postgresConnectionUrl({ + user: "authenticator", + password: opts.dbPassword, + host: dbHost, + port: opts.dbPort, + database: "postgres", + }), PGRST_DB_SCHEMAS: opts.schemas.join(","), PGRST_DB_EXTRA_SEARCH_PATH: opts.extraSearchPath.join(","), PGRST_DB_ANON_ROLE: "anon", diff --git a/packages/stack/src/services/realtime.ts b/packages/stack/src/services/realtime.ts index 4f4c209777..1383f8249f 100644 --- a/packages/stack/src/services/realtime.ts +++ b/packages/stack/src/services/realtime.ts @@ -7,6 +7,7 @@ interface DockerRealtimeOptions { readonly apiPort: number; readonly dbHost: string; readonly dbPort: number; + readonly dbPassword: string; readonly jwtSecret: string; readonly jwtJwks: string; readonly tenantId: string; @@ -47,7 +48,7 @@ export const makeRealtimeServiceDocker = (opts: DockerRealtimeOptions): ServiceD DB_HOST: opts.dbHost, DB_PORT: String(opts.dbPort), DB_USER: "postgres", - DB_PASSWORD: "postgres", + DB_PASSWORD: opts.dbPassword, DB_NAME: "postgres", DB_AFTER_CONNECT_QUERY: "SET search_path TO _realtime", DB_ENC_KEY: opts.encryptionKey, diff --git a/packages/stack/src/services/services.unit.test.ts b/packages/stack/src/services/services.unit.test.ts index 2db96bb9af..38d3a0e29e 100644 --- a/packages/stack/src/services/services.unit.test.ts +++ b/packages/stack/src/services/services.unit.test.ts @@ -1,4 +1,4 @@ -import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; import { describe, expect, it } from "vitest"; @@ -21,6 +21,7 @@ import { DEFAULT_VERSIONS, dockerImageForService } from "../versions.ts"; const JWT_SECRET = "super-secret-jwt-token-with-at-least-32-characters-long"; const DB_PORT = 54322; +const DB_PASSWORD = "postgres"; const API_PORT = 54321; const POSTGRES_BIN_PATH = `/cache/postgres/${DEFAULT_VERSIONS.postgres}/darwin-arm64`; const POSTGREST_BIN_PATH = `/cache/postgrest/${DEFAULT_VERSIONS.postgrest}/macos-aarch64`; @@ -34,6 +35,7 @@ describe("makePostgresService", () => { binPath: POSTGRES_BIN_PATH, dataDir: "/tmp/supabase/data", port: DB_PORT, + password: DB_PASSWORD, }); expect(def.name).toBe("postgres"); @@ -65,6 +67,44 @@ describe("makePostgresService", () => { expect(def.restart).toBe("unless-stopped"); expect(def.supervision).toBeDefined(); }); + + it("keeps runtime replication args for fresh micro-profile data dirs", () => { + const tempDir = mkdtempSync(path.join(tmpdir(), "stack-postgres-service-")); + const dataDir = path.join(tempDir, "data"); + const def = makePostgresService({ + binPath: POSTGRES_BIN_PATH, + dataDir, + port: DB_PORT, + password: DB_PASSWORD, + profile: "micro", + }); + + try { + expect(def.args).toContain("wal_level=logical"); + } finally { + rmSync(tempDir, { recursive: true, force: true }); + } + }); + + it("omits runtime replication args when the micro-profile overlay is installed", () => { + const tempDir = mkdtempSync(path.join(tmpdir(), "stack-postgres-service-")); + const dataDir = path.join(tempDir, "data"); + mkdirSync(dataDir, { recursive: true }); + writeFileSync(path.join(dataDir, "postgresql.conf"), "include_if_exists = 'micro.conf'\n"); + const def = makePostgresService({ + binPath: POSTGRES_BIN_PATH, + dataDir, + port: DB_PORT, + password: DB_PASSWORD, + profile: "micro", + }); + + try { + expect(def.args).not.toContain("wal_level=logical"); + } finally { + rmSync(tempDir, { recursive: true, force: true }); + } + }); }); describe("analyticsDockerRuntimeNetwork", () => { @@ -92,6 +132,7 @@ describe("makeStudioServiceDocker", () => { apiUrl: "http://host.docker.internal:54321", publicApiUrl: "http://127.0.0.1:54321", pgmetaUrl: "http://host.docker.internal:54322", + dbPassword: DB_PASSWORD, publishableKey: "sb_publishable_test", secretKey: "sb_secret_test", s3ProtocolAccessKeyId: LOCAL_S3_PROTOCOL_ACCESS_KEY_ID, @@ -123,6 +164,7 @@ describe("makePostgresService (dockerAccessible)", () => { binPath: POSTGRES_BIN_PATH, dataDir: path.join(tempDir, "data"), port: DB_PORT, + password: DB_PASSWORD, dockerAccessible: true, cleanupDataDirOnExit: true, }); @@ -166,6 +208,7 @@ describe("makePostgresServiceDocker", () => { image: dockerImageForService("postgres", DEFAULT_VERSIONS.postgres), dataDir: "/tmp/supabase/data", port: DB_PORT, + password: DB_PASSWORD, networkArgs: [...LINUX_HOST_GATEWAY_ARGS, "-p", `${DB_PORT}:${DB_PORT}`], jwtSecret: "test-jwt-secret-with-at-least-32-characters", jwtExpiry: 3600, @@ -209,6 +252,7 @@ describe("makePostgresServiceDocker", () => { image: dockerImageForService("postgres", DEFAULT_VERSIONS.postgres), dataDir: "/tmp/supabase/data", port: DB_PORT, + password: DB_PASSWORD, networkArgs: [...LINUX_HOST_GATEWAY_ARGS, "-p", `${DB_PORT}:${DB_PORT}`], jwtSecret: "test-jwt-secret-with-at-least-32-characters", jwtExpiry: 3600, @@ -231,6 +275,7 @@ describe("makePostgrestService", () => { const def = makePostgrestService({ binPath: POSTGREST_BIN_PATH, dbPort: DB_PORT, + dbPassword: DB_PASSWORD, port: API_PORT, schemas: ["public", "storage"], extraSearchPath: ["public", "extensions"], @@ -263,6 +308,7 @@ describe("makeAuthServiceNative", () => { const def = makeAuthServiceNative({ binPath: AUTH_BIN_PATH, dbPort: DB_PORT, + dbPassword: DB_PASSWORD, authPort: 9999, siteUrl: "http://localhost:3000", jwtSecret: JWT_SECRET, @@ -293,6 +339,7 @@ describe("makeAuthServiceDocker", () => { const def = makeAuthServiceDocker({ image: dockerImageForService("auth", DEFAULT_VERSIONS.auth), dbPort: DB_PORT, + dbPassword: DB_PASSWORD, authPort: 9999, siteUrl: "http://localhost:3000", jwtSecret: JWT_SECRET, @@ -408,6 +455,7 @@ describe("makePostgresInitService", () => { const def = makePostgresInitService({ postgresDir: POSTGRES_BIN_PATH, dbPort: DB_PORT, + dbPassword: DB_PASSWORD, autoExposeNewTables: true, }); @@ -426,6 +474,7 @@ describe("makePostgresInitService", () => { const def = makePostgresInitService({ postgresDir: POSTGRES_BIN_PATH, dbPort: DB_PORT, + dbPassword: DB_PASSWORD, autoExposeNewTables: true, }); const script = def.args?.[1] as string; @@ -436,6 +485,7 @@ describe("makePostgresInitService", () => { const def = makePostgresInitService({ postgresDir: "/cache/postgres/17/darwin-arm64", dbPort: DB_PORT, + dbPassword: DB_PASSWORD, autoExposeNewTables: true, }); const script = def.args?.[1] as string; @@ -447,6 +497,7 @@ describe("makePostgresInitService", () => { const def = makePostgresInitService({ postgresDir: "/cache/postgres/17/darwin-arm64", dbPort: DB_PORT, + dbPassword: DB_PASSWORD, autoExposeNewTables: true, }); const script = def.args?.[1] as string; @@ -462,6 +513,7 @@ describe("makePostgresInitService", () => { const def = makePostgresInitService({ postgresDir: "/cache/postgres/17/darwin-arm64", dbPort: DB_PORT, + dbPassword: DB_PASSWORD, autoExposeNewTables: true, }); const script = def.args?.[1] as string; @@ -475,6 +527,7 @@ describe("makePostgresInitService", () => { const def = makePostgresInitService({ postgresDir: POSTGRES_BIN_PATH, dbPort: DB_PORT, + dbPassword: DB_PASSWORD, autoExposeNewTables: true, }); const script = def.args?.[1] as string; @@ -486,6 +539,7 @@ describe("makePostgresInitService", () => { const def = makePostgresInitService({ postgresDir: POSTGRES_BIN_PATH, dbPort: DB_PORT, + dbPassword: DB_PASSWORD, autoExposeNewTables: false, }); const script = def.args?.[1] as string; @@ -581,6 +635,7 @@ describe("docker-backed auxiliary services", () => { nodeHost: "0.0.0.0", dbHost: "127.0.0.1", dbPort: DB_PORT, + dbPassword: DB_PASSWORD, apiKey: "test-api-key", backend: "postgres", networkArgs: ["-p", "54328:4000"], @@ -611,6 +666,7 @@ describe("docker-backed auxiliary services", () => { nodeHost: "0.0.0.0", dbHost: "host.docker.internal", dbPort: DB_PORT, + dbPassword: DB_PASSWORD, apiKey: "test-api-key", backend: "postgres", networkArgs: [...LINUX_HOST_GATEWAY_ARGS, "-p", "54328:4000"], @@ -631,6 +687,7 @@ describe("docker-backed auxiliary services", () => { hostAdminPort: 54329, dbHost: "127.0.0.1", dbPort: DB_PORT, + dbPassword: DB_PASSWORD, poolMode: "transaction", defaultPoolSize: 20, maxClientConn: 100, diff --git a/packages/stack/src/services/storage.ts b/packages/stack/src/services/storage.ts index 056a27605e..060d3f00f0 100644 --- a/packages/stack/src/services/storage.ts +++ b/packages/stack/src/services/storage.ts @@ -1,4 +1,5 @@ import type { ServiceDef } from "@supabase/process-compose"; +import { postgresConnectionUrl } from "../postgresCredentials.ts"; import { removePathOnOrphanCleanup } from "./docker-cleanup.ts"; import { dockerRunService, type ServiceDependency } from "./service-utils.ts"; @@ -8,6 +9,7 @@ interface DockerStorageOptions { readonly apiPort: number; readonly dbHost: string; readonly dbPort: number; + readonly dbPassword: string; readonly dataDir: string; readonly anonKey: string; readonly serviceKey: string; @@ -57,7 +59,13 @@ export const makeStorageServiceDocker = (opts: DockerStorageOptions): ServiceDef AUTH_JWT_SECRET: opts.jwtSecret, PGRST_JWT_SECRET: opts.jwtSecret, JWT_JWKS: opts.jwtJwks, - DATABASE_URL: `postgresql://supabase_storage_admin:postgres@${opts.dbHost}:${opts.dbPort}/postgres`, + DATABASE_URL: postgresConnectionUrl({ + user: "supabase_storage_admin", + password: opts.dbPassword, + host: opts.dbHost, + port: opts.dbPort, + database: "postgres", + }), FILE_SIZE_LIMIT: opts.fileSizeLimit, STORAGE_BACKEND: "file", FILE_STORAGE_BACKEND_PATH: STORAGE_DATA_DIR, diff --git a/packages/stack/src/services/studio.ts b/packages/stack/src/services/studio.ts index 7521fce3c7..d91853f277 100644 --- a/packages/stack/src/services/studio.ts +++ b/packages/stack/src/services/studio.ts @@ -8,6 +8,7 @@ interface DockerStudioOptions { readonly apiUrl: string; readonly publicApiUrl: string; readonly pgmetaUrl: string; + readonly dbPassword: string; readonly publishableKey: string; readonly secretKey: string; readonly s3ProtocolAccessKeyId: string; @@ -44,7 +45,7 @@ export const makeStudioServiceDocker = (opts: DockerStudioOptions): ServiceDef = PORT: String(opts.port), CURRENT_CLI_VERSION: "local", STUDIO_PG_META_URL: opts.pgmetaUrl, - POSTGRES_PASSWORD: "postgres", + POSTGRES_PASSWORD: opts.dbPassword, SUPABASE_URL: opts.apiUrl, SUPABASE_PUBLIC_URL: opts.publicApiUrl, AUTH_JWT_SECRET: opts.jwtSecret, diff --git a/packages/stack/tests/helpers/buildServices.ts b/packages/stack/tests/helpers/buildServices.ts index b8993c5446..cba1d8f2de 100644 --- a/packages/stack/tests/helpers/buildServices.ts +++ b/packages/stack/tests/helpers/buildServices.ts @@ -103,6 +103,7 @@ const baseConfig: ResolvedStackConfig = { port: 5432, dataDir: "/tmp/pg-data", version: DEFAULT_VERSIONS.postgres, + password: "postgres", autoExposeNewTables: true, }, postgrest: false, From 6137e34b2c008104ec47d631021b29f9be3e401e Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Wed, 8 Jul 2026 10:06:08 +0200 Subject: [PATCH 32/38] chore(cli): address micro stack review follow-ups --- packages/fleet/src/Fleet.ts | 19 +- packages/fleet/src/Fleet.unit.test.ts | 59 +++++ packages/fleet/src/PodManifest.ts | 17 +- packages/fleet/src/PodManifest.unit.test.ts | 14 ++ packages/fleet/src/PodRegistry.ts | 8 +- packages/fleet/src/PodRegistry.unit.test.ts | 20 +- packages/fleet/src/PortRegistry.ts | 10 +- packages/fleet/src/PortRegistry.unit.test.ts | 10 + packages/fleet/src/Provisioner.ts | 15 +- packages/fleet/src/Provisioner.unit.test.ts | 17 ++ .../src/TemplateStore.integration.test.ts | 5 +- packages/fleet/src/TemplateStore.ts | 222 +++++++++++------- .../fleet/src/cowClone.integration.test.ts | 12 + packages/fleet/src/cowClone.ts | 4 +- packages/stack/src/StackBuilder.ts | 32 ++- .../stack/src/StackLifecycleCoordinator.ts | 13 +- .../StackLifecycleCoordinator.unit.test.ts | 43 +++- packages/stack/src/index.ts | 3 +- packages/stack/src/serviceDependencies.ts | 19 ++ 19 files changed, 417 insertions(+), 125 deletions(-) create mode 100644 packages/fleet/src/Fleet.unit.test.ts create mode 100644 packages/stack/src/serviceDependencies.ts diff --git a/packages/fleet/src/Fleet.ts b/packages/fleet/src/Fleet.ts index 18867772d5..c1d355adc1 100644 --- a/packages/fleet/src/Fleet.ts +++ b/packages/fleet/src/Fleet.ts @@ -285,11 +285,16 @@ export async function createFleet(opts: FleetOptions = {}): Promise // into the freshly loaded PortRegistry from its manifest, which is the // mechanism `restore()` exists for (recovering from a quarantined/corrupt // port-state file). - for (const manifest of await pods.list()) { - await ports.restore(manifest.id, manifest.ports); - await reapStalePostmaster(pods.dataDir(manifest.id)); - await rm(runPidFile(manifest.id), { force: true }); - await registerEdge(manifest); + try { + for (const manifest of await pods.list()) { + await ports.restore(manifest.id, manifest.ports); + await reapStalePostmaster(pods.dataDir(manifest.id)); + await rm(runPidFile(manifest.id), { force: true }); + await registerEdge(manifest); + } + } catch (err) { + await proxy.close().catch(() => {}); + throw err; } const handle: FleetHandle = { @@ -314,9 +319,9 @@ export async function createFleet(opts: FleetOptions = {}): Promise await podLocks.withLock(id, async () => { await suspendLocked(id); await provisioner.destroy(id); + await proxy.unregister(id); + states.delete(id); }); - await proxy.unregister(id); - states.delete(id); }, async resetPod(id) { await podLocks.withLock(id, async () => { diff --git a/packages/fleet/src/Fleet.unit.test.ts b/packages/fleet/src/Fleet.unit.test.ts new file mode 100644 index 0000000000..fe4c73cf11 --- /dev/null +++ b/packages/fleet/src/Fleet.unit.test.ts @@ -0,0 +1,59 @@ +import { type AddressInfo, createServer } from "node:net"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createFleet } from "./Fleet.ts"; +import { PodRegistry } from "./PodRegistry.ts"; +import type { PodManifest } from "./PodManifest.ts"; + +function freePort(): Promise { + return new Promise((resolve, reject) => { + const server = createServer(); + server.on("error", reject); + server.listen(0, "127.0.0.1", () => { + const address = server.address() as AddressInfo; + server.close(() => resolve(address.port)); + }); + }); +} + +function expectPortAvailable(port: number): Promise { + return new Promise((resolve, reject) => { + const server = createServer(); + server.on("error", reject); + server.listen(port, "127.0.0.1", () => { + server.close(() => resolve()); + }); + }); +} + +function manifest(id: string, dbPort: number, apiPort: number): PodManifest { + return { + id, + versions: { postgres: "17.6.1.143" }, + services: {}, + flags: { supautils: false }, + ports: { dbPort, apiPort }, + createdAt: "2026-07-08T00:00:00.000Z", + }; +} + +describe("createFleet", () => { + it("closes registered edge listeners when startup reconciliation fails", async () => { + const root = await mkdtemp(join(tmpdir(), "fleet-unit-")); + try { + const pods = new PodRegistry(join(root, "pods")); + const dbPort = await freePort(); + const apiPort = await freePort(); + + await pods.write(manifest("pod-a", dbPort, apiPort)); + await pods.write(manifest("pod-b", dbPort, apiPort + 1)); + + await expect(createFleet({ root })).rejects.toThrow(/port already assigned/); + await expect(expectPortAvailable(dbPort)).resolves.toBeUndefined(); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/fleet/src/PodManifest.ts b/packages/fleet/src/PodManifest.ts index d82c019afa..0116112482 100644 --- a/packages/fleet/src/PodManifest.ts +++ b/packages/fleet/src/PodManifest.ts @@ -17,16 +17,29 @@ export interface PodManifest { const keyHash = (value: string): string => createHash("sha256").update(value).digest("hex").slice(0, 16); -export const baseTemplateKey = (postgresVersion: string): string => - `pg-${keyHash(postgresVersion)}`; +interface TemplateKeyOptions { + readonly postgresPassword?: string; +} + +const DEFAULT_POSTGRES_PASSWORD = "postgres"; + +export const baseTemplateKey = (postgresVersion: string, opts: TemplateKeyOptions = {}): string => { + const canonical = JSON.stringify({ + postgresVersion, + postgresPassword: opts.postgresPassword ?? DEFAULT_POSTGRES_PASSWORD, + }); + return `pg-${keyHash(canonical)}`; +}; export const templateKey = ( versions: Partial, enabledServices: ReadonlyArray = [], + opts: TemplateKeyOptions = {}, ): string => { const canonical = JSON.stringify({ versions: Object.fromEntries(Object.entries(versions).sort(([a], [b]) => a.localeCompare(b))), enabledServices: [...new Set(enabledServices)].sort((a, b) => a.localeCompare(b)), + postgresPassword: opts.postgresPassword ?? DEFAULT_POSTGRES_PASSWORD, }); return `tuple-${keyHash(canonical)}`; }; diff --git a/packages/fleet/src/PodManifest.unit.test.ts b/packages/fleet/src/PodManifest.unit.test.ts index 754c8bf834..4fd76f7bf6 100644 --- a/packages/fleet/src/PodManifest.unit.test.ts +++ b/packages/fleet/src/PodManifest.unit.test.ts @@ -25,6 +25,20 @@ describe("templateKey", () => { expect(baseTemplateKey("../17.6.1.143")).toMatch(/^pg-[a-f0-9]{16}$/); expect(baseTemplateKey("../17.6.1.143")).not.toContain(".."); }); + it("keys templates by the postgres password used to initialize the data dir", () => { + expect(baseTemplateKey("17.6.1.143", { postgresPassword: "one" })).not.toBe( + baseTemplateKey("17.6.1.143", { postgresPassword: "two" }), + ); + expect( + templateKey({ postgres: "17.6.1.143", auth: "2.192.0" }, ["auth"], { + postgresPassword: "one", + }), + ).not.toBe( + templateKey({ postgres: "17.6.1.143", auth: "2.192.0" }, ["auth"], { + postgresPassword: "two", + }), + ); + }); it("resolves default versions only for postgres and enabled services", () => { expect(resolveTemplateVersions({ postgres: "17.6.1.143" }, ["postgrest"])).toEqual({ diff --git a/packages/fleet/src/PodRegistry.ts b/packages/fleet/src/PodRegistry.ts index af6f766b4b..de63236aee 100644 --- a/packages/fleet/src/PodRegistry.ts +++ b/packages/fleet/src/PodRegistry.ts @@ -4,8 +4,12 @@ import type { PodManifest } from "./PodManifest.ts"; const POD_ID_RE = /^[A-Za-z0-9][A-Za-z0-9._-]*$/; +function isValidPodId(id: string): boolean { + return POD_ID_RE.test(id) && basename(id) === id; +} + function validatePodId(id: string): string { - if (!POD_ID_RE.test(id) || basename(id) !== id) { + if (!isValidPodId(id)) { throw new Error(`invalid pod id: ${id}`); } return id; @@ -38,7 +42,7 @@ export class PodRegistry { async list(): Promise { const entries = await readdir(this.podsRoot).catch(() => [] as string[]); - const manifests = await Promise.all(entries.map((id) => this.read(id))); + const manifests = await Promise.all(entries.filter(isValidPodId).map((id) => this.read(id))); return manifests.filter((m): m is PodManifest => m !== undefined); } diff --git a/packages/fleet/src/PodRegistry.unit.test.ts b/packages/fleet/src/PodRegistry.unit.test.ts index ede01df1ea..1d56cb0877 100644 --- a/packages/fleet/src/PodRegistry.unit.test.ts +++ b/packages/fleet/src/PodRegistry.unit.test.ts @@ -1,4 +1,4 @@ -import { mkdtemp } from "node:fs/promises"; +import { mkdtemp, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; @@ -12,4 +12,22 @@ describe("PodRegistry", () => { expect(() => pods.dataDir("nested/pod")).toThrow(/invalid pod id/); expect(() => pods.podDir("pod-a")).not.toThrow(); }); + + it("ignores non-pod entries while listing persisted pods", async () => { + const root = await mkdtemp(join(tmpdir(), "pods-")); + const pods = new PodRegistry(root); + const manifest = { + id: "pod-a", + versions: { postgres: "17.6.1.143" }, + services: {}, + flags: { supautils: false }, + ports: { dbPort: 55000, apiPort: 55001 }, + createdAt: "2026-07-08T00:00:00.000Z", + }; + + await pods.write(manifest); + await writeFile(join(root, ".DS_Store"), "not a pod"); + + await expect(pods.list()).resolves.toEqual([manifest]); + }); }); diff --git a/packages/fleet/src/PortRegistry.ts b/packages/fleet/src/PortRegistry.ts index 767699b853..60a1471f2d 100644 --- a/packages/fleet/src/PortRegistry.ts +++ b/packages/fleet/src/PortRegistry.ts @@ -17,6 +17,10 @@ function freshState(): PortState { return { basePort: DEFAULT_BASE_PORT, pods: {} }; } +function getOwnPodPorts(pods: Record, podId: string): PodPorts | undefined { + return Object.hasOwn(pods, podId) ? pods[podId] : undefined; +} + function isRecord(value: unknown): value is Record { return typeof value === "object" && value !== null && !Array.isArray(value); } @@ -96,12 +100,12 @@ export class PortRegistry { } get(podId: string): PodPorts | undefined { - return this.state.pods[podId]; + return getOwnPodPorts(this.state.pods, podId); } async allocate(podId: string): Promise { return this.withMutation(async () => { - const existing = this.state.pods[podId]; + const existing = getOwnPodPorts(this.state.pods, podId); if (existing) return existing; const used = new Set(Object.values(this.state.pods).flatMap((p) => [p.dbPort, p.apiPort])); let candidate = this.state.basePort; @@ -126,7 +130,7 @@ export class PortRegistry { */ async restore(podId: string, ports: PodPorts): Promise { await this.withMutation(async () => { - const existing = this.state.pods[podId]; + const existing = getOwnPodPorts(this.state.pods, podId); if (existing) { if (existing.dbPort === ports.dbPort && existing.apiPort === ports.apiPort) { return; diff --git a/packages/fleet/src/PortRegistry.unit.test.ts b/packages/fleet/src/PortRegistry.unit.test.ts index 34715f26a8..da2e7d2c76 100644 --- a/packages/fleet/src/PortRegistry.unit.test.ts +++ b/packages/fleet/src/PortRegistry.unit.test.ts @@ -29,6 +29,16 @@ describe("PortRegistry", () => { expect(c.dbPort).toBe(a1.dbPort); // freed ports are reusable }); + it("treats inherited property names as ordinary pod ids", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + const reg = await PortRegistry.load(file); + + const ports = await reg.allocate("constructor"); + + expect(ports).toEqual({ dbPort: 55000, apiPort: 55001 }); + expect(reg.get("constructor")).toEqual(ports); + }); + it("serializes concurrent mutations before persisting", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); diff --git a/packages/fleet/src/Provisioner.ts b/packages/fleet/src/Provisioner.ts index 45b8bd0697..fcf6293c94 100644 --- a/packages/fleet/src/Provisioner.ts +++ b/packages/fleet/src/Provisioner.ts @@ -1,5 +1,10 @@ import { rm } from "node:fs/promises"; -import type { ServiceName, VersionManifest } from "@supabase/stack"; +import { + SERVICE_NAMES, + validateEnabledServiceDependencies, + type ServiceName, + type VersionManifest, +} from "@supabase/stack"; import { cloneDir } from "./cowClone.ts"; import { resolveTemplateVersions, type PodManifest } from "./PodManifest.ts"; import type { PodRegistry } from "./PodRegistry.ts"; @@ -36,9 +41,11 @@ export class Provisioner { } const pgVersion = opts.versions.postgres; if (pgVersion === undefined) throw new Error("versions.postgres is required"); - const enabled = Object.entries(opts.services ?? {}) - .filter(([, on]) => on === true) - .map(([name]) => name as ServiceName); + const enabled = SERVICE_NAMES.filter((name) => opts.services?.[name] === true); + const dependencyError = validateEnabledServiceDependencies(new Set(enabled)); + if (dependencyError !== undefined) { + throw new Error(dependencyError); + } const resolvedVersions = resolveTemplateVersions(opts.versions, enabled); const template = opts.warm === true diff --git a/packages/fleet/src/Provisioner.unit.test.ts b/packages/fleet/src/Provisioner.unit.test.ts index ad965c18fc..e094c11770 100644 --- a/packages/fleet/src/Provisioner.unit.test.ts +++ b/packages/fleet/src/Provisioner.unit.test.ts @@ -102,6 +102,23 @@ describe("Provisioner (unit, fake deps)", () => { postgrest: DEFAULT_VERSIONS.postgrest, }); }); + + it("rejects invalid service dependency combinations before provisioning", async () => { + const templateDir = await mkdtemp(join(tmpdir(), "fleet-template-")); + await writeFile(join(templateDir, "PG_VERSION"), PG_VERSION); + const { p, ports, podsRoot } = await makeHarness(templateDir); + + await expect( + p.create({ + id: "bad", + versions: { postgres: PG_VERSION }, + services: { imgproxy: true }, + }), + ).rejects.toThrow(/imgproxy requires storage/); + + expect(ports.get("bad")).toBeUndefined(); + expect(await podsRootEntries(podsRoot)).not.toContain("bad"); + }); }); describe("fork", () => { diff --git a/packages/fleet/src/TemplateStore.integration.test.ts b/packages/fleet/src/TemplateStore.integration.test.ts index 06f1c4172a..95f1ba2cd5 100644 --- a/packages/fleet/src/TemplateStore.integration.test.ts +++ b/packages/fleet/src/TemplateStore.integration.test.ts @@ -1,6 +1,7 @@ import { mkdtemp, readFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { resolvePostgresPassword } from "@supabase/stack"; import { describe, expect, it } from "vitest"; import { baseTemplateKey } from "./PodManifest.ts"; import { TemplateStore } from "./TemplateStore.ts"; @@ -14,7 +15,9 @@ describe.skipIf(!process.env.FLEET_PG_TESTS)("TemplateStore", () => { const root = await mkdtemp(join(tmpdir(), "templates-")); const store = new TemplateStore(root); const first = await store.ensureBaseTemplate(POSTGRES_VERSION); - expect(first).toContain(baseTemplateKey(POSTGRES_VERSION)); + expect(first).toContain( + baseTemplateKey(POSTGRES_VERSION, { postgresPassword: resolvePostgresPassword() }), + ); // PGDATA got the micro profile const conf = await readFile(join(first, "postgresql.conf"), "utf8"); expect(conf).toContain("include_if_exists = 'micro.conf'"); diff --git a/packages/fleet/src/TemplateStore.ts b/packages/fleet/src/TemplateStore.ts index f6c12ad5b7..ff82e61b28 100644 --- a/packages/fleet/src/TemplateStore.ts +++ b/packages/fleet/src/TemplateStore.ts @@ -1,12 +1,24 @@ -import { mkdir, open, rename, rm, stat, unlink, writeFile } from "node:fs/promises"; +import { + mkdir, + mkdtemp, + open, + readFile, + rename, + rm, + stat, + unlink, + writeFile, +} from "node:fs/promises"; +import type { FileHandle } from "node:fs/promises"; import { join } from "node:path"; -import { createStack, installMicroProfile } from "@supabase/stack"; +import { createStack, installMicroProfile, resolvePostgresPassword } from "@supabase/stack"; import type { ServiceName, VersionManifest } from "@supabase/stack"; import { cloneDir } from "./cowClone.ts"; import { baseTemplateKey, resolveTemplateVersions, templateKey } from "./PodManifest.ts"; const LOCK_STALE_MS = 10 * 60 * 1000; const LOCK_POLL_MS = 250; +const LOCK_HEARTBEAT_MS = Math.floor(LOCK_STALE_MS / 3); function errorCode(error: unknown): string | undefined { if (typeof error !== "object" || error === null || !("code" in error)) return undefined; @@ -44,41 +56,50 @@ export class TemplateStore { } async ensureBaseTemplate(postgresVersion: string): Promise { - const key = baseTemplateKey(postgresVersion); + const postgresPassword = resolvePostgresPassword(); + const key = baseTemplateKey(postgresVersion, { postgresPassword }); if (await this.has(key)) return this.dataDir(key); return this.withLock(key, async () => { if (await this.has(key)) return this.dataDir(key); - const buildDir = join(this.root, `${key}.build`); - await rm(buildDir, { recursive: true, force: true }); - await mkdir(buildDir, { recursive: true }); + const buildDir = await this.createBuildDir(key); + let frozen = false; const buildDataDir = join(buildDir, "data"); - // One-shot stack: postgres only, non-provisioned -> postgres-init applies - // roles/schemas/baseline migrations exactly as it does for a normal stack. - const stack = await createStack({ - postgres: { version: postgresVersion, dataDir: buildDataDir }, - postgrest: false, - auth: false, - edgeRuntime: false, - realtime: false, - storage: false, - imgproxy: false, - mailpit: false, - pgmeta: false, - studio: false, - analytics: false, - vector: false, - pooler: false, - functions: false, - }); try { - await stack.start(); - await stack.ready(); + // One-shot stack: postgres only, non-provisioned -> postgres-init applies + // roles/schemas/baseline migrations exactly as it does for a normal stack. + const stack = await createStack({ + postgres: { version: postgresVersion, dataDir: buildDataDir }, + postgrest: false, + auth: false, + edgeRuntime: false, + realtime: false, + storage: false, + imgproxy: false, + mailpit: false, + pgmeta: false, + studio: false, + analytics: false, + vector: false, + pooler: false, + functions: false, + }); + try { + await stack.start(); + await stack.ready(); + } finally { + await stack.dispose(); + } + await installMicroProfile(buildDataDir); + await this.freeze(buildDir, key, { + key, + postgresVersion, + builtAt: new Date().toISOString(), + }); + frozen = true; + return this.dataDir(key); } finally { - await stack.dispose(); + if (!frozen) await rm(buildDir, { recursive: true, force: true }); } - await installMicroProfile(buildDataDir); - await this.freeze(buildDir, key, { key, postgresVersion, builtAt: new Date().toISOString() }); - return this.dataDir(key); }); } @@ -88,73 +109,84 @@ export class TemplateStore { ): Promise { const pgVersion = versions.postgres; if (pgVersion === undefined) throw new Error("versions.postgres is required"); + const postgresPassword = resolvePostgresPassword(); const base = await this.ensureBaseTemplate(pgVersion); if (enabledServices.length === 0) return base; const resolvedVersions = resolveTemplateVersions(versions, enabledServices); - const key = templateKey(resolvedVersions, enabledServices); + const key = templateKey(resolvedVersions, enabledServices, { postgresPassword }); if (await this.has(key)) return this.dataDir(key); return this.withLock(key, async () => { if (await this.has(key)) return this.dataDir(key); - const buildDir = join(this.root, `${key}.build`); - await rm(buildDir, { recursive: true, force: true }); - await mkdir(buildDir, { recursive: true }); + const buildDir = await this.createBuildDir(key); + let frozen = false; const buildDataDir = join(buildDir, "data"); - await cloneDir(base, buildDataDir); - // The clone is already post-init, so postgres-init is skipped (`provisioned: true`); - // each enabled service boots once and self-migrates against it. - const stack = await createStack({ - postgres: { - version: pgVersion, - dataDir: buildDataDir, - provisioned: true, - profile: "micro", - }, - postgrest: enabledServices.includes("postgrest") - ? { version: resolvedVersions.postgrest } - : false, - auth: enabledServices.includes("auth") ? { version: resolvedVersions.auth } : false, - edgeRuntime: enabledServices.includes("edge-runtime") - ? { version: resolvedVersions["edge-runtime"] } - : false, - realtime: enabledServices.includes("realtime") - ? { version: resolvedVersions.realtime } - : false, - storage: enabledServices.includes("storage") - ? { version: resolvedVersions.storage } - : false, - imgproxy: enabledServices.includes("imgproxy") - ? { version: resolvedVersions.imgproxy } - : false, - mailpit: enabledServices.includes("mailpit") - ? { version: resolvedVersions.mailpit } - : false, - pgmeta: enabledServices.includes("pgmeta") ? { version: resolvedVersions.pgmeta } : false, - studio: enabledServices.includes("studio") ? { version: resolvedVersions.studio } : false, - analytics: enabledServices.includes("analytics") - ? { version: resolvedVersions.analytics } - : false, - vector: enabledServices.includes("vector") ? { version: resolvedVersions.vector } : false, - pooler: enabledServices.includes("pooler") ? { version: resolvedVersions.pooler } : false, - functions: false, - }); try { - await stack.start(); - await stack.ready(); + await cloneDir(base, buildDataDir); + + // The clone is already post-init, so postgres-init is skipped (`provisioned: true`); + // each enabled service boots once and self-migrates against it. + const stack = await createStack({ + postgres: { + version: pgVersion, + dataDir: buildDataDir, + provisioned: true, + profile: "micro", + }, + postgrest: enabledServices.includes("postgrest") + ? { version: resolvedVersions.postgrest } + : false, + auth: enabledServices.includes("auth") ? { version: resolvedVersions.auth } : false, + edgeRuntime: enabledServices.includes("edge-runtime") + ? { version: resolvedVersions["edge-runtime"] } + : false, + realtime: enabledServices.includes("realtime") + ? { version: resolvedVersions.realtime } + : false, + storage: enabledServices.includes("storage") + ? { version: resolvedVersions.storage } + : false, + imgproxy: enabledServices.includes("imgproxy") + ? { version: resolvedVersions.imgproxy } + : false, + mailpit: enabledServices.includes("mailpit") + ? { version: resolvedVersions.mailpit } + : false, + pgmeta: enabledServices.includes("pgmeta") ? { version: resolvedVersions.pgmeta } : false, + studio: enabledServices.includes("studio") ? { version: resolvedVersions.studio } : false, + analytics: enabledServices.includes("analytics") + ? { version: resolvedVersions.analytics } + : false, + vector: enabledServices.includes("vector") ? { version: resolvedVersions.vector } : false, + pooler: enabledServices.includes("pooler") ? { version: resolvedVersions.pooler } : false, + functions: false, + }); + try { + await stack.start(); + await stack.ready(); + } finally { + await stack.dispose(); + } + await this.freeze(buildDir, key, { + key, + versions: resolvedVersions, + enabledServices, + builtAt: new Date().toISOString(), + }); + frozen = true; + return this.dataDir(key); } finally { - await stack.dispose(); + if (!frozen) await rm(buildDir, { recursive: true, force: true }); } - await this.freeze(buildDir, key, { - key, - versions: resolvedVersions, - enabledServices, - builtAt: new Date().toISOString(), - }); - return this.dataDir(key); }); } + private async createBuildDir(key: string): Promise { + await mkdir(this.root, { recursive: true }); + return mkdtemp(join(this.root, `${key}.build-`)); + } + private async freeze(buildDir: string, key: string, marker: unknown): Promise { const finalDir = join(this.root, key); await writeFile(join(buildDir, "template.json"), JSON.stringify(marker)); @@ -164,11 +196,18 @@ export class TemplateStore { private async withLock(key: string, body: () => Promise): Promise { const lockPath = join(this.root, `${key}.lock`); + const lockOwner = `${process.pid}:${Date.now()}:${Math.random()}`; + let lockHandle: FileHandle | undefined; await mkdir(this.root, { recursive: true }); for (;;) { try { const handle = await open(lockPath, "wx"); - await handle.close(); + try { + await handle.writeFile(lockOwner); + lockHandle = handle; + } finally { + if (lockHandle === undefined) await handle.close(); + } break; } catch (error) { if (errorCode(error) !== "EEXIST") throw error; @@ -180,9 +219,26 @@ export class TemplateStore { await new Promise((r) => setTimeout(r, LOCK_POLL_MS)); } } + const heartbeat = setInterval(() => { + if (lockHandle !== undefined) void this.refreshLock(lockHandle); + }, LOCK_HEARTBEAT_MS); try { return await body(); } finally { + clearInterval(heartbeat); + await this.releaseLock(lockPath, lockOwner); + await lockHandle?.close().catch(() => {}); + } + } + + private async refreshLock(lockHandle: FileHandle): Promise { + const now = new Date(); + await lockHandle.utimes(now, now).catch(() => {}); + } + + private async releaseLock(lockPath: string, lockOwner: string): Promise { + const currentOwner = await readFile(lockPath, "utf8").catch(() => undefined); + if (currentOwner === lockOwner) { await unlink(lockPath).catch(() => {}); } } diff --git a/packages/fleet/src/cowClone.integration.test.ts b/packages/fleet/src/cowClone.integration.test.ts index 096a9a461d..a65f249c6e 100644 --- a/packages/fleet/src/cowClone.integration.test.ts +++ b/packages/fleet/src/cowClone.integration.test.ts @@ -43,6 +43,18 @@ describe("cloneDir", () => { await expect(cloneDir(src, dest)).rejects.toThrow(/exists/); }); + it("creates the destination parent before attempting the CoW clone", async () => { + const root = await mkdtemp(join(tmpdir(), "cow-test-")); + const src = join(root, "src"); + await mkdir(src); + await writeFile(join(src, "file.txt"), "hello"); + + const dest = join(root, "missing-parent", "dest"); + await cloneDir(src, dest); + + expect(await readFile(join(dest, "file.txt"), "utf8")).toBe("hello"); + }); + it("does not silently mix stale CoW leftovers with fallback output when the CoW command fails after partially writing dest", async () => { const root = await mkdtemp(join(tmpdir(), "cow-test-")); const src = join(root, "src"); diff --git a/packages/fleet/src/cowClone.ts b/packages/fleet/src/cowClone.ts index 48882abe4e..694c5a7313 100644 --- a/packages/fleet/src/cowClone.ts +++ b/packages/fleet/src/cowClone.ts @@ -1,5 +1,6 @@ import { spawn } from "node:child_process"; -import { cp, rm, stat } from "node:fs/promises"; +import { cp, mkdir, rm, stat } from "node:fs/promises"; +import { dirname } from "node:path"; const run = (cmd: string, args: string[]): Promise => new Promise((resolve, reject) => { @@ -30,6 +31,7 @@ export async function cloneDir( () => false, ); if (exists) throw new Error(`cloneDir: destination already exists: ${dest}`); + await mkdir(dirname(dest), { recursive: true }); const cowCommand = options?.cowCommand ?? "cp"; let attemptedCow = false; diff --git a/packages/stack/src/StackBuilder.ts b/packages/stack/src/StackBuilder.ts index b84c134d29..543e728c26 100644 --- a/packages/stack/src/StackBuilder.ts +++ b/packages/stack/src/StackBuilder.ts @@ -12,6 +12,7 @@ import { dockerPortMapArgs, } from "./Platform.ts"; import type { ServiceResolution } from "./resolve.ts"; +import { validateEnabledServiceDependencies } from "./serviceDependencies.ts"; import { analyticsDockerRuntimeNetwork, makeAnalyticsServiceDocker } from "./services/analytics.ts"; import { makeAuthServiceDocker, makeAuthServiceNative } from "./services/auth.ts"; import { @@ -37,7 +38,7 @@ import { makeVectorServiceDocker } from "./services/vector.ts"; import type { PreparedStackArtifacts } from "./StackPreparation.ts"; import type { StackServiceProjectionCatalog } from "./StackStateProjection.ts"; import type { AllocatedPorts } from "./PortAllocator.ts"; -import type { ServiceName, VersionManifest } from "./versions.ts"; +import { SERVICE_NAMES, type ServiceName, type VersionManifest } from "./versions.ts"; export interface PostgresConfig { readonly port?: number; @@ -382,6 +383,10 @@ const resolvedConfigForService = ( service: Exclude, ) => (service === "edge-runtime" ? config.edgeRuntime : config[service]); +const nonPostgresServices = SERVICE_NAMES.filter( + (service): service is Exclude => service !== "postgres", +); + export const validateResolvedConfig = ( config: ResolvedStackConfig, ): Effect.Effect => @@ -399,26 +404,17 @@ export const validateResolvedConfig = ( } } - if (config.imgproxy !== false && config.storage === false) { - return yield* Effect.fail( - new StackBuildError({ - detail: "imgproxy requires storage to be enabled", - }), - ); - } - - if (config.vector !== false && config.analytics === false) { - return yield* Effect.fail( - new StackBuildError({ - detail: "vector requires analytics to be enabled", - }), - ); + const enabledServices = new Set(["postgres"]); + for (const service of nonPostgresServices) { + if (resolvedConfigForService(config, service) !== false) { + enabledServices.add(service); + } } - - if (config.studio !== false && config.pgmeta === false) { + const dependencyError = validateEnabledServiceDependencies(enabledServices); + if (dependencyError !== undefined) { return yield* Effect.fail( new StackBuildError({ - detail: "studio requires pgmeta to be enabled", + detail: dependencyError, }), ); } diff --git a/packages/stack/src/StackLifecycleCoordinator.ts b/packages/stack/src/StackLifecycleCoordinator.ts index 4d015fae53..4d4263e549 100644 --- a/packages/stack/src/StackLifecycleCoordinator.ts +++ b/packages/stack/src/StackLifecycleCoordinator.ts @@ -252,6 +252,7 @@ export class StackLifecycleCoordinator extends Context.Service< const info = stackInfoFor(config); const stateRef = yield* SubscriptionRef.make(initialPublicStates(config)); const phaseRef = yield* Ref.make("idle"); + const startInFlightRef = yield* Ref.make(false); const enableExtensionLock = yield* Semaphore.make(1); // Tracks every service that has actually been asked to start: the eager set from // start() under lazyServices, plus anything started later via startService (the @@ -589,8 +590,10 @@ export class StackLifecycleCoordinator extends Context.Service< Effect.succeed(runtimeState?.cleanupTargets ?? { dockerContainerNames: [] }), start: () => Effect.gen(function* () { + yield* Ref.set(startInFlightRef, true); yield* Ref.set(phaseRef, "starting"); const runtime = yield* ensureRuntime; + yield* Ref.set(phaseRef, "starting"); yield* configureFunctions(config); if (config.lazyServices === true) { // Only bring up postgres (and postgres-init, which depends on postgres and @@ -615,7 +618,7 @@ export class StackLifecycleCoordinator extends Context.Service< yield* runtime.orchestrator.waitAllReady(); } yield* Ref.set(phaseRef, "running"); - }), + }).pipe(Effect.ensuring(Ref.set(startInFlightRef, false))), stop: () => Effect.gen(function* () { if (runtimeState === undefined) { @@ -656,6 +659,14 @@ export class StackLifecycleCoordinator extends Context.Service< enableExtension: (name) => enableExtensionLock.withPermits(1)( Effect.gen(function* () { + const startInFlight = yield* Ref.get(startInFlightRef); + if (startInFlight || (yield* Ref.get(phaseRef)) === "starting") { + return yield* Effect.fail( + new StackBuildError({ + detail: `Cannot enable extension "${name}" while the stack is starting. Wait for start() to finish and retry.`, + }), + ); + } yield* Effect.promise(() => installPodConfOverlay(config.postgres.dataDir)); const currentLibraries = yield* Effect.promise(() => readPreloadLibraries(config.postgres.dataDir), diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts index 04eb5af6f8..2bf76170f3 100644 --- a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -4,7 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "@effect/vitest"; import { NodeServices } from "@effect/platform-node"; -import { Duration, Effect, Layer } from "effect"; +import { Duration, Effect, Exit, Fiber, Layer } from "effect"; import { mockChildProcessSpawner } from "../../process-compose/tests/helpers/mocks.ts"; import { mockBinaryResolver } from "../tests/helpers/mocks.ts"; import { defaultPublishableKey, defaultSecretKey, generateJwt } from "./JwtGenerator.ts"; @@ -139,6 +139,20 @@ function makeLazyConfig(dataDir: string, postgrestPort: number): ResolvedStackCo }; } +function makeSlowStartConfig(dataDir: string): ResolvedStackConfig { + return { + ...makeConfig(dataDir), + postgrest: { + port: defaultPorts.postgrestPort, + adminPort: defaultPorts.postgrestAdminPort, + schemas: ["public"], + extraSearchPath: ["public"], + maxRows: 1000, + version: DEFAULT_VERSIONS.postgrest, + }, + }; +} + interface FakeHealthyServer { readonly port: number; readonly stop: () => Promise; @@ -226,6 +240,33 @@ describe("StackLifecycleCoordinator enableExtension", () => { Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), ); }); + + it.live("rejects preload changes while the stack is still starting", () => { + const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-starting-test-")); + writeFileSync(join(dataDir, "postgresql.conf"), "# stock conf\n"); + const config = makeSlowStartConfig(dataDir); + const { layer, spawner } = setupLayer(config); + + return Effect.gen(function* () { + const stack = yield* Stack; + const startFiber = yield* stack.start().pipe(Effect.forkChild({ startImmediately: true })); + + yield* Effect.gen(function* () { + for (;;) { + if (spawner.spawned.length > 0) break; + yield* Effect.sleep(Duration.millis(10)); + } + + const exit = yield* stack.enableExtension("pg_cron").pipe(Effect.exit); + + expect(Exit.isFailure(exit)).toBe(true); + expect(yield* Effect.promise(() => readPreloadLibraries(dataDir))).toEqual([]); + }).pipe(Effect.ensuring(Fiber.interrupt(startFiber))); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), + ); + }); }); describe("StackLifecycleCoordinator lazyServices", () => { diff --git a/packages/stack/src/index.ts b/packages/stack/src/index.ts index ffd916d368..000abbc761 100644 --- a/packages/stack/src/index.ts +++ b/packages/stack/src/index.ts @@ -21,8 +21,9 @@ export type { } from "./StackBuilder.ts"; export type { ServiceName, VersionManifest } from "./versions.ts"; -export { DEFAULT_VERSIONS, fillServiceVersionManifest } from "./versions.ts"; +export { DEFAULT_VERSIONS, fillServiceVersionManifest, SERVICE_NAMES } from "./versions.ts"; export { postgresConnectionUrl, resolvePostgresPassword } from "./postgresCredentials.ts"; +export { validateEnabledServiceDependencies } from "./serviceDependencies.ts"; export type { ServiceResolution } from "./resolve.ts"; export type { PrefetchOptions, PrefetchResult } from "./prefetch.ts"; export type { ReadyOptions, StackHandle } from "./createStack.ts"; diff --git a/packages/stack/src/serviceDependencies.ts b/packages/stack/src/serviceDependencies.ts new file mode 100644 index 0000000000..026d50c67a --- /dev/null +++ b/packages/stack/src/serviceDependencies.ts @@ -0,0 +1,19 @@ +import type { ServiceName } from "./versions.ts"; + +export const validateEnabledServiceDependencies = ( + enabledServices: ReadonlySet, +): string | undefined => { + if (enabledServices.has("imgproxy") && !enabledServices.has("storage")) { + return "imgproxy requires storage to be enabled"; + } + + if (enabledServices.has("vector") && !enabledServices.has("analytics")) { + return "vector requires analytics to be enabled"; + } + + if (enabledServices.has("studio") && !enabledServices.has("pgmeta")) { + return "studio requires pgmeta to be enabled"; + } + + return undefined; +}; From e31c93f9f3bc55fce94ce1d39ac79acdcfb1464d Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Wed, 8 Jul 2026 10:11:29 +0200 Subject: [PATCH 33/38] chore(cli): address additional stack review feedback --- packages/fleet/src/Fleet.ts | 2 +- packages/fleet/src/TemplateStore.ts | 147 ++++++++++-------- packages/process-compose/src/Orchestrator.ts | 20 ++- .../src/Orchestrator.unit.test.ts | 71 +++++++++ 4 files changed, 171 insertions(+), 69 deletions(-) diff --git a/packages/fleet/src/Fleet.ts b/packages/fleet/src/Fleet.ts index c1d355adc1..f207495900 100644 --- a/packages/fleet/src/Fleet.ts +++ b/packages/fleet/src/Fleet.ts @@ -318,8 +318,8 @@ export async function createFleet(opts: FleetOptions = {}): Promise // to delete. await podLocks.withLock(id, async () => { await suspendLocked(id); - await provisioner.destroy(id); await proxy.unregister(id); + await provisioner.destroy(id); states.delete(id); }); }, diff --git a/packages/fleet/src/TemplateStore.ts b/packages/fleet/src/TemplateStore.ts index ff82e61b28..2e24d62f4a 100644 --- a/packages/fleet/src/TemplateStore.ts +++ b/packages/fleet/src/TemplateStore.ts @@ -19,6 +19,7 @@ import { baseTemplateKey, resolveTemplateVersions, templateKey } from "./PodMani const LOCK_STALE_MS = 10 * 60 * 1000; const LOCK_POLL_MS = 250; const LOCK_HEARTBEAT_MS = Math.floor(LOCK_STALE_MS / 3); +const STACK_BOOT_LOCK_KEY = "__stack-boot"; function errorCode(error: unknown): string | undefined { if (typeof error !== "object" || error === null || !("code" in error)) return undefined; @@ -65,30 +66,32 @@ export class TemplateStore { let frozen = false; const buildDataDir = join(buildDir, "data"); try { - // One-shot stack: postgres only, non-provisioned -> postgres-init applies - // roles/schemas/baseline migrations exactly as it does for a normal stack. - const stack = await createStack({ - postgres: { version: postgresVersion, dataDir: buildDataDir }, - postgrest: false, - auth: false, - edgeRuntime: false, - realtime: false, - storage: false, - imgproxy: false, - mailpit: false, - pgmeta: false, - studio: false, - analytics: false, - vector: false, - pooler: false, - functions: false, + await this.withStackBootLock(async () => { + // One-shot stack: postgres only, non-provisioned -> postgres-init applies + // roles/schemas/baseline migrations exactly as it does for a normal stack. + const stack = await createStack({ + postgres: { version: postgresVersion, dataDir: buildDataDir }, + postgrest: false, + auth: false, + edgeRuntime: false, + realtime: false, + storage: false, + imgproxy: false, + mailpit: false, + pgmeta: false, + studio: false, + analytics: false, + vector: false, + pooler: false, + functions: false, + }); + try { + await stack.start(); + await stack.ready(); + } finally { + await stack.dispose(); + } }); - try { - await stack.start(); - await stack.ready(); - } finally { - await stack.dispose(); - } await installMicroProfile(buildDataDir); await this.freeze(buildDir, key, { key, @@ -125,49 +128,59 @@ export class TemplateStore { try { await cloneDir(base, buildDataDir); - // The clone is already post-init, so postgres-init is skipped (`provisioned: true`); - // each enabled service boots once and self-migrates against it. - const stack = await createStack({ - postgres: { - version: pgVersion, - dataDir: buildDataDir, - provisioned: true, - profile: "micro", - }, - postgrest: enabledServices.includes("postgrest") - ? { version: resolvedVersions.postgrest } - : false, - auth: enabledServices.includes("auth") ? { version: resolvedVersions.auth } : false, - edgeRuntime: enabledServices.includes("edge-runtime") - ? { version: resolvedVersions["edge-runtime"] } - : false, - realtime: enabledServices.includes("realtime") - ? { version: resolvedVersions.realtime } - : false, - storage: enabledServices.includes("storage") - ? { version: resolvedVersions.storage } - : false, - imgproxy: enabledServices.includes("imgproxy") - ? { version: resolvedVersions.imgproxy } - : false, - mailpit: enabledServices.includes("mailpit") - ? { version: resolvedVersions.mailpit } - : false, - pgmeta: enabledServices.includes("pgmeta") ? { version: resolvedVersions.pgmeta } : false, - studio: enabledServices.includes("studio") ? { version: resolvedVersions.studio } : false, - analytics: enabledServices.includes("analytics") - ? { version: resolvedVersions.analytics } - : false, - vector: enabledServices.includes("vector") ? { version: resolvedVersions.vector } : false, - pooler: enabledServices.includes("pooler") ? { version: resolvedVersions.pooler } : false, - functions: false, + await this.withStackBootLock(async () => { + // The clone is already post-init, so postgres-init is skipped (`provisioned: true`); + // each enabled service boots once and self-migrates against it. + const stack = await createStack({ + postgres: { + version: pgVersion, + dataDir: buildDataDir, + provisioned: true, + profile: "micro", + }, + postgrest: enabledServices.includes("postgrest") + ? { version: resolvedVersions.postgrest } + : false, + auth: enabledServices.includes("auth") ? { version: resolvedVersions.auth } : false, + edgeRuntime: enabledServices.includes("edge-runtime") + ? { version: resolvedVersions["edge-runtime"] } + : false, + realtime: enabledServices.includes("realtime") + ? { version: resolvedVersions.realtime } + : false, + storage: enabledServices.includes("storage") + ? { version: resolvedVersions.storage } + : false, + imgproxy: enabledServices.includes("imgproxy") + ? { version: resolvedVersions.imgproxy } + : false, + mailpit: enabledServices.includes("mailpit") + ? { version: resolvedVersions.mailpit } + : false, + pgmeta: enabledServices.includes("pgmeta") + ? { version: resolvedVersions.pgmeta } + : false, + studio: enabledServices.includes("studio") + ? { version: resolvedVersions.studio } + : false, + analytics: enabledServices.includes("analytics") + ? { version: resolvedVersions.analytics } + : false, + vector: enabledServices.includes("vector") + ? { version: resolvedVersions.vector } + : false, + pooler: enabledServices.includes("pooler") + ? { version: resolvedVersions.pooler } + : false, + functions: false, + }); + try { + await stack.start(); + await stack.ready(); + } finally { + await stack.dispose(); + } }); - try { - await stack.start(); - await stack.ready(); - } finally { - await stack.dispose(); - } await this.freeze(buildDir, key, { key, versions: resolvedVersions, @@ -187,6 +200,10 @@ export class TemplateStore { return mkdtemp(join(this.root, `${key}.build-`)); } + private async withStackBootLock(body: () => Promise): Promise { + return this.withLock(STACK_BOOT_LOCK_KEY, body); + } + private async freeze(buildDir: string, key: string, marker: unknown): Promise { const finalDir = join(this.root, key); await writeFile(join(buildDir, "template.json"), JSON.stringify(marker)); diff --git a/packages/process-compose/src/Orchestrator.ts b/packages/process-compose/src/Orchestrator.ts index 3e0c698f60..a048c580e3 100644 --- a/packages/process-compose/src/Orchestrator.ts +++ b/packages/process-compose/src/Orchestrator.ts @@ -119,6 +119,7 @@ export class Orchestrator extends Context.Service< // FiberMap to track running service fibers — auto-interrupted on scope close const fibers = yield* FiberMap.make(); + const launchedServices = new Set(); // Helper: send a validated FSM event — only does the state transition const sendEvent = ( @@ -534,7 +535,8 @@ export class Orchestrator extends Context.Service< const svc = services.get(dependentName); if (svc === undefined) return false; const status = SubscriptionRef.getUnsafe(svc.state).status; - return status !== "Pending" && status !== "Stopped"; + if (status === "Pending") return launchedServices.has(dependentName); + return status !== "Stopped"; }; const collectDependents = (current: string): void => { for (const dependent of graph.dependentsOf(current)) { @@ -608,6 +610,7 @@ export class Orchestrator extends Context.Service< start: () => Effect.gen(function* () { for (const def of graph.startOrder) { + launchedServices.add(def.name); yield* FiberMap.run(fibers, def.name, runServiceSafe(def)); } }), @@ -621,11 +624,19 @@ export class Orchestrator extends Context.Service< const order = graph.startOrderFor(name); for (const d of order) { const svc = services.get(d.name); - const status = - svc === undefined ? undefined : SubscriptionRef.getUnsafe(svc.state).status; + const state = svc === undefined ? undefined : SubscriptionRef.getUnsafe(svc.state); + const status = state?.status; + if ( + (d.restart ?? defaults.restart) === "no" && + status === "Stopped" && + state?.exitCode === 0 + ) { + continue; + } if (status === "Stopped" || status === "Failed") { yield* resetService(d.name); } + launchedServices.add(d.name); yield* FiberMap.run(fibers, d.name, runServiceSafe(d), { onlyIfMissing: true }); } }), @@ -681,6 +692,7 @@ export class Orchestrator extends Context.Service< }), ), ); + launchedServices.clear(); }), stopService: (name: string) => @@ -694,6 +706,7 @@ export class Orchestrator extends Context.Service< yield* FiberMap.remove(fibers, name); // Force Stopped if still in Stopping (fiber was interrupted before ProcessExited) yield* sendEvent(name, { _tag: "ProcessExited", exitCode: 143 }); + launchedServices.delete(name); }), restartService: (name: string) => @@ -711,6 +724,7 @@ export class Orchestrator extends Context.Service< yield* resetService(affectedDef.name); } for (const affectedDef of affected) { + launchedServices.add(affectedDef.name); yield* FiberMap.run(fibers, affectedDef.name, runServiceSafe(affectedDef)); } }), diff --git a/packages/process-compose/src/Orchestrator.unit.test.ts b/packages/process-compose/src/Orchestrator.unit.test.ts index ddf55f40d6..49da134235 100644 --- a/packages/process-compose/src/Orchestrator.unit.test.ts +++ b/packages/process-compose/src/Orchestrator.unit.test.ts @@ -526,6 +526,34 @@ describe("Orchestrator", () => { }).pipe(Effect.provide(layer), Effect.scoped); }); + it.live("startService does not rerun completed one-shot dependencies", () => { + const { layer, proc } = setupOrchestrator( + [ + svc("postgres-init", { restart: "no" }), + svc("api", { + dependencies: [{ service: "postgres-init", condition: "completed" }], + }), + ], + { + perService: { + api: { exitDelay: "5 seconds" }, + "postgres-init": { exitDelay: "10 millis" }, + }, + }, + ); + return Effect.gen(function* () { + const orc = yield* Orchestrator; + yield* orc.startService("api"); + yield* proc.waitForSpawn("api"); + yield* waitForStopped(orc, "postgres-init"); + + yield* orc.startService("api"); + yield* Effect.sleep(Duration.millis(50)); + + expect(proc.spawned.map((s) => s.command)).toEqual(["postgres-init", "api"]); + }).pipe(Effect.provide(layer), Effect.scoped); + }); + it.live("restartService stops and restarts a service", () => { const { layer, proc } = setupOrchestrator([svc("a")], { exitDelay: "5 seconds", @@ -600,6 +628,49 @@ describe("Orchestrator", () => { }).pipe(Effect.provide(layer), Effect.scoped); }); + it.live("restartService includes launched dependents that are still pending", () => { + let dbReady = false; + const { layer, proc } = setupOrchestrator( + [ + svc("db", { + healthCheck: { + probe: { + _tag: "Exec", + command: "db-ready", + args: [], + }, + periodSeconds: 0.01, + }, + }), + svc("api", { + dependencies: [{ service: "db", condition: "healthy" }], + }), + ], + { + perService: { + api: { exitDelay: "5 seconds" }, + db: { exitDelay: "5 seconds" }, + "db-ready": { getExitCode: () => (dbReady ? 0 : 1) }, + }, + }, + ); + return Effect.gen(function* () { + const orc = yield* Orchestrator; + yield* orc.start(); + yield* proc.waitForSpawn("db"); + expect((yield* orc.getState("api")).status).toBe("Pending"); + + yield* orc.restartService("db"); + dbReady = true; + yield* proc.waitForSpawn("api"); + + const spawnedServices = proc.spawned + .map((s) => s.command) + .filter((command) => command === "api" || command === "db"); + expect(spawnedServices).toEqual(["db", "db", "api"]); + }).pipe(Effect.provide(layer), Effect.scoped); + }); + it.live("updateServiceDefinition restarts with the updated definition", () => { const { layer, proc } = setupOrchestrator([svc("a")], { exitDelay: "5 seconds", From c17517d5f1e709ddf4838f7204c003341d0e1357 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Wed, 8 Jul 2026 10:34:02 +0200 Subject: [PATCH 34/38] chore(cli): sync lockfile with current base --- pnpm-lock.yaml | 178 ++++++++++++++++++++++---------------------- pnpm-workspace.yaml | 4 +- 2 files changed, 91 insertions(+), 91 deletions(-) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index d62418845d..d029e46f75 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -37,8 +37,8 @@ catalogs: specifier: ^1.3.14 version: 1.3.14 '@typescript/native-preview': - specifier: 7.0.0-dev.20260629.1 - version: 7.0.0-dev.20260629.1 + specifier: 7.0.0-dev.20260630.1 + version: 7.0.0-dev.20260630.1 '@vitest/coverage-istanbul': specifier: ^4.1.9 version: 4.1.9 @@ -58,8 +58,8 @@ catalogs: specifier: ^1.72.0 version: 1.73.0 oxlint-tsgolint: - specifier: ^0.23.0 - version: 0.23.0 + specifier: ^0.24.0 + version: 0.24.0 tldts: specifier: ^7.4.5 version: 7.4.5 @@ -155,7 +155,7 @@ importers: version: 19.2.17 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260629.1 + version: 7.0.0-dev.20260630.1 '@vercel/detect-agent': specifier: ^1.2.3 version: 1.2.3 @@ -185,10 +185,10 @@ importers: version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.73.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.24.0) oxlint-tsgolint: specifier: 'catalog:' - version: 0.23.0 + version: 0.24.0 pg: specifier: ^8.22.0 version: 8.22.0 @@ -259,7 +259,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260629.1 + version: 7.0.0-dev.20260630.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -271,10 +271,10 @@ importers: version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.73.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.24.0) oxlint-tsgolint: specifier: 'catalog:' - version: 0.23.0 + version: 0.24.0 vitest: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) @@ -339,7 +339,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260629.1 + version: 7.0.0-dev.20260630.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -351,10 +351,10 @@ importers: version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.73.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.24.0) oxlint-tsgolint: specifier: 'catalog:' - version: 0.23.0 + version: 0.24.0 vitest: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) @@ -381,7 +381,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260629.1 + version: 7.0.0-dev.20260630.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -393,10 +393,10 @@ importers: version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.73.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.24.0) oxlint-tsgolint: specifier: 'catalog:' - version: 0.23.0 + version: 0.24.0 vitest: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) @@ -431,7 +431,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260629.1 + version: 7.0.0-dev.20260630.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -443,10 +443,10 @@ importers: version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.73.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.24.0) oxlint-tsgolint: specifier: 'catalog:' - version: 0.23.0 + version: 0.24.0 vitest: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) @@ -465,7 +465,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260629.1 + version: 7.0.0-dev.20260630.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -477,10 +477,10 @@ importers: version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.73.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.24.0) oxlint-tsgolint: specifier: 'catalog:' - version: 0.23.0 + version: 0.24.0 vitest: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) @@ -505,7 +505,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260629.1 + version: 7.0.0-dev.20260630.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -517,10 +517,10 @@ importers: version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.73.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.24.0) oxlint-tsgolint: specifier: 'catalog:' - version: 0.23.0 + version: 0.24.0 vitest: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) @@ -557,7 +557,7 @@ importers: version: 1.3.14 '@typescript/native-preview': specifier: 'catalog:' - version: 7.0.0-dev.20260629.1 + version: 7.0.0-dev.20260630.1 '@vitest/coverage-istanbul': specifier: 'catalog:' version: 4.1.9(vitest@4.1.9) @@ -569,10 +569,10 @@ importers: version: 0.57.0 oxlint: specifier: 'catalog:' - version: 1.73.0(oxlint-tsgolint@0.23.0) + version: 1.73.0(oxlint-tsgolint@0.24.0) oxlint-tsgolint: specifier: 'catalog:' - version: 0.23.0 + version: 0.24.0 vitest: specifier: 'catalog:' version: 4.1.9(@types/node@26.0.1)(@vitest/coverage-istanbul@4.1.9)(vite@8.0.14(@types/node@26.0.1)(esbuild@0.28.1)(jiti@2.7.0)(yaml@2.9.0)) @@ -1886,33 +1886,33 @@ packages: cpu: [x64] os: [win32] - '@oxlint-tsgolint/darwin-arm64@0.23.0': - resolution: {integrity: sha512-gOs9PVr2wEg4ox9z0aJo+RKhhImW86YL5N6yav8BK/rgPsIrwN/igSZ+pbRr723NFvUNKde9fgMhRA6JrXAOZw==} + '@oxlint-tsgolint/darwin-arm64@0.24.0': + resolution: {integrity: sha512-C2uMmwK5Bc4ri4ysZ6sA8Rcu+A5zBQTp6ml2u0CLLbRZp4kMFPV3yWk8B5DK9Aw7y9bbjogIm75tUwGLFzlsYQ==} cpu: [arm64] os: [darwin] - '@oxlint-tsgolint/darwin-x64@0.23.0': - resolution: {integrity: sha512-kjJ8B+7n4tB9VJdxS5A9GdJt6/bYpzbu4lXp2uO1S3sRmCB5gDEABlGoiePNApRWaW+xqL4b4xgiE727jSLhuA==} + '@oxlint-tsgolint/darwin-x64@0.24.0': + resolution: {integrity: sha512-Wgvt/1lRbDxmoNqWQKKcL+UIiqLmdJ+EWLpQa1qzoNVAfNB0PJpa82/8dH1twT/3rSs4zrP5TXPWl4juB71WuQ==} cpu: [x64] os: [darwin] - '@oxlint-tsgolint/linux-arm64@0.23.0': - resolution: {integrity: sha512-6dCZuKNu135seMXilkRk9SpCx6i1XgmiipYGalLij5WVRX6ZYS8c4xI7preN/zv9fCXhsQclTIMDu2Y/cytTjw==} + '@oxlint-tsgolint/linux-arm64@0.24.0': + resolution: {integrity: sha512-PB1rxII7KV83+ASY4sSkXtqvpij6ME66+QCRL49uksi/ofs2Rf/UVboYr095n0Rkbl2wgvlsHGl6DHC361jQUQ==} cpu: [arm64] os: [linux] - '@oxlint-tsgolint/linux-x64@0.23.0': - resolution: {integrity: sha512-3bdilnyA7kmSTjK27rvjIjSxL5SIg3wt7vwNiRkouWB83ytssyKnuGvxSYJxgMEmFpSutzaBzcCUM2jDtPGcgA==} + '@oxlint-tsgolint/linux-x64@0.24.0': + resolution: {integrity: sha512-xcz3CxKmjTQLREtE/UShh+ruWmm9nAb7UM9zKcD65BStiuYgOakAKkPHl4YS5DztpVcDrE0+HqbOolTlRKYWmw==} cpu: [x64] os: [linux] - '@oxlint-tsgolint/win32-arm64@0.23.0': - resolution: {integrity: sha512-j+OEp44SVYiQ+ZD+uttsX7u6L9SvmbbQ77SO1pSFCcJlsVMeCk8qZsjhKfGKuT/jIA+ipOJMVs/+pqUfObBWNw==} + '@oxlint-tsgolint/win32-arm64@0.24.0': + resolution: {integrity: sha512-A2i6ZGBec3i20S7RaxkgHc6r3HYtD5Mn7j/mb22NkTz14u0JuudvTu6JggAnbGMcv8+dBKQI//EasxSPJLD8pw==} cpu: [arm64] os: [win32] - '@oxlint-tsgolint/win32-x64@0.23.0': - resolution: {integrity: sha512-5MyjFuqf+g8OUPJBSGWHJtmoWnzFJYyOg4To9WMQshZYEWig/vtu7JtJ03VWnzHv9LJkAUeApY0gVCOywFR/iQ==} + '@oxlint-tsgolint/win32-x64@0.24.0': + resolution: {integrity: sha512-0ZbGd9qRB6zs82moekaKdEvncRANq49EAwfNX62JpTS46feXUhKAuoyVDvZMj6Rywejylrmmu79Wo6faYCo4Ew==} cpu: [x64] os: [win32] @@ -2898,50 +2898,50 @@ packages: '@types/ws@8.18.1': resolution: {integrity: sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg==} - '@typescript/native-preview-darwin-arm64@7.0.0-dev.20260629.1': - resolution: {integrity: sha512-wXRExZJweYoTzE4atRR7T5HwKJYkl6/KHxON0eF0iy2fvgLXDlyq4AQqhmV8mMx10PQKc/4sNbfhD4kjWWvm8A==} + '@typescript/native-preview-darwin-arm64@7.0.0-dev.20260630.1': + resolution: {integrity: sha512-zo1TYD32r+MKOTnzhB1YGr2Jrw14tzGR/rpfr6hRMDz0kV9k2CWVvtOYOmOtRdmuO4KWZcWtVX13QyaPGhA8Hw==} engines: {node: '>=16.20.0'} cpu: [arm64] os: [darwin] - '@typescript/native-preview-darwin-x64@7.0.0-dev.20260629.1': - resolution: {integrity: sha512-xkL/tETzUuDxfRkk2rD0/CjjjGLyOLHeE103T52rpgj0VNSXMnn7tTiw+TSdxnS61b8ImOrWgQJujhPFTp0jng==} + '@typescript/native-preview-darwin-x64@7.0.0-dev.20260630.1': + resolution: {integrity: sha512-57RYyZQQ6+drfu+CagVdqUvQwNesvuu/rJb/au66jv+figfpDOugD0S6ruQl1SxpFFfr0lCny3KEplsBdqFDEg==} engines: {node: '>=16.20.0'} cpu: [x64] os: [darwin] - '@typescript/native-preview-linux-arm64@7.0.0-dev.20260629.1': - resolution: {integrity: sha512-y4ey82krq4iLNHML4G1dUObBWY6gNAjVqaUHFDv7+LBu5bfAJy8VClBaLtAJce3siQQeB0/4SkN4XsAa0oZktQ==} + '@typescript/native-preview-linux-arm64@7.0.0-dev.20260630.1': + resolution: {integrity: sha512-QQxNyZ9rVbE6lUqctrrRiTtA5Z0w2FFBy8SkiP/eDkuRy2WXrVpMQYntNn6d4nO1nWTmWJR1z/CS+H2rsrl8gg==} engines: {node: '>=16.20.0'} cpu: [arm64] os: [linux] - '@typescript/native-preview-linux-arm@7.0.0-dev.20260629.1': - resolution: {integrity: sha512-QlTxO+O6yELc0NYZZYIbncZhkhw+K/vBoVg+mKipbEIK5b1E6cwW7ivWIrDp1FfssQ0Wu7zxincPappoci7KNw==} + '@typescript/native-preview-linux-arm@7.0.0-dev.20260630.1': + resolution: {integrity: sha512-FLJSSt3bUmRpzWlr5cDE+td40FoDy/MT+VDYk4JGouisJo7g7GPjuF+fYOfsKI/RbD9f6gDFTyFPAoTLVVR/9g==} engines: {node: '>=16.20.0'} cpu: [arm] os: [linux] - '@typescript/native-preview-linux-x64@7.0.0-dev.20260629.1': - resolution: {integrity: sha512-ckb4HyugC5beDpOMPOVpEx1H3l2Z2RgsGPxFOLGMU6LLIHQwlCaSdRjSfH8Hq8yffPgKuotTQLdA89cP1IC8xQ==} + '@typescript/native-preview-linux-x64@7.0.0-dev.20260630.1': + resolution: {integrity: sha512-WqNPvUBngGfnkunqn3IuIkkF6PA/HPSSbVucolO7stc9DxWZqfRals6/C8RTvuQaif9qt+bcY9U6lLXuDFyClA==} engines: {node: '>=16.20.0'} cpu: [x64] os: [linux] - '@typescript/native-preview-win32-arm64@7.0.0-dev.20260629.1': - resolution: {integrity: sha512-Ktgepu9p0blqrbiryzYrw11ddnbESXPdeGKURDG/wXESoHQETsMNxKWhqAo587ItNnWjKOBFxoEM22eP9OzKnw==} + '@typescript/native-preview-win32-arm64@7.0.0-dev.20260630.1': + resolution: {integrity: sha512-awATkrGGm2L1IraQgaw21VXWtAqv79DJ57No/J65Y+bL8InOVR2zu+zCH+5E44m8bh9IXwnShI7cAdvp02WNEw==} engines: {node: '>=16.20.0'} cpu: [arm64] os: [win32] - '@typescript/native-preview-win32-x64@7.0.0-dev.20260629.1': - resolution: {integrity: sha512-b2wmeIpFJs/peej3NG26320ya+iYQz21JzFYDrnvtsEeR/g66rB9uvp4Owo4J1AG/BcmRWC/nuwrBy3Jdb+UDA==} + '@typescript/native-preview-win32-x64@7.0.0-dev.20260630.1': + resolution: {integrity: sha512-M/+P66Gsss/CANV+MkKVFeNi9jljYYR3N2COf/WrVYcDwrHyiYHZYExfTw2cEbbkH8LcawXAekp9d23bevBR4Q==} engines: {node: '>=16.20.0'} cpu: [x64] os: [win32] - '@typescript/native-preview@7.0.0-dev.20260629.1': - resolution: {integrity: sha512-KImWFkxGUa/V78bUEAgzeJQT+6wKQXDDYHHrOavof8wnbMCpj86KuPNyBIPQMtoHiz2If8aNTums2m50oE3oqw==} + '@typescript/native-preview@7.0.0-dev.20260630.1': + resolution: {integrity: sha512-5OTvy2YpoDsOwAPqj4LkI4mrlAtc3mRzNdHAPp/1JWUvsKSRTzqAB2JPVD4qHUkAd9qY89yb758kc7fGyn3gbw==} engines: {node: '>=16.20.0'} hasBin: true @@ -5420,8 +5420,8 @@ packages: vite-plus: optional: true - oxlint-tsgolint@0.23.0: - resolution: {integrity: sha512-3mBv3CoPbh8dFbzfDGIWa2ytZjn2v+3EX4aKRXjIhsoGFzG8GCjfRirz3rwZf1wYbZzsNLTSgpw8VjQuWdp/jA==} + oxlint-tsgolint@0.24.0: + resolution: {integrity: sha512-giCk5sEvG02d5tzPmFMX3hem8ndzEEu1xvGYS5OwNfO2WGl6ZVxt5LjE0yiMDoz94INI7XkXwgFAQiydPvVHDw==} hasBin: true oxlint@1.73.0: @@ -7892,22 +7892,22 @@ snapshots: '@oxfmt/binding-win32-x64-msvc@0.57.0': optional: true - '@oxlint-tsgolint/darwin-arm64@0.23.0': + '@oxlint-tsgolint/darwin-arm64@0.24.0': optional: true - '@oxlint-tsgolint/darwin-x64@0.23.0': + '@oxlint-tsgolint/darwin-x64@0.24.0': optional: true - '@oxlint-tsgolint/linux-arm64@0.23.0': + '@oxlint-tsgolint/linux-arm64@0.24.0': optional: true - '@oxlint-tsgolint/linux-x64@0.23.0': + '@oxlint-tsgolint/linux-x64@0.24.0': optional: true - '@oxlint-tsgolint/win32-arm64@0.23.0': + '@oxlint-tsgolint/win32-arm64@0.24.0': optional: true - '@oxlint-tsgolint/win32-x64@0.23.0': + '@oxlint-tsgolint/win32-x64@0.24.0': optional: true '@oxlint/binding-android-arm-eabi@1.73.0': @@ -8782,36 +8782,36 @@ snapshots: dependencies: '@types/node': 26.0.1 - '@typescript/native-preview-darwin-arm64@7.0.0-dev.20260629.1': + '@typescript/native-preview-darwin-arm64@7.0.0-dev.20260630.1': optional: true - '@typescript/native-preview-darwin-x64@7.0.0-dev.20260629.1': + '@typescript/native-preview-darwin-x64@7.0.0-dev.20260630.1': optional: true - '@typescript/native-preview-linux-arm64@7.0.0-dev.20260629.1': + '@typescript/native-preview-linux-arm64@7.0.0-dev.20260630.1': optional: true - '@typescript/native-preview-linux-arm@7.0.0-dev.20260629.1': + '@typescript/native-preview-linux-arm@7.0.0-dev.20260630.1': optional: true - '@typescript/native-preview-linux-x64@7.0.0-dev.20260629.1': + '@typescript/native-preview-linux-x64@7.0.0-dev.20260630.1': optional: true - '@typescript/native-preview-win32-arm64@7.0.0-dev.20260629.1': + '@typescript/native-preview-win32-arm64@7.0.0-dev.20260630.1': optional: true - '@typescript/native-preview-win32-x64@7.0.0-dev.20260629.1': + '@typescript/native-preview-win32-x64@7.0.0-dev.20260630.1': optional: true - '@typescript/native-preview@7.0.0-dev.20260629.1': + '@typescript/native-preview@7.0.0-dev.20260630.1': optionalDependencies: - '@typescript/native-preview-darwin-arm64': 7.0.0-dev.20260629.1 - '@typescript/native-preview-darwin-x64': 7.0.0-dev.20260629.1 - '@typescript/native-preview-linux-arm': 7.0.0-dev.20260629.1 - '@typescript/native-preview-linux-arm64': 7.0.0-dev.20260629.1 - '@typescript/native-preview-linux-x64': 7.0.0-dev.20260629.1 - '@typescript/native-preview-win32-arm64': 7.0.0-dev.20260629.1 - '@typescript/native-preview-win32-x64': 7.0.0-dev.20260629.1 + '@typescript/native-preview-darwin-arm64': 7.0.0-dev.20260630.1 + '@typescript/native-preview-darwin-x64': 7.0.0-dev.20260630.1 + '@typescript/native-preview-linux-arm': 7.0.0-dev.20260630.1 + '@typescript/native-preview-linux-arm64': 7.0.0-dev.20260630.1 + '@typescript/native-preview-linux-x64': 7.0.0-dev.20260630.1 + '@typescript/native-preview-win32-arm64': 7.0.0-dev.20260630.1 + '@typescript/native-preview-win32-x64': 7.0.0-dev.20260630.1 '@ungap/structured-clone@1.3.2': {} @@ -11833,16 +11833,16 @@ snapshots: '@oxfmt/binding-win32-ia32-msvc': 0.57.0 '@oxfmt/binding-win32-x64-msvc': 0.57.0 - oxlint-tsgolint@0.23.0: + oxlint-tsgolint@0.24.0: optionalDependencies: - '@oxlint-tsgolint/darwin-arm64': 0.23.0 - '@oxlint-tsgolint/darwin-x64': 0.23.0 - '@oxlint-tsgolint/linux-arm64': 0.23.0 - '@oxlint-tsgolint/linux-x64': 0.23.0 - '@oxlint-tsgolint/win32-arm64': 0.23.0 - '@oxlint-tsgolint/win32-x64': 0.23.0 - - oxlint@1.73.0(oxlint-tsgolint@0.23.0): + '@oxlint-tsgolint/darwin-arm64': 0.24.0 + '@oxlint-tsgolint/darwin-x64': 0.24.0 + '@oxlint-tsgolint/linux-arm64': 0.24.0 + '@oxlint-tsgolint/linux-x64': 0.24.0 + '@oxlint-tsgolint/win32-arm64': 0.24.0 + '@oxlint-tsgolint/win32-x64': 0.24.0 + + oxlint@1.73.0(oxlint-tsgolint@0.24.0): optionalDependencies: '@oxlint/binding-android-arm-eabi': 1.73.0 '@oxlint/binding-android-arm64': 1.73.0 @@ -11863,7 +11863,7 @@ snapshots: '@oxlint/binding-win32-arm64-msvc': 1.73.0 '@oxlint/binding-win32-ia32-msvc': 1.73.0 '@oxlint/binding-win32-x64-msvc': 1.73.0 - oxlint-tsgolint: 0.23.0 + oxlint-tsgolint: 0.24.0 p-cancelable@2.1.1: {} diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index d42a59a931..eb8f8de15a 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -22,14 +22,14 @@ catalog: "@swc/core": "^1.15.43" "@tsconfig/bun": "^1.0.10" "@types/bun": "^1.3.14" - "@typescript/native-preview": "7.0.0-dev.20260629.1" + "@typescript/native-preview": "7.0.0-dev.20260630.1" "@vitest/coverage-istanbul": "^4.1.9" "effect": "4.0.0-beta.93" "knip": "^6.17.1" "nx": "^23.0.0" "oxfmt": "^0.57.0" "oxlint": "^1.72.0" - "oxlint-tsgolint": "^0.23.0" + "oxlint-tsgolint": "^0.24.0" "tldts": "^7.4.5" "vitest": "^4.1.9" From 6f5953c845e14f3ea4f0a3215dd24fdc1a6f93d4 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Wed, 8 Jul 2026 10:43:44 +0200 Subject: [PATCH 35/38] test(cli): relax core test timeout for ci --- apps/cli/vitest.config.ts | 2 ++ 1 file changed, 2 insertions(+) diff --git a/apps/cli/vitest.config.ts b/apps/cli/vitest.config.ts index a8c53dd4e5..333aca90db 100644 --- a/apps/cli/vitest.config.ts +++ b/apps/cli/vitest.config.ts @@ -45,6 +45,7 @@ export default defineConfig({ test: { name: "unit", include: ["**/*.unit.test.ts"], + testTimeout: 30_000, }, }, { @@ -52,6 +53,7 @@ export default defineConfig({ test: { name: "integration", include: ["**/*.integration.test.ts"], + testTimeout: 30_000, }, }, { From f69ab959edfe08307a8856b49b2fcad9a18e8979 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Wed, 8 Jul 2026 12:11:45 +0200 Subject: [PATCH 36/38] fix(fleet): harden micro stack lifecycle --- packages/fleet/src/Fleet.ts | 71 +++--- packages/fleet/src/Fleet.unit.test.ts | 27 ++- packages/fleet/src/PodManifest.ts | 4 +- packages/fleet/src/PodRegistry.ts | 210 +++++++++++++++++- packages/fleet/src/PodRegistry.unit.test.ts | 39 +++- packages/fleet/src/PortRegistry.ts | 84 +++++-- packages/fleet/src/PortRegistry.unit.test.ts | 78 ++++--- .../fleet/src/Provisioner.integration.test.ts | 2 +- packages/fleet/src/Provisioner.ts | 49 +++- packages/fleet/src/Provisioner.unit.test.ts | 45 +++- packages/fleet/src/TemplateStore.ts | 9 +- packages/process-compose/src/Orchestrator.ts | 9 +- .../src/Orchestrator.unit.test.ts | 41 ++++ .../src/DaemonServer.integration.test.ts | 16 +- packages/stack/src/DaemonServer.ts | 6 + packages/stack/src/StackBuilder.ts | 2 + .../stack/src/StackLifecycleCoordinator.ts | 5 +- .../StackLifecycleCoordinator.unit.test.ts | 30 +++ packages/stack/src/createStack.ts | 4 +- packages/stack/src/index.ts | 2 + packages/stack/src/services/pooler.ts | 3 +- packages/stack/src/services/postgres-init.ts | 1 + .../stack/src/services/services.unit.test.ts | 14 ++ 23 files changed, 646 insertions(+), 105 deletions(-) diff --git a/packages/fleet/src/Fleet.ts b/packages/fleet/src/Fleet.ts index f207495900..a6398855b5 100644 --- a/packages/fleet/src/Fleet.ts +++ b/packages/fleet/src/Fleet.ts @@ -56,13 +56,8 @@ interface WarmPod { const DB_PASSWORD = resolvePostgresPassword(); // Internal (in-process stack) ports are derived from the externally-visible, -// PortRegistry-owned port by a fixed +10_000 offset so the two ranges never -// collide. PortRegistry hands out ports starting at 55_000, so this scheme -// only stays valid while every allocated external port is < 55_536; beyond -// that the derived internal port would exceed the 16-bit port ceiling -// (65_535). Fine for the pod counts this phase targets; a later phase should -// allocate internal ports dynamically (e.g. port 0) if the fleet needs to -// scale past that. +// PortRegistry-owned ports by a fixed lower offset so the proxy-owned public +// range and the stack-owned internal range never collide. const INTERNAL_PORT_OFFSET = 10_000; /** @@ -114,7 +109,7 @@ export async function createFleet(opts: FleetOptions = {}): Promise const templates = new TemplateStore(join(root, "templates")); const pods = new PodRegistry(join(root, "pods")); const ports = await PortRegistry.load(join(root, "fleet-state.json")); - const provisioner = new Provisioner({ templates, pods, ports }); + const provisioner = new Provisioner({ templates, pods, ports, postgresPassword: DB_PASSWORD }); const states = new Map(); const warm = new Map(); @@ -138,12 +133,14 @@ export async function createFleet(opts: FleetOptions = {}): Promise const dbUrl = (manifest: PodManifest): string => postgresConnectionUrl({ user: "postgres", - password: DB_PASSWORD, + password: manifest.postgresPassword, host: "127.0.0.1", port: manifest.ports.dbPort, database: "postgres", }); + const internalPort = (externalPort: number): number => externalPort - INTERNAL_PORT_OFFSET; + const runPidFile = (id: string): string => join(pods.podDir(id), "run.pid"); async function withPodLocks(ids: ReadonlyArray, body: () => Promise): Promise { @@ -175,39 +172,41 @@ export async function createFleet(opts: FleetOptions = {}): Promise const manifest = await pods.read(id); if (manifest === undefined) throw new Error(`unknown pod: ${id}`); states.set(id, "waking"); - const internalDbPort = manifest.ports.dbPort + INTERNAL_PORT_OFFSET; + const internalDbPort = internalPort(manifest.ports.dbPort); const stack = await createStack({ + mode: "native", stackRoot: join(pods.podDir(id), "stack"), - port: manifest.ports.apiPort + INTERNAL_PORT_OFFSET, + port: internalPort(manifest.ports.apiPort), lazyServices: true, postgres: { dataDir: pods.dataDir(id), version: manifest.versions.postgres, port: internalDbPort, + password: manifest.postgresPassword, provisioned: true, profile: "micro", }, postgrest: - manifest.services.postgrest === true ? { version: manifest.versions.postgrest } : false, - auth: manifest.services.auth === true ? { version: manifest.versions.auth } : false, - realtime: - manifest.services.realtime === true ? { version: manifest.versions.realtime } : false, - edgeRuntime: - manifest.services["edge-runtime"] === true - ? { version: manifest.versions["edge-runtime"] } + manifest.services.postgrest === true + ? { + version: manifest.versions.postgrest, + port: internalPort(manifest.ports.postgrestPort), + } + : false, + auth: + manifest.services.auth === true + ? { version: manifest.versions.auth, port: internalPort(manifest.ports.authPort) } : false, - storage: - manifest.services.storage === true ? { version: manifest.versions.storage } : false, - imgproxy: - manifest.services.imgproxy === true ? { version: manifest.versions.imgproxy } : false, - mailpit: - manifest.services.mailpit === true ? { version: manifest.versions.mailpit } : false, - pgmeta: manifest.services.pgmeta === true ? { version: manifest.versions.pgmeta } : false, - studio: manifest.services.studio === true ? { version: manifest.versions.studio } : false, - analytics: - manifest.services.analytics === true ? { version: manifest.versions.analytics } : false, - vector: manifest.services.vector === true ? { version: manifest.versions.vector } : false, - pooler: manifest.services.pooler === true ? { version: manifest.versions.pooler } : false, + realtime: false, + edgeRuntime: false, + storage: false, + imgproxy: false, + mailpit: false, + pgmeta: false, + studio: false, + analytics: false, + vector: false, + pooler: false, functions: false, }); try { @@ -281,13 +280,13 @@ export async function createFleet(opts: FleetOptions = {}): Promise // process-group leader, so `reapStalePostmaster` can reliably signal the // whole tree via `-pid`. Phase 1 only ever runs postgres under a fleet pod // (postgres-only ready gate; no HTTP edge yet), so this fully covers what - // a stale pod could have left behind. The pod's ports are also re-seeded - // into the freshly loaded PortRegistry from its manifest, which is the - // mechanism `restore()` exists for (recovering from a quarantined/corrupt - // port-state file). + // a stale pod could have left behind. The pod port registry is reconciled + // to exactly the valid manifests on disk so stale allocations from deleted + // or skipped pods are pruned during startup. try { - for (const manifest of await pods.list()) { - await ports.restore(manifest.id, manifest.ports); + const manifests = await pods.list(); + await ports.reconcile(new Map(manifests.map((manifest) => [manifest.id, manifest.ports]))); + for (const manifest of manifests) { await reapStalePostmaster(pods.dataDir(manifest.id)); await rm(runPidFile(manifest.id), { force: true }); await registerEdge(manifest); diff --git a/packages/fleet/src/Fleet.unit.test.ts b/packages/fleet/src/Fleet.unit.test.ts index fe4c73cf11..ece6567248 100644 --- a/packages/fleet/src/Fleet.unit.test.ts +++ b/packages/fleet/src/Fleet.unit.test.ts @@ -3,6 +3,7 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; +import type { AllocatedPorts } from "@supabase/stack"; import { createFleet } from "./Fleet.ts"; import { PodRegistry } from "./PodRegistry.ts"; import type { PodManifest } from "./PodManifest.ts"; @@ -28,13 +29,37 @@ function expectPortAvailable(port: number): Promise { }); } +function ports(dbPort: number, apiPort: number): AllocatedPorts { + return { + dbPort, + apiPort, + authPort: apiPort + 1, + postgrestPort: apiPort + 2, + postgrestAdminPort: apiPort + 3, + edgeRuntimePort: apiPort + 4, + edgeRuntimeInspectorPort: apiPort + 5, + realtimePort: apiPort + 6, + storagePort: apiPort + 7, + imgproxyPort: apiPort + 8, + mailpitPort: apiPort + 9, + mailpitSmtpPort: apiPort + 10, + mailpitPop3Port: apiPort + 11, + pgmetaPort: apiPort + 12, + studioPort: apiPort + 13, + analyticsPort: apiPort + 14, + poolerPort: apiPort + 15, + poolerApiPort: apiPort + 16, + }; +} + function manifest(id: string, dbPort: number, apiPort: number): PodManifest { return { id, versions: { postgres: "17.6.1.143" }, services: {}, flags: { supautils: false }, - ports: { dbPort, apiPort }, + ports: ports(dbPort, apiPort), + postgresPassword: "postgres", createdAt: "2026-07-08T00:00:00.000Z", }; } diff --git a/packages/fleet/src/PodManifest.ts b/packages/fleet/src/PodManifest.ts index 0116112482..74ad9c9f6f 100644 --- a/packages/fleet/src/PodManifest.ts +++ b/packages/fleet/src/PodManifest.ts @@ -1,6 +1,7 @@ import { createHash } from "node:crypto"; import { fillServiceVersionManifest, + type AllocatedPorts, type ServiceName, type VersionManifest, } from "@supabase/stack"; @@ -10,7 +11,8 @@ export interface PodManifest { readonly versions: Partial; readonly services: Partial>; readonly flags: { readonly supautils: boolean }; - readonly ports: { readonly dbPort: number; readonly apiPort: number }; + readonly ports: AllocatedPorts; + readonly postgresPassword: string; readonly createdAt: string; } diff --git a/packages/fleet/src/PodRegistry.ts b/packages/fleet/src/PodRegistry.ts index de63236aee..a510912c6f 100644 --- a/packages/fleet/src/PodRegistry.ts +++ b/packages/fleet/src/PodRegistry.ts @@ -1,8 +1,32 @@ -import { mkdir, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { mkdir, readdir, readFile, rename, rm, writeFile } from "node:fs/promises"; import { basename, join } from "node:path"; +import { SERVICE_NAMES } from "@supabase/stack"; +import type { AllocatedPorts, ServiceName } from "@supabase/stack"; import type { PodManifest } from "./PodManifest.ts"; const POD_ID_RE = /^[A-Za-z0-9][A-Za-z0-9._-]*$/; +const SERVICE_NAME_SET = new Set(SERVICE_NAMES); + +function serviceNameFrom(value: string): ServiceName | undefined { + switch (value) { + case "postgres": + case "postgrest": + case "auth": + case "edge-runtime": + case "realtime": + case "storage": + case "imgproxy": + case "mailpit": + case "pgmeta": + case "studio": + case "analytics": + case "vector": + case "pooler": + return value; + default: + return undefined; + } +} function isValidPodId(id: string): boolean { return POD_ID_RE.test(id) && basename(id) === id; @@ -15,6 +39,175 @@ function validatePodId(id: string): string { return id; } +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function isFleetPort(value: unknown): value is number { + return typeof value === "number" && Number.isInteger(value) && value > 10_000 && value <= 65_535; +} + +function parsePorts(value: unknown): AllocatedPorts | undefined { + if (!isRecord(value)) return undefined; + if ( + !isFleetPort(value.dbPort) || + !isFleetPort(value.apiPort) || + !isFleetPort(value.authPort) || + !isFleetPort(value.postgrestPort) || + !isFleetPort(value.postgrestAdminPort) || + !isFleetPort(value.edgeRuntimePort) || + !isFleetPort(value.edgeRuntimeInspectorPort) || + !isFleetPort(value.realtimePort) || + !isFleetPort(value.storagePort) || + !isFleetPort(value.imgproxyPort) || + !isFleetPort(value.mailpitPort) || + !isFleetPort(value.mailpitSmtpPort) || + !isFleetPort(value.mailpitPop3Port) || + !isFleetPort(value.pgmetaPort) || + !isFleetPort(value.studioPort) || + !isFleetPort(value.analyticsPort) || + !isFleetPort(value.poolerPort) || + !isFleetPort(value.poolerApiPort) + ) { + return undefined; + } + return { + dbPort: value.dbPort, + apiPort: value.apiPort, + authPort: value.authPort, + postgrestPort: value.postgrestPort, + postgrestAdminPort: value.postgrestAdminPort, + edgeRuntimePort: value.edgeRuntimePort, + edgeRuntimeInspectorPort: value.edgeRuntimeInspectorPort, + realtimePort: value.realtimePort, + storagePort: value.storagePort, + imgproxyPort: value.imgproxyPort, + mailpitPort: value.mailpitPort, + mailpitSmtpPort: value.mailpitSmtpPort, + mailpitPop3Port: value.mailpitPop3Port, + pgmetaPort: value.pgmetaPort, + studioPort: value.studioPort, + analyticsPort: value.analyticsPort, + poolerPort: value.poolerPort, + poolerApiPort: value.poolerApiPort, + }; +} + +function parseVersions(value: unknown): PodManifest["versions"] | undefined { + if (!isRecord(value)) return undefined; + let postgres: string | undefined; + let postgrest: string | undefined; + let auth: string | undefined; + let edgeRuntime: string | undefined; + let realtime: string | undefined; + let storage: string | undefined; + let imgproxy: string | undefined; + let mailpit: string | undefined; + let pgmeta: string | undefined; + let studio: string | undefined; + let analytics: string | undefined; + let vector: string | undefined; + let pooler: string | undefined; + for (const [name, version] of Object.entries(value)) { + const service = serviceNameFrom(name); + if (service === undefined || typeof version !== "string") return undefined; + switch (service) { + case "postgres": + postgres = version; + break; + case "postgrest": + postgrest = version; + break; + case "auth": + auth = version; + break; + case "edge-runtime": + edgeRuntime = version; + break; + case "realtime": + realtime = version; + break; + case "storage": + storage = version; + break; + case "imgproxy": + imgproxy = version; + break; + case "mailpit": + mailpit = version; + break; + case "pgmeta": + pgmeta = version; + break; + case "studio": + studio = version; + break; + case "analytics": + analytics = version; + break; + case "vector": + vector = version; + break; + case "pooler": + pooler = version; + break; + } + } + return { + ...(postgres === undefined ? {} : { postgres }), + ...(postgrest === undefined ? {} : { postgrest }), + ...(auth === undefined ? {} : { auth }), + ...(edgeRuntime === undefined ? {} : { "edge-runtime": edgeRuntime }), + ...(realtime === undefined ? {} : { realtime }), + ...(storage === undefined ? {} : { storage }), + ...(imgproxy === undefined ? {} : { imgproxy }), + ...(mailpit === undefined ? {} : { mailpit }), + ...(pgmeta === undefined ? {} : { pgmeta }), + ...(studio === undefined ? {} : { studio }), + ...(analytics === undefined ? {} : { analytics }), + ...(vector === undefined ? {} : { vector }), + ...(pooler === undefined ? {} : { pooler }), + }; +} + +function parseServices(value: unknown): Partial> | undefined { + if (!isRecord(value)) return undefined; + const services: Partial> = {}; + for (const [name, enabled] of Object.entries(value)) { + const service = serviceNameFrom(name); + if (service === undefined || !SERVICE_NAME_SET.has(name) || typeof enabled !== "boolean") { + return undefined; + } + services[service] = enabled; + } + return services; +} + +function parseManifest(value: unknown): PodManifest | undefined { + if (!isRecord(value)) return undefined; + if (typeof value.id !== "string" || !isValidPodId(value.id)) return undefined; + const versions = parseVersions(value.versions); + const services = parseServices(value.services); + const ports = parsePorts(value.ports); + if (versions === undefined || services === undefined || ports === undefined) return undefined; + if (!isRecord(value.flags) || typeof value.flags.supautils !== "boolean") return undefined; + if (typeof value.postgresPassword !== "string" || value.postgresPassword.length === 0) { + return undefined; + } + if (typeof value.createdAt !== "string" || Number.isNaN(Date.parse(value.createdAt))) { + return undefined; + } + return { + id: value.id, + versions, + services, + flags: { supautils: value.flags.supautils }, + ports, + postgresPassword: value.postgresPassword, + createdAt: value.createdAt, + }; +} + /** * Persists pod manifests on disk, one per pod directory: `podsRoot//pod.json`. * The pod's data directory lives alongside it at `podsRoot//data`. @@ -32,12 +225,21 @@ export class PodRegistry { async read(id: string): Promise { const raw = await readFile(join(this.podDir(id), "pod.json"), "utf8").catch(() => undefined); - return raw === undefined ? undefined : (JSON.parse(raw) as PodManifest); + if (raw === undefined) return undefined; + try { + const manifest = parseManifest(JSON.parse(raw)); + return manifest?.id === id ? manifest : undefined; + } catch { + return undefined; + } } async write(manifest: PodManifest): Promise { - await mkdir(this.podDir(manifest.id), { recursive: true }); - await writeFile(join(this.podDir(manifest.id), "pod.json"), JSON.stringify(manifest, null, 2)); + const dir = this.podDir(manifest.id); + await mkdir(dir, { recursive: true }); + const tmp = join(dir, `pod.json.tmp-${process.pid}-${Date.now()}`); + await writeFile(tmp, JSON.stringify(manifest, null, 2)); + await rename(tmp, join(dir, "pod.json")); } async list(): Promise { diff --git a/packages/fleet/src/PodRegistry.unit.test.ts b/packages/fleet/src/PodRegistry.unit.test.ts index 1d56cb0877..dbec2bce32 100644 --- a/packages/fleet/src/PodRegistry.unit.test.ts +++ b/packages/fleet/src/PodRegistry.unit.test.ts @@ -1,9 +1,33 @@ -import { mkdtemp, writeFile } from "node:fs/promises"; +import { mkdir, mkdtemp, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; +import type { AllocatedPorts } from "@supabase/stack"; import { PodRegistry } from "./PodRegistry.ts"; +function ports(dbPort: number, apiPort: number): AllocatedPorts { + return { + dbPort, + apiPort, + authPort: apiPort + 1, + postgrestPort: apiPort + 2, + postgrestAdminPort: apiPort + 3, + edgeRuntimePort: apiPort + 4, + edgeRuntimeInspectorPort: apiPort + 5, + realtimePort: apiPort + 6, + storagePort: apiPort + 7, + imgproxyPort: apiPort + 8, + mailpitPort: apiPort + 9, + mailpitSmtpPort: apiPort + 10, + mailpitPop3Port: apiPort + 11, + pgmetaPort: apiPort + 12, + studioPort: apiPort + 13, + analyticsPort: apiPort + 14, + poolerPort: apiPort + 15, + poolerApiPort: apiPort + 16, + }; +} + describe("PodRegistry", () => { it("rejects ids that could escape the pod root", async () => { const pods = new PodRegistry(await mkdtemp(join(tmpdir(), "pods-"))); @@ -21,7 +45,8 @@ describe("PodRegistry", () => { versions: { postgres: "17.6.1.143" }, services: {}, flags: { supautils: false }, - ports: { dbPort: 55000, apiPort: 55001 }, + ports: ports(55000, 55001), + postgresPassword: "postgres", createdAt: "2026-07-08T00:00:00.000Z", }; @@ -30,4 +55,14 @@ describe("PodRegistry", () => { await expect(pods.list()).resolves.toEqual([manifest]); }); + + it("skips malformed manifests while listing persisted pods", async () => { + const root = await mkdtemp(join(tmpdir(), "pods-")); + const pods = new PodRegistry(root); + await mkdir(join(root, "bad")); + await writeFile(join(root, "bad", "pod.json"), "not-json"); + + await expect(pods.read("bad")).resolves.toBeUndefined(); + await expect(pods.list()).resolves.toEqual([]); + }); }); diff --git a/packages/fleet/src/PortRegistry.ts b/packages/fleet/src/PortRegistry.ts index 60a1471f2d..ff37fdb654 100644 --- a/packages/fleet/src/PortRegistry.ts +++ b/packages/fleet/src/PortRegistry.ts @@ -1,10 +1,8 @@ import { mkdir, readFile, rename, writeFile } from "node:fs/promises"; import { dirname } from "node:path"; +import { PORT_FIELDS, type AllocatedPorts } from "@supabase/stack"; -export interface PodPorts { - readonly dbPort: number; - readonly apiPort: number; -} +export type PodPorts = AllocatedPorts; interface PortState { readonly basePort: number; @@ -25,25 +23,50 @@ function isRecord(value: unknown): value is Record { return typeof value === "object" && value !== null && !Array.isArray(value); } -function isPositiveInteger(value: unknown): value is number { - return typeof value === "number" && Number.isInteger(value) && value > 0; +function isFleetPort(value: unknown): value is number { + return typeof value === "number" && Number.isInteger(value) && value > 10_000 && value <= 65_535; } function isValidState(value: unknown): value is PortState { if (!isRecord(value)) return false; const { basePort, pods } = value; - if (!isPositiveInteger(basePort)) return false; + if (!isFleetPort(basePort)) return false; if (!isRecord(pods)) return false; for (const ports of Object.values(pods)) { if (!isRecord(ports)) return false; - if (!isPositiveInteger(ports.dbPort) || !isPositiveInteger(ports.apiPort)) return false; + for (const field of PORT_FIELDS) { + if (!isFleetPort(ports[field])) return false; + } } return true; } +function allocatePortSet(next: () => number): PodPorts { + return { + dbPort: next(), + apiPort: next(), + authPort: next(), + postgrestPort: next(), + postgrestAdminPort: next(), + edgeRuntimePort: next(), + edgeRuntimeInspectorPort: next(), + realtimePort: next(), + storagePort: next(), + imgproxyPort: next(), + mailpitPort: next(), + mailpitSmtpPort: next(), + mailpitPop3Port: next(), + pgmetaPort: next(), + studioPort: next(), + analyticsPort: next(), + poolerPort: next(), + poolerApiPort: next(), + }; +} + /** - * Persistent registry mapping pod IDs to their allocated `{ dbPort, apiPort }` - * pair, backed by a single JSON state file on disk. + * Persistent registry mapping pod IDs to their allocated stack ports, backed + * by a single JSON state file on disk. * * Design assumptions: * - **Single owner process.** Exactly one `PortRegistry` instance (the fleet @@ -59,8 +82,7 @@ function isValidState(value: unknown): value is PortState { * quarantines the bad file (renaming it to `.corrupt`, replacing * any previous quarantine) and starts from fresh empty state. Since pod * ports are also duplicated in each pod's own manifest (`pod.json`), the - * daemon is expected to re-seed the registry after such a reset by calling - * `restore()` for each known pod. + * daemon reconciles the registry from valid manifests after such a reset. * - **In-process serialization.** Mutating operations are queued so concurrent * lifecycle calls in the owner process cannot interleave writes to the shared * temporary state file. @@ -107,14 +129,19 @@ export class PortRegistry { return this.withMutation(async () => { const existing = getOwnPodPorts(this.state.pods, podId); if (existing) return existing; - const used = new Set(Object.values(this.state.pods).flatMap((p) => [p.dbPort, p.apiPort])); + const used = new Set( + Object.values(this.state.pods).flatMap((p) => PORT_FIELDS.map((f) => p[f])), + ); let candidate = this.state.basePort; const next = (): number => { while (used.has(candidate)) candidate += 1; + if (candidate > 65_535) { + throw new Error("PortRegistry: exhausted fleet port range"); + } used.add(candidate); return candidate; }; - const ports: PodPorts = { dbPort: next(), apiPort: next() }; + const ports = allocatePortSet(next); this.state = { ...this.state, pods: { ...this.state.pods, [podId]: ports } }; await this.persist(); return ports; @@ -132,7 +159,7 @@ export class PortRegistry { await this.withMutation(async () => { const existing = getOwnPodPorts(this.state.pods, podId); if (existing) { - if (existing.dbPort === ports.dbPort && existing.apiPort === ports.apiPort) { + if (PORT_FIELDS.every((field) => existing[field] === ports[field])) { return; } throw new Error( @@ -141,9 +168,9 @@ export class PortRegistry { ); } - const restoredPorts = new Set([ports.dbPort, ports.apiPort]); + const restoredPorts = new Set(PORT_FIELDS.map((field) => ports[field])); for (const [otherPodId, otherPorts] of Object.entries(this.state.pods)) { - if (restoredPorts.has(otherPorts.dbPort) || restoredPorts.has(otherPorts.apiPort)) { + if (PORT_FIELDS.some((field) => restoredPorts.has(otherPorts[field]))) { throw new Error( `PortRegistry: cannot restore pod "${podId}" with ports ${JSON.stringify(ports)}; ` + `port already assigned to pod "${otherPodId}"`, @@ -165,6 +192,29 @@ export class PortRegistry { }); } + async reconcile(podPorts: ReadonlyMap): Promise { + await this.withMutation(async () => { + const nextPods: Record = {}; + const used = new Map(); + for (const [podId, ports] of podPorts) { + for (const field of PORT_FIELDS) { + const port = ports[field]; + const owner = used.get(port); + if (owner !== undefined) { + throw new Error( + `PortRegistry: cannot reconcile pod "${podId}" with ports ${JSON.stringify(ports)}; ` + + `port already assigned to pod "${owner}"`, + ); + } + used.set(port, podId); + } + nextPods[podId] = ports; + } + this.state = { ...this.state, pods: nextPods }; + await this.persist(); + }); + } + private async withMutation(body: () => Promise): Promise { const previous = this.mutationQueue; let release: () => void = () => {}; diff --git a/packages/fleet/src/PortRegistry.unit.test.ts b/packages/fleet/src/PortRegistry.unit.test.ts index da2e7d2c76..eaf754e974 100644 --- a/packages/fleet/src/PortRegistry.unit.test.ts +++ b/packages/fleet/src/PortRegistry.unit.test.ts @@ -2,8 +2,32 @@ import { mkdtemp, readFile, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; +import type { AllocatedPorts } from "@supabase/stack"; import { PortRegistry } from "./PortRegistry.ts"; +function ports(dbPort: number, apiPort: number): AllocatedPorts { + return { + dbPort, + apiPort, + authPort: apiPort + 1, + postgrestPort: apiPort + 2, + postgrestAdminPort: apiPort + 3, + edgeRuntimePort: apiPort + 4, + edgeRuntimeInspectorPort: apiPort + 5, + realtimePort: apiPort + 6, + storagePort: apiPort + 7, + imgproxyPort: apiPort + 8, + mailpitPort: apiPort + 9, + mailpitSmtpPort: apiPort + 10, + mailpitPop3Port: apiPort + 11, + pgmetaPort: apiPort + 12, + studioPort: apiPort + 13, + analyticsPort: apiPort + 14, + poolerPort: apiPort + 15, + poolerApiPort: apiPort + 16, + }; +} + describe("PortRegistry", () => { it("allocates unique port pairs and persists them", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); @@ -35,7 +59,8 @@ describe("PortRegistry", () => { const ports = await reg.allocate("constructor"); - expect(ports).toEqual({ dbPort: 55000, apiPort: 55001 }); + expect(ports).toEqual(expect.objectContaining({ dbPort: 55000, apiPort: 55001 })); + expect(new Set(Object.values(ports)).size).toBe(18); expect(reg.get("constructor")).toEqual(ports); }); @@ -91,18 +116,21 @@ describe("PortRegistry", () => { const badStructure = JSON.stringify({ basePort: 55000, pods: { - "pod-a": { dbPort: 55010, apiPort: "55011" }, + "pod-a": ports(55010, 55011), }, }); - await writeFile(file, badStructure); + const parsed = JSON.parse(badStructure); + parsed.pods["pod-a"].poolerApiPort = "55027"; + const raw = JSON.stringify(parsed); + await writeFile(file, raw); const reg = await PortRegistry.load(file); expect(reg.get("pod-a")).toBeUndefined(); const allocated = await reg.allocate("pod-a"); - expect(allocated).toEqual({ dbPort: 55000, apiPort: 55001 }); + expect(allocated).toEqual(expect.objectContaining({ dbPort: 55000, apiPort: 55001 })); const quarantined = await readFile(`${file}.corrupt`, "utf8"); - expect(quarantined).toBe(badStructure); + expect(quarantined).toBe(raw); }); it("overwrites any previous quarantine file", async () => { @@ -122,59 +150,59 @@ describe("PortRegistry", () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); - await reg.restore("pod-b", { dbPort: 55012, apiPort: 55013 }); + await reg.restore("pod-a", ports(55010, 55011)); + await reg.restore("pod-b", ports(55030, 55031)); - expect(reg.get("pod-a")).toEqual({ dbPort: 55010, apiPort: 55011 }); - expect(reg.get("pod-b")).toEqual({ dbPort: 55012, apiPort: 55013 }); + expect(reg.get("pod-a")).toEqual(expect.objectContaining({ dbPort: 55010, apiPort: 55011 })); + expect(reg.get("pod-b")).toEqual(expect.objectContaining({ dbPort: 55030, apiPort: 55031 })); // New allocations must skip the restored ports. const next = await reg.allocate("new-pod"); expect([next.dbPort, next.apiPort]).not.toContain(55010); expect([next.dbPort, next.apiPort]).not.toContain(55011); - expect([next.dbPort, next.apiPort]).not.toContain(55012); - expect([next.dbPort, next.apiPort]).not.toContain(55013); + expect([next.dbPort, next.apiPort]).not.toContain(55030); + expect([next.dbPort, next.apiPort]).not.toContain(55031); // Persisted, so a reload sees the restored allocation. const reloaded = await PortRegistry.load(file); - expect(reloaded.get("pod-a")).toEqual({ dbPort: 55010, apiPort: 55011 }); + expect(reloaded.get("pod-a")).toEqual( + expect.objectContaining({ dbPort: 55010, apiPort: 55011 }), + ); }); it("is idempotent when restoring identical ports for the same pod", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); - await expect( - reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }), - ).resolves.toBeUndefined(); - expect(reg.get("pod-a")).toEqual({ dbPort: 55010, apiPort: 55011 }); + await reg.restore("pod-a", ports(55010, 55011)); + await expect(reg.restore("pod-a", ports(55010, 55011))).resolves.toBeUndefined(); + expect(reg.get("pod-a")).toEqual(expect.objectContaining({ dbPort: 55010, apiPort: 55011 })); }); it("throws if the pod already has different ports recorded", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); - await expect(reg.restore("pod-a", { dbPort: 55010, apiPort: 55099 })).rejects.toThrow(); + await reg.restore("pod-a", ports(55010, 55011)); + await expect(reg.restore("pod-a", ports(55010, 55099))).rejects.toThrow(); }); it("throws if a port is already assigned to a different pod", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); - await expect(reg.restore("pod-b", { dbPort: 55010, apiPort: 55012 })).rejects.toThrow(); - await expect(reg.restore("pod-b", { dbPort: 55020, apiPort: 55011 })).rejects.toThrow(); + await reg.restore("pod-a", ports(55010, 55011)); + await expect(reg.restore("pod-b", ports(55010, 55030))).rejects.toThrow(); + await expect(reg.restore("pod-b", ports(55030, 55011))).rejects.toThrow(); }); it("throws if a restored db port collides with another pod's api port", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", { dbPort: 55010, apiPort: 55011 }); - await expect(reg.restore("pod-b", { dbPort: 55011, apiPort: 55012 })).rejects.toThrow(); - await expect(reg.restore("pod-b", { dbPort: 55012, apiPort: 55010 })).rejects.toThrow(); + await reg.restore("pod-a", ports(55010, 55011)); + await expect(reg.restore("pod-b", ports(55011, 55030))).rejects.toThrow(); + await expect(reg.restore("pod-b", ports(55030, 55010))).rejects.toThrow(); }); }); }); diff --git a/packages/fleet/src/Provisioner.integration.test.ts b/packages/fleet/src/Provisioner.integration.test.ts index 792f28506f..2d3ccb0552 100644 --- a/packages/fleet/src/Provisioner.integration.test.ts +++ b/packages/fleet/src/Provisioner.integration.test.ts @@ -15,7 +15,7 @@ describe.skipIf(!process.env.FLEET_PG_TESTS)("Provisioner", () => { const templates = new TemplateStore(join(root, "templates")); const pods = new PodRegistry(join(root, "pods")); const ports = await PortRegistry.load(join(root, "fleet-state.json")); - return { p: new Provisioner({ templates, pods, ports }), pods }; + return { p: new Provisioner({ templates, pods, ports, postgresPassword: "postgres" }), pods }; } it("creates, forks, resets, destroys", async () => { diff --git a/packages/fleet/src/Provisioner.ts b/packages/fleet/src/Provisioner.ts index fcf6293c94..f64a3923b4 100644 --- a/packages/fleet/src/Provisioner.ts +++ b/packages/fleet/src/Provisioner.ts @@ -1,4 +1,4 @@ -import { rm } from "node:fs/promises"; +import { rename, rm } from "node:fs/promises"; import { SERVICE_NAMES, validateEnabledServiceDependencies, @@ -11,6 +11,13 @@ import type { PodRegistry } from "./PodRegistry.ts"; import type { PortRegistry } from "./PortRegistry.ts"; import type { TemplateStore } from "./TemplateStore.ts"; +function errorCode(error: unknown): string | undefined { + if (typeof error !== "object" || error === null || !("code" in error)) return undefined; + return typeof error.code === "string" ? error.code : undefined; +} + +const NATIVE_FLEET_SERVICES = new Set(["postgres", "postgrest", "auth"]); + export interface CreatePodOptions { readonly id: string; readonly versions: Partial; @@ -31,11 +38,12 @@ export class Provisioner { readonly templates: TemplateStore; readonly pods: PodRegistry; readonly ports: PortRegistry; + readonly postgresPassword: string; }, ) {} async create(opts: CreatePodOptions): Promise { - const { templates, pods, ports } = this.deps; + const { templates, pods, ports, postgresPassword } = this.deps; if ((await pods.read(opts.id)) !== undefined) { throw new Error(`pod already exists: ${opts.id}`); } @@ -46,6 +54,12 @@ export class Provisioner { if (dependencyError !== undefined) { throw new Error(dependencyError); } + const unsupported = enabled.filter((service) => !NATIVE_FLEET_SERVICES.has(service)); + if (unsupported.length > 0) { + throw new Error( + `fleet native mode only supports postgrest and auth pods; unsupported services: ${unsupported.join(", ")}`, + ); + } const resolvedVersions = resolveTemplateVersions(opts.versions, enabled); const template = opts.warm === true @@ -60,6 +74,7 @@ export class Provisioner { services: opts.services ?? {}, flags: { supautils: opts.flags?.supautils ?? false }, ports: allocated, + postgresPassword, createdAt: new Date().toISOString(), }; await pods.write(manifest); @@ -74,14 +89,38 @@ export class Provisioner { /** Re-clones the pod's data dir from the base template of its postgres version. */ async reset(id: string): Promise { - const { templates, pods } = this.deps; + const { templates, pods, postgresPassword } = this.deps; const manifest = await pods.read(id); if (manifest === undefined) throw new Error(`unknown pod: ${id}`); const pgVersion = manifest.versions.postgres; if (pgVersion === undefined) throw new Error(`pod ${id} has no postgres version`); const template = await templates.ensureBaseTemplate(pgVersion); - await rm(pods.dataDir(id), { recursive: true, force: true }); - await cloneDir(template, pods.dataDir(id)); + const dataDir = pods.dataDir(id); + const tmpDataDir = `${dataDir}.reset-${process.pid}-${Date.now()}`; + const backupDataDir = `${dataDir}.backup-${process.pid}-${Date.now()}`; + let backedUp = false; + await cloneDir(template, tmpDataDir); + try { + await rename(dataDir, backupDataDir).then( + () => { + backedUp = true; + }, + (error: unknown) => { + if (errorCode(error) !== "ENOENT") throw error; + }, + ); + await rename(tmpDataDir, dataDir); + await pods.write({ ...manifest, postgresPassword }); + await rm(backupDataDir, { recursive: true, force: true }); + } catch (error) { + if (backedUp) { + await rm(dataDir, { recursive: true, force: true }).catch(() => {}); + await rename(backupDataDir, dataDir).catch(() => {}); + } + throw error; + } finally { + await rm(tmpDataDir, { recursive: true, force: true }).catch(() => {}); + } } /** Caller must ensure the source pod is stopped/suspended first. */ diff --git a/packages/fleet/src/Provisioner.unit.test.ts b/packages/fleet/src/Provisioner.unit.test.ts index e094c11770..ba1421fd8d 100644 --- a/packages/fleet/src/Provisioner.unit.test.ts +++ b/packages/fleet/src/Provisioner.unit.test.ts @@ -1,4 +1,4 @@ -import { mkdir, mkdtemp, readdir, writeFile } from "node:fs/promises"; +import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { DEFAULT_VERSIONS, type ServiceName, type VersionManifest } from "@supabase/stack"; @@ -37,7 +37,12 @@ async function makeHarness(templateDir: string) { const pods = new PodRegistry(podsRoot); const ports = await PortRegistry.load(join(root, "fleet-state.json")); const templates = fakeTemplateStore(templateDir); - return { p: new Provisioner({ templates, pods, ports }), pods, ports, podsRoot }; + return { + p: new Provisioner({ templates, pods, ports, postgresPassword: "secret-password" }), + pods, + ports, + podsRoot, + }; } async function podsRootEntries(podsRoot: string): Promise { @@ -54,6 +59,7 @@ describe("Provisioner (unit, fake deps)", () => { const manifest = await p.create({ id: "x", versions: { postgres: PG_VERSION } }); expect(manifest.id).toBe("x"); + expect(manifest.postgresPassword).toBe("secret-password"); expect(ports.get("x")).toEqual(manifest.ports); expect(await pods.read("x")).toEqual(manifest); }); @@ -119,6 +125,41 @@ describe("Provisioner (unit, fake deps)", () => { expect(ports.get("bad")).toBeUndefined(); expect(await podsRootEntries(podsRoot)).not.toContain("bad"); }); + + it("rejects services that cannot run in native fleet mode before provisioning", async () => { + const templateDir = await mkdtemp(join(tmpdir(), "fleet-template-")); + await writeFile(join(templateDir, "PG_VERSION"), PG_VERSION); + const { p, ports, podsRoot } = await makeHarness(templateDir); + + await expect( + p.create({ + id: "bad", + versions: { postgres: PG_VERSION }, + services: { storage: true }, + }), + ).rejects.toThrow(/native mode/); + + expect(ports.get("bad")).toBeUndefined(); + expect(await podsRootEntries(podsRoot)).not.toContain("bad"); + }); + }); + + describe("reset", () => { + it("keeps the live data dir when cloning the replacement template fails", async () => { + const templateDir = await mkdtemp(join(tmpdir(), "fleet-template-")); + await writeFile(join(templateDir, "PG_VERSION"), PG_VERSION); + const { p, pods } = await makeHarness(templateDir); + + await p.create({ id: "x", versions: { postgres: PG_VERSION } }); + await writeFile(join(pods.dataDir("x"), "marker.txt"), "still-here"); + await rm(templateDir, { recursive: true, force: true }); + + await expect(p.reset("x")).rejects.toThrow(); + + await expect(readFile(join(pods.dataDir("x"), "marker.txt"), "utf8")).resolves.toBe( + "still-here", + ); + }); }); describe("fork", () => { diff --git a/packages/fleet/src/TemplateStore.ts b/packages/fleet/src/TemplateStore.ts index 2e24d62f4a..40acec3368 100644 --- a/packages/fleet/src/TemplateStore.ts +++ b/packages/fleet/src/TemplateStore.ts @@ -70,7 +70,12 @@ export class TemplateStore { // One-shot stack: postgres only, non-provisioned -> postgres-init applies // roles/schemas/baseline migrations exactly as it does for a normal stack. const stack = await createStack({ - postgres: { version: postgresVersion, dataDir: buildDataDir }, + mode: "native", + postgres: { + version: postgresVersion, + dataDir: buildDataDir, + password: postgresPassword, + }, postgrest: false, auth: false, edgeRuntime: false, @@ -132,9 +137,11 @@ export class TemplateStore { // The clone is already post-init, so postgres-init is skipped (`provisioned: true`); // each enabled service boots once and self-migrates against it. const stack = await createStack({ + mode: "native", postgres: { version: pgVersion, dataDir: buildDataDir, + password: postgresPassword, provisioned: true, profile: "micro", }, diff --git a/packages/process-compose/src/Orchestrator.ts b/packages/process-compose/src/Orchestrator.ts index a048c580e3..11c8abe984 100644 --- a/packages/process-compose/src/Orchestrator.ts +++ b/packages/process-compose/src/Orchestrator.ts @@ -531,6 +531,7 @@ export class Orchestrator extends Context.Service< const restartClosureFor = (name: string): ReadonlyArray => { const names = new Set([name]); + const visited = new Set([name]); const shouldRestartDependent = (dependentName: string): boolean => { const svc = services.get(dependentName); if (svc === undefined) return false; @@ -540,9 +541,11 @@ export class Orchestrator extends Context.Service< }; const collectDependents = (current: string): void => { for (const dependent of graph.dependentsOf(current)) { - if (names.has(dependent.name)) continue; - if (!shouldRestartDependent(dependent.name)) continue; - names.add(dependent.name); + if (visited.has(dependent.name)) continue; + visited.add(dependent.name); + if (shouldRestartDependent(dependent.name)) { + names.add(dependent.name); + } collectDependents(dependent.name); } }; diff --git a/packages/process-compose/src/Orchestrator.unit.test.ts b/packages/process-compose/src/Orchestrator.unit.test.ts index 49da134235..38093bb8b0 100644 --- a/packages/process-compose/src/Orchestrator.unit.test.ts +++ b/packages/process-compose/src/Orchestrator.unit.test.ts @@ -628,6 +628,47 @@ describe("Orchestrator", () => { }).pipe(Effect.provide(layer), Effect.scoped); }); + it.live("restartService traverses stopped one-shot helpers to restart live dependents", () => { + const { layer, proc } = setupOrchestrator( + [ + svc("postgres"), + svc("postgres-init", { + restart: "no", + dependencies: [{ service: "postgres", condition: "started" }], + }), + svc("api", { + dependencies: [{ service: "postgres-init", condition: "completed" }], + }), + ], + { + perService: { + api: { exitDelay: "5 seconds" }, + postgres: { exitDelay: "5 seconds" }, + "postgres-init": { exitDelay: "10 millis" }, + }, + }, + ); + return Effect.gen(function* () { + const orc = yield* Orchestrator; + yield* orc.start(); + yield* proc.waitForSpawn("api"); + yield* waitForStopped(orc, "postgres-init"); + + yield* orc.restartService("postgres"); + yield* proc.waitForSpawn("api", 2); + + const spawnCounts = proc.spawned.reduce>((counts, record) => { + counts[record.command] = (counts[record.command] ?? 0) + 1; + return counts; + }, {}); + expect(spawnCounts).toEqual({ + api: 2, + postgres: 2, + "postgres-init": 1, + }); + }).pipe(Effect.provide(layer), Effect.scoped); + }); + it.live("restartService includes launched dependents that are still pending", () => { let dbReady = false; const { layer, proc } = setupOrchestrator( diff --git a/packages/stack/src/DaemonServer.integration.test.ts b/packages/stack/src/DaemonServer.integration.test.ts index 1f2ea9e431..74982f262b 100644 --- a/packages/stack/src/DaemonServer.integration.test.ts +++ b/packages/stack/src/DaemonServer.integration.test.ts @@ -4,6 +4,7 @@ import { Effect, Layer, ManagedRuntime, Stream } from "effect"; import * as http from "node:http"; import { afterAll, beforeAll, describe, expect, test } from "vitest"; import { DaemonServer } from "./DaemonServer.ts"; +import { StackBuildError } from "./errors.ts"; import { Stack, type StackInfo } from "./Stack.ts"; import { StackServiceState } from "./StackServiceState.ts"; @@ -79,9 +80,11 @@ function mockStack() { enableExtension: (name: string) => name === "unknown" ? Effect.fail(new ServiceNotFoundError({ name })) - : Effect.sync(() => { - serviceCalls.push(`enable-extension:${name}`); - }), + : name === "bad-build" + ? Effect.fail(new StackBuildError({ detail: "cannot enable while starting" })) + : Effect.sync(() => { + serviceCalls.push(`enable-extension:${name}`); + }), reloadFunctions: () => Effect.sync(() => { serviceCalls.push("reload-functions"); @@ -321,6 +324,13 @@ describe("DaemonServer", () => { expect(mock.serviceCalls).toContain("reload-edge-runtime"); }); + test("POST /extensions/:name/enable maps build errors to JSON 500", async () => { + const res = await fetch(`${url}/extensions/bad-build/enable`, { method: "POST" }); + expect(res.status).toBe(500); + const body = (await res.json()) as { error: string }; + expect(body.error).toContain("cannot enable while starting"); + }); + // ------------------------------------------------------------------------- // Error cases — service not found // ------------------------------------------------------------------------- diff --git a/packages/stack/src/DaemonServer.ts b/packages/stack/src/DaemonServer.ts index a9ec89d821..8c05c63e6c 100644 --- a/packages/stack/src/DaemonServer.ts +++ b/packages/stack/src/DaemonServer.ts @@ -166,6 +166,9 @@ export class DaemonServer extends Context.Service< Effect.catchTag("ServiceReadyError", (e) => Effect.succeed(HttpServerResponse.jsonUnsafe({ error: e.reason }, { status: 500 })), ), + Effect.catchTag("StackBuildError", (e) => + Effect.succeed(HttpServerResponse.jsonUnsafe({ error: e.detail }, { status: 500 })), + ), ), ), @@ -226,6 +229,9 @@ export class DaemonServer extends Context.Service< Effect.catchTag("ServiceReadyError", (e) => Effect.succeed(HttpServerResponse.jsonUnsafe({ error: e.reason }, { status: 500 })), ), + Effect.catchTag("StackBuildError", (e) => + Effect.succeed(HttpServerResponse.jsonUnsafe({ error: e.detail }, { status: 500 })), + ), ), ), diff --git a/packages/stack/src/StackBuilder.ts b/packages/stack/src/StackBuilder.ts index 543e728c26..34fdba9bf0 100644 --- a/packages/stack/src/StackBuilder.ts +++ b/packages/stack/src/StackBuilder.ts @@ -67,6 +67,8 @@ export interface PostgresConfig { } export interface PostgrestConfig { + readonly port?: number; + readonly adminPort?: number; readonly schemas?: ReadonlyArray; readonly extraSearchPath?: ReadonlyArray; readonly maxRows?: number; diff --git a/packages/stack/src/StackLifecycleCoordinator.ts b/packages/stack/src/StackLifecycleCoordinator.ts index 4d4263e549..cf78f796f7 100644 --- a/packages/stack/src/StackLifecycleCoordinator.ts +++ b/packages/stack/src/StackLifecycleCoordinator.ts @@ -618,7 +618,10 @@ export class StackLifecycleCoordinator extends Context.Service< yield* runtime.orchestrator.waitAllReady(); } yield* Ref.set(phaseRef, "running"); - }).pipe(Effect.ensuring(Ref.set(startInFlightRef, false))), + }).pipe( + Effect.onError(() => Ref.set(phaseRef, "stopped")), + Effect.ensuring(Ref.set(startInFlightRef, false)), + ), stop: () => Effect.gen(function* () { if (runtimeState === undefined) { diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts index 2bf76170f3..b98f032a92 100644 --- a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -8,6 +8,7 @@ import { Duration, Effect, Exit, Fiber, Layer } from "effect"; import { mockChildProcessSpawner } from "../../process-compose/tests/helpers/mocks.ts"; import { mockBinaryResolver } from "../tests/helpers/mocks.ts"; import { defaultPublishableKey, defaultSecretKey, generateJwt } from "./JwtGenerator.ts"; +import { StackBuildError } from "./errors.ts"; import { readPreloadLibraries } from "./pgconf.ts"; import type { AllocatedPorts } from "./PortAllocator.ts"; import { Stack } from "./Stack.ts"; @@ -267,6 +268,35 @@ describe("StackLifecycleCoordinator enableExtension", () => { Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), ); }); + + it.live("returns to stopped phase after start fails", () => { + const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-failed-start-test-")); + writeFileSync(join(dataDir, "postgresql.conf"), "# stock conf\n"); + const config = makeConfig(dataDir); + const resolver = mockBinaryResolver(); + const stackPreparationLayer = StackPreparation.layer.pipe(Layer.provide(resolver.layer)); + const failingBuilderLayer = Layer.succeed(StackBuilder, { + build: () => Effect.fail(new StackBuildError({ detail: "build failed" })), + }); + const layer = StackLifecycleCoordinator.layer(config).pipe( + Layer.provide(failingBuilderLayer), + Layer.provide(stackPreparationLayer), + Layer.provide(StackMetadataPersistence.noop), + Layer.provide(NodeServices.layer), + ); + + return Effect.gen(function* () { + const coordinator = yield* StackLifecycleCoordinator; + const startExit = yield* coordinator.start().pipe(Effect.exit); + + expect(Exit.isFailure(startExit)).toBe(true); + yield* coordinator.enableExtension("pg_cron"); + expect(yield* Effect.promise(() => readPreloadLibraries(dataDir))).toEqual(["pg_cron"]); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), + ); + }); }); describe("StackLifecycleCoordinator lazyServices", () => { diff --git a/packages/stack/src/createStack.ts b/packages/stack/src/createStack.ts index b55e18c376..e28fc3b415 100644 --- a/packages/stack/src/createStack.ts +++ b/packages/stack/src/createStack.ts @@ -512,8 +512,8 @@ export async function resolveConfig( apiPort: config.port, dbPort: postgresInput.port, authPort: authInput?.port, - postgrestPort: undefined, - postgrestAdminPort: undefined, + postgrestPort: postgrestInput?.port, + postgrestAdminPort: postgrestInput?.adminPort, edgeRuntimePort: edgeRuntimeInput?.port, edgeRuntimeInspectorPort: edgeRuntimeInput?.inspectorPort, realtimePort: realtimeInput?.port, diff --git a/packages/stack/src/index.ts b/packages/stack/src/index.ts index 000abbc761..a8e1aa217e 100644 --- a/packages/stack/src/index.ts +++ b/packages/stack/src/index.ts @@ -22,6 +22,8 @@ export type { export type { ServiceName, VersionManifest } from "./versions.ts"; export { DEFAULT_VERSIONS, fillServiceVersionManifest, SERVICE_NAMES } from "./versions.ts"; +export type { AllocatedPorts } from "./PortAllocator.ts"; +export { PORT_FIELDS } from "./PortAllocator.ts"; export { postgresConnectionUrl, resolvePostgresPassword } from "./postgresCredentials.ts"; export { validateEnabledServiceDependencies } from "./serviceDependencies.ts"; export type { ServiceResolution } from "./resolve.ts"; diff --git a/packages/stack/src/services/pooler.ts b/packages/stack/src/services/pooler.ts index aaa2f2a26e..6a17fa34c8 100644 --- a/packages/stack/src/services/pooler.ts +++ b/packages/stack/src/services/pooler.ts @@ -95,11 +95,12 @@ export const makePoolerServiceDocker = (opts: DockerPoolerOptions): ServiceDef = RUN_JANITOR: "true", ERL_AFLAGS: "-proto_dist inet_tcp", RLIMIT_NOFILE: "", + SUPAVISOR_TENANT_SCRIPT: tenantScript(opts), }, cmd: [ "/bin/sh", "-c", - `/app/bin/migrate && /app/bin/supavisor eval '${tenantScript(opts)}' && /app/bin/server`, + `/app/bin/migrate && /app/bin/supavisor eval "$SUPAVISOR_TENANT_SCRIPT" && /app/bin/server`, ], dependsOn: opts.dependencies, healthCheck: poolerHealthCheck(opts.hostAdminPort), diff --git a/packages/stack/src/services/postgres-init.ts b/packages/stack/src/services/postgres-init.ts index 3565570329..499fbd4214 100644 --- a/packages/stack/src/services/postgres-init.ts +++ b/packages/stack/src/services/postgres-init.ts @@ -124,6 +124,7 @@ WHERE rolname = ANY (ARRAY[ 'supabase_functions_admin', 'supabase_replication_admin', 'supabase_read_only_user', + 'pgbouncer', 'postgres' ])\\gexec EOSQL diff --git a/packages/stack/src/services/services.unit.test.ts b/packages/stack/src/services/services.unit.test.ts index 38d3a0e29e..5e6a02b3c3 100644 --- a/packages/stack/src/services/services.unit.test.ts +++ b/packages/stack/src/services/services.unit.test.ts @@ -493,6 +493,17 @@ describe("makePostgresInitService", () => { expect(script).toContain("already initialized"); }); + it("keeps the pooler database role password in sync", () => { + const def = makePostgresInitService({ + postgresDir: "/cache/postgres/17/darwin-arm64", + dbPort: DB_PORT, + dbPassword: DB_PASSWORD, + autoExposeNewTables: true, + }); + const script = def.args?.[1] as string; + expect(script).toContain("'pgbouncer'"); + }); + it("backfills auxiliary service schemas and internal databases", () => { const def = makePostgresInitService({ postgresDir: "/cache/postgres/17/darwin-arm64", @@ -716,5 +727,8 @@ describe("docker-backed auxiliary services", () => { expect(def.args).toContain(`PROXY_PORT_TRANSACTION=${poolerContainerPorts.transaction}`); expect(def.args).toContain(`54329:${poolerContainerPorts.admin}`); expect(def.args).toContain(`54330:${poolerContainerPorts.transaction}`); + const args = def.args ?? []; + expect(args.some((arg) => arg.startsWith("SUPAVISOR_TENANT_SCRIPT="))).toBe(true); + expect(args.join(" ")).toContain('supavisor eval "$SUPAVISOR_TENANT_SCRIPT"'); }); }); From 327e836a7cb4eb50461925f5e966fad53d1f4c83 Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Wed, 8 Jul 2026 13:51:57 +0200 Subject: [PATCH 37/38] fix(stack): stabilize password rotation --- packages/fleet/src/Fleet.ts | 13 ++-- packages/fleet/src/TemplateStore.ts | 9 ++- packages/stack/src/services/postgres-init.ts | 61 +++++++++++++------ .../stack/src/services/services.unit.test.ts | 32 ++++++++++ 4 files changed, 86 insertions(+), 29 deletions(-) diff --git a/packages/fleet/src/Fleet.ts b/packages/fleet/src/Fleet.ts index a6398855b5..705b8bd350 100644 --- a/packages/fleet/src/Fleet.ts +++ b/packages/fleet/src/Fleet.ts @@ -50,11 +50,6 @@ interface WarmPod { readonly internalDbPort: number; } -// Must match packages/stack/src/services/postgres.ts: the template build stores -// this password in the data dir, and pod connection URLs need to use the same -// value when exposing the suspended/warm pod. -const DB_PASSWORD = resolvePostgresPassword(); - // Internal (in-process stack) ports are derived from the externally-visible, // PortRegistry-owned ports by a fixed lower offset so the proxy-owned public // range and the stack-owned internal range never collide. @@ -105,11 +100,15 @@ const INTERNAL_PORT_OFFSET = 10_000; export async function createFleet(opts: FleetOptions = {}): Promise { const root = opts.root ?? join(homedir(), ".supabase"); const idleMs = opts.idleMs ?? 5 * 60_000; + // Must match packages/stack/src/services/postgres.ts: the template build stores + // this password in the data dir, and pod connection URLs need to use the same + // value when exposing the suspended/warm pod. + const postgresPassword = resolvePostgresPassword(); - const templates = new TemplateStore(join(root, "templates")); + const templates = new TemplateStore(join(root, "templates"), postgresPassword); const pods = new PodRegistry(join(root, "pods")); const ports = await PortRegistry.load(join(root, "fleet-state.json")); - const provisioner = new Provisioner({ templates, pods, ports, postgresPassword: DB_PASSWORD }); + const provisioner = new Provisioner({ templates, pods, ports, postgresPassword }); const states = new Map(); const warm = new Map(); diff --git a/packages/fleet/src/TemplateStore.ts b/packages/fleet/src/TemplateStore.ts index 40acec3368..b6e178b78a 100644 --- a/packages/fleet/src/TemplateStore.ts +++ b/packages/fleet/src/TemplateStore.ts @@ -43,7 +43,10 @@ function errorCode(error: unknown): string | undefined { * `wx` flag; a stale lock (holder crashed) is reclaimed after `LOCK_STALE_MS`. */ export class TemplateStore { - constructor(private readonly root: string) {} + constructor( + private readonly root: string, + private readonly postgresPassword = resolvePostgresPassword(), + ) {} private dataDir(key: string): string { return join(this.root, key, "data"); @@ -57,7 +60,7 @@ export class TemplateStore { } async ensureBaseTemplate(postgresVersion: string): Promise { - const postgresPassword = resolvePostgresPassword(); + const postgresPassword = this.postgresPassword; const key = baseTemplateKey(postgresVersion, { postgresPassword }); if (await this.has(key)) return this.dataDir(key); return this.withLock(key, async () => { @@ -117,7 +120,7 @@ export class TemplateStore { ): Promise { const pgVersion = versions.postgres; if (pgVersion === undefined) throw new Error("versions.postgres is required"); - const postgresPassword = resolvePostgresPassword(); + const postgresPassword = this.postgresPassword; const base = await this.ensureBaseTemplate(pgVersion); if (enabledServices.length === 0) return base; diff --git a/packages/stack/src/services/postgres-init.ts b/packages/stack/src/services/postgres-init.ts index 499fbd4214..9ee855c67e 100644 --- a/packages/stack/src/services/postgres-init.ts +++ b/packages/stack/src/services/postgres-init.ts @@ -30,13 +30,15 @@ function shellQuote(value: string): string { return `'${value.replaceAll("'", "'\"'\"'")}'`; } +const DEFAULT_POSTGRES_PASSWORD = "postgres"; + export const makePostgresInitService = (opts: PostgresInitOptions): ServiceDef => { const pgBinDir = `${opts.postgresDir}/bin`; const pgLibDir = `${opts.postgresDir}/lib`; const migrationsDir = `${opts.postgresDir}/share/supabase-cli/migrations`; const psql = `${pgBinDir}/psql -h 127.0.0.1 -p ${opts.dbPort}`; - const psqlOpts = `-v ON_ERROR_STOP=1 --no-password --no-psqlrc -v pgpass="$PGPASSWORD"`; + const psqlOpts = `-v ON_ERROR_STOP=1 --no-password --no-psqlrc -v pgpass="$TARGET_PGPASSWORD"`; const revokeStep = opts.autoExposeNewTables ? "" @@ -53,11 +55,29 @@ EOSQL // postgres-init time from ~5s to ~1s. const script = ` export PATH="${pgBinDir}:$PATH" -export PGPASSWORD=${shellQuote(opts.dbPassword)} +export TARGET_PGPASSWORD=${shellQuote(opts.dbPassword)} +export PGPASSWORD="$TARGET_PGPASSWORD" +DEFAULT_PGPASSWORD=${shellQuote(DEFAULT_POSTGRES_PASSWORD)} db="${migrationsDir}" +is_initialized() { + ${psql} -U supabase_admin -d postgres -tAc "SELECT 1 FROM pg_roles WHERE rolname='authenticator'" 2>/dev/null | grep -q 1 +} + # Check if already migrated (authenticator role created by initial-schema.sql) -if ${psql} -U supabase_admin -d postgres -tAc "SELECT 1 FROM pg_roles WHERE rolname='authenticator'" 2>/dev/null | grep -q 1; then +initialized=false +if is_initialized; then + initialized=true +elif [ "$TARGET_PGPASSWORD" != "$DEFAULT_PGPASSWORD" ]; then + export PGPASSWORD="$DEFAULT_PGPASSWORD" + if is_initialized; then + initialized=true + else + export PGPASSWORD="$TARGET_PGPASSWORD" + fi +fi + +if [ "$initialized" = "true" ]; then echo "Database already initialized, updating passwords..." else echo "Running Supabase migrations..." @@ -96,6 +116,25 @@ EOSQL ${psql} ${psqlOpts} -U supabase_admin -d postgres -c 'SELECT extensions.pg_stat_statements_reset(); SELECT pg_stat_reset();' || true ${revokeStep}fi +# Always update role passwords (idempotent) +${psql} ${psqlOpts} -U supabase_admin -d postgres <<'EOSQL' +SELECT format('ALTER ROLE %I WITH PASSWORD %L', rolname, :'pgpass') +FROM pg_roles +WHERE rolname = ANY (ARRAY[ + 'supabase_admin', + 'authenticator', + 'supabase_auth_admin', + 'supabase_storage_admin', + 'supabase_functions_admin', + 'supabase_replication_admin', + 'supabase_read_only_user', + 'pgbouncer', + 'postgres' +])\\gexec +EOSQL + +export PGPASSWORD="$TARGET_PGPASSWORD" + # Backfill schemas/databases used by docker-backed auxiliary services. ${psql} ${psqlOpts} -U postgres -d postgres <<'EOSQL' CREATE SCHEMA IF NOT EXISTS _realtime; @@ -112,22 +151,6 @@ ALTER SCHEMA _analytics OWNER TO postgres; CREATE SCHEMA IF NOT EXISTS _supavisor; ALTER SCHEMA _supavisor OWNER TO postgres; EOSQL - -# Always update role passwords (idempotent) -${psql} ${psqlOpts} -U supabase_admin -d postgres <<'EOSQL' -SELECT format('ALTER ROLE %I WITH PASSWORD %L', rolname, :'pgpass') -FROM pg_roles -WHERE rolname = ANY (ARRAY[ - 'authenticator', - 'supabase_auth_admin', - 'supabase_storage_admin', - 'supabase_functions_admin', - 'supabase_replication_admin', - 'supabase_read_only_user', - 'pgbouncer', - 'postgres' -])\\gexec -EOSQL `; return { diff --git a/packages/stack/src/services/services.unit.test.ts b/packages/stack/src/services/services.unit.test.ts index 5e6a02b3c3..6235ce4660 100644 --- a/packages/stack/src/services/services.unit.test.ts +++ b/packages/stack/src/services/services.unit.test.ts @@ -504,6 +504,38 @@ describe("makePostgresInitService", () => { expect(script).toContain("'pgbouncer'"); }); + it("keeps the supabase_admin database role password in sync", () => { + const def = makePostgresInitService({ + postgresDir: "/cache/postgres/17/darwin-arm64", + dbPort: DB_PORT, + dbPassword: DB_PASSWORD, + autoExposeNewTables: true, + }); + const script = def.args?.[1] as string; + expect(script).toContain("'supabase_admin'"); + }); + + it("separates the current auth password from the target role password", () => { + const def = makePostgresInitService({ + postgresDir: "/cache/postgres/17/darwin-arm64", + dbPort: DB_PORT, + dbPassword: "new-password", + autoExposeNewTables: true, + }); + const script = def.args?.[1] as string; + + expect(script).toContain("export TARGET_PGPASSWORD='new-password'"); + expect(script).toContain('export PGPASSWORD="$DEFAULT_PGPASSWORD"'); + expect(script).toContain('-v pgpass="$TARGET_PGPASSWORD"'); + + const updateRoles = script.indexOf("# Always update role passwords"); + const useTargetPassword = script.indexOf('export PGPASSWORD="$TARGET_PGPASSWORD"', updateRoles); + const backfill = script.indexOf("# Backfill schemas/databases", updateRoles); + expect(updateRoles).toBeGreaterThan(-1); + expect(useTargetPassword).toBeGreaterThan(updateRoles); + expect(backfill).toBeGreaterThan(useTargetPassword); + }); + it("backfills auxiliary service schemas and internal databases", () => { const def = makePostgresInitService({ postgresDir: "/cache/postgres/17/darwin-arm64", From 04dd2182fcda94ae0fb6408f840a51c95b6fce6c Mon Sep 17 00:00:00 2001 From: Julien Goux Date: Wed, 8 Jul 2026 16:19:32 +0200 Subject: [PATCH 38/38] fix(stack): harden lazy services and fleet ports --- packages/fleet/src/Fleet.ts | 25 ++-- packages/fleet/src/Fleet.unit.test.ts | 1 + packages/fleet/src/PodManifest.ts | 1 + packages/fleet/src/PodRegistry.ts | 11 +- packages/fleet/src/PodRegistry.unit.test.ts | 1 + packages/fleet/src/PortRegistry.ts | 112 ++++++++++++----- packages/fleet/src/PortRegistry.unit.test.ts | 114 +++++++++++++----- packages/fleet/src/Provisioner.ts | 6 +- packages/fleet/src/Provisioner.unit.test.ts | 26 +++- .../src/DaemonServer.integration.test.ts | 18 ++- packages/stack/src/DaemonServer.ts | 22 ++++ packages/stack/src/RemoteStack.ts | 45 +++---- .../stack/src/StackLifecycleCoordinator.ts | 12 ++ .../StackLifecycleCoordinator.unit.test.ts | 37 ++++++ packages/stack/src/services/pooler.ts | 5 +- packages/stack/src/services/postgres.ts | 1 + .../stack/src/services/services.unit.test.ts | 6 +- 17 files changed, 332 insertions(+), 111 deletions(-) diff --git a/packages/fleet/src/Fleet.ts b/packages/fleet/src/Fleet.ts index 705b8bd350..1e8e0b5682 100644 --- a/packages/fleet/src/Fleet.ts +++ b/packages/fleet/src/Fleet.ts @@ -50,11 +50,6 @@ interface WarmPod { readonly internalDbPort: number; } -// Internal (in-process stack) ports are derived from the externally-visible, -// PortRegistry-owned ports by a fixed lower offset so the proxy-owned public -// range and the stack-owned internal range never collide. -const INTERNAL_PORT_OFFSET = 10_000; - /** * Public facade tying together TemplateStore/PodRegistry/PortRegistry/Provisioner * (pod lifecycle + storage) with EdgeProxy/IdleMonitor (wake-on-connect, @@ -138,8 +133,6 @@ export async function createFleet(opts: FleetOptions = {}): Promise database: "postgres", }); - const internalPort = (externalPort: number): number => externalPort - INTERNAL_PORT_OFFSET; - const runPidFile = (id: string): string => join(pods.podDir(id), "run.pid"); async function withPodLocks(ids: ReadonlyArray, body: () => Promise): Promise { @@ -171,11 +164,12 @@ export async function createFleet(opts: FleetOptions = {}): Promise const manifest = await pods.read(id); if (manifest === undefined) throw new Error(`unknown pod: ${id}`); states.set(id, "waking"); - const internalDbPort = internalPort(manifest.ports.dbPort); + const { internalPorts } = manifest; + const internalDbPort = internalPorts.dbPort; const stack = await createStack({ mode: "native", stackRoot: join(pods.podDir(id), "stack"), - port: internalPort(manifest.ports.apiPort), + port: internalPorts.apiPort, lazyServices: true, postgres: { dataDir: pods.dataDir(id), @@ -189,12 +183,12 @@ export async function createFleet(opts: FleetOptions = {}): Promise manifest.services.postgrest === true ? { version: manifest.versions.postgrest, - port: internalPort(manifest.ports.postgrestPort), + port: internalPorts.postgrestPort, } : false, auth: manifest.services.auth === true - ? { version: manifest.versions.auth, port: internalPort(manifest.ports.authPort) } + ? { version: manifest.versions.auth, port: internalPorts.authPort } : false, realtime: false, edgeRuntime: false, @@ -284,7 +278,14 @@ export async function createFleet(opts: FleetOptions = {}): Promise // or skipped pods are pruned during startup. try { const manifests = await pods.list(); - await ports.reconcile(new Map(manifests.map((manifest) => [manifest.id, manifest.ports]))); + await ports.reconcile( + new Map( + manifests.map((manifest) => [ + manifest.id, + { ports: manifest.ports, internalPorts: manifest.internalPorts }, + ]), + ), + ); for (const manifest of manifests) { await reapStalePostmaster(pods.dataDir(manifest.id)); await rm(runPidFile(manifest.id), { force: true }); diff --git a/packages/fleet/src/Fleet.unit.test.ts b/packages/fleet/src/Fleet.unit.test.ts index ece6567248..3c586a5c4a 100644 --- a/packages/fleet/src/Fleet.unit.test.ts +++ b/packages/fleet/src/Fleet.unit.test.ts @@ -59,6 +59,7 @@ function manifest(id: string, dbPort: number, apiPort: number): PodManifest { services: {}, flags: { supautils: false }, ports: ports(dbPort, apiPort), + internalPorts: ports(dbPort - 10_000, apiPort - 10_000), postgresPassword: "postgres", createdAt: "2026-07-08T00:00:00.000Z", }; diff --git a/packages/fleet/src/PodManifest.ts b/packages/fleet/src/PodManifest.ts index 74ad9c9f6f..8fbcf5be3b 100644 --- a/packages/fleet/src/PodManifest.ts +++ b/packages/fleet/src/PodManifest.ts @@ -12,6 +12,7 @@ export interface PodManifest { readonly services: Partial>; readonly flags: { readonly supautils: boolean }; readonly ports: AllocatedPorts; + readonly internalPorts: AllocatedPorts; readonly postgresPassword: string; readonly createdAt: string; } diff --git a/packages/fleet/src/PodRegistry.ts b/packages/fleet/src/PodRegistry.ts index a510912c6f..bb4a82a5de 100644 --- a/packages/fleet/src/PodRegistry.ts +++ b/packages/fleet/src/PodRegistry.ts @@ -189,7 +189,15 @@ function parseManifest(value: unknown): PodManifest | undefined { const versions = parseVersions(value.versions); const services = parseServices(value.services); const ports = parsePorts(value.ports); - if (versions === undefined || services === undefined || ports === undefined) return undefined; + const internalPorts = parsePorts(value.internalPorts); + if ( + versions === undefined || + services === undefined || + ports === undefined || + internalPorts === undefined + ) { + return undefined; + } if (!isRecord(value.flags) || typeof value.flags.supautils !== "boolean") return undefined; if (typeof value.postgresPassword !== "string" || value.postgresPassword.length === 0) { return undefined; @@ -203,6 +211,7 @@ function parseManifest(value: unknown): PodManifest | undefined { services, flags: { supautils: value.flags.supautils }, ports, + internalPorts, postgresPassword: value.postgresPassword, createdAt: value.createdAt, }; diff --git a/packages/fleet/src/PodRegistry.unit.test.ts b/packages/fleet/src/PodRegistry.unit.test.ts index dbec2bce32..7dd9a6bc75 100644 --- a/packages/fleet/src/PodRegistry.unit.test.ts +++ b/packages/fleet/src/PodRegistry.unit.test.ts @@ -46,6 +46,7 @@ describe("PodRegistry", () => { services: {}, flags: { supautils: false }, ports: ports(55000, 55001), + internalPorts: ports(45000, 45001), postgresPassword: "postgres", createdAt: "2026-07-08T00:00:00.000Z", }; diff --git a/packages/fleet/src/PortRegistry.ts b/packages/fleet/src/PortRegistry.ts index ff37fdb654..5aa9c9a291 100644 --- a/packages/fleet/src/PortRegistry.ts +++ b/packages/fleet/src/PortRegistry.ts @@ -2,17 +2,24 @@ import { mkdir, readFile, rename, writeFile } from "node:fs/promises"; import { dirname } from "node:path"; import { PORT_FIELDS, type AllocatedPorts } from "@supabase/stack"; -export type PodPorts = AllocatedPorts; +export interface PodPorts { + readonly ports: AllocatedPorts; + readonly internalPorts: AllocatedPorts; +} interface PortState { readonly basePort: number; + readonly internalBasePort: number; readonly pods: Record; } const DEFAULT_BASE_PORT = 55000; +const DEFAULT_INTERNAL_BASE_PORT = 45000; +const INTERNAL_MAX_PORT = DEFAULT_BASE_PORT - 1; +const MAX_PORT = 65_535; function freshState(): PortState { - return { basePort: DEFAULT_BASE_PORT, pods: {} }; + return { basePort: DEFAULT_BASE_PORT, internalBasePort: DEFAULT_INTERNAL_BASE_PORT, pods: {} }; } function getOwnPodPorts(pods: Record, podId: string): PodPorts | undefined { @@ -24,24 +31,51 @@ function isRecord(value: unknown): value is Record { } function isFleetPort(value: unknown): value is number { - return typeof value === "number" && Number.isInteger(value) && value > 10_000 && value <= 65_535; + return ( + typeof value === "number" && Number.isInteger(value) && value > 10_000 && value <= MAX_PORT + ); +} + +function isValidPortSet(value: unknown): value is AllocatedPorts { + if (!isRecord(value)) return false; + for (const field of PORT_FIELDS) { + if (!isFleetPort(value[field])) return false; + } + return true; +} + +function isValidPodPorts(value: unknown): value is PodPorts { + if (!isRecord(value)) return false; + return isValidPortSet(value.ports) && isValidPortSet(value.internalPorts); } function isValidState(value: unknown): value is PortState { if (!isRecord(value)) return false; - const { basePort, pods } = value; - if (!isFleetPort(basePort)) return false; + const { basePort, internalBasePort, pods } = value; + if (!isFleetPort(basePort) || basePort < DEFAULT_BASE_PORT) return false; + if (!isFleetPort(internalBasePort) || internalBasePort > INTERNAL_MAX_PORT) return false; if (!isRecord(pods)) return false; - for (const ports of Object.values(pods)) { - if (!isRecord(ports)) return false; - for (const field of PORT_FIELDS) { - if (!isFleetPort(ports[field])) return false; - } + for (const allocation of Object.values(pods)) { + if (!isValidPodPorts(allocation)) return false; } return true; } -function allocatePortSet(next: () => number): PodPorts { +function allocatedPorts(allocation: PodPorts): ReadonlyArray { + return [ + ...PORT_FIELDS.map((field) => allocation.ports[field]), + ...PORT_FIELDS.map((field) => allocation.internalPorts[field]), + ]; +} + +function sameAllocation(a: PodPorts, b: PodPorts): boolean { + return PORT_FIELDS.every( + (field) => + a.ports[field] === b.ports[field] && a.internalPorts[field] === b.internalPorts[field], + ); +} + +function allocatePortSet(next: () => number): AllocatedPorts { return { dbPort: next(), apiPort: next(), @@ -76,7 +110,9 @@ function allocatePortSet(next: () => number): PodPorts { * - **No host-level port probing.** The registry never checks whether a port * is actually free on the host; it only tracks what it has handed out * itself. The daemon owns the 55000+ range by convention, so nothing else - * on the host is expected to bind those ports. + * on the host is expected to bind those ports. Public proxy listeners use + * the 55000+ range, while in-process stack services use separately allocated + * ports below that public range. * - **Corrupt-state recovery.** If the state file is missing, unreadable as * JSON, or structurally invalid, `load()` never throws. Instead it * quarantines the bad file (renaming it to `.corrupt`, replacing @@ -130,21 +166,37 @@ export class PortRegistry { const existing = getOwnPodPorts(this.state.pods, podId); if (existing) return existing; const used = new Set( - Object.values(this.state.pods).flatMap((p) => PORT_FIELDS.map((f) => p[f])), + Object.values(this.state.pods).flatMap((allocation) => allocatedPorts(allocation)), ); - let candidate = this.state.basePort; - const next = (): number => { - while (used.has(candidate)) candidate += 1; - if (candidate > 65_535) { + let publicCandidate = this.state.basePort; + let internalCandidate = this.state.internalBasePort; + const nextPublic = (): number => { + while (used.has(publicCandidate)) publicCandidate += 1; + if (publicCandidate > MAX_PORT) { throw new Error("PortRegistry: exhausted fleet port range"); } - used.add(candidate); - return candidate; + const port = publicCandidate; + used.add(port); + publicCandidate += 1; + return port; }; - const ports = allocatePortSet(next); - this.state = { ...this.state, pods: { ...this.state.pods, [podId]: ports } }; + const nextInternal = (): number => { + while (used.has(internalCandidate)) internalCandidate += 1; + if (internalCandidate > INTERNAL_MAX_PORT) { + throw new Error("PortRegistry: exhausted internal fleet port range"); + } + const port = internalCandidate; + used.add(port); + internalCandidate += 1; + return port; + }; + const allocation = { + ports: allocatePortSet(nextPublic), + internalPorts: allocatePortSet(nextInternal), + }; + this.state = { ...this.state, pods: { ...this.state.pods, [podId]: allocation } }; await this.persist(); - return ports; + return allocation; }); } @@ -159,7 +211,7 @@ export class PortRegistry { await this.withMutation(async () => { const existing = getOwnPodPorts(this.state.pods, podId); if (existing) { - if (PORT_FIELDS.every((field) => existing[field] === ports[field])) { + if (sameAllocation(existing, ports)) { return; } throw new Error( @@ -168,9 +220,16 @@ export class PortRegistry { ); } - const restoredPorts = new Set(PORT_FIELDS.map((field) => ports[field])); + const restoredPorts = allocatedPorts(ports); + if (new Set(restoredPorts).size !== restoredPorts.length) { + throw new Error( + `PortRegistry: cannot restore pod "${podId}" with ports ${JSON.stringify(ports)}; ` + + "allocation contains duplicate ports", + ); + } + const restoredPortSet = new Set(restoredPorts); for (const [otherPodId, otherPorts] of Object.entries(this.state.pods)) { - if (PORT_FIELDS.some((field) => restoredPorts.has(otherPorts[field]))) { + if (allocatedPorts(otherPorts).some((port) => restoredPortSet.has(port))) { throw new Error( `PortRegistry: cannot restore pod "${podId}" with ports ${JSON.stringify(ports)}; ` + `port already assigned to pod "${otherPodId}"`, @@ -197,8 +256,7 @@ export class PortRegistry { const nextPods: Record = {}; const used = new Map(); for (const [podId, ports] of podPorts) { - for (const field of PORT_FIELDS) { - const port = ports[field]; + for (const port of allocatedPorts(ports)) { const owner = used.get(port); if (owner !== undefined) { throw new Error( diff --git a/packages/fleet/src/PortRegistry.unit.test.ts b/packages/fleet/src/PortRegistry.unit.test.ts index eaf754e974..8f26eb3169 100644 --- a/packages/fleet/src/PortRegistry.unit.test.ts +++ b/packages/fleet/src/PortRegistry.unit.test.ts @@ -3,7 +3,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; import type { AllocatedPorts } from "@supabase/stack"; -import { PortRegistry } from "./PortRegistry.ts"; +import { PortRegistry, type PodPorts } from "./PortRegistry.ts"; function ports(dbPort: number, apiPort: number): AllocatedPorts { return { @@ -28,13 +28,29 @@ function ports(dbPort: number, apiPort: number): AllocatedPorts { }; } +function podPorts( + dbPort: number, + apiPort: number, + internalDbPort = dbPort - 10_000, + internalApiPort = apiPort - 10_000, +): PodPorts { + return { + ports: ports(dbPort, apiPort), + internalPorts: ports(internalDbPort, internalApiPort), + }; +} + describe("PortRegistry", () => { it("allocates unique port pairs and persists them", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); const a = await reg.allocate("pod-a"); const b = await reg.allocate("pod-b"); - expect(new Set([a.dbPort, a.apiPort, b.dbPort, b.apiPort]).size).toBe(4); + expect(new Set([a.ports.dbPort, a.ports.apiPort, b.ports.dbPort, b.ports.apiPort]).size).toBe( + 4, + ); + expect(a.internalPorts.dbPort).toBe(45000); + expect(a.internalPorts.apiPort).toBe(45001); const reloaded = await PortRegistry.load(file); expect(reloaded.get("pod-a")).toEqual(a); @@ -50,7 +66,8 @@ describe("PortRegistry", () => { await reg.release("pod-a"); expect(reg.get("pod-a")).toBeUndefined(); const c = await reg.allocate("pod-c"); - expect(c.dbPort).toBe(a1.dbPort); // freed ports are reusable + expect(c.ports.dbPort).toBe(a1.ports.dbPort); // freed ports are reusable + expect(c.internalPorts.dbPort).toBe(a1.internalPorts.dbPort); }); it("treats inherited property names as ordinary pod ids", async () => { @@ -59,8 +76,11 @@ describe("PortRegistry", () => { const ports = await reg.allocate("constructor"); - expect(ports).toEqual(expect.objectContaining({ dbPort: 55000, apiPort: 55001 })); - expect(new Set(Object.values(ports)).size).toBe(18); + expect(ports.ports).toEqual(expect.objectContaining({ dbPort: 55000, apiPort: 55001 })); + expect(ports.internalPorts).toEqual(expect.objectContaining({ dbPort: 45000, apiPort: 45001 })); + expect( + new Set([...Object.values(ports.ports), ...Object.values(ports.internalPorts)]).size, + ).toBe(36); expect(reg.get("constructor")).toEqual(ports); }); @@ -90,7 +110,8 @@ describe("PortRegistry", () => { // Loads successfully with fresh empty state. expect(reg.get("anything")).toBeUndefined(); const allocated = await reg.allocate("pod-a"); - expect(allocated.dbPort).toBeGreaterThanOrEqual(55000); + expect(allocated.ports.dbPort).toBeGreaterThanOrEqual(55000); + expect(allocated.internalPorts.dbPort).toBeGreaterThanOrEqual(45000); // Original bad bytes are preserved in the quarantine file. const quarantined = await readFile(`${file}.corrupt`, "utf8"); @@ -99,13 +120,17 @@ describe("PortRegistry", () => { it("quarantines a structurally invalid state file (bad basePort/pods)", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); - const badStructure = JSON.stringify({ basePort: "not-a-number", pods: [] }); + const badStructure = JSON.stringify({ + basePort: "not-a-number", + internalBasePort: 45000, + pods: [], + }); await writeFile(file, badStructure); const reg = await PortRegistry.load(file); expect(reg.get("anything")).toBeUndefined(); const allocated = await reg.allocate("pod-a"); - expect(allocated.dbPort).toBeGreaterThanOrEqual(55000); + expect(allocated.ports.dbPort).toBeGreaterThanOrEqual(55000); const quarantined = await readFile(`${file}.corrupt`, "utf8"); expect(quarantined).toBe(badStructure); @@ -115,19 +140,20 @@ describe("PortRegistry", () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const badStructure = JSON.stringify({ basePort: 55000, + internalBasePort: 45000, pods: { - "pod-a": ports(55010, 55011), + "pod-a": podPorts(55010, 55011), }, }); const parsed = JSON.parse(badStructure); - parsed.pods["pod-a"].poolerApiPort = "55027"; + parsed.pods["pod-a"].ports.poolerApiPort = "55027"; const raw = JSON.stringify(parsed); await writeFile(file, raw); const reg = await PortRegistry.load(file); expect(reg.get("pod-a")).toBeUndefined(); const allocated = await reg.allocate("pod-a"); - expect(allocated).toEqual(expect.objectContaining({ dbPort: 55000, apiPort: 55001 })); + expect(allocated.ports).toEqual(expect.objectContaining({ dbPort: 55000, apiPort: 55001 })); const quarantined = await readFile(`${file}.corrupt`, "utf8"); expect(quarantined).toBe(raw); @@ -150,22 +176,31 @@ describe("PortRegistry", () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", ports(55010, 55011)); - await reg.restore("pod-b", ports(55030, 55031)); + await reg.restore("pod-a", podPorts(55010, 55011, 45010, 45011)); + await reg.restore("pod-b", podPorts(55030, 55031, 45030, 45031)); - expect(reg.get("pod-a")).toEqual(expect.objectContaining({ dbPort: 55010, apiPort: 55011 })); - expect(reg.get("pod-b")).toEqual(expect.objectContaining({ dbPort: 55030, apiPort: 55031 })); + expect(reg.get("pod-a")?.ports).toEqual( + expect.objectContaining({ dbPort: 55010, apiPort: 55011 }), + ); + expect(reg.get("pod-a")?.internalPorts).toEqual( + expect.objectContaining({ dbPort: 45010, apiPort: 45011 }), + ); + expect(reg.get("pod-b")?.ports).toEqual( + expect.objectContaining({ dbPort: 55030, apiPort: 55031 }), + ); // New allocations must skip the restored ports. const next = await reg.allocate("new-pod"); - expect([next.dbPort, next.apiPort]).not.toContain(55010); - expect([next.dbPort, next.apiPort]).not.toContain(55011); - expect([next.dbPort, next.apiPort]).not.toContain(55030); - expect([next.dbPort, next.apiPort]).not.toContain(55031); + expect([next.ports.dbPort, next.ports.apiPort]).not.toContain(55010); + expect([next.ports.dbPort, next.ports.apiPort]).not.toContain(55011); + expect([next.ports.dbPort, next.ports.apiPort]).not.toContain(55030); + expect([next.ports.dbPort, next.ports.apiPort]).not.toContain(55031); + expect([next.internalPorts.dbPort, next.internalPorts.apiPort]).not.toContain(45010); + expect([next.internalPorts.dbPort, next.internalPorts.apiPort]).not.toContain(45011); // Persisted, so a reload sees the restored allocation. const reloaded = await PortRegistry.load(file); - expect(reloaded.get("pod-a")).toEqual( + expect(reloaded.get("pod-a")?.ports).toEqual( expect.objectContaining({ dbPort: 55010, apiPort: 55011 }), ); }); @@ -174,35 +209,52 @@ describe("PortRegistry", () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", ports(55010, 55011)); - await expect(reg.restore("pod-a", ports(55010, 55011))).resolves.toBeUndefined(); - expect(reg.get("pod-a")).toEqual(expect.objectContaining({ dbPort: 55010, apiPort: 55011 })); + await reg.restore("pod-a", podPorts(55010, 55011, 45010, 45011)); + await expect( + reg.restore("pod-a", podPorts(55010, 55011, 45010, 45011)), + ).resolves.toBeUndefined(); + expect(reg.get("pod-a")?.ports).toEqual( + expect.objectContaining({ dbPort: 55010, apiPort: 55011 }), + ); }); it("throws if the pod already has different ports recorded", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", ports(55010, 55011)); - await expect(reg.restore("pod-a", ports(55010, 55099))).rejects.toThrow(); + await reg.restore("pod-a", podPorts(55010, 55011, 45010, 45011)); + await expect(reg.restore("pod-a", podPorts(55010, 55099, 45010, 45099))).rejects.toThrow(); }); it("throws if a port is already assigned to a different pod", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", ports(55010, 55011)); - await expect(reg.restore("pod-b", ports(55010, 55030))).rejects.toThrow(); - await expect(reg.restore("pod-b", ports(55030, 55011))).rejects.toThrow(); + await reg.restore("pod-a", podPorts(55010, 55011, 45010, 45011)); + await expect(reg.restore("pod-b", podPorts(55010, 55030, 45030, 45031))).rejects.toThrow(); + await expect(reg.restore("pod-b", podPorts(55030, 55011, 45030, 45031))).rejects.toThrow(); + await expect(reg.restore("pod-b", podPorts(55030, 55031, 45010, 45030))).rejects.toThrow(); }); it("throws if a restored db port collides with another pod's api port", async () => { const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); const reg = await PortRegistry.load(file); - await reg.restore("pod-a", ports(55010, 55011)); - await expect(reg.restore("pod-b", ports(55011, 55030))).rejects.toThrow(); - await expect(reg.restore("pod-b", ports(55030, 55010))).rejects.toThrow(); + await reg.restore("pod-a", podPorts(55010, 55011, 45010, 45011)); + await expect(reg.restore("pod-b", podPorts(55011, 55030, 45030, 45031))).rejects.toThrow(); + await expect(reg.restore("pod-b", podPorts(55030, 55010, 45030, 45031))).rejects.toThrow(); + }); + + it("keeps internal allocations out of the public proxy range", async () => { + const file = join(await mkdtemp(join(tmpdir(), "ports-")), "state.json"); + await writeFile(file, JSON.stringify({ basePort: 65000, internalBasePort: 45000, pods: {} })); + const reg = await PortRegistry.load(file); + + const allocated = await reg.allocate("pod-high"); + + expect(allocated.ports.dbPort).toBe(65000); + expect(allocated.internalPorts.dbPort).toBe(45000); + expect(Object.values(allocated.internalPorts).every((port) => port < 55000)).toBe(true); }); }); }); diff --git a/packages/fleet/src/Provisioner.ts b/packages/fleet/src/Provisioner.ts index f64a3923b4..dc8709cf91 100644 --- a/packages/fleet/src/Provisioner.ts +++ b/packages/fleet/src/Provisioner.ts @@ -73,7 +73,8 @@ export class Provisioner { versions: resolvedVersions, services: opts.services ?? {}, flags: { supautils: opts.flags?.supautils ?? false }, - ports: allocated, + ports: allocated.ports, + internalPorts: allocated.internalPorts, postgresPassword, createdAt: new Date().toISOString(), }; @@ -137,7 +138,8 @@ export class Provisioner { const manifest: PodManifest = { ...source, id: newId, - ports: allocated, + ports: allocated.ports, + internalPorts: allocated.internalPorts, createdAt: new Date().toISOString(), }; await pods.write(manifest); diff --git a/packages/fleet/src/Provisioner.unit.test.ts b/packages/fleet/src/Provisioner.unit.test.ts index ba1421fd8d..9446b00518 100644 --- a/packages/fleet/src/Provisioner.unit.test.ts +++ b/packages/fleet/src/Provisioner.unit.test.ts @@ -60,7 +60,10 @@ describe("Provisioner (unit, fake deps)", () => { expect(manifest.id).toBe("x"); expect(manifest.postgresPassword).toBe("secret-password"); - expect(ports.get("x")).toEqual(manifest.ports); + expect(ports.get("x")).toEqual({ + ports: manifest.ports, + internalPorts: manifest.internalPorts, + }); expect(await pods.read("x")).toEqual(manifest); }); @@ -88,7 +91,10 @@ describe("Provisioner (unit, fake deps)", () => { // The duplicate-id pre-check throws before any (re-)allocation, so the // original pod's ports must still be intact. - expect(ports.get("dup")).toEqual(first.ports); + expect(ports.get("dup")).toEqual({ + ports: first.ports, + internalPorts: first.internalPorts, + }); }); it("records resolved default versions for enabled warm services", async () => { @@ -173,7 +179,11 @@ describe("Provisioner (unit, fake deps)", () => { expect(forked.id).toBe("dst"); expect(forked.ports).not.toEqual(source.ports); - expect(ports.get("dst")).toEqual(forked.ports); + expect(forked.internalPorts).not.toEqual(source.internalPorts); + expect(ports.get("dst")).toEqual({ + ports: forked.ports, + internalPorts: forked.internalPorts, + }); expect(await pods.read("dst")).toEqual(forked); }); @@ -209,8 +219,14 @@ describe("Provisioner (unit, fake deps)", () => { // The pre-check for an already-existing target throws before // (re-)allocating, so neither pod's ports should be disturbed. - expect(ports.get("src")).toEqual(source.ports); - expect(ports.get("dst")).toEqual(existing.ports); + expect(ports.get("src")).toEqual({ + ports: source.ports, + internalPorts: source.internalPorts, + }); + expect(ports.get("dst")).toEqual({ + ports: existing.ports, + internalPorts: existing.internalPorts, + }); }); }); }); diff --git a/packages/stack/src/DaemonServer.integration.test.ts b/packages/stack/src/DaemonServer.integration.test.ts index 74982f262b..76ccf6a9bb 100644 --- a/packages/stack/src/DaemonServer.integration.test.ts +++ b/packages/stack/src/DaemonServer.integration.test.ts @@ -46,6 +46,7 @@ const MOCK_LOGS: ReadonlyArray = [ function mockStack() { let stopped = false; + let waitAllReadyCalls = 0; const serviceCalls: string[] = []; const layer = Layer.succeed(Stack, { @@ -105,7 +106,10 @@ function mockStack() { allStateChanges: () => Stream.fromIterable(MOCK_STATES), waitReady: (name: string) => name === "unknown" ? Effect.fail(new ServiceNotFoundError({ name })) : Effect.void, - waitAllReady: () => Effect.void, + waitAllReady: () => + Effect.sync(() => { + waitAllReadyCalls += 1; + }), subscribeLogs: (name: string) => Stream.fromIterable(MOCK_LOGS.filter((l) => l.service === name)), subscribeAllLogs: (services?: ReadonlyArray) => @@ -130,6 +134,9 @@ function mockStack() { get stopped() { return stopped; }, + get waitAllReadyCalls() { + return waitAllReadyCalls; + }, serviceCalls, }; } @@ -216,6 +223,15 @@ describe("DaemonServer", () => { expect(text).toContain("postgres"); }); + test("GET /ready delegates readiness to the stack", async () => { + const before = mock.waitAllReadyCalls; + const res = await fetch(`${url}/ready`); + expect(res.status).toBe(200); + const body = (await res.json()) as { ok: boolean }; + expect(body.ok).toBe(true); + expect(mock.waitAllReadyCalls).toBe(before + 1); + }); + // ------------------------------------------------------------------------- // Logs // ------------------------------------------------------------------------- diff --git a/packages/stack/src/DaemonServer.ts b/packages/stack/src/DaemonServer.ts index 8c05c63e6c..4467613ba2 100644 --- a/packages/stack/src/DaemonServer.ts +++ b/packages/stack/src/DaemonServer.ts @@ -76,6 +76,28 @@ export class DaemonServer extends Context.Service< ), ), + // Ready: delegate readiness semantics to the stack implementation + HttpRouter.route( + "GET", + "/ready", + Effect.gen(function* () { + yield* stack.waitAllReady(); + return HttpServerResponse.jsonUnsafe({ ok: true }); + }).pipe( + Effect.catchTag("ServiceReadyError", (e) => + Effect.succeed( + HttpServerResponse.jsonUnsafe( + { error: e.reason, service: e.name }, + { status: 500 }, + ), + ), + ), + Effect.catchTag("StackBuildError", (e) => + Effect.succeed(HttpServerResponse.jsonUnsafe({ error: e.detail }, { status: 500 })), + ), + ), + ), + // Start: begin service startup HttpRouter.route( "POST", diff --git a/packages/stack/src/RemoteStack.ts b/packages/stack/src/RemoteStack.ts index 61132817e5..28dddacaba 100644 --- a/packages/stack/src/RemoteStack.ts +++ b/packages/stack/src/RemoteStack.ts @@ -3,6 +3,7 @@ import { Effect, Layer, Schema, Stream } from "effect"; import * as Sse from "effect/unstable/encoding/Sse"; import { HttpClientRequest, HttpClientResponse } from "effect/unstable/http"; import { Stack, StackInfoSchema } from "./Stack.ts"; +import { StackBuildError } from "./errors.ts"; import { StackServiceState, StackServiceStatusSchema } from "./StackServiceState.ts"; import { UnixHttpClient, UnixHttpClientError } from "./UnixHttpClient.ts"; @@ -34,6 +35,7 @@ const StatusResponseSchema = Schema.Struct({ const ServiceErrorResponseSchema = Schema.Struct({ error: Schema.String, + service: Schema.optionalKey(Schema.String), }); const StatusServiceEventSchema = Schema.fromJsonString(StatusServiceSchema); @@ -421,35 +423,20 @@ export const RemoteStack = { waitAllReady: () => withUnixHttpClient( Effect.gen(function* () { - // Check current state first - const { services } = yield* fetchStatus(socketPath, "/status"); - const allReady = services.every( - (s) => s.status === "Healthy" || s.status === "Running", - ); - if (allReady) return; - - // Track service readiness via SSE - const readySet = new Set( - services - .filter((s) => s.status === "Healthy" || s.status === "Running") - .map((s) => s.name), - ); - const totalCount = services.length; - - yield* withUnixHttpClient( - sseStream(socketPath, "/status/stream", (data) => { - const raw = decodeStatusServiceEvent(data); - return toServiceState(raw); - }).pipe( - Stream.takeUntil((s) => { - if (s.status === "Healthy" || s.status === "Running") { - readySet.add(s.name); - } - return readySet.size >= totalCount; - }), - Stream.runDrain, - ), - ); + const response = yield* unixResponse(socketPath, "/ready"); + if (response.status === 500) { + const body = yield* HttpClientResponse.schemaBodyJson(ServiceErrorResponseSchema)( + response, + ).pipe(Effect.orDie); + if (body.service !== undefined) { + return yield* new ServiceReadyError({ + name: body.service, + reason: body.error, + }); + } + return yield* new StackBuildError({ detail: body.error }); + } + yield* HttpClientResponse.filterStatusOk(response).pipe(Effect.orDie); }), ), diff --git a/packages/stack/src/StackLifecycleCoordinator.ts b/packages/stack/src/StackLifecycleCoordinator.ts index cf78f796f7..99348c9530 100644 --- a/packages/stack/src/StackLifecycleCoordinator.ts +++ b/packages/stack/src/StackLifecycleCoordinator.ts @@ -292,6 +292,17 @@ export class StackLifecycleCoordinator extends Context.Service< } return match; }); + const requireRunningForServiceStart = (name: string) => + Effect.gen(function* () { + const phase = yield* Ref.get(phaseRef); + if (phase !== "running") { + return yield* Effect.fail( + new StackBuildError({ + detail: `Cannot start service "${name}" while the stack is ${phase}. Call start() first and retry after it finishes.`, + }), + ); + } + }); let preparedArtifacts: PreparedStackArtifacts | undefined; let prepareDeferred: Deferred.Deferred | undefined; @@ -637,6 +648,7 @@ export class StackLifecycleCoordinator extends Context.Service< startService: (name) => Effect.gen(function* () { yield* requireKnownService(name); + yield* requireRunningForServiceStart(name); const runtime = yield* ensureRuntime; yield* runtime.orchestrator.startService(name); yield* markStarted([name]); diff --git a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts index b98f032a92..1320e51177 100644 --- a/packages/stack/src/StackLifecycleCoordinator.unit.test.ts +++ b/packages/stack/src/StackLifecycleCoordinator.unit.test.ts @@ -351,6 +351,43 @@ describe("StackLifecycleCoordinator lazyServices", () => { ); }); + it.live("rejects on-demand service starts before start()", () => { + const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-lazy-idle-test-")); + const config = makeLazyConfig(dataDir, defaultPorts.postgrestPort); + const { layer } = setupLayer(config); + + return Effect.gen(function* () { + const stack = yield* Stack; + const exit = yield* stack.startService("postgrest").pipe(Effect.exit); + + expect(Exit.isFailure(exit)).toBe(true); + expect(JSON.stringify(exit)).toContain("while the stack is idle"); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), + ); + }); + + it.live("rejects on-demand service starts after stop()", () => { + const dataDir = mkdtempSync(join(tmpdir(), "stack-lifecycle-coordinator-lazy-stopped-test-")); + const config = makeLazyConfig(dataDir, defaultPorts.postgrestPort); + const { layer } = setupLayer(config); + + return Effect.gen(function* () { + const stack = yield* Stack; + yield* stack.start(); + yield* stack.stop(); + + const exit = yield* stack.startService("postgrest").pipe(Effect.exit); + + expect(Exit.isFailure(exit)).toBe(true); + expect(JSON.stringify(exit)).toContain("while the stack is stopped"); + }).pipe( + Effect.provide(layer), + Effect.ensuring(Effect.sync(() => rmSync(dataDir, { recursive: true, force: true }))), + ); + }); + // Regression test: waitAllReady() used to unconditionally delegate to // orchestrator.waitAllReady() over the FULL graph startOrder. Under lazyServices, services // that were never started (e.g. postgrest, which stays "Pending" until the ApiProxy's diff --git a/packages/stack/src/services/pooler.ts b/packages/stack/src/services/pooler.ts index 6a17fa34c8..576fa87765 100644 --- a/packages/stack/src/services/pooler.ts +++ b/packages/stack/src/services/pooler.ts @@ -63,8 +63,9 @@ params = %{ }] } -if !Supavisor.Tenants.get_tenant_by_external_id(params["external_id"]) do - {:ok, _} = Supavisor.Tenants.create_tenant(params) +case Supavisor.Tenants.get_tenant_by_external_id(params["external_id"]) do + nil -> {:ok, _} = Supavisor.Tenants.create_tenant(params) + tenant -> {:ok, _} = Supavisor.Tenants.update_tenant(tenant, params) end`; export const makePoolerServiceDocker = (opts: DockerPoolerOptions): ServiceDef => diff --git a/packages/stack/src/services/postgres.ts b/packages/stack/src/services/postgres.ts index 122882c396..f7dee6ad0a 100644 --- a/packages/stack/src/services/postgres.ts +++ b/packages/stack/src/services/postgres.ts @@ -92,6 +92,7 @@ ALTER USER supabase_auth_admin WITH PASSWORD :'pgpass'; ALTER USER supabase_storage_admin WITH PASSWORD :'pgpass'; ALTER USER supabase_replication_admin WITH PASSWORD :'pgpass'; ALTER USER supabase_read_only_user WITH PASSWORD :'pgpass'; +ALTER USER pgbouncer WITH PASSWORD :'pgpass'; create schema if not exists _realtime; alter schema _realtime owner to postgres; SELECT 'CREATE DATABASE _supabase WITH OWNER postgres' diff --git a/packages/stack/src/services/services.unit.test.ts b/packages/stack/src/services/services.unit.test.ts index 6235ce4660..5c7f24f6d1 100644 --- a/packages/stack/src/services/services.unit.test.ts +++ b/packages/stack/src/services/services.unit.test.ts @@ -267,6 +267,7 @@ describe("makePostgresServiceDocker", () => { expect(script).toContain("\\connect _supabase"); expect(script).toContain("create schema if not exists _analytics;"); expect(script).toContain("create schema if not exists _supavisor;"); + expect(script).toContain("ALTER USER pgbouncer WITH PASSWORD :'pgpass';"); }); }); @@ -760,7 +761,10 @@ describe("docker-backed auxiliary services", () => { expect(def.args).toContain(`54329:${poolerContainerPorts.admin}`); expect(def.args).toContain(`54330:${poolerContainerPorts.transaction}`); const args = def.args ?? []; - expect(args.some((arg) => arg.startsWith("SUPAVISOR_TENANT_SCRIPT="))).toBe(true); + const tenantScriptArg = args.find((arg) => arg.startsWith("SUPAVISOR_TENANT_SCRIPT=")); + expect(tenantScriptArg).toBeDefined(); + expect(tenantScriptArg).toContain("Supavisor.Tenants.create_tenant(params)"); + expect(tenantScriptArg).toContain("Supavisor.Tenants.update_tenant(tenant, params)"); expect(args.join(" ")).toContain('supavisor eval "$SUPAVISOR_TENANT_SCRIPT"'); }); });