feat: micro Supabase stacks phase 1 — @supabase/fleet + stack micro profile#5819
feat: micro Supabase stacks phase 1 — @supabase/fleet + stack micro profile#5819jgoux wants to merge 40 commits into
Conversation
High-density Postgres + minimal stack design: pod architecture, micro.conf profile, CoW templates, fleet daemon with suspend-on-idle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds installMicroProfile/readPreloadLibraries/writePreloadLibraries to layer the micro postgres profile onto an existing PGDATA directory via include_if_exists lines in postgresql.conf, idempotently.
…f writes Address code review findings on the PGDATA conf-layering utilities: - fix TS2532 undefined-narrowing in readPreloadLibraries - match include_if_exists only when active (not commented out) so a disabled include no longer silently short-circuits profile install - accept flexible spacing/quoting in shared_preload_libraries parsing and trim comma-separated entries - writePreloadLibraries now updates/appends the preload line in place instead of clobbering the rest of pod.conf
….6.1.143 Wire postgres.provisioned (skip postgres-init for pre-initialized template clones) and postgres.profile (skip -c runtime args on "micro", since those settings live in micro.conf/pod.conf inside PGDATA) through StackBuilder and the native postgres service. Bump DEFAULT_VERSIONS.postgres to 17.6.1.143.
Adds enableExtension(name) across the coordinator, Stack service, RemoteStack/DaemonServer HTTP surface, and public StackHandle. When an extension requires shared_preload_libraries (per PRELOAD_REQUIRED_EXTENSIONS) and isn't already preloaded, it appends to pod.conf and restarts postgres; otherwise it's a no-op so plain CREATE EXTENSION keeps working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two concurrent enableExtension calls could both read the same shared_preload_libraries list, then write independently, silently dropping one extension when the second write clobbered the first. Concurrent restarts of the same "postgres" service also deadlocked the orchestrator. Guard the whole enableExtension body with a per-coordinator Effect Semaphore(1) so calls are serialized. Add a regression test that fires two enableExtension calls concurrently through the real StackLifecycleCoordinator (mocked binary resolver/spawner, temp PGDATA) and asserts both extensions land in pod.conf; without the fix the test times out instead of completing.
Adds StackConfig.lazyServices: when true, StackLifecycleCoordinator.start() only eagerly starts postgres (and postgres-init); every other service starts on demand the first time the ApiProxy routes a request to it, via a new ProxyConfig.ensureService hook memoized by lazyServices.ts so concurrent first requests trigger a single start. Deviates from the original sketch's `enabled: false` mechanic: process-compose's buildGraph excludes disabled ServiceDefs from the dependency graph entirely, so a lazily-disabled service can never be resurrected via startService/ updateServiceDefinition once the orchestrator is built. Services stay enabled in the graph; laziness is enforced by skipping the eager start in the coordinator instead.
StackLifecycleCoordinator.waitAllReady() unconditionally delegated to orchestrator.waitAllReady() over the full graph startOrder. Under lazyServices, services that are never started on demand (e.g. postgrest) never resolve their healthy deferred and never emit a Failed state, so ready() hung forever. Track the set of services that have actually been started (eager set from start(), plus startService/restartService/enableExtension/reload* calls). Under lazyServices, waitAllReady() now waits per-service on that snapshot instead of the full graph. Non-lazy behavior is unchanged: it still calls orchestrator.waitAllReady() over the whole graph.
Adds PodManifest interface plus templateKey/baseTemplateKey helpers that derive stable sha256-based cache keys from a partial VersionManifest, independent of key order.
cloneDir() now rm -rf's dest before falling back to fs.cp when the platform-specific cp -Rc/--reflink step exits non-zero, so a partially written dest from a failed CoW attempt can't silently mix with fresh fallback output. The fs.cp fallback also now passes verbatimSymlinks: true so relative symlinks in src aren't rewritten to absolute paths that point back into the source tree. Adds an internal, documented `cowCommand` test seam (CloneDirOptions) to force the CoW step to fail deterministically in tests, exercising the fallback-cleanup and symlink-preservation paths without depending on filesystem-specific CoW failure conditions.
Introduces PortRegistry as the single owner of the port space for all pods on a host, replacing the plan for a cross-stack filesystem scan (readReservedPorts()). Persists pod->port allocations to a JSON state file, is idempotent per pod, and reuses freed ports on release.
- load() no longer crashes on invalid JSON or structurally invalid
state (missing/NaN basePort, non-object pods); it quarantines the
bad file to `${stateFile}.corrupt` (overwriting any prior
quarantine) and starts from fresh empty state instead.
- Add restore(podId, ports) so the fleet daemon can re-seed known
allocations from each pod's manifest after a quarantine/reset,
without a full port scan. Idempotent for identical ports; throws on
conflicting ports for the same pod or ports already held by another
pod.
- Document the single-owner / no-port-probing / quarantine-and-restore
design assumptions on the class.
Adds TemplateStore, which builds golden Postgres data-dir templates by running @supabase/stack one-shot: base = postgres-only boot so postgres-init applies baseline migrations, then installMicroProfile freezes the result; warm = clone of base + enabled services booted once to self-migrate. Builds are concurrency-safe via a per-key wx-flag lockfile with a 10min stale threshold. Also exports installMicroProfile/readPreloadLibraries/writePreloadLibraries from stack's index so fleet can consume them.
PodRegistry persists pod manifests at podsRoot/<id>/pod.json. Provisioner creates pods by allocating ports and CoW-cloning a base or warm template into the pod's data dir, resets pods by re-cloning from the base template, forks a pod's data dir under a fresh id and ports, and destroys pods by removing their directory and releasing ports.
Provisioner.create() and fork() allocated ports before cloning the data directory and writing the pod manifest. If cloneDir or pods.write threw, the allocated port pair stayed recorded in the PortRegistry state file with no pod ever created to release it. Wrap the post-allocation steps in try/catch and release the ports (plus best-effort cleanup of any partially-written data/pod dir) before rethrowing. Pre-checks (duplicate id, unknown source) still run before allocation so failure cleanup can never release a pre-existing pod's ports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds EdgeProxy: binds a pod's external port once and keeps it bound for the pod's lifetime, awaiting wake() per accepted connection to get the live upstream before splicing bytes both ways. Emits connect/data/disconnect activity per pod for the idle monitor to consume. wake() rejection destroys the client without unhandled rejections; concurrent connections to the same suspended pod both succeed regardless of how many times wake() runs (dedup is the fleet layer's job). Buffers client bytes behind a real 'data' listener (not just pause()) before replaying them to the backend, since some runtimes start sockets flowing before user code attaches a listener and would otherwise silently drop data received while wake() is in flight.
Two review findings in EdgeProxy: (1) a client streaming data at a suspended pod during a slow wake() buffered without limit; add MAX_PREWAKE_BUFFER_BYTES (1 MiB) and destroy the client socket if it's exceeded before the backend is reachable. Fixing this exposed that the prior data+manual-array buffering never actually delivered data events while the socket was paused (confirmed under both Bun and Node), so the buffering mechanism was switched to readable+read(), which fires while paused and makes byte counting/capping actually work. (2) wake() resolving with a malformed upstream (e.g. an invalid port) could throw synchronously inside the .then() resolve handler, surfacing as an unhandled rejection; wrap that branch in try/catch routed to the same client-destroy path, and correct the doc comment's overstated guarantee. Also drop the redundant client.resume() after pipe() and note that the activity data listener is independent of pipe()'s internal listener. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds IdleMonitor: fires onIdle(podId) once a tracked pod has no open connections and no activity for idleMs. Open connections hold the pod warm indefinitely; any recordActivity call (re)arms the countdown; onIdle fires at most once per warm period and untracks the pod.
Ties together TemplateStore/PodRegistry/PortRegistry/Provisioner with EdgeProxy/IdleMonitor into createFleet(): pods stay suspended until their first proxied connection wakes an in-process @supabase/stack StackHandle, and idle warm pods are suspended again after idleMs with no open connections. Startup reconciliation kills stale run.pid processes from a previous daemon and re-seeds PortRegistry from each pod's manifest via restore(). Also exports PRELOAD_REQUIRED_EXTENSIONS from @supabase/stack's index so the suspended-pod enableExtension path can guard non-preload extensions without a restart.
Two race/leak fixes surfaced in review of the Fleet facade: - suspend() deleted from `warm` before awaiting stack.dispose(), and wakeUpstream()/destroyPod/resetPod/forkPod had no way to see an in-flight wake (tracked only in wakesInFlight, not yet `warm`) before mutating the same pod's data dir. Added a small per-pod async lock (PodLock, src/podLock.ts) and routed the wake body, suspend's full body, and the lifecycle-affecting sections of destroyPod/resetPod/ forkPod through it, so these can never interleave against the same pod's process/data dir. Wake dedup via wakesInFlight stays outside the lock so concurrent connections still share one wake. - Startup reconciliation killed `-run.pid` (the daemon's own pid), but process-compose/createStack spawn postgres `detached: true` in its own process group, so that kill missed stale postmasters entirely. Reap now keys off postgres's own ground truth, `<dataDir>/postmaster.pid` (src/reapStalePostmaster.ts): its first line is the postmaster pid, which is always its own process-group leader, so `-pid` reliably reaches the whole tree. run.pid is kept as a "was running" hint but no longer drives the kill decision. Also documented the enableExtension asymmetry: a suspended pod can only durably record intent for preload-required extensions; a non-preload extension request against a suspended pod is a silent no-op until the pod is woken. Adds a gated Fleet integration stress test interleaving suspend/wake/ query calls, plus unit tests for both new modules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the density guardrail E2E (register N pods, wake a subset, confirm the rest stay at zero processes, explicit suspend) gated behind FLEET_PG_TESTS with pod count tunable via FLEET_E2E_PODS. Mirrors packages/stack's e2e vitest project (fileParallelism: false) so nx picks up a test:e2e target for @supabase/fleet, and extends the package's knip entry points to cover tests/**/*.ts. Also adds the package README covering the public API, wake/suspend/fork lifecycle, and phase-1 limitations (native-only, kill-then-resuspend reconciliation, HTTP/realtime lazy-start gaps).
- Add enableExtension stub to Stack mocks (mocks.ts, running-stack.ts) to match the Stack service contract added in this branch. - Fix StackLifecycleCoordinator.unit.test.ts flakiness under Bun: widen the projected-status assertion to tolerate the transient "Restarting" state, and bind the fake healthy postgrest server on an OS-assigned port (listen(0)) instead of a hard-coded port to avoid EADDRINUSE under parallel test runs. - Remove unused effect and @supabase/process-compose dependencies from packages/fleet/package.json (zero imports), reformat with oxfmt, and move @tsconfig/bun into knip's ignoreDependencies (false positive — used only via tsconfig extends). - Sync pnpm-lock.yaml after the dependency removal.
![depthfirst-app[bot]](https://avatars.githubusercontent.com/in/1021002?s=60&v=4){kind=small}
The packages/fleet importer referenced oxfmt@0.56.0/oxlint@1.71.0 while the lockfile package set and catalogs snapshot had moved on, breaking frozen installs in CI (ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY). Verified with a clean-clone 'pnpm install --frozen-lockfile' from scratch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lockfile regenerated against merged manifests; verified with a from-scratch 'pnpm install --frozen-lockfile'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Supabase CLI previewnpx --yes https://pkg.pr.new/supabase/cli/supabase@327e836a7cb4eb50461925f5e966fad53d1f4c83Preview package for commit |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f561ded1ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ix flaky test The shared mock ChildProcessSpawner made every spawned process exit with code 0 after a fixed 10ms, including long-running daemons (postgres, postgrest). Under the orchestrator's `unless-stopped` restart policy this turned each daemon into a crash-loop: RestartTriggered -> exponential backoff -> respawn, repeating with growing backoff (1s, 2s, 4s...). The projected status never settled on Running/Healthy, so StackLifecycleCoordinator.unit.test.ts's poll-until-ready helper spun until the 5s test timeout (~30-40% of runs under Bun; never under Node, whose timer scheduling happened to keep the churn tighter). This is a mock-fidelity artifact, not a product bug: real daemons run until stopped, so the restart loop never engages in production. The Orchestrator's restart/waitReady/resetService logic is correct and unchanged. The mock now models a spawned process as a long-running daemon (exit resolves only on kill, reporting 143) when it is a supervised launch wrapping a real binary. Short-lived processes keep the 10ms auto-exit: health-check probes (pg_isready), one-shot init services launched via bash/sh (postgres-init, restart:"no"), and docker commands. This keeps it.live clocks progressing and one-shots completing. Also keep the pre-existing waitForReadyStatus test edits (same file, same flake theme): they replace two racy projected-status snapshot assertions with a poll-until-settled helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1c8854de5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9fc3f471e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69b4bc810e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69b4bc810e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e31c93f9f3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f5953c845
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Update Docker pgbouncer password too
When the stack uses Docker-backed Postgres with the pooler enabled and POSTGRES_PASSWORD is customized, makePoolerServiceDocker() now configures Supavisor's tenant with db_user: "pgbouncer" and db_password: opts.dbPassword, but this Docker bootstrap SQL still updates every bundled login except pgbouncer. The native postgres-init path includes that role, so the Docker path will leave pooler clients authenticating with the new password against a role that still has the image's default password.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Blacksmith runners detected OOM events on the following jobs:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 327e836a7c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| * other service is started on demand by the ApiProxy on the first request to its route, instead | ||
| * of starting everything up front. Default false: existing eager-start behavior is unchanged. | ||
| */ | ||
| readonly lazyServices?: boolean; |
There was a problem hiding this comment.
When lazyServices is enabled on a detached/daemon stack, the foreground coordinator has the new started-service-only waitAllReady() behavior, but checked RemoteStack.waitAllReady() and it still watches /status until every advertised service becomes Healthy/Running. Lazy services that have not received traffic deliberately stay Pending, so StackHandle.ready() against a daemon-backed lazy stack can hang after start() has correctly started only postgres; expose a daemon ready endpoint or mirror the started-service semantics in the remote client.
Useful? React with 👍 / 👎.
|
|
||
| const postgresDockerEnv = (opts: DockerPostgresOptions): Record<string, string> => ({ | ||
| POSTGRES_PASSWORD: "postgres", | ||
| POSTGRES_PASSWORD: opts.password, |
There was a problem hiding this comment.
The native postgres-init path now rotates pgbouncer, but in Docker-backed Postgres this changed the container to use a custom POSTGRES_PASSWORD while DOCKER_POSTGRES_SCHEMA_SQL still only updates postgres/auth/storage/etc. and skips the pgbouncer role. With mode: "docker" or an auto fallback plus pooler enabled and a custom password, Supavisor creates the tenant with opts.dbPassword but pgbouncer.get_auth still returns the old/default role password, so pooler clients cannot authenticate; include pgbouncer in the Docker schema password updates too.
Useful? React with 👍 / 👎.
| "users" => [%{ | ||
| "db_user" => "pgbouncer", | ||
| "db_password" => "postgres", | ||
| "db_password" => ${JSON.stringify(opts.dbPassword)}, |
There was a problem hiding this comment.
When a pooler-enabled stack is restarted after POSTGRES_PASSWORD changes, this puts the new password into the tenant params, but the script below only calls create_tenant if the tenant does not already exist. Existing _supabase tenant rows keep the old db_password while postgres-init rotates the pgbouncer role to the new password, so the pooler comes up with stale credentials; update/recreate the existing tenant user when the configured password changes.
Useful? React with 👍 / 👎.
| const ensureService = makeEnsureServiceMemo((name: ServiceName) => | ||
| Effect.runPromise( | ||
| Effect.gen(function* () { | ||
| yield* stack.startService(name); |
There was a problem hiding this comment.
With lazyServices enabled, the API proxy is bound as soon as createStack() builds the layer, and this unconditionally starts services on any matching request even if stack.start() has not run yet or after stack.stop() left the proxy alive. In those idle/stopped phases external traffic can resurrect Postgres/sidecars while lifecycle state still says stopped and can bypass start()-only setup such as function runtime configuration; make the proxy refuse/502 until the stack has been started, or move the lifecycle/config guard into startService.
Useful? React with 👍 / 👎.
| database: "postgres", | ||
| }); | ||
|
|
||
| const internalPort = (externalPort: number): number => externalPort - INTERNAL_PORT_OFFSET; |
There was a problem hiding this comment.
Fresh evidence after the per-pod port allocation fix is that internal stack ports are still derived by subtracting a fixed 10,000 from externally allocated ports. Because the external registry allocates upward from 55000 through 65535, a dense fleet eventually maps a later pod's internal port back into the proxy-owned external range (for example an external db port near 65008 becomes internal 55008), colliding with an existing EdgeProxy listener and making wake fail with EADDRINUSE; allocate or reserve a truly disjoint internal range instead of deriving it from public ports.
Useful? React with 👍 / 👎.
1 participant
What
Phase 1 of the micro Supabase stacks design (spec, plan): run 100+ Supabase-compatible Postgres pods on one small machine for agent worktrees, local dev, and eventually free-tier workloads.
@supabase/stackchangesmicro.ts+ PGDATA conf layering viamicro.conf/pod.confincludes): 16MB shared_buffers, jit off, fsync off,wal_level=logicalwith capped slot retention — settings live in conf files soALTER SYSTEMstill wins.postgres.provisioned): skips the per-bootpostgres-initmigration pass for data dirs cloned from a golden template.enableExtension(name)— preload-on-enable: appends toshared_preload_librariesinpod.confand restarts postgres (sub-second with fsync off); serialized per-coordinator with a semaphore. Exposed on StackHandle + daemon HTTP route.lazyServices:start()eagerly starts only postgres; the ApiProxy starts any other service on the first request to its route (memoized, concurrent-safe).ready()scopes to started services.17.6.1.143.New package:
@supabase/fleetcreateFleet()— a host-level pod orchestrator:forkPod: byte-identical, independently-diverging database branches for agent worktrees.postmaster.pidprocess groups; port registry re-seeded from pod manifests.Measured (gated e2e, Apple Silicon)
Testing
FLEET_PG_TESTS=1, run under Bun).FLEET_E2E_PODS=100green (~3min).check:allgreen (thecli-go:lint-checktimeout seen locally is a pre-existing environment artifact; no Go files touched).Follow-ups (non-blocking, from review)
IdleMonitortimers lackunref();templateKeyshould use ordinal sort; conf-value escaping; PortRegistry deep state validation; realtime WS lazy-start; shared multi-tenant Realtime (spec'd for a later phase).🤖 Generated with Claude Code