feat: consume @supabase/postgrest-typegen for type generation - #1084
Conversation
Integrate the extracted @supabase/postgrest-typegen package as the single
source of truth for type generation, replacing the embedded templates.
- src/lib/generators.ts: rewrite getGeneratorMetadata as a ~30-line adapter
over the package's introspect(). It wraps pgMeta.query into the package's
structural Queryable (throws on {error}), preserves the
Promise<PostgresMetaResult<GeneratorMetadata>> contract, surfaces the first
query error, and still ends the pool. Re-exports GeneratorMetadata from the
package.
- src/server/server.ts: getTypeOutput now calls getGeneratorMetadata +
generateTypescript/Go/Python/Swift, threading GENERATE_TYPES_DEFAULT_SCHEMA,
POSTGREST_VERSION, detect-1:1, and Swift access-control env values. Behavior
freeze: the CLI path still only supports included schemas.
- src/server/routes/generators/*.ts: swap `apply` template imports for the
package's generateX; query params, headers, and error shapes unchanged.
- Delete src/server/templates/*.ts and test/server/templates/go.test.ts;
re-point test/types.test.ts's pgTypeToTsType import and constants.ts's
AccessControl import to the package; drop the now-unused VALID_* constants.
- Keep PostgresMetaRelationships.ts and src/lib/sql/*.sql.ts (they back the
REST endpoints) — accepted temporary duplication.
`npm run check` passes. Full-suite byte-parity validation is Phase 2.3
(PGMETA-114). The dependency is pinned to 1.0.0-alpha.1; local validation
installs it from Verdaccio via an uncommitted scoped .npmrc, and the lockfile
is finalized when the package is published to npm (Phase 3).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B53a2ayu9d16u821FQ5Qqz
The package isn't published to npm yet; pin the dependency to the pkg.pr.new preview build for pg-toolbelt PR #302 so CI can install it. The lockfile must be regenerated (`npm install`) in an environment with network access to pkg.pr.new — the remote sandbox's egress allowlist blocks that host. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B53a2ayu9d16u821FQ5Qqz
The Go/Python/Swift generators emit objects in GeneratorMetadata order, so output depended on the order introspection returned rows (environment-dependent heap order). Apply the package's new sortGeneratorMetadata pass in the getGeneratorMetadata adapter so all four generators receive canonically-ordered metadata. Regenerate the typegen go/python snapshots accordingly: only ordering changes (the `a_view` view moves to its canonical oid position); struct/class contents are byte-identical. TypeScript and Swift sort internally and are unaffected. Requires @supabase/postgrest-typegen with sortGeneratorMetadata (supabase/pg-toolbelt#302). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B53a2ayu9d16u821FQ5Qqz
Follow-up to the sortGeneratorMetadata semantic-key change (supabase/pg-toolbelt#302): the canonical order is now schema+name based, so the Go/Python typegen snapshots are regenerated to alphabetical order. Pure reorder — struct/class contents are byte-identical. TypeScript/Swift sort internally and are unaffected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B53a2ayu9d16u821FQ5Qqz
…wofjdj-work # Conflicts: # src/server/routes/generators/typescript.ts # src/server/server.ts # src/server/templates/typescript.ts
|
@avallete I've fixed up the last things in here and marked it as ready for review. 😃 |
The postgrest-typegen refactor dropped the worker path added in #1102: the route calls generateTypescript() directly, the package formats with prettier inline and exposes no format hook, so format-pool, format-worker, the piscina dependency and the 503 load-shedding path were all left dead while CLAUDE.md still documented the feature. Rather than wait for a format hook upstream, hand the whole generateTypescript call to the worker. Measured on a synthetic public schema (12 columns and 2 foreign keys per table), 400 tables: wall clock 1059ms on a worker vs 1065ms inline, with the longest main-thread block dropping from ~1000ms to 21ms. Metadata crosses the boundary as a structured clone, which is plain JSON and costs nothing measurable. This also covers the ~10% of the cost that is string building rather than prettier, which a format-only hook would have left on the main thread. format-pool/format-worker are renamed to typegen-pool/typegen-worker since they no longer format, keeping the admission control, per-task timeout, idle pool and 503 shedding as they were. The PG_META_FORMAT_* env vars keep their names so existing deployments do not need reconfiguring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0192KEBCPggQgUT7JLPc3Cz1
Review: are #1101 and #1102 preserved?#1101 — kept. #1102 — not kept. Where the time actually goesMeasured against the released package on a synthetic public schema (12 columns and 2 foreign keys per table):
So a format hook upstream would recover most of it, but not all — the string building ahead of prettier is CPU-bound on the main thread too. Suggested fix, working against 0.1.0 todayHand the whole Pushed to While we're here: prettier is the wrong tool for this~90% of type generation is prettier, and the output is machine-generated code that nobody diffs by hand. Smaller things
Generated by Claude Code |
|
Correction to my review above: the #1102 fix is now in this PR, not on a separate branch. The PR description is updated to match: the "remaining blocker" section is replaced by what actually landed. Everything else in the review stands as written — #1101 verbatim in the package, the benchmark numbers, the oxfmt suggestion, and the smaller findings (sequential introspection queries in the CLI path, the deleted go template test, the forked introspection SQL, the CLAUDE.md prettier reflow, the deduped prettier pin). The "byte-identical output" wording is corrected in the description to "identical content, different order". Generated by Claude Code |
…the worker boundary
|
Let's hold unti: supabase/sdk#118 get merged |
…trospection (#118) ## Summary Adds an optional `format` callback to `GenerateTypescriptOptions` for the TypeScript generator, and switches the default formatter itself from prettier to oxfmt. This unblocks supabase/postgres-meta#1084, whose description flags that `@supabase/postgrest-typegen` formats inline with `prettier` and exposed no formatting hook yet, which that PR states should land here first. Tracked in Linear as SDK-1649. The review comment on that PR (supabase/postgres-meta#1084 (comment)) measured prettier at roughly 73-92% of `generateTypescript`'s time and made two asks against this package: 1. "A formatter-choice option... worth raising against `postgrest-typegen` — it already uses `oxfmt`/`oxlint` on its own source, so the dependency is familiar there." Addressed by the `format` option. 2. "~90% of type generation is prettier... `oxfmt` is roughly an order of magnitude faster on this kind of workload." Addressed by switching the default itself, not just making it overridable. Neither of these fully eliminates main-thread blocking on its own; that review's own benchmark achieved its largest win (~1000ms -> ~21ms main-thread block) by additionally wrapping the *entire* `generateTypescript()` call in a worker, which is an architecture choice for the consumer (postgres-meta) to make, not something needed here. ## Changes - `introspect()` now issues its ten introspection queries under `Promise.all` instead of awaiting them one at a time. The same review flagged that postgres-meta's CLI path (`supabase gen types`) went from parallel to sequential in the migration, adding one round trip per query on remote databases; this restores the old parallelism for every consumer. A pooled `Queryable` runs the queries in parallel, a single-connection one pipelines them. - `GenerateTypescriptOptions.format?: (code: string) => Promise<string>` — optional, defaults to a new `oxfmt`-backed formatter (`semi: false`, `printWidth: 80` to match prettier's default and minimize output churn). - `prettier` dropped as a dependency; `oxfmt` moves from a devDependency to a runtime dependency. - The nightly parity job against real postgres-meta (still prettier-formatted) now canonicalizes postgres-meta's TypeScript output through this package's own oxfmt formatter before diffing, so it keeps catching real content drift without flagging the formatter swap itself every night. - `test/parity/expected/typescript.txt` and the inline snapshots in `test/generation/typescript.test.ts` regenerated for the new formatter's output. Verified the only remaining differences from the previous prettier-formatted goldens are formatter style choices (confirmed by reformatting the old prettier golden through the new oxfmt formatter and diffing against a fresh regeneration; the only residual difference is oxfmt adding parentheses around a conditional type in a couple of generic-default positions, which is semantically inert). ## Test plan - [x] `bun run check-types` - [x] `bun run format-and-lint` - [x] `bun run knip` - [x] `bun run build` - [x] `bun run test` (93 pass, including a regression test asserting a custom `format` callback is invoked and its output is used verbatim)
0.2.0 formats with oxfmt instead of prettier (typescript snapshots regenerated; the only output change is oxfmt parenthesizing conditional types in generic-default positions), runs its introspection queries concurrently, and accepts a format hook, which stays unused here since the whole generateTypescript call already runs on the worker.
|
@avallete supabase/sdk#118 is merged and postgrest-typegen v0.2.0 is now released and depended on by this PR :) |
The typegen-pool/typegen-worker rename made the route's unchanged 503 load-shedding block show up as a diff against master because the error class inside it changed name. Keep master's file and identifier names (format-pool, format-worker, FormatQueueFullError, destroyFormatPool, isFormatPoolActive) so the only diff left in these files is the delegation itself: the worker task carries generator metadata into @supabase/postgrest-typegen's generateTypescript instead of a prettier payload, and the exported entry point is generateTypescriptTypes instead of format(code, options). No behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0192KEBCPggQgUT7JLPc3Cz1
… Args, trigger-writable views (#125) ## Summary Ports the worthwhile TypeScript generator fixes from postgres-meta's open template PRs into this package (the templates are being deleted in favor of this package in supabase/postgres-meta#1084, so open fixes there are triaged and re-landed here). Four fixes, one commit each: 1. **Stored generated columns omitted from Insert/Update** (from supabase/postgres-meta#1105): `GENERATED ALWAYS AS ... STORED` columns reject writes in Postgres, but only identity-ALWAYS columns were excluded; both now emit `?: never`. 2. **Non-nullable json narrowed to `NonNullable<Json>`** (from supabase/postgres-meta#1085): the emitted `Json` type includes `null`, so a NOT NULL json/jsonb column structurally permitted null. Known accepted edge: a NOT NULL jsonb column holding a JSON `'null'::jsonb` value still serializes as JS `null`, so Row is optimistic in that case; Insert/Update narrowing is fully sound. 3. **Zero-argument function Args typed `Record<PropertyKey, never>`** (the still-valid half of supabase/postgres-meta#1035): `Args: never` makes postgrest-js treat every zero-argument function as a computed field (`never extends { '': Row }` always holds), dropping same-named columns from `select('*')` results, and an uninhabited `Database` breaks sound type tooling. Verified against postgrest-js, whose `IsMatchingArgs` special-cases `Record<PropertyKey, never>`. 4. **Insert/Update types for INSTEAD OF trigger views** (from supabase/postgres-meta#1062, reimplemented): views made writable by INSTEAD OF triggers got no Insert/Update types. Views now carry `is_insert_enabled`/`is_update_enabled` computed via `pg_relation_is_updatable(oid, true)` (bit 8 INSERT, bit 4 UPDATE; also covers INSTEAD rules), gated independently, and column updatability counts triggers too (`pg_column_is_updatable(oid, attnum, true)` plus an explicit INSTEAD OF INSERT trigger check, since that function only considers the UPDATE event). The origin PR duplicated hand-rolled pg_trigger subqueries with one pair of wrong bit values and left trigger-writable columns degrading to `?: never`, visible in its own snapshot. The two new `PostgresView` fields are additive (metadata version stays 1), documented, and mirrored in the frozen equivalence contract. ## Triage of origin PRs | postgres-meta PR | Verdict | Reasoning | |---|---|---| | #1105 | Ported | Two-line correctness fix; `is_generated` was already introspected. | | #1085 | Ported | Nullability chokepoint fix; function returns and composite attributes untouched. | | #1035 | Ported (zero-arg half) | The computed-field-filtering half is superseded: this package introspects with `includeTableTypes: true`, so table/view row types already resolve (parity golden shows computed fields working). Only foreign-table row types remain uncovered; the PR's name-string matching is too fragile to port for that niche. | | #1062 | Reimplemented | Right idea, broken execution (wrong tgtype bits in one duplicated subquery pair, all-`never` Update output in its own snapshot). | | #1063 (TS part) | Skipped | Superseded: composite attributes already emit `| null` on main; the PR's remaining delta (`unknown | null`) is the identical type. | | #1048 (vector to `number[]`) | Skipped | Wrong as a global remap: PostgREST serializes pgvector as strings in responses, so Row types would regress; the reviewer asked for e2e evidence and got none. Needs input/output-aware mapping, a design discussion. | | #973 (`| string` numeric inserts) | Skipped | Maintainer requested changes: breaking for consumers expecting `number`; per-column overrides are the escape hatch. | | #573 | Skipped | Blanket `| null` on function args/returns is breaking (author concedes); the centralization half is superseded by the current generator; the domain-resolution gap is real but needs a metadata contract extension (feature-scale, raised separately). | | #750 (`Json` to `unknown`) | Skipped | Breaking; major-version decision. | | #1044 (int8 to `bigint`) | Skipped | Breaking, and incorrect without a custom JSON parser. | | #1083 (`bigint_as` option) | Skipped | Feature/option with API design questions, not a fix. | | #814 (json_schema constraint types) | Skipped | New feature. | ## Validation - Unit tests per fix, plus Docker-backed introspection integration tests proving a join view with an INSTEAD OF INSERT trigger introspects as insert-enabled/update-disabled with updatable columns, and auto-updatable views keep both flags. - Parity golden regenerated and reviewed line by line: the only change is 14 zero-argument functions switching `Args: never` to `Args: Record<PropertyKey, never>`. Fixes 1, 2 and 4 have no fixture-visible effect. - `check-types`, `format-and-lint`, `knip`, `build`, `test` (99 pass across 12 files) all green. - Note: the nightly parity job against real postgres-meta will show this intentional drift until postgres-meta consumes a release containing it (supabase/postgres-meta#1084 replaces the templates with this package, closing the gap).
…ls (#123) ## Summary Ports the Swift string literal escaping fix from postgres-meta's open template PRs into this package (the templates are being deleted in favor of this package in supabase/postgres-meta#1084, so open fixes there are triaged and re-landed here). Database-provided names were interpolated raw into Swift string literals, so a double quote, backslash (including interpolation sequences like `\(...)`), or line break in an enum label or column name produced Swift that does not compile (supabase/postgres-meta#1126). A `swiftStringLiteral` helper now escapes per Swift's string literal grammar (backslash, quote, tab, newline, carriage return, remaining C0 controls and DEL as `\u{n}`, plus U+2028/U+2029) and is applied in `generateEnum`, the single rendering point for both vulnerable positions: Postgres enum raw values and `CodingKeys` raw values. ## Triage of origin PRs | postgres-meta PR | Verdict | Reasoning | |---|---|---| | #1128 | Ported | The stronger duplicate: character-loop escaping covering controls and U+2028/U+2029, returns the complete quoted literal, and the author validated output with `swiftc -parse`. | | #1132 | Skipped | Duplicate; regex-based, misses U+2028/U+2029, and returns only inner text so call sites keep hand-placed quotes. The regex would also trip oxlint's control-character rule here. | Identifier positions (enum case names, property names) are already safe through the existing `formatForSwiftTypeName`/`formatForSwiftPropertyName` sanitization, so escaping is only needed at the literal seam. Pre-existing identifier gaps (all-punctuation names yielding empty case names, leading digits) exist upstream too and need a separate sanitization/dedup design; deliberately out of scope. ## Validation - Snapshot test with pathological labels (quote, backslash, `\(now)`, newline, tab, CR, BEL, U+2028) across enum raw values and CodingKeys in Select/Insert/Update, plus a pin that ordinary output stays byte-identical. - Generated pathological output passes `swiftc -parse`. - `check-types`, `format-and-lint`, `knip`, `build`, `test` (95 pass, parity 4/4) all green; parity golden unchanged.
## Summary Bumps the pinned pg-meta image from `v0.98.0` to `v0.99.0` in the shared service-image manifest (`apps/cli-go/pkg/config/templates/Dockerfile`, imported by the TypeScript CLI as its image source). postgres-meta v0.99.0 replaces the embedded type-generation templates with the shared [`@supabase/postgrest-typegen`](https://github.com/supabase/sdk/tree/main/packages/postgrest-typegen) package (supabase/postgres-meta#1084). This is part of a coordinated rollout with the hosted path (supabase/platform#37764) so `gen types` produces the same output locally and via `--project-id`. ## Relationship to supabase#6404 supabase#6404 makes `gen types` run postgrest-typegen in-process, removing the pg-meta container from that command entirely. This pin still matters independently of it: the same manifest entry provides the `pgmeta` service that `supabase start` runs for Studio's local API, and it covers `gen types` for any release cut before supabase#6404 lands. The two do not conflict (different files), and output is consistent either way since v0.99.0 serves the same generator package that supabase#6404 embeds. ## What changes for users Generated TypeScript output changes in two deliberate ways: deterministic metadata ordering (a one-time reordering diff when regenerating existing types) and oxfmt formatting instead of prettier (style-only). Content is otherwise unchanged. ## Validation - `go build ./...` passes in both modules. - The pre-existing `gen types` e2e tests pull this image tag directly, so CI exercises the new release; the image is published on Docker Hub and ECR Public.
Summary
Replaces postgres-meta's embedded type-generation templates/SQL with the released
@supabase/postgrest-typegenpackage (the same engine, extracted tosupabase/sdkand published to npm).src/lib/generators.tsbecomes a thin adapter that wrapspgMeta.queryinto the package's structuralQueryableand callsintrospect(), preserving the historicalgetGeneratorMetadatasignature and{ data, error }contract.Two deliberate behavior changes, both covered by tests:
sortGeneratorMetadatabefore generation, making output ordering deterministic (semantic sort instead of catalog order); the typegen snapshots were regenerated for this. The generated content is unchanged — sorting the lines oftest/server/typegen.tsbefore and after leaves only the new default-schema test as a real difference — but consumers regenerating types will see one large reordering diff.getGeneratorMetadatanow ends the connection pool on error paths too, where the previous implementation only ended it on success.Status
^0.2.0from npm (previously apkg.pr.newpreview build, then^0.1.0). 0.2.0 formats with oxfmt instead of prettier (typescript snapshots regenerated; the only output change is oxfmt parenthesizing conditional types in generic-default positions, which is semantically inert), runs its introspection queries concurrently (restoring the old CLI-path parallelism), and accepts aformathook, which stays unused here since the wholegenerateTypescriptcall already runs on the worker.master. The template fixes that landed after the extraction are already part of the released package with identical logic:ROWS 1return types whenSetofOptionsis emitted)PG_META_GENERATE_TYPES_DEFAULT_SCHEMAdirectly, so the/generators/typescriptroute honored it; the package takes it as an option, and the route wasn't passing it. The route now passesdefaultSchema, with a regression test.Worker-thread generation (#1102) preserved
#1102 added opt-in worker-thread formatting (
PG_META_FORMAT_IN_WORKER) with load shedding. The package formats internally, so formatting alone can no longer be intercepted on this side; instead the wholegenerateTypescriptcall is handed to the worker. Measured on a synthetic public schema (12 columns and 2 foreign keys per table), 400 tables: wall clock is unchanged vs inline, with the longest main-thread block dropping to ~20ms. Metadata crosses the thread boundary as a structured clone, which is plain JSON and costs nothing measurable.format-pool.ts/format-worker.jskeep their master names, identifiers (FormatQueueFullError,destroyFormatPool,isFormatPoolActive) and structure, so the only diff in them is the delegation itself: the worker task carries generator metadata into the package'sgenerateTypescriptinstead of a prettier payload, and the entry point isgenerateTypescriptTypes(metadata, options)instead offormat(code, options). Admission control, per-task timeout, idle pool, the 503 shedding path and thePG_META_FORMAT_*env vars are all unchanged from master.The worker task type excludes the package's
formatcallback option (Omit): functions cannot cross the structured-clone worker boundary, so the compiler guarantees one is never sent.Upstream changes (supabase/sdk#118, released as 0.2.0)
The remaining review findings are addressed in the package rather than here:
oxfmtand adds aformathook for callers that want to substitute their own.introspect()awaited its ten queries one at a time where the old CLI path usedPromise.all; feat(postgrest-typegen): formatter hook, oxfmt default, concurrent introspection sdk#118 runs them concurrently again.test/server/templates/go.test.ts: the unit cases (enum-array fallback and friends) are already mirrored upstream in the package'stest/generation/go.test.ts, alongside broader generation coverage.src/introspection/sql/are now the canonical typegen introspection queries, exercised by the package's own integration tests and a nightly parity job against real postgres-meta; the copies here undersrc/lib/sql/continue to serve the REST API.prettierstays forParser.ts(SQL formatting) and dev formatting.With oxfmt, inline
generateTypescriptbarely blocks the event loop at all (measured max block 11ms at 400 tables, 29ms at 1000, 51ms at 2000 — oxfmt's napi formatting runs off the JS thread), so the worker is belt-and-braces now; removing the machinery entirely is a candidate follow-up after this merges.Validation
npm run check(tsc) passes andprettier --checkpasses against the released package.format-pooltests pass against the released 0.2.0, covering worker/inline parity, the CLI opt-out, 503 load shedding and in-flight accounting.masterwhen run against the same database; ordering differs as described above.master(fixture-loading issues in the local Docker environment); the failure set on this branch matches themasterbaseline, and CI is the arbiter for the full suite.