diff --git a/.changeset/driver-sql-canonical-iso-read-door.md b/.changeset/driver-sql-canonical-iso-read-door.md new file mode 100644 index 0000000000..18f0c04e68 --- /dev/null +++ b/.changeset/driver-sql-canonical-iso-read-door.md @@ -0,0 +1,13 @@ +--- +"@objectstack/driver-sql": minor +--- + +The record read doors present the builtin audit stamps (`created_at`, `updated_at`) and every declared `Field.datetime` column as the canonical instant text `YYYY-MM-DDTHH:MM:SS.sssZ` on EVERY dialect — Postgres and MySQL now included, exactly as SQLite always has (ADR-0053 addendum D-F1..D-F3, #13973). + +**Consumer-visible change, Postgres and MySQL only.** An in-process consumer reading such a column off a `find()` / `findOne()` row, off the row `create()` / `update()` / `upsert()` / `bulkCreate()` / `bulkUpdate()` return, or out of `aggregate()` (`min` / `max`, a raw temporal group key) or `distinct()`, receives a `string` where it received a JS `Date`. The wire is unchanged: `JSON.stringify` already serialised that `Date` as the same ISO text, so REST, MCP and SDK callers see nothing move. A consumer that called a `Date` method directly on the field (`.getTime()`, `.toISOString()`, `.getFullYear()`) now fails loudly with a `TypeError` instead of silently working on one dialect; the sweep behind this change found none in the repository's non-test sources. A consumer that compared, sorted, keyed or formatted the value as text — the shape eight production-driver defects had (#13382, #13993–#13999) — is now correct by construction on every dialect. + +- **Where the fold happens.** At the driver's own read boundary (`formatOutput` for rows, `presentReadValue` for the aggregate/distinct doors). The `pg` and `mysql2` client parsers are untouched: a `Date` is still what the client materialises, and a raw knex read still hands it back. Only the driver's read doors changed. +- **The builtin audit columns gain an `aggregate()` / `distinct()` arm on every dialect.** `max(updated_at)` and `distinct('created_at')` had no read presentation at all before — on SQLite they even missed ADR-0074's legacy-row repair — and now present exactly what `find()` presents. +- **An Invalid `Date` is the one shape the fold hands through unchanged** (#14078: a MySQL zero `DATETIME`; a Postgres year past 275760). It has no canonical text; the fold never throws on it, and the consumer-side guards #14078 landed absorb it as before. + +The per-site canonicalisations landed for #13993–#13999 and #14078 stay correct and become no-ops on driver rows; nothing is removed here. diff --git a/docs/adr/0053-date-and-datetime-semantics.md b/docs/adr/0053-date-and-datetime-semantics.md index b8f200dfcf..bbcec4d337 100644 --- a/docs/adr/0053-date-and-datetime-semantics.md +++ b/docs/adr/0053-date-and-datetime-semantics.md @@ -1,6 +1,6 @@ # ADR-0053: `date` is a timezone-naive calendar day; `datetime` is an instant rendered in a reference timezone -**Status**: Accepted (2026-06-16) — Phase 1 + addendum D-A1 implemented (`sql-driver.ts` `toDateOnly` write/read/filter normalization; analytics `coerceTemporalFilterValue`), Phase 2 landing incrementally; D-A2 resolved 2026-07-30: `temporalFilterValue` + `temporalFilterColumnSql` are optional `IDataDriver` contract members with identity semantics, and analytics types its driver seam from the contract. **Partly superseded (2026-07-29, addendum D-B1..D-B4):** Phase 1's "`Field.datetime` stays stored as UTC epoch ms" is replaced by one canonical UTC instant per dialect — `YYYY-MM-DDTHH:MM:SS.sssZ` text on SQLite, `timestamptz` on Postgres, `DATETIME(3)` on MySQL — applied on write and to filter comparands alike (#3912, #3942). **Extended (2026-07-30, addendum D-C1..D-C3):** `Field.time` takes the same construction — canonical `HH:MM:SS[.fff]` wall-clock text, one function on write/filter/read, `TIME(3)` on MySQL, UTC `NOW()` defaults on every dialect (#3994). +**Status**: Accepted (2026-06-16) — Phase 1 + addendum D-A1 implemented (`sql-driver.ts` `toDateOnly` write/read/filter normalization; analytics `coerceTemporalFilterValue`), Phase 2 landing incrementally; D-A2 resolved 2026-07-30: `temporalFilterValue` + `temporalFilterColumnSql` are optional `IDataDriver` contract members with identity semantics, and analytics types its driver seam from the contract. **Partly superseded (2026-07-29, addendum D-B1..D-B4):** Phase 1's "`Field.datetime` stays stored as UTC epoch ms" is replaced by one canonical UTC instant per dialect — `YYYY-MM-DDTHH:MM:SS.sssZ` text on SQLite, `timestamptz` on Postgres, `DATETIME(3)` on MySQL — applied on write and to filter comparands alike (#3912, #3942). **Extended (2026-07-30, addendum D-C1..D-C3):** `Field.time` takes the same construction — canonical `HH:MM:SS[.fff]` wall-clock text, one function on write/filter/read, `TIME(3)` on MySQL, UTC `NOW()` defaults on every dialect (#3994). **Extended (2026-09-07, addendum D-F1..D-F3):** the READ side takes the same canon — every `@objectstack/driver-sql` record read door but `findWithWindowFunctions` (#16609) presents `Field.datetime` values and the builtin `created_at` / `updated_at` audit stamps as the canonical `YYYY-MM-DDTHH:MM:SS.sssZ` text on every dialect, folded at the driver's read boundary with the client parsers untouched; those doors never hand out a JS `Date` for those columns, save an Invalid `Date`, which has no canonical text and passes through unchanged (#13973, maintainer ruling B1 narrow, 2026-09-02). **Deciders**: ObjectStack Protocol Architects **Builds on**: [ADR-0032](./0032-unified-expression-layer.md) (unified expression layer — CEL dialect, `today()`/`daysFromNow()`), [ADR-0014](./0014-record-form-field-type.md) (field types) **Consumers**: `@objectstack/spec` (`Field.date`/`Field.datetime`), `@objectstack/driver-sql` (`coerceFilterValue`, `formatInput`/`formatOutput`, `dateFields`/`datetimeFields`), `@objectstack/formula` (`stdlib` time functions, `cel-engine` hydration), `@objectstack/objectql` (`applyFormulaPlan`), schedule/cron executors, report/analytics date bucketing, `sys-user-preference.timezone`. @@ -1020,3 +1020,153 @@ again — the #5499 freeze that used to be the other half of this condition was dissolved on 2026-08-11. The conversion itself is pinned by `mongodb-time-storage.test.ts`, which is unaffected: it is pure, needs no server, and still runs everywhere. + +--- + +## Addendum (2026-09-07) — the read door presents ONE instant shape on every dialect (D-F1..D-F3, #13973) + +> **Status:** landed. Extends D-B1's canon from the storage and comparand sides +> to the READ side, and ADR-0074's audit-column read repair from SQLite to every +> dialect. Provenance: maintainer ruling on #13973 — 「同意」 to the director +> seat's analysis recommending B1 (narrow), 2026-09-02, recorded on the card as +> comment 5507803003. The options that ruling declined are listed at the end. + +### What the read side actually was + +D-B1 gave `Field.datetime` one storage form per dialect and one comparand rule, +and said nothing about what the driver HANDS BACK. That was decided by the +client library. `formatOutput` folded every SQLite storage shape to +`YYYY-MM-DDTHH:MM:SS.sssZ` — inside `if (this.isSqlite)`. On Postgres and MySQL +a `timestamptz` / `DATETIME(3)` left the read door as node-pg's / mysql2's JS +`Date`; on SQLite (and the memory driver) as canonical text. The builtin audit +columns had the same gate around ADR-0074's repair, and the `aggregate()` / +`distinct()` presentation (`readPresentationKind`) had no arm for them on ANY +dialect. One column, two runtime types, through the same `any`-shaped record — +invisible to the type system, and to every test that never ran on a live +dialect. + +The #13973 census measured what that cost. 44 packages call a read door; 43 of +them had only ever seen the text side (the one CI job with a live Postgres and +MySQL runs `driver-sql` alone). Eight consumers were wrong on the production +default driver — #13382 in production (the OCC seam compared `String(v)` on +both sides and refused every guarded save), #13993–#13999 by reading (an +idempotency window that never expires; a migration that persisted +`Date.toString()`; a timeline sorted by weekday name; a `z.string()` field +holding a `Date`) — every one in the direction "expected the text, received a +`Date`", none the reverse. The driver comment stating the `Date` side as +deliberate ("`Field.datetime` depends on it") was checked on the tree and did +not hold: nothing on the read path consumed the `Date`. + +### D-F1 — Every read door presents the canonical instant text, on every dialect + +For the builtin audit columns (`created_at`, `updated_at`) and every declared +`Field.datetime` column, every record read door of `@objectstack/driver-sql` +listed here presents the value as `YYYY-MM-DDTHH:MM:SS.sssZ` text — on SQLite, +Postgres and MySQL alike, exactly as SQLite always did: + +- `find()`, `findOne()`, and the rows `create()`, `update()`, `upsert()`, + `bulkCreate()` and `bulkUpdate()` return (all through `formatOutput`, whose two + gates are now unconditional and whose audit-column arm folds a `Date`); +- `aggregate()` for `min` / `max` over such a column and for a raw temporal + group key, and `distinct()` over such a column (`presentReadValue`, whose + `datetime` arm is now unconditional, and `readPresentationKind`, which now + routes the two audit columns to the same `presentAuditTimestampOutput` + `formatOutput` applies — one presenter per column class, shared by every + door, so a value `find()` passes through as a number — ADR-0074 §3's epoch + INTEGER, or an author-declared non-temporal `created_at` — is that same + number here, never ISO text at this door alone). + +None of these doors hands out a JS `Date` for these columns, save the one +shape D-F3 names: an Invalid `Date`, which has no canonical text and passes +through unchanged. `findWithWindowFunctions` is not one of these doors (see +Consequences; #16609). Declared = enforced: +`sql-driver-13973-canonical-iso-read-door.test.ts` asserts it per cell of the +D-A3 driver axis for every door listed — `bulkCreate()` over the rows a +dialect's bulk insert returns (MySQL, with no `RETURNING`, returns none, and +that cell reads the batch back through `find()` instead) — the SQLite cell +everywhere, the Postgres and MySQL cells under `Temporal Conformance (live PG ++ MySQL)`, with the three-way zone skew guard so a `Z` that only survived +because every clock agreed cannot pass — and +`sql-driver-13567-audit-stamp-materialisation.test.ts` re-pins the audit +column at the OCC seam's door. The same file's §D pins, on SQLite, that +`aggregate()` / `distinct()` present the audit columns through the presenter +`find()` uses, on the two shapes where a different one would show (an +author-declared non-temporal `created_at`; a raw-written epoch INTEGER) — +SQLite because its type affinity is what lets a number sit in that column at +all; the `timestamptz` / `DATETIME(3)` the DDL types it as elsewhere cannot. + +Why text rather than "everything a `Date`" (the maintainer asked exactly this: +「为什么不能都用日期类型」): every platform with a metadata layer decides the +in-process type from the declared field type at its own read boundary, and +ObjectStack had already declared that type — D-B1's text — everywhere but this +one door. A `Date` is not a value type (`===` compares identity, a `Map` key is +by reference, it is mutable); `Invalid Date` is a `Date` whose `toISOString()` +throws; its local-zone methods answer differently on every host; JSON cannot +carry it; and it holds less precision than `timestamptz`. The canonical text +makes `String(v)`, `===`, a template, a sort and a `Map` key correct by +construction, which is what an author — human or AI — reaches for first. A +`Date` belongs at exactly two places: the client parser (D-F2) and the moment +arithmetic happens (`new Date(text).getTime()`, which every class (a) site in +the census already spells and which accepts both shapes). + +### D-F2 — Folded at the driver's read boundary, not at the client parser + +The pg and mysql2 type parsers are not touched. A `Date` stays the client-level +materialisation — a raw knex read of the same row still hands one back, and a +host's own `pg` clients keep the stock behaviour — and the driver canonicalises +in `formatOutput` / `presentReadValue`. This is the narrow form of B1: the +`date` parser installed by `withPostgresCalendarDayAsText` stays the one place +a clock is chosen, the instant types keep their stock parser, and nothing +outside the driver's own read doors moves. §C of +`sql-driver-13973-canonical-iso-read-door.test.ts` measures this on every live +cell: the raw read is a `Date`, the read door is text, and the two name the +same instant. + +### D-F3 — The one shape the fold cannot canonicalise passes through + +An Invalid `Date` — a `Date` whose time value is `NaN`, which #14078 measured +both live dialects to produce from rows already on disk (a MySQL zero +`DATETIME`; any Postgres year in 275760..294276) — has no canonical text. The +fold is total in the sense #14078 ruled for the shared consumer spelling: it +never throws, and it hands the client's Invalid `Date` through unchanged. It is +neither nulled (a stored value silently erased) nor spelled as the text +`Invalid Date` (a wire change: `JSON.stringify` already serialises the shape as +`null`); it leaves as the one `Date` the consumer-side guards #14078 landed +already absorb. `sql-driver-14078-invalid-date-materialisation.test.ts` pins +both halves: the control instant leaves as text, the Invalid `Date` leaves as +itself. + +### Consequences + +- **Consumer-visible, Postgres and MySQL only** (changeset `minor` for + `@objectstack/driver-sql`): in-process consumers receive a `string` where + they received a `Date`. The wire is unchanged — `JSON.stringify` already + serialised the `Date` as the same ISO text. A consumer that called a `Date` + method directly on such a field fails loudly with a `TypeError`; the census + for this addendum (its expressions are recorded on #13973's landing PR) + found none in non-test sources. A consumer that compared, sorted, keyed or + formatted the value as text is now correct on every dialect. +- The per-site canonicalisations landed for #13993–#13999 and the five total + `Date` arms #14078 landed become no-ops on driver rows. They stay correct and + are not removed here; retiring them is separate, deliberate work. +- ADR-0074's read repair is no longer SQLite-only in effect: its string arm is + unchanged and now runs on every dialect (a no-op on canonical text), and a + `Date` arm sits beside it. +- D-A3's matrix gains the read-shape cell. `Temporal Conformance (live PG + + MySQL)` is the job that proves it; its package set is not widened. +- Not covered: `findWithWindowFunctions`, which applies no read presentation of + any kind today (booleans, dates and JSON included) — a pre-existing gap of its + own, recorded rather than folded in. + +### Options not taken + +- **B1-full** — text at the pg parser. Same effect, one more decision reversed + (the parser is scoped per pool and would then differ from a host's own `pg` + clients), marginally better precision. Not taken: the driver's read boundary + is the layer that owns the declared type. +- **B2** — a union return type the consumer must narrow. Moot once the read + type is `string`; it only made sense if both shapes were kept. +- **B3** — accept the divergence and add a shared `Date` fixture. It accepts a + divergence this ADR had already declared away and leaves 43 packages + untested against the shape. Its fixture folds into the conformance cells + above. diff --git a/docs/adr/0074-canonical-audit-timestamp-storage-on-sqlite.md b/docs/adr/0074-canonical-audit-timestamp-storage-on-sqlite.md index b32ab26828..5607b380dd 100644 --- a/docs/adr/0074-canonical-audit-timestamp-storage-on-sqlite.md +++ b/docs/adr/0074-canonical-audit-timestamp-storage-on-sqlite.md @@ -1,6 +1,6 @@ # ADR-0074: Audit timestamps are stored in one canonical, timezone-explicit format on SQLite -**Status**: Accepted (2026-06-26) +**Status**: Accepted (2026-06-26). **Read side extended (2026-09-07):** the read repair this ADR added on SQLite is, since [ADR-0053 addendum D-F1](./0053-date-and-datetime-semantics.md) (#13973), one presentation on every dialect — `created_at` / `updated_at` leave every read door as the canonical ISO-Z text on Postgres and MySQL too, where they used to leave as the client library's `Date`. The storage decision below is unchanged. **Deciders**: ObjectStack Protocol Architects **Builds on**: [ADR-0053](./0053-date-and-datetime-semantics.md) (`datetime` is an instant stored as UTC) **Consumers**: `@objectstack/driver-sql` (`create`/`bulkCreate`/`upsert`/`update`, `formatOutput`), `@objectstack/objectql` (optimistic locking via `updated_at`, `sys_metadata` writes), report/analytics date bucketing, and any out-of-tree consumer of `created_at`/`updated_at` (notably the objectos kernel freshness probe). diff --git a/packages/drivers/driver-sql/src/sql-driver-11389-date-tz-skew.test.ts b/packages/drivers/driver-sql/src/sql-driver-11389-date-tz-skew.test.ts index f32ddf98a6..8673908336 100644 --- a/packages/drivers/driver-sql/src/sql-driver-11389-date-tz-skew.test.ts +++ b/packages/drivers/driver-sql/src/sql-driver-11389-date-tz-skew.test.ts @@ -462,8 +462,11 @@ declareDialectCell(PG_CELL, 'date wire form (#11389)', (cell) => { // SQL NULL survives as null; the year-boundary element is where a // pre-fix skew changed the year. expect(row.ds).toEqual([DAY, null, NEW_YEAR]); - // Untouched on purpose: an instant is exactly what a Date is for, and - // `Field.datetime` depends on it. + // Untouched on purpose (ADR-0053 D-F2): the CLIENT parser keeps + // materialising an instant as a `Date`, and the driver folds it to the + // canonical text at its own read doors, not here. This is a raw + // `execute`, past every read-side presentation, so the client's shape + // is what comes back — the #13973 ruling forbids changing that parser. expect(row.ts instanceof Date).toBe(true); expect((row.ts as Date).toISOString()).toBe(`${DAY}T00:00:00.000Z`); }); diff --git a/packages/drivers/driver-sql/src/sql-driver-13567-audit-stamp-materialisation.test.ts b/packages/drivers/driver-sql/src/sql-driver-13567-audit-stamp-materialisation.test.ts index 6640b2b1a6..897dc15c7e 100644 --- a/packages/drivers/driver-sql/src/sql-driver-13567-audit-stamp-materialisation.test.ts +++ b/packages/drivers/driver-sql/src/sql-driver-13567-audit-stamp-materialisation.test.ts @@ -31,40 +31,54 @@ * one — the layering runs the other way — so the seam's own regression suite * necessarily drives a hand-made `Date`. That is correct and catches a revert * of the repair. What it cannot assert is the half that made the bug real: - * **this driver, on these dialects, hands a `Date` out of its record read - * door.** That half is asserted here, where a live Postgres and a live MySQL - * actually exist (`Temporal Conformance (live PG + MySQL)`). + * what **this driver, on these dialects, hands out of its record read door.** + * That half is asserted here, where a live Postgres and a live MySQL actually + * exist (`Temporal Conformance (live PG + MySQL)`). * - * ## Not over-pinning: the driver states this decision deliberately + * ## What this file pinned, and what it pins now (#13973) * - * `withPostgresCalendarDayAsText` installs a text parser for `date` and - * `date[]` and says, in as many words, why the instant types are left alone: - * *"`timestamptz` / `timestamp` are deliberately untouched: those are instants, - * a `Date` is the right materialisation for them, and `Field.datetime` depends - * on it."* This file pins that stated decision at the door a consumer reads it - * through, and the SQLite side of the same asymmetry alongside it. + * As first landed, §B pinned the asymmetry as a coverage FACT: the two live + * cells handed `updated_at` out as a `Date`, SQLite as canonical text, and the + * driver's `withPostgresCalendarDayAsText` comment stated the `Date` side as + * deliberate. The #13973 census then measured the cost of that fact — 43 of + * the 44 packages that call a read door had only ever seen the text side, and + * eight consumers were wrong under the `Date` side (#13382 in production, + * #13993–#13999 by reading) — and the maintainer ruled B1 (narrow), 2026-09-02: + * the read door presents the canonical ISO-Z text on EVERY dialect (ADR-0053 + * D-F1), folded at the driver's own read boundary with the client parsers + * untouched (D-F2). §B now pins that contract; the SQLite cell's assertions + * simply apply to all three. The composed fact this file was written about is + * therefore inverted on purpose: `String(v)` of the driver's value IS the + * client's echoed token, on every dialect, by construction and no longer by + * accident. `sql-driver-13973-canonical-iso-read-door.test.ts` is the full + * conformance cell (both column classes, every read door); this file keeps + * the OCC-seam framing and the one column that seam reads. * * ## Which half rests on which evidence * * §A is pure JavaScript — `Date.prototype.toString` renders whole seconds in * the PROCESS's zone while `toISOString` renders milliseconds in UTC. It needs * no server, so it runs on every runner including Test Core, and it is the half - * that can be measured anywhere. + * that can be measured anywhere. It is kept as the record of WHY a `Date` + * leaving the door was a defect: the two spellings §A1 puts side by side are + * the two sides of the comparison #13382 made. * * §B is the DIALECT fact and only a live server can answer it: what - * `SqlDriver#findOne` puts in `updated_at`. The SQLite cell runs everywhere and - * pins the ISO-text side — the accident that kept every OCC pin green; the two - * live cells pin the `Date` side and are the reason this file lives in - * `driver-sql` rather than next to the seam. + * `SqlDriver#findOne` puts in `updated_at`. The SQLite cell runs everywhere; + * the two live cells are the reason this file lives in `driver-sql` rather + * than next to the seam, and §B3 is what keeps them a measurement rather than + * a restatement of the client library's behaviour — a raw knex read of the same + * row still materialises the dialect's `Date`, so the fold is provably the + * driver's. * * ## Why the audit column and not a declared `Field.datetime` * * `created_at` / `updated_at` are BUILTIN columns, so they are not in - * `datetimeFields` and no declared-field coercion reaches them. `formatOutput` - * repairs them only inside its `if (this.isSqlite)` arm - * (`repairNaiveUtcAuditTimestamp` over `AUDIT_TIMESTAMP_COLUMNS`), which is - * precisely why the two live dialects hand the raw client value through — and - * why `updated_at`, not some other datetime, is the value the OCC seam reads. + * `datetimeFields` and no declared-field coercion reaches them. Their read + * presentation is `presentAuditTimestampOutput` over `AUDIT_TIMESTAMP_COLUMNS` + * in `formatOutput` — SQLite-gated before #13973, which is precisely why the + * two live dialects handed the raw client value through — and `updated_at`, + * not some other datetime, is the value the OCC seam reads. */ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; @@ -79,17 +93,18 @@ const TABLE = 'os13567_stamp'; /** * How many create + read-back rounds each cell measures. * - * Sized for §B4's non-vacuity guard rather than for coverage: `String(Date)` - * drops milliseconds, which is only OBSERVABLE on a stamp that carried some. - * Postgres' `CURRENT_TIMESTAMP` is microsecond-precision and MySQL's `now(3)` - * is millisecond-precision, so a stamp landing on an exact `.000` is ~1 in - * 1000 — rare, but not impossible, and a run in which every stamp did would - * report a green that no truncation could have perturbed. Six independent - * rounds put that at ~1e-18, the same sizing `#11224` uses one file over. + * Sized for §B4's non-vacuity guard rather than for coverage: the fold keeps + * the milliseconds `String(Date)` would have dropped, which is only OBSERVABLE + * on a stamp that carried some. Postgres' `CURRENT_TIMESTAMP` is + * microsecond-precision and MySQL's `now(3)` is millisecond-precision, so a + * stamp landing on an exact `.000` is ~1 in 1000 — rare, but not impossible, + * and a run in which every stamp did would report a green that no truncation + * could have perturbed. Six independent rounds put that at ~1e-18, the same + * sizing `#11224` uses one file over. */ const ROUNDS = 6; -/** Canonical audit-timestamp text — the shape SQLite stores and returns. */ +/** Canonical audit-timestamp text — the shape every read door presents (ADR-0053 D-F1). */ const ISO_Z = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/; /** @@ -259,98 +274,104 @@ function measure(cell: DialectCell): void { } }); - if (cell.id === 'sqlite') { - it('§B1 hands `updated_at` back as canonical ISO-8601-Z TEXT', () => { - for (const row of rows) { - expect(typeof row.updated_at, `updated_at type for ${row.id}`).toBe('string'); - expect(row.updated_at).toMatch(ISO_Z); - } - }); + it('§B1 hands `updated_at` back as canonical ISO-8601-Z TEXT — on every dialect (ADR-0053 D-F1)', () => { + // The ruled shape (#13973 B1). Before it, this cell asserted the exact + // opposite for the two live dialects: `timestamptz` (Postgres, via + // node-pg's stock OID 1184 parser) and `DATETIME(3)` (MySQL, via mysql2's + // `parseDateTime` under the `timezone: 'Z'` pin) both materialise as a + // `Date`, and `formatOutput`'s audit-column presentation was SQLite-gated, + // so nothing downstream converted it. The fold is now unconditional and + // carries a `Date` arm, so the client's `Date` never leaves the door. + for (const row of rows) { + expect( + row.updated_at instanceof Date, + `${cell.label} returned updated_at as a JS Date ` + + `(${JSON.stringify(String(row.updated_at))}) — the read door is ruled to present the ` + + `canonical text on every dialect (ADR-0053 D-F1)`, + ).toBe(false); + expect(typeof row.updated_at, `updated_at type for ${row.id}`).toBe('string'); + expect(row.updated_at).toMatch(ISO_Z); + } + }); - it('§B2 `String()` of it IS the token a client echoes — the accident that hid the defect', () => { - // This is the control, and the reason this file exists: on the dialect - // every OCC pin was written against, the naive `String(v)` comparison - // matches BY ACCIDENT, because the stored value already is the - // canonical spelling the client was served. - const value = rows[0].updated_at as string; + it('§B2 `String()` of it IS the token a client echoes — by construction now, not by accident', () => { + // This used to be the SQLite-only control, and the reason this file + // exists: on the dialect every OCC pin was written against, the naive + // `String(v)` comparison matched BY ACCIDENT, because the stored value + // already was the canonical spelling the client was served. Under D-F1 it + // holds on every dialect, which is what makes the seam's text compare + // correct rather than lucky. + for (const row of rows) { + const value = row.updated_at as string; expect(String(value)).toBe(value); expect(new Date(value).toISOString()).toBe(value); - }); - } else { - it('§B1 hands `updated_at` back as a JS `Date`', () => { - // The composed fact. `timestamptz` (Postgres, via node-pg's stock OID - // 1184 parser — `withPostgresCalendarDayAsText` overrides only `date` - // and `date[]`) and `DATETIME(3)` (MySQL, via mysql2's `parseDateTime` - // under the `timezone: 'Z'` pin `withUtcSession` installs) both - // materialise as a `Date`, and `formatOutput`'s audit-column repair is - // SQLite-gated, so nothing downstream converts it. - for (const row of rows) { - expect( - row.updated_at instanceof Date, - `${cell.label} returned updated_at as ${typeof row.updated_at} ` + - `(${JSON.stringify(String(row.updated_at))}) — the OCC seam's Date handling is ` + - `pinned against a shape this dialect no longer produces`, - ).toBe(true); - } - // A real instant, not an Invalid Date dressed up as one. - for (const row of rows) expect(Number.isFinite((row.updated_at as Date).getTime())).toBe(true); - }); - - it('§B3 `String()` of it is NOT the token the client echoes, and follows the process zone', async () => { - const value = rows[0].updated_at as Date; - // What the GET served and what the client hands back as its next - // `If-Match`: a `Date` leaves this process as its ISO-8601 form. - const echoed = value.toISOString(); - expect(echoed).toMatch(ISO_Z); + } + }); + it('§B3 the presented text names the instant the CLIENT materialised — the fold changed the type and nothing else', async () => { + // Past every read-side presentation: what the client library hands the + // driver for the same row. On a live cell that is still a `Date` — the + // client parsers are untouched (D-F2) — so the fold is provably the + // driver's, and this cell is a measurement of the driver rather than of + // node-pg / mysql2. On SQLite the stored TEXT already is the presented + // text, the side of the old asymmetry every consumer was tested against. + const raw: any = await (driver as any).knex(TABLE).where('id', rows[0].id).first(); + expect(raw, 'raw read returned nothing').toBeTruthy(); + const presented = rows[0].updated_at as string; + if (cell.live) { + expect( + raw.updated_at instanceof Date, + `${cell.label} raw updated_at is ${typeof raw.updated_at} — the client parser was changed, ` + + `which the #13973 ruling forbids`, + ).toBe(true); + const value = raw.updated_at as Date; + expect(Number.isFinite(value.getTime())).toBe(true); + expect(value.toISOString()).toBe(presented); + + // And the two spellings #13382 compared are still two spellings: the + // client's `Date`, rendered through `String()`, follows the process + // zone and is NOT the token — which is exactly why it must never + // leave the door. const shanghai = await underProcessZone('Asia/Shanghai', () => String(value)); const newYork = await underProcessZone('America/New_York', () => String(value)); - for (const [tz, spelled] of [['Asia/Shanghai', shanghai], ['America/New_York', newYork]] as const) { expect(spelled, `${tz}: shape`).toMatch(WHOLE_SECONDS_AND_ZONE); - expect( - spelled === echoed, - `${tz}: a strict String() compare of the driver's value against the client's echoed ` + - `token is the comparison #13382 made — ${JSON.stringify(spelled)} vs ` + - `${JSON.stringify(echoed)}`, - ).toBe(false); + expect(spelled === presented, `${tz}: ${JSON.stringify(spelled)} vs ${JSON.stringify(presented)}`).toBe(false); } - - // One `Date`, two process zones, two spellings: the process zone is - // baked into the rendering. Asserted by DIFFERENCE rather than against - // a literal offset, so no tzdata value is pinned here. - expect( - shanghai, - `the same instant spelled identically under two different process zones — ` + - `${JSON.stringify(shanghai)}`, - ).not.toBe(newYork); + expect(shanghai).not.toBe(newYork); expect(offsetInSpelling(shanghai)).not.toBe(offsetInSpelling(newYork)); - }); - - it('§B4 `String()` drops the milliseconds the echoed token carries', async () => { - const withMillis = rows.filter((row) => (row.updated_at as Date).getMilliseconds() !== 0); - expect( - withMillis.length, - `none of the ${ROUNDS} stamps in this run carried sub-second digits, so nothing here ` + - `could have observed the millisecond loss — the cell measured nothing`, - ).toBeGreaterThan(0); + } else { + expect(typeof raw.updated_at).toBe('string'); + expect(raw.updated_at).toBe(presented); + } + }); - for (const row of withMillis) { - const value = row.updated_at as Date; - const echoed = value.toISOString(); - // The echoed token names the instant exactly … - expect(echoed, `${row.id}: echoed token`).toMatch(/\.\d{3}Z$/); - expect(Date.parse(echoed), `${row.id}: echoed token instant`).toBe(value.getTime()); - // … while its `String()` names one `getMilliseconds()` earlier. Read - // in the ambient zone the suite is running under, which is the zone - // the seam's `String(v)` would have used. - expect(Date.parse(String(value)), `${row.id}: spelled instant`).toBe( - value.getTime() - value.getMilliseconds(), - ); - expect(Date.parse(String(value))).not.toBe(Date.parse(echoed)); - } - }); - } + it('§B4 the presented text keeps the milliseconds `String(Date)` would have dropped', () => { + const withMillis = rows.filter((row) => !/\.000Z$/.test(row.updated_at as string)); + expect( + withMillis.length, + `none of the ${ROUNDS} stamps in this run carried sub-second digits, so nothing here ` + + `could have observed a millisecond loss — the cell measured nothing`, + ).toBeGreaterThan(0); + + for (const row of withMillis) { + const presented = row.updated_at as string; + // The presented token names the instant exactly, sub-second digits + // included … + expect(presented, `${row.id}: presented token`).toMatch(/\.\d{3}Z$/); + const instant = Date.parse(presented); + expect(Number.isFinite(instant)).toBe(true); + // … while the spelling the OCC seam used to compare against names one + // `getMilliseconds()` earlier. Read in the ambient zone the suite is + // running under, which is the zone the seam's `String(v)` would have + // used had a `Date` reached it. + const spelledAsDateWouldHaveBeen = String(new Date(instant)); + expect(Date.parse(spelledAsDateWouldHaveBeen), `${row.id}: spelled instant`).toBe( + instant - new Date(instant).getMilliseconds(), + ); + expect(Date.parse(spelledAsDateWouldHaveBeen)).not.toBe(instant); + } + }); }); } diff --git a/packages/drivers/driver-sql/src/sql-driver-13973-canonical-iso-read-door.test.ts b/packages/drivers/driver-sql/src/sql-driver-13973-canonical-iso-read-door.test.ts new file mode 100644 index 0000000000..af9d31415a --- /dev/null +++ b/packages/drivers/driver-sql/src/sql-driver-13973-canonical-iso-read-door.test.ts @@ -0,0 +1,563 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#13973] The read door presents ONE instant shape on EVERY dialect — + * ADR-0053 D-F1..D-F3, declared = enforced. + * + * ## The contract + * + * For the builtin audit columns (`created_at`, `updated_at`) and every declared + * `Field.datetime` column, `@objectstack/driver-sql`'s record read doors — + * `find()`, `findOne()`, and the rows `create()`, `update()`, `upsert()`, + * `bulkCreate()` and `bulkUpdate()` return — hand out the canonical instant + * text `YYYY-MM-DDTHH:MM:SS.sssZ`; `aggregate()` (`min` / `max`, and a raw + * temporal group key) and `distinct()` present the same two column classes + * the same way. None of those doors hands out a JS `Date` for those columns, + * on SQLite, Postgres or MySQL — with the one exception ADR-0053 D-F3 names: + * an Invalid `Date`, which has no canonical text and passes through as the + * `Date` it is (pinned by `sql-driver-14078-invalid-date-materialisation.test.ts`; + * never met here, because this fixture writes only valid instants). + * `findWithWindowFunctions` is not one of those doors: it applies no read + * presentation of any kind, D-F1 records it as not covered, and #16609 holds + * it. + * + * §A1–§A3 measure the four row doors on the fixture table; §A5–§A7 the three + * write doors whose return is a row, on a second table so their writes cannot + * move what §B/§C compare against. §D, on the SQLite cell alone, pins that + * `aggregate()` / `distinct()` present the audit columns through the SAME + * presenter `find()` uses — its own note says why SQLite is the whole + * coverage there and not a shortfall. + * + * ## Why this file exists next to #13567 + * + * `sql-driver-13567-audit-stamp-materialisation.test.ts` pinned the OLD + * asymmetry as a coverage fact: the live dialects handed `updated_at` out as a + * `Date` and SQLite as text, and 43 of the 44 packages that call a read door + * had only ever seen the text side — the state in which eight consumers were + * wrong on the production default driver (#13382 in production, #13993–#13999 + * by reading). The maintainer ruled (#13973, B1 narrow, 2026-09-02) that the + * SQLite presentation IS the contract, on every dialect. This file is that + * contract's conformance cell on the ADR-0053 D-A3 driver axis; the #13567 + * file keeps its OCC-seam framing and now pins the same shape. + * + * ## Non-vacuity, per cell + * + * A live cell that answered text because the CLIENT already produced text + * would prove nothing about the driver. §C reads the same row back through raw + * knex — past every read-side presentation — and asserts the client still + * materialises a `Date` there. That is the ruling's second clause, measured: + * the pg / mysql2 parsers are untouched (D-F2), and the fold happened at the + * driver's own read boundary. It is also this file's firing control: with the + * two `formatOutput` gates back inside `if (this.isSqlite)` and the + * `presentReadValue` arm back to `this.isSqlite ? … : value`, §A/§B go red on + * both live cells while §C stays green — the PR that landed this records that + * run against a real Postgres 16 and MySQL 8.0. + * + * The three-way zone skew (server, process, UTC pairwise different) is + * asserted on the live cells exactly as every other matrix consumer does, so a + * `Z` that survived only because every clock agreed cannot pass here. Every + * declared instant carries sub-second digits, so a fold that dropped + * milliseconds — what `String(Date)` does — would be visible too. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import type { DriverQuery } from '@objectstack/spec/contracts'; +import { SqlDriver } from './index.js'; +import { + DIALECT_CELLS, + assertThreeWayZoneSkew, + declareDialectCell, + readServerZone, + type DialectCell, +} from './live-dialect-matrix.testkit.js'; + +/** Driver options every write here uses — this fixture is not tenant-scoped. */ +const OPTS = { bypassTenantAudit: true } as any; + +const TABLE = 'os13973_read_door'; + +/** + * The table the write-door cells (§A5–§A7) write to. Separate from `TABLE` so + * an upsert, a bulk update and a bulk insert cannot move the `updated_at` + * values and the row count §B1/§B3 compare `aggregate()` / `distinct()` + * against — the cells stay order-independent. + */ +const TABLE_RETURNS = 'os13973_read_door_returns'; + +/** The canonical instant text — the ONE shape every read door presents. */ +const ISO_Z = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/; + +/** + * The declared `Field.datetime` instants, one per row. Each carries non-zero + * milliseconds (a fold that truncated would be visible), and the second is a + * duplicate of the first so `distinct()` has something to collapse. + */ +const CLOSED_AT = [ + '2026-01-10T09:00:00.123Z', + '2026-01-10T09:00:00.123Z', + '2026-02-14T21:30:45.678Z', + '2026-03-01T00:00:00.001Z', +] as const; + +/** `Field.date` control values — the calendar-day rule must not be disturbed. */ +const DUE_ON = ['2026-01-10', '2026-01-10', '2026-02-14', '2026-03-01'] as const; + +const INSTANT_COLUMNS = ['created_at', 'updated_at', 'closed_at'] as const; + +/** Sorted, de-duplicated — what `distinct()` over `closed_at` must answer. */ +const DISTINCT_CLOSED_AT = [...new Set(CLOSED_AT)].sort(); + +/** The one assertion this file is about, spelled once. */ +function expectCanonicalInstant(value: unknown, label: string): asserts value is string { + expect(value, `${label}: the read door did not return the column`).toBeDefined(); + expect(value, `${label}: null`).not.toBeNull(); + expect( + value instanceof Date, + `${label}: the read door handed out a JS Date (${String(value)}) — ADR-0053 D-F1 rules the ` + + `canonical text on every dialect`, + ).toBe(false); + expect(typeof value, `${label}: type`).toBe('string'); + expect(value, `${label}: shape`).toMatch(ISO_Z); +} + +function measure(cell: DialectCell): void { + describe(`#13973 — the read door presents one instant shape (${cell.label})`, () => { + let driver: SqlDriver; + let rows: any[] = []; + const createdReturns: any[] = []; + let updatedReturn: any; + + beforeAll(async () => { + driver = new SqlDriver(cell.config()); + // A live cell proves nothing unless server, process and UTC disagree — + // the same guard every other matrix consumer runs. + if (cell.live) assertThreeWayZoneSkew(cell, await readServerZone(cell, driver)); + await driver.execute(`drop table if exists ${TABLE}`).catch(() => {}); + await driver.execute(`drop table if exists ${TABLE_RETURNS}`).catch(() => {}); + // The DDL path, so the audit columns are the ones + // `createAuditTimestampColumn` produces (`timestamptz` on Postgres, + // `DATETIME(3)` on MySQL, TEXT on SQLite) and `closed_at` is a declared + // `Field.datetime` (`timestamptz` / `DATETIME(3)` / TEXT). + const fields = { + id: { type: 'text' }, + title: { type: 'string' }, + closed_at: { type: 'datetime' }, + due_on: { type: 'date' }, + amount: { type: 'number' }, + }; + await driver.initObjects([ + { name: TABLE, fields }, + { name: TABLE_RETURNS, fields }, + ] as any); + // Two rows for the write doors that operate on an EXISTING row (§A5's + // merge, §A6); the doors that insert (§A5's insert, §A7) bring their own. + for (const i of [0, 1]) { + await driver.create( + TABLE_RETURNS, + { id: `w${i}`, title: `write row ${i}`, closed_at: CLOSED_AT[i], due_on: DUE_ON[i], amount: 100 + i }, + OPTS, + ); + } + for (const [i, iso] of CLOSED_AT.entries()) { + // Alternate the WRITE shape — a JS `Date` and the canonical text — so + // the read shape is shown to be independent of how the row was written. + const closedAt = i % 2 === 0 ? new Date(iso) : iso; + createdReturns.push( + await driver.create( + TABLE, + { id: `r${i}`, title: `row ${i}`, closed_at: closedAt, due_on: DUE_ON[i], amount: i + 1 }, + OPTS, + ), + ); + } + updatedReturn = await driver.update(TABLE, 'r0', { title: 'row 0 (updated)' }, OPTS); + rows = await driver.find(TABLE, { orderBy: [{ field: 'id', order: 'asc' }] }, OPTS); + }, 60_000); + + afterAll(async () => { + await driver?.execute(`drop table if exists ${TABLE}`).catch(() => {}); + await driver?.execute(`drop table if exists ${TABLE_RETURNS}`).catch(() => {}); + await driver?.disconnect(); + }); + + it('§0 the fixture is non-vacuous: every row came back carrying every column under test', () => { + // Every assertion below reads these keys off these rows; a door that did + // not select a column would let them all pass having checked nothing. + expect(rows).toHaveLength(CLOSED_AT.length); + for (const row of rows) { + for (const col of [...INSTANT_COLUMNS, 'due_on'] as const) { + expect(row[col], `${row.id}.${col} missing from the find() row`).toBeDefined(); + expect(row[col], `${row.id}.${col} is null`).not.toBeNull(); + } + } + }); + + it('§A1 find(): the builtin audit columns are canonical ISO-Z text, never a Date', () => { + for (const row of rows) { + expectCanonicalInstant(row.created_at, `${row.id}.created_at`); + expectCanonicalInstant(row.updated_at, `${row.id}.updated_at`); + // A real, recent instant — the `Z` names UTC, not the server's or the + // process's zone (which the skew guard made different from each other + // and from UTC on a live cell). Ten minutes is generous for a stamp + // written seconds ago and far below any zone offset. + expect( + Math.abs(Date.now() - Date.parse(row.created_at)), + `${row.id}.created_at (${row.created_at}) is not the instant it was stamped at`, + ).toBeLessThan(10 * 60_000); + } + }); + + it('§A2 find(): a declared Field.datetime is canonical ISO-Z text naming the written instant, whichever shape wrote it', () => { + for (const [i, row] of rows.entries()) { + expectCanonicalInstant(row.closed_at, `${row.id}.closed_at`); + expect(row.closed_at, `${row.id}.closed_at (written as ${i % 2 === 0 ? 'a Date' : 'text'})`).toBe( + CLOSED_AT[i], + ); + } + }); + + it('§A3 findOne(), and the rows update() and create() return, present the same shape', async () => { + const one = await driver.findOne(TABLE, { where: { id: 'r1' } }, OPTS); + expect(one, 'findOne returned nothing').toBeTruthy(); + for (const col of INSTANT_COLUMNS) expectCanonicalInstant(one[col], `findOne ${col}`); + expect(one.closed_at).toBe(CLOSED_AT[1]); + + // `update()` reads the row back after the write, so its return is a + // whole row on every dialect. + expect(updatedReturn, 'update() returned nothing').toBeTruthy(); + for (const col of INSTANT_COLUMNS) expectCanonicalInstant(updatedReturn[col], `update() return ${col}`); + expect(updatedReturn.closed_at).toBe(CLOSED_AT[0]); + + // `create()` returns `returning('*')` where the dialect has it (MySQL + // has no RETURNING and hands back less than a row); whatever instant + // columns a dialect's return does carry must be the canonical text — + // the door is the same `formatOutput`. Which columns come back is the + // dialect's business and is not pinned here. + for (const [i, ret] of createdReturns.entries()) { + if (!ret || typeof ret !== 'object') continue; + for (const col of INSTANT_COLUMNS) { + if (ret[col] === undefined) continue; + expectCanonicalInstant(ret[col], `create() return ${i} ${col}`); + } + if (ret.closed_at !== undefined) expect(ret.closed_at).toBe(CLOSED_AT[i]); + } + }); + + it('§A4 the Field.date control is untouched — a calendar day stays YYYY-MM-DD', () => { + for (const [i, row] of rows.entries()) { + expect(row.due_on, `${row.id}.due_on`).toBe(DUE_ON[i]); + } + }); + + // §A5–§A7: the three remaining row doors D-F1 lists. Each is covered by + // construction — the same `formatOutput` call — and is measured here anyway, + // because "declared = enforced" is a statement about cells, not about call + // graphs. `expectCanonicalInstant` carries §0's guard (defined, non-null) + // inside it, so a door that returned a row WITHOUT its audit columns fails + // here rather than passing over nothing. + + it('§A5 upsert(): the row it hands back presents the same shape — merged onto an existing row, and inserted', async () => { + // `upsert()` reads the row back after its statement on every dialect + // (`readback.first()` → `formatOutput`), so its return is a whole row and + // the guard applies unqualified. + const before = await driver.findOne(TABLE_RETURNS, { where: { id: 'w0' } }, OPTS); + expect(before, 'the seed row is missing').toBeTruthy(); + const merged = await driver.upsert(TABLE_RETURNS, { id: 'w0', title: 'write row 0 (merged)' }, undefined, OPTS); + const inserted = await driver.upsert( + TABLE_RETURNS, + { id: 'u0', title: 'upserted row', closed_at: CLOSED_AT[2], due_on: DUE_ON[2], amount: 200 }, + undefined, + OPTS, + ); + for (const [label, ret, closedAt] of [ + ['merged w0', merged, CLOSED_AT[0]], + ['inserted u0', inserted, CLOSED_AT[2]], + ] as const) { + expect(ret, `upsert() ${label} returned nothing`).toBeTruthy(); + expect(ret.id, `upsert() ${label} returned a row that is not the one written`).toBe(label.split(' ')[1]); + for (const col of INSTANT_COLUMNS) expectCanonicalInstant(ret[col], `upsert() ${label} ${col}`); + expect(ret.closed_at, `upsert() ${label} closed_at`).toBe(closedAt); + } + // `created_at` is insert-only under a merge (ADR-0074 §2): the merged + // return names the instant the row was created at, in the same spelling. + expect(merged.created_at).toBe(before.created_at); + }); + + it('§A6 bulkUpdate(): every row it hands back presents the same shape', async () => { + // Loops `update()`, which reads each row back after its statement — a + // whole row per entry on every dialect, so the guard applies unqualified. + const ret = await driver.bulkUpdate( + TABLE_RETURNS, + [ + { id: 'w0', data: { title: 'write row 0 (bulk)' } }, + { id: 'w1', data: { title: 'write row 1 (bulk)' } }, + ], + OPTS, + ); + expect(ret, 'bulkUpdate() did not return one row per update').toHaveLength(2); + for (const [i, r] of ret.entries()) { + expect(r.id, `bulkUpdate() return ${i}`).toBe(`w${i}`); + for (const col of INSTANT_COLUMNS) expectCanonicalInstant(r[col], `bulkUpdate() return w${i} ${col}`); + expect(r.closed_at, `bulkUpdate() return w${i} closed_at`).toBe(CLOSED_AT[i]); + // A fresh stamp, in UTC — the same recency bound §A1 puts on `find()`. + expect(Math.abs(Date.now() - Date.parse(r.updated_at)), `w${i}.updated_at is not the instant of the update`).toBeLessThan(10 * 60_000); + } + }); + + it('§A7 bulkCreate(): whatever rows its return carries present the same shape, and the batch lands canonical', async () => { + const batch = [ + { id: 'b0', title: 'batch row 0', closed_at: new Date(CLOSED_AT[0]), due_on: DUE_ON[0], amount: 300 }, + { id: 'b1', title: 'batch row 1', closed_at: CLOSED_AT[3], due_on: DUE_ON[3], amount: 301 }, + ]; + const expectedClosedAt: Record = { b0: CLOSED_AT[0], b1: CLOSED_AT[3] }; + const ret = await driver.bulkCreate(TABLE_RETURNS, batch, OPTS); + expect(Array.isArray(ret), `bulkCreate() answered ${JSON.stringify(ret)}`).toBe(true); + // `insert(rows).returning('*')` hands back a whole row per element where + // the dialect has RETURNING (Postgres, SQLite) and knex's insert-id + // placeholder where it has not (MySQL) — the same fact §A3 records for + // `create()`. Which of the two a dialect answers is its business and is + // not pinned. What IS pinned: an element that is a row carries every + // instant column, presents it canonically, and names a row of THIS + // batch; and the return is all rows or none, so a door that dropped part + // of a batch could not pass as "the dialect has no RETURNING". + const rowReturns: any[] = ret.filter((r: unknown) => !!r && typeof r === 'object'); + expect( + rowReturns.length === 0 || rowReturns.length === batch.length, + `bulkCreate() returned ${rowReturns.length} row(s) for a batch of ${batch.length}`, + ).toBe(true); + for (const r of rowReturns) { + expect(Object.keys(expectedClosedAt), `bulkCreate() returned a row outside the batch: ${r.id}`).toContain(r.id); + for (const col of INSTANT_COLUMNS) expectCanonicalInstant(r[col], `bulkCreate() return ${r.id} ${col}`); + expect(r.closed_at, `bulkCreate() return ${r.id} closed_at`).toBe(expectedClosedAt[r.id]); + } + // The leg that measures something on EVERY dialect, RETURNING or not: + // the batch landed, its rows read back canonical through `find()`, and — + // where the return carried a row — the return and the row agree value + // for value, so the return door presents what the read door presents. + const landed = (await driver.find(TABLE_RETURNS, { orderBy: [{ field: 'id', order: 'asc' }] }, OPTS)).filter( + (row: any) => row.id in expectedClosedAt, + ); + expect(landed, 'the batch did not land').toHaveLength(batch.length); + for (const row of landed) { + for (const col of INSTANT_COLUMNS) expectCanonicalInstant(row[col], `find() after bulkCreate ${row.id} ${col}`); + expect(row.closed_at).toBe(expectedClosedAt[row.id]); + } + for (const r of rowReturns) { + const row = landed.find((l: any) => l.id === r.id); + for (const col of INSTANT_COLUMNS) expect(r[col], `bulkCreate() return vs find() ${r.id}.${col}`).toBe(row[col]); + } + }); + + it('§B1 aggregate(): min/max over a declared datetime AND over the audit columns are canonical ISO-Z text', async () => { + const query: DriverQuery = { + aggregations: [ + { function: 'min', field: 'closed_at', alias: 'earliest' }, + { function: 'max', field: 'closed_at', alias: 'latest' }, + { function: 'max', field: 'created_at', alias: 'newest_created' }, + { function: 'max', field: 'updated_at', alias: 'newest_updated' }, + { function: 'count', field: 'closed_at', alias: 'n' }, + ], + }; + const res: any[] = await driver.aggregate(TABLE, query, OPTS); + expect(Array.isArray(res) && res.length === 1, `aggregate() answered ${JSON.stringify(res)}`).toBe(true); + const row = res[0]; + expectCanonicalInstant(row.earliest, 'min(closed_at)'); + expectCanonicalInstant(row.latest, 'max(closed_at)'); + expect(row.earliest).toBe(DISTINCT_CLOSED_AT[0]); + expect(row.latest).toBe(DISTINCT_CLOSED_AT[DISTINCT_CLOSED_AT.length - 1]); + // The audit columns had NO aggregate arm before #13973, on any dialect. + expectCanonicalInstant(row.newest_created, 'max(created_at)'); + expectCanonicalInstant(row.newest_updated, 'max(updated_at)'); + // Agrees with what find() presents for the same column, value for value. + const last = (values: string[]): string => [...values].sort()[values.length - 1]; + expect(row.newest_created).toBe(last(rows.map((r) => r.created_at))); + expect(row.newest_updated).toBe(last(rows.map((r) => r.updated_at))); + // A numeric aggregate over a datetime stays a number. + expect(Number(row.n)).toBe(CLOSED_AT.length); + }); + + it('§B2 aggregate(): a raw temporal group key is the canonical text', async () => { + const query: DriverQuery = { + groupBy: ['closed_at'], + aggregations: [{ function: 'sum', field: 'amount', alias: 'total' }], + }; + const res: any[] = await driver.aggregate(TABLE, query, OPTS); + const keys = res.map((r) => r.closed_at); + expect(keys.length).toBe(DISTINCT_CLOSED_AT.length); + for (const key of keys) expectCanonicalInstant(key, 'groupBy closed_at key'); + expect([...keys].sort()).toEqual(DISTINCT_CLOSED_AT); + }); + + it('§B3 distinct(): a declared datetime AND the audit columns are canonical ISO-Z text', async () => { + const closed = await driver.distinct(TABLE, 'closed_at', undefined, OPTS); + expect(closed.length).toBe(DISTINCT_CLOSED_AT.length); + for (const v of closed) expectCanonicalInstant(v, 'distinct(closed_at)'); + expect([...closed].sort()).toEqual(DISTINCT_CLOSED_AT); + + for (const col of ['created_at', 'updated_at'] as const) { + const values = await driver.distinct(TABLE, col, undefined, OPTS); + expect(values.length, `distinct(${col}) is empty`).toBeGreaterThan(0); + for (const v of values) expectCanonicalInstant(v, `distinct(${col})`); + // Agrees with the find() presentation of the same column, value for value. + expect([...values].sort()).toEqual([...new Set(rows.map((r) => r[col]))].sort()); + } + }); + + it("§C the fold is the driver's, not the client's: raw knex still materialises the dialect's own shape", async () => { + const raw: any = await (driver as any).knex(TABLE).where('id', 'r0').first(); + expect(raw, 'raw read returned nothing').toBeTruthy(); + if (cell.live) { + // Postgres (`timestamptz`, node-pg's stock OID 1184 parser) and MySQL + // (`DATETIME(3)`, mysql2 under the `timezone: 'Z'` pin) hand a `Date` + // to the driver — D-F2: the client parser is untouched, the driver + // folds at its own boundary. This is what makes §A/§B a measurement + // rather than a restatement of the client's behaviour. + for (const col of INSTANT_COLUMNS) { + expect( + raw[col] instanceof Date, + `${cell.label} raw ${col} is ${typeof raw[col]} (${String(raw[col])}) — the client parser ` + + `was changed, which the #13973 ruling forbids`, + ).toBe(true); + expect(Number.isNaN((raw[col] as Date).getTime()), `${col}: Invalid Date`).toBe(false); + } + // Same instant, two spellings: the fold changed the TYPE and nothing else. + expect((raw.closed_at as Date).toISOString()).toBe(rows[0].closed_at); + expect((raw.updated_at as Date).toISOString()).toBe(rows[0].updated_at); + expect((raw.created_at as Date).toISOString()).toBe(rows[0].created_at); + } else { + // SQLite has no temporal type: the stored TEXT is already the presented + // text, which is the side of the old asymmetry every consumer was + // tested against. + for (const col of INSTANT_COLUMNS) expect(typeof raw[col], `sqlite raw ${col}`).toBe('string'); + expect(raw.closed_at).toBe(rows[0].closed_at); + } + }); + }); +} + +// A matrix that silently finds zero cells reports OK — every cell is declared +// EITHER WAY, measured when it is provisioned and a named skip when it is not +// (a named RED under `OS_EXPECT_LIVE_DIALECT_MATRIX=1`). +for (const cell of DIALECT_CELLS) { + declareDialectCell(cell, 'canonical ISO read door (#13973)', measure); +} + +/** + * §D — `find()`, `distinct()` and `aggregate()` share ONE presenter for the + * audit columns. + * + * `readPresentationKind` routes `created_at` / `updated_at` to the same + * `presentAuditTimestampOutput` that `formatOutput` applies to a `find()` row — + * not to `normalizeSqliteDatetimeOutput`. The two presenters differ on exactly + * one input class, a NUMBER: the audit presenter passes it through (ADR-0074 + * §3), the datetime fold turns it into ISO text. #13973's first cut routed the + * audit columns to the datetime fold, and the contract review of PR #16619 + * reproduced the divergence that made: an author-declared `created_at: number` + * read `1700000000000` off `find()` and `"2023-11-14T22:13:20.000Z"` off + * `distinct()` and `max()`. Two reachable shapes carry a number there: + * + * D1 an author-declared non-temporal `created_at` — `applySystemFields` lets + * the declaration win (objectql `registry.ts`, "Author-declared fields + * win") and `AUDIT_FIELD_GOVERNANCE` forces only `readonly` / `system`, + * never `type` — holding the number the author wrote; + * D2 an epoch-ms INTEGER raw-written into the builtin, undeclared audit + * column, the pre-ADR-0074 shape a raw insert leaves, which ADR-0074 §3 + * declares `find()` hands through untouched. + * + * ## Why the SQLite cell is the whole coverage, not a shortfall + * + * The driver's DDL never types the audit column from the declaration: a + * declared `created_at` is skipped (`builtinColumns`) and + * `createAuditTimestampColumn` runs, so on Postgres the column is a + * `timestamptz` and on MySQL a `DATETIME(3)` — neither can hold a number, and + * the write that would put one there is refused by the server. SQLite's type + * affinity is what lets a number sit in that column at all, so SQLite is the + * only dialect on which the three doors CAN disagree, and the only one on + * which this pin measures anything; a live cell would exercise the D-F1 shape + * §B1/§B3 already cover and nothing of §D. Firing control: route the audit + * columns back to the `datetime` kind in `readPresentationKind` and §D1/§D2 go + * red on their `distinct()` and `aggregate()` legs — ISO text where `find()` + * answers the number — while every §A/§B/§C cell stays green. + */ +describe('#13973 §D — find(), distinct() and aggregate() present the audit columns through one presenter (sqlite)', () => { + const SQLITE = DIALECT_CELLS.find((c) => c.id === 'sqlite'); + const T_DECLARED = 'os13973_declared_audit'; + const T_RAW = 'os13973_raw_audit'; + /** 2023-11-14T22:13:20.000Z as epoch ms — the review's own value. */ + const EPOCH = 1_700_000_000_000; + let driver: SqlDriver; + + beforeAll(async () => { + expect(SQLITE, 'the matrix lost its SQLite cell').toBeDefined(); + driver = new SqlDriver(SQLITE!.config()); + for (const t of [T_DECLARED, T_RAW]) await driver.execute(`drop table if exists ${t}`).catch(() => {}); + await driver.initObjects([ + // D1: the author declares the audit column non-temporal. + { name: T_DECLARED, fields: { id: { type: 'text' }, created_at: { type: 'number' }, n: { type: 'number' } } }, + // D2: the audit column is the builtin, undeclared one. + { name: T_RAW, fields: { id: { type: 'text' }, n: { type: 'number' } } }, + ] as any); + await driver.create(T_DECLARED, { id: 'd0', created_at: EPOCH, n: 1 }, OPTS); + await driver.create(T_DECLARED, { id: 'd1', created_at: EPOCH + 1, n: 2 }, OPTS); + await driver.create(T_RAW, { id: 'x0', n: 1 }, OPTS); + // Past the driver's write door, as a raw insert would leave it: an epoch + // INTEGER in both builtin audit columns. + await (driver as any).knex(T_RAW).where('id', 'x0').update({ created_at: EPOCH, updated_at: EPOCH }); + }); + + afterAll(async () => { + for (const t of [T_DECLARED, T_RAW]) await driver?.execute(`drop table if exists ${t}`).catch(() => {}); + await driver?.disconnect(); + }); + + /** The three doors' answers for one column, in the shape each door hands out. */ + async function threeDoors(table: string, col: 'created_at' | 'updated_at') { + const query: DriverQuery = { aggregations: [{ function: 'max', field: col, alias: 'newest' }] }; + const [rows, distinct, agg] = await Promise.all([ + driver.find(table, { orderBy: [{ field: 'id', order: 'asc' }] }, OPTS), + driver.distinct(table, col, undefined, OPTS), + driver.aggregate(table, query, OPTS) as Promise, + ]); + return { find: rows.map((r: any) => r[col]), distinct: [...distinct].sort(), max: agg[0]?.newest }; + } + + it('§D1 an author-declared non-temporal created_at: the number find() presents is what distinct() and max() present', async () => { + const doors = await threeDoors(T_DECLARED, 'created_at'); + // The `find()` side, stated rather than assumed: the declared type wins + // (the numeric repair is a no-op on a number, the audit presenter passes + // it through), so the row carries the number the author wrote. + expect(doors.find, 'find()').toEqual([EPOCH, EPOCH + 1]); + // `toEqual` is type-strict: "2023-11-14T22:13:20.000Z" is not 1700000000000. + expect(doors.distinct, 'distinct(created_at) disagrees with find()').toEqual([EPOCH, EPOCH + 1]); + expect(doors.max, 'max(created_at) disagrees with find()').toBe(EPOCH + 1); + expect(typeof doors.max).toBe('number'); + }); + + it('§D2 an epoch INTEGER raw-written into the builtin audit columns: the three doors agree, on both columns', async () => { + for (const col of ['created_at', 'updated_at'] as const) { + const doors = await threeDoors(T_RAW, col); + // ADR-0074 §3: a number passes the audit presenter untouched on `find()`. + expect(doors.find, `find() ${col}`).toEqual([EPOCH]); + expect(doors.distinct, `distinct(${col}) disagrees with find()`).toEqual([EPOCH]); + expect(doors.max, `max(${col}) disagrees with find()`).toBe(EPOCH); + } + }); + + it('§D3 the control: the same two doors still fold a Field.datetime number and a legacy naive audit string to ISO text, as find() does', async () => { + // A raw zone-naive `CURRENT_TIMESTAMP` string in the undeclared audit + // column is the shape ADR-0074 repairs on `find()`; §D must not have + // bought the number agreement by losing that repair at these doors. + await driver.create(T_RAW, { id: 'x1', n: 2 }, OPTS); + await (driver as any).knex(T_RAW).where('id', 'x1').update({ updated_at: '2026-01-10 09:00:00' }); + const legacy = await driver.findOne(T_RAW, { where: { id: 'x1' } }, OPTS); + expect(legacy.updated_at).toBe('2026-01-10T09:00:00.000Z'); + const distinct = await driver.distinct(T_RAW, 'updated_at', undefined, OPTS); + expect(distinct).toContain('2026-01-10T09:00:00.000Z'); + expect(distinct).toContain(EPOCH); + const query: DriverQuery = { aggregations: [{ function: 'max', field: 'updated_at', alias: 'newest' }] }; + const agg: any[] = await driver.aggregate(T_RAW, query, OPTS); + // SQLite `max()` over mixed INTEGER/TEXT storage orders TEXT above INTEGER, + // so the newest is the legacy string — presented through the same repair. + expect(agg[0].newest).toBe('2026-01-10T09:00:00.000Z'); + }); +}); diff --git a/packages/drivers/driver-sql/src/sql-driver-14078-invalid-date-materialisation.test.ts b/packages/drivers/driver-sql/src/sql-driver-14078-invalid-date-materialisation.test.ts index 34d6c422ea..c170a4a776 100644 --- a/packages/drivers/driver-sql/src/sql-driver-14078-invalid-date-materialisation.test.ts +++ b/packages/drivers/driver-sql/src/sql-driver-14078-invalid-date-materialisation.test.ts @@ -48,11 +48,16 @@ * ## The column, and why the write door is bypassed * * `updated_at` — the same builtin audit column `#13567` measures one file over, - * for the same reason: it is a BUILTIN, so it is not in `datetimeFields`, no - * declared-field coercion reaches it, and `formatOutput`'s audit repair - * (`repairNaiveUtcAuditTimestamp`) sits inside `if (this.isSqlite)`. On the two - * live dialects the read door therefore hands back whatever the client library - * produced, unmodified — which is precisely what is being measured. + * for the same reason: it is a BUILTIN, so it is not in `datetimeFields` and no + * declared-field coercion reaches it. When this file was written, + * `formatOutput`'s audit repair sat inside `if (this.isSqlite)`, so on the two + * live dialects the read door handed back whatever the client library produced, + * unmodified. Since #13973 (ADR-0053 D-F1) that door folds a valid `Date` to + * the canonical ISO-Z text on every dialect — §B1's control now reads back as + * text — and the Invalid `Date` is the ONE shape the fold hands through + * unchanged (D-F3): it has no canonical text, and the fold is total in the + * sense this card ruled (it never throws). §B2/§B3 therefore still measure + * exactly what the client library produced for it, through the same door. * * The value is written with **raw knex**, not through `create`/`update`. That * is deliberate and is stated rather than hidden: the ObjectQL write door @@ -63,11 +68,12 @@ * * ## Pinned as OBSERVED, not as desired * - * These assertions describe what the drivers do today. They are not a claim - * that it is correct, and they take no side on the shared spelling. A client - * upgrade that changed a materialisation would redden this file — which is the - * point: the decision, whichever way it goes, is being made against a reading - * that must stay true. + * The Invalid-`Date` assertions describe what the drivers do today. They are + * not a claim that it is correct, and they take no side on the shared spelling. + * A client upgrade that changed a materialisation would redden this file — + * which is the point: the decision, whichever way it goes, is being made + * against a reading that must stay true. (Ruled since: option B, 2026-09-02 — + * every copy of the shared spelling carries a total `Date` arm.) */ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; @@ -389,15 +395,23 @@ function measure(cell: DialectCell): void { return; } - it('§B1 the control value materialises as a REAL instant', () => { + it('§B1 the control value materialises as a REAL instant — and leaves the read door as canonical text', () => { const label = cell.id === 'pg' ? 'year 0001' : 'year 1000'; const probe = probes.get(label)!; expect(probe.stored, `${label} was refused: ${probe.refusal}`).toBe(true); - expect(probe.readBack instanceof Date, `${label}: got ${typeof probe.readBack}`).toBe(true); - const value = probe.readBack as Date; - expect(Number.isNaN(value.getTime()), `${label}: ${String(value)}`).toBe(false); - // The control's whole job: the shared spelling renders this one fine, so - // anything §B2 finds is about the value, not about the door. + // [#13973] ADR-0053 D-F1: a valid instant is folded to the canonical + // text at the driver's read boundary, so the control no longer reaches + // any consumer's `Date` arm at all — the string arm returns it first. + expect( + probe.readBack instanceof Date, + `${label}: the read door handed out a JS Date (${String(probe.readBack)})`, + ).toBe(false); + expect(typeof probe.readBack, `${label}: got ${typeof probe.readBack}`).toBe('string'); + expect(probe.readBack, `${label}: shape`).toMatch(/^-?\d{4,6}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/); + const value = new Date(probe.readBack as string); + expect(Number.isNaN(value.getTime()), `${label}: ${String(probe.readBack)}`).toBe(false); + // The control's whole job: this instant is one the fold CAN canonicalise, + // so anything §B2/§B3 finds is about the value, not about the door. expect(renderThroughSharedSpelling(value).threw).toBe(false); }); diff --git a/packages/drivers/driver-sql/src/sql-driver.ts b/packages/drivers/driver-sql/src/sql-driver.ts index 91337b2b75..67e1068bd4 100644 --- a/packages/drivers/driver-sql/src/sql-driver.ts +++ b/packages/drivers/driver-sql/src/sql-driver.ts @@ -264,8 +264,9 @@ const NUMERIC_SCALAR_TYPES = new Set([ /** * The builtin audit-timestamp columns every managed object carries. They are * stamped to a single canonical instant format on SQLite (see - * `stampInsertTimestamps`/`update`) and read-repaired by - * `repairNaiveUtcAuditTimestamp`. + * `stampInsertTimestamps`/`update`) and presented on read — on EVERY dialect — + * as that same canonical instant text by {@link presentAuditTimestampOutput} + * ([ADR-0053 D-F1], #13973). */ const AUDIT_TIMESTAMP_COLUMNS = ['created_at', 'updated_at'] as const; @@ -302,6 +303,52 @@ function repairNaiveUtcAuditTimestamp(value: unknown): unknown { return Number.isNaN(d.getTime()) ? value : d.toISOString(); } +/** + * Fold a client library's `Date` — what node-pg materialises a `timestamptz` + * as and what mysql2 materialises a `DATETIME(3)` as — into the canonical + * instant text, `YYYY-MM-DDTHH:MM:SS.sssZ`. + * + * Total in the sense #14078 ruled for the shared consumer spelling: the ONE + * `Date` shape `toISOString()` refuses — an Invalid `Date`, whose time value + * is `NaN` — is handed back UNCHANGED rather than thrown on. Both live + * dialects are measured to produce it from rows that are already on disk (a + * MySQL zero `DATETIME`; a Postgres year past 275760, inside the server's + * range and outside JS's — `sql-driver-14078-invalid-date-materialisation.test.ts`), + * and it has no canonical text to fold to. It is neither nulled (that erases a + * stored value silently) nor spelled as the text `Invalid Date` (that would + * change the wire, where `JSON.stringify` already serialises it as `null`); it + * leaves as the client's `Date`, which is exactly the shape the ruled-B + * consumer arms (`canonicalIsoInstant` and kin) already absorb. + * [ADR-0053 D-F3]. + */ +function isoFromValidDate(value: Date): unknown { + return Number.isNaN(value.getTime()) ? value : value.toISOString(); +} + +/** + * What a record read door hands out for a builtin audit timestamp, on every + * dialect ([ADR-0053 D-F1], #13973): the canonical instant text. + * + * Two arms, one per shape the dialects actually produce. A `Date` — the + * client-level materialisation on Postgres (`timestamptz`) and MySQL + * (`DATETIME(3)`), which the driver deliberately does NOT alter at the client + * parser ([ADR-0053 D-F2]; see {@link SqlDriver.withPostgresCalendarDayAsText}) + * — is folded by {@link isoFromValidDate}. A string — SQLite's TEXT, canonical + * post-ADR-0074 or a legacy zone-naive `CURRENT_TIMESTAMP` — takes ADR-0074's + * {@link repairNaiveUtcAuditTimestamp}, which is idempotent on the canonical + * form. Every other shape passes through untouched, exactly as before. + * + * Before #13973 the string arm ran inside `if (this.isSqlite)` and the `Date` + * arm did not exist, so the two live dialects handed the client's `Date` + * through the read door — the divergence the #13973 census measured eight + * consumers to be wrong under (#13382 in production, #13993–#13999 by reading), + * every one in the direction "expected the text, received a `Date`". + */ +function presentAuditTimestampOutput(value: unknown): unknown { + if (value instanceof Date) return isoFromValidDate(value); + return repairNaiveUtcAuditTimestamp(value); +} + /** * Whether a field's `defaultValue` is the framework's `'NOW()'` convention * ("use the database clock at insert time"). Case-insensitive, whitespace @@ -339,8 +386,15 @@ const isNowDefaultValue = isNowDefaultToken; * ADR-0074's `repairNaiveUtcAuditTimestamp` for the string shapes (the single * source of the zone-naive→UTC rules) and adds the INTEGER epoch-ms / `Date` * folding, mirroring the read-repair the `Field.date`/numeric-scalar paths do. - * SQLite-only: Postgres/MySQL store a real zone-aware TIMESTAMP and never carry - * this ambiguity. + * + * Runs on EVERY dialect since #13973 ([ADR-0053 D-F1]): the name records the + * storage forms it was written to fold (SQLite's INTEGER epoch and naive TEXT), + * not where it runs. Postgres and MySQL store a real `timestamptz` / + * `DATETIME(3)` and never carry the mixed-storage ambiguity, but their client + * libraries hand the column back as a JS `Date`, and that `Date` used to leave + * the read door as-is — the SQLite-gated fold was the whole of the dialect + * divergence the #13973 census measured. The `Date` arm below is the one those + * two dialects take, and the client parser stays untouched ([ADR-0053 D-F2]). */ function normalizeSqliteDatetimeOutput(value: unknown): unknown { if (value == null) return value; @@ -350,11 +404,10 @@ function normalizeSqliteDatetimeOutput(value: unknown): unknown { const d = new Date(value); return Number.isNaN(d.getTime()) ? value : d.toISOString(); } - // A JS `Date` is never returned by better-sqlite3 here, but normalize one - // defensively so any caller-shaped row also reads back canonical. - if (value instanceof Date) { - return Number.isNaN(value.getTime()) ? value : value.toISOString(); - } + // The client-level materialisation on Postgres and MySQL (never what + // better-sqlite3 returns): fold it to the canonical text, an Invalid `Date` + // passing through — see `isoFromValidDate` for why that is the total arm. + if (value instanceof Date) return isoFromValidDate(value); if (typeof value !== 'string') return value; const s = value.trim(); if (s === '') return value; @@ -4084,8 +4137,15 @@ export type SqlWindowFunctionQuery = Omit & { * presents. One entry per rule `formatOutput` applies to a scalar column; the * read paths that bypass `formatOutput` (`aggregate`, `distinct`) name the rule * per column instead. See {@link SqlDriver.readPresentationKind}. + * + * `audit_timestamp` is the builtin `created_at` / `updated_at` rule — + * {@link presentAuditTimestampOutput}, the presenter `formatOutput` applies to + * those two columns — kept apart from `datetime` because the two presenters + * differ on a number: the datetime fold turns an epoch INTEGER into ISO text, + * the audit presenter passes it through (ADR-0074 §3), and a `find()` row + * takes the latter ([ADR-0053 D-F1], #13973). */ -export type ReadPresentationKind = 'datetime' | 'date' | 'time' | 'boolean' | 'number'; +export type ReadPresentationKind = 'datetime' | 'date' | 'time' | 'boolean' | 'number' | 'audit_timestamp'; /** * Journal modes the driver knows how to ask a file-backed SQLite database for. @@ -5246,9 +5306,20 @@ export class SqlDriver implements IDataDriver { * optional peer dependency and is never imported here — `setTypeParser` / * `getTypeParser` are read off the `pg.Client` knex hands the hook. * - * `timestamptz` / `timestamp` are deliberately untouched: those are - * instants, a `Date` is the right materialisation for them, and - * `Field.datetime` depends on it. + * `timestamptz` / `timestamp` are deliberately untouched here: those are + * instants, and a `Date` is the right materialisation for them at the + * CLIENT layer, where the wire text is parsed. It is not, however, what the + * driver hands out of its read doors: [ADR-0053 D-F1/D-F2] (#13973) fold that + * `Date` to the canonical `YYYY-MM-DDTHH:MM:SS.sssZ` text in `formatOutput` + * / `presentReadValue`, so every dialect presents the instant the way + * SQLite always has. This comment used to add that "`Field.datetime` + * depends on" the `Date` materialisation; #13973 checked that on the tree + * and it did not hold — nothing on the read path consumed the `Date` (the + * SQLite-gated `formatOutput` never touched a datetime column on Postgres), + * and every in-repo reader of the value already accepted the text form. + * Canonicalising at the driver's own read boundary rather than at this + * parser is the ruled shape: one clock stays in play here for `date`, the + * instant types keep their stock parser, and no host `pg` client is touched. * * A host's existing `pool.afterCreate` is chained rather than replaced, * exactly as in {@link withUtcSession}. @@ -12761,6 +12832,28 @@ export class SqlDriver implements IDataDriver { * row, asked one field at a time so the paths that return raw builder output * can ask it too. `null` means the stored form already IS the presented form. * + * The temporal kinds run on every dialect. For `datetime` that is [ADR-0053 + * D-F1] (#13973): the fold was SQLite-only before, so `aggregate()` and + * `distinct()` handed Postgres' and MySQL's `Date` through here exactly as + * `find()` did. The builtin audit columns take their OWN rule, + * `audit_timestamp` — the same `presentAuditTimestampOutput` `formatOutput` + * applies to them — unless the author declared the column temporal (the + * engine's `applySystemFields` declares both as `Field.datetime`), in which + * case the temporal rule wins here exactly as `formatOutput`'s datetime fold + * runs after its audit fold there. They are not in `datetimeFields` by + * themselves (builtin, not declared), and before #13973 this function had + * NO arm for them on ANY dialect, so `max(updated_at)` and + * `distinct('created_at')` diverged from `find()` even on SQLite, where they + * missed ADR-0074's legacy-row repair. The arm is the audit presenter and + * not the `datetime` one because the two differ on a NUMBER: ADR-0074 §3 + * passes an epoch INTEGER through on a `find()` row, and a `created_at` the + * author declared non-temporal (`applySystemFields` lets the declaration + * win; `AUDIT_FIELD_GOVERNANCE` forces only `readonly` / `system`, never + * `type`) leaves `find()` as the number it is — routed to the datetime fold, + * both became ISO text at this door and nowhere else (the #16619 contract + * review's finding; `sql-driver-13973-canonical-iso-read-door.test.ts` §D + * pins the agreement). + * * The boolean rule runs on SQLite AND MySQL — the two dialects that store a * declared boolean as a number (INTEGER 0/1, `tinyint(1)`) — because * `formatOutput` gates its row reads that way (#11782; SQLite-only before, @@ -12776,6 +12869,7 @@ export class SqlDriver implements IDataDriver { if (!table) return null; const temporal = this.temporalFieldKind(table, field); if (temporal) return temporal; + if ((AUDIT_TIMESTAMP_COLUMNS as readonly string[]).includes(field)) return 'audit_timestamp'; if ((this.isSqlite || this.isMysql) && this.booleanFields[table]?.includes(field)) { return 'boolean'; } @@ -12785,16 +12879,38 @@ export class SqlDriver implements IDataDriver { } /** - * Present one value exactly the way `formatOutput` presents it on a `find()` - * row, for the read paths that return raw builder output instead - * (`aggregate`, `distinct` — #3797 for instants, #3849 for scalars). - * - * The dialect gating mirrors `formatOutput`: the `Field.datetime` repair and - * the numeric coercion are SQLite-only, the boolean coercion runs on SQLite - * and MySQL (#11782 — the two dialects whose stored boolean is a number), - * and the `Field.date` → `YYYY-MM-DD` collapse runs everywhere. + * Present one value through the presenter `formatOutput` applies to that + * column class on a `find()` row, for the read paths that return raw builder + * output instead (`aggregate`, `distinct` — #3797 for instants, #3849 for + * scalars): one presenter per kind, and it is the function `formatOutput` + * itself calls, never a re-derivation of it. + * + * The dialect gating mirrors `formatOutput`: the three temporal folds and + * the audit-stamp fold run everywhere (`datetime` and `audit_timestamp` + * since #13973, [ADR-0053 D-F1] — the former SQLite-only and the latter + * absent before, which handed the two live dialects' `Date` through), the + * numeric coercion is SQLite-only, and the boolean coercion runs on SQLite + * and MySQL (#11782 — the two dialects whose stored boolean is a number). * {@link readPresentationKind} does the dialect gating for the scalar kinds, * so by the time one arrives here the dialect is settled. + * + * Where this is NOT `formatOutput` to the letter: a row walk COMPOSES. A + * `created_at` an author declared `number` takes the SQLite numeric repair + * and then the audit presenter there, while here it takes the audit + * presenter alone (`readPresentationKind` answers one kind per column). The + * two differ on any TEXT that `Number()` accepts but SQLite's NUMERIC + * affinity leaves as TEXT — hex, binary and octal literals (`'0x10'`, + * `'0b101'`, `'0o17'`) and `'Infinity'` — which `find()` reads as the number + * (`16`, `5`, `15`, `Infinity`) and this door as the string; that is + * reachable through `create()` / `update()` on the driver's own DDL with an + * author-declared non-temporal audit column, not only through a hand-made + * TEXT-affinity column (measured in the #16619 contract review). A decimal + * or exponent spelling (`'1700000000000'`, `'1e3'`, `'.5'`) is folded to + * INTEGER/REAL by the affinity before it is read back, and a number passes + * both presenters untouched, so those agree. The B1 ruling did not decide + * that residual shape — it lies outside its column classes — and nothing + * here closes it. `sql-driver-13973-canonical-iso-read-door.test.ts` §D + * pins the agreement on the shapes the ruling covers. */ protected presentReadValue(kind: ReadPresentationKind, value: any): any { if (value == null) return value; @@ -12807,7 +12923,23 @@ export class SqlDriver implements IDataDriver { // exactly what `find()` presents (#3994, the F6 gap of the #3849 fix). return this.toTimeOnly(value); case 'datetime': - return this.isSqlite ? normalizeSqliteDatetimeOutput(value) : value; + // Every dialect ([ADR-0053 D-F1]): the same fold `formatOutput` applies + // to a `find()` row, so `min`/`max`/`distinct` over a declared + // `Field.datetime` (the audit columns too, when the engine declares + // them so) present the canonical text and never the client's `Date`. + return normalizeSqliteDatetimeOutput(value); + case 'audit_timestamp': + // Every dialect ([ADR-0053 D-F1]): the ONE presenter `formatOutput` + // applies to `created_at` / `updated_at` — a `Date` folds to the + // canonical text, a zone-naive legacy string is repaired (ADR-0074), + // and a number passes through as it does on a `find()` row (ADR-0074 + // §3) — so `max(created_at)` and `distinct('updated_at')` answer the + // value `find()` answers, type for type. `normalizeSqliteDatetimeOutput` + // would fold the number too, and did between #13973's first cut and + // its contract review: an author-declared `created_at: number` read + // `1700000000000` off `find()` and `"2023-11-14T22:13:20.000Z"` off + // `distinct()` — the divergence the conformance file's §D pins closed. + return presentAuditTimestampOutput(value); case 'boolean': return Boolean(value); case 'number': { @@ -16629,32 +16761,46 @@ export class SqlDriver implements IDataDriver { } } - // Builtin audit timestamps: repair any legacy/raw row stored as a - // zone-naive, space-separated string (CURRENT_TIMESTAMP or the pre-fix - // UPDATE stamp) to canonical ISO-8601 with `Z`, so reads are unambiguous - // and uniform regardless of when/how the row was written. Idempotent on - // already-canonical values; mirrors the legacy-row read-repair the - // `Field.date`/numeric paths already do. See `repairNaiveUtcAuditTimestamp`. - for (const col of AUDIT_TIMESTAMP_COLUMNS) { - if (data[col] !== undefined) data[col] = repairNaiveUtcAuditTimestamp(data[col]); - } + } - // Present every `Field.datetime` value as one canonical instant — - // ISO-8601 with an explicit `Z` — regardless of its on-disk storage form. - // A SQLite `datetime` column mixes forms: an explicit value bound as a JS - // `Date` is stored as INTEGER epoch ms, while a `defaultValue: 'NOW()'` - // slot is TEXT (canonical ISO-`Z` post-fix, or a legacy timezone-naive - // `CURRENT_TIMESTAMP` string). Without this, reads leak the raw integer or - // a zone-naive string that `Date.parse` mis-reads as LOCAL time. Folds all - // shapes to UTC ISO-`Z` and transparently repairs legacy rows with no data - // migration — mirroring the `Field.date`/numeric read-repairs above and - // the audit-column repair just above. See `normalizeSqliteDatetimeOutput`. - const datetimeFields = this.datetimeFields[object]; - if (datetimeFields && datetimeFields.size > 0) { - for (const field of datetimeFields) { - if (data[field] !== undefined) { - data[field] = normalizeSqliteDatetimeOutput(data[field]); - } + // [ADR-0053 D-F1] (#13973) — the two instant classes present as ONE shape + // on EVERY dialect: the canonical `YYYY-MM-DDTHH:MM:SS.sssZ` text. Both + // loops below sat inside the `if (this.isSqlite)` arm above until #13973, + // and that gate WAS the divergence the card's census measured: on + // Postgres and MySQL the client library's `Date` left this door as-is, + // while SQLite handed out text — one value, two runtime types, through the + // same `any`-shaped record, invisible to the type system. Eight consumers + // were wrong under it (#13382 in production, #13993–#13999 by reading), + // all in the direction "expected the text, received a `Date`", none the + // reverse. The client parsers are NOT touched ([ADR-0053 D-F2]); the + // driver canonicalises at its own read boundary, here and in + // `presentReadValue` for the `aggregate()`/`distinct()` doors. + + // Builtin audit timestamps. On SQLite: repair any legacy/raw row stored as + // a zone-naive, space-separated string (CURRENT_TIMESTAMP or the pre-fix + // UPDATE stamp) to canonical ISO-8601 with `Z`, idempotent on + // already-canonical values (ADR-0074). On Postgres/MySQL: fold the + // client's `Date`. See `presentAuditTimestampOutput`. + for (const col of AUDIT_TIMESTAMP_COLUMNS) { + if (data[col] !== undefined) data[col] = presentAuditTimestampOutput(data[col]); + } + + // Present every `Field.datetime` value as one canonical instant — + // ISO-8601 with an explicit `Z` — regardless of its on-disk storage form. + // A SQLite `datetime` column mixes forms: an explicit value bound as a JS + // `Date` is stored as INTEGER epoch ms, while a `defaultValue: 'NOW()'` + // slot is TEXT (canonical ISO-`Z` post-fix, or a legacy timezone-naive + // `CURRENT_TIMESTAMP` string). Without this, reads leak the raw integer or + // a zone-naive string that `Date.parse` mis-reads as LOCAL time. Folds all + // shapes to UTC ISO-`Z` and transparently repairs legacy rows with no data + // migration — mirroring the `Field.date`/numeric read-repairs above and + // the audit-column repair just above. A Postgres/MySQL `Date` takes the + // same fold. See `normalizeSqliteDatetimeOutput`. + const datetimeFields = this.datetimeFields[object]; + if (datetimeFields && datetimeFields.size > 0) { + for (const field of datetimeFields) { + if (data[field] !== undefined) { + data[field] = normalizeSqliteDatetimeOutput(data[field]); } } } diff --git a/scripts/adr-anchors/packages__drivers__driver-sql__src__sql-driver.ts.json b/scripts/adr-anchors/packages__drivers__driver-sql__src__sql-driver.ts.json index 251e903868..dcf5f7716b 100644 --- a/scripts/adr-anchors/packages__drivers__driver-sql__src__sql-driver.ts.json +++ b/scripts/adr-anchors/packages__drivers__driver-sql__src__sql-driver.ts.json @@ -1,7 +1,8 @@ { "file": "packages/drivers/driver-sql/src/sql-driver.ts", "adrs": [ + "ADR-0053", "ADR-0120" ], - "invariant": "The bare-composite → NULL-safe tightening migrates through the ceremony (ADR-0120 D4): a `recreate_index` gated by the duplicate pre-flight probe — clean data grades it `safe` (dev autoMigrate may apply), duplicates BLOCK it with a row report and the old index stays in place; apply re-probes, so even --allow-destructive cannot drop a constraint whose replacement is not creatable. Storage stays NULL — GLOBAL_TENANT is an index-key fold, never written to the organization column." + "invariant": "ADR-0053 D-F1/D-F2/D-F3: `formatOutput`'s audit-column and `Field.datetime` folds and `presentReadValue`'s `datetime` arm run on EVERY dialect — a read door presents `created_at`/`updated_at` and every declared `datetime` as the canonical `YYYY-MM-DDTHH:MM:SS.sssZ` text and never the client library's `Date`; the fold lives at the driver's read boundary, never at the pg/mysql2 type parser, and an Invalid `Date` passes through unchanged. Re-gating either fold on `isSqlite` reverts a maintainer ruling (#13973). ADR-0120 D4: the bare-composite → NULL-safe tightening migrates through the ceremony — a `recreate_index` gated by the duplicate pre-flight probe: clean data grades it `safe` (dev autoMigrate may apply), duplicates BLOCK it with a row report and the old index stays in place; apply re-probes, so even --allow-destructive cannot drop a constraint whose replacement is not creatable. Storage stays NULL — GLOBAL_TENANT is an index-key fold, never written to the organization column." }