diff --git a/.changeset/8714-test-tsconfig-parity-census.md b/.changeset/8714-test-tsconfig-parity-census.md new file mode 100644 index 0000000000..320226b976 --- /dev/null +++ b/.changeset/8714-test-tsconfig-parity-census.md @@ -0,0 +1,11 @@ +--- +--- + +Adds `scripts/tsconfig-test-parity-census.mjs` (`pnpm census:tsconfig-test-parity`) and +`docs/audits/2026-09-test-tsconfig-parity-census.md`, the measurement objectui#8714 asked +for: every workspace `tsconfig.test.json` resolved through TypeScript's own config +resolver and compared axis by axis, plus the two ambient-type routes that live in the test +sources rather than in any config. + +Internal tooling and documentation only — no published package source, entry point or +publish-contract field is touched, so nothing releases from this change. diff --git a/docs/audits/2026-09-test-tsconfig-parity-census.md b/docs/audits/2026-09-test-tsconfig-parity-census.md new file mode 100644 index 0000000000..80a2cf0e3a --- /dev/null +++ b/docs/audits/2026-09-test-tsconfig-parity-census.md @@ -0,0 +1,331 @@ +# Audit: test-`tsconfig` parity census across the workspace (2026-09) + +**Question** (objectui#8714): the repo has found **three** per-package `tsconfig` +divergences — each by a developer tripping over it, none by any instrument. How many +packages differ, on how many axes, with how many distinct values each? + +**Why it exists**: the card that triggered this named one instance — +`plugin-timeline/tsconfig.test.json` omits `@testing-library/jest-dom` from +`compilerOptions.types` while `plugin-map`'s names it, so a jest-dom matcher there is +**green under vitest and `TS2339` under `tsc`**. Its own reading was that the tractable +card is not *"add jest-dom to plugin-timeline"* but *"nothing asserts that sibling +packages' test type-programs agree."* That question is unanswerable without this number, +and nobody had it. + +⛔ **This audit changes no `tsconfig`, adds no gate, and rules nothing.** Some +divergences are deliberate and load-bearing — one of them is measured below. Normalising +them would be a repo-wide behaviour change dressed as tidying. What lands with this audit +is the *instrument*: `scripts/tsconfig-test-parity-census.mjs` +(`pnpm census:tsconfig-test-parity`), report-only, re-runnable. + +**Instrument**: TypeScript's own config resolver +(`ts.getParsedCommandLineOfConfigFile`), so `extends` chains are flattened the way `tsc` +flattens them, and every reading below is taken from the *resolved* program rather than +from the config text. + +**Precondition**: every `tsc -p` reading here was taken with the dependency closure +**built** (`pnpm build`, 43/43 tasks green). An unbuilt tree gives `TS2307` for every +workspace import and makes all 38 type-programs look broken in the same way — a +precondition, never a result. + +--- + +## Summary + +| | | +| --- | --- | +| test type-programs in the workspace | **42** | +| …with a dedicated `tsconfig.test.json` (this census) | **38** | +| …that fuse tests into the package's ordinary `tsconfig.json` | **4** | +| program root files across the 38 | **2,477** | +| of the 38, compiling green today | **38 / 38** (measured, exit 0, 0 errors each) | +| divergent axes across all 38 | **23** | +| behavioural axes within the 35-package sibling cohort | **4** (`types`, `lib`, `include`, `exclude`) | +| distinct values on those four | 6 / 4 / 3 / 2 | +| **live cost already paid, in workarounds** | **2 sites** — see *Already paid* | +| **files typed only by leakage from another file** | **83** | + +⭐ **The headline**: it is not *"two packages differ on one key"* (three one-line fixes and +a note). It is **35 sibling packages, four behavioural axes, no axis with fewer than two +values and one with six** — plus two ambient-type routes that are **not in any `tsconfig` +at all**, which is why three separate cards each found "the" divergence and none of them +found the set. + +--- + +## Population, and proof it is not empty + +``` +$ pnpm census:tsconfig-test-parity | head -1 +# test-tsconfig parity census — 38 projects, 2477 program files +``` + +38 files, discovered from disk (`packages/*`, `apps/*`, `examples/*`), not from a list — +a list is a thing to forget, and a package outside the census reads exactly like a package +that agrees with its siblings. A census that reads zero because its glob is wrong is +indistinguishable from a uniform repo, so the script **exits non-zero on an empty +population** and `scripts/__tests__/tsconfig-test-parity-census.test.ts` runs that leg. + +**Two cohorts**, and the split matters because it explains 19 of the 23 axes: + +| cohort | n | `extends` | note | +| --- | --- | --- | --- | +| **A** | 35 | `../../tsconfig.json` (repo root) | the real sibling comparison | +| **B** | 3 | its own package `tsconfig.json` | `examples/console-starter`, `examples/schema-catalog`, `packages/vscode-extension` — a different base, so they diverge on 24 axes among themselves; not comparable to A | + +**The 4 fused programs** (no `tsconfig.test.json`; tests compiled by the package's build +config, under the build's options) are a structural axis in their own right: + +| package | test files | `types` | `lib` | +| --- | --- | --- | --- | +| `apps/console` | 93 | `["vitest/globals", "@testing-library/jest-dom"]` | `[ES2020, DOM, DOM.Iterable]` | +| `packages/data-objectstack` | 60 | `["node"]` | inherited `[ES2020, DOM, DOM.Iterable]` | +| `packages/cli` | 16 | *unset* | `["ES2020"]` — **no DOM** | +| `packages/test-support` | 5 | `["node", "vitest/globals"]` | inherited | + +`vitest/globals` appears in exactly **2 of the 42** programs. `apps/console` extends +nothing at all. + +--- + +## The matrix — cohort A (35 sibling packages) + +### 1. `compilerOptions.types` — 6 effective values (7 spellings) + +| value | n | packages | +| --- | --- | --- | +| *unset* | 11 | collaboration fields mobile plugin-designer plugin-editor plugin-markdown **plugin-timeline** plugin-tree providers react-runtime sdui-parser | +| `["node"]` | 9 | core create-plugin layout permissions plugin-dashboard plugin-detail plugin-gantt plugin-list types | +| `["node", "@testing-library/jest-dom"]` | 7 | auth components i18n plugin-chatbot plugin-form plugin-grid react | +| `["@testing-library/jest-dom", "node"]` | 6 | plugin-calendar plugin-charts plugin-kanban **plugin-map** plugin-report plugin-view | +| `["node", "vite/client", "@testing-library/jest-dom"]` | 1 | app-shell | +| `["node", "vite/client"]` | 1 | runner | + +The last two spellings are the same set in different order — the census normalises sets, +so those two rows are **one** effective value, and 6 is the honest count. + +**`types` is a switch, not a list**: naming it at all turns OFF automatic `@types/*` +inclusion. So *unset* and `["node"]` are not "one has jest-dom, one does not" — they are +two different mechanisms. + +### 2. `compilerOptions.lib` — 4 effective values + +| value | n | packages | +| --- | --- | --- | +| `[ES2020, DOM, DOM.Iterable]` | 27 | (the rest) | +| `[ES2022, DOM, DOM.Iterable]` | 4 | app-shell components mobile plugin-list | +| `[ES2020, DOM]` | 3 | i18n plugin-form types | +| `[ES2022, DOM]` | 1 | plugin-grid | + +**6 of the 38** programs are on ES2022. That number matters — see *Already paid*. + +### 3. `include` — 3 effective values + +| value | n | +| --- | --- | +| `["src/**/*.test.ts", "src/**/*.test.tsx"]` | 28 | +| `+ "src/**/*.d.ts"` | 6 — auth components plugin-charts plugin-dashboard plugin-grid plugin-map | +| `+ "src/**/*.bench.ts"` | 1 — core | + +### 4. `exclude` — 2 values + +Unset in 34; `["src/browser-process-shim.d.ts"]` in `components`. + +### Same program, different spelling (reported, not counted as divergence) + +`composite` (`false` ×32 / unset ×3) and `declaration` (`false` ×11 / unset ×24). Both +default to `false` in TypeScript, so the two spellings compile identically. The census +prints them marked `[same program, different spelling]` rather than hiding them, because +"why does my package say this and my neighbour's not" is a real question with a boring +answer. + +### Uniform across cohort A (the good news) + +`strict`, `module`, `moduleResolution` (`bundler`), `target` (ES2020), `jsx` (`react-jsx`), +`paths` (`{}` — every sibling drops the root source-tree paths so `@object-ui/*` resolves +through built `.d.ts`), `noEmit`, `skipLibCheck`, `isolatedModules`, `resolveJsonModule`, +`baseUrl`, `noUnusedLocals` (`false`), `noUnusedParameters` (`true`), +`useDefineForClassFields`. + +--- + +## The two axes that are not in any `tsconfig` + +This is the part a `diff` of the config files cannot produce, and it is why the card's +framing ("nothing compares them") was right about the *problem* and incomplete about the +*surface*. + +### Route 3 — jest-dom typed by a side-effect import in a **different file** + +`import '@testing-library/jest-dom'` is a global augmentation. Once **any** file in the +program has it, the matchers are typed for **every** file in that program. So a package +can be green with no `types` entry and no import of its own. + +| package | `types` names jest-dom | files using matchers | files importing it | **files typed only by leakage** | +| --- | --- | --- | --- | --- | +| `plugin-detail` | no (`["node"]`) | 55 | 24 | **40** | +| `plugin-dashboard` | no (`["node"]`) | 37 | 22 | **16** | +| `fields` | no (*unset*) | 86 | 93 | **15** | +| `plugin-list` | no (`["node"]`) | 22 | 14 | **12** | + +**83 files** across four packages compile today because a *different* file imports +jest-dom. Deleting the last importing file from `plugin-detail` turns 40 files `TS2339` in +one commit, and nothing in the package names the dependency. + +### Route 4 — Node globals from a `/// ` + +`sdui-parser` and `react-runtime` both leave `types` **unset**. Measured with a probe file +(`const _x: string = process.env.NODE_ENV ?? '';`) compiled by each project: + +``` +react-runtime TS2591: Cannot find name 'process'. +sdui-parser (clean) +``` + +`--explainFiles` names the cause: + +``` +node_modules/.../@types/node/index.d.ts + Type library referenced via 'node' from file + 'packages/sdui-parser/src/__tests__/dashboard-widget-options-census.test.ts' +``` + +One triple-slash directive on line 1 of one test file types Node globals for the whole +package. **`sdui-parser` is the only one of the 38 whose Node types arrive this way.** +Probed across all 11 `types`-unset projects plus two `["node"]` controls: 10 have no Node +globals, `sdui-parser` has them, both controls have them (control lit). + +--- + +## Already paid — LIVE vs LATENT + +A divergence matters only where code depends on it. Both known axes are **LATENT for +compilation** (all 38 programs are green) and **LIVE as workaround cost** — which is the +expensive kind, because the cost is invisible in CI. + +| axis | status | evidence | +| --- | --- | --- | +| jest-dom in `plugin-timeline` | **LATENT** — 0 of its 26 program files use a matcher | but **paid**: `ObjectTimeline.expandFls-7429.test.tsx:147` uses `expect(...).toBeTruthy()` with a comment, from PR #8713, *because* the matcher would not type-check | +| `lib` ES2020 | **LATENT** — of the 20 `.at(` matches inside ES2020 programs, 6 are comments and the other 14 sit on `any`-typed receivers (`makeAdapter()` returns `as Record`), so none is a real `TS2550` | but **paid**: **5 comment sites in `plugin-designer`** documenting index arithmetic written instead of `.at(-1)` | +| ambient `.d.ts` in `include` | **LATENT and deliberate** | see below | + +**Control leg for the `lib` reading** — a zero-hit sweep is evidence only if a lit control +fires. A probe file added to `plugin-timeline`'s program with a genuinely typed array: + +``` +packages/plugin-timeline/src/zzprobe.test.ts(5,15): error TS2550: Property 'at' does not + exist on type 'number[]'. … Try changing the 'lib' compiler option to 'es2022' or later. +packages/plugin-timeline/src/zzprobe.test.ts(9,16): error TS2339: Property + 'toBeInTheDocument' does not exist on type 'Assertion'. +``` + +Both fire. So the sweep's zeros are readings, not blindness — and the card's `TS2339` +claim reproduces exactly. + +⚠️ **One of the paid workarounds encodes a false belief.** +`plugin-designer/src/__tests__/DashboardEditor.i18nTitle.test.tsx` states: + +> `.at()` type-checks **nowhere in this repo** even though every runtime it ships on has it. + +**6 of the 38 programs are on ES2022**, where it does. That comment is what the absence of +a census costs: a developer generalised correctly from their own package and wrote the +generalisation into the tree as documentation. + +--- + +## Deliberate or accidental — and why you cannot tell by reading + +| divergence | verdict | reason | +| --- | --- | --- | +| `permissions` omits jest-dom | **deliberate** | its config says so: *"nothing here needs the jest-dom matchers, which these suites do not use"* | +| `app-shell` raises `lib` to ES2022 | **deliberate** | its config says so, and says why it was raised there and not in the package build config | +| `plugin-map` includes `src/**/*.d.ts` | **deliberate** | its config explains the `TS2882` on a CSS side-effect import | +| `plugin-view` / `plugin-report` **exclude** their `src/global.d.ts` | **deliberate — but undocumented, and only discoverable by measurement** | adding `src/**/*.d.ts` to `plugin-view`'s `include` (the shape 6 siblings carry) turns it **RED with 7 errors**: its `global.d.ts` declares `const process`, which collides with `@types/node`'s. The exclusion is load-bearing; nothing says so | +| `auth`, `plugin-charts`, `plugin-grid` include `src/**/*.d.ts` | **accidental (inert)** | those patterns match **zero files on disk**. Copied along with the config | +| `composite: false` / `declaration: false` present in some, absent in others | **accidental (inert)** | identical program either way | +| `plugin-timeline` omits jest-dom | **inconclusive — and that is the finding** | it does not name `types` at all, so it is not "the sibling that forgot a line"; it is one of 11 packages using a different mechanism | +| `sdui-parser` gets Node types from a triple-slash | **accidental** | no sibling does it; the directive is on line 1 of one file and nothing points at it | + +⭐ **The generalisable result**: three of these eight needed a `tsc` run to classify. **You +cannot read deliberate-vs-accidental off the config text.** That is the strongest argument +for an instrument and the strongest argument against normalising by hand. + +--- + +## The one-line fix, and why it was not made + +objectui#8714 authorised adding `@testing-library/jest-dom` to `plugin-timeline` **only +if** the census showed the omission accidental — *"its siblings uniformly name jest-dom +and nothing explains the exception."* + +**The condition fails.** 21 of 35 cohort-A siblings do not name it; `permissions` +documents its omission on purpose; `fields` uses matchers in 86 files with `types` unset +entirely. There is no uniform sibling practice for `plugin-timeline` to have deviated +from — it is a data point in the matrix, not a bug. **No `tsconfig` was edited.** + +Measured, for whoever rules on it (each leg run against a probe file using +`process.env`, restored after): + +| plugin-timeline `types` | result | +| --- | --- | +| *unset* (today) | jest-dom matchers `TS2339`; `process` `TS2591` | +| `["@testing-library/jest-dom"]` — the literal one line | matchers type-check; `process` still `TS2591` | +| `["node", "@testing-library/jest-dom"]` — the shape 7 siblings use | matchers type-check; `process` resolves; project green | + +If triage rules "fix the instance", the third row is the shape to use, not the first. + +--- + +## Proposed gate — ⛔ NOT implemented here + +If triage wants one, this is what it should assert. Implementing it is a separate card. + +**It should not assert uniformity.** Four of the eight rows above are deliberate, and a +gate that flattens them is the repo-wide behaviour change this audit refused to make. +What it can assert without ruling anything: + +1. **Every divergence is declared.** A `tsconfig.test.json` whose effective value on a + listed axis (`lib`, `types`, `include`, `exclude`) differs from the cohort's modal + value must carry a comment on that key. `permissions`, `app-shell`, `plugin-map` + already pass; `plugin-view`'s load-bearing `d.ts` exclusion and `i18n` / `plugin-form` + / `types` dropping `DOM.Iterable` do not. + **Would have caught**: this card, objectui#8691, and the false comment in + `plugin-designer`. +2. **No file's matchers may be typed only by another file's import.** The `leaning` + column must be 0 — fixed either by the importing file's package naming jest-dom in + `types`, or by each file importing it. **Would have caught**: 83 files across 4 + packages, today. +3. **Inert declarations fail.** An `include` pattern matching zero files, or an option + restating its own default, is deleted. **Would have caught**: 3 dead `d.ts` patterns, + 35 redundant `composite`/`declaration` lines. +4. **Ratchet, not big bang.** Every rule above has live violations; each needs a declared + allowlist that can only shrink, in the shape `check-type-check-coverage.mjs` already + uses. + +--- + +## Method — re-runnable + +```bash +pnpm build # PRECONDITION: unbuilt => TS2307 everywhere +pnpm census:tsconfig-test-parity # the matrix +pnpm census:tsconfig-test-parity -- --json # machine-readable + +# baseline: every test project compiles today +for f in $(git ls-files '*tsconfig.test.json'); do + ./node_modules/.bin/tsc -p "$f" > /tmp/b.log 2>&1 + printf '%-46s exit=%s errors=%s\n' "$f" "$?" "$(grep -c 'error TS' /tmp/b.log)" +done +``` + +## What this audit does NOT claim + +- **Not** that any of the 23 axes is wrong. It reports; triage rules. +- **Not** that the 38 are all the test type-programs — 4 more are fused into build + configs (see *Population*), and `scripts/__tests__/*.test.ts` are compiled by a 43rd + program, `tsconfig.scripts.json` (`lib: ES2022`, `types: ["node"]`). +- **Not** anything about `objectui#8710`'s module-resolution axis: that divergence is + between Vite's alias table and the `exports` map, not between two `tsconfig`s, so this + instrument cannot see it. `moduleResolution` is uniformly `bundler` across all 38. +- **Not** that a zero in the leakage column means a package is safe — it means no file + there uses a matcher without its own import *today*. diff --git a/package.json b/package.json index c5a4732083..bf9bcad980 100644 --- a/package.json +++ b/package.json @@ -51,6 +51,7 @@ "check:esm-specifiers": "node scripts/check-node-esm-load.mjs --specifiers-only", "check:node-esm-load": "node scripts/check-node-esm-load.mjs", "census:body-dialect": "node scripts/body-dialect-census.mjs", + "census:tsconfig-test-parity": "node scripts/tsconfig-test-parity-census.mjs", "check:control-bytes": "node scripts/check-control-bytes.mjs", "check:action-ref-convention": "node scripts/check-action-ref-convention.mjs", "check:published-dist": "node scripts/check-published-dist-tooling.mjs", diff --git a/scripts/__tests__/tsconfig-test-parity-census.test.ts b/scripts/__tests__/tsconfig-test-parity-census.test.ts new file mode 100644 index 0000000000..bb3beadb93 --- /dev/null +++ b/scripts/__tests__/tsconfig-test-parity-census.test.ts @@ -0,0 +1,166 @@ +/** + * ObjectUI + * Copyright (c) 2024-present ObjectStack Inc. + * + * This source code is licensed under the MIT license found in the + * LICENSE file in the root directory of this source tree. + */ + +/** + * Pins for `scripts/tsconfig-test-parity-census.mjs` (objectui#8714). + * + * The card this census answers exists because three divergences were each + * found by a developer tripping over them. The failure mode of the CURE is + * the same shape as the disease: a census that reads zero because its + * discovery glob is wrong is indistinguishable from a repo whose test + * type-programs agree, and it prints that lie with exit 0. + * + * So the population is asserted FIRST and the emptiness leg is OBSERVED here + * rather than reasoned about: `main` is run against a tree with no + * `tsconfig.test.json` in it and must return non-zero. Without that leg every + * other assertion in this file would still pass over an empty set. + * + * The rest pins the three readings a plain `diff` of the config text cannot + * make, each of which was measured on this tree before it was encoded here: + * spellings that are the same program, the jest-dom augmentation leaking + * program-wide from one file's import, and Node globals arriving through a + * `/// ` instead of `compilerOptions.types`. + */ + +import { describe, it, expect } from 'vitest'; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { + discoverTestProjects, + buildMatrix, + jestDomRoute, + nodeTypesRoute, + main, + FALSE_BY_DEFAULT, + NOT_AN_AXIS, + DEFAULT_ROOT, +} from '../tsconfig-test-parity-census.mjs'; + +function tree(files: Record): string { + const root = mkdtempSync(join(tmpdir(), 'parity-census-')); + for (const [rel, body] of Object.entries(files)) { + const abs = join(root, rel); + mkdirSync(join(abs, '..'), { recursive: true }); + writeFileSync(abs, body); + } + return root; +} + +describe('the population is real before anything is said about it', () => { + it('finds the workspace test projects, and names two the card turns on', () => { + const found = discoverTestProjects(DEFAULT_ROOT); + // A floor, not the exact count: packages are added often and this test is + // not the place to relitigate that. What it refuses is a collapse. + expect(found.length).toBeGreaterThan(20); + expect(found).toContain('packages/plugin-map/tsconfig.test.json'); + expect(found).toContain('packages/plugin-timeline/tsconfig.test.json'); + }); + + it('FAILS on an empty population instead of reporting a uniform repo', () => { + // The whole reason this file exists. Every other assertion here passes + // vacuously over an empty set; this is the one that does not. + const empty = tree({ 'packages/only/tsconfig.json': '{}' }); + try { + expect(discoverTestProjects(empty)).toEqual([]); + expect(main(['--root', empty])).toBe(1); + } finally { + rmSync(empty, { recursive: true, force: true }); + } + }); +}); + +describe('two spellings of the same program are not a divergence', () => { + const project = (pkg: string, options: Record) => ({ + project: `packages/${pkg}/tsconfig.test.json`, + package: `packages/${pkg}`, + options, + include: ['src/**/*.test.ts'], + exclude: undefined, + fileNames: [], + }); + + it('reads an omitted `composite` and an explicit `false` as one program', () => { + expect(FALSE_BY_DEFAULT.has('composite')).toBe(true); + const [axis] = buildMatrix([project('a', { composite: false }), project('b', {})]); + expect(axis.axis).toBe('composite'); + expect(axis.spellings).toBe(2); + expect(axis.effective).toBe(1); + }); + + it('reads `types` as a SET — sibling packages spell the same pair both ways', () => { + const axes = buildMatrix([ + project('a', { types: ['node', '@testing-library/jest-dom'] }), + project('b', { types: ['@testing-library/jest-dom', 'node'] }), + ]); + expect(axes.find((a) => a.axis === 'types')).toMatchObject({ spellings: 2, effective: 1 }); + }); + + it('does not report per-project paths as axes — they differ by construction', () => { + expect(NOT_AN_AXIS.has('configFilePath')).toBe(true); + const axes = buildMatrix([ + project('a', { configFilePath: '/x/a', lib: ['lib.es2020.d.ts'] }), + project('b', { configFilePath: '/x/b', lib: ['lib.es2022.d.ts'] }), + ]); + expect(axes.map((a) => a.axis)).toEqual(['lib']); + }); +}); + +describe('the routes that are not in the tsconfig text', () => { + it('counts files whose matchers are typed only by ANOTHER file s import', () => { + const root = tree({ + 'a.test.ts': "import '@testing-library/jest-dom';\nexpect(el).toBeInTheDocument();\n", + // No import of its own. It compiles today only because `a.test.ts` is in + // the same program — delete that file and this one turns TS2339. + 'b.test.ts': 'expect(el).toHaveTextContent("x");\n', + }); + try { + const p = { options: {}, fileNames: ['a.test.ts', 'b.test.ts'] }; + expect(jestDomRoute(root, p)).toMatchObject({ + route: 'side-effect import', + matcherFiles: 2, + importFiles: 1, + matcherWithoutOwnImport: 1, + }); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + it('does not count leaning when `compilerOptions.types` names jest-dom', () => { + const root = tree({ 'a.test.ts': 'expect(el).toBeInTheDocument();\n' }); + try { + const p = { options: { types: ['@testing-library/jest-dom'] }, fileNames: ['a.test.ts'] }; + expect(jestDomRoute(root, p)).toMatchObject({ + route: 'compilerOptions.types', + matcherWithoutOwnImport: 0, + }); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + it('sees Node globals arriving through a triple-slash directive', () => { + const root = tree({ + 'a.test.ts': '/// \nconst x = process.env.HOME;\n', + 'b.test.ts': 'const y = 1;\n', + }); + try { + const p = { options: {}, fileNames: ['a.test.ts', 'b.test.ts'] }; + expect(nodeTypesRoute(root, p)).toMatchObject({ + route: 'triple-slash reference in one file', + directiveFiles: ['a.test.ts'], + }); + const named = { options: { types: ['node'] }, fileNames: ['b.test.ts'] }; + expect(nodeTypesRoute(root, named).route).toBe('compilerOptions.types'); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/scripts/tsconfig-test-parity-census.mjs b/scripts/tsconfig-test-parity-census.mjs new file mode 100644 index 0000000000..f68b1f881e --- /dev/null +++ b/scripts/tsconfig-test-parity-census.mjs @@ -0,0 +1,264 @@ +#!/usr/bin/env node +/** + * ObjectUI + * Copyright (c) 2024-present ObjectStack Inc. + * + * This source code is licensed under the MIT license found in the + * LICENSE file in the root directory of this source tree. + */ + +/** + * Test-`tsconfig` parity census (objectui#8714). + * + * ## What this answers + * + * Every workspace package hand-maintains its own `tsconfig.test.json`, and + * until this script NOTHING compared them. Three divergences had been found, + * each by a developer tripping over it rather than by any instrument: + * + * - objectui#8691 — `lib` level (`packages/permissions` inherits `ES2020`, + * where `.at(-1)` is TS2550; `packages/app-shell` sets `ES2022`). + * - objectui#8710 — module resolution (Vite alias vs the `exports` map). + * - objectui#8714 — ambient types (`plugin-map` names + * `@testing-library/jest-dom`, `plugin-timeline` does not, so a matcher + * there is green under vitest and TS2339 under `tsc`). + * + * The shared failure shape is GREEN UNDER ONE TOOL, RED UNDER ANOTHER, so the + * author's evidence and CI's disagree and nothing in the package says why. + * This prints the whole matrix at once so the fourth axis is read off an + * instrument instead of being discovered next month by the same accident. + * + * ## This is a REPORT, not a gate + * + * It asserts nothing about which values are correct and it never fails on a + * divergence — some of them are deliberate and load-bearing (measured: adding + * `src/**\/*.d.ts` to `plugin-view`'s `include`, which six siblings do carry, + * turns that project RED with 7 errors, because its own `src/global.d.ts` + * redeclares `process`). Deciding which rows are defects is triage's, and a + * gate is a separate card. The one thing this DOES refuse is a silent zero: + * an empty population exits non-zero, because a census that reads nothing + * because it is blind is indistinguishable from a repo that is uniform. + * + * ## Why the tsconfig files are not the whole answer + * + * Two of the routes that decide a test program's ambient types are not in any + * tsconfig, and both were measured on this tree: + * + * - A bare `import '@testing-library/jest-dom'` registers the matcher + * augmentation PROGRAM-WIDE, so a file with no import of its own compiles + * because a DIFFERENT file in the same package has one. Four packages lean + * on that today for 54 files between them. + * - A `/// ` in one test file does the same for + * Node globals: `sdui-parser` and `react-runtime` both leave `types` + * unset, and `process` resolves in one and is TS2591 in the other. + * + * So the census reads those routes off the resolved program files, not off the + * config text. `scripts/__tests__/tsconfig-test-parity-census.test.ts` pins + * the instrument against throwaway trees, including the empty-population case. + * + * ## Boundary — 38 is not the number of test type-programs + * + * This reads `tsconfig.test.json` files. Four packages have no such file and + * compile their tests inside their ORDINARY `tsconfig.json` instead + * (`packages/cli`, `packages/data-objectstack`, `packages/test-support`, + * `apps/console`), so the workspace has 42 test type-programs and this census + * sees 38 of them. `scripts/check-type-check-coverage.mjs` is the instrument + * that asks whether every package's tests are compiled AT ALL; this one asks + * whether the ones with a dedicated project agree with each other. Reading 38 + * as "every test program" is the mistake this paragraph exists to prevent. + * + * Run: node scripts/tsconfig-test-parity-census.mjs [--json] [--root DIR] + */ + +import { readFileSync, existsSync } from 'node:fs'; +import { join, dirname, relative, sep } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { globSync } from 'node:fs'; +import ts from 'typescript'; +import { isEntrypoint } from './invoked-as.mjs'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +export const DEFAULT_ROOT = join(HERE, '..'); + +/** + * Options whose absence is the same PROGRAM as an explicit `false`, so the two + * spellings must not be reported as a divergence. Every entry is a TypeScript + * default that does not depend on any other option (`useDefineForClassFields` + * is deliberately absent — its default follows `target`). + */ +export const FALSE_BY_DEFAULT = new Set([ + 'composite', + 'declaration', + 'declarationMap', + 'sourceMap', + 'allowJs', + 'checkJs', +]); + +/** + * Options TypeScript injects or resolves to an absolute path. They differ per + * project BY CONSTRUCTION — reporting them would bury the four real axes under + * 38 rows of "this file is at a different path than that file". + */ +export const NOT_AN_AXIS = new Set(['configFilePath', 'pathsBasePath', 'baseUrl', 'outDir', 'rootDir', 'typeRoots']); + +const JEST_DOM_MATCHER = + /\.(toBeInTheDocument|toHaveTextContent|toHaveClass|toHaveAttribute|toBeVisible|toBeDisabled|toBeEnabled|toBeChecked|toHaveValue|toHaveFocus|toBeEmptyDOMElement|toContainElement|toContainHTML|toHaveStyle|toBeRequired|toBeInvalid|toBeValid|toHaveAccessibleName|toHaveAccessibleDescription|toHaveDisplayValue|toBePartiallyChecked|toHaveErrorMessage|toHaveRole|toHaveAccessibleErrorMessage|toHaveSelection)\(/; +const JEST_DOM_IMPORT = /import\s+['"]@testing-library\/jest-dom(\/vitest)?['"]/; +const NODE_TRIPLE_SLASH = /\/\/\/\s*/; + +/** + * Every `tsconfig.test.json` under the workspace, repo-relative and sorted. + * + * Discovered from disk rather than listed, for the reason + * `check-type-check-coverage.mjs` records: a list is a thing to forget, and a + * package that lands outside the census reads exactly like a package that + * agrees with its siblings. + */ +export function discoverTestProjects(root = DEFAULT_ROOT) { + return globSync(['packages/*/tsconfig.test.json', 'apps/*/tsconfig.test.json', 'examples/*/tsconfig.test.json'], { + cwd: root, + }) + .map((p) => p.split(sep).join('/')) + .sort(); +} + +/** Flatten one project's `extends` chain through TypeScript's own resolver. */ +export function resolveProject(root, relPath) { + const abs = join(root, relPath); + const host = { + ...ts.sys, + onUnRecoverableConfigFileDiagnostic: (d) => { + throw new Error(ts.flattenDiagnosticMessageText(d.messageText, ' ')); + }, + }; + const parsed = ts.getParsedCommandLineOfConfigFile(abs, {}, host); + if (!parsed) throw new Error(`could not parse ${relPath}`); + return { + project: relPath, + package: dirname(relPath), + options: parsed.options, + include: parsed.raw?.include, + exclude: parsed.raw?.exclude, + fileNames: parsed.fileNames.map((f) => relative(root, f).split(sep).join('/')), + }; +} + +/** + * How this program gets jest-dom's matcher types, and how many of its files + * would lose them if the last importing file went away. + */ +export function jestDomRoute(root, project) { + const named = (project.options.types ?? []).includes('@testing-library/jest-dom'); + let matcherFiles = 0; + let importFiles = 0; + let matcherWithoutOwnImport = 0; + for (const f of project.fileNames) { + const p = join(root, f); + if (!existsSync(p)) continue; + const text = readFileSync(p, 'utf8'); + const usesMatcher = JEST_DOM_MATCHER.test(text); + const imports = JEST_DOM_IMPORT.test(text); + if (usesMatcher) matcherFiles += 1; + if (imports) importFiles += 1; + if (usesMatcher && !imports && !named) matcherWithoutOwnImport += 1; + } + const route = named ? 'compilerOptions.types' : importFiles > 0 ? 'side-effect import' : 'none'; + return { route, named, matcherFiles, importFiles, matcherWithoutOwnImport }; +} + +/** How this program gets Node globals: the `types` field, or one file's directive. */ +export function nodeTypesRoute(root, project) { + if ((project.options.types ?? []).includes('node')) return { route: 'compilerOptions.types', directiveFiles: [] }; + const directiveFiles = project.fileNames.filter((f) => { + const p = join(root, f); + return existsSync(p) && NODE_TRIPLE_SLASH.test(readFileSync(p, 'utf8')); + }); + if (project.options.types === undefined && directiveFiles.length > 0) { + return { route: 'triple-slash reference in one file', directiveFiles }; + } + return { route: 'none', directiveFiles }; +} + +const spell = (v) => (v === undefined ? '' : JSON.stringify(v)); + +/** Group projects by the value they give each axis; flag same-program spellings. */ +export function buildMatrix(projects) { + const keys = new Set(); + for (const p of projects) for (const k of Object.keys(p.options)) if (!NOT_AN_AXIS.has(k)) keys.add(k); + const axes = []; + const read = (p, k) => (k === 'include' || k === 'exclude' ? p[k] : p.options[k]); + for (const k of [...keys, 'include', 'exclude'].sort()) { + const byValue = new Map(); + for (const p of projects) { + const v = spell(read(p, k)); + if (!byValue.has(v)) byValue.set(v, []); + byValue.get(v).push(p.package); + } + if (byValue.size < 2) continue; + const effective = new Set( + [...byValue.keys()].map((v) => { + if (v === '') return FALSE_BY_DEFAULT.has(k) ? 'false' : v; + const parsed = JSON.parse(v); + return Array.isArray(parsed) ? JSON.stringify([...parsed].map(String).sort()) : v; + }), + ); + axes.push({ axis: k, spellings: byValue.size, effective: effective.size, byValue: [...byValue] }); + } + return axes.sort((a, b) => b.effective - a.effective || a.axis.localeCompare(b.axis)); +} + +/** CLI entry. Exported so the emptiness leg can be OBSERVED failing in a test. */ +export function main(argv) { + const rootFlag = argv.indexOf('--root'); + const root = rootFlag === -1 ? DEFAULT_ROOT : argv[rootFlag + 1]; + const files = discoverTestProjects(root); + if (files.length === 0) { + console.error( + 'tsconfig-test-parity-census: found NO tsconfig.test.json under packages/*, apps/* or examples/*.\n' + + 'That is a blind instrument, not a uniform repo — check --root and the workspace layout.', + ); + return 1; + } + const projects = files.map((f) => resolveProject(root, f)); + const axes = buildMatrix(projects); + const rows = projects.map((p) => ({ + package: p.package, + programFiles: p.fileNames.length, + lib: p.options.lib, + types: p.options.types, + jestDom: jestDomRoute(root, p), + node: nodeTypesRoute(root, p), + })); + + if (argv.includes('--json')) { + console.log(JSON.stringify({ population: files.length, axes, projects: rows }, null, 2)); + return 0; + } + + console.log(`# test-tsconfig parity census — ${files.length} projects, ${projects.reduce((a, p) => a + p.fileNames.length, 0)} program files\n`); + console.log(`## divergent axes (${axes.length})\n`); + for (const a of axes) { + const note = a.effective === 1 ? ' [same program, different spelling]' : ''; + console.log(`### ${a.axis} — ${a.spellings} spellings, ${a.effective} effective${note}`); + for (const [value, pkgs] of a.byValue.sort((x, y) => y[1].length - x[1].length)) { + console.log(` ${value} x${pkgs.length}`); + console.log(` ${pkgs.join(' ')}`); + } + console.log(''); + } + console.log('## ambient-type route per project (not visible in the tsconfig text)\n'); + console.log( + `${'package'.padEnd(32)}${'files'.padStart(6)} ${'jest-dom route'.padEnd(22)}${'leaning'.padStart(8)} node route`, + ); + for (const r of rows) { + console.log( + `${r.package.padEnd(32)}${String(r.programFiles).padStart(6)} ${r.jestDom.route.padEnd(22)}${String(r.jestDom.matcherWithoutOwnImport).padStart(8)} ${r.node.route}`, + ); + } + return 0; +} + +if (isEntrypoint(import.meta.url)) { + process.exit(main(process.argv.slice(2))); +}