feat(spec)!: publish the $-prefix key ban the normalized filter enforces, and make the ratchet able to see it

.changeset/18670-project-operator-key-pattern.md

@@ -0,0 +1,33 @@
+---
+"@objectstack/spec": minor
+---
+
+**BREAKING (published artifact narrows)** — `packages/spec/json-schema/**` now states the `$`-prefix key ban a normalized field condition enforces, so a validator reading the published files stops answering PASS on `{"$and":[{"$bogus":{"$eq":1}}]}` at `data/NormalizedFilter` — a document the runtime refuses by name (#18670 item 2, the fifth arm).
+
+Clause-②: yes (narrowing)
+
+One named pattern joins the closed list, and only one:
+
+- **`banned-key-pattern` — "no document may carry a key matching this pattern"**, emitted as `propertyNames` with a `not` over a `pattern`. `NormalizedFilter`'s `$and` / `$or` members and its `$not` operand each admit a field condition whose keys are field names (`amount`, `account.name`) and never `$`-prefixed operators. The published file now says so at all three nodes.
+
+**Scoped, and the scope is mechanical.** The ban is over an OPEN set of names, which is why the existing `banned-keys` arm cannot express it — a finite list that merely sampled the set would be wider than the rule. The pattern arm that can express it is bounded by a second closed list: `BannedKeyPattern` is a union of the pattern strings this package publishes, exactly one today (`^\$`), so a call site cannot invent a regex because there is no `string` to pass, and widening it is the same reviewed decision that adding an arm is. That is what answers the standing objection to a regex-shaped declaration — its over-reach cannot be read off the declaration the way a key list's can, so the bound is on how few declarations exist rather than on trusting the next caller.
+
+**The rows retired, by name.** `packages/spec/dropped-refinements.baseline.json`, entry `data/NormalizedFilter`:
+
+| row | before | after |
+|:---|:---|:---|
+| `lazy.$and.element.options[0]` | dropped | **deleted** — reads `projected`, arm `banned-key-pattern` |
+| `lazy.$or.element.options[0]` | dropped | **deleted** — reads `projected`, arm `banned-key-pattern` |
+| `lazy.$not.options[0]` | dropped | **deleted** — reads `projected`, arm `banned-key-pattern` |
+
+**⛔ Not a behaviour change, and no document the runtime accepts becomes refused.** The arm is EXACT rather than approximate. A JSON object's properties are exactly its own enumerable string-keyed ones and `propertyNames` judges exactly those names; JSON Schema specifies `pattern` as an ECMA-262 regular expression evaluated as a SEARCH, which is `RegExp.prototype.test` and nothing else — so the same source text decides the same set of names on both sides. It is presence and never value: a matching key present with a `null` value is present to both. Measured with ajv 8 (draft 2020-12) on the generated file, the verdict vector moves in one direction only: the three `$`-prefixed specimens go `true` → `false`, and every document the runtime accepts — `{}`, the empty combinators `{"$and":[]}` / `{"$or":[{}]}` / `{"$not":{}}`, a nested group, an ordinary field condition — is accepted before and after. Across the published tree, **1530 of 1535 files are byte-identical**; one file changes what it accepts, two change annotation only, and the remaining two are the bundle and the build-input hash.
+
+**The predicate and the keyword are ONE string.** `bannedKeyPattern` compiles its `RegExp` from the declared pattern, so the keyword the file publishes and the rule the runtime enforces cannot come to mean different things — the construction `requiredOneOf`, `dependentRequired` and `bannedKeys` already use, and the reason this arm needs no drift pin either. The `RegExp` carries no flags, which is part of the equality rather than a style choice: a JSON Schema `pattern` has none to carry, and `g` would make `test` stateful through `lastIndex` so a key's verdict would depend on which keys were judged before it.
+
+**A ratchet repair ships with it, and it is what made the rows exist to delete.** The detector decided `dropped` vs `projected` on a two-rung projection ladder while the generator publishes on a three-rung one — a node whose every io direction refuses over an unrepresentable member still reaches its file when that member sits in a union position, because the emit loop drops the branch and publishes the rest. Nine PUBLISHED sites therefore read `undecidable`, the one verdict the ledger does not count: they held no row, carried no `x-dropped-refinements`, and no repair of them could ever have deleted a row. The three nodes this arm closes were three of the nine. The detector now carries the generator's third rung and reports which rung answered, so a differential can never compare a pruned projection with an unpruned one; and a published site that still cannot be adjudicated fails the build by name, so the blind spot cannot reopen in silence.
+
+⚠️ **The ledger therefore GREW before it shrank, and the growth is the point.** Six sites became visible that were previously uncounted — `data/FieldOperators` and `data/RangeOperator` gained `$between.items[0]` / `[1]`, `data/NormalizedFilter` gained the same pair under `$not`, and `data/RangeOperator` entered the ledger as a published schema that had been holding no entry at all — then this arm deleted three. Net across the change: **204 entries / 560 sites → 205 / 566**, with the census at **566 dropped / 205 published schemas / 360 projected** (224 `non-blank-string`, 129 `required-one-of`, 3 `banned-key-pattern`, 2 `dependent-required`, 2 `banned-keys`) and **0 undecidable**, down from 9. Those two files gain annotation only: `x-` keywords are ignored by every validator, so the set of documents they accept is unchanged.
+
+⭐ **Superseding a sibling entry in this same release.** `18670-project-banned-keys.md` records that the `$`-prefix sites 「stay unprojected … they carry NO annotation and hold NO ledger row: published yet unratcheted」. That was a correct reading of its own tree and is no longer true of this one: the sites are projected, the blind spot is closed, and the population it described is empty. The earlier entry is left as the record of what it landed.
+
+<!-- adr-0087: not-required (no-migration-prescription) Nothing an author can write is removed, renamed or re-spelled: no spec key, no export and no config field changes, and the accepted set of metadata documents is byte-for-byte what it was. What changed is a machine-readable DECLARATION catching up with the runtime it always described, so there is nothing for `objectstack migrate meta` to rewrite and no stored representation to convert. -->

packages/spec/dropped-refinements.baseline.json

@@ -2,10 +2,10 @@
   "description": "Shrink-only ledger of every PUBLISHED JSON Schema that is STILL WIDER than the Zod type it was generated from, because a rule written as `.refine()` reaches the runtime and not the file (#18670). `z.toJSONSchema()` has no arm for a `custom` check: a plain record, the same record with a `.refine()`, and the same record with an ABORTING `.refine()` all project byte-identically (measured on zod 4.4.3, the version packages/spec resolves). So a document one of these files ACCEPTS can still be refused at parse time, and an author -- or an AI -- validating against packages/spec/json-schema/** finds out a release later. Each `sites` path is a position under that schema at which a refinement is dropped; the same paths are written onto the artifact itself as `x-dropped-refinements`. Item 2 closed the first patterns: a refinement DECLARED through the closed list in src/shared/refinement-projection.ts is emitted into the published file, reads `projected` rather than `dropped`, and its row LEAVES this ledger in the same PR -- which is why the ledger shrinks and never grows on a repair. Every refinement outside that closed list stays here, and adding an arm to the list is a public-contract decision, not a refactor. Hand-edited on purpose and with no `gen:` script: a generator would let a new gap be admitted by running a command instead of by a decision, which is the silence this ledger exists to end. Adding, removing or moving a site fails packages/spec/scripts/build-schemas.ts until the line moves with it, and the failure prints the corrected entry in full. ⛔ Do not delete or weaken a refinement to shorten this file -- the runtime rule is correct; it is the projection that is silent, and the remedy is to teach the closed list a NAMED pattern, never to drop the rule.",
   "measured": {
     "zod": "4.4.3",
-    "publishedSchemasWithDroppedRefinements": 205,
-    "droppedRefinementSites": 565,
-    "refinementSitesThatDidProject": 366,
-    "refinementSitesWithNoJsonFormToCompare": 9
+    "publishedSchemasWithDroppedRefinements": 206,
+    "droppedRefinementSites": 571,
+    "refinementSitesThatDidProject": 369,
+    "refinementSitesWithNoJsonFormToCompare": 0
   },
   "entries": {
     "ai/BlueprintField": {
@@ -614,6 +614,8 @@
     },
     "data/FieldOperators": {
       "sites": [
+        "$between.items[0]",
+        "$between.items[1]",
         "$in",
         "$nin"
       ]
@@ -677,6 +679,8 @@
     },
     "data/NormalizedFilter": {
       "sites": [
+        "lazy.$not.options[0].valueType.$between.items[0]",
+        "lazy.$not.options[0].valueType.$between.items[1]",
         "lazy.$not.options[0].valueType.$in",
         "lazy.$not.options[0].valueType.$nin"
       ]
@@ -745,6 +749,12 @@
         "in.where.options[0].lazy.options[2].items[0]"
       ]
     },
+    "data/RangeOperator": {
+      "sites": [
+        "$between.items[0]",
+        "$between.items[1]"
+      ]
+    },
     "data/ReferenceIdValue": {
       "sites": [
         ""

packages/spec/scripts/build-schemas.ts

@@ -3582,6 +3582,58 @@ if (hasDroppedRefinementProblems(droppedRefinementProblems)) {
   process.exit(1);
 }
 
+// ⭐ #18670 item 2 — the ratchet's OWN blind spot, held at zero.
+//
+// A site reaches this census only from the successful-emit path above, so every
+// entry here describes a PUBLISHED file. An `undecidable` site is therefore a
+// node that is published and whose rule the comparison could not read: it holds
+// no ledger row, it is named in no `x-dropped-refinements`, and no repair of it
+// could ever delete a row. That is strictly worse than a declared gap — the gap
+// is real and the instrument built to count it cannot see it — so it is refused
+// rather than reported, on the same principle as every ratchet above: growth is
+// a reviewed line in a diff, never a number that moved.
+//
+// This population was NINE sites when the item was ruled, all of them nodes the
+// published file reaches through `projectByPruningUnionBranches` while the
+// detector's ladder stopped at the two strict rungs. Giving the detector the
+// generator's own third rung took it to ZERO and turned those nine into
+// ordinary `dropped` rows, which is the reading this check now pins.
+const undecidableEntries = refinementCensus.filter((entry) => entry.undecidable.length > 0);
+if (undecidableEntries.length > 0) {
+  const undecidableTotal = undecidableEntries.reduce((sum, entry) => sum + entry.undecidable.length, 0);
+  console.error(
+    `\n❌ ${undecidableTotal} PUBLISHED refinement site(s) across ${undecidableEntries.length} schema(s) ` +
+      `carry a rule this build could not adjudicate (#18670 item 2):`,
+  );
+  for (const entry of undecidableEntries) {
+    console.error(`     + ${entry.defKey}  (${entry.undecidable.length} site(s))`);
+    for (const site of entry.undecidable) {
+      console.error(`         ${site.path === '' ? '<the export itself>' : site.path}  (${site.nodeType})`);
+    }
+  }
+  console.error(
+    `\n   These files ARE published, so the rule is enforced by the runtime and the file says\n` +
+      `   nothing about it — the same gap ${DROPPED_REFINEMENTS_BASELINE_FILE} exists to hold. What is\n` +
+      `   different, and worse, is that an undecidable site holds NO ledger row: it is invisible to\n` +
+      `   the ratchet, absent from \`x-dropped-refinements\`, and no repair of it can delete a row.\n` +
+      `   ⛔ Leaving it uncounted is the one outcome this check exists to refuse.\n\n` +
+      `   Two remedies, in order of preference:\n\n` +
+      `     1. TEACH THE LADDER. \`projectOrNull\` in scripts/lib/dropped-refinements.ts must project\n` +
+      `        the way THIS generator does, or it answers a question about a file nobody publishes.\n` +
+      `        It carries three rungs — output, input, and the branch-pruning pass — because those\n` +
+      `        are the three attempts the emit loop above makes. If your export reaches its file by\n` +
+      `        a fourth route, the ladder owes that route too, and the site becomes an ordinary\n` +
+      `        \`dropped\` row you declare in the ledger.\n` +
+      `     2. GIVE THE LEDGER AN \`undecidable\` ROW SHAPE. If a site genuinely cannot be judged,\n` +
+      `        the population stops being empty and needs recording rather than refusing — an entry\n` +
+      `        shape of its own, ratcheted in both directions like \`sites\`. That is a decision about\n` +
+      `        what the ledger counts, ⛔ not a refactor: take it, don't assume it.\n\n` +
+      `   ⛔ Do not delete or weaken the refinement to clear this line, and ⛔ do not stop\n` +
+      `   publishing the export — the rule is correct and the file is wanted.`,
+  );
+  process.exit(1);
+}
+
 // The accepted population, reported in full on every run — the same discipline
 // as the never-published ledger above, and for the same reason: a population
 // that passes in silence is the silence this ratchet was built to end.
@@ -3602,8 +3654,9 @@ if (droppedSiteTotal > 0) {
       `     the refinement, and ⛔ never an open-ended translator over the whole population.`,
   );
   console.log(
-    `     Also measured this run: ${projectedSiteTotal} refinement site(s) DID reach the file, ` +
-      `${undecidableSiteTotal} had no JSON form on either side to compare.`,
+    `     Also measured this run: ${projectedSiteTotal} refinement site(s) DID reach the file, and ` +
+      `${undecidableSiteTotal} published site(s) could not be adjudicated — a population the check\n` +
+      `     above holds at zero, so this figure is a reading of it and not a tolerance.`,
   );
 }
 

packages/spec/scripts/lib/dropped-refinements.ts

@@ -94,6 +94,9 @@ import {
   projectableRefinementsOf,
   zodDefOf,
 } from './refinement-projection';
+// The generator's THIRD projection attempt. The detector owes the same ladder
+// for the same reason it owes the same `override` — see `projectOrNull`.
+import { projectByPruningUnionBranches } from './union-branch-projection';
 
 /** File name of the committed ledger, resolved against the package root. */
 export const DROPPED_REFINEMENTS_BASELINE_FILE = 'dropped-refinements.baseline.json';
@@ -134,8 +137,17 @@ export interface RefinementSite {
   /**
    * `dropped` — removing the refinements leaves the projection byte-identical.
    * `projected` — the projection changed, so the rule DID reach a reader.
-   * `undecidable` — the node has no JSON form in either io direction, so the
-   * comparison has no two sides. Reported, never counted as a gap.
+   * `undecidable` — no rung of the generator's own projection ladder gives this
+   * node a JSON form, so the comparison has no two sides.
+   *
+   * ⛔ `undecidable` is NOT a third way to be fine. A node can be undecidable
+   * here and PUBLISHED all the same — its export reaches the file through a
+   * route this comparison could not reproduce — and such a node states nothing
+   * about its rule while holding zero ledger rows, which is the ratchet's own
+   * blind spot rather than an absence of gap. Since #18670's second acceptance
+   * item the ladder carries the generator's third rung so far fewer nodes land
+   * here, and the ones that still do are counted: see
+   * {@link DroppedRefinementsEntry.undecidableSites}.
    */
   readonly verdict: 'dropped' | 'projected' | 'undecidable';
   /**
@@ -229,8 +241,15 @@ function withoutCustomChecks(schema: z.ZodType): z.ZodType | null {
   return stripped;
 }
 
+/** One side of the differential: which ladder rung answered, and what it said. */
+interface LadderReading {
+  /** `output` / `input` — a strict pass; `pruned` — the branch-dropping pass. */
+  readonly rung: 'output' | 'input' | 'pruned';
+  readonly text: string;
+}
+
 /**
- * `toJSONSchema` in the generator's own io ladder, or `null` when neither side
+ * `toJSONSchema` in the generator's own io ladder, or `null` when no rung of it
  * has a JSON form.
  *
  * ⭐ It projects through `projectPublishedJsonSchema` — the SAME call the
@@ -248,15 +267,27 @@ function withoutCustomChecks(schema: z.ZodType): z.ZodType | null {
  * `projected` behind a green ledger while the published file went wide in
  * silence. The helper's own docblock carries that measurement.
  */
-function projectOrNull(schema: z.ZodType): string | null {
+function projectOrNull(schema: z.ZodType): LadderReading | null {
   for (const io of ['output', 'input'] as const) {
     try {
-      return JSON.stringify(projectPublishedJsonSchema(schema, { io }));
+      return { rung: io, text: JSON.stringify(projectPublishedJsonSchema(schema, { io })) };
     } catch {
       // Try the other direction — the generator does the same, for the same reason.
     }
   }
-  return null;
+  // THIRD rung — the generator's own third attempt (#16431 (a)), owed here for
+  // the same reason the `override` is. A node whose every io direction refuses
+  // over one unrepresentable member still REACHES the published file when that
+  // member sits in a union position: `build-schemas.ts` drops the branch and
+  // publishes the rest. Stopping the ladder at two rungs therefore asked a
+  // different question than "what does the published file say about this rule"
+  // — it asked what a projection nobody publishes says — and answered
+  // `undecidable`, which is the one verdict the ledger does not count. Measured
+  // on `data/NormalizedFilter`: the three record nodes carrying the `$`-prefix
+  // ban published as bare objects, held ZERO ledger rows, and no repair of them
+  // could ever have deleted a row.
+  const pruned = projectByPruningUnionBranches(schema);
+  return pruned ? { rung: 'pruned', text: JSON.stringify(pruned.schema) } : null;
 }
 
 /** One node's raw differential and the verdict adjudicated from it. */
@@ -306,7 +337,16 @@ function readProjection(schema: z.ZodType): NodeProjectionReading {
   const before = projectOrNull(schema);
   const after = projectOrNull(stripped);
   if (before === null || after === null) return { verdict: 'undecidable', projectionMoved: false };
-  if (before === after) return { verdict: 'dropped', projectionMoved: false };
+  // ⛔ Two rungs are not two readings of one question. A differential whose
+  // sides were answered by different passes compares a pruned projection with
+  // an unpruned one, and the bytes then differ over the branches one side
+  // dropped — a `projected` verdict earned by the ladder rather than by the
+  // rule. Nothing observed reaches here (a `custom` check cannot change which
+  // types a subtree carries, so both sides refuse and survive alike), which is
+  // exactly why it is written down: the day it stops holding, this reads
+  // `undecidable` and is counted, instead of reading `projected` and vanishing.
+  if (before.rung !== after.rung) return { verdict: 'undecidable', projectionMoved: false };
+  if (before.text === after.text) return { verdict: 'dropped', projectionMoved: false };
   const stated = projectableRefinementsOf(schema).length;
   const total = customChecksOf(schema).length;
   return { verdict: total === stated ? 'projected' : 'dropped', projectionMoved: true };