feat(spec): refuse null members in list-comparand positions — $in/$nin members, $between bounds (#13357) (#13673)

* feat(spec): refuse null members in list-comparand positions ($in/$nin members, $between bounds)

Implements the 2026-08-31 maintainer ruling on the null-list-comparand axis
(option C): the contract refuses the shape at the validation entrance —
both the runtime door (assertListComparandShapes, inside parseFilterAST and
the engine seam) and the schema door (SetOperatorSchema / RangeOperatorSchema)
— so the three backend camps' divergence over what a null member matches
becomes constructively unreachable. The refusal text prescribes the ruling's
explicit spelling for absence. Negative pins witness that no engine verb and
no compile-face caller can hand the shape to a driver or the reference
matcher. The empty list stays a declared predicate; only null is carved out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F3jdziLbAPGeceVNmSox5L

* docs(spec): changeset + regenerated filter reference for the null list-member refusal

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F3jdziLbAPGeceVNmSox5L

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Steedos <zhuangjianguo@steedos.com>

Author: 3 people

.changeset/null-list-comparand-refused.md

@@ -0,0 +1,44 @@
+---
+"@objectstack/spec": minor
+---
+
+feat(spec): refuse `null` in list-comparand positions — `$in` / `$nin` members and `$between` bounds (#13357, #13495)
+
+**BREAKING** accept-set narrowing on the filter contract, shipped as `minor`
+under the repo's launch-window convention for breaking changes. Maintainer
+ruling 2026-08-31 (option C): the contract refuses the shape loudly at the
+validation entrance, and the cross-backend divergence it used to reach becomes
+constructively unreachable — ⛔ no cross-backend alignment (#5299 stays
+declined), and the reference matcher's own answers for these shapes are sealed
+behind the refusal, not repaired.
+
+What is refused, and where:
+
+- **Runtime door** (`assertListComparandShapes`, run inside `parseFilterAST`
+  and at the engine seam on every verb): a `null` member of `$in` / `$nin`,
+  and a `null` `$between` bound, are refused with the platform envelope
+  (`INVALID_FILTER` / 400). Previously the shape reached the backends, where
+  the SQL family, the mingo path and the reference matcher answered it three
+  ways — the matcher even disagreed with itself across the two readings of
+  "no value" (#13357's table).
+- **Schema door** (`SetOperatorSchema` / `FieldOperatorsSchema`): a `null`
+  member is refused at parse time with a pointed message, the same
+  check-not-type-change mechanism as the #7596 `{ $field }` member refusal.
+  A `null` `$between` endpoint never parsed (the endpoint union is
+  `number | Date | string`); it now gets the pointed message instead of zod's
+  generic union text.
+
+The refusal text prescribes the ruling's explicit spelling: "one of […] OR has
+no value" is `{"$or": [{"$in": […]}, {"$null": true}]}`, and `{"$null": false}`
+is the has-a-value half. The carve-out is null-shaped and nothing wider:
+`$in: []` / `$nin: []` stay the declared predicates they are, every non-null
+member type keeps parsing (#5041's and #5234's member questions stand
+untouched), and `$eq: null` is a separate surface (#13494, ruled separable).
+
+**Migration.** A filter refused by the new checks was already answered
+inconsistently across backends, so it had no portable meaning to preserve.
+Spell the intent explicitly: `$or: [{$in: […]}, {$null: true}]` for
+"one of […] or empty", `{$null: false}` (or `$and` with it) for the
+has-a-value direction, and `$gte` / `$lte` for a half-open range.
+
+<!-- adr-0087: not-required (no-migration-prescription) A validity narrowing over existing keys: no key is removed, renamed or re-shaped, so there is no tombstone and nothing mechanical for `objectstack migrate meta` to rewrite. The refusal reaches an affected author at the parse/query site carrying the remedy; which explicit spelling matches the author's intent ($or with $null:true, $null:false, or a half-open range) is an authoring decision no migration entry can perform — and the ruling's evidence base measured zero authored occurrences of the refused shape. -->

content/docs/references/data/filter.mdx

@@ -123,8 +123,8 @@ Type: `[FilterArray](#filterarray)[]`
 
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
-| **$in** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. |
-| **$nin** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. |
+| **$in** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. null is NOT a member shape either: state absence explicitly with the null predicate — "one of […] OR has no value" is `{ "$or": [{ "$in": […] }, { "$null": true }] }`. |
+| **$nin** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. null is NOT a member shape either: state absence explicitly with the null predicate — "one of […] OR has no value" is `{ "$or": [{ "$in": […] }, { "$null": true }] }`. |
 
 
 ---

packages/drivers/driver-memory/src/memory-null-list-member-unreachable.test.ts

@@ -0,0 +1,112 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#13357] Ruling point 3's NEGATIVE pin, matcher side: a refused null list
+ * member cannot reach this package's reference matcher.
+ *
+ * # What was ruled (2026-08-31, option C)
+ *
+ * #13357 measured that the reference matcher answers `$in: [null]` /
+ * `$nin: [null]` differently across the two readings of "no value" (a stored
+ * `null` vs an absent key) while `$null` / `$ne: null` agree — and that the
+ * SQL family answers the same filters a third way. The ruling REFUSES the
+ * shape at the contract's validation entrance (`@objectstack/spec`,
+ * `assertListComparandShapes`, run inside `parseFilterAST` and at the engine
+ * seam) instead of aligning the backends: the divergence becomes
+ * constructively unreachable, ⛔ deliberately not repaired (「⛔ 不单独修一个
+ * 到不了的路径」), so NOTHING in this file asserts what the matcher would
+ * have answered. `memory-matcher-null-value-and-comparand.test.ts` keeps
+ * those arms deliberately absent for the same reason.
+ *
+ * # What this file pins, and its honest boundary
+ *
+ * A direct caller of this driver — this repo's own conformance suites, an
+ * embedder — compiles its filter with `parseFilterAST` and hands the result
+ * over (`filter-comparand-shape.ts`'s #9228 section is the ruling that put
+ * the gate on that face for exactly this caller). This file drives that
+ * pipeline end to end and pins that for every refused shape it ABORTS at the
+ * compile face, on BOTH fixture readings, before any row is consulted: the
+ * evaluation step is provably never reached because the compile step throws.
+ * The engine half (every verb, driver-call witness) is pinned in
+ * `@objectstack/objectql`'s `engine-filter-array-lowering.test.ts`; the
+ * wire/protocol face runs the same `parseFilterAST`.
+ *
+ * The boundary, stated rather than hidden: `match()` and
+ * `InMemoryDriver.find()` remain plain library functions — a caller that
+ * skips the compile face meets only this package's own
+ * `assertFilterConditionShape`, which is deliberately NOT extended to the
+ * null-member rule (⛔ 不做跨后端对齐工程). That boundary is the same one
+ * every #5869 refusal has had since #9228, and it is not widened here.
+ */
+
+import { describe, it, expect } from 'vitest';
+import { parseFilterAST } from '@objectstack/spec/data';
+
+import { match } from './memory-matcher.js';
+
+type Refusal = Error & { code?: string; status?: number };
+
+/** The card's own fixture, in both readings of "no value" (#13357). */
+const NULLED_ROWS: Array<Record<string, unknown>> = [
+  { id: '1', name: 'a' },
+  { id: '3', name: null },
+];
+const MISSING_ROWS: Array<Record<string, unknown>> = [
+  { id: '1', name: 'a' },
+  { id: '3' },
+];
+
+/**
+ * The direct-caller pipeline, exactly as the module note describes it: compile
+ * first, evaluate second. The refusal has to land in step one — if compile
+ * returns, the matcher HAS been reached and the pin below fails on the
+ * sentinel rather than on a missing throw.
+ */
+function compileThenMatch(rows: Array<Record<string, unknown>>, where: unknown): string[] {
+  const condition = parseFilterAST(where);
+  return rows.filter((row) => match(row, condition)).map((row) => String(row.id));
+}
+
+const refusalOf = (run: () => unknown): Refusal => {
+  try {
+    run();
+  } catch (e) {
+    return e as Refusal;
+  }
+  throw new Error('expected the compile face to refuse this filter, but it returned');
+};
+
+describe('[#13357] a refused null list member cannot reach the matcher (ruled 2026-08-31)', () => {
+  it.each([
+    ['$in: [null]', { name: { $in: [null] } }],
+    ['$nin: [null]', { name: { $nin: [null] } }],
+    ['$between: [null, null]', { name: { $between: [null, null] } }],
+    ['$between: [null, max]', { name: { $between: [null, 'z'] } }],
+    ['$between: [min, null]', { name: { $between: ['a', null] } }],
+  ])('%s aborts at the compile face on BOTH readings of "no value"', (_label, where) => {
+    // Record-independent by construction — the compile face never sees a row —
+    // so the two readings that split the matcher (#13357's table) cannot even
+    // be posed. Driving both anyway is the point of the pin: neither fixture
+    // gets an answer, so there is no divergence left to observe.
+    for (const rows of [NULLED_ROWS, MISSING_ROWS]) {
+      const err = refusalOf(() => compileThenMatch(rows, where));
+      expect(err.code, _label).toBe('INVALID_FILTER');
+      expect(err.status, _label).toBe(400);
+    }
+  });
+
+  it('the pipeline itself is real — a legal list compiles and the matcher answers', () => {
+    // Positive control: without it, the refusals above would also "pass" if
+    // compileThenMatch were broken outright.
+    expect(compileThenMatch(NULLED_ROWS, { name: { $in: ['a'] } })).toEqual(['1']);
+    expect(compileThenMatch(MISSING_ROWS, { name: { $nin: ['a'] } })).toEqual(['3']);
+  });
+
+  it('an EMPTY list still passes the same face — the refusal is null-shaped, not list-shaped', () => {
+    // `$in: []` / `$nin: []` are declared predicates ("matches nothing" /
+    // "matches everything") and PR #13630 pins them downstream; the carve-out
+    // must not catch them.
+    expect(compileThenMatch(NULLED_ROWS, { name: { $in: [] } })).toEqual([]);
+    expect(compileThenMatch(NULLED_ROWS, { name: { $nin: [] } })).toEqual(['1', '3']);
+  });
+});

packages/objectql/src/engine-filter-array-lowering.test.ts

@@ -483,6 +483,69 @@ describe('Door 2 lowers FilterArray to FilterCondition before the driver (#5158)
     expect(err.message).toMatch(/where\.amount\.\$between/);
   });
 
+  // ── [#13357] the null carve-out, ruled 2026-08-31: refused at this seam, ──
+  // ── so the drivers' three-way null-member divergence is UNREACHABLE ──────
+  //
+  // Ruling point 3's negative pin, engine half: the witness is the recording
+  // driver's call log, not the thrown envelope alone — an envelope asserted
+  // without the log would also pass if a DRIVER had thrown it, i.e. if the
+  // refusal had not preceded the dispatch, which is the one thing these
+  // assertions exist to prove. The compile-face half (`parseFilterAST`, both
+  // input forms) is pinned in `@objectstack/spec`'s
+  // `filter-comparand-shape.test.ts`; the matcher-side statement lives in
+  // driver-memory's `memory-null-list-member-unreachable.test.ts`. ⛔ Nothing
+  // here asserts what any backend WOULD have answered — the divergence is
+  // sealed, not reconciled (#5299 stays declined).
+
+  it.each([
+    ['$in: [null]', { stage: { $in: [null] } }],
+    ['$nin: [null]', { stage: { $nin: [null] } }],
+    ['$between: [null, null]', { amount: { $between: [null, null] } }],
+    ['$between: [null, max]', { amount: { $between: [null, 20] } }],
+  ])('a null list member is refused on EVERY verb before any driver call — %s', async (_l, where) => {
+    await expect(engine.find('deal', { where }))
+      .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+    await expect(engine.findOne('deal', { where }))
+      .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+    await expect(engine.count('deal', { where } as unknown as EngineCountOptions))
+      .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+    await expect(engine.aggregate('deal', {
+      where: where as unknown as EngineAggregateOptions['where'],
+      groupBy: ['stage'],
+      aggregations: [{ function: 'count', field: 'id', alias: 'n' }],
+    })).rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+    await expect(engine.update('deal', { amount: 1 }, { where, multi: true } as any))
+      .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+    await expect(engine.delete('deal', { where, multi: true } as any))
+      .rejects.toMatchObject({ status: 400, code: 'INVALID_FILTER' });
+    // The negative half: refused BEFORE the store — no read, no write, no row
+    // moved. (The count() control below adds its own read, so it runs after.)
+    expect(reads).toHaveLength(0);
+    expect(writes).toHaveLength(0);
+    expect(await engine.count('deal')).toBe(3);
+  });
+
+  it('the null-member refusal is not vacuous — the same list WITHOUT null reaches the driver', async () => {
+    // Positive control for the zero-call reading above: one member removed,
+    // same operator, same field, and the dispatch happens.
+    const rows = await engine.find('deal', { where: { stage: { $in: ['won'] } } });
+    expect(reads).toHaveLength(1);
+    expect(lastWhere()).toEqual({ stage: { $in: ['won'] } });
+    expect(rows.map((r: any) => r.id).sort()).toEqual(['d1', 'd3']);
+  });
+
+  it('a nested null member is refused at its own path, engine prefix and all', async () => {
+    const err = await engine.find(
+      'deal',
+      { where: { $or: [{ stage: { $nin: [null] } }] } },
+    ).then(() => null, (e: any) => e);
+    expect(err?.status).toBe(400);
+    expect(err?.code).toBe('INVALID_FILTER');
+    expect(err.message).toMatch(/^find\('deal'\): /);
+    expect(err.message).toContain('where.$or[0].stage.$nin[0]');
+    expect(reads).toHaveLength(0);
+  });
+
   // ── what must KEEP working: the declared list shapes ───────────────────
 
   it('a proper list comparand still reaches the driver untouched', async () => {

packages/spec/src/data/filter-comparand-shape.test.ts

@@ -98,6 +98,68 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922
     expect(err.status).toBe(400);
   });
 
+  // ── the null carve-out, ruled 2026-08-31 (#13357; $between is #13495) ──
+
+  it.each([
+    ['$in, lowered array form', [['stage', 'in', [null]]]],
+    ['$in, object passthrough', { stage: { $in: [null] } }],
+    ['$nin, lowered array form', [['stage', 'not_in', [null]]]],
+    ['$nin, object passthrough', { stage: { $nin: [null] } }],
+    ['$in with a real neighbour', { stage: { $in: ['won', null] } }],
+  ])('refuses a null list MEMBER — %s', (_label, where) => {
+    const err = refusalOf(() => parseFilterAST(where));
+    expect(err.code).toBe(StandardErrorCode.enum.INVALID_FILTER);
+    expect(err.status).toBe(400);
+  });
+
+  it.each([
+    ['[null, null]', { at: { $between: [null, null] } }],
+    ['[null, max]', { at: { $between: [null, '2026-07-15'] } }],
+    ['[min, null]', { at: { $between: ['2026-07-01', null] } }],
+  ])('refuses a null $between BOUND — %s', (_label, where) => {
+    const err = refusalOf(() => parseFilterAST(where));
+    expect(err.code).toBe(StandardErrorCode.enum.INVALID_FILTER);
+    expect(err.status).toBe(400);
+  });
+
+  it('the null-member refusal prescribes the ruling\'s explicit spelling', () => {
+    // 2026-08-31: 「等于 X 或为空」的合法拼法是显式的 $or + $null — the
+    // refusal must spell it out, in both halves, and still name operator,
+    // field, position and authoring spellings (the #5346/#5348 contract).
+    const err = refusalOf(() => parseFilterAST({ stage: { $nin: [null] } }));
+    expect(err.message).toMatch(/^Operator "\$nin" on field "stage"/);
+    expect(err.message).toContain('where.stage.$nin[0]');
+    expect(err.message).toContain('{"$or": [{"stage": {"$in": […]}}, {"stage": {"$null": true}}]}');
+    expect(err.message).toContain('{"$null": false}');
+    expect(err.message).toMatch(/Authoring spellings: nin, not_in, notin/);
+    expect(err.message).toMatch(/UNFILTERED result set/);
+  });
+
+  it('the null-bound refusal points at the offending index and the working alternatives', () => {
+    const err = refusalOf(() => parseFilterAST({ at: { $between: ['2026-07-01', null] } }));
+    expect(err.message).toMatch(/^Operator "\$between" on field "at" requires two non-null bounds/);
+    expect(err.message).toContain('where.at.$between[1]');
+    expect(err.message).toContain('"$gte"/"$lte"');
+    expect(err.message).toContain('{"at": {"$null": true}}');
+    expect(err.message).toMatch(/UNFILTERED result set/);
+  });
+
+  it('a null member is refused at its own path inside $and / $or / $not too', () => {
+    expect(refusalOf(() => parseFilterAST({ $not: { stage: { $in: [null] } } })).message)
+      .toContain('where.$not.stage.$in[0]');
+    expect(refusalOf(() => parseFilterAST({ $or: [{ stage: { $nin: [null] } }] })).message)
+      .toContain('where.$or[0].stage.$nin[0]');
+  });
+
+  it('refuses ONLY null — falsy and empty-ish members are values, not absence', () => {
+    // The carve-out is null-shaped and nothing wider: #5041's and #5234's
+    // member questions stand untouched, and every falsy VALUE keeps working.
+    expect(parseFilterAST({ n: { $in: [0, false, ''] } })).toEqual({ n: { $in: [0, false, ''] } });
+    expect(parseFilterAST({ n: { $nin: [0, false, ''] } })).toEqual({ n: { $nin: [0, false, ''] } });
+    expect(parseFilterAST({ at: { $between: ['', ''] } })).toEqual({ at: { $between: ['', ''] } });
+    expect(parseFilterAST({ n: { $between: [0, 0] } })).toEqual({ n: { $between: [0, 0] } });
+  });
+
   // ── the wording contract (#5346 / #5348), unchanged by the move ────────
 
   it('names the operator, the field, what arrived, where, and the fix', () => {
@@ -134,6 +196,13 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922
       [['stage', 'not_in', 'won']],
       [['stage', 'in', 'won']],
       [['amount', 'between', 5]],
+      // The 2026-08-31 null carve-out (#13357/#13495): the prescribed $or +
+      // $null spelling makes these the LONGEST messages this door assembles,
+      // so they live inside the same unrelaxed bound.
+      { stage: { $in: [null] } },
+      { stage: { $nin: [null] } },
+      { close_date: { $between: [null, null] } },
+      { close_date: { $between: ['2026-07-01', null] } },
     ]) {
       const err = refusalOf(() => parseFilterAST(where, "find('deal')"));
       expect(err.message.length, JSON.stringify(where)).toBeLessThan(500);