Skip to content

Verify the whole collection shape at run open, so no query fails on it #768

Description

@ddeboer

What

Replace InPlaceRebuild’s two special-cased run-open checks (reference fields, and since #765 facet companions) with one generic shape check: diff the collection definition built from the SearchType against the live collection, and fail – after the lock, before any write – on every field that is missing or mis-shaped in a way some schema-admitted query would trip over.

The invariant it buys, stated positively: if openRun succeeds, no query fails because of the collection’s shape. This completes a symmetry the package already has on its other two legs – assertValidQuery guarantees nothing schema-invalid reaches the engine, and searchSchema validates the declaration itself; the collection ↔ schema contract is the one leg that is only partially checked today. BlueGreenRebuild needs none of this: it creates fresh collections from the definition every run, so it holds the invariant by construction.

Material, per field the definition declares

  • exists at all – a missing search companion breaks every text search on the collection (query_by names an unknown field), a missing facet field breaks every facet on it, a missing sort key breaks every ordered query; today only references and facet companions are caught;
  • type – a mismatch fails imports document by document, mid-run;
  • facetfacet_by on a non-facet field errors;
  • sortsort_by on a non-sort field errors;
  • reference – wrong or missing target breaks joins (the pre-existing check).

Report all mismatches in one error, with the existing drop-and-rebuild instruction, so recovering is one decision rather than a fail-fix-fail loop.

Deliberately not checked (friction without error-freeness)

  • optional, stem/locale – degrade ranking or strictness, error nothing;
  • extra columns the schema no longer declares – stale but harmless;
  • index: false display fields – stored either way.

Full drift detection stays a feature of its own, as the current code comments say.

Consequences

  • Less code, not more: the two special-cased checks and the facetCompanionsOf helper (which re-derives companion names the definition already carries) dissolve into one table-driven diff over the definition’s fields; the schema threading added for the companion check in feat(search): narrow a reference’s facet to the keys its target admits #765 goes with them. The one real maintenance surface is the materiality table – which centralises engine-error knowledge that today lives implicitly across the compiler and the two checks.
  • Behaviour change for existing in-place deployments, by design: a collection that predates some field’s searchable/facetable/sortable flag – possibly running “fine” only because nothing queries that field yet – now refuses at run open until dropped once. That is the check working; such a collection is a query-time incident waiting for its first user.

Out of scope

Heal instead of fail. Typesense can alter a live collection (add fields via a schema PATCH, re-indexing from stored values), so openRun could add missing fields itself instead of asking for a drop. Attractive, but a philosophy change (the writer mutating a live production index), reference fields may be excluded from alteration, and pre-policy documents carry no companion values until reprocessed – so healing the schema does not heal the data. Its semantics on Typesense 30 need verifying first; a separate issue if wanted.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions