fix(search)!: fan out a qualified edge into one entry per tuple - #801
Open
ddeboer wants to merge 2 commits into
Open
fix(search)!: fan out a qualified edge into one entry per tuple#801ddeboer wants to merge 2 commits into
ddeboer wants to merge 2 commits into
Conversation
A weld asks whether ONE entry satisfies every condition. An entry whose leaves hold arrays stands for every combination at once, so it answers the weld with none of them – and Typesense 30.2, the current stable release, does not answer at all: the search hangs indefinitely, which reaches a consumer as an unexplained error. - refuse a nested field declaring both filterable and array, so a leaf a weld can name states one value per entry - fan an edge out into one entry per combination of its weldable leaves, on the framed node before projection, so each leaf still passes through transform, folding and the facet companion unchanged - write a nested identity companion under the arity of its reference, since that companion is the leaf a weld actually names; a top-level companion stands for the whole document and is unchanged - bound the fan-out with maxEntries (default 100), per ADR 12: a cartesian product over an edge's values is a bound in the data's own units - pin the engine guarantee against a real container, the array-shaped hang included, so a future engine bump reports when it is gone Measured at 1,000,000 documents: the welded filter answers in 14–19 ms where it previously never returned, and the fanned-out shape is no slower than the array shape on 31.0, so the eventual upgrade unwinds nothing. Indexing is ~15 % slower for ~50 % more entries. BREAKING CHANGE: a nested field may no longer declare both “filterable” and “array”; searchSchema refuses it. Declare the leaf single-valued – the projection now emits one entry per combination.
Three defects found reviewing the fan-out, two of them introduced by it and each invisible without a live engine. - restore the already-a-list case when widening a nested leaf. Only Reference Types are held to the single-valued rule, and the same path declares the fields of the Root Type a local lookup nests, where array and filterable meet on an ordinary facet – the leaf was emitted with no type at all, which the engine refuses outright - declare a nested identity companion by what its path yields rather than always as a list. Under a single-valued edge the parent is an object, nothing flattens, and Typesense rejects the scalar the projection writes against a declared list: the whole import fails - expand every tuple the cap keeps. Stopping mid-expansion left the remaining weldable leaves holding their lists, and a single-valued leaf then kept the first value and dropped the rest – losing data quietly rather than fanning it out Two existing tests asserted the arity rule inverted, stating that a companion is always a list and that a single value is what the engine rejects. A live import says the opposite, so both now read the other way. An identity on a Reference Type's field is already refused, and a nested filterable lookup can no longer be an array, so a nested companion never holds more than one id; no change was needed there. The entry budget is per node carrying the reference rather than per document, so nested edges compound it – documented rather than changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #798.
A welded co-element filter – this agent in this role – hangs Typesense 30.2 indefinitely whenever the nested entry’s leaves hold arrays, which is every document a qualified relation produced in practice. Through the GraphQL surface that reached a consumer as
Unexpected error.What this changes, and why it is not a workaround
The hang is why this got noticed. It is not why the shape changes.
A weld asks whether one entry satisfies every condition. An entry holding
{ role: ["etser", "etcher"], creator_id: ["p1", "p3"] }stands for four (role, agent) pairs at once, so it answers the weld with none of them – and the weld silently degenerates into the cross-product it exists to exclude. That entry is not a valid input the engine mishandles; it is a shape that never had a meaning, and one ADR 24’s own words – one entry per edge – already exclude.So: a weldable nested leaf is single-valued, and multiplicity moves to the entry list.
searchSchemarefuses a nested field declaring bothfilterableandarray.output-only leaves are untouched – nothing welds them, so they may carry a list for display.creator_id: ["p1"]inside each entry, which is exactly what hangs 30.2. Top-level companions stand for the whole document and are unchanged.maxEntries(default 100) bounds the entries one document stores, per ADR 12: a cartesian product over an edge’s values is a bound in the data’s own units, which is not a bound.Full reasoning, measurements and the rejected alternatives are in the new ADR 26.
Measured
Against live containers, 1 000 000 documents, same data and same query:
Two things that decided the design. The upgrade unwinds nothing: the fanned-out shape is no slower on 31.0 than the array shape it replaces, so when 31.0 goes stable this is routine maintenance rather than a migration back. And facets stay on the real fields: the other shape that works on 30.2 is an index-time composite
role|agentkey, faster at 3 ms but faceting into 2 142 125 buckets in 1.4 s against 15 buckets in 48 ms.Cost: indexing is ~15 % slower (54 s against 47 s per 1 000 000 documents) for ~50 % more entries. A batch cost, not a request cost.
Upstream
Typesense fixed this in the 31.0 release candidates –
31.0.rc1hangs,rc5throughrc14answer correctly – but 31.0 has no stable release. Bisect and trigger posted to the two open reports, typesense#2469 (where the reporter had been told it was the expected cost of&&) and typesense#2964.Notes for review
filterableandarray. Threesearch-pipelinefixtures carried exactly that shape and are updated here – the same edit a deployment makes.packages/search-typesense/vite.config.tsmoves a coverage threshold up (95.69 → 95.82) from a full-package run.arrayValueType’scase 'string[]'became unreachable once indexed nested leaves are single-valued, and a collection-definition test covering a combination the schema now refuses (the refusal itself is asserted inschema.test.ts).welded-filter.integration.test.tsis the test that would have caught this: it asserts the weld returns strictly less than the same conditions unwelded, and pins the array-shaped hang so a future engine bump reports when it is gone.