diff --git a/docs/decisions/0026-fan-out-a-qualified-edge-into-one-entry-per-tuple.md b/docs/decisions/0026-fan-out-a-qualified-edge-into-one-entry-per-tuple.md new file mode 100644 index 00000000..663078e5 --- /dev/null +++ b/docs/decisions/0026-fan-out-a-qualified-edge-into-one-entry-per-tuple.md @@ -0,0 +1,199 @@ +# 26. Fan out a qualified edge into one entry per tuple + +Date: 2026-09-02 + +## Status + +Proposed + +Amends [ADR 24 (Carry data on a reference edge)](./0024-carry-data-on-a-reference-edge.md), +which introduced the welded co-element filter but left the shape of an entry +under-specified. Relates to +[ADR 18 (Filter across several fields with one clause)](./0018-filter-across-several-fields-with-one-clause.md) +and [ADR 12 (Bound memory by the unit of work, not the input)](./0012-bound-memory-by-the-unit-of-work-not-the-input.md). + +## Context + +[ADR 24](./0024-carry-data-on-a-reference-edge.md) says the shape is **one entry +per edge**, and gives the welded filter that only an edge can answer: _this +agent in this role_, rather than “this role appears, and this agent appears, +somewhere in this document”. Unwelded, a work where X is the publisher and +somebody else is the photographer matches `role: fotograaf && creator: X`. That +false positive is the entire reason the nested entry exists. + +What the ADR did not say is what an entry holds when the graph gives an edge +more than one value. Nothing stopped a nested leaf declaring `array: true`, and +two ordinary modelling facts push straight at it: + +- a role may be stated once per language, so `role` arrives as two literals; +- the edge’s endpoint may be multi-valued, so `creator` arrives as two IRIs. + +So the projection wrote entries like +`{ "role": ["etser", "etcher"], "creator_id": ["p1", "p3"] }`, and every +qualified relation in production carried them. + +**Two things are wrong with that, and only one of them is a bug in somebody +else’s code.** + +### The engine hangs + +`typesense/typesense:30.2` – the current stable release – **hangs indefinitely** +on a welded filter whenever the entry holds arrays. Measured against a live +container, one document, no LDE code in the path: + +``` +c.{role:=etser && aid:=p1} with role:["etser"] aid:["p1"] → no response, ever +c.{role:=etser && aid:=p1} with role:"etser" aid:"p1" → 1 hit, 5 ms +``` + +The trigger is narrower than “the fields hold arrays”: the hang needs both +conditions to find a match somewhere in the document **and** at least one matched +value to be array-valued. A single array-valued leaf on either side is enough – +`role:"etser" aid:["p1"]` hangs exactly as hard as both-arrays. A document that +matches neither condition is never slow, however array-shaped. At 1 000 000 +documents the welded filter never returns, while either condition alone answers +in 16 ms. + +Through the GraphQL surface that reaches a consumer as `Unexpected error`, the +5 s engine timeout retried once. + +Typesense fixed it in the 31.0 release candidates – `31.0.rc1` hangs, `rc5` +through `rc14` answer correctly – but 31.0 has no stable release, and the +neighbouring upstream reports +([typesense#2469](https://github.com/typesense/typesense/issues/2469), +[typesense#2964](https://github.com/typesense/typesense/issues/2964)) are both +still open. + +### The entry has no tuple to test + +The engine bug is the reason this got noticed. It is not the reason to change +the shape. + +A weld asks a question about **one element**: _is there an entry whose role is +`etser` and whose agent is `p1`_. An entry holding +`{ role: ["etser", "etcher"], creator_id: ["p1", "p3"] }` has no single answer – +it stands for four (role, agent) pairs at once, and the weld silently degenerates +into the cross-product it existed to exclude. Welding inside such an entry is the +same mistake as not welding at all, one level down. + +So the array-valued entry is not a valid input the engine mishandles. It is a +shape that never had a meaning, which no engine could have answered, and which +[ADR 24](./0024-carry-data-on-a-reference-edge.md)’s own words – _one entry per +edge_ – already exclude. + +## Decision + +**A weldable nested leaf is single-valued, and multiplicity moves to the entry +list.** + +Three parts. + +### 1. A `filterable` nested field may not declare `array: true` + +Refused by `searchSchema`, beside the Roles nesting already cannot serve. A +nested leaf that a weld can name states one value per entry, and a declaration +saying otherwise is refused at startup rather than producing entries no filter +can read. + +`output`-only nested leaves are untouched: nothing welds them, so an entry may +carry a list for display. + +### 2. The projection fans out one entry per tuple + +Where the graph gives an edge several values for a weldable leaf, the projection +emits **one entry per combination**, each leaf single-valued: + +```jsonc +// the graph +{ "role": ["etser", "etcher"], "creator": ["p1", "p3"] } + +// the entries +[ { "role": "etser", "creator_id": "p1" }, + { "role": "etser", "creator_id": "p3" }, + { "role": "etcher", "creator_id": "p1" }, + { "role": "etcher", "creator_id": "p3" } ] +``` + +Fan-out happens on the **framed node**, before the entry is projected, so each +leaf passes through `transform`, folding and the facet companion exactly as a +single-valued field always has. Nothing downstream learns a new shape. + +Nothing is dropped: the four entries carry what the two arrays carried. What +changes is that each one now answers the weld. + +### 3. Language variants are labels, not values + +A role stated once per language is **one** role. Declaring `role` multi-valued to +hold `"etser"@nl` and `"etcher"@en` models a labelling accident as data, and +fan-out would then emit two entries for one relation and facet them into two +buckets. + +Index the role’s canonical IRI, single-valued, and resolve labels at the surface +like every other reference. Measured on 1 000 000 documents this halves the +entries per document (3.74 against 7.49) and gives 15 facet buckets rather than +30 that split one role across languages. + +### What bounds it + +A cartesian product over an edge’s own values is a bound stated in the data’s own +units, which [ADR 12](./0012-bound-memory-by-the-unit-of-work-not-the-input.md) +says is not a bound. An inline reference therefore declares `maxEntries` +(default 100): entries past it are dropped and reported, rather than a +pathological edge multiplying a document until the run dies. + +## Consequences + +**The weld works on the engine we run.** No release to wait for, no release +candidate in production. + +**It costs nothing to upgrade later.** Measured at 1 000 000 documents, both end +states are the same speed, and the fanned-out shape is no slower on 31.0 than the +array shape it replaces: + +| | welded filter, p50 | role alone | facet | +| ------------------------ | ------------------ | ---------- | ----- | +| 30.2 + fan-out | 14–19 ms | 15.9 ms | 30 ms | +| 31.0 + arrays | 13–16 ms | 15.8 ms | 64 ms | +| 30.2 + arrays _(before)_ | **hangs** | 15.8 ms | 30 ms | +| 31.0 + fan-out | 13–16 ms | 16.1 ms | 28 ms | + +So this is not a trade the 31.0 upgrade unwinds. When 31.0 goes stable it is +routine maintenance, not a migration back. + +**Indexing is ~15 % slower** – 54 s against 47 s for 1 000 000 documents, +consistent across both engine versions – because a document carries about 50 % +more entries. A batch cost, not a request cost. + +**Filters and facets stay on the real fields.** The alternative that also works on +30.2 is to weld at index time into a composite `role|agent` key and filter it with +one condition. It is faster (3 ms against 17 ms) and worse: at 1 000 000 documents +faceting that key yields 2 142 125 buckets in 1.4 s, against 15 buckets in 48 ms +for the role itself. It also needs a separator that can never occur in an IRI, and +forces the schema to declare which pairs are weldable, narrowing the query surface +from _any two edge fields_ to _the declared pairs_. Rejected. + +**A schema that declared a weldable leaf `array: true` now fails at startup**, with +a message saying to fan out instead. We are pre-release; there is no migration. + +**One array-valued weldable leaf re-arms the hang**, which is why part 1 is a +refusal rather than a convention. A deployment cannot opt out of it by accident, +and the failure is at startup rather than a query that never returns. + +**A companion’s declared type follows its path, not the leaf.** The flat id a +weld actually names holds one value per entry, but its declared type describes +the whole path across the document: `string[]` under an `object[]` edge, +`string` under a single-valued one. Typesense enforces that strictly wherever +nothing widens the path – a companion declared `string[]` under a single-valued +edge fails the import outright, for every document carrying such an edge. This +is not visible in a collection definition, only against a live engine, which is +why the integration test asserts the import rather than the declaration. + +**A single-valued edge still cannot be welded on 30.2.** Where the reference is +not `array`, the stored parent is `object` rather than `object[]`, and the +engine hangs on `credit.{…}` over it whatever the leaves hold – so this is not +the array defect above and fan-out does not address it. It costs nothing today: +a qualified relation is multi-valued by nature and every real edge declares +`array: true`, which is the shape [ADR 24](./0024-carry-data-on-a-reference-edge.md) +describes. A deployment that genuinely wants one qualified edge per document +should still declare it `array: true` and rely on the entries, until 31.0 is +stable. diff --git a/docs/reference/search.md b/docs/reference/search.md index c10305cf..00308f86 100644 --- a/docs/reference/search.md +++ b/docs/reference/search.md @@ -532,6 +532,41 @@ Welding on identity needs the endpoint's `filterable`, which fans out its id as a leaf beside the stored object – an engine welds conditions on an entry's own leaf fields only. That is a physical detail: you write the logical field. +**A weldable leaf is single-valued.** A nested field declaring `filterable` may +not also declare `array`; `searchSchema` refuses it. A weld asks whether _one_ +entry satisfies every condition, and an entry holding a list stands for each +combination at once – so it answers the weld with none of them, and the weld +degenerates into the cross-product it exists to exclude. + +Multiplicity belongs to the entry list instead. Where the graph gives an edge +several roles or several endpoints, the projection emits **one entry per +combination**: + +```jsonc +// the graph +{ "role": ["etser", "etcher"], "creator": ["p1", "p3"] } + +// the entries +[ { "role": "etser", "creator_id": "p1" }, + { "role": "etser", "creator_id": "p3" }, + { "role": "etcher", "creator_id": "p1" }, + { "role": "etcher", "creator_id": "p3" } ] +``` + +Nothing is dropped, and each entry now answers the weld. Two consequences worth +knowing when you declare an edge: + +- A role stated once per language is **one** role, not two. Index its canonical + IRI single-valued and resolve labels at the surface; declaring the label + multi-valued makes fan-out emit an entry per language and splits one role + across two facet buckets. +- `maxEntries` on the inline reference caps the entries one document stores + (default 100), so a pathological edge cannot multiply a document without + bound. An `output`-only nested leaf is untouched by all of this: nothing welds + it, so it may carry a list for display. + +See [ADR 26](../decisions/0026-fan-out-a-qualified-edge-into-one-entry-per-tuple). + Out of scope for now: faceting an edge's own values, which the current engine cannot serve correctly. diff --git a/packages/search-pipeline/test/extraction-roundtrip.integration.test.ts b/packages/search-pipeline/test/extraction-roundtrip.integration.test.ts index 938c5faf..e1bf58d7 100644 --- a/packages/search-pipeline/test/extraction-roundtrip.integration.test.ts +++ b/packages/search-pipeline/test/extraction-roundtrip.integration.test.ts @@ -89,9 +89,10 @@ const creatorRole = defineSearchType({ name: 'CreatorRole', fields: [ { + // Single-valued: a leaf a weld can name states one value per entry, and + // an edge the graph gave several roles fans out (ADR 26). name: 'role', kind: 'keyword', - array: true, output: true, filterable: true, path: `<${SCHEMA}roleName>`, diff --git a/packages/search-pipeline/test/extraction.test.ts b/packages/search-pipeline/test/extraction.test.ts index 334f9b72..0ed63c05 100644 --- a/packages/search-pipeline/test/extraction.test.ts +++ b/packages/search-pipeline/test/extraction.test.ts @@ -102,9 +102,10 @@ const creatorRole = defineSearchType({ name: 'CreatorRole', fields: [ { + // Single-valued: a leaf a weld can name states one value per entry, and + // an edge the graph gave several roles fans out (ADR 26). name: 'role', kind: 'keyword', - array: true, output: true, filterable: true, path: `<${SCHEMA}roleName>`, diff --git a/packages/search-pipeline/test/nested-fanout.integration.test.ts b/packages/search-pipeline/test/nested-fanout.integration.test.ts index 32ea4a46..ef62cec4 100644 --- a/packages/search-pipeline/test/nested-fanout.integration.test.ts +++ b/packages/search-pipeline/test/nested-fanout.integration.test.ts @@ -46,9 +46,10 @@ const creatorRole = defineSearchType({ name: 'CreatorRole', fields: [ { + // Single-valued: a leaf a weld can name states one value per entry, and + // an edge the graph gave several roles fans out (ADR 26). name: 'role', kind: 'keyword', - array: true, output: true, filterable: true, path: `<${SCHEMA}roleName>`, diff --git a/packages/search-typesense/src/collection-definition.ts b/packages/search-typesense/src/collection-definition.ts index acd99b1e..cf5529a8 100644 --- a/packages/search-typesense/src/collection-definition.ts +++ b/packages/search-typesense/src/collection-definition.ts @@ -438,7 +438,7 @@ function nestedFields( index: false, optional: true, }, - ...nestedIdentityFields(prefix, field, schema), + ...nestedIdentityFields(prefix, field, schema, flattensToArray), ); continue; } @@ -458,7 +458,7 @@ function nestedFields( // an engine welds conditions on an entry's LEAF fields only. Its // identity companion is that leaf, so it sits beside the object rather // than inside it. - ...nestedIdentityFields(prefix, field, schema), + ...nestedIdentityFields(prefix, field, schema, flattensToArray), ); continue; } @@ -527,6 +527,7 @@ function nestedIdentityFields( prefix: string, field: SearchField, schema: SearchSchema, + flattensToArray: boolean, ): CollectionFieldSchema[] { const names = physicalFields(field, schema); if (names.identity === undefined) { @@ -535,11 +536,16 @@ function nestedIdentityFields( return [ { name: nestedFieldName(prefix, names.identity), - // Always a list, whatever the enclosing reference's arity: the projection - // writes it with `setArray`, and an indexed field's declared type is - // checked against what is stored – a `string` here rejects every - // document carrying such an edge, at import. - type: 'string[]', + // Typed by what the PATH yields across the document, exactly as the `id` + // beside it is, and by the same two routes every other declaration here + // uses: an `object[]` ancestor multiplying the entries, or the field's + // own `array` making each entry hold a list. A weldable leaf is + // single-valued (ADR 26), so a companion that a weld names contributes + // one id per entry – but a multi-valued reference reached through a + // locally-nested Root Type is not weldable and still harvests a list. + // Getting either route wrong rejects the document at import: Typesense + // enforces the declared arity wherever nothing widens the path. + type: flattensToArray || field.array === true ? 'string[]' : 'string', index: true, optional: true, }, @@ -613,6 +619,10 @@ function nestedLeafFields( // `object[]` flattens it into one. `typesenseValueType` honours `array` only // for the string-shaped kinds, so a multi-valued nested `integer` would // otherwise be declared scalar and rejected at import. + // + // A weldable leaf of a Reference Type is single-valued (ADR 26), but that + // narrows nothing here: this same path declares the fields of the Root Type a + // `local` lookup nests, where `array` and `filterable` may legitimately meet. const storesAList = flattensToArray || field.array === true; fields.push({ name: nestedFieldName(prefix, field.name), @@ -654,8 +664,11 @@ function arrayValueType(type: ValueType): CollectionFieldSchema['type'] { return 'float[]'; case 'bool': return 'bool[]'; - // Already a list: a multi-valued declaration under a multi-valued ancestor - // flattens no further. + // Already a list, and reachable even though a weldable leaf of a Reference + // Type is single-valued (ADR 26): this path also declares the fields of the + // Root Type a `local` lookup nests, and `searchSchema` constrains only + // Reference Types. A root keyword may be `array` and `filterable` at once – + // an ordinary facet – so flattening it must widen no further. case 'string[]': return 'string[]'; } diff --git a/packages/search-typesense/test/qualified-relation.test.ts b/packages/search-typesense/test/qualified-relation.test.ts index 038d7c7b..5ad0682d 100644 --- a/packages/search-typesense/test/qualified-relation.test.ts +++ b/packages/search-typesense/test/qualified-relation.test.ts @@ -148,17 +148,19 @@ describe('a single-valued edge', () => { }); const singleSchema = searchSchema(singleWork, person, creatorEdge); - it('declares the nested identity companion as a list', () => { - // The projection writes it with `setArray` whatever the arity, and an - // indexed field's declared type is checked at import – a `string` here - // rejects every document carrying such an edge. + it('declares the nested identity companion as a single value', () => { + // Nothing flattens the path under a single-valued edge – the parent is + // `object`, not `object[]` – and the companion holds one id per entry + // (ADR 26). Declaring `string[]` here is what rejects the document: checked + // against a live engine, the import fails outright, because Typesense + // enforces the declared arity wherever no ancestor widens it. const fields = buildCollectionDefinition(singleWork, { schema: singleSchema }).fields ?? []; expect( fields.find((field) => field.name === 'creator.creator_id'), - ).toMatchObject({ type: 'string[]', index: true }); + ).toMatchObject({ type: 'string', index: true }); }); }); @@ -287,11 +289,12 @@ describe('a local lookup that reaches back', () => { fields.find((field) => field.name === 'creator.made.id'), ).toMatchObject({ type: scalar, index: false }); // The identity companion sits BESIDE the object, indexed, because that is - // the leaf a filter can weld on – always a list, whatever the arity of - // the edge it hangs off. + // the leaf a filter can weld on – and it takes the arity of the path that + // reaches it, exactly as the `id` above does: one id per entry (ADR 26), + // widened only where an `object[]` ancestor multiplies the entries. expect( fields.find((field) => field.name === 'creator.made_id'), - ).toMatchObject({ type: 'string[]', index: true }); + ).toMatchObject({ type: scalar, index: true }); // The descent still stops: the cut type's own fields are not walked // again. expect( @@ -337,12 +340,23 @@ describe('nested fields of other kinds', () => { filterable: true, }, { + // Output-only, so it may stay a list: nothing welds it, and a weldable + // leaf is single-valued (ADR 26). name: 'source', kind: 'keyword', path: `${SCHEMA_ORG}isBasedOn`, array: true, output: true, - filterable: true, + }, + { + // Searchable rather than filterable, so it may stay a list too: free + // text is not a weld, and its folded companion is what gets indexed. + name: 'attribution', + kind: 'keyword', + path: `${SCHEMA_ORG}creditText`, + array: true, + output: true, + searchable: { weight: 1 }, }, ], }); @@ -415,30 +429,23 @@ describe('nested fields of other kinds', () => { ); }); - it('gives a searchable nested keyword its folded companion', () => { - expect(richField('credit.note_search')).toMatchObject({ - type: 'string[]', - }); - }); - - it('widens a multi-valued nested numeric under a single-valued edge', () => { - // Two ways a list arrives – the field declares one, or an ancestor - // flattens it – and `typesenseValueType` honours only the string-shaped - // kinds' own `array`, so this one needs widening on its own account. - const countEdge = defineSearchType({ - name: 'CountEdge', + it('stems a language-tagged nested text field in its own locale', () => { + // The counterpart of the `und` case above: a declared locale stems in + // itself, never in `defaultLocale`, so a Dutch note is not stemmed as if + // it were English. + const taggedEdge = defineSearchType({ + name: 'TaggedEdge', fields: [ { - name: 'position', - kind: 'integer', - path: `${SCHEMA_ORG}position`, - array: true, - output: true, - filterable: true, + name: 'note', + kind: 'text', + path: `${SCHEMA_ORG}description`, + locales: ['nl'], + searchable: { weight: 1 }, }, ], }); - const singleEdgeWork = defineSearchType({ + const taggedWork = defineSearchType({ name: 'Work', class: `${SCHEMA_ORG}CreativeWork`, fields: [ @@ -446,32 +453,229 @@ describe('nested fields of other kinds', () => { name: 'credit', kind: 'reference', path: `${SCHEMA_ORG}creator`, + array: true, output: true, - ref: { strategy: 'inline', typeName: 'CountEdge' }, + ref: { strategy: 'inline', typeName: 'TaggedEdge' }, }, ], }); const fields = - buildCollectionDefinition(singleEdgeWork, { - schema: searchSchema(singleEdgeWork, countEdge), + buildCollectionDefinition(taggedWork, { + schema: searchSchema(taggedWork, taggedEdge), + defaultLocale: 'en', }).fields ?? []; expect( - fields.find((field) => field.name === 'credit.position'), - ).toMatchObject({ type: 'int64[]', index: true }); + fields.find((field) => field.name === 'credit.note_search_nl'), + ).toMatchObject({ type: 'string[]', stem: true, locale: 'nl' }); + }); + + it('gives a searchable nested keyword its folded companion', () => { + expect(richField('credit.note_search')).toMatchObject({ + type: 'string[]', + }); }); it.each([ ['credit.position', 'int64[]'], ['credit.certainty', 'float[]'], ['credit.disputed', 'bool[]'], - // Already a list on its own: flattening does not double it. - ['credit.source', 'string[]'], ])('widens indexed nested %s to %s', (name, type) => { // An engine checks an indexed field's declared type against what is // stored, and the `object[]` above these flattens each value into a list. expect(richField(name)).toMatchObject({ type, index: true }); }); + + it('types a nested identity companion by what its path yields', () => { + // The companion holds one id per entry (ADR 26), so its declared type is + // decided by the ancestors, exactly as the `id` beside it is. Declaring + // `string[]` unconditionally makes Typesense reject, at import, every + // document whose edge is single-valued: nothing flattens the path there, + // and the engine enforces the declared arity. + const person = defineSearchType({ + name: 'Person', + class: `${SCHEMA_ORG}Person`, + fields: [ + { + name: 'label', + kind: 'text', + path: `${SCHEMA_ORG}name`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + ], + }); + const edge = defineSearchType({ + name: 'IdentifiedEdge', + fields: [ + { + name: 'agent', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + output: true, + filterable: true, + ref: { strategy: 'lookup', target: 'Person', local: true }, + }, + ], + }); + const workWith = (array: boolean) => + defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'credit', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + ...(array ? { array: true } : {}), + output: true, + ref: { strategy: 'inline', typeName: 'IdentifiedEdge' }, + }, + ], + }); + const companion = (array: boolean) => { + const type = workWith(array); + return ( + buildCollectionDefinition(type, { + schema: searchSchema(type, person, edge), + }).fields ?? [] + ).find((field) => field.name === 'credit.agent_id'); + }; + + expect(companion(true)).toMatchObject({ type: 'string[]' }); + expect(companion(false)).toMatchObject({ type: 'string' }); + }); + + it('declares a multi-valued nested companion as a list, unflattened', () => { + // The companion's other route to a list: the reference itself is `array`, + // so one entry harvests several ids even where no ancestor multiplies the + // entries. Reachable through a locally-nested Root Type, whose fields the + // single-valued rule does not constrain – and the projection writes a list + // there, so declaring `string` rejects the document at import. + const org = defineSearchType({ + name: 'Membership', + fields: [ + { + name: 'org', + kind: 'reference', + path: `${SCHEMA_ORG}memberOf`, + output: true, + ref: { strategy: 'lookup', target: 'Person' }, + }, + ], + }); + const nestedRoot = defineSearchType({ + name: 'Person', + class: `${SCHEMA_ORG}Person`, + fields: [ + { + name: 'label', + kind: 'text', + path: `${SCHEMA_ORG}name`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: 'affiliation', + kind: 'reference', + path: `${SCHEMA_ORG}affiliation`, + array: true, + output: true, + filterable: true, + ref: { strategy: 'inline', typeName: 'Membership', identity: 'org' }, + }, + ], + }); + // Single-valued, so nothing above flattens: only the field's own `array` + // makes this a list. + const work = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + output: true, + ref: { strategy: 'lookup', target: 'Person', local: true }, + }, + ], + }); + const fields = + buildCollectionDefinition(work, { + schema: searchSchema(work, nestedRoot, org), + }).fields ?? []; + + expect( + fields.find((field) => field.name === 'creator.affiliation_id'), + ).toMatchObject({ type: 'string[]', index: true }); + }); + + it('widens an indexed nested leaf of a local lookup’s own root type', () => { + // `searchSchema` constrains Reference Types, so a weldable leaf there is + // single-valued – but this same path also declares the fields of the Root + // Type a `local` lookup nests, where `array` and `filterable` meet on an + // ordinary facet. Widening must still produce a type. + const agent = defineSearchType({ + name: 'Agent', + class: `${SCHEMA_ORG}Person`, + fields: [ + { + name: 'label', + kind: 'text', + path: `${SCHEMA_ORG}name`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: 'nationality', + kind: 'keyword', + path: `${SCHEMA_ORG}nationality`, + array: true, + output: true, + filterable: true, + }, + ], + }); + const work = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + array: true, + output: true, + ref: { strategy: 'lookup', target: 'Agent', local: true }, + }, + ], + }); + const fields = + buildCollectionDefinition(work, { + schema: searchSchema(work, agent), + }).fields ?? []; + + expect( + fields.find((field) => field.name === 'creator.nationality'), + ).toMatchObject({ type: 'string[]', index: true }); + }); + + it('does not double a nested list that is already one', () => { + // A leaf a weld can name is single-valued (ADR 26), so a nested list is + // either output-only or searchable. Flattening one under the `object[]` + // widens it once, not twice – `string[]`, never `string[][]`. + expect(richField('credit.source')).toMatchObject({ + type: 'string[]', + index: false, + }); + expect(richField('credit.attribution_search')).toMatchObject({ + type: 'string[]', + }); + }); }); describe('a facet policy over the companion', () => { diff --git a/packages/search-typesense/test/welded-filter.integration.test.ts b/packages/search-typesense/test/welded-filter.integration.test.ts new file mode 100644 index 00000000..b0a7eeac --- /dev/null +++ b/packages/search-typesense/test/welded-filter.integration.test.ts @@ -0,0 +1,156 @@ +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import type { Client } from 'typesense'; +import { TypesenseContainer } from './typesense-container.js'; + +/** + * The engine guarantee the welded co-element filter rests on, pinned against a + * real Typesense rather than reasoned about – *this agent in this role*, the + * one question a qualified edge buys over two flat fields. + * + * **Why an integration test and not a compiler assertion.** The filter this + * compiles to is valid, documented syntax, and the compiler emitted it + * correctly all along. What no unit test could see is that the engine’s answer + * depends on the *shape of the stored values*: on `typesense/typesense:30.2` a + * weld over entries whose leaves hold arrays never returns at all – no error, + * no timeout, no response – while every unwelded variant answers in + * milliseconds. Only a live engine says so, which is why the entries below are + * written in both shapes and both are asserted. + * + * The trap, asserted here because it is silent in the other direction: + * `role:=X && agent:=Y` written OUTSIDE the braces matches a work where X and Y + * occur in *different* entries. That false positive is the whole reason the + * weld exists, so a weld that merely returns is not enough – it has to return + * strictly less. + * + * See [ADR 26](../../../docs/decisions/0026-fan-out-a-qualified-edge-into-one-entry-per-tuple.md) + * and [#798](https://github.com/ldelements/lde/issues/798). + */ +describe('a welded co-element filter', () => { + const container = new TypesenseContainer(); + let client: Client; + + const collection = 'works'; + const etser = 'http://vocab.example/role/etser'; + const drukker = 'http://vocab.example/role/drukker'; + const rembrandt = 'http://data.example/agent/rembrandt'; + const other = 'http://data.example/agent/other'; + + /** The filter a compiled {@link WeldedCriterion} produces. */ + const welded = (role: string, agent: string) => + `credit.{role:=\`${role}\` && agent_id:=\`${agent}\`}`; + + const found = async (filterBy: string) => { + const result = await client + .collections(collection) + .documents() + .search({ q: '*', query_by: '', filter_by: filterBy }, {}); + return (result.hits ?? []) + .map((hit) => (hit.document as { id: string }).id) + .sort(); + }; + + beforeAll(async () => { + client = await container.start(); + await client.collections().create({ + name: collection, + enable_nested_fields: true, + fields: [ + { name: 'credit', type: 'object[]' }, + { name: 'credit.role', type: 'string[]' }, + { name: 'credit.agent_id', type: 'string[]' }, + ], + }); + await client + .collections(collection) + .documents() + .import( + [ + // Fanned out: one entry per (role, agent) tuple, every leaf a single + // value – what the projection now writes. + { + id: 'fanned-match', + credit: [{ role: etser, agent_id: rembrandt }], + }, + // The false positive the weld exists to exclude: both values are + // present in the document, in different entries. + { + id: 'fanned-cross', + credit: [ + { role: etser, agent_id: other }, + { role: drukker, agent_id: rembrandt }, + ], + }, + ], + { action: 'create' }, + ); + }, 120_000); + + afterAll(async () => { + await container.stop(); + }); + + it('matches only the work whose ONE entry satisfies both conditions', async () => { + expect(await found(welded(etser, rembrandt))).toEqual(['fanned-match']); + }); + + it('returns strictly less than the same conditions unwelded', async () => { + // Unwelded, the cross-matched work comes back too – Rembrandt is on it, and + // so is the etser role, just never together. This is the assertion that + // makes the weld worth its cost. + expect( + await found( + `credit.role:=\`${etser}\` && credit.agent_id:=\`${rembrandt}\``, + ), + ).toEqual(['fanned-cross', 'fanned-match']); + }); + + it('answers a weld naming no matching tuple', async () => { + expect(await found(welded(drukker, other))).toEqual([]); + }); + + it('accepts a single-valued edge’s companion, declared as one value', async () => { + // A declared type only matters if the engine agrees with it, and here it + // disagreed: under a single-valued edge the parent is `object`, nothing + // flattens the path, and Typesense rejects a scalar companion declared + // `string[]` – the whole import fails. Pinned live, because the collection + // definition alone cannot show it. + const single = 'single_edge_works'; + await client.collections().create({ + name: single, + enable_nested_fields: true, + fields: [ + { name: 'credit', type: 'object' }, + { name: 'credit.role', type: 'string' }, + // One id per entry, and one entry: no ancestor widens the path. + { name: 'credit.agent_id', type: 'string' }, + ], + }); + const result = await client + .collections(single) + .documents() + .import([{ id: 'w1', credit: { role: etser, agent_id: rembrandt } }], { + action: 'create', + }); + + expect(JSON.stringify(result)).toContain('"success":true'); + }, 60_000); + + it('hangs on 30.2 where an entry holds arrays, which is why we fan out', async () => { + // The defect this shape exists to avoid, pinned so a future engine bump + // tells us when it is gone. Typesense answers every OTHER form of this + // query in milliseconds; welded over array-valued leaves it never responds, + // so the client's own 5 s timeout is what ends the call. + await client + .collections(collection) + .documents() + .import( + [{ id: 'arrayed', credit: [{ role: [etser], agent_id: [rembrandt] }] }], + { action: 'create' }, + ); + + // Either condition alone still answers instantly over the same document. + expect(await found(`credit.role:=\`${etser}\``)).toContain('arrayed'); + + await expect(found(welded(etser, rembrandt))).rejects.toThrow(); + }, 60_000); +}); diff --git a/packages/search-typesense/vite.config.ts b/packages/search-typesense/vite.config.ts index 5015fb5f..61247c5e 100644 --- a/packages/search-typesense/vite.config.ts +++ b/packages/search-typesense/vite.config.ts @@ -28,7 +28,7 @@ export default mergeConfig( // projection naming what no lookup reaches are unreachable through // the port, since `assertValidQuery` rejects such a query first. // They hold for a direct caller, and are exercised as one. - branches: 95.69, + branches: 95.87, statements: 99.48, }, }, diff --git a/packages/search/src/adapter.ts b/packages/search/src/adapter.ts index 3437e99f..7d66cd92 100644 --- a/packages/search/src/adapter.ts +++ b/packages/search/src/adapter.ts @@ -40,6 +40,7 @@ export { labelTargetNameOf, documentKeyOf, DEFAULT_LABEL_FIELD, + DEFAULT_MAX_ENTRIES, isRangeFacet, isAbsoluteIri, isoToUnixSeconds, diff --git a/packages/search/src/project.ts b/packages/search/src/project.ts index 1a4e32ee..e8c519fa 100644 --- a/packages/search/src/project.ts +++ b/packages/search/src/project.ts @@ -7,6 +7,7 @@ import { } from './frame-by-type.js'; import { assertTypeInSchema, + DEFAULT_MAX_ENTRIES, displayFieldName, documentKeyOf, fieldNamed, @@ -155,11 +156,12 @@ function projectFields( searchType: SearchType, schema: SearchSchema | undefined, context: ProjectionContext, + nested = false, ): ProjectedNode { const id = documentIdOf(node, searchType); const document: ProjectedNode = id === undefined ? {} : { id }; for (const field of searchType.fields) { - applyField(document, node, field, searchType, schema, context); + applyField(document, node, field, searchType, schema, context, nested); } return document; } @@ -260,6 +262,7 @@ function applyField( searchType: SearchType, schema: SearchSchema | undefined, context: ProjectionContext, + nested: boolean, ): void { // The three value sources, mutually exclusive by declaration // (`validateSearchType`): a projection value, a computed value, a graph path. @@ -319,7 +322,15 @@ function applyField( // project nothing rather than fall through and emit the referent IRIs under // the field name (the wrong shape). if (schema !== undefined) { - applyInlineReference(document, node, alias, field, schema, context); + applyInlineReference( + document, + node, + alias, + field, + schema, + context, + nested, + ); } return; } @@ -342,7 +353,7 @@ function applyField( // a filter can reach – an engine welds conditions on an entry's LEAF // fields only. `filterable` therefore fans out the id beside the object, // exactly as an inline reference's identity companion does. - applyLocalIdentity(document, endpoints, field, schema); + applyLocalIdentity(document, endpoints, field, schema, nested); return; } } @@ -644,6 +655,7 @@ function applyInlineReference( field: ReferenceField & { readonly ref: { readonly typeName: string } }, schema: SearchSchema, context: ProjectionContext, + nested: boolean, ): void { // Resolves for a schema that declares the referent (always so for the schema a // type is projected through); a type framed against a foreign schema that @@ -660,7 +672,7 @@ function applyInlineReference( schema, context, ); - applyIdentityCompanion(document, referents, field, schema); + applyIdentityCompanion(document, referents, field, schema, nested); } /** @@ -684,6 +696,7 @@ function applyIdentityCompanion( referents: readonly ProjectedNode[], field: ReferenceField, schema: SearchSchema, + nested: boolean, ): void { const names = physicalFields(field, schema); if (names.identity === undefined) { @@ -707,14 +720,51 @@ function applyIdentityCompanion( if (ids.length === 0) { return; } - setArray(document, names.identity, ids); + setIdentity(document, names.identity, ids, field, nested); // Same rule as every other facetable reference: where the target declares a // facet policy, the facet reads a narrowed companion of its own, so an // excluded id is never seen by the engine rather than merely unlabelled. const policy = inheritedFacetKeys(field, schema); if (policy !== undefined) { - setArray(document, names.facet as string, ids.filter(policy.only)); + setIdentity( + document, + names.facet as string, + ids.filter(policy.only), + field, + nested, + ); + } +} + +/** + * Write an identity companion under the arity of the reference it belongs to – + * `array` decides the shape here exactly as it does for every other kind + * ({@link applyFacet}). + * + * It matters most where the companion is a **nested** leaf. That is the field a + * weld actually names – the endpoint's own id is a level deeper than a weld can + * reach – and a weld asks whether ONE entry satisfies every condition. A + * companion holding a list inside an entry stands for each of its ids at once, + * so the entry answers the weld with none of them; a Typesense 30.2 engine does + * not answer at all, and hangs (ADR 26). The entry fans out instead + * ({@link tuplesOf}), which leaves exactly one id per entry for this to write. + * + * A top-level companion is unaffected: it is a flat field standing for the whole + * document rather than for one entry, so an `array` reference's companion holds + * every id its entries reference, as it always has. + */ +function setIdentity( + document: ProjectedNode, + name: string, + ids: readonly string[], + field: ReferenceField, + nested: boolean, +): void { + if (!nested || field.array === true) { + setArray(document, name, ids); + return; } + setString(document, name, ids[0]); } /** @@ -730,6 +780,7 @@ function applyLocalIdentity( endpoints: readonly ProjectedNode[], field: ReferenceField, schema: SearchSchema, + nested: boolean, ): void { const names = physicalFields(field, schema); if (names.identity === undefined) { @@ -744,9 +795,7 @@ function applyLocalIdentity( .map((endpoint) => endpoint.id) .filter((id): id is string => typeof id === 'string'), ); - if (ids.length > 0) { - setArray(document, names.identity, ids); - } + setIdentity(document, names.identity, ids, field, nested); } /** The id a value under an identity field carries: the value itself when the @@ -781,9 +830,26 @@ function applyNestedReferents( schema: SearchSchema, context: ProjectionContext, ): readonly ProjectedNode[] { - const referents = values - .filter(isObject) - .map((referent) => projectFields(referent, nestedType, schema, context)) + // One node may stand for several entries: a weldable leaf is single-valued, + // so an edge the graph gave several roles or several endpoints fans out into + // one entry per combination BEFORE it is projected (ADR 26). The budget spans + // every edge THIS node states, rather than resetting per edge – a node holds + // as many edges as the graph gives it, so a per-edge cap would still let the + // entries grow with the input. + const limit = isInlineReference(field) + ? (field.ref.maxEntries ?? DEFAULT_MAX_ENTRIES) + : Number.POSITIVE_INFINITY; + const nodes: FramedNode[] = []; + for (const value of values.filter(isObject)) { + if (nodes.length >= limit) { + break; + } + nodes.push(...tuplesOf(value, nestedType, field, limit - nodes.length)); + } + const referents = nodes + .map((referent) => + projectFields(referent, nestedType, schema, context, true), + ) // Fields, not identity, are what makes something a referent: a literal // value object under the alias (dirty source data), or a node this // reference type reads nothing from, projects nothing and is no referent. @@ -797,6 +863,80 @@ function applyNestedReferents( return referents; } +/** + * Split one framed edge node into the entries it stands for: the cartesian + * product of its **weldable** leaves’ values, one value each. + * + * A weld asks whether ONE entry satisfies every condition, so a leaf a weld can + * name (`filterable` – `searchSchema` refuses `array` on one) states a single + * value. An edge the graph gave two roles and two endpoints is therefore four + * entries rather than one entry holding two lists, which stands for all four at + * once and answers the weld with none of them. See + * [ADR 26](../../docs/decisions/0026-fan-out-a-qualified-edge-into-one-entry-per-tuple.md). + * + * Done on the **framed node**, before projection, so each leaf reaches + * {@link applyField} single-valued and passes through `transform`, folding and + * the facet companion exactly as a single-valued field always has – no + * downstream step learns that fan-out happened. Only the weldable aliases are + * split: an `output`-only leaf keeps its list, because nothing welds it, and it + * is shared unchanged across the entries the node fans out to. + * + * Inline references only. A {@link ReferenceStrategy.local local} lookup runs + * through this same body but nests the endpoint’s **own Root Type**, whose + * fields are multi-valued for reasons of their own – fanning one out would + * split a person across their `sameAs` values. What a weld names there is the + * flat `${name}_id` companion, which {@link applyLocalIdentity} already writes + * beside the object. + * + * `limit` is what remains of the document’s entry budget + * ({@link ReferenceStrategy.maxEntries}, {@link DEFAULT_MAX_ENTRIES}), so the + * product stops growing mid-way rather than being built and then trimmed – a + * bound in the data’s own units is not a bound (ADR 12), and one pathological + * edge would otherwise multiply a document until the run dies. + */ +function tuplesOf( + node: FramedNode, + nestedType: SearchType, + field: ReferenceField, + limit: number, +): readonly FramedNode[] { + if (!isInlineReference(field)) { + return [node]; + } + const weldable = nestedType.fields + .filter((nested) => nested.filterable === true) + .map((nested) => irAlias(nestedType, nested)) + // A leaf the frame carries at most one value for is already a tuple + // position; splitting it would copy the node to no purpose. + .filter((alias) => valuesOf(node, alias).length > 1); + if (weldable.length === 0) { + return [node]; + } + let tuples: FramedNode[] = [node]; + for (const alias of weldable) { + const values = valuesOf(node, alias); + const grown: FramedNode[] = []; + for (const tuple of tuples) { + if (grown.length >= limit) { + break; + } + for (const value of values) { + if (grown.length >= limit) { + break; + } + grown.push({ ...tuple, [alias]: value }); + } + } + // Capped per alias rather than by returning from inside this loop, so every + // tuple that survives is split across EVERY weldable alias. Returning early + // would hand back tuples whose remaining aliases still held their lists, + // and a single-valued leaf then keeps the first value and drops the rest – + // silent data loss in place of the fan-out this exists to perform. + tuples = grown; + } + return tuples; +} + // --- Framed-IR readers: read a field’s value off the framed node by its // {@link irAlias IR Alias} key. Internal to projection – a `derive` reads the // projected document, never the node, so `path` stays the whole statement of diff --git a/packages/search/src/schema.ts b/packages/search/src/schema.ts index 9386bc64..6a605342 100644 --- a/packages/search/src/schema.ts +++ b/packages/search/src/schema.ts @@ -293,6 +293,30 @@ export type ReferenceStrategy = * keyed on a label, so two endpoints that share a label are never merged. */ readonly identity?: string; + /** + * Cap on the entries this reference stores **per node carrying it** – + * across every edge that node states, and so per document for a + * reference declared on a Root Type. Entries past it are dropped. + * Defaults to 100 ({@link DEFAULT_MAX_ENTRIES}). Must be a positive + * integer: the budget is counted off one entry at a time, so a cap that + * is not one can never be reached. + * + * A reference nested inside another reference type is budgeted per + * *parent entry* rather than per document, so a schema nesting one edge + * inside another admits up to the product of their caps. Bounded, but + * multiplicatively: set the inner cap with the outer one in mind. + * + * A weldable leaf is single-valued, so an edge whose graph values are + * multi-valued fans out into one entry per combination + * ([ADR 26](../../docs/decisions/0026-fan-out-a-qualified-edge-into-one-entry-per-tuple.md)). + * That product is bounded by the edge’s own data, and a bound in the + * data’s own units is not a bound + * ([ADR 12](../../docs/decisions/0012-bound-memory-by-the-unit-of-work-not-the-input.md)): + * one pathological edge would otherwise multiply a document until the run + * dies. Raise it for a corpus whose edges are legitimately wide; the + * default is far above what a qualified relation produces in practice. + */ + readonly maxEntries?: number; }; /** An IRI-valued reference to another entity, resolved at the surface. */ @@ -1206,10 +1230,20 @@ function assertNoInlineCycle( * Field}, the reading device that is the other half of an inline reference’s * job. * + * **`filterable` with `array` is refused**, for a reason about meaning rather + * than about any engine: a weld asks whether ONE entry satisfies every + * condition, so a leaf a weld can name states one value per entry. A leaf + * holding a list stands for every combination at once and answers the weld with + * none of them. Multiplicity belongs to the entry list instead – the projection + * emits one entry per combination + * ({@link ReferenceStrategy.maxEntries maxEntries}). An `output`-only nested + * leaf is untouched: nothing welds it, so it may carry a list for display. + * * Checked schema-wide, like the label sources and for the same reason: a single * declaration cannot see whether it is a Reference Type at all. * - * See [ADR 24](../../docs/decisions/0024-carry-data-on-a-reference-edge.md). + * See [ADR 24](../../docs/decisions/0024-carry-data-on-a-reference-edge.md) and + * [ADR 26](../../docs/decisions/0026-fan-out-a-qualified-edge-into-one-entry-per-tuple.md). */ function assertServiceableNestedFields( referenceTypes: ReadonlyMap, @@ -1233,6 +1267,11 @@ function assertServiceableNestedFields( `Nested field “${referenceType.name}.${field.name}” declares a label source, which an inline reference cannot serve; declare a “lookup” on the nested reference instead of resolving a label for it.`, ); } + if (field.filterable === true && field.array === true) { + throw new Error( + `Nested field “${referenceType.name}.${field.name}” declares both “filterable” and “array”: a weld asks whether ONE entry satisfies every condition, and an entry holding a list has no single value to test – it stands for each combination at once, so the weld degenerates into the cross-product it exists to exclude. Declare the field single-valued; the projection emits one entry per combination (see “maxEntries”).`, + ); + } } } } @@ -1288,6 +1327,14 @@ const UNSERVICEABLE_INLINE_ROLES = ['searchable', 'sortable'] as const; * {@link SearchTypeBase.labelField}. */ export const DEFAULT_LABEL_FIELD = 'label'; +/** + * Entries one document stores for an inline reference that declares no + * {@link ReferenceStrategy.maxEntries maxEntries} of its own. Well above what a + * qualified relation produces – a measured corpus averages under four – so the + * default bounds the pathological case without truncating a real one. + */ +export const DEFAULT_MAX_ENTRIES = 100; + /** The `name` the type serves its label under: its declared * {@link SearchTypeBase.labelField}, else `label`. */ export function labelFieldNameOf(searchType: SearchType): string { @@ -1407,6 +1454,7 @@ export interface SearchTypeIssue { | 'invalid-locale' | 'missing-ref' | 'missing-ref-type-name' + | 'invalid-max-entries' | 'ref-not-allowed' | 'text-requires-locales' | 'locales-not-allowed' @@ -1610,6 +1658,18 @@ export function validateSearchType( ) { issue('missing-ref-type-name'); } + // The fan-out budget is counted off one entry at a time, so a fractional + // or non-positive cap is never reached and the cartesian product in + // `tuplesOf` grows with the data – exactly the unbounded case the cap + // exists to prevent (ADR 12). A cap that cannot bind is worse than none, + // because the declaration says otherwise. + if ( + field.ref?.strategy === 'inline' && + field.ref.maxEntries !== undefined && + !(Number.isInteger(field.ref.maxEntries) && field.ref.maxEntries > 0) + ) { + issue('invalid-max-entries'); + } // A join addresses the referent's collection – the one a lookup's // `target` or an idOnly's `labelSource` names. With neither, the flag // states an edge to nowhere. diff --git a/packages/search/test/qualified-relation.test.ts b/packages/search/test/qualified-relation.test.ts index 9a91dff0..bac4463e 100644 --- a/packages/search/test/qualified-relation.test.ts +++ b/packages/search/test/qualified-relation.test.ts @@ -146,6 +146,256 @@ describe('an edge that carries data and resolves a lookup', () => { }); }); +describe('fanning an edge out into one entry per tuple', () => { + // A weld asks whether ONE entry satisfies every condition, so a leaf a weld + // can name holds one value. An edge the graph gave several fans out (ADR 26). + + /** Both leaves weldable, so both are tuple positions. */ + const weldableEdge = defineSearchType({ + name: 'CreatorEdge', + fields: [ + { + name: 'role', + kind: 'keyword', + path: `${SCHEMA_ORG}name`, + output: true, + filterable: true, + }, + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + output: true, + filterable: true, + ref: { strategy: 'lookup', target: 'Person', local: true }, + }, + ], + }); + + const twoRoles = { + '@id': 'https://ex/work/2', + [workKey('creator')]: [ + { + [edgeKey('role')]: [{ '@value': 'etser' }, { '@value': 'drukker' }], + [edgeKey('creator')]: [ + { '@id': 'https://a/1', [personKey('sameAs')]: [{ '@id': RKD }] }, + ], + }, + ], + }; + + it('splits a multi-valued weldable leaf across entries', () => { + const entries = entriesOf(projectDocument(twoRoles, work, schema)); + + expect(entries).toHaveLength(2); + expect(entries.map((entry) => entry.role)).toEqual(['etser', 'drukker']); + // Every entry keeps the endpoint the edge stated: the tuple is what fans + // out, not the edge's other values. + expect( + entries.map((entry) => (entry.creator as SearchDocument).id), + ).toEqual([RKD, RKD]); + }); + + it('takes the product where two weldable leaves are multi-valued', () => { + const twoOfEach = { + '@id': 'https://ex/work/3', + [workKey('creator')]: [ + { + [edgeKey('role')]: [{ '@value': 'etser' }, { '@value': 'drukker' }], + [edgeKey('creator')]: [ + { '@id': 'https://a/1', [personKey('sameAs')]: [{ '@id': RKD }] }, + { '@id': 'https://a/2' }, + ], + }, + ], + }; + const entries = entriesOf( + projectDocument( + twoOfEach, + work, + searchSchema(work, person, weldableEdge), + ), + ); + + expect(entries.map((entry) => [entry.role, entry.creator_id])).toEqual([ + ['etser', RKD], + ['etser', 'https://a/2'], + ['drukker', RKD], + ['drukker', 'https://a/2'], + ]); + }); + + it('leaves an output-only list on the entry', () => { + // Nothing welds it, so it needs no tuple position – and splitting the entry + // over it would multiply entries for a value no filter can name. + const noteEdge = defineSearchType({ + name: 'CreatorEdge', + fields: [ + { + name: 'role', + kind: 'keyword', + path: `${SCHEMA_ORG}name`, + output: true, + filterable: true, + }, + { + name: 'note', + kind: 'keyword', + path: `${SCHEMA_ORG}description`, + array: true, + output: true, + }, + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + output: true, + ref: { strategy: 'lookup', target: 'Person', local: true }, + }, + ], + }); + const annotated = { + '@id': 'https://ex/work/6', + [workKey('creator')]: [ + { + [edgeKey('role')]: [{ '@value': 'etser' }], + [edgeKey('note')]: [ + { '@value': 'gesigneerd' }, + { '@value': 'ovaal' }, + ], + }, + ], + }; + const entries = entriesOf( + projectDocument(annotated, work, searchSchema(work, person, noteEdge)), + ); + + expect(entries).toHaveLength(1); + expect(entries[0].note).toEqual(['gesigneerd', 'ovaal']); + }); + + it('stops at the cap once earlier edges have spent it', () => { + // The cap bounds the DOCUMENT, not each edge: a document holds as many + // edges as the graph states, so a per-edge cap would still let the entries + // grow with the input (ADR 12). `node` states two edges; a budget of one + // is spent by the first, and the second contributes nothing. + const cappedWork = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + array: true, + output: true, + filterable: true, + ref: { + strategy: 'inline', + typeName: 'CreatorEdge', + identity: 'creator', + maxEntries: 1, + }, + }, + ], + }); + const entries = entriesOf( + projectDocument( + node, + cappedWork, + searchSchema(cappedWork, person, creatorEdge), + ), + ); + + expect(entries).toHaveLength(1); + expect(entries[0].role).toBe('etser'); + }); + + it('fully expands every entry it keeps when the cap bites', () => { + // The cap drops whole tuples, never half-expanded ones. Capping by + // returning mid-expansion would hand back entries whose remaining weldable + // leaves still held their lists – and a single-valued leaf then keeps the + // first value and drops the rest, losing data silently instead of fanning + // it out. + const cappedWork = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + array: true, + output: true, + filterable: true, + ref: { + strategy: 'inline', + typeName: 'CreatorEdge', + identity: 'creator', + maxEntries: 3, + }, + }, + ], + }); + const wide = { + '@id': 'https://ex/work/9', + [workKey('creator')]: [ + { + [edgeKey('role')]: [ + { '@value': 'etser' }, + { '@value': 'drukker' }, + { '@value': 'uitgever' }, + ], + [edgeKey('creator')]: [ + { '@id': 'https://a/1', [personKey('sameAs')]: [{ '@id': RKD }] }, + { '@id': 'https://a/2' }, + ], + }, + ], + }; + const entries = entriesOf( + projectDocument( + wide, + cappedWork, + searchSchema(cappedWork, person, weldableEdge), + ), + ); + + expect(entries).toHaveLength(3); + // Both leaves participate in every kept tuple; none is pinned to its first + // value because expansion stopped early. + expect(entries.map((entry) => [entry.role, entry.creator_id])).toEqual([ + ['etser', RKD], + ['etser', 'https://a/2'], + ['drukker', RKD], + ]); + }); + + it('does not fan a local lookup out over the endpoint’s own fields', () => { + // A `local` lookup nests the endpoint's own Root Type, whose fields are + // multi-valued for reasons of their own – `sameAs` here. Splitting on those + // would scatter one person across entries; what a weld names is the flat + // companion beside the object. + const twoAlignments = { + '@id': 'https://ex/work/7', + [workKey('creator')]: [ + { + [edgeKey('role')]: [{ '@value': 'etser' }], + [edgeKey('creator')]: [ + { + '@id': 'https://a/1', + [personKey('sameAs')]: [{ '@id': RKD }, { '@id': 'https://a/9' }], + }, + ], + }, + ], + }; + const entries = entriesOf(projectDocument(twoAlignments, work, schema)); + + expect(entries).toHaveLength(1); + }); +}); + describe('the identity companion', () => { it('harvests the ids the entries reference', () => { // The flat field an engine filters and facets in the nested object’s @@ -428,6 +678,131 @@ describe('welding conditions to one entry', () => { }); }); +describe('nesting is where a node is projected, not what type it is', () => { + // A Root Type reached by a `local` lookup is nested exactly as a Reference + // Type is – it just happens to have a collection of its own elsewhere. Its + // companions must therefore be written under the nested rule too. Deciding + // that from the TYPE rather than from the projection context reads a + // locally-nested root as a root, and the arity it writes then disagrees with + // the one the collection declares: the import fails for every such document. + const inner = defineSearchType({ + name: 'Membership', + fields: [ + { + name: 'org', + kind: 'reference', + path: `${SCHEMA_ORG}memberOf`, + output: true, + ref: { strategy: 'lookup', target: 'Person' }, + }, + ], + }); + const nestedRoot = defineSearchType({ + name: 'Person', + class: `${SCHEMA_ORG}Person`, + key: { field: 'sameAs' }, + fields: [ + { + name: 'label', + kind: 'text', + path: `${SCHEMA_ORG}name`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: 'sameAs', + kind: 'reference', + path: `${SCHEMA_ORG}sameAs`, + array: true, + }, + { + name: 'affiliation', + kind: 'reference', + path: `${SCHEMA_ORG}affiliation`, + output: true, + filterable: true, + ref: { strategy: 'inline', typeName: 'Membership', identity: 'org' }, + }, + ], + }); + const work = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + output: true, + ref: { strategy: 'lookup', target: 'Person', local: true }, + }, + ], + }); + + it('writes a single-valued companion inside a locally-nested root type', () => { + const node = { + '@id': 'https://ex/work/10', + [workKey('creator')]: [ + { + '@id': 'https://p/1', + [personKey('sameAs')]: [{ '@id': RKD }], + [alias('Person', 'affiliation')]: [ + { [alias('Membership', 'org')]: [{ '@id': 'https://o/1' }] }, + ], + }, + ], + }; + const document = projectDocument( + node, + work, + searchSchema(work, nestedRoot, inner), + ); + const endpoint = document.creator as SearchDocument; + + // A single value, matching what the collection declares for this path - + // not the one-element list a root-level companion would carry. + expect(endpoint.affiliation_id).toBe('https://o/1'); + }); +}); + +describe('a local lookup at the root', () => { + it('harvests every endpoint into the flat companion', () => { + // A top-level companion stands for the whole DOCUMENT rather than for one + // entry, so nothing welds it and an `array` reference's companion holds + // every id its endpoints carry – unchanged by the nested rule (ADR 26). + const rootLookup = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + array: true, + output: true, + filterable: true, + ref: { strategy: 'lookup', target: 'Person', local: true }, + }, + ], + }); + const twoEndpoints = { + '@id': 'https://ex/work/8', + [workKey('creator')]: [ + { '@id': 'https://a/1', [personKey('sameAs')]: [{ '@id': RKD }] }, + { '@id': 'https://a/2' }, + ], + }; + const document = projectDocument( + twoEndpoints, + rootLookup, + searchSchema(rootLookup, person), + ); + + expect(document.creator_id).toEqual([RKD, 'https://a/2']); + }); +}); + describe('the identity companion of a local lookup', () => { // Its own id is a level deeper than a condition can be welded to, so // `filterable` fans it out as a leaf beside the stored object. @@ -460,49 +835,60 @@ describe('the identity companion of a local lookup', () => { ); const [identified] = document.creator as readonly SearchDocument[]; - expect(identified.creator_id).toEqual([RKD]); + expect(identified.creator_id).toBe(RKD); expect((identified.creator as SearchDocument).id).toBe(RKD); }); - it('holds only the endpoint a single-valued reference stores', () => { - // A single-valued reference keeps the first endpoint and drops the rest; - // a companion holding a dropped one's id would match a filter whose hit - // then shows a different endpoint. - const twoEndpoints = { - '@id': 'https://ex/work/4', - [workKey('creator')]: [ - { - [edgeKey('creator')]: [ - { '@id': 'https://a/1', [personKey('sameAs')]: [{ '@id': RKD }] }, - { '@id': 'https://a/2' }, - ], - }, - ], - }; - const singleEndpointEdge = defineSearchType({ - name: 'CreatorEdge', + it('holds only the entries the cap admits', () => { + // The one place entries are still dropped: a companion holding an id from + // a dropped entry would match a filter whose hit then shows no such entry. + const cappedWork = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, fields: [ { name: 'creator', kind: 'reference', path: `${SCHEMA_ORG}creator`, + array: true, output: true, filterable: true, - ref: { strategy: 'lookup', target: 'Person', local: true }, + ref: { + strategy: 'inline', + typeName: 'CreatorEdge', + identity: 'creator', + maxEntries: 2, + }, }, ], }); + const threeEndpoints = { + '@id': 'https://ex/work/4', + [workKey('creator')]: [ + { + [edgeKey('creator')]: [ + { '@id': 'https://a/1', [personKey('sameAs')]: [{ '@id': RKD }] }, + { '@id': 'https://a/2' }, + { '@id': 'https://a/3' }, + ], + }, + ], + }; const document = projectDocument( - twoEndpoints, - work, - searchSchema(work, person, singleEndpointEdge), + threeEndpoints, + cappedWork, + searchSchema(cappedWork, person, filterableEdge), ); - const [entry] = document.creator as readonly SearchDocument[]; - expect(entry.creator_id).toEqual([RKD]); + expect(document.creator).toHaveLength(2); + expect(document.creator_id).toEqual([RKD, 'https://a/2']); }); - it('holds every endpoint a multi-valued reference stores', () => { + it('fans a multi-valued endpoint out into one entry per endpoint', () => { + // The endpoint is what a weld names, so it is single-valued per entry: an + // edge the graph gave two endpoints is two entries, not one entry holding + // both. One entry holding both stands for either pairing and answers the + // weld with neither (ADR 26). const jointEdge = defineSearchType({ name: 'CreatorEdge', fields: [ @@ -510,7 +896,6 @@ describe('the identity companion of a local lookup', () => { name: 'creator', kind: 'reference', path: `${SCHEMA_ORG}creator`, - array: true, output: true, filterable: true, ref: { strategy: 'lookup', target: 'Person', local: true }, @@ -533,9 +918,13 @@ describe('the identity companion of a local lookup', () => { work, searchSchema(work, person, jointEdge), ); - const [entry] = document.creator as readonly SearchDocument[]; + const entries = document.creator as readonly SearchDocument[]; - expect(entry.creator_id).toEqual([RKD, 'https://a/2']); + expect(entries).toHaveLength(2); + expect(entries.map((entry) => entry.creator_id)).toEqual([ + RKD, + 'https://a/2', + ]); }); it('is absent where the endpoint is not identified', () => { diff --git a/packages/search/test/schema.test.ts b/packages/search/test/schema.test.ts index ea70f23c..4b1a12c3 100644 --- a/packages/search/test/schema.test.ts +++ b/packages/search/test/schema.test.ts @@ -1141,7 +1141,6 @@ describe('searchSchema validation', () => { { name: 'contentUrl', kind: 'keyword', - array: true, output: true, path: 'https://schema.org/contentUrl', ...field, @@ -1231,6 +1230,63 @@ describe('searchSchema validation', () => { ).not.toThrow(); }); + it.each([ + ['a fraction', 2.5], + ['zero', 0], + ['a negative', -1], + ])('rejects maxEntries that is %s', (_label, maxEntries) => { + // The budget is counted off one entry at a time, so a cap that is not a + // positive integer is never reached and the fan-out grows with the data – + // the unbounded case the cap exists to prevent. A cap that cannot bind is + // worse than none, because the declaration claims otherwise. + expect(() => + searchSchema( + datasetNesting({ + strategy: 'inline', + typeName: 'MediaObject', + maxEntries, + }), + mediaObjectWith({}), + ), + ).toThrow(/invalid-max-entries/u); + }); + + it('accepts a positive integer maxEntries', () => { + expect(() => + searchSchema( + datasetNesting({ + strategy: 'inline', + typeName: 'MediaObject', + maxEntries: 5, + }), + mediaObjectWith({}), + ), + ).not.toThrow(); + }); + + it('rejects a nested field declaring both filterable and array', () => { + // A weld asks whether ONE entry satisfies every condition, so a leaf a + // weld can name holds one value. A leaf holding a list stands for every + // combination at once and answers the weld with none of them – the + // projection fans the entry out instead (ADR 26). + expect(() => + searchSchema( + datasetNesting({ strategy: 'inline', typeName: 'MediaObject' }), + mediaObjectWith({ filterable: true, array: true }), + ), + ).toThrow(/declares both “filterable” and “array”/u); + }); + + it('accepts an output-only nested field declaring array', () => { + // Nothing welds it, so an entry may carry a list for display. + expect(() => + searchSchema( + datasetNesting({ strategy: 'inline', typeName: 'MediaObject' }), + mediaObjectWith({ array: true }), + ), + ).not.toThrow(); + }); + it.each([ ['searchable', { searchable: { weight: 1 } }], ['sortable', { sortable: true }], diff --git a/packages/search/vite.config.ts b/packages/search/vite.config.ts index 3066345a..92874588 100644 --- a/packages/search/vite.config.ts +++ b/packages/search/vite.config.ts @@ -12,7 +12,7 @@ export default mergeConfig( thresholds: { functions: 100, lines: 100, - branches: 99.71, + branches: 99.72, statements: 100, }, },