Since #785, a canonically keyed collection records which datasets reference each document, in a private referenced_by: [{ dataset, run }]. That column holds the answer to a question consumers can currently ask of no keyed type: which datasets reference this term – and its inverse, show me the Getty terms this publisher uses.
It is also the honest version of a question ADR 25 had to forbid. A keyed type may not declare a from: 'dataset' field, because on a shared document that field would hold whichever dataset wrote last and a filter on any of the others would silently miss it. referenced_by has no such problem: it is the whole set, and it is correct by construction.
What is already true
Nothing about the engine or the stored data needs to change. referenced_by.dataset is declared facet: true, and test/membership-filters.integration.test.ts pins both halves against a real Typesense:
facet_by=referenced_by.dataset returns one bucket per referring dataset;referenced_by.dataset:=[A,C] filters by containment.
What is missing is only the declaration a deployment writes and the surface it produces. Responses are assembled from outputFields(searchType), so today the column is invisible by design.
The crux: a field whose values come from bookkeeping
This would be the first declared field whose values come from the writer rather than from the graph.
Every field today is projection-populated; from: 'dataset' is filled by the projection and merely reasserted by the writer, which is why the writer's stamp “only ever reasserts it”. Membership has no projection source at all – the writer is the only thing that writes it, and the projection must never touch it. So the design question is how @lde/search expresses that category, and what a deployment writes to opt in. Worth settling before any code: it is a new kind of field, not a new field.
Naming should stay in the voice ADR 25 established – which datasets reference this document, plural – and never dataset, singular, which is the conflation the ADR removed.
Scope
In: faceting and filtering a keyed type by its referring datasets, with label resolution through the existing lookup strategy (the stored values are dataset IRIs, which are ids in the datasets collection – ADR 20 already covers this).
Out (for a first cut): a joinable membership reference. Typesense carries reference: on top-level fields only (collection-definition.ts:226), so cross-collection filtering through membership would need a flat companion beside the nested column – the identity-companion pattern inline references already use (${name}_id, plus a narrowed ${name}_facet where the target declares a facetKeys policy). Known move, but it earns its own issue if a consumer asks for it.
Unaffected: the sweep. maxSweepableSources faceting is the writer's own bookkeeping and is untouched by anything a query does.
Since #785, a canonically keyed collection records which datasets reference each document, in a private
referenced_by: [{ dataset, run }]. That column holds the answer to a question consumers can currently ask of no keyed type: which datasets reference this term – and its inverse, show me the Getty terms this publisher uses.It is also the honest version of a question ADR 25 had to forbid. A keyed type may not declare a
from: 'dataset'field, because on a shared document that field would hold whichever dataset wrote last and a filter on any of the others would silently miss it.referenced_byhas no such problem: it is the whole set, and it is correct by construction.What is already true
Nothing about the engine or the stored data needs to change.
referenced_by.datasetis declaredfacet: true, andtest/membership-filters.integration.test.tspins both halves against a real Typesense:facet_by=referenced_by.datasetreturns one bucket per referring dataset;referenced_by.dataset:=[A,C]filters by containment.What is missing is only the declaration a deployment writes and the surface it produces. Responses are assembled from
outputFields(searchType), so today the column is invisible by design.The crux: a field whose values come from bookkeeping
This would be the first declared field whose values come from the writer rather than from the graph.
Every field today is projection-populated;
from: 'dataset'is filled by the projection and merely reasserted by the writer, which is why the writer's stamp “only ever reasserts it”. Membership has no projection source at all – the writer is the only thing that writes it, and the projection must never touch it. So the design question is how@lde/searchexpresses that category, and what a deployment writes to opt in. Worth settling before any code: it is a new kind of field, not a new field.Naming should stay in the voice ADR 25 established – which datasets reference this document, plural – and never
dataset, singular, which is the conflation the ADR removed.Scope
In: faceting and filtering a keyed type by its referring datasets, with label resolution through the existing
lookupstrategy (the stored values are dataset IRIs, which are ids in the datasets collection – ADR 20 already covers this).Out (for a first cut): a joinable membership reference. Typesense carries
reference:on top-level fields only (collection-definition.ts:226), so cross-collection filtering through membership would need a flat companion beside the nested column – the identity-companion pattern inline references already use (${name}_id, plus a narrowed${name}_facetwhere the target declares afacetKeyspolicy). Known move, but it earns its own issue if a consumer asks for it.Unaffected: the sweep.
maxSweepableSourcesfaceting is the writer's own bookkeeping and is untouched by anything a query does.