You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 185650f
Browse filesBrowse the repository at this point in the historyBrowse files
improvement(knowledge): maintain keyword projections only for search indexes (#8333)
- Since #8314, workspace knowledge search ranks on `embedding`/`document`. The only live reader of `embedding_keyword_search` and `embedding_keyword_tin` is dormant indexed search, and only for bases that are search indexes. Yet every chunk write still upserted a keyword row, ran a Tin `DELETE` on insert, and every document insert fanned an ACL/connector sync out to the projections
- Script migration `0025_scope_keyword_projections`, all `CREATE OR REPLACE` in one transaction with the 0024 lock-timeout retry:
- `sync_embedding_keyword_search` takes the shared membership lock and writes keyword rows only for `is_search_index` bases. On an update that moves a chunk out of a search index, it deletes the chunk's row
- A new flip trigger backfills a base's keyword rows when it becomes a search index and deletes them when it stops being one, under the exclusive membership lock. It is separate from the Tin flip trigger, which only exists where Tin is installed. Its upsert skips unchanged rows
- The Tin chunk trigger skips its `DELETE` on insert, since a new chunk can't have a Tin row
- `projection_source_acl_sync` becomes `AFTER UPDATE OF connector_id, acl ... WHEN` the value actually changed. A new document has no chunks yet (FK), so the insert arm never matched anything
- The projector's keyword page writes only for search-index bases, keyed on the base flag
- Fresh installs, `db:push`, and late Tin adoption end with the same trigger definitions: 0019 installs the final guarded, key-share-locked Tin triggers, and 0025 re-runs after 0016. A `db:push` re-run of the legacy 0016 backfill may refill keyword rows for non-search bases; they're unread and left in place like the existing ones
- Both adoption backfills read their chunks `FOR KEY SHARE`, so a chunk delete racing an adoption is waited out instead of failing it
- Rollback floor: v0.9.1 and earlier ranked workspace keyword search through `embedding_keyword_search`. Once 0025 has run, don't roll back below v0.9.2, and self-hosters should upgrade through v0.9.2 or later
- Existing keyword rows for non-search bases are left in place (unread) and removed separately
Copy file name to clipboardExpand all lines: apps/sim/lib/knowledge/__integration__/knowledge-projection.integration.ts
+41-2Lines changed: 41 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -24,6 +24,7 @@ import {
24
24
credentialGroupEnrollment,
25
25
document,
26
26
embedding,
27
+
embeddingKeywordSearch,
27
28
embeddingKeywordTin,
28
29
embeddingSearch,
29
30
knowledgeBase,
@@ -383,7 +384,6 @@ beforeAll(async () => {
383
384
sql.raw(`CREATE SCHEMA tin;
384
385
CREATE FUNCTION tin.full_score(tid) RETURNS double precision LANGUAGE sql IMMUTABLE AS 'SELECT 1.0::float8';
385
386
CREATE FUNCTION knowledge_tin_base_token(text) RETURNS text LANGUAGE sql IMMUTABLE AS $$SELECT 'kb'$$;
386
-
CREATE FUNCTION knowledge_tin_membership_key(text) RETURNS bigint LANGUAGE sql IMMUTABLE AS $$SELECT hashtextextended('embedding_keyword_tin:' || $1, 0)$$;
387
387
CREATE FUNCTION knowledge_tin_stream(vector tsvector) RETURNS text LANGUAGE sql IMMUTABLE AS $$
388
388
SELECT coalesce(string_agg(entry.lexeme, ' ' ORDER BY position), '')
389
389
FROM unnest(vector) AS entry(lexeme, positions, weights), unnest(entry.positions) AS position
@@ -400,7 +400,6 @@ afterAll(async () => {
400
400
sql.raw(`DROP OPERATOR IF EXISTS ==> (text, text);
401
401
DROP FUNCTION IF EXISTS tin_fixture_match(text, text);
402
402
DROP FUNCTION IF EXISTS knowledge_tin_base_token(text);
403
-
DROP FUNCTION IF EXISTS knowledge_tin_membership_key(text);
404
403
DROP FUNCTION IF EXISTS knowledge_tin_stream(tsvector);
Copy file name to clipboardExpand all lines: apps/sim/lib/sim-search/indexed/README.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@ While the gate is off:
14
14
- Indexed-only surfaces refuse with `SearchIndexDormantError` (a `409`): the Stats report and connecting a source that crawls into a search index. The indexed document page is not found.
15
15
- A knowledge search that names a search-index knowledge base (the Knowledge block, v1, v2, Sim's knowledge tool) still answers from the documents it already holds, decided on each document exactly as a workspace knowledge base is.
16
16
- Nothing crawls into search indexes: content syncs, member syncs, and processing recovery skip them (`lib/knowledge/connectors/indexing-policy.ts`).
17
-
- The projector owes search-index documents nothing: their marks are released with the rest, and it writes no Tin keyword rows.
17
+
- The projector owes search-index documents nothing: their marks are released with the rest, and it writes no Tin keyword rows. The GIN keyword projection follows `is_search_index` alone, so it keeps its search-index rows either way.
18
18
19
19
## Layout
20
20
@@ -28,7 +28,7 @@ The dormant UI sits in `indexed/` folders next to the component that picks it fr
28
28
## Re-enabling
29
29
30
30
1. Set `SIM_SEARCH_LIVE=false` in both the app and the Trigger.dev environment, and deploy. The container entrypoint (`apps/sim/bootstrap.ts`) mirrors it to `NEXT_PUBLIC_SIM_SEARCH_LIVE` for the client; crawling, processing, and projection read it in whichever process runs them.
31
-
2. Confirm the Tin objects exist (`0019_tin_keyword_projection`, `0024_knowledge_projection_async`), backfill `embedding_keyword_tin`for every search-index knowledge base, and build its index.
31
+
2. Confirm the keyword projection objects exist (`0019_tin_keyword_projection`, `0024_knowledge_projection_async`, `0025_scope_keyword_projections`). Both keyword projections, `embedding_keyword_search` and `embedding_keyword_tin`, hold only search-index rows, written by the chunk triggers and by the trigger on `knowledge_base.is_search_index`. Backfill both for every search-index knowledge base whose rows were removed while dormant, and build the Tin index.
32
32
3. Resume and fully resync the connectors of search-index knowledge bases, so content that went stale while dormant is indexed again.
33
33
34
34
Projection rows written before projections carried their document's source and ACL are decided on their document until they are rewritten.
0 commit comments