[core] Fall back to raw vector search when scalar index cannot fully evaluate filter - #9884
Open
LuciferYang wants to merge 1 commit into
Open
LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
…evaluate filter
Vector search with a WHERE filter delegates pre-filtering to the scalar
global index. The index answers with all-or-nothing bitmaps and the index
read path has no final-read filter, so the bitmap is trusted as the exact
match set. Two cases broke that assumption:
- The index could not evaluate the predicate at all (no scalar index
files, or an unsupported function such as IS NOT NULL on a
multivalue-indexed array column). scalarMatchedRows returned an empty
bitmap, which was AND-ed into every index split, so all index-covered
rows silently vanished from the result.
- The index could evaluate only some of the filter's fields, e.g.
"a = 1 AND tags IS NOT NULL" where the array conjunct is unsupported.
The evaluator drops the unsupported conjunct and returns a superset
matching "a = 1" alone, polluting the vector top-K with non-matching
rows.
Make scalarMatchedRows return null ("cannot decide exactly") in both
cases: when scanWithCoverage is absent, and when the contributing fields
do not cover every field the filter references. Then route the ranges the
raw splits do not already cover through the raw search, where the exact
final-read filter decides; the scalar pre-filter is cached for preFilters
to reuse. Every reader entry (local, batch, Spark, Flink) demotes right
after splitting its splits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9883
Vector search with a WHERE filter delegates pre-filtering to the scalar global index. The index returns all-or-nothing bitmaps, and the index read path has no final-read filter to correct them, so it trusts each bitmap as the exact match set. Two cases broke that assumption:
IS NOT NULLon a multivalue-indexed array column.scalarMatchedRowsreturned an empty bitmap, which was AND-ed into every index split, so every index-covered row silently vanished from the result.a = 1 AND tags IS NOT NULLwhere the array conjunct is unsupported. The evaluator drops that conjunct and returns a superset matchinga = 1alone, which polluted the vector top-K with non-matching rows and displaced matching ones.scalarMatchedRowsnow returnsnull, meaning "cannot decide exactly", in both cases: whenscanWithCoveragefinds nothing, and when the contributing fields do not cover every field the filter references.demoteUncoverableIndexSplitsthen routes the ranges no raw split already covers through the raw search, where the exact final-read filter decides; ranges an existing raw split covers keep the current dedup. The computed scalar pre-filter is cached forpreFiltersto reuse. Every reader entry point (local, batch, Spark, Flink) demotes right after splitting its splits.Tests
Added two cases to
VectorSearchBuilderTest, both asserting the exact row ids the search returns:testVectorSearchFilterIndexCannotEvaluateuses a filter the index cannot evaluate at all (IS NOT NULLon a multivalue-indexed array). The old empty-bitmap handling returns an empty result; the fallback returns the closest rows that match the filter.testVectorSearchFilterPartiallyEvaluableAnduses a partially-evaluableAND(id >= 0 AND tags IS NOT NULL). The old superset handling puts a row that fails the filter into the top-K; the fallback returns only matching rows.