Skip to content

[core] Fall back to raw vector search when scalar index cannot fully evaluate filter - #9884

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/vector-scalar-prefilter-unsupported
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/vector-scalar-prefilter-unsupported

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #9883

Vector search with a WHERE filter delegates pre-filtering to the scalar global index. The index returns all-or-nothing bitmaps, and the index read path has no final-read filter to correct them, so it trusts each bitmap as the exact match set. Two cases broke that assumption:

  • The index could not evaluate the predicate at all: no scalar index files, or an unsupported function such as IS NOT NULL on a multivalue-indexed array column. scalarMatchedRows returned an empty bitmap, which was AND-ed into every index split, so every index-covered row silently vanished from the result.
  • The index could evaluate only some of the filter's fields, for example a = 1 AND tags IS NOT NULL where the array conjunct is unsupported. The evaluator drops that conjunct and returns a superset matching a = 1 alone, which polluted the vector top-K with non-matching rows and displaced matching ones.

scalarMatchedRows now returns null, meaning "cannot decide exactly", in both cases: when scanWithCoverage finds nothing, and when the contributing fields do not cover every field the filter references. demoteUncoverableIndexSplits then routes the ranges no raw split already covers through the raw search, where the exact final-read filter decides; ranges an existing raw split covers keep the current dedup. The computed scalar pre-filter is cached for preFilters to reuse. Every reader entry point (local, batch, Spark, Flink) demotes right after splitting its splits.

Tests

Added two cases to VectorSearchBuilderTest, both asserting the exact row ids the search returns:

  • testVectorSearchFilterIndexCannotEvaluate uses a filter the index cannot evaluate at all (IS NOT NULL on a multivalue-indexed array). The old empty-bitmap handling returns an empty result; the fallback returns the closest rows that match the filter.
  • testVectorSearchFilterPartiallyEvaluableAnd uses a partially-evaluable AND (id >= 0 AND tags IS NOT NULL). The old superset handling puts a row that fails the filter into the top-K; the fallback returns only matching rows.

…evaluate filter

Vector search with a WHERE filter delegates pre-filtering to the scalar
global index. The index answers with all-or-nothing bitmaps and the index
read path has no final-read filter, so the bitmap is trusted as the exact
match set. Two cases broke that assumption:

- The index could not evaluate the predicate at all (no scalar index
  files, or an unsupported function such as IS NOT NULL on a
  multivalue-indexed array column). scalarMatchedRows returned an empty
  bitmap, which was AND-ed into every index split, so all index-covered
  rows silently vanished from the result.

- The index could evaluate only some of the filter's fields, e.g.
  "a = 1 AND tags IS NOT NULL" where the array conjunct is unsupported.
  The evaluator drops the unsupported conjunct and returns a superset
  matching "a = 1" alone, polluting the vector top-K with non-matching
  rows.

Make scalarMatchedRows return null ("cannot decide exactly") in both
cases: when scanWithCoverage is absent, and when the contributing fields
do not cover every field the filter references. Then route the ranges the
raw splits do not already cover through the raw search, where the exact
final-read filter decides; the scalar pre-filter is cached for preFilters
to reuse. Every reader entry (local, batch, Spark, Flink) demotes right
after splitting its splits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter

1 participant