Skip to content

[core] Search fresh level-0 files in PK vector search full modes - #9896

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/pk-vector-search-level0-files
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/pk-vector-search-level0-files

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #9894

PrimaryKeyVectorRead handed the bucket search only the files PrimaryKeyIndexSourcePolicy.shouldRead accepts (complete COMPACT level>0 outputs), so Level-0 APPEND files, the rows written since the last compaction, were omitted by every search mode, including full and detail, which are documented to exact-scan unindexed data files. This passes all data files of the split (filesToSearch) to the bucket search instead. The exact fallback then covers the files the ANN segments do not, FAST keeps ignoring them, and the ANN state builder (PkVectorBucketIndexState.fromActiveDataFiles) still applies the compact-file policy internally, so the ANN segment state is unchanged and only the exact-scan file set widens.

Note: on a first-row merge-engine table with deletion vectors disabled, exact-scanning uncompacted files in full/detail mode can surface a superseded row version, because the exact scan returns physical rows without a merge-on-read dedup. This is a pre-existing property of the exact scan (it already applied to superseded versions promoted into compacted files); including level-0 files makes it more frequent rather than introducing it. The deletion-vector-enabled path is unaffected, since superseded rows are deletion-vector-marked and excluded. Tracked in #9895.

Tests

The synchronous write harness compacts eagerly and cannot hold a stable level-0 window, so the coverage pins the two halves of the chain separately:

  • PrimaryKeyVectorReadTest#testFilesToSearchIncludesAppendLevel0Files: filesToSearch returns an APPEND level-0 file that shouldRead rejects. Against the old filtered list the append file is absent, so this fails.
  • PrimaryKeyVectorReadTest#testFullModeBucketSearchExactScansAppendLevel0Files: the bucket search exact-scans that level-0 file in full mode and returns it, while FAST still skips it.

PrimaryKeyVectorRead handed only PrimaryKeyIndexSourcePolicy-filtered
files (complete COMPACT level>0 outputs) to the bucket search, so
Level-0 APPEND files — freshly written rows between compactions — were
omitted by every search mode, including full and detail, which are
documented to scan unindexed data files exactly. Pass all data files
of the split instead: the exact fallback covers the files the ANN
segments do not, while FAST keeps ignoring them, and the ANN state
builder keeps applying the compact-file policy itself.

The synchronous write harness cannot keep a stable level-0 window (the
write path compacts eagerly), so the regression coverage pins both
halves of the chain: filesToSearch must include an APPEND level-0
file that shouldRead rejects, and the bucket search must exact-scan
exactly such a file in full mode while FAST still skips it.

Assisted-by: GLM-5.3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Primary-key vector search omits fresh level-0 files in full/detail search modes

1 participant