[core] Search fresh level-0 files in PK vector search full modes - #9896
Open
LuciferYang wants to merge 1 commit into
Open
LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
PrimaryKeyVectorRead handed only PrimaryKeyIndexSourcePolicy-filtered files (complete COMPACT level>0 outputs) to the bucket search, so Level-0 APPEND files — freshly written rows between compactions — were omitted by every search mode, including full and detail, which are documented to scan unindexed data files exactly. Pass all data files of the split instead: the exact fallback covers the files the ANN segments do not, while FAST keeps ignoring them, and the ANN state builder keeps applying the compact-file policy itself. The synchronous write harness cannot keep a stable level-0 window (the write path compacts eagerly), so the regression coverage pins both halves of the chain: filesToSearch must include an APPEND level-0 file that shouldRead rejects, and the bucket search must exact-scan exactly such a file in full mode while FAST still skips it. Assisted-by: GLM-5.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9894
PrimaryKeyVectorReadhanded the bucket search only the filesPrimaryKeyIndexSourcePolicy.shouldReadaccepts (complete COMPACT level>0 outputs), so Level-0 APPEND files, the rows written since the last compaction, were omitted by every search mode, includingfullanddetail, which are documented to exact-scan unindexed data files. This passes all data files of the split (filesToSearch) to the bucket search instead. The exact fallback then covers the files the ANN segments do not,FASTkeeps ignoring them, and the ANN state builder (PkVectorBucketIndexState.fromActiveDataFiles) still applies the compact-file policy internally, so the ANN segment state is unchanged and only the exact-scan file set widens.Note: on a
first-rowmerge-engine table with deletion vectors disabled, exact-scanning uncompacted files infull/detailmode can surface a superseded row version, because the exact scan returns physical rows without a merge-on-read dedup. This is a pre-existing property of the exact scan (it already applied to superseded versions promoted into compacted files); including level-0 files makes it more frequent rather than introducing it. The deletion-vector-enabled path is unaffected, since superseded rows are deletion-vector-marked and excluded. Tracked in #9895.Tests
The synchronous write harness compacts eagerly and cannot hold a stable level-0 window, so the coverage pins the two halves of the chain separately:
PrimaryKeyVectorReadTest#testFilesToSearchIncludesAppendLevel0Files:filesToSearchreturns an APPEND level-0 file thatshouldReadrejects. Against the old filtered list the append file is absent, so this fails.PrimaryKeyVectorReadTest#testFullModeBucketSearchExactScansAppendLevel0Files: the bucket search exact-scans that level-0 file infullmode and returns it, whileFASTstill skips it.