[core][flink] Support building Lumina global index on deletion-vector tables - #8930
Open
XiaoHongbo-Hope wants to merge 1 commit into
Open
Conversation
XiaoHongbo-Hope
force-pushed
the
feature/lumina-index-support-dv-tables
branch
from
July 30, 2026 13:11
1ed7eb5 to
7b704f2
Compare
… tables Building a generic global index (incl. Lumina vector) on a deletion-vectors.enabled table was rejected to avoid indexing deleted rows. That guard predates read-side DV handling: vector search removes DV-deleted rows at query time via the live-row pre-filter, so the index can be a superset of physical rows and never needs rebuilding when DVs change. - Add opt-in GlobalIndexerFactory.supportsDeletionVectors() (default false), enabled only for Lumina (legacy lumina-vector-ann inherits it); all other index types stay rejected. Gate the build guard on it. The build still reads physical rows without DVs, so row-ids stay dense. - Pin the whole DV vector search to one snapshot: the index scan resolves it once, carries it in VectorScan.Plan, and the read threads it into the live-row filter, scalar/raw pre-filter and batch read, so index row-ids and the live-row bitmap always come from the same table state. - Drop concurrently-built global indexes on materialize compaction: the dropper now scans at the latest snapshot, so an index committed after the compaction planned (over pre-materialization row-ids) is dropped too. - Add a deterministic test: plan at S0, delete a row at S1, then read stays pinned to S0.
XiaoHongbo-Hope
force-pushed
the
feature/lumina-index-support-dv-tables
branch
from
July 30, 2026 13:19
7b704f2 to
ef31d32
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Building a generic global index (incl. Lumina vector) on a
deletion-vectors.enabledtable fails fast inGenericGlobalIndexBuilder.scan(). That guard predates read-side DV handling and is now too conservative: vector search already filters DV-deleted rows at query time via the live-row pre-filter (GlobalIndexLiveRowFilter/AbstractDataEvolutionVectorRead.preFilters), reading the current snapshot's DVs. So the index can be a superset of physical rows — deleted rows are removed on read, and it never needs rebuilding when DVs change.Changes
GlobalIndexerFactory.supportsDeletionVectors()(defaultfalse), mirroringsupportsFullTextSearch().LuminaVectorGlobalIndexerFactory; legacylumina-vector-anninherits). Other types stay rejected.!deletionVectorsEnabled() || supportsDeletionVectors(). Build still reads physical rows without DVs, so row-ids stay dense.Tests
Follow-up: E2E (pre-existing DV before build; new DV after build without rebuild; consistency after DV materialize/compaction).