Skip to content

fix(index): rebind vector readers to current object store #7944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wjones127 merged 2 commits into from
Jul 30, 2026
+591 −112
Merged

fix(index): rebind vector readers to current object store #7944

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5f3afb3
fix(index): rebind vector readers to current object store
u70b3 Jul 27, 2026
f27fc13
ci: rerun checks
u70b3 Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 20 additions & 14 deletions rust/lance/src/index.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -703,6 +703,11 @@ impl CacheKey for LegacyVectorIndexCacheKey<'_> {
/// Used for v0.3+ indices that support serialization. This key has a codec,
/// so custom cache backends can serialize the state to disk/Redis/etc.
/// Legacy indices use `LegacyVectorIndexCacheKey` instead (in-memory only).
///
/// Note: legacy entries hold live readers bound to the object store that
/// opened them, so they are only valid for credential setups that refresh
/// internally (e.g. a credentials provider). Deployments that pass fresh
/// static credentials per dataset open should use v0.3+ index formats.
#[derive(Debug, Clone)]
pub(crate) struct IvfIndexStateCacheKey<'a> {
uuid: &'a Uuid,
Expand Down Expand Up @@ -2438,7 +2443,9 @@ impl DatasetIndexInternalExt for Dataset {
let tailing_bytes = read_last_block(reader.as_ref()).await?;
let (major_version, minor_version) = read_version(&tailing_bytes)?;

// Namespace the index cache by the UUID of the index.
// Namespace the index cache by the UUID of the index. v2+ partition
// entries are store-free and remain reusable across object-store
// generations alongside their serializable state.
let index_cache = self.index_cache.for_index(uuid, frag_reuse_uuid.as_ref());

// Extract the cacheable state before type-erasing to Arc<dyn VectorIndex>.
Expand Down Expand Up @@ -2497,8 +2504,8 @@ impl DatasetIndexInternalExt for Dataset {

(0, 3) | (2, _) => {
let scheduler = ScanScheduler::new(
self.object_store.clone(),
SchedulerConfig::max_bandwidth(&self.object_store),
object_store.clone(),
SchedulerConfig::max_bandwidth(&object_store),
);
let cached_size = file_sizes
.get(INDEX_FILE_NAME)
Expand Down Expand Up @@ -2532,7 +2539,7 @@ impl DatasetIndexInternalExt for Dataset {
"IVF_FLAT" => match element_type {
DataType::Float16 | DataType::Float32 | DataType::Float64 => {
let ivf = IVFIndex::<FlatIndex, FlatQuantizer>::try_new(
self.object_store.clone(),
object_store.clone(),
index_dir,
uuid.to_owned(),
frag_reuse_index,
Expand All @@ -2545,7 +2552,7 @@ impl DatasetIndexInternalExt for Dataset {
}
DataType::UInt8 => {
let ivf = IVFIndex::<FlatIndex, FlatBinQuantizer>::try_new(
self.object_store.clone(),
object_store.clone(),
index_dir,
uuid.to_owned(),
frag_reuse_index,
Expand All @@ -2564,7 +2571,7 @@ impl DatasetIndexInternalExt for Dataset {

"IVF_PQ" => {
let ivf = IVFIndex::<FlatIndex, ProductQuantizer>::try_new(
self.object_store.clone(),
object_store.clone(),
index_dir,
uuid.to_owned(),
frag_reuse_index,
Expand All @@ -2578,7 +2585,7 @@ impl DatasetIndexInternalExt for Dataset {

"IVF_SQ" => {
let ivf = IVFIndex::<FlatIndex, ScalarQuantizer>::try_new(
self.object_store.clone(),
object_store.clone(),
index_dir,
uuid.to_owned(),
frag_reuse_index,
Expand All @@ -2592,8 +2599,8 @@ impl DatasetIndexInternalExt for Dataset {

"IVF_RQ" => {
let ivf = IVFIndex::<FlatIndex, RabitQuantizer>::try_new(
self.object_store.clone(),
self.indices_dir(),
object_store.clone(),
index_dir,
uuid.to_owned(),
frag_reuse_index,
self.metadata_cache.as_ref(),
Expand All @@ -2607,7 +2614,7 @@ impl DatasetIndexInternalExt for Dataset {
"IVF_HNSW_FLAT" => match element_type {
DataType::UInt8 => {
let ivf = IVFIndex::<HNSW, FlatBinQuantizer>::try_new(
self.object_store.clone(),
object_store.clone(),
index_dir,
uuid.to_owned(),
frag_reuse_index,
Expand All @@ -2620,7 +2627,7 @@ impl DatasetIndexInternalExt for Dataset {
}
_ => {
let ivf = IVFIndex::<HNSW, FlatQuantizer>::try_new(
self.object_store.clone(),
object_store.clone(),
index_dir,
uuid.to_owned(),
frag_reuse_index,
Expand All @@ -2635,7 +2642,7 @@ impl DatasetIndexInternalExt for Dataset {

"IVF_HNSW_SQ" => {
let ivf = IVFIndex::<HNSW, ScalarQuantizer>::try_new(
self.object_store.clone(),
object_store.clone(),
index_dir,
uuid.to_owned(),
frag_reuse_index,
Expand All @@ -2649,7 +2656,7 @@ impl DatasetIndexInternalExt for Dataset {

"IVF_HNSW_PQ" => {
let ivf = IVFIndex::<HNSW, ProductQuantizer>::try_new(
self.object_store.clone(),
object_store.clone(),
index_dir,
uuid.to_owned(),
frag_reuse_index,
Expand Down Expand Up @@ -2684,7 +2691,6 @@ impl DatasetIndexInternalExt for Dataset {
io_stats.add_scan_stats(&open_stats);
}
if let Some(ivf_entry) = ivf_entry {

@wjones127 wjones127 Jul 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue(blocking): Legacy (v0.1/v0.2) live vector indices no longer get any in-memory index-object cache (the old else branch inserting CachedLegacyVectorIndex is gone) — every query on an old-format index now reopens and reconstructs the full index each time. That's the correct fix for the credential-staleness bug, but for high-QPS users still on pre-v2 index files this could be a real latency/IO regression.

TBH the stale credentials thing seems to me to be an edge case. (Most users I know that have long-lived processes configure a CredentialsProvider that can do the refresh internally.) I lean towards still having that bug for old indices than dropping support for index caching on them. What do you think about that?

@u70b3 u70b3 Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — restored the in-memory cache for legacy (v0.1/v0.2) live indices, so their caching behavior is unchanged from before this PR. The staleness caveat is now documented on the cache key: legacy entries assume credentials that refresh internally (e.g. a credentials provider); deployments passing fresh static credentials per dataset open should use v0.3+ index formats.

One clarification for the record: even with the legacy index-object cache removed, partition data stayed cached (the store-free partition entries), so the per-query cost was a few metadata reads rather than a full index rebuild. Either way, keeping the legacy cache is the better trade — thanks for the pushback.

The A → B → A rotation test now pins the version-specific behavior: V3 readers rebind to the current store, while a cached legacy index keeps reading through its original store.

let state_key = IvfIndexStateCacheKey::new(uuid, frag_reuse_uuid.as_ref());
self.index_cache
.insert_with_key(&state_key, Arc::new(ivf_entry))
.await;
Expand Down
Loading
Loading