fix(index): avoid caching store-bound scalar indices#7962
Conversation
github-actions
Bot
added
A-index
📝 WalkthroughWalkthroughIndex caching now avoids retaining store-bound scalar and vector objects across object-store generations. Portable IVF state remains cacheable, while readers are reopened against the current store. Regression tests verify cache isolation, current-store IO, and secret-free cache keys. ChangesIndex cache isolation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Dataset
participant IndexCache
participant ObjectStore
participant VectorIndex
Dataset->>IndexCache: open index
IndexCache-->>Dataset: portable IVF state or cache miss
Dataset->>ObjectStore: open readers for current store
ObjectStore-->>VectorIndex: index data
VectorIndex-->>Dataset: reconstructed index
Dataset->>IndexCache: store portable IVF state
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
f1a2cca to
f6fdf31
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
rust/lance/src/index/vector/ivf/v2.rs (1)
6570-6585: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd
#[cfg_attr(coverage, coverage(off))]to this test utility.
NoPartitionCacheBackendand itsCacheBackendimpl are test-only scaffolding and should be excluded from coverage instrumentation.♻️ Suggested annotations
+#[cfg_attr(coverage, coverage(off))] #[derive(Debug)] struct NoPartitionCacheBackend(lance_core::cache::MokaCacheBackend);#[async_trait::async_trait] +#[cfg_attr(coverage, coverage(off))] impl lance_core::cache::CacheBackend for NoPartitionCacheBackend {As per coding guidelines: "disable coverage for test utilities with
#[cfg_attr(coverage, coverage(off))]".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/index/vector/ivf/v2.rs` around lines 6570 - 6585, Add #[cfg_attr(coverage, coverage(off))] to the NoPartitionCacheBackend test utility and its CacheBackend implementation, excluding both scaffolding components from coverage instrumentation.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/lance-index-core/src/scalar.rs`:
- Around line 249-257: Expand the documentation for
can_cache_store_bound_indices with a compiling usage example that demonstrates
both true and false implementations, and link to the relevant cache and
index-opening APIs using their actual symbols and signatures. Keep the example
synchronized with the public contract and clarify when implementers should
override the default.
In `@rust/lance/src/index.rs`:
- Around line 2942-2975: Replace the manually constructed test fixtures in the
batch setup around RecordBatchIterator with arrow_array’s record_batch!() macro.
Remove the redundant Schema, Arc, and RecordBatch::try_new boilerplate while
preserving the existing id and text values and resulting RecordBatchIterator
behavior.
---
Nitpick comments:
In `@rust/lance/src/index/vector/ivf/v2.rs`:
- Around line 6570-6585: Add #[cfg_attr(coverage, coverage(off))] to the
NoPartitionCacheBackend test utility and its CacheBackend implementation,
excluding both scaffolding components from coverage instrumentation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: 6335ac2e-99de-4a65-815f-d9354fafa37f
📒 Files selected for processing (5)
rust/lance-index-core/src/scalar.rsrust/lance-index/src/scalar/lance_format.rsrust/lance-index/src/scalar/registry.rsrust/lance/src/index.rsrust/lance/src/index/vector/ivf/v2.rs
| /// Whether a live index holding this store can be shared through a session cache. | ||
| /// | ||
| /// Implementations whose readers capture connection state or renewable | ||
| /// credentials should return `false`. Serializable index state can still be | ||
| /// cached and reconstructed with the current store. | ||
| fn can_cache_store_bound_indices(&self) -> bool { | ||
| true | ||
| } | ||
|
|
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add a usage example for this public cache-safety contract.
Document both return values with a compiling example and links to the relevant cache/opening API so implementers can safely override this method.
As per coding guidelines, “Document all public APIs with examples and links to relevant structs and methods; keep examples synchronized with actual signatures.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rust/lance-index-core/src/scalar.rs` around lines 249 - 257, Expand the
documentation for can_cache_store_bound_indices with a compiling usage example
that demonstrates both true and false implementations, and link to the relevant
cache and index-opening APIs using their actual symbols and signatures. Keep the
example synchronized with the public contract and clarify when implementers
should override the default.
Source: Coding guidelines
| let test_dir = TempStrDir::default(); | ||
| let schema = Arc::new(Schema::new(vec![ | ||
| Field::new("id", DataType::Int32, false), | ||
| Field::new("text", DataType::Utf8, false), | ||
| ])); | ||
| let batches = vec![ | ||
| RecordBatch::try_new( | ||
| schema.clone(), | ||
| vec![ | ||
| Arc::new(Int32Array::from(vec![0, 1, 2, 3])), | ||
| Arc::new(StringArray::from(vec![ | ||
| "alpha common", | ||
| "beta common", | ||
| "gamma common", | ||
| "alpha common", | ||
| ])), | ||
| ], | ||
| ) | ||
| .unwrap(), | ||
| RecordBatch::try_new( | ||
| schema.clone(), | ||
| vec![ | ||
| Arc::new(Int32Array::from(vec![4, 5, 6, 7])), | ||
| Arc::new(StringArray::from(vec![ | ||
| "beta common", | ||
| "gamma common", | ||
| "alpha common", | ||
| "beta common", | ||
| ])), | ||
| ], | ||
| ) | ||
| .unwrap(), | ||
| ]; | ||
| let reader = RecordBatchIterator::new(batches.into_iter().map(Ok), schema); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Use record_batch!() for the test fixtures.
This manually repeats Schema, Arc, and RecordBatch::try_new boilerplate. Build these batches with record_batch!() instead.
As per coding guidelines, “Use record_batch!() from arrow_array to construct RecordBatch in tests instead of manual Schema/Arc/try_new boilerplate.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rust/lance/src/index.rs` around lines 2942 - 2975, Replace the manually
constructed test fixtures in the batch setup around RecordBatchIterator with
arrow_array’s record_batch!() macro. Remove the redundant Schema, Arc, and
RecordBatch::try_new boilerplate while preserving the existing id and text
values and resulting RecordBatchIterator behavior.
Source: Coding guidelines
# Conflicts: # rust/lance/src/index.rs
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
rust/lance/src/index.rs (1)
2400-2557: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
IVF_RQopens against the wrong directory.Every other IVF variant branch (
IVF_FLAT,IVF_PQ,IVF_SQ,IVF_HNSW_FLAT,IVF_HNSW_SQ,IVF_HNSW_PQ) passes the locally-resolvedindex_dir(fromself.indice_files_dir(&index_meta), which honorsindex_meta.base_idfor e.g. shallow-cloned datasets).IVF_RQinstead passesself.indices_dir(), ignoring any base redirection. For anIVF_RQindex whose metadata points at a different base (cloned dataset), this will resolve to the wrong location and fail to open.🐛 Proposed fix
"IVF_RQ" => { let ivf = IVFIndex::<FlatIndex, RabitQuantizer>::try_new( object_store.clone(), - self.indices_dir(), + index_dir, uuid.to_owned(), frag_reuse_index, self.metadata_cache.as_ref(), index_cache, file_sizes, ) .await?; Ok(wrap_ivf(ivf)) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/index.rs` around lines 2400 - 2557, Update the IVF_RQ branch in the index-opening match to pass the locally resolved index_dir to IVFIndex::try_new, matching every other IVF variant. Do not use self.indices_dir(), so index metadata base redirection remains honored.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@rust/lance/src/index.rs`:
- Around line 2400-2557: Update the IVF_RQ branch in the index-opening match to
pass the locally resolved index_dir to IVFIndex::try_new, matching every other
IVF variant. Do not use self.indices_dir(), so index metadata base redirection
remains honored.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: 5c7e1498-9d16-4f7a-9f13-3d6df6327c64
📒 Files selected for processing (1)
rust/lance/src/index.rs
1 participant
Summary
IndexStoreimplementations declare whether live indices holding their readers are safe to share through a Session cacheScalarIndexcache forLanceIndexStore, so FTS and other default scalar plugins always bind readers to the current object storeLifecycle
Depends on #7944. Together, the two PRs remove the known store-bound vector and scalar entries from Session caches. This resolves stale-generation retention deterministically without adding TTL/TTI or process-local object-store identities.
Testing
cargo fmt --all -- --checkcargo check -p lance-index-core -p lance-index -p lance --libcargo test -p lance --lib test_scalar_cache_uses_current_object_store -- --nocapturecargo test -p lance-index test_btree_index_state_reconstruct_and_plugin_cachecargo test -p lance-index test_bitmap_cache_fast_pathcargo clippy --all --tests --benches -- -D warningsThe current #7944 base contains an unrelated blob merge-insert test compile error (
Option<BlobFile>::read). The scalar regression and full clippy were run with a local-only six-line unwrap fix for that base error; that workaround is not included here.Closes #7958
Closes #7959