fix: filter stale scalar index fragments#7825
Conversation
Keep fragment coverage filtering separate from deletion masks so stale row addresses cannot reach take operations without expanding sparse deletion masks. AI Disclosure: This code was written in part by an AI agent.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Represent excluded legacy fragments as compact full-fragment blocks so all mask consumers enforce index coverage without expanding sparse deletion masks. Strengthen legacy vector and manifest boundary coverage. AI Disclosure: This code was written in part by an AI agent.
| unique_ids.len(), | ||
| ids.len() | ||
| actual_ids, | ||
| expected_ids.iter().map(String::as_str).collect(), |
There was a problem hiding this comment.
This assertion does not compile at the current head: the collect() result is ambiguous (E0283) because HashSet can compare against multiple borrowed forms. The same type error prevents the Rust build, clippy, MSRV, ARM, and no-lock targets from reaching the regression tests, so the fix cannot be validated in its current form.
| .chain(&self.excluded_fragments) | ||
| .flat_map(|fragments| fragments.iter()); | ||
| let mut fragment_block_list = RowAddrTreeMap::new(); | ||
| for fragment_id in blocked_fragments { |
There was a problem hiding this comment.
For an old or partial index, excluded_fragments can cover nearly 0..=max_fragment_id. This loop expands every historical ID into a RowAddrTreeMap entry on query startup, so memory and CPU scale with dataset history rather than current rows or index coverage. A high-churn dataset with a small current index can therefore create an unbounded vector/FTS query-time resource regression.
2 participants
Closes #7803
Keep fragment coverage filtering separate from deletion masks so stale row addresses cannot reach take operations without expanding sparse deletion masks.