feat(lance-table): Bε-tree manifest prototype for backfill scaling (#7499)#7848
Draft
jackye1995 wants to merge 7 commits into
Draft
feat(lance-table): Bε-tree manifest prototype for backfill scaling (#7499)#7848jackye1995 wants to merge 7 commits into
jackye1995 wants to merge 7 commits into
Conversation
Prototype for discussion lance-format#7499: the manifest root buffers fragment actions (AddFragment/RemoveFragment/AddDataFile/RemoveDataFile/AddDeletionFile) in an ε-buffer and flushes batches into tabular Lance-file child nodes routed by fragment-id range, instead of rewriting the full fragment list on every commit. Adds an honest flat-manifest baseline (real write_manifest_file_to_path / read_manifest) and a criterion bench (betree_backfill) run against S3 Standard and S3 Express. Scope: two-level fixed-partition tree; child-level ε-buffers (3+ level recursion) and byte-based split/merge self-balancing are follow-ups.
Contributor
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
github-actions
Bot
added
A-python
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: QUIET Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Replace the two-level fixed-partition prototype with a real recursive Bε-tree per the write-optimization literature (Bender et al.; BetrFS; TokuDB): - msn-ordered messages buffered per node (newest-wins), flushed to the fullest child gated at B/16 (the amortization threshold); - byte-based leaf split and fanout-based internal split with root grow (+height); - borrow/coalesce merge on underflow with root shrink (-height); type-aware underflow (leaves by bytes, internal by child count); - immutable copy-on-write nodes: leaves are Lance files, internal/root are protobuf; every touched root-to-leaf node is rewritten and the root repoints. Ideal byte-based defaults: epsilon=1/2, ~10 MiB nodes, fanout 16.
- Decompose the leaf Lance schema from one opaque DataFragment protobuf blob per fragment into one row per data file with each field its own column (path, versions, size, base_id, field ids). Lets Lance compress each column independently (RLE on versions, dictionary/FSST on paths) — ~2x smaller leaves than raw protobuf at scale (confirms Xuanwo's columnar observation). - Add streaming bootstrap (bootstrap_generate) that packs leaves from a fragment generator without holding the whole fragment list, so the tree can reach billion-data-file scale within a bounded memory budget. - Add an at-scale bench mode (BASE_FILES_PER_FRAGMENT): bootstrap N fragments each with many data files, report leaf compression vs the (infeasible) flat manifest, tree height, per-commit write, and cold-open — up to ~1B data files.
A full backfill column at N=1M is up to 100K commits; a 200-commit sample is enough to characterize per-commit write at scale and keeps the billion-file run tractable. Count added files exactly for the cold-open verification.
Add flush-depth tracking (CommitStats.max_flush_depth) and an internal-node-size report (BeTree::internal_node_sizes), plus a DEEP_FLUSH_COMMITS bench mode that drives a sustained backfill cascading flushes into internal nodes. At B=10 MiB this reaches max_flush_depth 6 with internal buffers filling to ~9.8 MiB — i.e. internal nodes are only tiny in the cold/bootstrap tree; under sustained writes they carry near-full ε-buffers, and per-commit write is bounded by ~height x B.
… nodes Two fixes so B and fanout are the only tuning knobs and both are safe at any value: - min_flush was hardcoded to B/16, silently assuming fanout 16. At higher fanout the ε-buffer spreads thinner than the gate so flushes never fire and the tree degrades to split-only. Derive it as B/fanout (BeTreeConfig::min_flush_bytes), the natural "fair share" of a full buffer across fanout children; keep an optional override for tests. At fanout 16 this is unchanged (B/16). - A fully-emptied leaf/internal node was kept with a phantom ChildRef whose min_key=0 (unwrap_or(0) on an empty range), which sorts to the front and breaks child_index_for's binary search — low-key removes misroute to it and no-op, resurrecting deleted fragments. Aggressive flushing (small min_flush) exposed this. Fix: drop an emptied node from its parent. Add a regression test plus a sparse-delete merge test that exercises real coalescing.
Rename the two tuning knobs to say exactly what they are: - target_node_bytes -> max_node_bytes (the byte size at which a node splits) - fanout -> max_children_per_node (the branching factor: split above it, merge below a quarter of it) - ChildRef.fanout_used -> num_children (a node's current direct-child count) Pure rename + doc cleanup (proto fields, config, tree, bench, tests); no behavior change. The FANOUT / NODE_SIZE_MB bench env vars are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype for the Bε-tree manifest counter-proposal in discussion #7499.
Instead of rewriting the full fragment list on every commit, the manifest root buffers
fragment_actions(AddFragment/RemoveFragment/AddDataFile/RemoveDataFile/AddDeletionFile) in an ε-buffer and flushes batches into tabular Lance-file child nodes routed by fragment-id range. This targets the 2D (fragments × columns) growth from add-column / backfill, which is the real manifest-scaling problem.What's here
protos/betree.proto— prototype root/child/action messages (merge into thelance.tablepbmodule).rust/lance-table/src/betree/— root (protobuf: child refs + inline ε-buffer), children as Lance v2 files, action set, commit → overflow → parallel flush, materialize / cold-open.write_manifest_file_to_path/read_manifest.benches/betree_backfill.rs— criterion bench (IO-tracked), runs local / S3 Standard / S3 Express.Benchmark (N=1M fragments, realistic Lance data-file names,
c7i.16xlargein us-east-1)Realistic fine-grained backfill (F fragments/commit → N/F commits, as GPU-computed embeddings trickle in). Flat rewrites the full ~82 MiB manifest on every commit regardless of F:
Scope (draft)
Two-level, fixed-partition tree. Child-level ε-buffers (3+ level recursion) and byte-based split/merge self-balancing are in progress. Not for merge — this exists to inform the #7499 design.