Repository navigation
feat: parallel rollout WAL files with master batch commits - #303
Merged
Merged
Conversation
added 5 commits
October 4, 2026 03:51
beinan
marked this pull request as ready for review
October 4, 2026 04:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rollout WAL merges currently serialize each worker's complete delete/index/append cycle against the shared base table. Add an opt-in append-only path where workers prepare immutable files concurrently and the master publishes bounded groups in one transaction, with atomic MemWAL generation watermarks for recovery.
ROLLOUT_APPEND_TARGETS. Generic/context merges keep their existing semantics. Default concurrency is 4 and each staged prefix retains a 64 MiB byte cap and the worker's shared slots/memory budget.This path relies on rollout's immutable, non-reused ID contract; it is not an upsert or cross-shard ingest-deduplication API. Workers must be deployed with compatible owned-merge configuration before enabling targets. Index coverage remains handled by normal maintenance. See
docs/rollout-parallel-append.mdfor configuration, recovery and retention details.Validation:
af0dee7083574d45aaba895812c85d92e46aacb3: Rust tests, Python tests, coverage, clippy, three-platform test wheels and release packages, and distribution validation. This includes all 63 master etcd HA tests.No production deployment or activation is included.