Repository navigation
feat: let master schedule low-count WAL tails automatically - #322
Merged
Merged
Conversation
beinan
marked this pull request as ready for review
October 6, 2026 06:30
This was referenced Oct 6, 2026
beinan
added a commit
that referenced
this pull request
Oct 6, 2026
Keep explicitly configured owned tables under continuous master catch-up coverage, including low-count tails after successful slices. Persist the canonical target set in the shared policy so failover cannot silently discard desired coverage; fresh stats, cooldowns, fencing and global capacity still gate every new Job. If a previously confirmed Job disappears, exact terminal child-container evidence can retire its inventory reservation as a failure. Missing or running children and identity mismatches remain unresolved. Execution/claim/target-lock records are retained for normal storage recovery before successor writes. Rebased onto merged #321; this PR contains only the continuous coverage and missing-Job changes. New cases cover continued service after restart, mismatched replica policy, stale post-success stats and missing-Job identity/terminal checks. Formatting passed. Local catchup suite: 33/33 passed with real etcd and native storage. Combined with #321, #322 and #324: all 170 master library tests passed, including real-etcd tests. CI remains in progress. Existing external publisher adoption and production activation remain separate, qualified per-target migrations; no owner keys are reset by this change. Co-authored-by: Beinan Wang <>
beinan
added a commit
that referenced
this pull request
Oct 6, 2026
A persistent publisher can repeatedly claim another merge pass before a prepared compaction obtains commit ownership, causing the compactor to exhaust its wait budget and rebuild files. Publish a commit-ready hint only after nonempty output is prepared; subsequent merge claims yield to the matching live preparer. The hint shares the preparation claim lease and is checked in the existing admission snapshot and final claim CAS. Preparation stays concurrent, admitted merges are not cancelled, and unresolved merge executions can still be claimed for storage recovery. Normal exclusive commit ownership and the existing wait budget remain in force. Validation: formatting/diff checks passed. All three local real-etcd tests passed for native pass handoff, lease-loss cleanup/stale publisher rejection, and recovery bypass. Combined with #321, #322 and #323: all 170 master library tests passed, including real-etcd tests. CI remains in progress. Every scheduling/catchup contender needs the updated binary to honor this fairness hint; production rollout is separate. Co-authored-by: Beinan Wang <>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The normal eight-generation merge trigger can strand one to seven sealed WAL generations indefinitely after writes stop. Add an opt-in master sweep that serves these tails through the existing scheduler using fresh in-memory stats, without opening datasets or adding I/O to reads/writes.
A durable etcd CAS cursor reserves bounded pages across replicas and survives failover. Active tasks, cooldowns, draining targets and dedicated owners retain their existing handling. Unknown enqueue outcomes are not immediately replayed; later rotations reevaluate current demand. Maintenance-only masters stay disabled. This is periodic low-count service, not an estimate of generation creation age.
Validation: formatting/diff checks passed; added fresh/stale selection tests plus real-etcd multi-replica pagination, restart and external-owner isolation tests. Local tests passed on the current head: 3/3 including real-etcd replicas, restart/cursor and external-owner isolation. CI is in progress. Production activation is separate from the source change.