Repository navigation
fix: reconcile catch-up deferrals without failure backoff - #321
Merged
Merged
Conversation
beinan
marked this pull request as ready for review
October 6, 2026 06:19
beinan
added a commit
that referenced
this pull request
Oct 6, 2026
Keep explicitly configured owned tables under continuous master catch-up coverage, including low-count tails after successful slices. Persist the canonical target set in the shared policy so failover cannot silently discard desired coverage; fresh stats, cooldowns, fencing and global capacity still gate every new Job. If a previously confirmed Job disappears, exact terminal child-container evidence can retire its inventory reservation as a failure. Missing or running children and identity mismatches remain unresolved. Execution/claim/target-lock records are retained for normal storage recovery before successor writes. Rebased onto merged #321; this PR contains only the continuous coverage and missing-Job changes. New cases cover continued service after restart, mismatched replica policy, stale post-success stats and missing-Job identity/terminal checks. Formatting passed. Local catchup suite: 33/33 passed with real etcd and native storage. Combined with #321, #322 and #324: all 170 master library tests passed, including real-etcd tests. CI remains in progress. Existing external publisher adoption and production activation remain separate, qualified per-target migrations; no owner keys are reset by this change. Co-authored-by: Beinan Wang <>
beinan
added a commit
that referenced
this pull request
Oct 6, 2026
A persistent publisher can repeatedly claim another merge pass before a prepared compaction obtains commit ownership, causing the compactor to exhaust its wait budget and rebuild files. Publish a commit-ready hint only after nonempty output is prepared; subsequent merge claims yield to the matching live preparer. The hint shares the preparation claim lease and is checked in the existing admission snapshot and final claim CAS. Preparation stays concurrent, admitted merges are not cancelled, and unresolved merge executions can still be claimed for storage recovery. Normal exclusive commit ownership and the existing wait budget remain in force. Validation: formatting/diff checks passed. All three local real-etcd tests passed for native pass handoff, lease-loss cleanup/stale publisher rejection, and recovery bypass. Combined with #321, #322 and #323: all 170 master library tests passed, including real-etcd tests. CI remains in progress. Every scheduling/catchup contender needs the updated binary to honor this fairness hint; production rollout is separate. Co-authored-by: Beinan Wang <>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a controller-owned catch-up Job exits because its table is busy before claim, the controller currently counts it as a merge failure and adds exponential backoff. It can also miss that terminal result entirely when a different maintenance execution now owns the table.
This change writes a bounded, Pod-UID/Job/attempt-bound termination receipt for proven pre-claim deferrals, reconciles terminal Jobs before sampling execution progress, and preserves real failure debt while scheduling a short jittered retry. Missing or mismatched receipts remain unresolved; other owners are never revoked. Actual readmission follows the configured controller tick and existing ownership/cooldown checks.
Validation: formatting and diff checks passed. Local catchup tests passed: 30/30, including real-etcd and native storage cases. CI is in progress. Added receipt identity/ambiguity regressions, real-etcd failure-debt and stale-controller checks, mocked-Kubernetes plus real-etcd reconciliation against another active execution, and native busy-to-success storage coverage. No production enablement or adoption of external publishers is included.