fix: clear stale MoveFailed once the cluster is balanced - #6
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
eightynine01
force-pushed
the
fix/clear-stale-movefailed-degraded
branch
from
August 25, 2026 23:11
bc9bc8c to
47205e9
Compare
reconcileRebalance raised Degraded=True/MoveFailed when a move command failed, but the only path that lowered it again was the active-move settle branch. When the plan later became empty - the cluster reached balance on its own, or the move stopped being necessary - the function returned phaseRunning and left the condition standing. Observed live from 2026-08-22 to 2026-08-26: Degraded=True for four days while all 21 collections reported green and the shard named in the condition message was Active on both peers. The message pointed at a consensus commit timeout that had long since stopped mattering. A condition that can be raised but never lowered is worse than no condition, because it hides the next real fault behind a red light the operator has learned to ignore. The balanced branch now clears the condition, but only when the reason is the one this function owns. DrainBlocked and ImmutableFieldChanged are raised elsewhere and lowering them here would swallow those signals. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbPL3ACkTc18bnwYUd7G3w Signed-off-by: phil <phil@keiailab.com>
eightynine01
force-pushed
the
fix/clear-stale-movefailed-degraded
branch
from
August 25, 2026 23:16
47205e9 to
3ade8e7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
reconcileRebalanceraisesDegraded=Truewith reasonMoveFailedwhen a movecommand fails, but the only path that lowers it again is the active-move settle
branch. When the plan later becomes empty — the cluster reached balance on its
own, or the move stopped being necessary — the function returns
phaseRunningand leaves the condition standing.
The balanced branch now clears the condition, but only when the reason is the
one this function owns.
DrainBlockedandImmutableFieldChangedare raisedelsewhere; lowering them here would swallow those signals.
Why
Observed on a production cluster from 2026-08-22 to 2026-08-26:
Degraded=Truefor four days while every collection reported
greenand the shard named in thecondition message was
Activeon both peers. The message pointed at a consensuscommit timeout that had long since stopped mattering.
A condition that can be raised but never lowered is worse than no condition,
because it hides the next real fault behind a red light the operator has learned
to ignore.
Tests
internal/controller/rebalance_degraded_test.go:TestClearMoveFailed_자기사유만회수— the owned reason is cleared, backoff reset.TestClearMoveFailed_타사유는불가침—DrainBlocked/ImmutableFieldChangedsurvive untouched, backoff preserved.
TestReconcileRebalance_균형이면Degraded회수— the real entry point, with abalanced observation, clears a pre-existing
MoveFailed.Verified the guard fails without the fix:
go vet ./...clean,gofmtclean.TestControllersneeds envtest binaries andfails identically on an unmodified tree in this environment.
🤖 Generated with Claude Code
https://claude.ai/code/session_01VbPL3ACkTc18bnwYUd7G3w