Skip to content

fix: clear stale MoveFailed once the cluster is balanced - #6

Merged
KeiaiLab-PHIL merged 1 commit into
mainfrom
fix/clear-stale-movefailed-degraded
Aug 25, 2026
Merged

fix: clear stale MoveFailed once the cluster is balanced#6
KeiaiLab-PHIL merged 1 commit into
mainfrom
fix/clear-stale-movefailed-degraded

Conversation

@eightynine01

Copy link
Copy Markdown
Contributor

What

reconcileRebalance raises Degraded=True with reason MoveFailed when a move
command fails, but the only path that lowers it again is the active-move settle
branch. When the plan later becomes empty — the cluster reached balance on its
own, or the move stopped being necessary — the function returns phaseRunning
and leaves the condition standing.

The balanced branch now clears the condition, but only when the reason is the
one this function owns. DrainBlocked and ImmutableFieldChanged are raised
elsewhere; lowering them here would swallow those signals.

Why

Observed on a production cluster from 2026-08-22 to 2026-08-26: Degraded=True
for four days while every collection reported green and the shard named in the
condition message was Active on both peers. The message pointed at a consensus
commit timeout that had long since stopped mattering.

A condition that can be raised but never lowered is worse than no condition,
because it hides the next real fault behind a red light the operator has learned
to ignore.

Tests

internal/controller/rebalance_degraded_test.go:

  • TestClearMoveFailed_자기사유만회수 — the owned reason is cleared, backoff reset.
  • TestClearMoveFailed_타사유는불가침DrainBlocked / ImmutableFieldChanged
    survive untouched, backoff preserved.
  • TestReconcileRebalance_균형이면Degraded회수 — the real entry point, with a
    balanced observation, clears a pre-existing MoveFailed.

Verified the guard fails without the fix:

--- FAIL: TestReconcileRebalance_균형이면Degraded회수
    균형 도달 시 MoveFailed 회수 실패: status=True reason=MoveFailed

go vet ./... clean, gofmt clean. TestControllers needs envtest binaries and
fails identically on an unmodified tree in this environment.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VbPL3ACkTc18bnwYUd7G3w

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@eightynine01
eightynine01 force-pushed the fix/clear-stale-movefailed-degraded branch from bc9bc8c to 47205e9 Compare August 25, 2026 23:11
reconcileRebalance raised Degraded=True/MoveFailed when a move command
failed, but the only path that lowered it again was the active-move
settle branch. When the plan later became empty - the cluster reached
balance on its own, or the move stopped being necessary - the function
returned phaseRunning and left the condition standing.

Observed live from 2026-08-22 to 2026-08-26: Degraded=True for four days
while all 21 collections reported green and the shard named in the
condition message was Active on both peers. The message pointed at a
consensus commit timeout that had long since stopped mattering. A
condition that can be raised but never lowered is worse than no
condition, because it hides the next real fault behind a red light the
operator has learned to ignore.

The balanced branch now clears the condition, but only when the reason
is the one this function owns. DrainBlocked and ImmutableFieldChanged
are raised elsewhere and lowering them here would swallow those signals.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VbPL3ACkTc18bnwYUd7G3w
Signed-off-by: phil <phil@keiailab.com>
@eightynine01
eightynine01 force-pushed the fix/clear-stale-movefailed-degraded branch from 47205e9 to 3ade8e7 Compare August 25, 2026 23:16
@KeiaiLab-PHIL
KeiaiLab-PHIL merged commit 71562cf into main Aug 25, 2026
14 of 15 checks passed
@KeiaiLab-PHIL
KeiaiLab-PHIL deleted the fix/clear-stale-movefailed-degraded branch August 25, 2026 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants