Repository navigation
fix: Wait for the cache to show a StatefulSet shrink before the next pass - #71
Merged
Merged
Conversation
…pass replicaCounts applies the larger of the declared count and the one the cached StatefulSet carries. A pass that read the StatefulSet from before shedRetiredPods lowered it applied the old count again, then shrank it a second time once it found the member already gone; the StatefulSet controller could recreate the shed pod in between. The MemgraphCluster status write that ends the shedding pass requeues the cluster on a separate watch stream, so that pass can start before the StatefulSet event reaches the cache. shedRetiredPods now records the UID and generation the apply returned in an in-memory expectation (internal/expectations), and a pass whose cached StatefulSet is the same object at a lower generation ends before applying anything, until the StatefulSet event clears it. A restarted operator lists into a fresh cache that already holds the shrink, so nothing is persisted.
kubectl wait fails at once on an object that does not exist yet, and the operator creates the Deployment a moment after the patch, so the spec raced it. It now polls the Available condition, like the init container spec polls its pod.
as51340
enabled auto-merge (squash)
October 9, 2026 13:28
as51340
disabled auto-merge
October 9, 2026 13:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
replicaCountsappliesmax(declared, cached spec.replicas). AftershedRetiredPodslowers a StatefulSet, the next pass can still read the pre-shrink copy from the informer cache. The status write that ends the shedding pass requeues the cluster on the MemgraphCluster watch stream, which isn't ordered with the StatefulSet stream. That pass then:The end state is correct, so this is churn rather than data loss. But a retired coordinator or data instance pod can briefly come back on its retained volume.
Fix
This is the Kubernetes expectations pattern, shaped like ECK's
ExpectedStatefulSetUpdatesand narrowed to the one write whose stale read undoes it:internal/expectations:StatefulSets.Expect(name, uid, generation)andSatisfied(name, cached). An expectation is satisfied once the cached copy is gone, is a different object, or has reached the generation. It's forgotten once satisfied.shedRetiredPodsrecords the UID and generation its apply returned. To get them,applynow returns the written object, andappliedGenerationsbecameappliedObjects.replicaCountschecks each role's cached StatefulSet. If either is behind, the pass ends before applying anything and returnsRequeueAfter: requeueWhilePending. TheOwnsStatefulSet event normally requeues it first.Every other cached read stays as it is. The apply restates desired state, pod deletes carry a UID precondition,
rollout'sStalecheck covers status, and membership is read live over Bolt. CLAUDE.md gets a convention bullet with this reasoning.Tests
internal/expectationscover older and newer generations, a missing StatefulSet, a replaced UID, forgetting a satisfied expectation, and a later write replacing an earlier one.laggingCacheclient, then checks that the StatefulSet's generation doesn't change and that the pass converges once the cache catches up. With the gate disabled the test fails: the generation goes 2 → 4, grown back and then shrunk again.make testandmake lintpass.