peer: hard-bound the breaker map under failed-address churn (fixes #54) - #58
Conversation
The #9 sweep only removes information-free entries, so a stream of distinct addresses that each carry failure state (below-threshold failures, open circuits, half-open probes in flight) grew the peer map past maxTrackedPeers without bound, contradicting the documented "bounded" contract. Make the cap a hard bound: when the sweep frees nothing, evict the least-recently-active entry. Activity is stamped only by request paths (Allow and the Record* reports) so read-only observers (State/Healthy/ OpenCount) cannot launder stale entries into recent use. The victim is by construction the address no request has touched for the longest time — almost always a departed node; if it is genuinely sick and still in rotation, the next request re-creates the entry and re-accumulates the failures, so breaker behavior for active peers is unchanged. Regression tests churn cap+100 dirty addresses in three flavors and pin the bound plus victim selection (stalest first, active peer preserved); both fail on the pre-fix code (map reaches 4196+ entries). Fixes #54
|
@copilot please review this PR. Focus points:
Please do NOT push commits to this branch; leave review comments only. |
Reviewed at
|
What
The breaker peer-map cap (
maxTrackedPeers = 4096, added for #9) was not a hard bound: the sweep only removes information-free entries (closed, zero failures, no probes in flight), so a stream of distinct addresses that each carry failure state grewpeerspast 4096 without bound, contradicting the documented "peer map is bounded" contract. This PR makes the cap a true hard bound via least-recently-active eviction when the sweep frees nothing.Triage evidence (issue verified authentic)
Verified against the exact commit the issue names (
84aac4e, current main at review time):git show 84aac4e:internal/peer/breaker.go—entryLockedinvokessweepLockedand then unconditionally inserts;sweepLockeddeletes an entry only whenstate == BreakerClosed && failures == 0 && inFlight == 0. A churned address with any failure/open/half-open state is never evicted. Exactly as the issue describes.TestBreakerSweepsCleanEntrieschurns only clean addresses, so it verified the information-free sweep, not the bound under failed-address churn — also as the issue states.docs/peer.mdclaimed "The peer map is bounded" — false under failed-address churn. Doc/code contract mismatch, not a deliberate minimal-design stance (nothing in the trust model requires unbounded retention of departed-address health state).Classification: implementation (residual design gap after #9; doc/code contract mismatch fixed on the code side).
Fixes
breakerEntrygains alastActivestamp, written only by request paths (Allow— including rejected attempts — and the threeRecord*reports). Read-only observers (State/Healthy/OpenCount) never stamp, so a metrics scrape cannot launder every entry into "recently used" (this matters:OpenCountiterates the whole map per scrape).entryLocked: at the cap, sweep first; if the sweep freed nothing,evictStalestLockeddrops the entry with the oldestlastActive— by construction the address no request has touched for the longest time, i.e. almost always a departed node. If it is genuinely sick and still in rotation, the next request re-creates the entry and re-accumulates the failures, so breaker behavior for active peers is unchanged.Client, engine wiring, metrics) unaffected.docs/peer.md§3.5 rewritten to document the hard bound, the stamping rule, and why evicting a dirty entry is safe; test-inventory table updated.Regression tests
Both fail on the pre-fix code (verified by restoring
origin/main'sbreaker.gounder the new tests: map reached 4196+ entries,FAILin all four subtests):TestBreakerCapHardUnderFailedChurn(table-driven, three flavors of dirty entries: below-threshold failures / open circuits / half-open probes in flight): churnsmaxTrackedPeers + 100distinct dirty addresses, assertslen(peers) <= maxTrackedPeers, and asserts the most-recently-touched address keeps correct breaker behavior (still open / still half-open / still closed-below-threshold).TestBreakerEvictionDropsStalest: fills the map to the cap with dirty entries on an injected clock, keeps one sick peer actively routed-to (freshest stamp), churns 100 more dirty addresses, then asserts: bound holds, the stalest entry was the first victim, and the active peer's open circuit survived.Verification (actual commands + results, Go 1.22.12,
-count=1)Pre-fix failure proof (old
breaker.go+ new tests):Fixes #54