streaming: generation-qualified stream lifecycle and exact publication - #82
Merged
Conversation
Streams bind lazily to one Redis-owned generation with a distinct physical key per explicit recreation. Every metadata mutation - publication, event removal, consumer-group recovery, consumer registration, keep-alive refresh, stale recovery, acknowledgement, and destruction - verifies the exact generation atomically in Redis, so destroyed-generation metadata can never be resurrected by a concurrent sink. Adds idempotent AddOnce publication and side-effect-free Snapshot reads, Redis-owned TTL/deadline retention, and a fenced stale-recovery lease. Exported option structs keep their v1 fields as a stable ordered prefix; unkeyed literals are pinned internally and keyed v1 construction is pinned for external compatibility.
Delete the orphaned rollbackStreamRegistration helper (atomic registration made Redis rollback unnecessary), keep the exported Stream.MaxLen immutable after construction so concurrent readers never race with generation binding (the canonical bound lives in the private snapshot), and pin the v1 option fields to their leading positions with a reflect-based prefix check that keyed literals cannot provide.
The hasher closure is shared by two nodes and invoked from concurrent routing and rebalance goroutines; its job counter must be atomic. Latent on main, exposed by the streaming lifecycle's timing changes under -race.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contract
Streams bind lazily to one Redis-owned generation; every explicit recreation gets a distinct physical key, so stale handles can never touch a later incarnation. Every metadata mutation — publication, event removal, consumer-group recovery, consumer registration, keep-alive refresh, stale recovery, acknowledgement, destruction — verifies
state + generation + physical_keyatomically in Redis, which makes destroyed-generation metadata impossible to resurrect (the shipped state-only lifecycle fence could not prevent an unfenced keep-alive tick or membership append from recreating deleted maps).New capabilities: idempotent
AddOncepublication and side-effect-freeSnapshotreads (generation-scoped, Lua-linearized), Redis-owned TTL/deadline retention (expiry applied in the same operation as the write; metadata cannot outlive data), and a fenced Redis-time stale-recovery lease forXAUTOCLAIM/stale-consumer cleanup.Binding is write-once: a bound
*Streamis an immutable capability for one exact generation, so post-bind reads need no locking. Consumer registration is one atomic script (consumer group entry + membership + initial keep-alive), removing the previous three-step attach and its rollback surface.Compatibility
Stream.MaxLenstays the immutable construction value (v1 semantics); the generation's canonical bound is tracked privately.Destroycontract narrowed: an unbound handle for an absent name returnsErrStreamNotFoundinstead of silently succeeding. In-repo callers verified.Acceptance
go test ./... -raceagainst live Redis: green.TestDestroyedGenerationMetadataCannotBeRecreated), consumer rotation rollback, recovery cursor sharing, expired-stream recovery.Pre-existing non-blocking findings (max/delay shadowing in testing.go, BUSYGROUP string matching, etc.) are tracked on the
wip/full-hardeningbacklog, not here.