fix(api): make WatchSandbox loss-aware and resumable - #3209
Draft
letv1nnn wants to merge 9 commits into
Draft
Conversation
…nating Broadcast lag on the status, log, and platform receivers was converted to a RESOURCE_EXHAUSTED status that terminated the whole watch stream. Lag is recoverable: the receiver resumes at the oldest surviving message. Emit a SandboxStreamWarning and continue streaming instead; keep terminating on Closed. Add helpers and unit tests covering the warning payload and receiver recovery after lag. Partially addresses NVIDIA#3055 (cursor/resume follow up separately). Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
…numbers Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Allocate cursors from a single SeqAllocator shared by the log and platform event buses, so a sandbox's merged watch stream carries unique, strictly increasing cursors. A single resume_after_cursor can then unambiguously locate a client's position across both sources. Rewrite both publish paths to allocate the sequence, stamp event.cursor, send, and append to the tail under one lock. This removes the previous get_mut().expect() TOCTOU race where a concurrent remove() between the two lock sections could panic. Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Add tail_after() to the log and platform event buses, returning every buffered event newer than a client's resume cursor. Each PerSandbox now tracks last_trimmed_seq (the highest seq it has evicted) so a resume is reported as an unrecoverable ResumeGap only when this bus dropped an event the client still needs. Judging gaps by evictions, not by the tail's oldest seq, is required under the shared cursor space: each bus's tail is non-contiguous in the global sequence because the other bus owns the missing seqs, so comparing against tail.front() would flag false gaps. Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
… buses Wire resume_after_cursor into the watch producer. On a non-zero cursor, replay events strictly after it from both the log and platform buses, merge by shared cursor, and emit in order before entering the live loop. A trimmed range on either bus is an unrecoverable gap and terminates the stream with OUT_OF_RANGE carrying the requested and earliest-available cursors, distinct from recoverable lag which warns and continues. Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Add handler-level tests for the resumable watch stream: replay strictly after the client cursor, merge log and platform events in shared-cursor order, suppress duplicates when resuming at the latest cursor, and terminate with OUT_OF_RANGE when the requested cursor has been trimmed. Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make
WatchSandboxloss-aware and resumable. Recoverable broadcast lag now emits aSandboxStreamWarningand keeps streaming instead of terminating withRESOURCE_EXHAUSTED. Every resumable event (log line, platform event) carries a monotonic per-sandboxcursor, and a reconnecting client can passresume_after_cursorto replay only what it missed. Unrecoverable loss (a trimmed cursor) terminates the stream withOUT_OF_RANGEso gaps are never silent.Draft / work in progress — server, proto, and docs are complete; SDK reconnect helpers and e2e coverage are still pending (see TODO).
Related Issue
Refs #3055 (partial; issue stays open until SDK helpers land).
Changes
SandboxStreamWarningand continue onRecvError::Lagged;Closedstays terminal. Addslag_warning/lag_warning_eventhelpers.TracingLogBusandPlatformEventBusdraw sequence numbers from one per-sandboxSeqAllocator, so the merged stream is ordered in a single cursor space. Each buffered event is stamped at publish time.SandboxStreamEvent.cursorandWatchSandboxRequest.resume_after_cursor.cursor = 0marks non-resumable events (status snapshots, warnings);resume_after_cursor = 0means no cursor resume (tail-limited replay vialog_tail_lines/event_tail).tail_after(cursor)returns events after a cursor or aResumeGapwhen the requested cursor was trimmed. The producer replays both resumable sources after the cursor, merged in cursor order, then enters live delivery. A gap terminates withOUT_OF_RANGEcarrying the requested and earliest-available cursors.TracingLogBus::removeclears both resumable per-sandbox maps before resetting the shared allocator, closing a publish-during-teardown window.proto,architecture/gateway.md, anddocs/observability/accessing-logs.mdx.TODO (follow-up)
resume_after_cursor, surface warnings, signal gap).stop_on_terminalERROR regression.Testing
cargo test -p openshell-server— all pass, including resume replay, cross-source merge order, duplicate suppression, gap →OUT_OF_RANGE, init-race single-delivery, and lag-warning regression.mise run pre-commitChecklist