Repository navigation
Commit 85402d0
authored
fix(mothership): end refused replay turns cleanly and bound oversized stream events (#8450)
* fix(mothership): bound replay frames and end refused turns instead of handing them off
A leased Chat stream persists every event to its Redis replay buffer before
delivering or dispatching it. When the buffer refused a write (a frame over
the 1 MiB single-write ceiling, or a stream past its 32 MiB budget), the
writer threw a generic error, the controller classified it as a takeover and
released its lock without finalizing, and every reconnect started a recovery
controller that re-received and re-refused the same event until the run
deadline. The refused tool call was never dispatched, so the worker waited.
Refusals:
- The writer throws StreamReplayBudgetExhaustedError and rejects later
publishes the same way, but still delivers the turn's terminal error and
complete events unpersisted; flush no longer rethrows the refusal.
- The controller aborts with that error rather than a supersession, the
lifecycle classifies it as an error (not a user cancel), the run is
finalized as an error with code replay_budget_exhausted, and the worker is
told to stop. Reconnects then see a terminal run and start no controller.
- Only a genuine StreamControllerSupersededError still hands the run off.
Compaction, applied only to the copy the writer delivers and persists:
- Payloads over 256 KiB have long string leaves cut to an 8 KiB head in
place, then medium strings to 500 characters, and only then the largest
unprotected subtrees replaced by { omitted, bytes }, targeting 128 KiB.
A payload-level streamTruncation marker lists every changed field.
- Identity fields, UI-read keys, arguments of client-executable calls, file
preview events, and generate_api_key results are never compacted; such a
frame that stays too large is refused and ends the turn cleanly.
- Oversized assistant text splits into contiguous exact pieces, and each tool
call forwards only the head of its argument deltas.
* fix(mothership): render stream-omitted tool values honestly
A call frame whose whole argument object was omitted by stream compaction no longer replaces a tool node's arguments, and the permission card and generic tool output render omitted values as "Too large to show (N KB)" instead of raw marker JSON.
* refactor(mothership): replace replay compaction with one string-truncation pass
Compaction is now a single linear walk: when a payload's strings could
serialize past 256 KiB, string leaves longer than 8K units are cut to their
head with an inline "…[truncated, N total]" note, copying only what changes.
Identity and UI-read keys, file previews, generate_api_key results, and the
arguments of calls the browser executes are never cut; anything still over
the write ceiling is refused and ends the turn. The subtree omission, second
cut pass, truncation marker, text splitting, argument-delta cap, and the UI
stub handling are removed.
One predicate, isClientExecutedToolCall, now names the tool calls the browser
starts from the call frame; both the stream compaction and the client
dispatch use it, and the workflow tool names move beside it.
Also fixes three defects in the refusal path:
- A superseded controller that is refused an oversized frame no longer
expires its successor's stream or clears its abort marker; teardown keys on
whether this controller handed off, not on the abort reason.
- A refusal after a user Stop stays a cancellation: the abort reason decides.
- The per-user hourly ceiling gets its own message instead of telling the
user to continue immediately.
* fix(mothership): stop stale approval cards, raw backend bodies, and errored-turn spinners
- A replayed call frame is history and never gates anything, so Sim now clears
the worker's awaiting_approval stamp on it as it already does for live
frames. Replays no longer render Allow/Skip cards when approvals are off.
- Stop settles every unfinished tool row (pending, executing, or awaiting
approval) through one shared predicate, isUnsettledToolState, on the
persisted, live, and pending-message paths.
- An errored turn settles its unfinished tool rows as errored when it is
persisted, so they no longer reload as spinners.
- A failed backend response no longer puts the upstream body in the error
shown to users: 5xx and non-JSON bodies say the service is temporarily
unavailable, 4xx uses the worker's displayMessage or a generic message.
The status and body stay on the error for logs.
* fix(mothership): never forward an approval stamp on a frame that cannot be gated
The worker stamps awaiting_approval on resolved integration calls and relies
on Sim to keep or clear it. Sim cleared it on live complete frames that were
not gated, but a partial frame returned before that decision and kept the
stamp. Only a live, complete call frame can be held behind a prompt, so every
replayed or partial frame now drops the stamp at the one point each frame
passes through before it is persisted and delivered. With approvals off, no
frame, replay, snapshot, or persisted row can carry the stamp to the client.
* fix(mothership): keep retrying an unreachable worker through a task replacement
A worker task replacement leaves the load balancer answering 502/504 for
about a minute. Stream retries stopped after three attempts, about four
seconds, and ended the turn with an error. While no worker answers (a 5xx or
no connection), retries now continue with the existing backoff for up to two
minutes from the first failure, still bounded by the run deadline and by
Stop. Each attempt re-sends the same message identity, which the worker
treats as a reattach rather than a second run. Streams that end without a
terminal keep the three-retry limit.
* fix(mothership): settle runs whose terminal events fail, and tighten compaction and cleanup
- A controller that lost its lock while handling an ordinary failure no
longer expires its successor's stream or clears its abort marker: teardown
skips cleanup when this controller handed off or was superseded.
- The terminal run status is now written even when publishing the terminal
events fails. The write stays scoped to this controller's token and never
replaces a terminal status, so a successor's run is left alone. Before, such
a failure left the run non-terminal with nothing to settle it.
- A failed backend response is described by its status alone: 5xx says the
service is temporarily unavailable, anything else that the request could
not be processed.
- Compaction no longer exempts whole subtrees by key name. Every string the
UI reads for identity, status, titles, or targets is far shorter than the
cut, and long text keeps its head, so only client-executed call arguments,
file previews, and generate_api_key results are left whole.
* fix(mothership): settle tool rows left unfinished by any finished turn
- A completed turn now persists its unfinished tool rows settled as the live
view settled them at complete, so they no longer reload as spinners.
- A stored message's unfinished tool rows render as interrupted, which settles
rows persisted before these fixes; the live streaming message is unchanged.
- The client's error finalize settles pending and awaiting-approval rows too:
finalizeResidualToolCalls reports whether it settled anything, and that
decides whether the snapshot is rewritten.
* fix(mothership): harden replay compaction, stream retries, teardown, and backend errors
- Assistant text is never compacted: its length is part of the text receipt a
replacement controller and the worker check. When cutting strings alone
cannot bound an event, long arrays keep their first 100 items and a count.
- Only a gateway error page (502/503/504 without a JSON body) or a failed
connection counts as an unreachable worker for the two-minute window; a
JSON 5xx from a reachable worker keeps the three-retry limit. A retry that
streams again resets the budget for the next outage.
- A controller cleans up its stream's buffer and abort marker only while it
still holds the chat lock, checked just before cleanup and before releasing
the lock, instead of inferring ownership from flags.
- A controller superseded while publishing its terminal events leaves the run
to its successor; other publish failures still settle the run.
- A 4xx shows the worker's own short, plain reason (rate limit, BYOK, plan
mode, model selection, version skew) and hides internal identifiers.
- File preview snapshots are spaced by size, so a large file's patch preview
can no longer fill the stream's replay budget; small files keep their pace.
* fix(mothership): separate retry budgets, cap preview totals, and clean up only ended turns
- An unreachable worker and a reachable one now have independent retry
budgets. Any answer from the worker restarts only the two-minute
unreachable window; the three-retry, 30 s budget for failures of a worker
that answered keeps its original semantics, so a failure that repeats
after every reattach still stops, and a long outage no longer spends the
replacement's retries.
- Intermediate file preview content is capped at 8M characters per edit;
past it the preview holds until the final snapshot, which is always sent.
- A controller cleans up its stream only when it finalized the turn and
still holds the lock, so a run it leaves recoverable keeps its buffer and
any pending Stop.
- Remove the unreachable generate_api_key compaction exemption, avoid a
second copy of trimmed arrays, and correct the size-estimate comment.
* fix(mothership): clean up turns whose terminal publish failed and cap previews per turn in bytes
- A controller that ended its turn but could not publish the terminal events
still cleans up its stream: the run is settled either way, so the buffer no
longer lingers for its full TTL. Only a superseded controller skips cleanup,
and the lease check still protects a successor.
- The intermediate file preview cap is now 8 MiB of UTF-8 per stream loop,
across all of its edits, instead of characters per edit. Each edit's final
snapshot is still always sent.
* fix(mothership): bound preview frames per write and per turn, and narrow unreachable retries
- A file preview frame whose content would not fit one replay write, or that
would exceed the turn's preview budget, is skipped instead of sent: the
preview holds its last content and the client loads the stored file on
completion. A completion too large to send whole drops its tool output.
The per-frame limit derives from the replay buffer's single-write ceiling,
and the 8 MiB budget now covers every preview frame in the turn, finals
included, across all legs. A large file preview no longer ends the turn.
- Only a request that fails before any response headers counts as an
unreachable worker. A TypeError raised after the response began, including
one from handling delivered events, keeps the three-retry budget, so a
deterministic failure can no longer retry until the run deadline.
- A worker that cannot be reached is reported with the generic "temporarily
unavailable" message; the network error stays on the error's cause.
- Retry logs and spans count retries from both budgets.
* fix(mothership): omit the largest field of an event no cut can bound, and persist closed lanes
- An event whose many short values survive every string and array cut, such
as an object with thousands of keys, now has its largest bulk field
replaced by a size note when it would still exceed one replay write,
instead of ending the turn. Identity fields and client-executed arguments
are never replaced; such an event is still refused cleanly.
- finalizeResidualToolCalls reports closing an open subagent lane as a
change, so the error finalizer persists it instead of leaving a stale lane.
* fix(mothership): keep sibling fields when omitting bulk, and never cache a skipped preview as file content
- The last-resort omission for an event no cut can bound now walks down from
the largest field while one child holds most of its parent's bytes and
replaces only that node, so the ids, resources, and status fields beside
the bulk keep driving the UI's resource updates.
- On completion the client seeds the file content cache from preview text
only when it received the completed version. When the server skipped the
final content frame, the text is an earlier draft, so the stored file loads
instead.
* fix(mothership): keep omitting bulk until an event fits one replay write
A result with more than one large object of short fields left the second
one over the write limit after the first was omitted, so the event was
refused and the turn ended. Omission now repeats until the event fits or
nothing large enough is left; each pass removes at least 64 KiB, so it
always terminates.
* fix(mothership): shed oversized replay events in one pass and retry mid-body stream cuts
- Replace the replay-compaction omit loop with one bounded pass over memoized
exact serialized sizes: recurse into the smallest child that covers the
overage, otherwise drop the largest children whole, keeping small siblings
- Map a non-abort body-read failure from the worker stream to a retryable
WorkerStreamInterruptedError on the reachable budget, with the generic
message and the original error on cause
- Log the cause message on retry warnings and orchestration failures
- Count replayed file preview content toward a restored turn's preview budget
* refactor(mothership): tidy replay compaction and stream retry after review
- Share the replay write ceiling between compaction and file previews
- Scan for the smallest sufficient child only once one can cover the need,
and reuse one size memo across every compaction step
- Stop retrying a raw TypeError; fetch and body-read failures are already
wrapped as WorkerUnreachableError and WorkerStreamInterruptedError
- Use absolute imports, a private retry counter, a typed unsettled-state set,
and hasUnsettledTool naming
- Cover a timed-out body read and the preview completion edge cases
* fix(mothership): bound preview metadata frames and settle stopped rows on every save
- Compact every preview phase except content and completion, which the preview
adapter already bounds; a model-written patch search string could otherwise
exceed one replay write in an edit_meta frame
- Settle unfinished tool rows as stopped in buildPersistedAssistantMessage for a
cancelled turn, so background and API callers that persist the result
directly never save pending, executing, or awaiting-approval rows1 parent 10726ce commit 85402d0
41 files changed
Lines changed: 3334 additions & 312 deletions
File tree
- apps/sim
- app
- api/copilot/confirm
- workspace/[workspaceId]/home/hooks
- preview
- stream
- lib/mothership
- chat
- request
- context
- go
- handlers
- lifecycle
- session
- tools
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
45 | 46 | | |
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
49 | 50 | | |
50 | 51 | | |
51 | | - | |
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| |||
Lines changed: 28 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
Lines changed: 10 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
12 | 16 | | |
13 | 17 | | |
14 | 18 | | |
| |||
18 | 22 | | |
19 | 23 | | |
20 | 24 | | |
21 | | - | |
| 25 | + | |
22 | 26 | | |
23 | 27 | | |
24 | 28 | | |
| |||
142 | 146 | | |
143 | 147 | | |
144 | 148 | | |
145 | | - | |
146 | | - | |
| 149 | + | |
| 150 | + | |
147 | 151 | | |
148 | 152 | | |
149 | | - | |
| 153 | + | |
150 | 154 | | |
151 | 155 | | |
152 | 156 | | |
153 | 157 | | |
154 | 158 | | |
155 | 159 | | |
156 | | - | |
| 160 | + | |
157 | 161 | | |
158 | 162 | | |
159 | 163 | | |
| |||
Lines changed: 41 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
4 | 7 | | |
5 | 8 | | |
6 | 9 | | |
| |||
94 | 97 | | |
95 | 98 | | |
96 | 99 | | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
Lines changed: 14 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
Lines changed: 5 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
16 | 19 | | |
17 | 20 | | |
18 | 21 | | |
| |||
381 | 384 | | |
382 | 385 | | |
383 | 386 | | |
384 | | - | |
| 387 | + | |
385 | 388 | | |
386 | 389 | | |
387 | 390 | | |
| |||
Lines changed: 0 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | 7 | | |
11 | 8 | | |
12 | 9 | | |
| |||
Lines changed: 20 additions & 51 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
18 | | - | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
| |||
188 | 190 | | |
189 | 191 | | |
190 | 192 | | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | | - | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
199 | 203 | | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
241 | 210 | | |
242 | 211 | | |
243 | 212 | | |
| |||
Lines changed: 37 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
Lines changed: 11 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
| |||
68 | 69 | | |
69 | 70 | | |
70 | 71 | | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
80 | | - | |
| 80 | + | |
81 | 81 | | |
| 82 | + | |
82 | 83 | | |
83 | 84 | | |
84 | 85 | | |
| |||
93 | 94 | | |
94 | 95 | | |
95 | 96 | | |
| 97 | + | |
96 | 98 | | |
97 | 99 | | |
98 | 100 | | |
99 | | - | |
| 101 | + | |
| 102 | + | |
100 | 103 | | |
101 | 104 | | |
102 | 105 | | |
| |||
105 | 108 | | |
106 | 109 | | |
107 | 110 | | |
| 111 | + | |
108 | 112 | | |
109 | 113 | | |
110 | 114 | | |
| |||
0 commit comments