fix: stop y-websocket connection leaks in shared editing and reap idle /rtc connections at the gateway#6240
fix: stop y-websocket connection leaks in shared editing and reap idle /rtc connections at the gateway#6240aicam wants to merge 3 commits into
Conversation
Two distinct leaks on the SharedModel -> WebsocketProvider path exhausted the shared-editing (/rtc) connection cap and broke collaborative cursors: - Per-edit (dominant): the throwaway WorkflowGraph built for validation on every edit unconditionally created a SharedModel, opening a new /rtc socket that was never destroyed and reconnected forever. Thread an enableSharedEditing flag through WorkflowGraph -> SharedModel and pass false from ValidationWorkflowService.getValidTexeraGraph() so validation/compilation graphs run local-only (standalone Awareness, no WebsocketProvider). - Per-navigation: SharedModel.destroy() called wsProvider.disconnect() (only when connected), which leaves y-websocket's reconnect timer running so the provider is never GC'd. Call wsProvider.destroy() unconditionally instead. wsProvider is now optional; add ?. at its remaining access sites and pass enableSharedEditing=false in mock fixtures so unit tests open no sockets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
Give the shared-editing (y-websocket) /rtc path its own HTTPRoute and attach a BackendTrafficPolicy that drops idle upstream connections and raises the per-cluster max_connections circuit breaker above Envoy's 1024 default. The y-websocket reference server is single-process and every browser tab holds a long-lived WebSocket, so abandoned/leaked realtime sockets can pile up against the gateway's per-cluster connection cap and starve legitimate co-editing traffic. This is defense-in-depth for the client-side connection leak fixed in the frontend. Both the idle timeout (default 5m) and the connection ceiling (default 8192) are configurable via yWebsocketServer.trafficPolicy; set it to null to omit the policy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6240 +/- ##
============================================
+ Coverage 60.46% 60.62% +0.15%
Complexity 3370 3370
============================================
Files 1133 1133
Lines 43985 44142 +157
Branches 4782 4827 +45
============================================
+ Hits 26596 26760 +164
+ Misses 15937 15929 -8
- Partials 1452 1453 +1
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR addresses runaway /rtc (y-websocket) WebSocket growth caused by frontend shared-editing objects being created without proper teardown (and, in one path, created unintentionally for throwaway validation graphs). It also adds an Envoy Gateway policy as defense-in-depth to reap idle /rtc upstream connections and raise the connection ceiling for that cluster.
Changes:
- Added an
enableSharedEditingswitch so throwaway/validationWorkflowGraphinstances do not create aWebsocketProvider(avoids per-edit leaked/rtcsockets). - Fixed shared-editing teardown by destroying the provider (
wsProvider.destroy()) rather than only disconnecting, eliminating reconnect-timer “zombie” providers. - Split
/rtcinto its own GatewayHTTPRouteand added an optionalBackendTrafficPolicy(idle timeout + increasedmaxConnections) configurable via Helm values.
Reviewed changes
Copilot reviewed 5 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| frontend/src/app/workspace/service/workflow-graph/model/shared-model.ts | Makes wsProvider optional, supports local-only mode, and ensures provider is fully destroyed on teardown. |
| frontend/src/app/workspace/service/workflow-graph/model/workflow-graph.ts | Threads enableSharedEditing into SharedModel construction to avoid unintended network connections. |
| frontend/src/app/workspace/service/validation/validation-workflow.service.ts | Builds validation graphs with enableSharedEditing=false to prevent per-edit WebSocket leaks. |
| frontend/src/app/workspace/service/workflow-graph/model/workflow-action.service.ts | Updates provider access for optionality (but one call site still needs full null-safe access). |
| frontend/src/app/workspace/service/execute-workflow/mock-workflow-plan.ts | Updates fixtures to create local-only graphs (no sockets during tests/fixtures). |
| bin/k8s/templates/base/gateway/gateway-routes.yaml | Moves /rtc to its own HTTPRoute so policies can target it without affecting other routes. |
| bin/k8s/templates/base/shared-editing-server/shared-editing-server-backend-traffic-policy.yaml | Adds optional Envoy Gateway BackendTrafficPolicy for /rtc (idle reaping + circuit breaker headroom). |
| bin/k8s/values.yaml | Adds yWebsocketServer.trafficPolicy knobs (connectionIdleTimeout, maxConnections). |
| bin/k8s/values-development.yaml | Mirrors the new yWebsocketServer.trafficPolicy knobs for dev values. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (this.texeraGraph.sharedModel.wsProvider?.shouldConnect) { | ||
| this.texeraGraph.sharedModel.wsProvider.disconnect(); | ||
| } |
aglinxinyuan
left a comment
There was a problem hiding this comment.
You manually verified the behavior after the fix. Could you also verify it before the fix? Specifically, we'd like to confirm that there were consistently multiple WebSocket connections before this change.
Also, there's no need to list the files changed in the PR description.
Envoy Gateway's BackendTrafficPolicy exposes the connection idle timeout under
spec.timeout.http.connectionIdleTimeout; spec.timeout.tcp only has connectTimeout.
The previous timeout.tcp.connectionIdleTimeout was rejected by the CRD schema
("field not declared in schema"), which blocked the whole release from applying.
Verified against a live Envoy Gateway install on minikube.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
do we need to backport the fix to v1.2? |
Yicong-Huang
left a comment
There was a problem hiding this comment.
why do we need a enableSharedEditing config? Please find a better way to pass the config. Bundling it with WorkflowGraph is not ideal.
| validOperators, | ||
| validLinks, | ||
| texeraGraph.getAllCommentBoxes(), | ||
| /* enableSharedEditing */ false |
There was a problem hiding this comment.
-1 on this change. WorkflowGraph should not describe if shared editing is enabled.
What changes were proposed in this PR?
Fixes two distinct
y-websocket(/rtc) connection leaks in the shared-editing code path, plus a gateway-side idle-connection reaper as defense-in-depth. Both leaks live onSharedModel→WebsocketProvider, and together they let the number of open realtime connections grow far out of proportion to the number of users until the shared-editing connection cap is exhausted and collaborative cursors/awareness stop syncing.Leak B — throwaway validation graph opens a socket per edit (dominant).
On every operator/link/property edit,
WorkflowCompilingServicebuilds a throwawayWorkflowGraphviaValidationWorkflowService.getValidTexeraGraph()just to produce a logical plan for compilation.WorkflowGraph's constructor unconditionally creates aSharedModel, which unconditionally opens a new auto-connecting/rtcWebSocket in a random room. That graph is read once and never destroyed, so each edit leaks a socket that reconnects forever. A single user editing a workflow accumulates dozens of permanent sockets.Fix: thread an
enableSharedEditingflag fromWorkflowGraphintoSharedModel. Whenfalse,SharedModelskips theWebsocketProviderentirely and uses a standaloneAwarenessso downstream reads ofawareness/clientIdkeep working.ValidationWorkflowService.getValidTexeraGraph()now passesfalse, so validation/compilation graphs never touch the network.Leak A — incomplete provider teardown (per-navigation).
SharedModel.destroy()calledwsProvider.disconnect()(and only when the socket happened to be OPEN). Iny-websocket@1.4.0,disconnect()does not clear the provider's_checkInterval/_resyncIntervalreconnect timers, so the provider is never garbage-collected and, if it was mid-reconnect at destroy time (common during workflow switches), keeps re-opening a zombie socket for the life of the tab.Fix: call
wsProvider.destroy()unconditionally, which clears those timers, removes thebeforeunloadhandler, broadcasts awareness removal (peers drop our cursor), and closes the socket.Supporting frontend changes.
wsProvideris now optional (WebsocketProvider | undefined), so its remaining access sites inWorkflowActionService(setTempWorkflow/resetTempWorkflow) use?., and themock-workflow-plan.tsfixtures passenableSharedEditing = falseso unit tests open no sockets. Behavior for the real collaborative graph is unchanged: it still constructs aWebsocketProviderexactly as before.Gateway change — reap idle
/rtcconnections (defense-in-depth).The y-websocket reference server is single-process and every browser tab holds a long-lived WebSocket, so abandoned/leaked realtime sockets can pile up against the gateway's per-cluster
max_connectionscircuit breaker (Envoy's default is 1024) and starve legitimate co-editing traffic. This PR gives the/rtcpath its ownHTTPRoute(so a policy can target just that cluster without affecting the other static routes) and attaches an Envoy GatewayBackendTrafficPolicythat:connectionIdleTimeout(default5m), andcircuitBreaker.maxConnections(default8192) for headroom.Both knobs are configurable under
yWebsocketServer.trafficPolicyinvalues.yaml; settingtrafficPolicytonullomits the policy. This is complementary to the frontend fix — the frontend fix stops creating the leaked sockets; the gateway policy caps the blast radius of any that still linger.Files changed:
frontend/src/app/workspace/service/workflow-graph/model/shared-model.ts— optional/local-mode provider; correctdestroy().frontend/src/app/workspace/service/workflow-graph/model/workflow-graph.ts— threadenableSharedEditing.frontend/src/app/workspace/service/validation/validation-workflow.service.ts— passfalsefor the throwaway graph.frontend/src/app/workspace/service/workflow-graph/model/workflow-action.service.ts—wsProvider?.null-safety.frontend/src/app/workspace/service/execute-workflow/mock-workflow-plan.ts— fixtures open no sockets.bin/k8s/templates/base/gateway/gateway-routes.yaml— split/rtcinto its ownHTTPRoute.bin/k8s/templates/base/shared-editing-server/shared-editing-server-backend-traffic-policy.yaml— newBackendTrafficPolicy.bin/k8s/values.yaml,bin/k8s/values-development.yaml—yWebsocketServer.trafficPolicyknobs.Any related issues, documentation, discussions?
Fixes #6239.
How was this PR tested?
Existing frontend unit tests continue to pass; the
mock-workflow-plan.tsfixtures were updated to construct their graphs local-only so tests no longer open real/rtcsockets.Manual verification, client-side (browser DevTools → Network → WS filter):
/rtcWebSocket (unchanged).The same behavior is observable server-side by counting established connections to the y-websocket server while a single user edits: after this change the count stays flat (≈ number of open collaborative tabs) and drops when tabs close, instead of climbing with every edit.
Gateway change:
helm templatewas run to confirm the chart renders valid manifests — the/rtcHTTPRouteand theBackendTrafficPolicy(withconnectionIdleTimeoutandcircuitBreaker.maxConnections) are emitted with the policy correctly targeting only the shared-editing route, and the policy is omitted whenyWebsocketServer.trafficPolicyisnull.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)