fix(connections): keep table loads working while a database switch reconnects

[datlechin]

Stacked on #2824. Review and merge that first; this branch targets fix/connections-gate-ownership.

Problem

On an engine that reconnects to change database (PostgreSQL, Redshift, CockroachDB), a table tab could fail or come up empty while its connection switched database.

1. A table load queued behind the switch was refused. Tabs A and B are on app. A slow load in A holds the driver, the user picks orders, and the switch queues. Paging B decides its route while app is still the browsed database, so it queues for the session driver behind the switch. The switch moves the browsed database to orders, and when B's turn comes it fails with "This tab is on app. Switch the connection to that database to run it." Running B again works, on a pooled connection.
2. A table on another database opened empty with no error. A table tab on reports loads through a pooled connection that is still dialing. The user switches to sales, the switch's reconnect closes every pooled connection and cancels every open in progress, libpq's connect throws CancellationError, and the tab treats that as a user cancel: no rows, no error. Every idle pooled connection to other databases was closed too.

Root cause

1. The route is a judgement about session state (resolvedBrowseDatabase), and every execution caller makes it before taking the FIFO session driver gate. Inside the gate pin can only refuse. For user SQL that refusal is right, because the session the statement was written against is gone. An app-built table SELECT depends on nothing the session holds, and at that moment executionRoute would answer .pooled.
2. Pool invalidation lived in DatabaseTreeMetadataService.handleReconnect and ran for every reconnect. A pooled entry depends only on the session's effective connection plus its own database. A reconnect-based switch over a direct connection changes neither. What does change it is a rebuilt tunnel, which binds a new local port, or a recovery from a connection that stopped answering.

Fix

Table reads follow the route they waited through.

• New DatabaseManager.withTableReadDriver(scope:cancellation:_:), beside withScopedDriver and modelled on withMetadataDriver, resolves executionRoute itself.
• On .sessionDriver it takes the session driver's turn. After the existing isUsable check it asks executionRoute again inside the gate. If the answer is no longer .sessionDriver, it returns without running the body, leaves the gate, and dispatches again: .pooled goes to MetadataConnectionPool, .unavailable throws its message.
• The gate is never held across a pooled read. A moved turn only carries a route that does not queue on the gate again, so the re-dispatch ends.
• The second decision goes through executionRoute, so it reads session.resolvedBrowseDatabase exactly as the first one did, never the driver's own connection.
• withScopedDriver and pin are unchanged, so every other caller is still refused. The tracked-cancellation wrapper and the gate turn (verify, isUsable, gate, trackOperation, second isUsable) are now private helpers both paths share.
• Opt-in is MainContentCoordinator.withExecutionDriver(scope:isTableTab:_:), used by executeQueryInternal and by Fetch All. It keys on tab.tabType == .table only, never on QueryClassifier, isAutoLoad or the cancellation policy, because an editor SELECT can read a temp table or sit inside the user's open transaction. First load, pagination, filters, sort and refresh all reach executeQueryInternal through executeTableTabQueryDirectly. A driver-built EXPLAIN run from a table tab also goes through executeQueryInternal, so it takes this path too; its SQL is built from the tab's own SELECT.

The owner of the transport fences the pool.

• handleReconnect no longer touches the pool. MetadataConnectionPool gained beginTransportReplacement(connectionId:) and endTransportReplacement(connectionId:).
• begin withdraws pending opens tagged .transportReplaced, closes idle entries, defers in-flight ones and parks new acquires. end resumes them. Replacements nest by count.
• A caller whose open was withdrawn as transport-replaced acquires again after end, whether the withdrawn open threw (libpq's CancellationError) or returned without keeping its entry. It checks its own cancellation first, so a stopped load does not reopen. A parked caller that is cancelled leaves at once, through tickets like SessionDriverGate's.
• closeAll(connectionId:database:) (rename) and closeAll(connectionId:) (disconnect) tag .closed: the waiter still fails and nothing reopens. A caller parked for a replacement whose scope the close matches is failed too, so it never wakes to open a connection for a session or a database that has gone.
• reconnectSession calls begin only when the connection has an active tunnel kind, or when the session was not .live as the reconnect started. Its only caller is the reconnect-based database switch, so the liveness check is what tells a switch that recovers a failing session from a plain one; no purpose parameter was added for a single caller. performHealthMonitorReconnect always calls it, being a recovery. recoverDeadTunnel is left exactly as it was; see "Found in review, not fixed here".
• Each end is a defer placed right after its begin, so it runs after the replacement effective connection is installed and on every exit: a thrown error, a cancellation and a declined password prompt.
• Opening a pooled driver moved into a static openSessionDriver(for:) that the pool takes as its opener, so a DEBUG pool can stand in a test opener.

Tests

Every schedule is ordered with explicit handshakes: a latch the holder opens from inside the gate, waiterCount and the new transportWaiterCount polled with a bound. Nothing counts yields to decide an order. Every injected session sets status = .connected.

• DatabaseSwitchLeaseOrderingTests, PostgreSQL session on app with a pooled app connection seeded:
  • tableReadFollowsTheSwitchOntoThePool: hold the gate, queue a table read for app, move browse to orders, release. The body runs on the pooled driver.
  • otherWorkQueuedThroughASwitchIsStillRefused (guard): the same schedule through withScopedDriver(route: executionRoute) still throws queryFailed naming app, and the body never runs. This is what keeps a COMMIT from being re-routed.
  • reroutedTableReadDoesNotHoldTheGate: while the re-routed read blocks on a latch inside the pool, sessionDriverGate.withExclusiveAccess acquires straight away and nothing is queued.
  • tableReadBehindADeadDriverIsRefused (guard): a holder that marks the session .unreachable makes the read throw notConnected, and it never reaches the pool.
  • tableReadWithNowhereToGoNamesItsDatabase (guard): a fake reconnect-switch type that cannot pool throws the unavailable message naming app.
• MetadataConnectionPoolTransportReplacementTests, on an isolated pool with a recording opener:
  • withdrawnOpenIsRetriedAfterTheReplacement: a lease waiting on a slow open survives begin and end and runs on the second driver.
  • leaseDuringAReplacementWaitsToOpen: after begin, a new lease calls no opener until end, then exactly one.
  • overlappingReplacementsHoldUntilTheLastEnds: two begins need two ends.
  • cancelledWaiterStopsWaiting: a parked lease that is cancelled throws CancellationError at once and leaves no waiter.
  • closingADatabaseFailsItsWaiter (guard): closeAll(connectionId:database:) withdrawing a pending open fails its waiter, and the opener runs once.
  • closingAConnectionFailsParkedLeases (added in review): a disconnect during a replacement fails the parked lease at once, and it opens nothing.
  • closingADatabaseFailsOnlyItsParkedLeases (added in review): closing shop fails the lease parked for shop, and the one parked for reports still runs after end.
• New SwitchDatabasePooledConnectionTests:
  • switchLeavesOtherDatabasesPooled: a direct session, a pooled reports connection, a failing switchDatabase. pooledDriverCount stays 1, disconnectCallCount stays 0, and no replacement is left open.
  • treeReconnectLeavesThePool: handleReconnect no longer closes an injected entry.
  • recoveringSwitchClosesThePool (guard): the same failing switch on a session that stopped answering still closes the pooled connection, and the fence is released after the failure.
  • healthReconnectClosesThePool (guard): performHealthMonitorReconnect closes the pooled connection and releases the fence.

Negative control. The base branch's versions of the 8 changed production files were swapped in. Only the seams the new tests call were added back, each with the base behaviour: withTableReadDriver delegating to withScopedDriver(route: executionRoute(for:)), beginTransportReplacement as closeAll(connectionId:), endTransportReplacement doing nothing, the waiter count and isReplacingTransport answering 0 and false, and an opener override. Only the 3 suites with new cases were run: 20 cases, 8 failed, 12 passed. The 8 failures are the 8 non-guard cases above. The 6 guard cases and the 6 existing cases in DatabaseSwitchLeaseOrderingTests passed. The files were then restored, and each one's hash matched its committed blob.

The three cases added in round one were controlled the same way against the first commit: its MetadataConnectionPool.swift and DatabaseManager+Tunnel.swift swapped in, the two suites holding them run. 12 cases, 3 failed, 9 passed, and the 3 failures were exactly those three. Both files were restored and their hashes matched. Round two then cut the tunnel recovery change back, and its case went with it; the two pool cases remain.

No UI automation: the change is in driver and pool ordering, which the unit tests control deterministically and a UI test could only reach through timing.

Verification

All through .claude/skills/fix-issue/scripts/verify.sh.

• generate: PASS.
• build (TablePro scheme): PASS.
• lint: PASS, 0 violations on the 11 Swift files of the first commit, again on the 4 changed in round one, and on the files touched by the cut-back.
• On the final commit (a241591), the spec suites plus DatabaseManagerTunnelTests: PASS, 116 of 116. The same set on b2e1759, before the cut-back removed one case: PASS, 117 of 117.
• On the final commit, every other suite referencing a changed type, plus the gate and switch suites from fix(connections): fail queued driver work on disconnect and gate Redis database selection #2824: PASS, 167 of 167 (the same 27 suites listed below).
• The three new suites run a second time on the first commit to check for flakiness: PASS, 20 of 20.
• After the round one fixes, MetadataConnectionPoolTransportReplacementTests, SwitchDatabasePooledConnectionTests, DatabaseSwitchLeaseOrderingTests, MetadataConnectionPoolTests, MetadataConnectionPoolIdleEvictionTests, MetadataConnectionPoolPlanTests, MetadataConnectionPoolLostEntryTests, HealthMonitorReconnectTests, DatabaseManagerTunnelTests, DatabaseManagerSchemaChangeRoutingTests, ScopedDriverRoutingTests, DatabaseManagerDisconnectTests: PASS, 91 of 91.
• Tests on the first commit, spec suites: PASS, 104 of 104. DatabaseSwitchLeaseOrderingTests, SwitchDatabasePooledConnectionTests, ScopedDriverRoutingTests, ScopedDriverPinningTests, MetadataConnectionPoolTests, MetadataConnectionPoolIdleEvictionTests, MetadataConnectionPoolLostEntryTests, MetadataConnectionPoolPlanTests, MetadataConnectionPoolTransportReplacementTests, SwitchDatabaseReconnectFailureTests, DatabaseManagerSchemaChangeRoutingTests, CancelledExecutionOwnershipTests, ConnectionVerificationTests, PluginDriverAdapterLostConnectionTests, HealthMonitorReconnectTests.
• Tests on the first commit, every other suite referencing a changed type, plus the gate and switch suites from fix(connections): fail queued driver work on disconnect and gate Redis database selection #2824: PASS, 167 of 167. ScopedDriverCancellationTests, DatabaseManagerDisconnectTests, TriggerInfoMappingTests, StructureTabTriggersTests, TriggerApplyStrategyTests, TriggerEditingBridgeTests, TriggerApplyExecutionTests, SchemaProviderRegistryTests, SchemaRefreshAfterWriteTests, DatabaseTreeCatalogRefreshPlanTests, DatabaseTreeMetadataServiceTests, DatabaseTreeMetadataServiceRefreshTests, DatabaseTreeMetadataServiceRefreshDatabasesTests, DatabaseTreeMetadataServiceRefreshObjectsTests, SchemaRefreshServiceTests, PaginationCoordinatorTests, MainContentCoordinatorLazyLoadTests, StructureEditingSessionTests, TableStructureLoaderScopeTests, ForeignKeyReferenceMenusTests, QueryCompletionProfileRegistryTests, SessionDriverGateTests, RedisDatabaseSelectionGateTests, SwitchDatabaseTests, SwitchSchemaTests, DatabaseManagerDatabaseSwitchTests, DatabaseManagerSessionTests.

Review. Codex is out of usage until 2026-09-19, so both rounds used Claude's code-review skill, reading the committed branch diff against fix/connections-gate-ownership. Every finding was checked against the code.

Round one, 4 findings:

• Medium: recoverDeadTunnel held the pool across its whole backoff loop, up to 10 attempts with waits reaching 120 seconds, so every pooled read on the connection spun with no error for the whole recovery where it used to fail at once. Real, and introduced here. Fixed in round one by holding the pool around each connect attempt only, with a test. Round two showed that shape leaves the dead port reachable during the backoff, so the change was cut back; see round two.
• Medium: nothing on teardown failed a caller parked for a replacement, and the replacement count is keyed by connection id, so a parked read from a session that ended could wake onto a reopened one. Real. Fixed: a .closed withdrawal fails the parked callers whose scope it matches, as the gate drain does for gate waiters. Tests: closingAConnectionFailsParkedLeases, closingADatabaseFailsOnlyItsParkedLeases. A stale count now lasts at most one reconnect. See below.
• Low: withdrawing an open does not wake the callers awaiting it, so after a driver that ignores cancellation they wait for that open to finish, bounded by the pool's 15 second connect and 60 second preparation deadlines. Real, and the same on the base branch, where a cancelled open was awaited the same way. The doc comment now says callers retry once the withdrawn open has returned. See below.
• Low: a rename during a replacement could not reach work the replacement had moved aside. The parked half is fixed by the second fix. The other half is below.

Round two, reading both commits, 4 findings:

• Medium: holding the pool only around each recovery attempt leaves the backoff before the first attempt uncovered. For at least two seconds the session still carries the dead tunnel's port, a pooled open dials it, and another connection's new tunnel can have bound that port. Pooled connections built on the dead tunnel also stay until the first attempt. Real, and the per-attempt hold's own comment claimed the opposite. Round one had shown that holding across the whole recovery hangs pooled reads for minutes, so rather than try a third shape the tunnel recovery change was cut back: recoverDeadTunnel is byte for byte the base branch's, and its test went with it. See "Found in review, not fixed here".
• Medium-low: a failed tunneled reconnect wakes parked callers onto an effectiveConnection that still names the replaced port, and on the health monitor path that repeats at every retry. Real, and already true on the base branch, where nothing parked them at all; it is the second item under "Found while investigating fix(connections): hold the session driver while a database switch reconnects #2820".
• Low: reconnectSession holds the pool across its password prompts. Real. The approved design ends the hold after a declined prompt, so this is that design's cost and was not changed. See below.
• Low: a caller parked for a replacement and failed by a rename gets CancellationError, which a table tab reports as a user cancel. Real, and the same as a pending open withdrawn by a rename on the base branch, so it is folded into that item below. A disconnect failing a parked caller with CancellationError matches how the gate drain reports queued work.

No third round was run.

Found while investigating #2820

• The first load after a switch now opens a pooled connection where it used to fail at once. The pool holds up to 6 per connection, and one costs 800 to 1900 ms across the internet.
• A failed tunneled reconnect closes its new tunnel (closeReconnectTunnels) but leaves effectiveConnection naming that port, so leases resumed by end dial a port no tunnel owns, which another connection's tunnel could bind.
• A rename still withdraws a pending open with CancellationError, which a table tab reports as a user cancel: blank, no error. A caller parked for a replacement and failed by the rename gets the same error. A typed error for the rename withdrawal would cover both.

Found in review, not fixed here

• A caller already awaiting an open when a replacement withdraws it keeps waiting until that open returns. With a driver whose connect ignores cancellation, that is up to the pool's own connect and preparation deadlines, and Stop does not reach the caller while it waits, because awaiting an unstructured task does not forward cancellation. Both were already true on the base branch. Waking those callers needs the withdrawal to resume them directly, which is a change to how every pending open is awaited.
• A database rename that runs while a replacement is in progress cannot reach callers of an open the replacement already withdrew: that open has left the pending list, so they retry after end and can open a backend on the database being renamed, which PostgreSQL then refuses to rename. Nothing fences new acquires during a rename today either.
• The replacement count is keyed by connection id, not by session. A disconnect and reopen during one reconnect leaves the reopened session's pooled work waiting until that reconnect returns. A replacement owned by a ticket that a disconnect clears, as SessionDriverGate owns its turns, would close this.
• A tunnel recovery does nothing about the pool, as before this change. Pooled connections built on the dead tunnel stay until they fail, and an open during the recovery dials the dead port, which another connection's tunnel can bind and then serve with this connection's credentials. Holding the pool across the whole recovery left pooled reads spinning for minutes (round one), and holding it per attempt left the backoff open (round two). The shape that would work is for the recovery to retire the pool at once, fail pooled opens fast while the tunnel is down, and hold them only while the new tunnel is being built.
• reconnectSession holds the pool across its password prompts, so on a tunneled connection whose password changed, pooled reads for that connection wait until the sheet is answered.

https://claude.ai/code/session_01SNC28GzXZvLwbgAB8G583A