Repository navigation
[SSH] Fail fast when the driver proxy refuses the tunnel - #7057
Merged
Merged
Conversation
When an administrator disables port forwarding to serverless compute, the driver proxy answers every request to the tunnel server with 403 FEATURE_DISABLED. ssh connect treated that like a server that was still starting, kept polling /metadata for the full 5-minute timeout, and then failed with a generic "status code 403" error while the compute it had started stayed up. A 403 from /metadata is now final: connect stops on the first one with the proxy's own message, both in the wait loop and on the reuse path (where it would otherwise submit another server), and cancels the run it submitted. Every other non-200, such as 400 INVALID_STATE from a stale metadata.json, is still retried. The test server models the port_fwd_serverless_enabled setting so the acceptance test reaches the 403 through the setting rather than a stub. DECO-28889 Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The settings v2 backend decodes the PATCH body as a protobuf message, in
which false is the default, so {"boolean_val": {"value": false}} arrives
empty and is rejected with "must contain at least one subfield with a
non-default value" (seen on dogfood). The fake accepted it, so the
acceptance test turned the setting off with a request the real API
refuses. The fake now rejects it the same way, and the test sends the
name in the body, which is what the real API needs.
Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Oct 14, 2026
Collaborator
Integration test reportCommit: 86d1e8b
Top 10 slowest tests (at least 2 minutes):
|
louis-johnston-db
approved these changes
Oct 15, 2026
anton-107
enabled auto-merge
October 15, 2026 12:11
rclarey
approved these changes
Oct 15, 2026
Collaborator
Integration test reportCommit: 4d4ed66
16 interesting tests: 12 flaky, 4 FAIL
Top 50 slowest tests (at least 2 minutes):
|
Collaborator
Integration test reportCommit: 4d4ed66
83 interesting tests: 83 FAIL
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
databricks ssh connectnow stops at the first 403 from the driver proxy and prints the proxy's own message, instead of polling/metadatauntil the 5-minute server start timeout. It also cancels the run it submitted, since no client can ever reach that server.getServerMetadataturns a 403 into the newerrDriverProxyForbidden, carrying the response body with its JSON quotes removed. It isn't joined witherrServerMetadata, so the first call (the reuse path) returns it as is instead of submitting a new server.cancelSSHServerRun). A server found running on the reuse path is never cancelled.400 INVALID_STATEfrom ametadata.jsonthat points at a terminated cluster.libs/testservergains a settings v2 PATCH handler, and its fake driver proxy returns the realFEATURE_DISABLEDbody whileport_fwd_serverless_enabledisfalse.Why
When an administrator turns off
port_fwd_serverless_enabled, the driver proxy answers every request to the tunnel server with403and the body"FEATURE_DISABLED: …"(a JSON string, not the usual error object). The client counted any non-200 as "server not up yet", so the user watched "Waiting for the SSH server to start..." for 5 minutes and then gotserver is not ok, status code 403, while the serverless compute stayed up.This is PR 1 of DECO-28889. PR 2 adds a check of
port_fwd_serverless_enabledbefore any compute is started, with this 403 handling as the fallback when the setting can't be read. PR 3 adds telemetry error categories for both.Tests
TestMetadataResponseErrorcovers a 403 with the real body, a non-JSON body and an empty body, plus a 400 and a 502 that are still retried.acceptance/ssh/connect-serverless-feature-disabledturns the setting off through the API, then checks the error and the recordedruns/cancel. It finishes in 1.6 s, so connect gives up on the first 403. All otheracceptance/sshtests still pass.1035981371775948endedTERMINATED / CANCELED.0907-190337-c3qelmbu) still connects and runs the remote command.This pull request and its description were written by Isaac.