Repository navigation
Retry workspace get-status on transient failures during sync - #7044
Merged
Merged
Conversation
Collaborator
Integration test reportCommit: 6684fff
Top 3 slowest tests (at least 2 minutes):
|
EnsureRemotePathIsUsable validates the sync target with workspace/get-status. A transient server error or a dropped connection (e.g. REFUSED_STREAM, #6472) on that read aborted the whole deploy. Route the calls through libs/retry with a new IsTransientRead predicate: on top of the 5xx/408 that IsTransient covers, an idempotent read is also safe to replay after a transport-level failure (*url.Error). Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
force-pushed
the
janniklasrose/retry-workspace-get-status
branch
from
October 14, 2026 10:59
cb37d03 to
d339309
Compare
janniklasrose
marked this pull request as ready for review
October 14, 2026 11:05
janniklasrose
commented
Oct 14, 2026
andrewnester
approved these changes
Oct 14, 2026
| @@ -0,0 +1,4 @@ | |||
| Warn: retrying after 504 Gateway Timeout from GET /api/2.0/workspace/get-status | |||
Contributor
There was a problem hiding this comment.
Do we want to log it at Warn level? Maybe just a debug one?
Member
Author
There was a problem hiding this comment.
following #7014 here but we can do that as follow-up for both. agree that customers don't need to see these internal details
Contributor
There was a problem hiding this comment.
Given that it's rare I personally would not mind seeing some progress instead of just CLI appears being blocked.
Matching every *url.Error restarted the SDK's RetryTimeout loop (5m default) for the transport errors it already retries (connection reset/refused, i/o timeout, TLS), turning a persistent failure's ~5m into ~15m. Retry only the HTTP/2 REFUSED_STREAM the SDK leaves unhandled (#6472). Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
enabled auto-merge
October 14, 2026 12:50
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Oct 14, 2026
## Release v1.21.0 ### CLI * Support pipeline IDs in `pipelines history`. ([#6995](#6995)) * Show a hint for previewing more rows in `pipelines show`. ([#6995](#6995)) * `databricks environments setup-local` now uses the environment in `UV_PROJECT_ENVIRONMENT` when it is set, as `uv sync` does. ([#7026](#7026)) * `databricks environments setup-local` now supports uv workspace members: it uses the shared `.venv` at the workspace root and writes the dependency constraints to the root `pyproject.toml`. ([#7018](#7018)) ### AI Runtime * Limit `databricks air run` submission preflight validation to one five-second attempt before continuing when the service is unavailable. ([#6994](#6994)) ### Bundles * Bundle file uploads now use the multipart `POST /api/2.0/workspace/import` endpoint instead of the deprecated `POST /api/2.0/workspace-files/import-file`, raising the per-workspace upload rate limit and improving deploy throughput under concurrency. ([#6149](#6149)) * Diagnostic paths now quote map keys that are not identifiers, e.g. `resources.jobs['${var.env}_job']`. ([#6980](#6980)) * No longer include `CLAUDE.md` in the bundle templates, now that Claude supports `AGENTS.md`. ([#7000](#7000)) * Fix `bundle deployment bind` silently binding over a deployment state it could not read, which could take over a resource that was already managed by the bundle. ([#6331](#6331)) * `databricks bundle deploy`, `databricks bundle sync` and `databricks sync` now retry file, library, state and metadata uploads that fail with a transient server error (5xx or 408), instead of failing. ([#7014](#7014)) * `databricks bundle deploy`, `databricks bundle sync` and `databricks sync` now retry the workspace path check (`workspace/get-status`) when it fails with a transient server error (5xx or 408) or an HTTP/2 `REFUSED_STREAM`, instead of failing. ([#7044](#7044)) * Fixed `experimental.immutable_folder` failing validation for targets with `mode: development` ("file_path must start with '~/' or contain the current username"). ([#7027](#7027)) * Fixed `experimental.immutable_folder` failing with `experimental.deployment_history` ("workspace_info.file_path must be an absolute workspace path") and failing `bundle validate` when top-level `permissions` are set. ([#7001](#7001)) * Translate relative `.env` file paths in a job's `environment_variables[].spec.files` to their deployed workspace locations. ([#6985](#6985)) * Resolve job `project_environment` paths relative to the YAML file that defines them. ([#6983](#6983)) ### Dependency Updates * Bump Go toolchain to 1.26.9. ([#6996](#6996)) * Update Go SDK to 0.188.0. ([#7038](#7038))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
EnsureRemotePathIsUsablevalidates the sync target with aworkspace/get-statusread. Its calls now go throughlibs/retry(added in #7014) with a newretry.IsTransientReadpredicate: on top of the 5xx/408 errorsIsTransientalready covers, an idempotent read is also safe to replay after a transport-level failure that produced no HTTP response (a*url.Error, e.g. the HTTP/2REFUSED_STREAMin #6472).IsTransient(used for the write path) is left unchanged.Why
#6472: a sporadic HTTP/2
REFUSED_STREAMon theget-statuscall duringbundle deployaborted the deploy and left the bundle out of sync with its artifacts. #7014 added upload retries; this extends the same approach to the get-status read that was the actual point of failure.Tests
IsTransientRead(5xx/408, aREFUSED_STREAM*url.Error, and the non-retriable cases).cmd/sync/get-status-504) injecting a 504 onGET /api/2.0/workspace/get-statusduringdatabricks sync; the read is retried and the sync succeeds. The transport-error branch is unit-tested rather than acceptance-tested: the HTTP testserver can't produce a real stream error, and Go's transport auto-retries idempotent GETs on reused connections, which makes a forced-disconnect fault non-deterministic to assert.Builds on #7014 (now merged).
This pull request and its description were written by Isaac.