Skip to content

Allow changing deployment history on existing file state - #7198

Merged
shreyas-goenka merged 1 commit into
mainfrom
shreyas-goenka_data/stack/deployment-history-setting-unenforced
Oct 27, 2026
Merged

Allow changing deployment history on existing file state#7198
shreyas-goenka merged 1 commit into
mainfrom
shreyas-goenka_data/stack/deployment-history-setting-unenforced

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Stop enforcing the setting for DMS. No longer needed because we now we do best effort history tracking

@shreyas-goenka shreyas-goenka mentioned this pull request Oct 27, 2026
Draft
@github-actions

github-actions Bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: 6548a02f7 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0

test branch main (814bdfe) latest (v1.22.1)
TestAccept/bundle/dms/existing-state/DMS= ✅ ❌ ❌
TestAccept/bundle/migrate/auto/dms-migration/DMS= ✅ ❌ ❌
TestAccept/bundle/dms/existing-state/DMS=: branch ✅ | main (814bdfe) ❌ | latest (v1.22.1) ❌

main (814bdfe):

acceptance_test.go:1341: Diff:
        --- bundle/dms/existing-state/output.txt
        +++ /tmp/TestAcceptbundledmsexisting-stateDMS=4021839367/002/output.txt
        @@ -1,26 +1,9 @@
         
         === Enable history on existing remote file state without replacing the job
         >>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/files...
        -Files: 3 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
        +Error: deployment history setting (true) does not match the existing state (false)
         
        ->>> jq {features: (.features // {}), job_id: .state["resources.jobs.one"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "job_id": "[ONE_ID]"
        -}
        +Update experimental.deployment_history to match the existing deployment, or run "databricks bundle destroy" to start over
         
        -=== Disable history without replacing the job
        ->>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/files...
        -Files: 0 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
         
        ->>> jq {features: (.features // {}), job_id: .state["resources.jobs.one"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {},
        -  "job_id": "[ONE_ID]"
        -}
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.setup: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-existing-state-4eiokfxedbej3lyzibm75lhw64/default/files...
        LOG.setup: Created jobs.one
        LOG.setup: Files: 3 uploaded, 0 deleted
        LOG.setup: Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/dms/existing-state/output.txt
        +++ /tmp/TestAcceptbundledmsexisting-stateDMS=1236352417/002/output.txt
        @@ -1,26 +1,9 @@
         
         === Enable history on existing remote file state without replacing the job
         >>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/files...
        -Files: 3 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
        +Error: deployment history setting (true) does not match the existing state (false)
         
        ->>> jq {features: (.features // {}), job_id: .state["resources.jobs.one"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "job_id": "[ONE_ID]"
        -}
        +Update experimental.deployment_history to match the existing deployment, or run "databricks bundle destroy" to start over
         
        -=== Disable history without replacing the job
        ->>> DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-existing-state-[UNIQUE_NAME]/default/files...
        -Files: 0 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
         
        ->>> jq {features: (.features // {}), job_id: .state["resources.jobs.one"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {},
        -  "job_id": "[ONE_ID]"
        -}
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.setup: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-existing-state-s6pcfbega5cpjmt4x63ycrpzlq/default/files...
        LOG.setup: Created jobs.one
        LOG.setup: Files: 3 uploaded, 0 deleted
        LOG.setup: Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
TestAccept/bundle/migrate/auto/dms-migration/DMS=: branch ✅ | main (814bdfe) ❌ | latest (v1.22.1) ❌

main (814bdfe):

acceptance_test.go:1341: Diff:
        --- bundle/migrate/auto/dms-migration/output.txt
        +++ /tmp/TestAcceptbundlemigrateautodms-migrationDMS=3532796553/002/output.txt
        @@ -5,24 +5,9 @@
         === Terraform state migrates to file-backed direct state with shadow history
         >>> DATABRICKS_BUNDLE_ENGINE=direct DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true [CLI] bundle deploy
         Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Migrated 1 resource to direct deployment engine.
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
        +Error: deployment history setting (true) does not match the existing state (false)
         
        ->>> jq {features, job_id: .state["resources.jobs.test_job"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "job_id": "[TEST_JOB_ID]"
        -}
        +Update experimental.deployment_history to match the existing deployment, or run "databricks bundle destroy" to start over
         
        ->>> find.py \.databricks/.*(\.tfstate|resources.*\.json|\.wal)
        -.databricks/bundle/default/resources.json
        -.databricks/bundle/default/terraform/terraform.tfstate.backup
         
        ->>> MSYS_NO_PATHCONV=1 [CLI] workspace get-status /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate.backup -o json
        -{
        -  "object_type": "FILE"
        -}
        +Exit code: 1
        
FAIL

latest (v1.22.1): same output as main (814bdfe)

@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-setting-unenforced branch from 80c67af to 1c97efb Compare October 27, 2026 16:52
@shreyas-goenka
shreyas-goenka marked this pull request as ready for review October 27, 2026 16:57
@shreyas-goenka
shreyas-goenka requested review from a team as code owners October 27, 2026 16:57
denik
denik approved these changes Oct 27, 2026
View reviewed changes
Comment thread acceptance/bundle/dms/existing-state/script Outdated
title "Wiping the local cache still refuses after pulling the non-DMS state from the workspace"
# Create ordinary file state, then force the next command to pull it from the workspace.
$CLI bundle deploy &> LOG.setup
job_id=$(jq -r '.state["resources.jobs.one"].__id__' .databricks/bundle/default/resources.json)

@denik denik Oct 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have read_id

Base automatically changed from shreyas-goenka_data/shreyas/history-best-effort-default to main October 27, 2026 17:08
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-setting-unenforced branch from 1c97efb to ffda837 Compare October 27, 2026 17:15
@github-actions github-actions Bot added the DABs DABs related issues label Oct 27, 2026
This was referenced Oct 27, 2026
Merged
Merged
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 6548a02

Run: 40600457740

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 283 16 5:20
✅​ aws-windows-latest-4core-16gb 285 14 3:38
✅​ azure linux-2core-8gb 282 16 5:33
✅​ azure-windows-latest-4core-16gb 284 14 3:39
✅​ gcp linux-2core-8gb 283 16 5:32
✅​ gcp-windows-latest-4core-16gb 285 14 6:19
Top 6 slowest tests (at least 2 minutes):
duration env testname
5:56 gcp-windows-latest-4core-16gb TestAccept
3:49 azure linux-2core-8gb TestAccept
3:41 gcp linux-2core-8gb TestAccept
3:36 aws linux-2core-8gb TestAccept
3:17 aws-windows-latest-4core-16gb TestAccept
3:17 azure-windows-latest-4core-16gb TestAccept

@shreyas-goenka
Remove deployment-history configuration enforcement
6548a02 
Addresses denik review comment 4387609280. Existing file state can enable or disable shadow history without being rejected.
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-setting-unenforced branch from ffda837 to 6548a02 Compare October 27, 2026 23:11
@shreyas-goenka
shreyas-goenka added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit 5e52c4c Oct 27, 2026
27 checks passed
@shreyas-goenka
shreyas-goenka deleted the shreyas-goenka_data/stack/deployment-history-setting-unenforced branch October 27, 2026 23:57
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 5e52c4c

Run: 40605625695

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 1467 1217 130:45
🔄​ aws-windows-latest-4core-16gb 3 1389 1242 137:13
✅​ azure linux-2core-8gb 1316 1269 108:13
✅​ azure-windows-latest-4core-16gb 1241 1294 108:26
🔄​ gcp linux-2core-8gb 3 1300 1274 114:03
🔄​ gcp-windows-latest-4core-16gb 2 1226 1299 118:17
6 interesting tests: 6 flaky
Test Name aws-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
🔄​ TestAccept 🔄​f 🔄​f 🔄​f
🔄​ TestAccept/bundle/local_state_staleness ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/local_state_staleness/DMS=true ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/apps/lifecycle-started-toggle 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS= 🔄​f ✅​p ✅​p
🔄​ TestAccept/cmd/workspace/import-zip-auto ✅​p ✅​p 🔄​f
Top 50 slowest tests (at least 2 minutes):
duration env testname
14:45 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
11:19 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
10:58 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
10:48 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:12 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
9:42 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:52 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
8:44 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
8:36 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
8:33 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
8:20 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
8:06 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
7:57 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:27 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
7:27 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
7:24 azure linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
7:24 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:22 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
7:21 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:17 azure linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
7:15 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:08 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:02 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:55 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:54 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:52 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:46 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:45 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:44 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:44 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:32 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:27 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:26 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:26 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:18 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:06 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:55 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:54 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:53 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:51 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:37 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:37 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:22 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
5:20 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
5:19 gcp-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
5:16 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
5:14 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
5:10 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
5:10 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
5:02 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@denik denik denik approved these changes
@renaudhartert-db renaudhartert-db Awaiting requested review from renaudhartert-db renaudhartert-db is a code owner automatically assigned from databricks/eng-deco-cli
@pietern pietern Awaiting requested review from pietern pietern is a code owner automatically assigned from databricks/eng-deco-dabs

Assignees

No one assigned

Labels

DABs DABs related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shreyas-goenka @eng-dev-ecosystem-bot @denik