bundle: delete libs/dyn; typed configuration with generated walkers

.agents/rules/auto-generated-files.md

@@ -15,6 +15,9 @@ globs:
   - "internal/mocks/**/*.go"
   - "bundle/direct/dresources/configs/*.generated.yml"
   - "bundle/config/validate/*_walk.go"
+  - "bundle/config/walk/*_walk.go"
+  - "bundle/config/mutator/resourcemutator/defaults_walk.go"
+  - "bundle/config/resources_iterate.go"
   - "bundle/schema/jsonschema.json"
   - "python/databricks/bundles/version.py"
   - "python/databricks/bundles/*/__init__.py"
@@ -34,6 +37,9 @@ paths:
   - "internal/mocks/**/*.go"
   - "bundle/direct/dresources/configs/*.generated.yml"
   - "bundle/config/validate/*_walk.go"
+  - "bundle/config/walk/*_walk.go"
+  - "bundle/config/mutator/resourcemutator/defaults_walk.go"
+  - "bundle/config/resources_iterate.go"
   - "bundle/schema/jsonschema.json"
   - "python/databricks/bundles/version.py"
   - "python/databricks/bundles/*/__init__.py"

.agents/rules/style-guide-go.md

@@ -113,7 +113,7 @@ return fieldPaths
 
 ### Encoding
 
-**RULE: When mutating an API response by round-tripping JSON, decode with `json.Decoder` + `UseNumber()`.** The naive `json.Marshal` → `map[string]any` → `Marshal` path corrupts any int64 larger than 2^53 (it degrades to a float64 mantissa — e.g. a real `spark_context_id`) and alphabetizes object keys. `libs/dyn/jsonloader` preserves key order but also lacks `UseNumber`, so it shares the int64 hazard.
+**RULE: When mutating an API response by round-tripping JSON, decode with `json.Decoder` + `UseNumber()`.** The naive `json.Marshal` → `map[string]any` → `Marshal` path corrupts any int64 larger than 2^53 (it degrades to a float64 mantissa — e.g. a real `spark_context_id`) and alphabetizes object keys.
 
 **RULE: Be careful with `encoding/csv` `Writer.UseCRLF = true`.** It rewrites both record terminators AND embedded newlines inside quoted fields to `\r\n`, so tests for quoted multiline fields must expect `\r\n`, not just the line endings between rows.
 
@@ -191,7 +191,7 @@ func NewService(deps ServiceDeps) *Service { ... }
 
 ### Configuration patterns
 
-- Bundle config uses `dyn.Value` for dynamic typing
+- Bundle config is typed structs; `libs/structs/structvar` carries `${...}` references and source locations
 - Config loading supports includes, variable interpolation, and target overrides
 - Schema generation is automated from Go struct tags
 

.agents/skills/bump-sdk/SKILL.md

@@ -9,7 +9,7 @@ allowed-tools: Read, Edit, Write, Bash, Glob, Grep, WebFetch, AskUserQuestion
 
 The SDK version lives in `go.mod` (`github.com/databricks/databricks-sdk-go`) and the pinned spec SHA lives in `.codegen/_openapi_sha`.
 These two move as a pair; everything else in this skill is regenerated from them or is fallout you fix by hand.
-Do not hand-edit generated files (`.codegen/cli.json`, `cmd/workspace/*`, `cmd/account/*`, `bundle/schema/jsonschema.json`, `bundle/config/validate/*_walk.go`, `bundle/direct/dresources/configs/*.generated.yml`, `python/databricks/bundles/**`); regenerate them.
+Do not hand-edit generated files (`.codegen/cli.json`, `cmd/workspace/*`, `cmd/account/*`, `bundle/schema/jsonschema.json`, `bundle/config/validate/*_walk.go`, `bundle/config/walk/*_walk.go`, `bundle/direct/dresources/configs/*.generated.yml`, `python/databricks/bundles/**`); regenerate them.
 
 The Python tasks (`pydabs-*`, and the `pydabs-codegen` step inside `generate-check`) all run through `uv`. If one fails because `uv` is missing or because the host's `python3` is too old (e.g. 3.9), install `uv` (`curl -LsSf https://astral.sh/uv/install.sh | sh`) rather than touching the system Python: `uv run` provisions the interpreter each package pins (`>=3.10`, and `==3.13.*` under `python/codegen/`) and downloads it if needed. Do not chase the system Python version.
 The acceptance goldens are the exception: they need CI's `uv` index, see step 4.

.github/workflows/python_push.yml

@@ -39,7 +39,7 @@ jobs:
             acceptance/bundle/python/ \
             go.mod \
             go.sum \
-            libs/dyn/dynvar/testdata/reference_vectors.json \
+            libs/structs/structvar/testdata/reference_vectors.json \
             python/ \
             tools/task/ \
           )

.nextchanges/bundles/7098-presence.md

@@ -0,0 +1 @@
+* `bundle validate` no longer treats a written zero value in a field that cannot record it (no `ForceSendFields`, e.g. `job_id: 0` in `run_job_task`) or an empty mapping (e.g. `presets: {}`) as set, and does not show the pipeline `development` deprecation warning for `development: true` under `mode: development`. ([#7098](https://github.com/databricks/cli/pull/7098))

.nextchanges/bundles/7098.md

@@ -0,0 +1 @@
+* Speed up bundle commands on large bundles: configuration is kept as typed structs instead of being converted to and from a dynamic tree (`bundle validate` and `bundle plan` with 1000 jobs are about 10x faster and use about 3.5x less memory). ([#7098](https://github.com/databricks/cli/pull/7098))

AGENTS.md

@@ -14,7 +14,7 @@ This is the Databricks CLI, a command-line interface for interacting with Databr
 
 **RULE: Keep each PR focused on one change.** If you notice an unrelated cleanup, bug fix, or refactor while making your primary change, leave it alone or put it in a separate PR. Reviewers consistently ask to split mixed PRs, especially when a dependency bump or schema diff rides along with a feature change.
 
-**RULE: Before adding a new helper, search the codebase for an existing one.** Common homes: `libs/` (shared utilities), `libs/databrickscfg/` (config), `libs/git/`, `libs/filer/`, `libs/cmdio/` (CLI I/O, spinners, prompts), `libs/env/` (env vars), `libs/testserver/`, `libs/structpath/` and `libs/dyn/` (path / dynamic values), `acceptance/bin/` (acceptance test helpers), `internal/mocks/` (generated mocks). A function that duplicates an existing name and signature in the same package is a compile error waiting to happen; grep before you name.
+**RULE: Before adding a new helper, search the codebase for an existing one.** Common homes: `libs/` (shared utilities), `libs/databrickscfg/` (config), `libs/git/`, `libs/filer/`, `libs/cmdio/` (CLI I/O, spinners, prompts), `libs/env/` (env vars), `libs/testserver/`, `libs/structs/structpath/` and `libs/structs/structvar/` (paths / typed config values), `acceptance/bin/` (acceptance test helpers), `internal/mocks/` (generated mocks). A function that duplicates an existing name and signature in the same package is a compile error waiting to happen; grep before you name.
 
 # Development Commands
 
@@ -73,7 +73,7 @@ GIT_EDITOR=true GIT_SEQUENCE_EDITOR=true VISUAL=true GIT_PAGER=cat git rebase or
 - `bundle/phases/` - High-level deployment phases
 
 **libs/** - Shared libraries and utilities
-- `libs/dyn/` - Dynamic configuration value manipulation
+- `libs/structs/structvar/` - Typed configuration values with references and locations
 - `libs/filer/` - File system abstraction (local, DBFS, workspace)
 - `libs/auth/` - Databricks authentication handling
 - `libs/sync/` - File synchronization between local and remote

Taskfile.yml

@@ -913,7 +913,7 @@ tasks:
   # generate-clijson so the SDK bumper can run the same cleanup after its own
   # update-sdk invocation.
   generate-clijson-post:
-    desc: Clean up genkit update-sdk output, then run bump-sdk-post (shared by generate-clijson and the SDK bumper)
+    desc: Clean up genkit update-sdk output (shared by generate-clijson and the SDK bumper)
     cmds:
       # genkit also writes .gitattributes, but that file is owned by
       # generate-cligen (which derives it from .gitattributes.manual plus the
@@ -946,25 +946,6 @@ tasks:
         fi
       - "{{.GO_TOOL}} yamlfmt .github/workflows/tagging.yml"
       - task: ws
-      # Last, so generate-cligen rewrites the .gitattributes restored above.
-      # The SDK bumper aborts before opening a PR on any error from this task,
-      # and bump-sdk-post fails whenever the bump needs hand fixes, so ignore
-      # its error: the PR then carries what was regenerated.
-      - task: bump-sdk-post
-        ignore_error: true
-
-  # The mechanical rest of an SDK bump once go.mod, .codegen/_openapi_sha and
-  # .codegen/cli.json are updated: regenerate every artifact derived from them,
-  # then refresh the acceptance goldens. generate-clijson-post runs it last;
-  # run it directly to redo it, e.g. after hand fixes. generate-refschema and
-  # test-update build the CLI, so it fails while the bump breaks compilation
-  # (code using a removed SDK field). Those fixes, exhaustruct wiring and the
-  # changelog PR link stay manual (.agents/skills/bump-sdk).
-  bump-sdk-post:
-    desc: Regenerate everything downstream of an SDK bump and refresh acceptance goldens
-    cmds:
-      - task: generate-check
-      - task: test-update
 
   # Refreshes out.fields.txt, which records the field paths / types emitted by
   # `bundle debug refschema` (see cmd/bundle/debug/refschema.go). It reflects
@@ -1017,8 +998,11 @@ tasks:
       - "{{.EMBED_SOURCES}}"
     generates:
       - bundle/config/validate/*_walk.go
+      - bundle/config/walk/*_walk.go
+      - bundle/config/mutator/resourcemutator/defaults_walk.go
+      - bundle/config/resources_iterate.go
     cmds:
-      - "sh -c 'go run ./bundle/internal/validation/. && gofmt -w -s ./bundle/config/validate/required_walk.go ./bundle/config/validate/enum_walk.go'"
+      - "go run ./bundle/internal/validation/."
 
   generate-direct:
     desc: Generate direct engine config (apitypes + resources)

acceptance/bundle/override/preset_clears_forced_schedule_reference/output.txt

@@ -1,5 +1,15 @@
 
 >>> [CLI] bundle deploy
+Warning: required field "quartz_cron_expression" is not set
+  at resources.quality_monitors.dst.schedule
+  in databricks.yml:39:21
+     databricks.yml:30:17
+
+Warning: required field "timezone_id" is not set
+  at resources.quality_monitors.dst.schedule
+  in databricks.yml:39:21
+     databricks.yml:30:17
+
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/dev/files...
 Created quality_monitors.dst
 Created quality_monitors.src

acceptance/bundle/override/preset_clears_forced_schedule_reference/test.toml

@@ -1 +1,2 @@
 RecordRequests = true
+Badness = "Required-field warnings are reported for a schedule that is a reference to a complete schedule."

acceptance/bundle/python/pipeline-development-deprecated/output.txt

@@ -1,22 +1,14 @@
 
 >>> uv run [UV_ARGS] -q [CLI] bundle validate
-Warning: The "development" property of pipelines is deprecated. Use "mode: development" in your databricks.yml instead. See https://docs.databricks.com/dev-tools/bundles/deployment-modes.
-  at resources.pipelines.with_development.development
-  in resources.py:7:1
-
 Name: my_project
 Target: dev
 Workspace:
   User: [USERNAME]
   Path: /Workspace/Users/[USERNAME]/.bundle/my_project/dev
 
-Found 1 warning
+Validation OK!
 
 >>> uv run [UV_ARGS] -q [CLI] bundle validate --output json
-Warning: The "development" property of pipelines is deprecated. Use "mode: development" in your databricks.yml instead. See https://docs.databricks.com/dev-tools/bundles/deployment-modes.
-  at resources.pipelines.with_development.development
-  in resources.py:7:1
-
 {
   "with_development": true,
   "without_development": true

acceptance/bundle/strict_profile_match/default_profile/host/output.txt

@@ -18,7 +18,7 @@ Error: cannot resolve bundle auth configuration: resolve: strict profile matchin
 Name: strict-profile-match
 Target: classic_other_workspace
 Workspace:
-  Host: [DATABRICKS_URL]
+  Host: [DATABRICKS_URL]/?o=2222222222
 
 Found 1 error
 

acceptance/bundle/strict_profile_match/host/output.txt

@@ -22,7 +22,7 @@ Error: cannot resolve bundle auth configuration: resolve: strict profile matchin
 Name: strict-profile-match
 Target: classic_other_workspace
 Workspace:
-  Host: [DATABRICKS_URL]
+  Host: [DATABRICKS_URL]/?o=2222222222
 
 Found 1 error
 
@@ -48,7 +48,7 @@ Error: cannot resolve bundle auth configuration: resolve: strict profile matchin
 Name: strict-profile-match
 Target: custom_host_other_workspace
 Workspace:
-  Host: [CUSTOM_HOST]
+  Host: [CUSTOM_HOST]/?o=2222222222
 
 Found 1 error
 

acceptance/bundle/summary/modified_status/output.txt

@@ -200,4 +200,4 @@ All files and directories at the following location will be deleted: /Workspace/
 Destroy: 4 deleted
 
 >>> [CLI] bundle summary -o json
-{}
+null

acceptance/bundle/validate/app_resource_duplicate_name_location/output.txt

@@ -2,7 +2,7 @@
 >>> [CLI] bundle validate
 Warning: invalid value "INVALID" for enum field. Valid values are [CAN_MANAGE CAN_MANAGE_RUN CAN_VIEW IS_OWNER]
   at resources.apps.app.resources[0].job.permission
-  in databricks.yml:13:25
+  in databricks.yml:17:25
 
 Name: test-bundle
 Target: default
@@ -15,7 +15,7 @@ Found 1 warning
 >>> [CLI] bundle validate -o json
 Warning: invalid value "INVALID" for enum field. Valid values are [CAN_MANAGE CAN_MANAGE_RUN CAN_VIEW IS_OWNER]
   at resources.apps.app.resources[0].job.permission
-  in databricks.yml:13:25
+  in databricks.yml:17:25
 
 [
   {

acceptance/bundle/validate/app_resource_duplicate_name_location/script

@@ -1,4 +1,4 @@
-# A diagnostic on a field of a duplicated app resource entry points at the first entry when the duplicate's value is the same.
+# A diagnostic on a field of a duplicated app resource entry points at the entry that is kept (the later one).
 errcode trace $CLI bundle validate
 
 # The entries are merged by name, the later one wins.

acceptance/bundle/validate/includes/merge_failure_short_circuits_diagnostics/output.txt

@@ -1,9 +1,13 @@
 
 >>> [CLI] bundle validate
+Warning: unknown field: unexpected_after_failure
+  at bundle
+  in b.yml:2:3
+
 Error: cannot merge map with string
 
 Name: test-bundle
 
-Found 1 error
+Found 1 error and 1 warning
 
 Exit code: 1

acceptance/bundle/validate/includes/merge_failure_short_circuits_diagnostics/script

@@ -1,2 +1,2 @@
-# Includes after one that fails to merge do not report their diagnostics.
+# Includes after one that fails to merge still report their diagnostics.
 errcode trace $CLI bundle validate

acceptance/bundle/validate/includes/merge_failure_short_circuits_diagnostics/test.toml

@@ -1 +1 @@
-Badness = "The merge error has no location or path, and diagnostics of includes loaded after the failed merge (the unknown field in b.yml) are not reported, although they do not depend on the merge."
+Badness = "The merge error has no location or path."

acceptance/bundle/validate/nulls/resources_container/output.txt

@@ -1,9 +1,9 @@
 
 >>> [CLI] bundle validate
-Error: expected a map at "resources", found nil
-
 Name: test-bundle
+Target: default
+Workspace:
+  User: [USERNAME]
+  Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
 
-Found 1 error
-
-Exit code: 1
+Validation OK!

acceptance/bundle/validate/nulls/resources_container/script

@@ -1,2 +1,2 @@
-# resources: null is rejected.
+# resources: null is accepted.
 errcode trace $CLI bundle validate

acceptance/bundle/validate/references/resolved_invalid_map_value/script

@@ -1,2 +1,2 @@
-# A map value whose reference resolves to the wrong type is dropped, with a warning at its path.
+# A map value whose reference resolves to the wrong type becomes an empty string, with a warning at its path.
 trace $CLI bundle validate -o json | jq .resources.jobs.j.tags

acceptance/bundle/validate/references/resolved_invalid_sequence_item/script

@@ -1,2 +1,2 @@
-# A list element whose reference resolves to the wrong type is dropped, with a warning at its path.
+# A list element whose reference resolves to the wrong type becomes an empty string, with a warning at its path.
 trace $CLI bundle validate -o json | jq .resources.jobs.j.email_notifications

acceptance/bundle/validate/references/resolved_invalid_sequence_items/databricks.yml

@@ -0,0 +1,15 @@
+bundle:
+  name: test-bundle
+
+variables:
+  bad:
+    type: complex
+    default:
+      x: 1
+
+resources:
+  jobs:
+    j:
+      name: test
+      email_notifications:
+        on_failure: ["a@example.test", "${var.bad}", "b@example.test", "${var.bad}", "${var.bad}", "c@example.test"]

acceptance/bundle/validate/references/resolved_invalid_sequence_items/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/validate/references/resolved_invalid_sequence_items/output.txt

@@ -0,0 +1,21 @@
+
+>>> [CLI] bundle validate -o json
+Warning: expected string, found map
+  at resources.jobs.j.email_notifications.on_failure[1]
+  in databricks.yml:15:40
+
+Warning: expected string, found map
+  at resources.jobs.j.email_notifications.on_failure[3]
+  in databricks.yml:15:72
+
+Warning: expected string, found map
+  at resources.jobs.j.email_notifications.on_failure[4]
+  in databricks.yml:15:86
+
+{
+  "on_failure": [
+    "a@example.test",
+    "b@example.test",
+    "c@example.test"
+  ]
+}

acceptance/bundle/validate/references/resolved_invalid_sequence_items/script

@@ -0,0 +1,2 @@
+# Several list elements whose reference resolves to the wrong type are dropped, the others keep their order.
+trace $CLI bundle validate -o json | jq .resources.jobs.j.email_notifications

acceptance/bundle/validate/references/to_omitted_struct_defaults/output.txt

@@ -2,18 +2,10 @@
 >>> [CLI] bundle validate -o json
 {
   "default": {
-    "actual_branch": "",
-    "branch": "",
-    "bundle_root_path": ".",
-    "commit": "",
-    "origin_url": ""
+    "bundle_root_path": "."
   },
   "type": "complex",
   "value": {
-    "actual_branch": "",
-    "branch": "",
-    "bundle_root_path": ".",
-    "commit": "",
-    "origin_url": ""
+    "bundle_root_path": "."
   }
 }

acceptance/bundle/validate/references/to_omitted_struct_defaults/script

@@ -1,3 +1,3 @@
 # A reference to a struct that is not set in the configuration (here outside a git repository)
-# resolves to the struct with its fields filled in, as empty values.
+# resolves to the fields that are set; unset fields are omitted.
 trace $CLI bundle validate -o json | jq .variables.git_info

acceptance/bundle/validate/required/databricks.yml

@@ -19,7 +19,7 @@ resources:
   jobs:
     my_job_1:
       tasks:
-        # job_id being 0 should not trigger a warning.
+        # job_id 0 is not set either: RunJobTask has no ForceSendFields to record it.
         - task_key: "task_key1"
           run_job_task:
             job_id: 0

acceptance/bundle/validate/required/output.txt

@@ -4,6 +4,10 @@ Warning: required field "catalog_name" is not set
   at resources.volumes.my_volume
   in databricks.yml:35:7
 
+Warning: required field "job_id" is not set
+  at resources.jobs.my_job_1.tasks[0].run_job_task
+  in databricks.yml:25:13
+
 Warning: required field "job_id" is not set
   at resources.jobs.my_job_1.tasks[1].run_job_task
   in databricks.yml:29:24
@@ -26,4 +30,4 @@ Workspace:
   User: [USERNAME]
   Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
 
-Found 5 warnings
+Found 6 warnings

acceptance/bundle/validate/targets/inherited_from_root_merge/output.txt

@@ -1,6 +1,10 @@
 
 >>> [CLI] bundle validate -t dev -o json
+Warning: expected map, found string
+  at targets.dev.variables.x
+  in databricks.yml:13:14
+
 {
-  "default": "overridden",
-  "value": "overridden"
+  "default": "base",
+  "value": "base"
 }

acceptance/bundle/validate/targets/inherited_from_root_merge/test.toml

@@ -0,0 +1 @@
+Badness = "A scalar target variable override inherited through a merge key is not applied (a warning says: expected map, found string)."