bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations by denik · Pull Request #6954 · databricks/cli

denik · 2026-10-06T22:39:05Z

Changes

Bundle configuration no longer goes through libs/dyn; the package is deleted.

config.Root is the typed struct plus a structvar side: pure ${...} references in non-string fields and a tree of source locations (one node per YAML value). YAML is decoded straight into the structs, with the same warnings as before.
Mutators read through structvar.View and write with Set/Merge/Override/… instead of converting the whole configuration to a dyn.Value and back. Paths are structpath everywhere, including diag.Diagnostic.Paths.
Variable resolution, ${...} parsing and the resource-reference resolver run on typed values (structvar.Resolve, same semantics and error text as dynvar).
Generate commands write YAML with the new libs/structs/structyaml (identical output); --json and variable-overrides.json use structvar.ParseJSON (exact int64, same error positions).
Location lookups on config.Root take a path node and say which value they return: NearestLocations/NearestLocation fall back to the closest ancestor with a location (for diagnostics, so values set by mutators still point somewhere); ExactLocations/ExactLocation return where the value is written. Paths and patterns are built from parts (NewPath/NewPattern) instead of formatted and parsed.

Adds structpath.NewPattern(prev, parts...) with structpath.AnyKey / structpath.AnyIndex wildcards, the pattern counterpart of NewPath.

Why

Converting the configuration typed↔dyn around mutators dominated command time on large bundles.

./task bench-1k-summary (1000 jobs, wall seconds, DMS= variant), main → this PR:

benchmark	wall	user CPU	peak RSS
validate	12.6s → 1.29s (9.8x)	21.1s → 2.0s	655MB → 183MB
plan	19.2s → 1.77s (10.8x)	25.5s → 2.9s	670MB → 179MB
deploy (1000 creates)	16.4s → 2.33s (7.0x)	33.4s → 8.3s	676MB → 266MB

(deploy/DMS=true at 1000 jobs fails on main too: the test server caps a deployment version at 800 operations.)

Tests

Existing unit and acceptance tests; one golden changed (path quoting above). Parity tests against libs/dyn (decode, view, writes, merge/override, walk) passed on every bundle fixture before dyn was deleted.

This pull request and its description were written by Isaac.

eng-dev-ecosystem-bot · 2026-10-07T00:11:25Z

Integration test report

Commit: 586b4f7

Run: 38878078039

	Env	✅pass	🙈skip	Time
✅	aws linux-2core-8gb	276	15	5:30
✅	aws-windows-latest-4core-16gb	278	13	3:33
✅	azure linux-2core-8gb	275	15	5:21
✅	azure-windows-latest-4core-16gb	277	13	3:35
✅	gcp linux-2core-8gb	276	15	5:18
✅	gcp-windows-latest-4core-16gb	278	13	3:40

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
3:46	aws linux-2core-8gb	TestAccept
3:44	gcp linux-2core-8gb	TestAccept
3:41	azure linux-2core-8gb	TestAccept
3:34	gcp-windows-latest-4core-16gb	TestAccept
3:29	azure-windows-latest-4core-16gb	TestAccept
3:27	aws-windows-latest-4core-16gb	TestAccept

NewPattern builds a pattern from parts like NewPath does for paths, so callers don't nest the single-step constructors or parse pattern strings. Co-authored-by: Isaac <no-reply@databricks.com>

…r refs and locations Bundle configuration is the typed struct plus structvar (pure ${...} references in non-string fields and a tree of source locations). YAML and JSON decode straight into the structs; mutators, validators, variable resolution, configsync and generate work on structvar views and structpath paths. libs/dyn is deleted. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

… test Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

- configsync: compare int and int64 values as equal when restoring variable references (variables decode as int, remote fields are int64). - schema annotations: merge object/sequence defaults as documented; accept an empty annotations file. - python mutator: use ResourceKeySet.AddPattern instead of a duplicate helper. - structvar.Set: don't rewrite an unchanged leaf location node (avoids cloning every ancestor per write). - ResolveConfigAgainstState: resolve all references in one call so lookups are shared. Co-authored-by: Isaac <no-reply@databricks.com>

…ctpath Co-authored-by: Isaac <no-reply@databricks.com>

- apply_default_task_source: an explicit source: "" is user-set; detect it by its definition location (not every task type has ForceSendFields). - pipeline_development_deprecated: warn on any user-written development value, including a reference resolving to an unset field. - rewriteShorthands: a later merged map takes precedence over an earlier one, and merge keys are expanded before detecting a full variable definition. - required: look dashboards up by a path that round-trips keys with path syntax. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

…om parts Earlier rebases resolved conflicts with #6980 in favor of string paths, reverting call sites that main already builds with structpath.NewPath. Restore main's versions where this PR has no reason to change them, and make the app permission checks take a path node instead of parsing a string. Co-authored-by: Isaac <no-reply@databricks.com>

… parts Root's location lookups take a *structpath.PathNode and say which value they return: NearestLocations/NearestLocation fall back to the closest ancestor with a location (for diagnostics), ExactLocations/ExactLocation return where the value is written (empty if a mutator set it). The string variants are gone, so callers build paths from parts instead of formatting and parsing them. Literal MustParsePattern/MustParsePath calls in production code are replaced with NewPattern/NewPath, and the app permission checks read the resource view directly. Co-authored-by: Isaac <no-reply@databricks.com>

IsForceSend boxed the ForceSendFields slice on every call and FieldValues built a slice per struct; both now run on every View traversal rather than once per conversion. Iterate with reflect and yield the fields instead. BenchmarkWalkReadOnly (10k jobs): 7.16M -> 4.77M allocs. Co-authored-by: Isaac <no-reply@databricks.com>

… fields Walking a configuration examines every field of every struct, so the per-field cost dominates: - eachField/eachMapItem/eachElem take a callback instead of returning an iterator, so walking no longer allocates closures per node (MapItems and Sequence wrap them). - StructInfo keeps a field list with the index, Go name, omitempty, the governing ForceSendFields and whether the key is a dotted path component, so a visit does no lookups by name; GetStructInfo is lock-free. - ForceSendFields are read once per struct visit, not once per zero field. - An unset field that cannot be present (not a struct or interface, no location, no reference) is skipped without building its view. BenchmarkWalkReadOnly (10k jobs): 316ms -> 183ms, 4.77M -> 1.58M allocs. Co-authored-by: Isaac <no-reply@databricks.com>

The path visitors used by TranslatePaths and NormalizePaths iterate the typed resources with plain field access and pass the value as a *string, so the consumers rewrite it in place instead of looking the path up again with Set. Rules are applied one at a time over sorted resource keys, as the patterns were, so the visit order is unchanged. Paths are only built for the values that are visited. 1000 jobs (main / #6954 / this): TranslatePaths 384 / 28 / 8 ms, NormalizePaths 395 / 50 / 26 ms. Co-authored-by: Isaac <no-reply@databricks.com>

walk.Strings visits the strings of the configuration with plain field access (generated, one function per struct type that contains strings), filtered inline: WithReferences passes the strings that contain "${". The callback can return a replacement, which is stored in the field. RewriteWorkspacePrefix, TFOnlyReferences and reverse interpolation use it instead of walking every value of the configuration; pure references in non-string fields come from Root.References. 1000 jobs (main / #6954 / this): RewriteWorkspacePrefix 114 / 19 / 1.8 ms, TFOnlyReferences 91 / 19 / 1.9 ms. Co-authored-by: Isaac <no-reply@databricks.com>

github-actions Bot added DABs DABs related issues PyDABs labels Oct 6, 2026

denik force-pushed the denik/bundle-struct-no-dyn branch from 13fac18 to b2a1c10 Compare October 7, 2026 09:41

github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 7, 2026

denik changed the title ~~bundle: make the typed configuration the source of truth (no per-mutator dyn round trip)~~ bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations Oct 7, 2026

denik mentioned this pull request Oct 7, 2026

diag: own Location (move from libs/dyn) #6974

Merged

denik force-pushed the denik/bundle-struct-no-dyn branch from 6ee2caa to 68af62f Compare October 7, 2026 12:40

denik changed the base branch from main to denik/diag-location-structpath October 7, 2026 12:40

denik force-pushed the denik/bundle-struct-no-dyn branch from baccaac to 7086a71 Compare October 7, 2026 14:08

denik mentioned this pull request Oct 7, 2026

diag: use structpath for diagnostic paths #6980

Merged

denik force-pushed the denik/bundle-struct-no-dyn branch from 7086a71 to 4c4f0b3 Compare October 7, 2026 14:33

denik force-pushed the denik/bundle-struct-no-dyn branch from 4c4f0b3 to 3bb81b3 Compare October 7, 2026 14:38

denik force-pushed the denik/bundle-struct-no-dyn branch 2 times, most recently from 722c97c to 649bde3 Compare October 7, 2026 16:17

denik force-pushed the denik/bundle-struct-no-dyn branch from 649bde3 to 67d1321 Compare October 7, 2026 21:11

denik mentioned this pull request Oct 7, 2026

lint: forbid parsing structpath paths that can be built from parts #6984

Draft

denik force-pushed the denik/bundle-struct-no-dyn branch from 67d1321 to d49f829 Compare October 8, 2026 07:52

denik force-pushed the denik/bundle-struct-no-dyn branch from d49f829 to b22dd28 Compare October 8, 2026 11:18

denik force-pushed the denik/bundle-struct-no-dyn branch from b22dd28 to 799731f Compare October 8, 2026 11:34

denik force-pushed the denik/bundle-struct-no-dyn branch 2 times, most recently from 2ae530d to 819660f Compare October 15, 2026 14:04

structpath: add NewPattern with AnyKey/AnyIndex wildcards

6674d4b

NewPattern builds a pattern from parts like NewPath does for paths, so callers don't nest the single-step constructors or parse pattern strings. Co-authored-by: Isaac <no-reply@databricks.com>

denik mentioned this pull request Oct 15, 2026

structpath: add NewPattern with AnyKey/AnyIndex wildcards #7074

Closed

denik and others added 19 commits October 15, 2026 17:35

structvar: map YAML locations in the YAML source; drop mappedSource

21fa868

Co-authored-by: Isaac <no-reply@databricks.com>

Use NewStringKeysSlice for single key paths

dc8c9b1

Co-authored-by: Isaac <no-reply@databricks.com>

Build literal diagnostic paths with NewStringKeysSlice

a703bfc

Co-authored-by: Isaac <no-reply@databricks.com>

Restack fixes: structvar grammar for structpath's PureReferenceToPath…

6b66d26

… test Co-authored-by: Isaac <no-reply@databricks.com>

Restore import block after restack

9f08470

Co-authored-by: Isaac <no-reply@databricks.com>

Use structpath.NewPath for key/index paths

18b64b8

Co-authored-by: Isaac <no-reply@databricks.com>

Build the target preset path in one NewPath call

0671d48

Co-authored-by: Isaac <no-reply@databricks.com>

GetLocationsOf on the location tree

abb002f

Co-authored-by: Isaac <no-reply@databricks.com>

Minimal import edits; drop unused diag import

fbe25ac

Co-authored-by: Isaac <no-reply@databricks.com>

structvar: test the YAML decoder with the yamlloader fixtures

6476693

Co-authored-by: Isaac <no-reply@databricks.com>

structvar: add the YAML test helpers

b189c77

Co-authored-by: Isaac <no-reply@databricks.com>

Port the job environment_variables files pattern (#6985) to structpath

1ed2dd9

Co-authored-by: Isaac <no-reply@databricks.com>

Port the job environments project_environment pattern (#6983) to stru…

3ca9616

…ctpath Co-authored-by: Isaac <no-reply@databricks.com>

Drop the embedded JobSettings literal flagged by embedlit

3ecf7d6

Co-authored-by: Isaac <no-reply@databricks.com>

denik force-pushed the denik/bundle-struct-no-dyn branch from b7e9bbb to 1bcb66d Compare October 15, 2026 15:46

denik mentioned this pull request Oct 16, 2026

acc: record configuration edge cases ahead of the libs/dyn removal #7085

Draft

denik mentioned this pull request Oct 17, 2026

bundle: delete libs/dyn; typed configuration with generated walkers #7098

Draft

Navigation Menu

Repository navigation

bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations - #6954

bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations#6954
denik wants to merge 22 commits into
mainfrom
denik/bundle-struct-no-dyn

denik commented Oct 6, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

denik commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

denik commented Oct 6, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Oct 7, 2026 •

edited

Loading