PyDABs codegen: autogenerate core resource wiring by Sankalp-Mittal · Pull Request #6397 · databricks/cli

Sankalp-Mittal · 2026-08-27T13:17:06Z

Summary

Generate the databricks.bundles.core per-resource wiring instead of hand-writing it, so adding a PyDABs resource needs no manual core/ edits — only RESOURCE_NAMESPACE. Stacked on #6408.

Changes

Each wired resource is rendered from a template (codegen/codegen/wiring_resource.py.tmpl, a string.Template) into its own core/_generated/<resource_plural>.py — the add_<x>() + collection-property mixin, the <x>_mutator decorator, and a lazy _resource_type(). A generated core/_generated/__init__.py collects them into _GeneratedResources (mixed into Resources), _all_resource_types(), and the mutator re-exports. core/__init__.py is generated too.
Resources.add_resource / add_resources / __init__ are now data-driven off _ResourceType.all() (single self._resources dict), so they don't grow per resource.
Duplicate-name error switched to uniform for resource '<type>' wording, dropping the a/an special case.

Behavior-preserving apart from that wording. Generalizing the hardcoded jobs.ForEachTask recursion handling is left as a follow-up.

Tests

Unit + codegen_tests pass; ./task pydabs-codegen is idempotent (no _models/ diff); pyright and ruff clean.

This pull request and its description were written by Isaac.

eng-dev-ecosystem-bot · 2026-08-27T13:34:00Z

Integration test report

Commit: cfeb0a7

Run: 33180617335

	Env	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
💚	aws linux	1	1	274	1207	4:28
💚	aws windows	1	1	276	1205	3:40
💚	azure linux	1	1	273	1207	4:37
💚	azure windows	1	1	275	1205	3:26
💚	gcp linux	1	1	274	1207	6:03
💚	gcp windows	1	1	276	1205	3:46

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
3:35	aws windows	TestAccept
3:27	gcp windows	TestAccept
3:15	azure windows	TestAccept

github-actions · 2026-08-27T22:52:07Z

Waiting for approval

Based on git history, these people are best suited to review:

@denik -- recent work in python/codegen/codegen/, python/

Eligible reviewers: @andrewnester, @anton-107, @janniklasrose, @lennartkats-db, @pietern, @rclarey, @renaudhartert-db, @rugpanov, @shreyas-goenka, @simonfaltum

_{Suggestions based on git history. See OWNERS for ownership rules.}

The PyDABs codegen writes databricks/bundles/<ns>/_models/*.py and each namespace __init__.py, but those files carried no generated-file marker. - Prepend a "# Code generated by pydabs-codegen. DO NOT EDIT." header in both writer paths (_write_code, _write_exports) and regenerate. - Add a nested python/databricks/bundles/.gitattributes marking the generated _models trees and namespace __init__.py files as linguist-generated (core/ and resources/ stay hand-written). Addresses review comments on #6335. Co-authored-by: Isaac <no-reply@databricks.com>

Match all namespace __init__.py files and unset the hand-written core one, instead of listing each generated namespace. New resource namespaces are then marked automatically without touching .gitattributes. Co-authored-by: Isaac <no-reply@databricks.com>

Wire the catalog resource into the core package: add_catalog, the catalogs collection property, catalog_mutator, and the _ResourceType registration, mirroring the other resources. The catalog model was already generated (resources.Catalog is in RESOURCE_NAMESPACE) but never wired, so it was not user-addable. Add a catalog case to test_resources.py. Catalog is supported only on the direct deployment engine (there is no terraform converter for it); this matches YAML-defined catalogs. Co-authored-by: Isaac <no-reply@databricks.com>

Validate that a YAML-defined catalog and a Python-added catalog coexist, serialize, and are mutated by catalog_mutator. Restricted to the direct deployment engine, since catalogs have no terraform converter. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Rewrite Resources.add_resource and Resources.add_resources to iterate _ResourceType.all() instead of enumerating each resource type by hand, so they no longer need per-resource edits when a resource is added. Behavior-preserving. Co-authored-by: Isaac <no-reply@databricks.com>

Replace the per-type Resources._jobs/_pipelines/... attributes with a single self._resources dict keyed by plural name, populated by iterating _ResourceType.all(). __init__ no longer needs a per-resource line; the add_X methods and properties read/write self._resources[plural]. Behavior-preserving. Co-authored-by: Isaac <no-reply@databricks.com>

Replace the per-type "for a job" / "for an alert" phrasing in the duplicate-resource-name error with a uniform "for resource '<type>'". This removes the grammatical-article special case so the wiring can be generated uniformly. Drops the now-unused TestCase.article field. Co-authored-by: Isaac <no-reply@databricks.com>

Add codegen/generated_wiring.py, which emits the per-resource wiring for databricks.bundles.core into a new core/_generated/ package: the _ResourceType registry (_resource_types.py), the Resources add_*/property methods (_resources.py, a mixin), and the *_mutator decorators (_resource_mutators.py). main.py calls it after the model-generation loop. The core package __init__ is now generated too (static exports plus the generated mutator exports); mark core/_generated/ generated in .gitattributes and wipe it before regenerating. The hand-written wiring in _resource_type.py / _resources.py / _resource_mutator.py is removed in the next commit. Co-authored-by: Isaac <no-reply@databricks.com>

Resources now inherits the generated _GeneratedResources mixin instead of defining per-resource add_*/property methods; _ResourceType.all() returns the generated tuple; and _resource_mutator.py keeps only the ResourceMutator dataclass, with the decorators now generated. Tests import the mutators from the public databricks.bundles.core instead of the internal module. Behavior-preserving: the generated wiring reproduces what was hand-written. Co-authored-by: Isaac <no-reply@databricks.com>

Address review feedback that the f-string-based generator was hard to reason about. Each wired resource now gets its own _generated/<plural>.py rendered from wiring_resource.py.tmpl (a string.Template with $-placeholders, so the generated shape reads like real Python and there is no brace-escaping). The generated _generated/__init__.py collects the per-resource mixins into _GeneratedResources, exposes _all_resource_types(), and re-exports the mutators. Per-resource files stay small as more resources are onboarded instead of growing single aggregate modules. Behavior unchanged: same generated API, tests green, regeneration idempotent. Co-authored-by: Isaac <no-reply@databricks.com>

…-autogen-wiring # Conflicts: # python/databricks/bundles/.gitattributes # python/databricks/bundles/core/__init__.py # python/databricks/bundles/core/_resource_mutator.py # python/databricks/bundles/core/_resource_type.py # python/databricks/bundles/core/_resources.py # python/databricks_tests/core/test_resources.py

Sankalp-Mittal and others added 2 commits August 27, 2026 10:03

remove LOADED_NAMESPACES and instead run a coverage in jsonschema.py

3f3f58c

Merge branch 'main' into sankalp-mittal/automate-pybads-resources

a2b4f2f

Sankalp-Mittal force-pushed the sankalp-mittal/pydabs-autogen-wiring branch from 7f66409 to 1acc08f Compare August 27, 2026 21:22

Sankalp-Mittal changed the base branch from sankalp-mittal/pydabs-mark-generated-files to sankalp-mittal/pydabs-catalogs August 27, 2026 21:22

Sankalp-Mittal marked this pull request as ready for review August 27, 2026 22:51

Sankalp-Mittal changed the title ~~PyDABs codegen: autogenerate core resource wiring [WIP]~~ PyDABs codegen: autogenerate core resource wiring Aug 28, 2026

Sankalp-Mittal requested a review from janniklasrose August 28, 2026 09:08

Sankalp-Mittal and others added 6 commits August 28, 2026 14:19

Merge branch 'main' into sankalp-mittal/automate-pybads-resources

c23cdb1

Add changelog entry for PyDABs catalogs support

686b343

Co-authored-by: Isaac <no-reply@databricks.com>

Sankalp-Mittal force-pushed the sankalp-mittal/pydabs-catalogs branch from 0b7588b to 686b343 Compare August 28, 2026 12:33

Sankalp-Mittal and others added 8 commits August 28, 2026 12:34

remove the useless article generator

9ce5799

remove articles completely from the code

e1c2b7a

Sankalp-Mittal force-pushed the sankalp-mittal/pydabs-autogen-wiring branch from 1def02a to d385c7f Compare August 28, 2026 12:35

Base automatically changed from sankalp-mittal/pydabs-catalogs to main August 28, 2026 14:15

Sankalp-Mittal force-pushed the sankalp-mittal/pydabs-autogen-wiring branch from d385c7f to 6feddc6 Compare August 28, 2026 14:26

Sankalp-Mittal force-pushed the sankalp-mittal/pydabs-autogen-wiring branch from 6feddc6 to cfeb0a7 Compare August 28, 2026 14:31

Navigation Menu

Uh oh!

PyDABs codegen: autogenerate core resource wiring - #6397

PyDABs codegen: autogenerate core resource wiring#6397
Sankalp-Mittal wants to merge 17 commits into
mainfrom
sankalp-mittal/pydabs-autogen-wiring

Sankalp-Mittal commented Aug 27, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Aug 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Sankalp-Mittal commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Tests

Uh oh!

eng-dev-ecosystem-bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

github-actions Bot commented Aug 27, 2026

Waiting for approval

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sankalp-Mittal commented Aug 27, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Aug 27, 2026 •

edited

Loading