diff --git a/.agents/rules/template-versions.md b/.agents/rules/template-versions.md new file mode 100644 index 0000000000..e9bbd9ec75 --- /dev/null +++ b/.agents/rules/template-versions.md @@ -0,0 +1,24 @@ +--- +description: Version pins in bundle template versions.tmpl files (DBR, DB Connect, serverless env, Python) +globs: + - "libs/template/templates/**/library/versions.tmpl" +paths: + - "libs/template/templates/**/library/versions.tmpl" +--- + +# Bundle template version pins + +Each bundle template pins the runtime versions a freshly initialized project +ships with in `library/versions.tmpl`. The `default` template pins the full set +— `latest_lts_dbr_version`, `conservative_db_connect_version_spec`, +`serverless_environment_version`, `python_version_spec`, and +`default_python_version`; the SQL templates (`dbt-sql`, `default-sql`) define +only a subset. + +**RULE: Keep `conservative_db_connect_version_spec` (in `default/`) at the lowest version that still works. Bump it only when the pinned DBR release falls out of support — never just to match the newest serverless environment version.** The DB Connect client is only forward-compatible: it reaches compute of its own version and higher. The lowest working pin therefore maximizes the range of DBR versions a customer can connect to, whereas a high pin rules out customers on older DBR. Customers can upgrade themselves after initializing the template. This is why the pin typically lags the newest release. See PR #3897 and PR #6378 for prior history. + +**RULE: In `default/`, keep `serverless_environment_version`, the Python pins (`python_version_spec` / `default_python_version`), and `conservative_db_connect_version_spec` mutually compatible.** They form one set: a serverless environment version dictates a runtime Python version, and the DB Connect pin must support that Python. For example, environment version 5 uses Python 3.12, and DB Connect 16.4 supports Python 3.12. When you change any one, cross-check the other two against the [serverless environment version release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/) and the [DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements). + +**RULE: `serverless_environment_version` is the one macro that must stay in sync across templates — it is pinned to the same value (`5`) in both `default/` and `dbt-sql/`, so bump it in both.** Other macros that appear in more than one template hold intentionally different values and must NOT be synced: `latest_lts_dbr_version` is `16.4` in `default/` but `15.4` in the SQL templates, and each SQL template pins its own `latest_lts_db_connect_version_spec` (a distinct macro from `default/`'s `conservative_db_connect_version_spec`). Change only the templates that define a given macro, and only for the same reason. + +Changing a version pin changes rendered template output, so regenerate the acceptance goldens afterward (see [auto-generated-files.md](auto-generated-files.md)). diff --git a/libs/template/templates/default/library/versions.tmpl b/libs/template/templates/default/library/versions.tmpl index 2c58247bc6..b41af62ce7 100644 --- a/libs/template/templates/default/library/versions.tmpl +++ b/libs/template/templates/default/library/versions.tmpl @@ -5,19 +5,20 @@ 16.4.x-scala2.12 {{- end}} -{{/* A conservative version of DB Connect for local development. +{{/* A conservative DB Connect pin for local development. * - * DB Connect is only forward-compatible: a client connects to compute of the - * same version and higher. We keep this conservative (low) so a freshly - * initialized project can reach the widest range of DBR versions; a high - * version would rule out customers on older DBR. We use 16.4 rather than an - * older release because DBR 15 is no longer supported and used Python 3.11, - * whereas 16.4 uses Python 3.12 (matching the serverless environment and the - * template's Python pin). Customers can move to a newer version themselves - * after initializing the template. + * The DB Connect client is only forward-compatible (it reaches compute of its + * own version and higher), so keep this at the LOWEST version that still + * works: bump it only when the pinned DBR release falls out of support, never + * to match the newest serverless environment version. A higher pin rules out + * customers on older DBR; they can upgrade themselves after init. * - * See https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements - * for DB Connect release notes and version compatibility. + * Current floor is 16.4 (DBR 15 is out of support and ran Python 3.11; 16.4 + * runs Python 3.12, matching serverless env 5 and python_version_spec). + * + * See .agents/rules/template-versions.md for the full upgrade rule and the + * env-version / Python / DB Connect consistency invariant. + * https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements */}} {{define "conservative_db_connect_version_spec" -}} >=16.4,<16.5