chore: upgrade Databricks CLI to 1.9.0, DQX to 0.15.0, and track uv.lock

[andre-salvati]

What?

Version bumps across the deploy toolchain and dev dependencies, plus uv.lock is now tracked.

• Databricks CLI / bundles 0.298.0 → 1.9.0 (databricks.yml, setup-cli action, databricks-bundles)
• DQX 0.12.0 → 0.15.0
• Tooling: ruff hook v0.15.22, setup-uv@v9 + uv 0.11.31, pytest 9.1.1, pytest-cov 7.1.0, coverage 7.15.2, pre-commit 4.6.1, pydantic 2.13.4
• Unchanged on purpose: pyspark 4.1.0, pandas 2.2.3, numpy 2.1.3, pyarrow 21.0.0

Why?

The CLI was ~10 months behind and databricks-bundles tracks it in lockstep. DQX was three minor versions behind.

The serverless-mirror pins are not upgraded despite newer PyPI releases (pyspark 4.2.0, pandas 3.0.5, numpy 2.5.1, pyarrow 25.0.0). Serverless environment version 5 — the runtime the wheel executes on — ships Spark 4.1.0 with exactly pandas 2.2.3 / numpy 2.1.3 / pyarrow 21.0.0. Bumping them would put unit tests ahead of production. Environment version 5 is the newest available; there is no Spark 4.2 serverless environment yet. A comment in pyproject.toml now records this so the pins aren't "helpfully" bumped later.

How?

uv.lock is now tracked. The blanket *.lock ignore dated to the 2024 pipenv migration and predates uv; it silently excluded the lockfile, so CI resolved dependencies from scratch on every run. databricks-sdk isn't declared in pyproject.toml at all — it arrives transitively via DQX under ~=0.73, a 27-minor-version window — so CI could install a different SDK on two runs of the same commit. Tracking the lock also makes the existing cache-dependency-glob: "uv.lock" in CI meaningful; previously it hashed a file absent from every fresh checkout.

A uv lock --check CI step now runs before make sync, failing fast if a PR edits pyproject.toml without re-locking (otherwise make sync re-resolves silently in CI's checkout and the committed lock goes stale).

Auth migration (operator action). CLI v1.0.0 moved OAuth tokens from ~/.databricks/token-cache.json to the OS keyring. Profiles with auth_type = databricks-cli (dev, account) need a one-time databricks auth login --profile <name>. Service-principal profiles (DEFAULT, staging, prod) authenticate via client_id/client_secret and are unaffected — CI needs no change.

Validation?

Run on dev: make deploy env=dev (Deployment complete!) and make run env=dev (TERMINATED SUCCESS). The nested job1 run confirmed extract_source2 — the DQX task — SUCCESS, alongside run_sdp (SDP pipeline, full refresh).

• 16 unit tests pass on pytest 9.1.1
• ruff lint + format pass
• bundle validate OK on staging and prod, including the mode: production deployer-identity gate
• Job generator produces jobs.yml for all three targets on databricks-bundles 1.9.0
• uv sync --locked exit 0 — lock exactly satisfies pyproject.toml

Impact in prod

• No table schema/data change — no production impact.
• Schema change — classify and declare the remediation:

DQX 0.15 adds three fields to the _errors / _warnings structs on raw.order_quarantine: rule_fingerprint: string, rule_set_fingerprint: string, skipped: boolean.

	struct fields
old (DQX 0.12)	name, message, columns, filter, function, run_time, run_id, user_metadata
new (DQX 0.15)	above + rule_fingerprint, rule_set_fingerprint, skipped

Because every medallion write uses overwriteSchema=false, the widened struct is rejected against a table created under 0.12 → job1 extract_source2 fails. Remediation is 🟠 make drop env=<env> yes=--yes before the first run on that env (quarantine is a rebuildable DQX output — no source-of-truth data is lost).

Chosen strategy: drop + recreate.

Staging — done. make drop env=staging (17 objects) → make deploy env=staging → make run env=staging all passed, and CI on this branch is green. The recreated staging.raw.order_quarantine now carries the 11-field struct. This is what validates the migration end-to-end; the dev run could not, because dev_….raw.order_quarantine was created fresh mid-run (create path, not the overwriteSchema=false drift path).

Prod — pending, one manual step after merge. prod.raw.order_quarantine still holds the old 8-field struct (10,000 rows). Before the next prod job1 run: make drop env=prod yes=--yes. Note this rebuilds the full prod medallion (~44M rows) from external_source and re-freezes product_name against current values, so historical frozen names do not survive — schedule it deliberately.