feat!: enforce Databricks Runtime compatibility

[Tomoscorbin]

What changed

• Enforce the Spark backend's Databricks Runtime range at engine construction: >=16.4,<19.
• Fail closed for older, newer, unavailable, or malformed DBR versions while preserving lazy PySpark imports.
• Publish the range in a new runtime compatibility reference and append it automatically to future GitHub release notes.
• Add a Free Edition Spark release-smoke notebook and a manual release checklist; SQL warehouse compatibility remains separate from the DBR gate.

Why

A newly released DBR should not be able to run an existing Delta Engine release before that runtime family has been reviewed. The guard compares the Spark environment's current_version().dbr_version with the release's declared range before any table work begins.

Databricks' unified release model can still update a supported major family without changing its major number. The docs therefore distinguish the enforced range from the narrower live-test evidence.

Validation

• uv run pytest -q — 1018 passed, 64 deselected (96.54% coverage)
• uv run ruff check .
• uv run mypy .
• uv run lint-imports
• uv run --group docs sphinx-build -W -b html docs docs/_build/html
• uv build
• uvx twine check --strict dist/*
• Installed-wheel smoke tests

Before release

Run the Free Edition Spark smoke notebook and the existing SQL warehouse live suite, then record the observed versions and candidate commit in the compatibility evidence table.