databricks: support Lakehouse//RT, auto-detecting the SEA backend

[dey-abhishek]

databricks: support Lakehouse//RT, auto-detecting the SEA backend (backward compatible with DBSQL)

Problem

Lakehouse//RT (real-time) Databricks warehouses only speak the Statement Execution
API (SEA) protocol and reject the Thrift/HiveServer2 protocol that databricks-sql-go
uses by default:

BAD_REQUEST: Lakehouse/RT is not supported for Thrift protocol.
Please update your Databricks SQL Driver version to the latest version,
which supports the Statement Execution API protocol  (HTTP 400)

Rill currently pins databricks-sql-go v1.10.0, which predates the driver's SEA/kernel
backend, so Rill cannot connect to Lakehouse//RT warehouses at all — the connector fails
at open, before any query runs.

What this changes

1. Bump databricks-sql-go v1.10.0 → v1.15.1. v1.15.0 introduced the opt-in
   SEA/kernel backend (useKernel=true / WithUseKernel(true)). The default remains
   Thrift, so this bump alone changes no behavior.
2. Auto-detect Lakehouse//RT (no configuration needed). On first connect the
   connector probes once; if the warehouse rejects Thrift (the RT signal), it
   transparently switches to the SEA backend. The probe result is cached and shared by
   both the OLAP path and the warehouse ingest path (effectiveDSN). DBSQL warehouses
   accept Thrift and are left unchanged. This means a user can add an RT warehouse in
   the UI with no special knowledge — no use_kernel, no UseThriftClient — and it
   just works.
3. use_kernel connector property (default false) as an explicit override. Forces
   SEA and skips the probe. Rarely needed given auto-detection, but useful to force SEA.
4. warehouse.go ingest fallback. The bulk-ingest path prefers
   GetArrowIPCStreams (unchanged Thrift path). The SEA backend doesn't implement IPC
   streams — it exports Arrow C Data — and returns an error wrapping
   dbsqlerr.ErrNotSupportedByKernel. On that sentinel we fall back to
   GetArrowBatches and re-serialize each record to a self-contained Arrow IPC stream
   (via the driver's Arrow v12 ipc.Writer), which the existing v18 ipc.Reader →
   parquet path consumes untouched. This bridges the driver's Arrow v12 ↔ Rill's Arrow
   v18 without changing the DBSQL path. Rill's OLAP path (olap.go, sqlx rows) was
   already SEA-compatible and is unchanged.
5. information_schema.Lookup de-JOINed. The per-table schema lookup JOINed
   information_schema.tables and columns; that join forces a shuffle that RT's Photon
   rejects (PHOTON_INTERNAL_ERROR, retry unsupported), breaking the schema browser.
   Split into two filtered point-lookups (no shuffle) — verified against a real RT
   warehouse (the JOIN fails, both point-lookups succeed); equivalent on DBSQL.

Backward compatibility

• Default builds (no databricks_kernel tag) compile unchanged; use_kernel defaults to
  false, so DBSQL warehouses are byte-for-byte on the same Thrift path.
• If use_kernel: true is set in a build without the kernel backend linked, the driver
  fails at connect with a clear error wrapping ErrKernelNotCompiled (no silent Thrift
  fallback, no crash).

Build / distribution note (needs a maintainer decision)

The SEA backend links a native library and is gated behind -tags databricks_kernel
with CGO. Rill already builds with CGO enabled (DuckDB, confluent-kafka), so the only
additional cost is adding the tag and linking the per-platform kernel archive
(databricks-sql-kernel-bindings, ~60–95 MB/platform). This PR makes RT support
available and correct when built with the tag, and a graceful error otherwise. How
(and whether) to enable the tag in shipped release binaries is left as a follow-up for
maintainers.

Testing

• New unit tests runtime/drivers/databricks/dsn_internal_test.go: TestResolveDSN
  (backend selection — no useKernel by default, set when opted in, DSN pass-through,
  ?/& joining, no duplicate), TestRTRequiresSEA (only the Thrift-not-supported
  message triggers the switch — 403/refused/nil don't), and TestWithUseKernel.
• Auto-detection verified live against a real Lakehouse//RT warehouse with no
  use_kernel set: the connector logged the Thrift→SEA switch and both the connector
  and a model reconciled; a regular DBSQL warehouse stayed on Thrift and reconciled.
• Manual, both paths, end to end (kernel-tag build, macOS arm64):
  • Lakehouse//RT (use_kernel: true): connector + a SELECT current_catalog(), now()
    model reconcile successfully and materialize into DuckDB.
  • Regular DBSQL serverless warehouse (use_kernel unset, Thrift): a
    SELECT current_catalog(), current_version(), now() model reconciles successfully —
    current_version() works here but is UNRESOLVED_ROUTINE on RT, confirming the
    Thrift path is unaffected.

Checklist:

• Covered by tests (unit test for DSN resolution; manual e2e for both backends)
• Ran it and it works as intended
• Reviewed the diff before requesting a review
• Checked for unhandled edge cases (missing kernel build → clear error; no duplicate param)
• Linked the issues it closes
• Checked if the docs need to be updated (updated the Databricks connector doc)
• Intend to cherry-pick into the release branch
• I'm proud of this work!