refactor(kernel): thin azure-sp-m2m — kernel owns Azure resolution + optional mgmt token

Make the `azure-sp-m2m` bridge thin, matching the kernel becoming the
Azure-aware auth core. The connector now forwards
`auth_type='azure-sp-m2m'` + `azure_client_id` / `azure_client_secret`
(and optional `azure_tenant_id` / `azure_workspace_resource_id`) straight
to the kernel Session, instead of constructing the Entra token endpoint
and `{app_id}/.default` scope itself.

Behavior changes on the kernel path (Thrift parity):
- `azure_tenant_id` is now OPTIONAL — the kernel auto-discovers the
  tenant from the workspace's `/aad/auth` redirect when omitted, exactly
  as the Thrift backend does. (Previously the kernel path required it.)
- `azure_workspace_resource_id` is now honored as an optional add-on:
  forward it and the kernel fetches an Azure-management token and sends
  the `X-Databricks-Azure-SP-Management-Token` +
  `X-Databricks-Azure-Workspace-Resource-Id` pair, so an SP with only an
  Azure RBAC role (not a workspace member) can authenticate. (Previously
  it was dropped with a warning.)

Also thread the `azure_*` connection kwargs into `kernel_auth_options`
in session.py — without this the bridge never received them and
`azure-sp-m2m` failed at session-open with "requires azure_client_id".
Adds a regression test for that threading, and rewrites the bridge tests
for thin forwarding (tenant optional, resource id forwarded). Drops the
now-unused `get_effective_azure_login_app_id` import and
`_AZURE_AAD_LOGIN_HOST` constant.

Co-authored-by: Isaac
Signed-off-by: eric-wang-1990 <e.wang@databricks.com>

Author: eric-wang-1990

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 # Unreleased
 - Kernel backend (`use_kernel=True`): OAuth U2M with `auth_type="databricks-oauth"` now forwards the connector's `databricks-sql-python` OAuth-app bundle (`client_id` + `sql offline_access` scopes + redirect port) into the kernel, so a bare U2M connection authenticates as `databricks-sql-python` — parity with the Thrift path — instead of inheriting the kernel's own `databricks-sql-connector` default. A caller-supplied `oauth_client_id` (with its coupled `oauth_redirect_port`) is honored, as is a caller-supplied `oauth_scopes`; absent one, the connector default (`sql offline_access`) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full `8020..8024` range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass `oauth_redirect_port` (with `oauth_client_id`) to pick a free one on a port collision (PECOBLR-4040)
-- Kernel backend (`use_kernel=True`): **Azure Entra (Azure AD) OAuth is now supported.** Both connector Azure auth types route onto the kernel's generic OAuth flows: `auth_type="azure-oauth"` (Azure AD U2M) forwards the Azure app bundle (client id `96eecda7-…`, redirect port `8030`, and the `{app_id}/user_impersonation offline_access` delegated scope, honoring `DATABRICKS_AZURE_TENANT_ID`) to the kernel's U2M browser flow; `auth_type="azure-sp-m2m"` (Azure service principal) forwards the Azure credentials with an Entra v2.0 token endpoint and the `{app_id}/.default` scope to the kernel's M2M flow. `azure-sp-m2m` requires an explicit `azure_tenant_id` on the kernel path (it does not auto-discover the tenant as the Thrift path does), and the Azure management-token header / `azure_workspace_resource_id` are not applied — the Databricks-audience token authenticates service principals that are workspace principals, matching the Go and Node SQL connectors (PECOBLR-4141; PECOBLR-4120)
+- Kernel backend (`use_kernel=True`): **Azure Entra (Azure AD) OAuth is now supported.** The kernel is the Azure-aware auth core (it owns the endpoints, scopes, app ids, and tenant discovery); the connector forwards the selector and Azure credentials unchanged, so `connect()` is byte-identical between the Thrift and `use_kernel=True` paths. `auth_type="azure-oauth"` (Azure AD U2M) runs the kernel's browser flow against the workspace v2.0 authorize/token endpoints with the Azure app client id (`96eecda7-…`), redirect port `8030`, and the `{app_id}/user_impersonation offline_access` scope. `auth_type="azure-sp-m2m"` (Azure service principal) forwards `azure_client_id` / `azure_client_secret`; the kernel builds the Entra v2.0 token endpoint and the `{app_id}/.default` scope, and **auto-discovers the tenant** from the workspace's `/aad/auth` redirect when `azure_tenant_id` is omitted (matching Thrift). `azure_workspace_resource_id` is an optional add-on: set it and the kernel additionally sends the Azure management-token header pair (`X-Databricks-Azure-SP-Management-Token` + `X-Databricks-Azure-Workspace-Resource-Id`) so an SP that holds only an Azure RBAC role (not a workspace member) can authenticate; omit it and the SP authenticates with the data token alone (PECOBLR-4141; PECOBLR-4120)
 
 # 4.4.0 (2026-07-22)
 - Raised the minimum supported Python version to 3.10, dropping the end-of-life 3.8/3.9, to update the lockfile and clear CVE-flagged dependencies in the repo (databricks/databricks-sql-python#798)

CONNECTION_PARAMETERS.md

@@ -69,16 +69,16 @@ to change without notice.
 | Option                                              | Type                 | Thrift | Kernel | Default Value             | Note                                                                                                                                                            |
 | --------------------------------------------------- | -------------------- | :----: | :----: | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `access_token` (PAT)                                | `str`                |   ✅   |   ✅   | `None`                    | Personal Access Token / bearer token. The default auth mode when set; otherwise auth falls back to OAuth.                                                       |
-| `auth_type`                                         | `str`                |   ✅   |   ✅   | `None` ⇒ Databricks OAuth | `databricks-oauth` (U2M), `azure-oauth` (Azure AD U2M), or `azure-sp-m2m` (Azure service-principal M2M). All three work on the kernel path; `azure-oauth` / `azure-sp-m2m` kernel support added in #919 (they route onto the kernel's generic OAuth flows with Azure values). |
+| `auth_type`                                         | `str`                |   ✅   |   ✅   | `None` ⇒ Databricks OAuth | `databricks-oauth` (U2M), `azure-oauth` (Azure AD U2M), or `azure-sp-m2m` (Azure service-principal M2M). All three work on the kernel path; `azure-oauth` / `azure-sp-m2m` kernel support added in #919 (the connector forwards the selector + Azure credentials and the kernel owns Azure resolution). |
 | `oauth_client_id` (U2M)                             | `str`                |   ✅   |   ✅   | built-in client id        | Custom U2M client id. Forwarded on both; when absent, each path applies its own built-in default.                                                              |
 | `oauth_redirect_port` (U2M)                         | `int`                |   ✅   |   ✅   | `None`                    | Localhost redirect port for the browser flow. On **both** backends it is only honored when a custom `oauth_client_id` is also supplied — then that single port becomes the redirect URI. With the built-in client id (or when omitted) the connector uses the full registered range 8020–8024 and binds the first free port, so a bare `oauth_redirect_port` has no effect. (Thrift: `auth.py` `oauth_redirect_port_range`; Kernel: same logic, forwarded as `redirect_ports`.) |
 | `oauth_client_secret` (OAuth M2M)                   | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only in practice.** The Thrift auth path never reads `oauth_client_secret`; use `credentials_provider` or an Azure service principal for M2M on Thrift. |
 | `oauth_scopes`                                      | `List[str]`          |   ❌   |   ✅   | `["sql","offline_access"]`| **Thrift ignores custom scopes** — it always uses the built-in scope set. Only the kernel honors a custom `oauth_scopes`.                                       |
 | `credentials_provider`                              | `CredentialsProvider`|   ✅   |   ❌   | `None`                    | Custom external credentials provider. **Rejected on the kernel path** (`NotSupportedError`) — it is an opaque token source, so the kernel cannot own the token lifecycle; use `oauth_client_id` + `oauth_client_secret` for M2M, or the Thrift backend. |
 | `identity_federation_client_id`                     | `str`                |   ✅   |   ✅   | `None`                    | Workload identity / token-federation client id (kernel support added in #910).                                                                                 |
 | `experimental_oauth_persistence`                    | `OAuthPersistence`   |   ✅   |   ❌   | `None`                    | **Thrift-only.** The kernel owns its own token lifecycle and does not accept a persistence store.                                                              |
-| `azure_client_id` / `azure_client_secret` / `azure_tenant_id` | `str` | ✅ | ✅ | `None` | Azure service-principal (Entra ID M2M), selected by `auth_type="azure-sp-m2m"`. On the kernel path these route onto OAuth M2M with an Entra v2.0 token endpoint + the Databricks-resource `.default` scope (#919). **`azure_tenant_id` is required on the kernel path** — unlike Thrift, it is not auto-discovered from the workspace. |
-| `azure_workspace_resource_id`                       | `str`                |   ✅   |   ⚠️   | `None`                    | Thrift sends this with the Azure SP **management token** (`X-Databricks-Azure-SP-Management-Token`) to authorize an SP that has an Azure RBAC role but is not a workspace member. **Not applied on the kernel path** — the management-token flow is unsupported there (matching the Go and Node SQL drivers, which don't use it); add the SP as a workspace principal instead. Setting it on the kernel path logs a warning and is otherwise ignored. |
+| `azure_client_id` / `azure_client_secret` / `azure_tenant_id` | `str` | ✅ | ✅ | `None` | Azure service-principal (Entra ID M2M), selected by `auth_type="azure-sp-m2m"`. On the kernel path the connector forwards these to the kernel, which owns Azure resolution (Entra v2.0 token endpoint + the Databricks-resource `.default` scope) (#919). **`azure_tenant_id` is optional on the kernel path too** — like Thrift, the kernel auto-discovers it from the workspace's `/aad/auth` redirect when omitted. |
+| `azure_workspace_resource_id`                       | `str`                |   ✅   |   ✅   | `None`                    | Optional add-on for `azure-sp-m2m`. When set, the SP **management token** (`X-Databricks-Azure-SP-Management-Token` + `X-Databricks-Azure-Workspace-Resource-Id`) is sent to authorize an SP that has an Azure RBAC role but is not a workspace member. On the kernel path the connector forwards it and the kernel fetches the management token and emits the header pair; omit it and the SP authenticates with the Databricks-audience data token alone. |
 | `_use_cert_as_auth` (+ `_tls_client_cert_file`)     | `bool`               |   ✅   |   ❌   | `False`                   | Authenticate with a TLS client certificate instead of a token. Thrift-only.                                                                                    |
 | `username` / `password`                             | `str`                |   ❌   |   ❌   | `None`                    | **Removed.** Basic auth is no longer supported; passing either raises `ValueError`.                                                                            |
 

src/databricks/sql/backend/kernel/auth_bridge.py

@@ -21,23 +21,25 @@
   ``oauth_client_id`` / ``oauth_redirect_port`` overriding it) is
   forwarded to the kernel's ``auth_type='oauth-u2m'`` and the kernel
   runs the browser flow itself.
-- **Azure Entra (Azure AD)** — both Azure auth types route to the
-  kernel's *generic* OAuth flows with Azure values as overrides (the
-  kernel needs no Azure-specific code):
-
-  - ``azure-oauth`` (U2M) → ``oauth-u2m`` with the Azure app client id
-    (``96eecda7-…``), redirect port ``8030``, and the AAD delegated scope
-    ``{app_id}/user_impersonation offline_access`` (via
-    ``AzureOAuthEndpointCollection``, honoring ``DATABRICKS_AZURE_TENANT_ID``).
-    The kernel discovers endpoints via the workspace ``/oidc`` redirector,
-    which an Azure workspace redirects to Entra (PECOBLR-4120).
-  - ``azure-sp-m2m`` (M2M) → ``oauth-m2m`` with the Azure service-principal
-    credentials, an Entra v2.0 ``token_url``, and the
-    ``{effective_app_id}/.default`` scope (PECOBLR-4141). ``azure_tenant_id``
-    is required (the kernel path does not auto-discover it). The Azure
-    management-token header and ``azure_workspace_resource_id`` are **not**
-    applied on the kernel path — no SQL connector uses them; an SP that is
-    not a workspace member (RBAC-only) is unsupported here.
+- **Azure Entra (Azure AD)** — both Azure auth types forward the selector
+  and Azure credentials to the KERNEL, which is the Azure-aware auth core
+  (it owns the endpoints, scopes, app ids, and tenant discovery). The
+  binding stays thin — it does not construct endpoints or scopes:
+
+  - ``azure-oauth`` (U2M) → forward ``auth_type='azure-oauth'`` (plus any
+    optional ``oauth_client_id`` / ``oauth_redirect_port`` passthrough). The
+    kernel pins the workspace v2.0 authorize/token endpoints, the Azure app
+    client id (``96eecda7-…``), port ``8030``, and the
+    ``{app_id}/user_impersonation offline_access`` scope (PECOBLR-4120).
+  - ``azure-sp-m2m`` (M2M) → forward ``auth_type='azure-sp-m2m'`` with the
+    Azure service-principal ``azure_client_id`` / ``azure_client_secret``.
+    The kernel builds the Entra v2.0 token endpoint and the
+    ``{effective_app_id}/.default`` scope, and auto-discovers the tenant from
+    the workspace's ``/aad/auth`` redirect when ``azure_tenant_id`` is omitted
+    (Thrift parity). ``azure_workspace_resource_id`` is an optional add-on:
+    forward it and the kernel additionally sends the Azure management-token
+    header pair, so an RBAC-only SP (not a workspace member) can authenticate
+    (PECOBLR-4141).
 
 ``identity_federation_client_id`` is forwarded with whichever auth shape
 wins resolution. It selects mandatory SP-wide workload-identity token
@@ -72,15 +74,9 @@
     PYSQL_OAUTH_SCOPES,
 )
 from databricks.sql.auth.authenticators import AccessTokenAuthProvider, AuthProvider
-from databricks.sql.auth.common import get_effective_azure_login_app_id
 from databricks.sql.auth.token_federation import TokenFederationProvider
 from databricks.sql.exc import NotSupportedError, ProgrammingError
 
-# Entra (Azure AD) v2.0 token endpoint template. The kernel's generic M2M
-# provider sends the credentials as ``scope`` (v2.0), so we point it at the
-# v2.0 endpoint (the connector's own SP path uses the v1.0 ``resource`` form).
-_AZURE_AAD_LOGIN_HOST = "https://login.microsoftonline.com"
-
 logger = logging.getLogger(__name__)
 
 
@@ -175,7 +171,7 @@ def kernel_auth_kwargs(
          ``oauth_client_secret`` together.
 
     (The Azure Entra auth types — ``azure-oauth`` and ``azure-sp-m2m`` —
-    are routed to the kernel's generic OAuth flows up front, before these
+    are forwarded to the kernel's Azure-aware flows up front, before these
     guards; see the module docstring.)
     1. **OAuth M2M** — ``oauth_client_id`` + ``oauth_client_secret``
        both present → forward raw creds to the kernel's ``oauth-m2m``.
@@ -234,51 +230,42 @@ def kernel_auth_kwargs(
             kwargs["identity_federation_client_id"] = federation_client_id
         return kwargs
 
-    # azure-sp-m2m (Azure service principal, client-credentials): forward to
-    # oauth-m2m with the Azure app credentials, an Entra v2.0 token endpoint,
-    # and the {effective_app_id}/.default scope. The kernel sends the client
-    # secret via HTTP Basic (which Entra v2.0 accepts) and, because a
-    # token_url override is set, skips workspace OIDC discovery. PECOBLR-4141.
+    # azure-sp-m2m (Azure service principal, client-credentials): forward the
+    # selector + Azure SP credentials; the KERNEL owns Azure resolution (it is
+    # the auth core). The kernel builds the Entra v2.0 token endpoint
+    # (`{login}/{tenant}/oauth2/v2.0/token`) and the `{effective_app_id}/.default`
+    # scope, and — when azure_tenant_id is omitted — auto-discovers the tenant
+    # from the workspace's /aad/auth redirect, matching the Thrift backend
+    # (so connect() is byte-identical between Thrift and use_kernel=True).
+    # PECOBLR-4141.
     #
-    # NOT applied on the kernel path: the Azure management-token header
-    # (X-Databricks-Azure-SP-Management-Token) and azure_workspace_resource_id.
-    # No SQL connector (Go, Node) uses them; the Databricks-audience token
-    # authenticates SPs that are workspace principals (the SQL norm). An SP with
-    # only an Azure RBAC role (not a workspace member) is unsupported here.
+    # azure_workspace_resource_id is an optional add-on: forward it and the
+    # kernel additionally fetches an Azure-management token and sends the
+    # X-Databricks-Azure-SP-Management-Token + X-Databricks-Azure-Workspace-
+    # Resource-Id pair, so an SP that holds only an Azure RBAC role (not a
+    # workspace member) can authenticate. Omit it (the common case) and the SP
+    # authenticates with the Databricks-audience data token alone.
     if auth_type == "azure-sp-m2m":
         azure_client_id = opts.get("azure_client_id")
         azure_client_secret = opts.get("azure_client_secret")
-        azure_tenant_id = opts.get("azure_tenant_id")
         if not (azure_client_id and azure_client_secret):
             raise ProgrammingError(
                 "auth_type='azure-sp-m2m' requires azure_client_id and "
                 "azure_client_secret."
             )
-        if not azure_tenant_id:
-            # The Thrift path auto-discovers the tenant from the workspace's
-            # /aad/auth redirect; the kernel path does not make that call, so
-            # require it explicitly rather than silently guessing.
-            raise NotSupportedError(
-                "use_kernel=True auth_type='azure-sp-m2m' requires an explicit "
-                "azure_tenant_id (the kernel path does not auto-discover the "
-                "Azure tenant from the workspace as the Thrift backend does)."
-            )
-        if opts.get("azure_workspace_resource_id"):
-            logger.warning(
-                "azure_workspace_resource_id is ignored on use_kernel=True: the "
-                "Azure management-token flow (X-Databricks-Azure-SP-Management-"
-                "Token) is not applied on the kernel path. The Databricks-"
-                "audience token authenticates service principals that are "
-                "workspace principals; an RBAC-only SP is unsupported here."
-            )
-        app_id = get_effective_azure_login_app_id(hostname or "")
         kwargs = {
-            "auth_type": "oauth-m2m",
-            "client_id": azure_client_id,
-            "client_secret": azure_client_secret,
-            "token_url": f"{_AZURE_AAD_LOGIN_HOST}/{azure_tenant_id}/oauth2/v2.0/token",
-            "oauth_scopes": [f"{app_id}/.default"],
+            "auth_type": "azure-sp-m2m",
+            "azure_client_id": azure_client_id,
+            "azure_client_secret": azure_client_secret,
         }
+        # Optional passthroughs: the kernel auto-discovers the tenant when
+        # absent, and sends the data token alone when no resource id is set.
+        azure_tenant_id = opts.get("azure_tenant_id")
+        if azure_tenant_id:
+            kwargs["azure_tenant_id"] = azure_tenant_id
+        azure_workspace_resource_id = opts.get("azure_workspace_resource_id")
+        if azure_workspace_resource_id:
+            kwargs["azure_workspace_resource_id"] = azure_workspace_resource_id
         if federation_client_id:
             kwargs["identity_federation_client_id"] = federation_client_id
         return kwargs

src/databricks/sql/session.py

@@ -177,6 +177,19 @@ def _create_backend(
                 "identity_federation_client_id": kwargs.get(
                     "identity_federation_client_id"
                 ),
+                # Azure Entra SP credentials for the azure-sp-m2m path. The
+                # kernel owns Azure resolution (endpoint/scope/tenant discovery),
+                # so these raw kwargs are the only source; without threading them
+                # the bridge would fail with "requires azure_client_id". The
+                # tenant and workspace-resource-id are optional (kernel
+                # auto-discovers the tenant; the resource id gates the optional
+                # management token). Kernel-only; Thrift / SEA are unaffected.
+                "azure_client_id": kwargs.get("azure_client_id"),
+                "azure_client_secret": kwargs.get("azure_client_secret"),
+                "azure_tenant_id": kwargs.get("azure_tenant_id"),
+                "azure_workspace_resource_id": kwargs.get(
+                    "azure_workspace_resource_id"
+                ),
             }
             # Forward the connector's retry-tuning kwargs so the kernel's
             # own retry policy honours them (the kernel owns the retry