refactor(kernel): drop azure-sp-m2m management token on the kernel path

Mirror the kernel-side simplification: the kernel's `azure-sp-m2m` no longer
supports the Azure management-token flow (RBAC-only SPs), so stop forwarding
`azure_workspace_resource_id` to the kernel. The SP must be a workspace member
on `use_kernel=True`.

`azure_workspace_resource_id` is still a valid connection parameter for the
Thrift path (unchanged). On the kernel path, setting it now logs a warning
(rather than silently dropping a security-relevant auth parameter and failing
later with an opaque 403) and the SP authenticates with the Databricks-audience
data token alone. Drops the threading through `kernel_auth_options` in
session.py. RBAC-only-SP support is a documented follow-up.

Co-authored-by: Isaac
Signed-off-by: eric-wang-1990 <e.wang@databricks.com>

Author: eric-wang-1990

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 # Unreleased
 - Kernel backend (`use_kernel=True`): OAuth U2M with `auth_type="databricks-oauth"` now forwards the connector's `databricks-sql-python` OAuth-app bundle (`client_id` + `sql offline_access` scopes + redirect port) into the kernel, so a bare U2M connection authenticates as `databricks-sql-python` — parity with the Thrift path — instead of inheriting the kernel's own `databricks-sql-connector` default. A caller-supplied `oauth_client_id` (with its coupled `oauth_redirect_port`) is honored, as is a caller-supplied `oauth_scopes`; absent one, the connector default (`sql offline_access`) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full `8020..8024` range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass `oauth_redirect_port` (with `oauth_client_id`) to pick a free one on a port collision (PECOBLR-4040)
-- Kernel backend (`use_kernel=True`): **Azure Entra (Azure AD) OAuth is now supported.** The kernel is the Azure-aware auth core (it owns the endpoints, scopes, app ids, and tenant discovery); the connector forwards the selector and Azure credentials unchanged, so `connect()` is byte-identical between the Thrift and `use_kernel=True` paths. `auth_type="azure-oauth"` (Azure AD U2M) runs the kernel's browser flow against the workspace v2.0 authorize/token endpoints with the Azure app client id (`96eecda7-…`), redirect port `8030`, and the `{app_id}/user_impersonation offline_access` scope. `auth_type="azure-sp-m2m"` (Azure service principal) forwards `azure_client_id` / `azure_client_secret`; the kernel builds the Entra v2.0 token endpoint and the `{app_id}/.default` scope, and **auto-discovers the tenant** from the workspace's `/aad/auth` redirect when `azure_tenant_id` is omitted (matching Thrift). `azure_workspace_resource_id` is an optional add-on: set it and the kernel additionally sends the Azure management-token header pair (`X-Databricks-Azure-SP-Management-Token` + `X-Databricks-Azure-Workspace-Resource-Id`) so an SP that holds only an Azure RBAC role (not a workspace member) can authenticate; omit it and the SP authenticates with the data token alone (PECOBLR-4141; PECOBLR-4120)
+- Kernel backend (`use_kernel=True`): **Azure Entra (Azure AD) OAuth is now supported.** The kernel is the Azure-aware auth core (it owns the endpoints, scopes, app ids, and tenant discovery); the connector forwards the selector and Azure credentials unchanged, so `connect()` is byte-identical between the Thrift and `use_kernel=True` paths. `auth_type="azure-oauth"` (Azure AD U2M) runs the kernel's browser flow against the workspace v2.0 authorize/token endpoints with the Azure app client id (`96eecda7-…`), redirect port `8030`, and the `{app_id}/user_impersonation offline_access` scope. `auth_type="azure-sp-m2m"` (Azure service principal) forwards `azure_client_id` / `azure_client_secret`; the kernel builds the Entra v2.0 token endpoint and the `{app_id}/.default` scope, and **auto-discovers the tenant** from the workspace's `/aad/auth` redirect when `azure_tenant_id` is omitted (matching Thrift). The service principal authenticates with the Databricks-audience data token, so it must be a workspace member; the kernel path does not yet support the Azure management-token flow (for an RBAC-only SP), so `azure_workspace_resource_id` is ignored with a warning on `use_kernel=True` (it still applies on the Thrift path) (PECOBLR-4141; PECOBLR-4120)
 
 # 4.4.0 (2026-07-22)
 - Raised the minimum supported Python version to 3.10, dropping the end-of-life 3.8/3.9, to update the lockfile and clear CVE-flagged dependencies in the repo (databricks/databricks-sql-python#798)

CONNECTION_PARAMETERS.md

@@ -78,7 +78,7 @@ to change without notice.
 | `identity_federation_client_id`                     | `str`                |   ✅   |   ✅   | `None`                    | Workload identity / token-federation client id (kernel support added in #910).                                                                                 |
 | `experimental_oauth_persistence`                    | `OAuthPersistence`   |   ✅   |   ❌   | `None`                    | **Thrift-only.** The kernel owns its own token lifecycle and does not accept a persistence store.                                                              |
 | `azure_client_id` / `azure_client_secret` / `azure_tenant_id` | `str` | ✅ | ✅ | `None` | Azure service-principal (Entra ID M2M), selected by `auth_type="azure-sp-m2m"`. On the kernel path the connector forwards these to the kernel, which owns Azure resolution (Entra v2.0 token endpoint + the Databricks-resource `.default` scope) (#919). **`azure_tenant_id` is optional on the kernel path too** — like Thrift, the kernel auto-discovers it from the workspace's `/aad/auth` redirect when omitted. |
-| `azure_workspace_resource_id`                       | `str`                |   ✅   |   ✅   | `None`                    | Optional add-on for `azure-sp-m2m`. When set, the SP **management token** (`X-Databricks-Azure-SP-Management-Token` + `X-Databricks-Azure-Workspace-Resource-Id`) is sent to authorize an SP that has an Azure RBAC role but is not a workspace member. On the kernel path the connector forwards it and the kernel fetches the management token and emits the header pair; omit it and the SP authenticates with the Databricks-audience data token alone. |
+| `azure_workspace_resource_id`                       | `str`                |   ✅   |   ⚠️   | `None`                    | For `azure-sp-m2m`. When set, the SP **management token** (`X-Databricks-Azure-SP-Management-Token` + `X-Databricks-Azure-Workspace-Resource-Id`) is sent to authorize an SP that has an Azure RBAC role but is not a workspace member. **Not yet supported on the kernel path** — the SP must be a workspace member there; setting it on `use_kernel=True` logs a warning and is otherwise ignored (RBAC-only-SP support is a planned follow-up). Applies normally on the Thrift path. |
 | `_use_cert_as_auth` (+ `_tls_client_cert_file`)     | `bool`               |   ✅   |   ❌   | `False`                   | Authenticate with a TLS client certificate instead of a token. Thrift-only.                                                                                    |
 | `username` / `password`                             | `str`                |   ❌   |   ❌   | `None`                    | **Removed.** Basic auth is no longer supported; passing either raises `ValueError`.                                                                            |
 

src/databricks/sql/backend/kernel/auth_bridge.py

@@ -36,9 +36,9 @@
     The kernel builds the Entra v2.0 token endpoint and the
     ``{effective_app_id}/.default`` scope, and auto-discovers the tenant from
     the workspace's ``/aad/auth`` redirect when ``azure_tenant_id`` is omitted
-    (Thrift parity). ``azure_workspace_resource_id`` is an optional add-on:
-    forward it and the kernel additionally sends the Azure management-token
-    header pair, so an RBAC-only SP (not a workspace member) can authenticate
+    (Thrift parity). The SP must be a workspace member (the kernel path does not
+    yet support the Azure management-token flow, so ``azure_workspace_resource_id``
+    is ignored with a warning; RBAC-only SPs are a planned follow-up)
     (PECOBLR-4141).
 
 ``identity_federation_client_id`` is forwarded with whichever auth shape
@@ -239,12 +239,10 @@ def kernel_auth_kwargs(
     # (so connect() is byte-identical between Thrift and use_kernel=True).
     # PECOBLR-4141.
     #
-    # azure_workspace_resource_id is an optional add-on: forward it and the
-    # kernel additionally fetches an Azure-management token and sends the
-    # X-Databricks-Azure-SP-Management-Token + X-Databricks-Azure-Workspace-
-    # Resource-Id pair, so an SP that holds only an Azure RBAC role (not a
-    # workspace member) can authenticate. Omit it (the common case) and the SP
-    # authenticates with the Databricks-audience data token alone.
+    # The SP authenticates with the Databricks-audience data token, so it must be
+    # a workspace member. The kernel path does not yet support the Azure
+    # management-token flow (for an RBAC-only SP), so azure_workspace_resource_id
+    # is ignored with a warning here — see below.
     if auth_type == "azure-sp-m2m":
         azure_client_id = opts.get("azure_client_id")
         azure_client_secret = opts.get("azure_client_secret")
@@ -258,14 +256,22 @@ def kernel_auth_kwargs(
             "azure_client_id": azure_client_id,
             "azure_client_secret": azure_client_secret,
         }
-        # Optional passthroughs: the kernel auto-discovers the tenant when
-        # absent, and sends the data token alone when no resource id is set.
+        # Optional passthrough: the kernel auto-discovers the tenant when absent.
         azure_tenant_id = opts.get("azure_tenant_id")
         if azure_tenant_id:
             kwargs["azure_tenant_id"] = azure_tenant_id
-        azure_workspace_resource_id = opts.get("azure_workspace_resource_id")
-        if azure_workspace_resource_id:
-            kwargs["azure_workspace_resource_id"] = azure_workspace_resource_id
+        # azure_workspace_resource_id drives the Azure management-token flow on
+        # the Thrift path (for an SP with only an Azure RBAC role, not a
+        # workspace member). The kernel path does NOT support it yet — the SP
+        # must be a workspace member — so warn rather than silently dropping a
+        # security-relevant auth parameter and failing later with an opaque 403.
+        if opts.get("azure_workspace_resource_id"):
+            logger.warning(
+                "azure_workspace_resource_id is ignored on use_kernel=True: the "
+                "Azure SP management-token flow is not yet supported on the "
+                "kernel path. The service principal must be a workspace member "
+                "(the Databricks-audience token alone authenticates it)."
+            )
         if federation_client_id:
             kwargs["identity_federation_client_id"] = federation_client_id
         return kwargs

src/databricks/sql/session.py

@@ -181,15 +181,13 @@ def _create_backend(
                 # kernel owns Azure resolution (endpoint/scope/tenant discovery),
                 # so these raw kwargs are the only source; without threading them
                 # the bridge would fail with "requires azure_client_id". The
-                # tenant and workspace-resource-id are optional (kernel
-                # auto-discovers the tenant; the resource id gates the optional
-                # management token). Kernel-only; Thrift / SEA are unaffected.
+                # tenant is optional (the kernel auto-discovers it). Kernel-only;
+                # Thrift / SEA are unaffected. (azure_workspace_resource_id is not
+                # forwarded — the kernel path does not yet support the Azure
+                # management-token flow; the bridge warns if it is set.)
                 "azure_client_id": kwargs.get("azure_client_id"),
                 "azure_client_secret": kwargs.get("azure_client_secret"),
                 "azure_tenant_id": kwargs.get("azure_tenant_id"),
-                "azure_workspace_resource_id": kwargs.get(
-                    "azure_workspace_resource_id"
-                ),
             }
             # Forward the connector's retry-tuning kwargs so the kernel's
             # own retry policy honours them (the kernel owns the retry

tests/unit/test_kernel_auth_bridge.py

@@ -537,21 +537,22 @@ def test_azure_sp_m2m_tenant_optional_kernel_autodiscovers(self):
         }
         assert "azure_tenant_id" not in kwargs
 
-    def test_azure_sp_m2m_forwards_workspace_resource_id(self):
-        # An optional add-on: when set, the kernel fetches an Azure-management
-        # token and emits the X-Databricks-Azure-* header pair (for an SP with
-        # only an Azure RBAC role, not a workspace member). The binding forwards
-        # it rather than dropping it.
+    def test_azure_sp_m2m_ignores_workspace_resource_id_with_warning(self, caplog):
+        # The kernel path does not yet support the Azure management-token flow
+        # (for an RBAC-only SP), so azure_workspace_resource_id is NOT forwarded
+        # — but we warn rather than silently dropping a security-relevant param.
         opts = dict(
             self._CREDS,
             azure_workspace_resource_id="/subscriptions/s/resourceGroups/rg/workspace/w",
         )
-        kwargs = kernel_auth_kwargs(
-            _FakeOAuthProvider(), opts, hostname="adb-1.azuredatabricks.net"
-        )
-        assert (
-            kwargs["azure_workspace_resource_id"]
-            == "/subscriptions/s/resourceGroups/rg/workspace/w"
+        with caplog.at_level("WARNING"):
+            kwargs = kernel_auth_kwargs(
+                _FakeOAuthProvider(), opts, hostname="adb-1.azuredatabricks.net"
+            )
+        assert "azure_workspace_resource_id" not in kwargs
+        assert any(
+            "azure_workspace_resource_id is ignored" in r.message
+            for r in caplog.records
         )
 
     def test_azure_sp_m2m_omits_workspace_resource_id_when_absent(self):

tests/unit/test_session.py

@@ -512,7 +512,6 @@ def test_azure_sp_m2m_kwargs_threaded_into_kernel_auth_options(self):
                 azure_client_id="azure-sp",
                 azure_client_secret="azure-secret",
                 azure_tenant_id="tenant-123",
-                azure_workspace_resource_id="/subscriptions/s/rg/w",
                 enable_telemetry=False,
             )
             try:
@@ -522,7 +521,6 @@ def test_azure_sp_m2m_kwargs_threaded_into_kernel_auth_options(self):
                 assert opts["azure_client_id"] == "azure-sp"
                 assert opts["azure_client_secret"] == "azure-secret"
                 assert opts["azure_tenant_id"] == "tenant-123"
-                assert opts["azure_workspace_resource_id"] == "/subscriptions/s/rg/w"
             finally:
                 conn.close()