fix(kernel): preserve empty metadata filters (#933)

* fix(kernel): preserve empty metadata filters

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

* fix(kernel): handle empty exact catalog filters

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

* refactor(kernel): simplify metadata filter forwarding

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

* refactor(kernel): forward metadata filters unchanged

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

* fix(kernel): retain catalog wildcard normalization

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

* chore(kernel): update pinned revision

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

* chore(kernel): pin merged metadata fix

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

* docs(metadata): clarify table catalog semantics

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

* docs(metadata): scope filter semantics by backend

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

* docs(metadata): keep shared API docs backend-neutral

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

---------

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

Author: vuanhphung

CHANGELOG.md

@@ -1,6 +1,7 @@
 # Release History
 
 # Unreleased
+- Kernel metadata filters no longer collapse empty strings to `None`; empty patterns therefore match nothing. Existing `%`/`*` catalog wildcard handling is unchanged (PECOBLR-4221).
 - Kernel backend (`use_kernel=True`): OAuth **M2M with a JWT private-key client assertion** (RFC 7523) is now supported. Pass `oauth_client_id` + `oauth_jwt_key_file` + `oauth_jwt_kid` (with optional `oauth_jwt_passphrase` for an encrypted PKCS#8 key, `oauth_jwt_algorithm` defaulting to `RS256`, `oauth_scopes`, and `token_url` for the IdP token endpoint) and the connector routes them to the kernel's `auth_type="oauth-m2m-jwt"`, which signs a short-lived assertion with the private key instead of sending a client secret. The kernel owns the token lifecycle. A private-key file is treated as unambiguous JWT M2M intent and is mutually exclusive with `oauth_client_secret` / `credentials_provider` (both raise `NotSupportedError`). Verified end-to-end against an Azure Databricks workspace with the service principal's public certificate registered on its Entra ID app registration. Requires `databricks-sql-kernel >= 0.2.0` with JWT support.
 - Kernel backend (`use_kernel=True`): OAuth U2M with `auth_type="databricks-oauth"` now forwards the connector's `databricks-sql-python` OAuth-app bundle (`client_id` + `sql offline_access` scopes + redirect port) into the kernel, so a bare U2M connection authenticates as `databricks-sql-python` — parity with the Thrift path — instead of inheriting the kernel's own `databricks-sql-connector` default. A caller-supplied `oauth_client_id` (with its coupled `oauth_redirect_port`) is honored, as is a caller-supplied `oauth_scopes`; absent one, the connector default (`sql offline_access`) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full `8020..8024` range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass `oauth_redirect_port` (with `oauth_client_id`) to pick a free one on a port collision (PECOBLR-4040)
 - Kernel backend (`use_kernel=True`): **Azure Entra (Azure AD) service-principal M2M is now supported.** `auth_type="azure-sp-m2m"` forwards `azure_client_id` / `azure_client_secret`; the kernel is the Azure-aware auth core — it builds the Entra v2.0 token endpoint and the `{app_id}/.default` scope, and **auto-discovers the tenant** from the workspace's `/aad/auth` redirect when `azure_tenant_id` is omitted (matching Thrift). The `Authorization` bearer is the Databricks-audience data token, which alone authenticates a workspace-member SP. Set `azure_workspace_resource_id` and the kernel also sends the Azure SP management token (`X-Databricks-Azure-SP-Management-Token`) + `X-Databricks-Azure-Workspace-Resource-Id` header (matching the JDBC driver), so a service principal with an Azure RBAC role but no workspace membership can authenticate; omit it and no ARM management-scope token is fetched. Azure AD **U2M** (`auth_type="azure-oauth"`) now routes to the kernel's OAuth U2M flow, identically to `auth_type="databricks-oauth"`: the kernel runs the in-house workspace-federated browser flow, which Azure workspaces support (the workspace federates login to Entra). It forwards the connector's `databricks-sql-python` OAuth app, not the Thrift Azure app (`96eecda7` / port 8030), which is registered for Thrift's direct-Entra flow the kernel does not perform (PECOBLR-4141; PECOBLR-4120)

KERNEL_REV

@@ -1 +1 @@
-628abd6f5045897efcadb38ec77a1e9e0c23544e
+d64009eb59404c1b082cb020296337f96dc0d4d7

src/databricks/sql/backend/kernel/client.py

@@ -126,35 +126,9 @@ def _is_not_found(exc: BaseException) -> bool:
     )
 
 
-def _none_if_blank(value: Optional[str]) -> Optional[str]:
-    """Map an empty/whitespace-only metadata filter to ``None``
-    ("match all"), matching the Thrift backend's effective behaviour.
-
-    The kernel's ``Identifier`` / ``LikePattern`` reject ``""`` with
-    ``InvalidArgument`` (-> ``ProgrammingError``); ``None`` is the
-    kernel's canonical "match all". Applied to schema / table / column
-    *pattern* args (which otherwise keep ``%`` / ``_`` as real LIKE
-    wildcards)."""
-    if value is None:
-        return None
-    return value if value.strip() else None
-
-
 def _catalog_or_none(value: Optional[str]) -> Optional[str]:
-    """Normalise a catalog filter: ``None`` / blank / ``'%'`` / ``'*'``
-    all mean "all catalogs" -> ``None``.
-
-    This makes ``columns(catalog='%')`` behave like
-    ``tables(catalog='%')`` / ``schemas(catalog='%')`` — the kernel
-    already treats blank/``%``/``*`` as "all catalogs" for SHOW SCHEMAS
-    / SHOW TABLES (``is_null_or_wildcard``) but treats the catalog as an
-    exact identifier for SHOW COLUMNS, so the three diverged. Normalising
-    connector-side makes them symmetric. This intentionally diverges from
-    raw-Thrift literalness (Thrift treats ``%`` as a literal catalog
-    name) in favour of JDBC "catalog is exact-or-all, not a pattern" +
-    internal consistency. Catalog is the only arg normalised this way;
-    schema/table/column patterns keep ``%`` / ``*`` as LIKE wildcards."""
-    if value is None or not value.strip() or value in ("%", "*"):
+    """Map supported all-catalog wildcards to the kernel's unset filter."""
+    if value is None or value in ("%", "*"):
         return None
     return value
 
@@ -1024,7 +998,7 @@ def get_schemas(
         try:
             stream = self._kernel_session.metadata().list_schemas(
                 catalog=_catalog_or_none(catalog_name),
-                schema_pattern=_none_if_blank(schema_name),
+                schema_pattern=schema_name,
             )
             return self._make_result_set(stream, cursor, self._synthetic_command_id())
         except Exception as exc:
@@ -1051,8 +1025,8 @@ def get_tables(
             # through preserves streaming for large schemas.
             stream = self._kernel_session.metadata().list_tables(
                 catalog=_catalog_or_none(catalog_name),
-                schema_pattern=_none_if_blank(schema_name),
-                table_pattern=_none_if_blank(table_name),
+                schema_pattern=schema_name,
+                table_pattern=table_name,
                 table_types=table_types if table_types else None,
             )
             return self._make_result_set(stream, cursor, self._synthetic_command_id())
@@ -1081,9 +1055,9 @@ def get_columns(
             # the user's perspective.
             stream = self._kernel_session.metadata().list_columns(
                 catalog=_catalog_or_none(catalog_name),
-                schema_pattern=_none_if_blank(schema_name),
-                table_pattern=_none_if_blank(table_name),
-                column_pattern=_none_if_blank(column_name),
+                schema_pattern=schema_name,
+                table_pattern=table_name,
+                column_pattern=column_name,
             )
             return self._make_result_set(stream, cursor, self._synthetic_command_id())
         except Exception as exc:

tests/e2e/test_kernel_backend.py

@@ -356,17 +356,31 @@ def test_metadata_columns(conn):
         assert len(rows) > 0
 
 
-# ── Metadata filter normalization (batch 3) ───────────────────────
+# ── Metadata filter semantics ─────────────────────────────────────
 
 
-def test_schemas_with_empty_string_filter_matches_all(conn):
-    """An empty-string schema pattern normalizes to match-all rather
-    than raising ``ProgrammingError`` (kernel rejects ``""``) — locks
-    ``_none_if_blank`` on the pattern args."""
+def test_schemas_with_empty_string_filter_matches_nothing(conn):
+    """An empty string is a real pattern, distinct from absent ``None``."""
     with conn.cursor() as cur:
         cur.schemas(catalog_name="main", schema_name="")
-        rows = cur.fetchall()
-        assert len(rows) > 0
+        assert cur.fetchall() == []
+
+
+@pytest.mark.parametrize(
+    "empty_filter", ["schema_name", "table_name", "column_name"]
+)
+def test_columns_with_empty_string_filter_matches_nothing(conn, empty_filter):
+    filters = {
+        "catalog_name": "system",
+        "schema_name": "information_schema",
+        "table_name": "tables",
+        "column_name": "table_catalog",
+    }
+    filters[empty_filter] = ""
+
+    with conn.cursor() as cur:
+        cur.columns(**filters)
+        assert cur.fetchall() == []
 
 
 def test_tables_table_types_filter_is_case_insensitive(conn):

tests/unit/test_kernel_client.py

@@ -1786,15 +1786,12 @@ def test_sync_execute_leaves_rowcount_default_when_num_modified_rows_none():
 
 
 # ---------------------------------------------------------------------------
-# Metadata filter normalization — wildcard catalog + empty-string patterns
+# Metadata filter semantics
 # ---------------------------------------------------------------------------
 
 
-@pytest.mark.parametrize("wildcard", ["%", "*", "", "   "])
-def test_get_columns_normalizes_wildcard_catalog_to_none(wildcard):
-    """``catalog_name`` of ``%``/``*``/blank → ``None`` (all-catalogs),
-    matching JDBC exact-or-all semantics and keeping the three metadata
-    methods symmetric."""
+@pytest.mark.parametrize("catalog_wildcard", ["%", "*"])
+def test_get_columns_normalizes_all_catalog_wildcard(catalog_wildcard):
     c = _make_client()
     c._kernel_session = MagicMock()
     list_columns = c._kernel_session.metadata.return_value.list_columns
@@ -1808,7 +1805,7 @@ def test_get_columns_normalizes_wildcard_catalog_to_none(wildcard):
         max_rows=1,
         max_bytes=1,
         cursor=cursor,
-        catalog_name=wildcard,
+        catalog_name=catalog_wildcard,
         schema_name="s",
         table_name="t",
         column_name="c",
@@ -1822,10 +1819,8 @@ def test_get_columns_normalizes_wildcard_catalog_to_none(wildcard):
     )
 
 
-def test_get_schemas_normalizes_blank_pattern_to_none():
-    """An empty-string schema pattern → ``None`` (match-all), mapping
-    the kernel's ``InvalidArgument``-on-``""`` to Thrift's effective
-    match-all. ``%``/``*`` stay as real LIKE wildcards on patterns."""
+def test_get_schemas_preserves_empty_pattern():
+    """An empty pattern is distinct from the absent ``None`` filter."""
     c = _make_client()
     c._kernel_session = MagicMock()
     list_schemas = c._kernel_session.metadata.return_value.list_schemas
@@ -1843,7 +1838,139 @@ def test_get_schemas_normalizes_blank_pattern_to_none():
         schema_name="",
     )
 
-    list_schemas.assert_called_once_with(catalog="main", schema_pattern=None)
+    list_schemas.assert_called_once_with(catalog="main", schema_pattern="")
+
+
+def test_get_schemas_preserves_empty_catalog():
+    c = _make_client()
+    c._kernel_session = MagicMock()
+    list_schemas = c._kernel_session.metadata.return_value.list_schemas
+    list_schemas.return_value = _stream_with_schema()
+    cursor = MagicMock()
+    cursor.arraysize = 100
+    cursor.buffer_size_bytes = 1024
+
+    c.get_schemas(
+        session_id=MagicMock(),
+        max_rows=1,
+        max_bytes=1,
+        cursor=cursor,
+        catalog_name="",
+        schema_name="ignored",
+    )
+
+    list_schemas.assert_called_once_with(catalog="", schema_pattern="ignored")
+
+
+def test_get_tables_preserves_empty_patterns():
+    c = _make_client()
+    c._kernel_session = MagicMock()
+    list_tables = c._kernel_session.metadata.return_value.list_tables
+    list_tables.return_value = _stream_with_schema()
+    cursor = MagicMock()
+    cursor.arraysize = 100
+    cursor.buffer_size_bytes = 1024
+
+    c.get_tables(
+        session_id=MagicMock(),
+        max_rows=1,
+        max_bytes=1,
+        cursor=cursor,
+        catalog_name="",
+        schema_name="",
+        table_name="",
+    )
+
+    list_tables.assert_called_once_with(
+        catalog="",
+        schema_pattern="",
+        table_pattern="",
+        table_types=None,
+    )
+
+
+def test_get_columns_preserves_empty_patterns():
+    c = _make_client()
+    c._kernel_session = MagicMock()
+    list_columns = c._kernel_session.metadata.return_value.list_columns
+    list_columns.return_value = _stream_with_schema()
+    cursor = MagicMock()
+    cursor.arraysize = 100
+    cursor.buffer_size_bytes = 1024
+
+    c.get_columns(
+        session_id=MagicMock(),
+        max_rows=1,
+        max_bytes=1,
+        cursor=cursor,
+        catalog_name="main",
+        schema_name="",
+        table_name="",
+        column_name="",
+    )
+
+    list_columns.assert_called_once_with(
+        catalog="main",
+        schema_pattern="",
+        table_pattern="",
+        column_pattern="",
+    )
+
+
+def test_get_columns_preserves_empty_catalog():
+    c = _make_client()
+    c._kernel_session = MagicMock()
+    list_columns = c._kernel_session.metadata.return_value.list_columns
+    list_columns.return_value = _stream_with_schema()
+    cursor = MagicMock()
+    cursor.arraysize = 100
+    cursor.buffer_size_bytes = 1024
+
+    c.get_columns(
+        session_id=MagicMock(),
+        max_rows=1,
+        max_bytes=1,
+        cursor=cursor,
+        catalog_name="",
+        schema_name="ignored",
+        table_name="table",
+        column_name="column",
+    )
+
+    list_columns.assert_called_once_with(
+        catalog="",
+        schema_pattern="ignored",
+        table_pattern="table",
+        column_pattern="column",
+    )
+
+
+def test_get_columns_preserves_whitespace_for_kernel_validation():
+    c = _make_client()
+    c._kernel_session = MagicMock()
+    list_columns = c._kernel_session.metadata.return_value.list_columns
+    list_columns.return_value = _stream_with_schema()
+    cursor = MagicMock()
+    cursor.arraysize = 100
+    cursor.buffer_size_bytes = 1024
+
+    c.get_columns(
+        session_id=MagicMock(),
+        max_rows=1,
+        max_bytes=1,
+        cursor=cursor,
+        catalog_name="   ",
+        schema_name="   ",
+        table_name="   ",
+        column_name="   ",
+    )
+
+    list_columns.assert_called_once_with(
+        catalog="   ",
+        schema_pattern="   ",
+        table_pattern="   ",
+        column_pattern="   ",
+    )
 
 
 def test_get_schemas_keeps_wildcard_pattern():