Add ad-hoc agent/MCP query demo and deployable MCP server

demos/agent_mcp_app/.gitignore

@@ -0,0 +1,9 @@
+
+.databricks
+
+# Synced/built by build_wheel.sh from the repo's own src/ -- not checked in,
+# since they'd otherwise drift from the repo's actual source. Run
+# build_wheel.sh before deploying (see README.md).
+impulse_reporting/
+impulse_query_engine/
+wheels/*.whl

demos/agent_mcp_app/README.md

@@ -0,0 +1,85 @@
+# mcp-impulse-agent
+
+A custom MCP server, hosted as a Databricks App, exposing ad-hoc Impulse
+queries as agent tools: `list_channels`, `list_containers`,
+`preview_histogram`, `preview_histogram_2d`, `preview_stats`, and
+`preview_point_values`. See `demos/agent_mcp_query` for the notebook this
+was built from, including a full write-up of the design decisions (why not
+Genie/managed MCP, the safety model for virtual-signal expression trees, and
+the latency work that got steady-state calls down to 2-7s).
+
+## Prerequisites
+
+- A Databricks workspace with Unity Catalog and serverless compute enabled.
+- An already-loaded Impulse silver layer to point this at.
+- `databricks` CLI authenticated to your workspace.
+
+## Setup
+
+1. **Build the wheel** (ships this repo's `impulse_reporting`/
+   `impulse_query_engine` to the remote serverless workers TSAL compiles
+   Python UDFs onto, and to the app's own process):
+   ```bash
+   ./build_wheel.sh
+   ```
+   Re-run this whenever `src/` changes -- it always resyncs from the
+   canonical source, so the bundled copies here never drift.
+
+2. **Configure `app.yaml`**: replace the `CATALOG`/`SCHEMA`/`TABLE_PREFIX`
+   placeholders with wherever your silver layer lives.
+
+3. **Create and deploy the app**:
+   ```bash
+   databricks apps create mcp-impulse-agent   # name must start with mcp-
+   databricks sync . /Workspace/Users/<you>/mcp-impulse-agent
+   databricks apps deploy mcp-impulse-agent \
+     --source-code-path /Workspace/Users/<you>/mcp-impulse-agent
+   ```
+
+4. **Grant the app's service principal Unity Catalog access** (client ID
+   from `databricks apps get mcp-impulse-agent`):
+   ```sql
+   GRANT USE CATALOG, BROWSE ON CATALOG <catalog> TO `<service_principal_client_id>`;
+   GRANT USE SCHEMA, SELECT, MODIFY, CREATE TABLE ON SCHEMA <catalog>.<schema> TO `<service_principal_client_id>`;
+   ```
+
+5. **Test it**: the app is automatically discoverable as a custom MCP
+   server in AI Playground (Workspace sidebar → Playground → Tools
+   dropdown), since its name starts with `mcp-`.
+
+## Implementation notes
+
+- **Dependency pins matter.** `databricks-connect==18.2.*` is required --
+  `17.0.1` has a pyspark incompatibility and `18.3.1` is unsupported on
+  serverless compute. `databricks-sdk==0.106.0` must match Impulse's own
+  pin, since its telemetry code reads a private `Config._product_info`
+  attribute that only exists on that version. Don't pin `pyspark`/
+  `delta-spark` separately -- let `databricks-connect` provide them.
+- **Why the wheel exists at all:** TSAL compiles to Python UDFs that run on
+  remote serverless workers, not in the app's own process, so
+  `impulse_reporting`/`impulse_query_engine` have to be shipped there
+  explicitly via `DatabricksEnv().withDependencies("local:<wheel>")` --
+  see `server/main.py`'s `get_spark()`.
+- **Latency:** AI Playground enforces a ~55s per-call timeout. Persisting
+  results to Delta and reading them back took 33-43s per call -- too slow.
+  Reading `report.aggregation_dfs[...]` directly (already in final fact
+  schema, no join needed) plus going sinkless (`unity_sink` is `Optional`
+  on Impulse's `Report` config) cut steady-state latency to 2-7s.
+
+## Known limitations
+
+- **No per-user (on-behalf-of-user) authorization.** All callers share this
+  app's single service principal's Unity Catalog permissions. On-behalf-of-
+  user auth was attempted and hit a platform inconsistency between the
+  Databricks Apps `user_api_scopes` API and the Databricks Connect runtime's
+  actual scope requirements -- parked pending further investigation.
+- **Distance/custom-weighted histograms are not exposed.** Impulse supports
+  weighting histograms by distance or an arbitrary signal
+  (`HistogramDistance`/`HistogramCustomWeights`), not just duration.
+  Verification found these produce numerically incorrect results (off by
+  several orders of magnitude) when the weight signal is derived via
+  `resample()`+`cumtrapz()` -- traced to the `synchronized()`+`diff()`
+  interaction inside `HistogramCustomWeights.build()` in the query engine,
+  not something fixable from this MCP layer. `preview_histogram`/
+  `preview_histogram_2d` raise a clear error if you ask for anything other
+  than duration weighting, rather than silently returning wrong numbers.

demos/agent_mcp_app/app.yaml

@@ -0,0 +1,12 @@
+command: ['uv', 'run', 'mcp-server']
+# CATALOG/SCHEMA/TABLE_PREFIX must point at an already-loaded Impulse silver
+# layer. Replace these placeholders before deploying, or override them via
+# the app's "Environment variables" configuration in the Databricks Apps UI
+# without editing this file.
+env:
+  - name: CATALOG
+    value: "<your-catalog>"
+  - name: SCHEMA
+    value: "<your-schema>"
+  - name: TABLE_PREFIX
+    value: "<your-table-prefix>"

demos/agent_mcp_app/build_wheel.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Syncs impulse_reporting/impulse_query_engine from this repo's src/ (so the
+# app's own process can import them -- Databricks Apps only deploy the
+# source-code-path directory itself, not sibling paths, so these can't just
+# be sys.path-referenced at runtime) and builds a wheel from the same source
+# into ./wheels/ (so the app can also ship this code to remote serverless
+# workers -- TSAL compiles to Python UDFs that run there, not in the app's
+# own process; see server/main.py's get_spark()).
+#
+# Run this once before `databricks sync` + `databricks apps deploy`, and
+# again whenever the repo's src/ changes -- it always overwrites the local
+# copies, so they can't silently drift from the canonical source. Assumes
+# this directory lives two levels under the repo root (e.g.
+# demos/agent_mcp_app/) -- adjust REPO_ROOT if you've moved it.
+set -euo pipefail
+cd "$(dirname "$0")"
+REPO_ROOT="../.."
+
+echo "Syncing impulse_reporting/impulse_query_engine from $REPO_ROOT/src..."
+rm -rf impulse_reporting impulse_query_engine
+cp -r "$REPO_ROOT/src/impulse_reporting" .
+cp -r "$REPO_ROOT/src/impulse_query_engine" .
+
+echo "Building wheel..."
+rm -f wheels/databricks_impulse-*.whl
+mkdir -p wheels
+python3 -m pip install -q build
+python3 -m build --wheel --outdir wheels "$REPO_ROOT"
+echo "Built: $(ls wheels/databricks_impulse-*.whl)"

demos/agent_mcp_app/pyproject.toml

@@ -0,0 +1,29 @@
+[project]
+name = "mcp-impulse-agent"
+version = "0.1.0"
+requires-python = "==3.12.*"
+dependencies = [
+    "mcp>=1.0.0",
+    "databricks-connect==18.2.*",
+    "databricks-sdk==0.106.0",
+    "lz4==4.4.5",
+    "nptyping==2.5.0",
+    "pandas==2.2.3",
+    "pyarrow==19.0.1",
+    "pydantic==2.11.7",
+    "scipy==1.15.1",
+    "zstandard>=0.25.0",
+    "uvicorn>=0.30.0",
+    "starlette>=0.37.0",
+]
+
+[project.scripts]
+mcp-server = "server.main:main"
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["server*", "impulse_reporting*", "impulse_query_engine*"]

demos/agent_mcp_app/requirements.txt

@@ -0,0 +1 @@
+uv