feat: Genie Agent managed-memory agent panel + bootstrap runbook

.env.example

@@ -29,9 +29,23 @@ BETTER_AUTH_SECRET=your_better_auth_secret_here
 
 # Better Auth URL (base URL for auth endpoints)
 # Development: http://localhost:3000
-# Production: https://your-domain.com
+# Production: https://your-domain.com (must match the Databricks OAuth redirect URI)
 BETTER_AUTH_URL=http://localhost:3000
-NEXT_PUBLIC_BETTER_AUTH_URL=http://localhost:3000
+
+# NEXT_PUBLIC_BETTER_AUTH_URL — client-side auth base URL.
+#
+# ⚠️  DO NOT set this for Vercel preview deployments.
+#     Preview URLs are dynamic (e.g. https://my-app-<hash>-team.vercel.app) and
+#     this variable is baked in at build time. If it is set to a different origin
+#     than the page serving the request, the browser blocks auth API calls with a
+#     CORS error. Leave it unset: the auth client automatically uses
+#     window.location.origin, which is always correct for any deployment URL.
+#
+# Only set this if you have a stable, canonical URL (e.g. a custom domain
+# attached to every deployment) AND that URL is also registered as a
+# Databricks OAuth redirect URI.
+#
+# NEXT_PUBLIC_BETTER_AUTH_URL=https://your-domain.com
 
 # Encryption key for OAuth token storage (AES-256-GCM)
 # Must be a 32-byte hex-encoded string (64 characters)
@@ -54,6 +68,21 @@ DATABRICKS_APP_URL=https://your-code-editor-app.databricksapps.com
 # Databricks notebook editor app URL (Marimo)
 DATABRICKS_NOTEBOOK_APP_URL=https://your-notebook-editor-app.databricksapps.com
 
+# =============================================================================
+# Agent Panel Configuration (managed-memory OpenAI agent)
+# =============================================================================
+
+# Toggle the slide-out Agent panel in the SSO-SPN org view.
+# Set to "true" to show the panel; anything else hides it.
+NEXT_PUBLIC_AGENT_ENABLED=false
+
+# Deployed Databricks App URL for the managed-memory agent (Genie + memory).
+# The Vercel-native reverse proxy at /api/agent-proxy forwards to this app,
+# minting the current user's (or guest's) mapped SPN token as a bearer. No Go
+# proxy / Cloud Run is required for the agent. Build/deploy the app from the
+# vendor/app-templates submodule via scripts/assemble_agent.sh.
+DATABRICKS_AGENT_APP_URL=https://your-agent-app.databricksapps.com
+
 # =============================================================================
 # Embedded Dashboard Configuration
 # =============================================================================
@@ -85,7 +114,9 @@ SPN_AUTH_DATABRICKS_ACCOUNTS_URL=https://accounts.cloud.databricks.com
 # Databricks Workspace URL for SPN tokens
 SPN_AUTH_DATABRICKS_WORKSPACE_URL=https://your-workspace.cloud.databricks.com
 
-# Okta OAuth Configuration for SPN Auth
+# Okta OAuth Configuration for SPN Auth (optional — leave unset if not using Okta)
+# When all three are set, an Okta provider is registered for SPN identity mapping.
+# When any are absent, the provider is skipped (no build error).
 SPN_AUTH_OKTA_CLIENT_ID=your_okta_client_id
 SPN_AUTH_OKTA_CLIENT_SECRET=your_okta_client_secret
 SPN_AUTH_OKTA_ISSUER=https://your-tenant.okta.com/oauth2/default

.github/pull_request_template.md

@@ -0,0 +1,33 @@
+## What & why
+
+<!-- What changed, and the problem it solves. Link the issue: Closes #NNN -->
+
+## Bootstrap / runbook changes
+
+<!-- Delete this whole section if you did not touch BOOTSTRAP.md, README.md,
+     scripts/bootstrap.sh, or scripts/lib/**. -->
+
+GitHub Actions is currently disabled on this repo, so the invariant guard does **not** run
+automatically. Run it locally and paste the result:
+
+```bash
+bash scripts/check-runbook-invariants.sh
+```
+
+- [ ] Guard passes (or this PR does not touch the runbook/bootstrap)
+- [ ] pnpm is still installed via `npm install -g pnpm@<pinned>` — **not** `corepack enable`/`prepare` (ENV-0, #69)
+- [ ] Phase 0 of `BOOTSTRAP.md` still has runnable commands, not prose-only guidance
+- [ ] Corporate-network logic still lives only in `scripts/lib/corp-network.sh`, sourced by both consumers
+- [ ] `README.md` and `BOOTSTRAP.md` still agree on the pnpm install method
+
+> Why these exist: the corepack guard was deleted once and then silently reintroduced by a
+> docs-sync commit, breaking setup for everyone on a network that blocks public npm. See
+> `AGENTS.md` → "Bootstrap invariants".
+
+## Testing
+
+<!-- What you ran, and what you did NOT verify. Be explicit about the gaps. -->
+
+## Risk / rollback
+
+<!-- Blast radius, and how to revert if this misbehaves. -->

.github/workflows/phase8-origin-guard.yml

@@ -0,0 +1,21 @@
+name: Phase 8 origin guard
+
+on:
+  pull_request:
+    branches:
+      - genie-agent
+  push:
+    branches:
+      - genie-agent
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@v4
+      - name: Run hermetic Phase 8 origin tests
+        run: bash tests/test_phase8_origin.sh

.github/workflows/pypi-proxy-guard.yml

@@ -0,0 +1,21 @@
+name: PyPI proxy guard
+
+on:
+  pull_request:
+    branches:
+      - genie-agent
+  push:
+    branches:
+      - genie-agent
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@v4
+      - name: Run hermetic proxy guard tests
+        run: bash tests/test_pypi_proxy_guard.sh

.github/workflows/runbook-invariants.yml

@@ -0,0 +1,63 @@
+name: runbook invariants
+
+# Guards the ENV-0 regression: the runbook must not require corepack (blocked wherever
+# public npm is blocked), and the corporate-network bridges must stay reachable from
+# BOOTSTRAP.md rather than living only in bootstrap.sh.
+
+on:
+  pull_request:
+    paths:
+      - BOOTSTRAP.md
+      - README.md
+      - package.json
+      - scripts/**
+      - .github/workflows/runbook-invariants.yml
+  push:
+    branches: [genie-agent]
+  workflow_dispatch:
+
+jobs:
+  invariants:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install ripgrep
+        run: sudo apt-get update && sudo apt-get install -y ripgrep
+
+      - name: Check runbook invariants
+        run: bash scripts/check-runbook-invariants.sh
+
+      - name: Shell syntax check
+        run: |
+          bash -n scripts/bootstrap.sh
+          bash -n scripts/lib/corp-network.sh
+          bash -n scripts/check-runbook-invariants.sh
+
+      # BOOTSTRAP.md Phase 0a asks users to source the library from their own shell, and
+      # zsh is the macOS default. Verify it is genuinely portable, not just bash-valid.
+      - name: Install zsh
+        run: sudo apt-get install -y zsh
+
+      - name: Library must be sourceable from zsh
+        run: |
+          zsh -n scripts/lib/corp-network.sh
+          zsh -c "source scripts/lib/corp-network.sh; ok ok; note note; warn warn; fail fail"
+
+  # Behavioural counterpart to the static checks above. Adapted from the fresh-HOME test on
+  # fix/corepack-mkdir-home-bin (#67) and reframed to assert the outcome rather than the
+  # install mechanism. Runs on macOS because that is what the runbook targets.
+  # NOTE: GitHub runners have unrestricted npm access, so this covers fresh-HOME and the
+  # version pin — NOT the blocked-registry case, which needs a corporate-network VM.
+  phase1a-fresh-home:
+    runs-on: macos-15
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "22"
+      - name: Install ripgrep
+        run: brew install ripgrep
+      - name: Fresh-HOME Phase 1a regression
+        run: bash scripts/test-bootstrap-phase1a.sh

.gitignore

@@ -62,3 +62,14 @@ go/databricks-proxy
 go/proxy
 # Local skills and personal tooling
 .local/
+
+# Assembled agent (ephemeral, from vendor submodule + agent/ overlay)
+agent-build/
+
+# Python build artifacts
+__pycache__/
+*.pyc
+.env*
+
+# Bootstrap runner local state (secrets, resume markers)
+.firefly-bootstrap/

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "vendor/app-templates"]
+	path = vendor/app-templates
+	url = https://github.com/databricks/app-templates.git

.vercelignore

@@ -0,0 +1,5 @@
+# Agent build inputs/outputs are not part of the Next.js frontend build.
+# Leading slash anchors to repo root so we DON'T exclude src/components/agent/.
+/agent/
+/agent-build/
+/vendor/

AGENTS.md

@@ -16,4 +16,66 @@
 - Custom API attribution/user-agent logic appears in `databricks-apps/code-editor/app.py` via explicit `User-Agent` headers.
 - Pipeline Studio encodes Databricks SQL AI functions (`ai_query`, `ai_extract`), Delta Change Data Feed, and three-level UC table names in `src/lib/pipeline-to-sql.ts`.
 - Unity Catalog Volumes are first-class in the product via `src/lib/databricks-volumes-api.ts`, including `catalog.schema.volume` parsing and `/Volumes/...` path generation.
-- The repo exposes Databricks pipelines via REST API routes, but there is no obvious repository-local IaC or CI/CD automation checked in.
+- The repo exposes Databricks pipelines via REST API routes. The only checked-in CI is `.github/workflows/runbook-invariants.yml`; GitHub Actions is currently **disabled at the repository level**, so it does not run — the invariants below are enforced by convention and by running the guard script manually.
+- The managed-memory Agent panel embeds a Databricks App (Genie + memory) via a Vercel-native reverse proxy at `src/app/api/agent-proxy/[[...path]]/route.ts` (mints the user/guest SPN token, forwards HTTP + SSE to `DATABRICKS_AGENT_APP_URL`) — distinct from the Go proxy used by the code/notebook editors. UI: `src/components/agent/agent-panel.tsx` (gated by `NEXT_PUBLIC_AGENT_ENABLED`).
+- The agent app source is the `vendor/app-templates` git submodule (`agent-openai-agents-sdk` + `e2e-chatbot-app-next`) plus the local `agent/` overlay, merged by `scripts/assemble_agent.sh` into the gitignored `agent-build/`. Keep `vendor/**` pristine; put deltas in `agent/`.
+
+## Bootstrap invariants — do not regress (ENV-0, #69)
+
+Read this before editing `BOOTSTRAP.md`, `README.md`, or `scripts/bootstrap.sh`.
+Verify any change with `bash scripts/check-runbook-invariants.sh`.
+
+- **Never require `corepack enable` or `corepack prepare`.** corepack fetches its package
+  manager from `registry.npmjs.org` and ignores the configured npm registry, so it fails
+  wherever public npm is blocked or blackholed. Install pnpm with
+  `npm install -g pnpm@<pinned-version>`, which honors the user's own registry. The version
+  **must** be pinned: pnpm's `latest` dist-tag has shipped a 12.x alpha that ignores
+  `onlyBuiltDependencies` (`ERR_PNPM_IGNORED_BUILDS`).
+- **Phase 0 of `BOOTSTRAP.md` must contain runnable commands, not prose.** Corporate-network
+  handling lives in `scripts/lib/corp-network.sh` and is sourced by *both* `bootstrap.sh` and
+  the Phase 0a block in `BOOTSTRAP.md`. Describing a bridge without giving the reader a
+  command to run it is the exact defect that caused #69 — the runbook is followed
+  top-to-bottom by agents (that is what the README's "Open in Cursor" badge launches), so an
+  unexecuted prose bridge means the install runs unbridged.
+- **Never duplicate the bridge logic.** One implementation in `scripts/lib/corp-network.sh`;
+  both consumers source it. Two copies drift, and the drift is invisible until someone on a
+  restricted network runs the doc path.
+- **`README.md` and `BOOTSTRAP.md` must agree** on how pnpm is installed. They contradicted
+  each other for months (README said "not corepack" while the runbook required corepack).
+- **Never probe a registry with `npm ping`.** Corporate mirrors proxy package routes but not
+  service endpoints — `npm-proxy.dev.databricks.com` returns HTTP 404 for `/-/ping` while
+  serving `npm view pnpm@10.34.5 version` fine. Probe the real operation instead.
+- **Never work around a blocked registry** with `--registry https://registry.npmjs.org`, by
+  disabling TLS verification, or by editing `/etc/hosts`. Bridge the user's own configured
+  mirror; never hardcode a mirror URL.
+- **Every helper `BOOTSTRAP.md` calls must be defined in a `scripts/lib/*.sh` that the
+  runbook itself sources.** Not in `bootstrap.sh`, not left to the reader. The runbook
+  called `store_secret` / `read_secret` for ten days with no definition anywhere in the
+  file, while `bootstrap.sh`'s version took `(VARNAME, _, KEY)` and every runbook call site
+  used `$(read_secret KEY)` — so even copying it across did not work. A headless agent
+  invented its own, used bash-only `${!key}`, and got `bad substitution` under zsh.
+- **Runnable blocks must parse under bash *and* zsh.** zsh is the macOS default and the
+  reader pastes these into their own shell. No `${!var}`, no `declare -F` as a function
+  test, no `mapfile`/`readarray` (macOS also ships bash 3.2).
+- **A code block must do what it looks like it does.** `command -v X || { # install X }`
+  and `{ : ; }` shipped as the Phase 1b/1e installers: they read as installs and are
+  no-ops, which is invisible on any machine that already has the tool.
+- **Never parse structured output with `grep`.** The `sync.exclude` list opens with a
+  comment naming `pyproject.toml` and `vendor-wheels/`, so `grep -q pyproject.toml` reports
+  it excluded when it is not — `bootstrap.sh` failed a correct config that way, and an
+  agent's `-\s` regex read the same list as empty and passed. Use
+  `check_sync_exclude_rules`. Same rule for CLI JSON: use the shared parsers.
+- **Never derive an id from a `create` response you then act on.** `neonctl projects
+  create` succeeds server-side before any parse of its output can fail, so a parse bug
+  orphans a real project. Create, then look the id up by name; fail closed if it is empty.
+
+Historical note: this regressed once already. `e91322d` deleted the "do not use corepack"
+guard and its rationale, then `99f2cc2` — a docs-sync commit — reintroduced corepack as the
+first executable command in the runbook. Deleting the reason is what made reintroducing the
+bug look reasonable. Keep the rationale attached to the rule.
+
+The same `99f2cc2` is also the origin of three of the defects listed above: it back-ported
+`bootstrap.sh`'s *idempotency* fix for Neon while dropping the *parsing* fix from the same
+block, replaced self-contained `keyring` one-liners with calls to functions it did not
+carry across, and added the two comment-only installers. Blanket "sync the docs to the
+script" commits are how this class of bug arrives. Edit the shared library instead.