pi-based harness

.github/workflows/e2e.yml

@@ -43,7 +43,7 @@ jobs:
       # snapshot dir is owned by the runner user, so without this Metabase
       # cannot write the per-stack snapshot and bootstrap fails.
       - name: Make snapshot dir writable by Metabase container
-        run: chmod 777 tests/e2e/snapshots
+        run: chmod 777 packages/cli/tests/e2e/snapshots
 
       - name: Run e2e against ${{ matrix.stack }}
         run: bun run e2e:matrix --stack=${{ matrix.stack }}
@@ -54,5 +54,5 @@ jobs:
         uses: actions/upload-artifact@v4
         with:
           name: metabase-logs-${{ matrix.stack }}
-          path: metabase-${{ matrix.stack }}.log
+          path: packages/cli/metabase-${{ matrix.stack }}.log
           if-no-files-found: ignore

.github/workflows/lint.yml

@@ -42,5 +42,5 @@ jobs:
       - uses: actions/checkout@v4
       - uses: stbenjam/skillsaw@v0
         with:
-          path: skill-data
+          path: packages/cli/skill-data
           strict: true

.github/workflows/release-alpha.yml

@@ -35,7 +35,7 @@ jobs:
       - name: Compute alpha version and dist-tag
         id: version
         run: |
-          BASE=$(node -p "require('./package.json').version")
+          BASE=$(node -p "require('./packages/cli/package.json').version")
           BRANCH="${{ github.event.pull_request.head.ref }}"
           SLUG=$(echo "$BRANCH" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g; s/--*/-/g; s/^-//; s/-$//')
           SHORT_SHA=$(git rev-parse --short HEAD)
@@ -46,12 +46,14 @@ jobs:
           echo "Will publish @metabase/cli@$VERSION with dist-tag $TAG"
 
       - name: Set package version
+        working-directory: packages/cli
         run: npm version --no-git-tag-version --allow-same-version "${{ steps.version.outputs.version }}"
 
       - name: Build
         run: bun run build
 
       - name: Publish to NPM
+        working-directory: packages/cli
         run: npm publish --tag "${{ steps.version.outputs.tag }}" --access public
         env:
           NODE_AUTH_TOKEN: ${{ secrets.NPM_RELEASE_TOKEN }}

.github/workflows/release.yml

@@ -33,7 +33,7 @@ jobs:
       - name: Check whether local version was already published
         id: check
         run: |
-          LOCAL=$(node -p "require('./package.json').version")
+          LOCAL=$(node -p "require('./packages/cli/package.json').version")
           LATEST=$(npm view @metabase/cli version 2>/dev/null || echo "none")
           ALREADY_PUBLISHED=$(npm view @metabase/cli versions --json 2>/dev/null | node -e "
             let raw = '';
@@ -55,6 +55,7 @@ jobs:
 
       - name: Publish to NPM
         if: steps.check.outputs.changed == 'true'
+        working-directory: packages/cli
         run: npm publish --tag latest --access public
         env:
           NODE_AUTH_TOKEN: ${{ secrets.NPM_RELEASE_TOKEN }}

.gitignore

@@ -9,6 +9,6 @@ coverage/
 .tsdown/
 plan
 .dev-state/
-tests/e2e/.bootstrap*.json
-tests/e2e/snapshots/*.sql
+packages/cli/tests/e2e/.bootstrap*.json
+packages/cli/tests/e2e/snapshots/*.sql
 metabase-*.log

CLAUDE.md

@@ -2,8 +2,12 @@
 
 Metabase CLI. TypeScript ESM. citty + native `fetch` + Zod + @clack/prompts. oxlint + oxfmt. vitest. tsdown.
 
+Bun-workspaces monorepo: `packages/cli` is `@metabase/cli` (everything below), `packages/agent` is `@metabase/agent`. All paths in this file are relative to `packages/cli/` unless they start with a repo-root marker (`packages/`, `docs/`, `.github/`, `tsconfig.base.json`, `bun.lock`).
+
 ## Hard rules
 
+- **`mb` and `mb-agent` are two products, and neither knows the other exists.** They share code — `@metabase/agent` imports `@metabase/cli`'s library exports (the HTTP client, the config resolver, the error taxonomy) — and sharing is right whenever the thing shared is genuinely product-neutral. But no user-facing surface of one may name, describe, or hand the user the other. Concretely: the `mb` binary is never on the model's PATH and never appears in the agent's system prompt, its tool descriptions, or its skills; the agent's capability is exactly its curated tools, so a Metabase surface a tool doesn't reach is a gap to close with a tool, never a `bash` fallback. Skills are the sharp edge of this: a skill that teaches `mb card create --body …` teaches the agent a command it cannot run, so the two products own their skills separately (`packages/cli/skill-data/`, `packages/agent/skill-data/`) and the duplication is deliberate. The same JSON grammar appearing in both is fine; a shared file that speaks one product's surface to the other's model is not. Resolving the CLI's binary inside `mb-agent auth` (which spawns `mb auth` to run the login flow) is the one sanctioned exception — internal plumbing on a path the model never sees.
+- **No evals.** This repo has no eval suite, no scored task list, no model-vs-model baseline harness, and none is to be added. Correctness is held by the test tiers below — unit, e2e, and the agent's smokes. Don't propose an eval, don't scaffold one, and don't reintroduce `packages/agent/evals/`.
 - No `as` type casts: `as X`, `as unknown as X`, `as never`, `as any`. Use type guards in `core/errors.ts`, or Zod `.parse` at boundaries. `as const` is allowed (it's a const assertion, not a cast — it narrows literal types without bypassing checking).
 - No `any`. No `Record<string, unknown>` for API responses — every cross-network value gets a named Zod schema in `src/domain/` and is parsed at the boundary. The sole exemption is `src/output/projection.ts`, the canonical structurally-untyped envelope for dot-path `--fields a,b.c.d` projection over already-validated payloads.
 - No `!` non-null assertions. Restructure with helpers (see `core/config.ts` `pickField()` for the pattern).
@@ -26,12 +30,14 @@ Metabase CLI. TypeScript ESM. citty + native `fetch` + Zod + @clack/prompts. oxl
 
 ## Layout
 
+Repo root holds the workspace glue: a private `package.json` (`workspaces: ["packages/*"]`, scripts that delegate to `packages/cli` via `--cwd`), `tsconfig.base.json` (the shared strict compiler options; each package's `tsconfig.json` extends it), `bun.lock`, `CLAUDE.md`, `docs/`, `.github/`. `packages/agent/` is the pi-based agent harness (`mb-agent`); it consumes `@metabase/cli`'s library exports, reaches Metabase through its own curated tools (`src/tools/`), loads its own `packages/agent/skill-data/` as pi skills, and documents both in `packages/agent/docs/` (`toolset.md`, `prompt.md`). It is a separately published product that authenticates separately: the CLI's resolver, profile format and login flow, run against its own profile store (`MB_PROFILE_STORE=agent` → keychain service `metabase-agent`, `profiles.agent.json`), so no agent run acts under the credential a human logged `mb` in with; model-provider keys sit in pi's `AuthStorage` over an OS-keychain backend. `packages/agent/docs/auth.md` is the contract, and `packages/agent/docs/gaps.md` is the register of Metabase surfaces its tools do not yet reach — each one a tool to build. Everything below is inside `packages/cli/`.
+
 - `src/cli.ts` — `runMain` entry.
 - `src/main.ts` — root citty command, lazy `subCommands`.
 - `src/commands/` — CLI shell only. No HTTP, no parsing, no formatting.
 - `src/core/` — pure logic, no CLI deps.
-  - `auth/` — credential storage + verify, plus the OAuth login flow: `credential.ts` (discriminated `Credential` union), `pkce.ts`, `callback-server.ts` (loopback redirect), `oauth-login.ts` (orchestration), `oauth-session.ts` (refresh/revoke).
-  - `env.ts` — `readEnv(canonical)` resolves a CLI env var by its canonical `MB_` name, falling back to the deprecated `METABASE_` alias and recording the legacy hit so the command shell warns once per run (`consumeLegacyEnvWarnings`, flushed in `commands/runtime.ts`). Every CLI env-var name (`MB_URL`, `MB_API_KEY`, `MB_PROFILE`, `MB_VERBOSE`, `MB_CLI_SKIP_PREFLIGHT`, `MB_CLI_DISABLE_KEYRING`) is a const here; reads go through `readEnv`, never raw `process.env[...]`.
+  - `auth/` — credential storage + verify, plus the OAuth login flow: `credential.ts` (discriminated `Credential` union), `pkce.ts`, `callback-server.ts` (loopback redirect), `oauth-login.ts` (orchestration), `oauth-session.ts` (refresh/revoke). `store.ts` names the profile store the process runs under (`MB_PROFILE_STORE`, default `cli`): its keychain service (`metabase-<store>`) and its profiles file. Every secret `storage.ts` reads or writes belongs to that store, so a second product embedding this library (the agent) cannot read, refresh, overwrite or clear a credential a human logged `mb` in with — including when both stores hold a profile of the same name. Only the default store degrades to a plaintext `profiles.json` when the keychain is gone (a CLI must still authenticate on a vault-less box); every other store is keyring-only and a write that cannot reach the keychain throws `ConfigError` rather than putting a secret on disk.
+  - `env.ts` — `readEnv(canonical)` resolves a CLI env var by its canonical `MB_` name, falling back to the deprecated `METABASE_` alias and recording the legacy hit so the command shell warns once per run (`consumeLegacyEnvWarnings`, flushed in `commands/runtime.ts`). Every CLI env-var name (`MB_URL`, `MB_API_KEY`, `MB_PROFILE`, `MB_PROFILE_STORE`, `MB_VERBOSE`, `MB_CLI_SKIP_PREFLIGHT`, `MB_CLI_DISABLE_KEYRING`) is a const here; reads go through `readEnv`, never raw `process.env[...]`.
   - `config.ts` — flag → env → stored resolver. Profile-aware (`resolveProfileName`, `resolveConfig`). Credential/profile env reads (`MB_URL`/`MB_API_KEY`/`MB_PROFILE`) live here, via `core/env.ts`.
   - `errors.ts` — `isNotFoundError`, `errorMessage` (Node error type guards used outside the HTTP boundary).
   - `http/` — the HTTP boundary. `client.ts` wraps native `fetch` with `requestParsed(schema, path, opts)` (the ONLY typed-JSON path), `requestRaw`, `requestStream`. Retries are idempotency-aware: GET/HEAD/OPTIONS retry on retryable status codes by default; POST/PUT/PATCH/DELETE never retry on status (only on network/timeout). Callers may override via `RequestOptions.idempotent`. `errors.ts` owns the discriminated `MetabaseError` taxonomy and `toMetabaseError(unknown)`. `sanitize.ts` runs at `HttpError` construction — secret redaction is not optional. `retry.ts` is the backoff math; it is also the only `core/http/` site allowed to drive a `setTimeout`-based wait loop (via `node:timers/promises`) outside `src/runtime/poll.ts`. `oauth.ts` is the OAuth protocol boundary (RFC 8414 discovery with same-origin endpoint pinning, dynamic client registration, token exchange/refresh/revocation); its schemas are protocol envelopes, not `src/domain/` resources. Nothing outside this directory may import a third-party HTTP library or call `fetch` directly; this is enforced by `tests/structure.test.ts`.
@@ -112,7 +118,7 @@ Local prerequisites for e2e: `bun run e2e:up && bun run e2e:bootstrap` (~1 minut
 Lives under `tests/e2e/`. The whole point is to run the **built `dist/cli.mjs`** against a real Metabase via docker compose, with no mocks.
 
 - `tests/e2e/run-cli.ts` — `runCli({ args, configHome, env, stdin, timeoutMs })` is the ONLY way an e2e test invokes the CLI. It spawns `node dist/cli.mjs` via `execa`, with an isolated `XDG_CONFIG_HOME` (per-call temp dir by default), `MB_CLI_DISABLE_KEYRING=1`, and stripped env (no inherited `MB_*`/`METABASE_*` from the developer's shell). Tests never call `execa`/`child_process` directly; never call `fetch` against Metabase (that's bootstrap's job).
-- `tests/e2e/bootstrap-data.ts` — sole owner of the `Bootstrap` Zod schema and the stack-scoped `BOOTSTRAP_FILE_PATH` (`.bootstrap.<stack>.json`). The schema carries `seeded` (entity ids the bootstrap **discovers** — warehouse db/collection/card/dashboard/dashcard plus warehouse table & field ids resolved by name, never pinned) and `server` (the probed `{ version, tokenFeatures }`). The writer (`tests/e2e/setup/bootstrap.ts`) imports this schema; do not redeclare it. Tests read admin creds via `readBootstrap()` (async); they read seeded entity ids via the `SEEDED` const in `tests/e2e/seed/seeded.ts` (a sync `seededIds()` read, mirroring Metabase's own `cypress_sample_instance_data` pattern) — never hard-code an entity id, never invoke the setup wizard themselves, never hard-code an API key.
+- `tests/e2e/bootstrap-data.ts` — sole owner of the `Bootstrap` Zod schema and the stack-scoped `BOOTSTRAP_FILE_PATH` (`.bootstrap.<stack>.json`). The schema carries `seeded` (entity ids the bootstrap **discovers** — the warehouse db, the `E2E Default` and `Finance` collections with their cards, dashboards and dashcards, plus warehouse table & field ids resolved by name, never pinned) and `server` (the probed `{ version, tokenFeatures }`). The writer (`tests/e2e/setup/bootstrap.ts`) imports this schema; do not redeclare it. Tests read admin creds via `readBootstrap()` (async); they read seeded entity ids via the `SEEDED` const in `tests/e2e/seed/seeded.ts` (a sync `seededIds()` read, mirroring Metabase's own `cypress_sample_instance_data` pattern) — never hard-code an entity id, never invoke the setup wizard themselves, never hard-code an API key.
 - `tests/e2e/setup/bootstrap.ts` — standalone script invoked by `bun run e2e:bootstrap` and by `tests/e2e/setup/global-setup.ts`. Idempotent: reuses `.bootstrap.<stack>.json` when the stored key still authenticates, otherwise calls `/api/setup` (or logs in directly if already setup), mints a fresh admin API key, discovers seeded ids, and probes the server. The Metabase HTTP responses it parses are setup-only — their schemas live colocated here, not in `src/domain/`.
 - `tests/e2e/setup/global-setup.ts` — vitest globalSetup. Verifies `dist/cli.mjs` exists, then spawns `bootstrap.ts` once per `bun run test:e2e`.
 - `tests/e2e/defaults.ts` — sole owner of `DEFAULT_E2E_BASE_URL`/`resolveE2EBaseUrl()` (reads `METABASE_CLI_E2E_URL`), `DEFAULT_E2E_STACK`/`resolveStackId()` (reads `METABASE_CLI_E2E_STACK`, default `default`), and `resolveSnapshotName()` (`cli_<stack>`). Anything needing a base URL, stack id, or snapshot name imports from here.
@@ -148,6 +154,6 @@ Running e2e — the suite is slow (~3–5 minutes for a full run, ~hundreds of m
 
 ## Commands
 
-Install: `bun install` (npm 7 too old). Typecheck `npx tsc --noEmit`. Lint `npx oxlint`. Format `npx oxfmt` (`--check` to verify). Build `npx tsdown`.
+Install: `bun install` from the repo root (npm 7 too old). Run everything else from `packages/cli`, or from the root via the delegating scripts (same names). Typecheck `npx tsc --noEmit`. Lint `npx oxlint`. Format `npx oxfmt` (`--check` to verify). Build `npx tsdown`.
 
 Tests: `bun run test` (unit), `bun run test:e2e` (e2e — requires `bun run build && bun run e2e:up && bun run e2e:bootstrap` first). Iteration: `bun run test:watch` for unit, `bun run test:e2e:watch` for e2e (Metabase stays up, restart not needed between runs). Stack control: `bun run e2e:up`, `bun run e2e:down`, `bun run e2e:logs`.