[Coly010]

Current Behavior

The cli suite has no eval for the resolve loop around the CLI's database checks: run supabase db lint / supabase db advisors against a local database, then fix what they report. That loop is what users asked for in FDBKIN-21143 and FDBKIN-3130, and what supabase/cli#3839 (db advisors shipped) and supabase/cli#5868 (--local missed the API-exposure lints; fixed in 2.110.0-beta.30) shaped.

Expected Behavior

Adds evals/cli/resolve-database-005-fix-lint-findings (CLI-2711). The prompt is "We're about to launch and I want the database in good shape first. Can you run Supabase's database checks on our local database and fix whatever they flag?" It names no command, finding, object or file (fixtures.test.ts guards the word list).

The seeded orders app (local/) has three migrations that apply cleanly but plant exactly five findings, plus a seed with two users' orders and src/orders.ts as the app contract (unit_price, rpc('order_total', { p_order_id })):

• db lint: public.order_total(bigint) still reads oi.price after a later migration renamed it to unit_price (plpgsql_check error).
• db advisors: rls_disabled_in_public and policy_exists_rls_disabled on public.order_items, auth_rls_initplan on the orders policy, function_search_path_mutable on the updated_at trigger function.

No stack is running at the start (projectRunning: false, needsDocker: false).

#	Check	Proves
1	local stack is running for this project	There is a local database to check (shared lib stack resolution + select 1)
2	ran supabase db lint or db advisors	The agent found the CLI's own checks instead of guessing
3	db lint reports no errors or warnings in the project functions	The plpgsql error is fixed (same plpgsql_check predicate and level filter as db lint)
4	advisors no longer flag the seeded findings	The four advisor findings are gone (splinter predicates for the four seeded lint types)
5	the app still reads only its own orders and totals	Fixes kept the app working without widening access: each seeded user's orders, unit_price items and order_total totals as authenticated; order_total returns nothing for orders the caller can't read; anon reads nothing; the trigger still stamps updated_at
6	every migration file is recorded in the local migration history	Migration files on disk were applied through the CLI's history
7	replaying the migrations and seed reproduces the fixed database	The fixes live in migrations: public objects are dropped and the on-disk migrations + seed replayed in a rolled-back transaction, then 3–5 rerun
8	no container-runtime detours	Shared detour judge over executed commands
9	metrics	Diagnostics only (CLI versions, lint/advisors/reset runs, new and edited migrations, probe counts); always passes
10	final report matches what the checks and database show	The agent's summary is honest, judged against the probe's ground truth

Notes

• The scorer never runs the CLI itself (2.x target flags differ from 3.0's --remote); it mirrors db lint's plpgsql_check call and the splinter predicates in probe.sql, all in one psql session inside begin … rollback, so nothing it does persists.
• Valid alternatives pass: order_total rewritten as language sql, search_path = pg_catalog, alter policy, also revoking anon's grants, editing the seeded migrations then db reset, a migration wrapped in begin; … commit;, a trigger stamping clock_timestamp().
• Cheats fail: dropping order_total or set_updated_at, using (true), revoking authenticated's grants, renaming unit_price back, a security definer order_total that returns other users' totals, fixing only via psql / db query (replay fails), applying a migration by hand (history fails).
• db diff -f without a later db reset or migration repair fails the history check by design: the file isn't recorded, so the next migration up would re-run it.
• nodaemon / absent are red by design at the stack check: supabase start defaults to the Docker backend (CLI-2534), as for build-database-005.
• Known limitations (README): other advisor lint types a fix might introduce aren't scored; Supabase-owned schemas are ignored; the replay keeps live extensions, so it can disagree with db reset for hand-installed extensions or non-idempotent create extension; history compares versions only.

Verified locally

• Scorer unit tests: pnpm --filter @supabase-evals/framework exec vitest run --config vitest.cli-lib.config.ts evals/cli/resolve-database-005-fix-lint-findings (9 files, incl. EVAL.test.ts wiring).
• pnpm --silent eval -- list --experiment-suite cli --eval resolve-database-005-fix-lint-findings → pinned, stable, beta, next, nodaemon, absent.
• The real probe script (buildProbeScript → psql → parseProbeOutput → checks) run against supabase/postgres:17.11.0.004 for 16 migration variants: the seeded state fails 3/4/5/7 with the five findings named; the correct fix and each valid alternative pass; each cheat fails with a specific note.
• pnpm format:check && pnpm typecheck && pnpm test (test:framework stops after its 8th smoke check without a judge key; test:cli-lib run separately, green).

Results

Latest refresh (run 39946200701, after the failed-runs fix, 3 runs per environment):

Environment	CLI	Passed	Failures
pinned	2.117.0	1/3	agent stopped to ask before starting a stack (two runs)
stable	2.122.0	2/3	agent ran only db advisors, never db lint, so it missed the order_total error
beta	2.122.0-beta.15	3/3
next	3.0.0-next.23	0/3	agent stopped to ask before starting a stack (two runs); advisors-only run as on stable (one run)
nodaemon	2.122.0-beta.15	0/3	by design: no Docker daemon, so no stack starts (CLI-2534)
absent	2.122.0-beta.15	0/3	by design: no container runtime

Every Docker-arm failure is an agent gap I've seen in earlier refreshes, and the report judge passed each truthful report. Pass rates on pinned and next move between refreshes with how often the agent asks before running supabase start: the previous two refreshes (run 39941047477, run 39785046175) passed pinned and next 3/3.

All 14 db lint / db advisors runs that exited non-zero in this refresh were DbConnectErrors before a stack was up (nodaemon, absent, and pinned run 2 before it started the stack). None were counted, and the check's notes list them as failed runs. No --fail-on run found anything and exited non-zero this time; that case is covered by unit tests built from earlier refreshes' output.

I read every failing transcript and one passing transcript per environment across the refreshes. The passing runs ran db lint and db advisors against the local stack (or found the order_total bug through their own RLS smoke test), fixed all five findings in a new migration, applied it with migration up, and re-checked. No false passes.

Scorer fixes made along the way, each with regression tests:

• the report judge failed truthful "I wrote a migration but couldn't apply or verify it" reports in the no-Docker environments. A file written by a patch tool shows only its path in the transcript, and the rubric wanted a successful replay for any migration claim. The ground truth now carries the SQL of every new or edited migration, and only "applied/verified" claims need the replay;
• the report judge also failed reports that never mentioned a lint finding the agent never saw because it never ran db lint (stable, first refresh: an agent gap the lint, behaviour and replay checks already score). The rubric now holds the report to findings from checks the agent ran, and still fails a report that calls an unrun check clean;
• ran supabase db lint or db advisors counted help runs (db lint help) and remote targets (--linked, --project-ref, --remote, a non-local --db-url). It now counts only runs against the local stack, notes any remote runs, and the reset / migration up / db query metrics use the same filter;
• it also counted runs that failed before checking anything (a DbConnectError, a usage dump for a bad flag). A run now counts only when its output shows check results ("results", No issues found, No schema errors found, or lint's Linting schema:), so a --fail-on run that exits non-zero because it found something still counts. The reset / migration up / db query metrics stay attempt counts.

Braintrust (latest): pinned · stable · beta · next · nodaemon · absent

Related Issue(s)

Closes CLI-2711

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Oct 23, 2026 11:40am UTC

[avallete]

Cross-review (Codex) finding — SIGNIFICANT, confirmed:

evals/cli/resolve-database-005-fix-lint-findings/commands.ts:17 — countDatabaseCheckRuns counts any db lint / db advisors invocation by subcommand only. supabase db lint --help or a --linked (remote) run both count, so "ran supabase db lint or db advisors" can pass without the local database ever being checked. Suggest excluding help invocations and remote-targeted flags.

[Coly010]

thanks Andrew, you were right on this one - help and remote runs were both slipping through. Fixed in 4334af5:

• ran supabase db lint or db advisors now only counts runs against the local database. --linked, --project-ref, --remote and any --db-url that isn't the local stack are skipped, and show up as remote runs in the notes
• help runs (--help, -h, help) never count
• the reset / migration up / db query metrics had the same gap, so they use the same filter now, with regression tests for each

the latest refresh shows it working - runs that only ran db lint --help now fail that check

[avallete]

Thanks, the help/remote fix looks good. Re-review (Codex) raised one minor edge case, non-blocking — address here or in a follow-up as you prefer:

• evals/cli/resolve-database-005-fix-lint-findings/commands.ts:78 — a db lint / db advisors invocation that failed (non-zero exit, no findings printed) still counts toward "ran supabase db lint or db advisors".