From 3213b1046d9373d64c6c9793c59045c5cb8bc931 Mon Sep 17 00:00:00 2001 From: parth Date: Tue, 13 Oct 2026 23:46:16 +0530 Subject: [PATCH] Report the BIRD Mini-Dev results in the README and evaluation history The README's benchmark section now gives the two full Mini-Dev runs (258/500 baseline, 284/500 final, official EX), notes that BIRD's own script reports the same 56.80, how the changes were found and measured (from the baseline's misses, validated on dev, test run once), the EX on questions the hand review does not flag, and the current PostgreSQL entries of the Mini-Dev leaderboard for context. EVALUATION_PLAN.md records the test baseline, the final run and the partial Groq run in its history, and no longer says that BIRD results are not committed. --- EVALUATION_PLAN.md | 46 ++++++++++++++++++++++++++++++++++++++++++---- README.md | 22 ++++++++++++++++++++-- 2 files changed, 62 insertions(+), 6 deletions(-) diff --git a/EVALUATION_PLAN.md b/EVALUATION_PLAN.md index 8e11cca..e7d477c 100644 --- a/EVALUATION_PLAN.md +++ b/EVALUATION_PLAN.md @@ -150,7 +150,8 @@ all), with cryptic column names, dirty values and no column comments. Difficulty * Data: `python -m scripts.bird download` (pinned, SHA-256 verified) and `python -m scripts.bird load` (database `bird` next to the OWID one, one schema per BIRD database, SELECT granted to - sql_agent). See data/README.md. Nothing from BIRD is committed. + sql_agent). See data/README.md. BIRD's data is not committed; run results, which quote it, are + (CC BY-SA 4.0, listed in evaluation/results/BIRD_RUNS.md). * Ground truth: `python -m evaluation.bird build-expected` runs BIRD's SQL (the PostgreSQL version, Hugging Face revision f65faf4) on the read-only account. All 500 run and all 500 are scored: BIRD runs raise the agent's row cap to 50,000 (--max-rows; the app's MAX_ROWS is unchanged), and @@ -225,12 +226,50 @@ Evaluation History Record every run here with its report. Never add numbers that were not produced by a run. +2026-10-13 — BIRD Mini-Dev, COMPLETE, final test run (run 20261013-122627-bird-database) +Commit: a226871 (tag bird-minidev-final) | Dataset: BIRD Mini-Dev PostgreSQL f65faf4 | Model: +gpt-oss-120b (Cerebras) | scope: database | evidence: on | answers: template | 1 candidate | default +reasoning effort | max retries 2 | row cap 50,000 +Results: evaluation/results/20261013-122627-bird-database.jsonl +* Official EX 284/500 (56.8%); Soft F1 58.0%; simple 105/148, moderate 132/250, challenging 47/102; + SQL execution success 492/500; retries 27/500; no timeouts, no safety violations; 4,440 tokens + per question +* BIRD's own evaluation_ex.py, run on the exported answers: 56.80, the same per difficulty (steps + in evaluation/results/BIRD_RUNS.md); 56.60 with the July 2025 package's older ground-truth file, + which still has BIRD's earlier SQL for B0879 +* Against the baseline below: 48 fixed, 22 broken. Hand review (35 questions): EX on the 465 + unflagged questions 282/465 (60.6%) +* Changes since the baseline, each measured on the dev split first: schema retrieval uses the hint + and adds linking tables; an unknown plan intent no longer rejects a reply; joins inferred for + columns named like a declared foreign key; NUMERIC casts in divisions computed in floating point; + phone and e-mail columns visible to BIRD runs; answers that concatenate columns sent to repair. + They came from reading the baseline's 242 misses, so this run is not independent of the test + set; it was run once +* Dev runs in between, including what was reverted or not kept (a prompt rule for hint formulas, + reasoning effort high, deterministic SQL rewrites): evaluation/results/BIRD_RUNS.md + +2026-10-12 — BIRD Mini-Dev, COMPLETE, test baseline (run 20261012-094139-bird-database) +Commit: 6f85d14 | Dataset and settings as in the final run above +Results: evaluation/results/20261012-094139-bird-database.jsonl +* Official EX 258/500 (51.6%); Soft F1 52.0%; simple 96/148, moderate 124/250, challenging 38/102; + SQL execution success 464/500; retries 35/500; no timeouts, no safety violations; 4,216 tokens per + question +* Largest miss shapes: different values 101, different columns 29, value format 28, unanswerable 28. + All 242 misses were read; the fixes in the final run above came from them + +2026-10-05 to 2026-10-07 — BIRD Mini-Dev, PARTIAL (run 20261005-184558-bird-database) — not comparable +Commit: 6960b34 | Model: openai/gpt-oss-120b (Groq free tier) | otherwise as above +Results: evaluation/results/20261005-184558-bird-database.jsonl +* 87 of 500 scored (debit_card_specializing, student_club, thrombosis_prediction) before the run + stopped with its database container. Official EX 54/87; Soft F1 60.9%. Three databases: not a + result + 2026-10-05 — BIRD dev split, smoke run of 10 questions (run 20261005-160047-bird-dev-database) — a pipeline check, not a result Commit: 4808884 (working tree had an unrelated uncommitted change) | Dataset: BIRD dev 2025-11-06 3c11fb1 | Model: openai/gpt-oss-120b (Groq) | scope: database | evidence: on | answers: template | --sample 10 (9 simple, 1 moderate) | row cap 50,000 -Results: evaluation/results/20261005-160047-bird-dev-database.jsonl (local, not committed) +Results: evaluation/results/20261005-160047-bird-dev-database.jsonl * Purpose: the Phase 0 pipeline end to end with a real model: 10/10 scored, no provider failures, explanation, plan, checks and token counts recorded for every question * Official EX 6/10; Soft F1 72.9%; project scorer 10/10; no retries, no timeouts @@ -242,8 +281,7 @@ Results: evaluation/results/20261005-160047-bird-dev-database.jsonl (local, not 2026-10-05 — BIRD Mini-Dev, PARTIAL (run 20261005-103021-bird-database) — not comparable Commit: 2dbb325 (working tree had uncommitted changes) | Dataset: BIRD Mini-Dev PostgreSQL f65faf4 | Model: openai/gpt-oss-120b (Groq) | scope: database | evidence: on | answers: template | row cap 1,000 -Results: evaluation/results/20261005-103021-bird-database.jsonl (local; BIRD-derived results are -not committed until the licence question is decided) +Results: evaluation/results/20261005-103021-bird-database.jsonl * Planned: formula_1, california_schools, thrombosis_prediction (142 questions at the time; 13 of the 500 were excluded by the 1,000-row cap). Scored: 96 (46 formula_1, 50 thrombosis_prediction); 5 not run (HTTP 429), california_schools never reached: Groq's 200,000 tokens/day limit diff --git a/README.md b/README.md index c4a00f0..3670e41 100644 --- a/README.md +++ b/README.md @@ -197,8 +197,26 @@ takes [BIRD Mini-Dev](https://github.com/bird-bench/mini_dev) (500 questions wit over 11 unfamiliar databases, PostgreSQL version): `python -m evaluation.run --dataset bird`, after loading it as described in [data/README.md](data/README.md#benchmark-data-bird-mini-dev). BIRD runs are scored with BIRD's official execution accuracy and Soft F1 (reproduced exactly, checked against -BIRD's own scripts), with a hand review of questionable ground truth shown beside them. Results -are recorded in [EVALUATION_PLAN.md](EVALUATION_PLAN.md) only once a run exists. +BIRD's own scripts), with a hand review of questionable ground truth shown beside them. + +Full Mini-Dev runs (PostgreSQL, all 500 questions; gpt-oss-120b on Cerebras, BIRD's hints on, one +candidate, at most two repairs): + +| Run | Code | Official EX | Simple / moderate / challenging | +|---|---|---|---| +| Baseline, 2026-10-12 | 6f85d14 | 258/500 (51.6%) | 96/148 · 124/250 · 38/102 | +| Final, 2026-10-13 | a226871 | **284/500 (56.8%)** | 105/148 · 132/250 · 47/102 | + +BIRD's own `evaluation_ex.py` reports the same 56.80 for the final run (steps in +[evaluation/results/BIRD_RUNS.md](evaluation/results/BIRD_RUNS.md)). The changes between the two +runs came from reading the baseline's 242 misses; each was measured on BIRD's separate dev +questions, and the test set was run once more at the end, so the final number is validated on +dev but not independent of the test set. On the 465 questions the hand review does not flag, EX is +60.6%. For context, as of 2026-10-13 the PostgreSQL column of the +[official Mini-Dev leaderboard](https://bird-bench.github.io/) lists 65.80 (Ontology2SQL), 56.80 +(SmartQuery + GPT-5.1) and 50.80 (TA + GPT-4). Every run, with what was kept, reverted or not +kept, is in [BIRD_RUNS.md](evaluation/results/BIRD_RUNS.md) and +[EVALUATION_PLAN.md](EVALUATION_PLAN.md). ## Known limitations