Commit afc8ad0
committed
chore(kpi): freeze the baseline — 52.6% P@1 macro, labels 12336a824b18
The comparison point every future ranker or content change is measured from. Frozen
from a clean tree at ranker 64c0d5f, content e02b754, so it is reproducible by anyone
who checks out those two commits and runs `npm run kpi`.
P@1 52.6% macro (95% CI [42.7, 62.5]) / 61.6% micro
MRR@target 64.5% / 72.5%
recall@6 85.9% / 90.4%
nDCG@10 63.8% / 74.3%
target never returned 3.8%, buried 11-50 3.8%, reference page in top 6 66.7%
by source natural 63.1%, seo 81.3%, question 43.5%, intent 100.0%, peer 71.4%
985 headline queries over 42 macro groups, with the 246 `seo` keywords measured and
stored but held out of the headline.
Read a future delta against this with `npm run kpi -- --compare
scripts/search-kpi/data/gold-baseline.json`, which reports McNemar on P@1 and a paired
bootstrap CI on MRR@target rather than two means — and which refuses outright if the
labels have moved, because a delta across two different label sets is not a result.1 parent e02b754 commit afc8ad0
1 file changed
Lines changed: 10 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
3 | 13 | | |
4 | 14 | | |
5 | 15 | | |
| |||
0 commit comments