feat(kpi): significance testing, a holdout half, and nDCG@10

The harness was being asked to resolve differences it could not. Decisions have
rested on natural-macro moves of 0.1–0.5 points, and its own README records a
change that moved the mean by +0.0 while 260 queries churned. "19 better / 7
worse" is a better signal than the mean and still not an answer.

lib/stats.js reports the same per-query deltas three ways: a seeded paired
bootstrap 95% CI, a Wilcoxon signed-rank p-value (non-parametric because the
per-query metric is discrete, bounded and mostly exactly zero — where a t-test
misbehaves), and the same test over TOPIC MEANS, because the headline is a macro
average and a claim about a macro has to be tested over topics. Paired
throughout: both rankers see the same queries, so the variance that matters is
the variance of the differences.

It also calibrates the evidence already in use. 19 better / 7 worse of 2,281 at
equal magnitudes is p = 0.019 — significant, so the churn count was worth more
than it looked. 12 better / 10 worse is p = 0.68 and means nothing.

lib/split.js cuts each set in two and both halves print by default, because
every constant in the ranker was chosen by sweeping it against these sets and a
training score that reads as a measurement is the failure this prevents. The cut
is BY TOPIC, never by query: the natural harvest expands each seed a–z, so
`imqueue rpc` and `imqueue rpc example` are near-twins on the same page, and
splitting by query would put twins on both sides and make the holdout agree with
the fit by construction. Artificial is cut by TARGET PAGE — its `bucket` is a
query SHAPE, and cutting on shape produced halves made of different populations
and read the difference as a 5.6-point fitting gap on a ranker nobody had tuned
against that set at all.

The result is good news that was not guaranteed:

  natural      fit 87.8%  holdout 90.3%   (-2.5, holdout BETTER)
  artificial   fit 98.5%  holdout 98.5%   (+0.0)
  question     fit 62.4%  holdout 60.6%   (+1.8, inside the noise of 56 queries)

No detectable overfitting after twenty rounds of hand-tuning.

nDCG@10 is reported beside the linear metric rather than replacing it. The -10
per position rule says something true — nobody scrolls to row eleven — but it
overstates #5 against #7 and understates #1 against #2; the two together say
more than either. One relevant document per query, because `expect` lists
ALTERNATIVES, and textbook DCG summing over them would reward returning three
spellings of the same answer. `grades` is honoured where a query carries one, so
a graded label set needs no further harness change.

compare.js gains the question set. It measured two of the three populations, and
the missing one is the only set that has ever caught a regression the other two
could not see.

check-search-ranking.js gains three cases for the ranker's new relaxation pass,
including the gate itself — a query that returns results must never be rewritten.
The safety argument for spelling correction is worth exactly as much as that
assertion.

Pins search-ranker 6fc2ca8.

Author: Mikhus

scripts/check-search-ranking.js

@@ -741,6 +741,64 @@ if (!df || !ranker.state.t2.docs) {
   }
 }
 
+// ---- relaxation fires ONLY on an empty result set ------------------------------
+// The whole safety argument for spelling correction is the gate: a query that returns
+// something is scored by identical code, so the feature cannot move the KPI. That argument is
+// worth exactly as much as this assertion — remove the gate and the ranker starts silently
+// answering questions nobody asked, with no test objecting.
+//
+// The ordering case is here too. `nestjs.microservices cqrs` is answered by splitting the
+// compound alone, and a single combined relaxation pass also "corrected" `cqrs` to `cars`
+// (one substitution, and `cars` is in five sections) — announcing a query about CQRS as a
+// query about cars. Confident rewrites have to be tried before guesses.
+{
+  const RELAXED = [
+    {
+      query: 'nestjs.microservices cqrs',
+      corrected: 'nestjs microservices cqrs',
+      protects: 'a dotted compound is split, and the unknown word beside it is left alone',
+      regression: 'returned zero results; then returned results "for nestjs microservices cars"',
+    },
+    {
+      query: 'imqeueue',
+      corrected: 'imqueue',
+      protects: 'one transposed key is corrected against the corpus vocabulary',
+      regression: '29% of one-transposed-key queries returned an empty result set',
+    },
+    {
+      query: 'watcherChekcDelay',
+      corrected: null,
+      protects: 'a query that already returns something is never rewritten',
+      regression: 'n/a — this is the gate the safety argument rests on',
+    },
+  ];
+
+  for (const testCase of RELAXED) {
+    const q = ranker.parseQuery(testCase.query);
+    const hits = ranker.search(q);
+
+    if (!hits.length) {
+      fail(`"${testCase.query}" returns nothing — ${testCase.protects}`);
+      continue;
+    }
+    if (testCase.corrected === null) {
+      if (q.corrected) {
+        fail(`"${testCase.query}" was rewritten to "${q.corrected}" although it had results — `
+          + 'relaxation must fire only on an empty result set');
+      } else {
+        pass(`"${testCase.query}" is not rewritten — ${testCase.protects}`);
+      }
+      continue;
+    }
+    if (q.corrected !== testCase.corrected) {
+      fail(`"${testCase.query}" was rewritten to "${q.corrected}", expected `
+        + `"${testCase.corrected}" — ${testCase.protects}`);
+    } else {
+      pass(`"${testCase.query}" → "${q.corrected}" — ${testCase.protects}`);
+    }
+  }
+}
+
 if (failures) {
   console.error(`\n${failures} search ranking check(s) failed.`);
   process.exit(1);

scripts/search-kpi/README.md

@@ -4,8 +4,9 @@ A number that says whether a change to the ranker made search better or worse, m
 before we have any query logs of our own.
 
 ```bash
-npm run kpi:search          # the numbers
+npm run kpi:search          # the numbers — all three sets
 npm run kpi:search:worst    # plus the 40 worst misses, with what was expected
+npm run kpi:compare         # this working tree's ranker vs the pinned one, with significance
 ```
 
 Both query sets are committed, so the measurement is reproducible and a change in the number
@@ -21,6 +22,16 @@ npm run kpi:search:gen      # regenerate the 10,000 artificial queries from curr
 Position 1 scores 100%, and every position below it costs 10 points, so position 11 and
 "not returned at all" both score 0. Nobody scrolls to the eleventh row of a site search.
 
+**nDCG@10 is reported next to it**, and is the standard one. The linear −10 metric above is this
+project's own and it overstates the difference between #5 and #7 while understating #1 against #2;
+nDCG's log discount is closer to how clicks actually fall off. Both are printed because they
+disagree in a useful direction: a change that lifts a query from #4 to #2 is +20 of 100 in accuracy
+terms and a large nDCG move.
+
+One relevant document per query, deliberately — `expect` lists *alternatives*, so the ideal ranking
+puts one of them first, not all of them. Summing gains over the alternatives (textbook DCG) would
+reward a ranker for returning three spellings of the same answer.
+
 Position is read from the flat merged list — what `/search/` renders as "Everything". The
 dialog also splits results into Answers/Docs/API groups, so a hit at flat position 4 can be
 the first row of its own group there. Flat position is the pessimistic reading and the one
@@ -88,11 +99,58 @@ edition, which is the right shape for that risk rather than an average.
 
 | | natural | artificial | question |
 |---|---|---|---|
-| micro | **94.0%** | 89.9% | 64.1% |
-| macro | **88.9%** | 94.5% | **61.1%** |
+| micro | **94.3%** | 90.8% | 64.5% |
+| macro | **89.1%** | 95.3% | **61.5%** |
+| nDCG@10 | 90.7% | 86.6% | 60.1% |
 | recall@6 | — | — | 66.1% (micro) / 62.9% (macro) |
-| never found | — | — | 19.1% |
-| typos (reported apart) | — | 36.4% | — |
+| never found | — | — | 18.3% |
+| typos (reported apart) | — | **55.2%** | — |
+
+## Is that delta real?
+
+A change of 0.1–0.5 macro points has been enough to keep or drop a ranker change here, and the
+section below records one that moved the mean by **+0.0** while 260 queries churned. So every
+comparison now reports the same per-query deltas three ways (`lib/stats.js`):
+
+- a **paired bootstrap 95% CI** on the mean delta, seeded so it is reproducible — if it straddles
+  zero the change is *unmeasured*, whatever the point estimate says;
+- a **Wilcoxon signed-rank** p-value, non-parametric because the per-query metric is discrete,
+  bounded and mostly exactly zero, which is where a t-test misbehaves;
+- the same test **over topic means**, because the headline is a macro average and a claim about a
+  macro has to be tested over topics rather than over queries.
+
+Paired is the load-bearing word: both rankers see the same queries, so the variance that matters is
+the variance of the *differences*. Comparing two independent CIs on the means would call almost
+everything a tie.
+
+Calibration worth keeping in mind: **19 better / 7 worse out of 2,281, at equal magnitudes, is
+p = 0.019** — significant. The churn count was a better signal than it looked. **12 better / 10
+worse is p = 0.68** and means nothing.
+
+## Fit and holdout
+
+Every constant in the ranker was chosen by sweeping it against these sets, which makes every number
+above a **training** score. Each set is therefore cut in two and both halves are printed by default
+(`lib/split.js`).
+
+The cut is **by topic, never by query**: the natural harvest expands each seed a–z, so
+`imqueue rpc` and `imqueue rpc example` are near-twins answered by the same page, and splitting by
+query would put twins on both sides and make the holdout agree with the fit by construction. The
+artificial set is cut by **target page** for the same reason — its `bucket` field is a query *shape*
+(title-salient, body-salient), and cutting on shape produced two halves made of different
+populations and read their difference as a 5.6-point fitting gap on a ranker never tuned against it.
+
+Measured, and it is good news that was not guaranteed:
+
+| set | fit | holdout | gap |
+|---|---|---|---|
+| natural (55 topics) | 87.8% | **90.3%** | −2.5 |
+| artificial (1,237 pages) | 98.5% | 98.5% | +0.0 |
+| question (18 topics) | 62.4% | 60.6% | +1.8 |
+
+Natural's holdout is *better* than its fit, and artificial's halves are identical. **There is no
+detectable overfitting** in twenty rounds of hand-tuning — the weights generalise across topics they
+were not fitted on. The question set's +1.8 is inside the noise of 56 queries.
 
 The question set's weakest topics, and they point the same way the diagnosis above does — every
 one of them is answered by an API symbol page, which has no question-shaped text to compete with:
@@ -195,6 +253,47 @@ additionally a family of ion-channel genes.
 - **Google autocomplete is not our traffic.** It is web-search intent, scored here as if it
   were site-search intent. Real logs will disagree.
 - **The natural set is skewed** toward whatever Google has many completions for.
-- **Typos are reported separately** and never folded into the headline, because the ranker
-  has no fuzzy matching at all — mixing them in would move the KPI for a reason unrelated to
-  relevance weighting.
+- **Typos are reported separately** and never folded into the headline. They are now partly
+  answered — see below — but a spelling correction moves the number for a reason unrelated to
+  relevance weighting, so mixing them in would make the headline mean two things.
+
+## The relaxation pass, and why it could not regress anything
+
+A query that returns **nothing** gets a second attempt: dotted compounds the corpus does not
+contain are split into parts it does (`nestjs.microservices` → `nestjs microservices`), and unknown
+words are corrected against the corpus vocabulary by restricted Damerau-Levenshtein distance —
+restricted, because the measured typo class is one transposed key, which plain Levenshtein scores as
+two edits.
+
+**The gate is the whole safety argument.** It runs only when the ranked list is empty, so any query
+that returns at least one result today is scored by byte-identical code and cannot move. Measured
+against that prediction:
+
+| | before | after |
+|---|---|---|
+| typo accuracy | 36.2% | **55.2%** |
+| typo empty result set | 29.1% | **5.8%** |
+| typo never found | 54.3% | 34.3% |
+| natural | 94.2% | 94.3% (2 queries improved, 0 worse) |
+| artificial | 90.8% | 90.8% (**0 queries changed**) |
+| question | 64.5% | 64.5% (0 changed) |
+
+The two natural queries that moved were returning nothing before. `check-search-ranking.js` asserts
+the gate, because the safety argument is worth exactly as much as that assertion.
+
+Two design notes that cost a measurement each:
+
+1. **Confident rewrites are tried before guesses.** `nestjs.microservices cqrs` is answered by
+   splitting alone. A single combined pass also "corrected" `cqrs` to `cars` — one substitution, and
+   `cars` is in five sections — and announced a query about CQRS as a query about cars.
+2. **A df floor does not separate good corrections from bad ones.** `cars` has df 5, and 67% of the
+   prose vocabulary has df ≤ 5, so any threshold that rejects `cars` rejects most legitimate
+   corrections too. The ordering above is what fixes it.
+
+Corrections are **announced** (`3 results for “nestjs microservices cqrs”` in the status line, which
+is already an aria-live region) and the highlighter marks the corrected term, not the misspelling.
+A search that quietly answers a different question is worse than one that finds nothing.
+
+What this deliberately does **not** fix: a typo whose query still returned something irrelevant.
+That is the larger half of the typo gap (34.3% still never found) and it needs a change that can
+regress, so it needs to be measured rather than argued.

scripts/search-kpi/compare.js

@@ -26,7 +26,8 @@ const os = require('node:os');
 const path = require('node:path');
 const { execFileSync } = require('node:child_process');
 
-const { load, page, accuracyFor } = require('./lib/harness');
+const { load, page, accuracyFor, ndcgFor } = require('./lib/harness');
+const { verdict } = require('./lib/stats.js');
 
 const ROOT = path.join(__dirname, '..', '..');
 const { RANKER_DIR } = require('../lib/ranker.js');
@@ -104,31 +105,78 @@ function main() {
   const before = load(DIR, baselineFile);
   const after = load(DIR);
 
-  const readJson = (file) => JSON.parse(fs.readFileSync(path.join(__dirname, 'data', file), 'utf8'));
+  const readJson = (file) => {
+    const at = path.join(__dirname, 'data', file);
+
+    return fs.existsSync(at) ? JSON.parse(fs.readFileSync(at, 'utf8')) : null;
+  };
+  const setOf = (file, key) => {
+    const data = readJson(file);
+
+    return data ? data[key] : null;
+  };
   const sets = {
-    natural: readJson('natural-judged.json').judged,
-    artificial: readJson('artificial-queries.json').main,
+    natural: setOf('natural-judged.json', 'judged'),
+    artificial: setOf('artificial-queries.json', 'main'),
+    // The chat-shaped set was measured only by questions.js --ref, so a comparison run reported
+    // two of the three populations. It is the one that caught a real regression the other two
+    // could not see, which makes leaving it out of the default comparison the wrong default.
+    question: setOf('question-queries.json', 'queries'),
   };
 
   let dirty = false;
 
   for (const [name, cases] of Object.entries(sets)) {
+    if (!cases) {
+      console.log(`\n=== ${name}: SET NOT PRESENT, not measured ===`);
+      continue;
+    }
+
     const better = [];
     const worse = [];
+    // Per-query deltas, INCLUDING the zeros — see lib/stats.js. `byTopic` carries the same deltas
+    // grouped, because the headline number is a macro average and a claim about a macro has to be
+    // tested over topics rather than over queries.
+    const deltas = [];
+    const ndcgDeltas = [];
+    const byTopic = new Map();
 
     for (const testCase of cases) {
       const expect = (Array.isArray(testCase.expect) ? testCase.expect : [testCase.expect])
         .map(page);
       const b = rankOf(before, testCase.query, expect);
       const a = rankOf(after, testCase.query, expect);
+      const delta = accuracyFor(a) - accuracyFor(b);
+      const topic = testCase.label || testCase.bucket || '(none)';
 
-      if (accuracyFor(a) > accuracyFor(b)) better.push({ ...testCase, b, a });
-      if (accuracyFor(a) < accuracyFor(b)) worse.push({ ...testCase, b, a });
+      deltas.push(delta);
+      ndcgDeltas.push((ndcgFor(a, 1, 1) - ndcgFor(b, 1, 1)) * 100);
+
+      if (!byTopic.has(topic)) byTopic.set(topic, []);
+      byTopic.get(topic).push(delta);
+
+      if (delta > 0) better.push({ ...testCase, b, a });
+      if (delta < 0) worse.push({ ...testCase, b, a });
     }
 
-    console.log(`\n=== ${name} (n = ${cases.length}) vs ${REF} ===`);
-    console.log(`better: ${better.length}   worse: ${worse.length}   `
-      + `unchanged: ${cases.length - better.length - worse.length}`);
+    // One number per topic, so each topic weighs the same as every other — the macro definition.
+    const topicDeltas = [...byTopic.values()]
+      .map((list) => list.reduce((x, y) => x + y, 0) / list.length);
+
+    const micro = verdict(deltas);
+    const macro = verdict(topicDeltas);
+
+    console.log(`\n=== ${name} (n = ${cases.length}, ${byTopic.size} topics) vs ${REF} ===`);
+    console.log(`  accuracy micro   ${micro.line}`);
+    console.log(`  accuracy macro   ${macro.line}`);
+    console.log(`  nDCG@10 micro    ${verdict(ndcgDeltas).line}`);
+
+    // The honest reading, spelled out because "unmeasured" is the result most likely to be
+    // misread as "safe". It means this set cannot tell, not that nothing happened.
+    if (!micro.significant && !macro.significant && (better.length || worse.length)) {
+      console.log(`  → ${better.length + worse.length} queries moved and neither average clears `
+        + 'zero: this change is UNMEASURED on this set, not neutral.');
+    }
 
     if (!worse.length) {
       continue;

scripts/search-kpi/lib/harness.js

@@ -74,6 +74,35 @@ function accuracyFor(position) {
   return position >= 1 && position <= 10 ? 100 - 10 * (position - 1) : 0;
 }
 
+// nDCG@10, reported ALONGSIDE the accuracy above rather than instead of it.
+//
+// The linear −10-per-position metric is this project's own invention and it says something true —
+// "nobody scrolls to the eleventh row" — but it discounts positions 5 to 10 far more harshly than
+// any reader behaves, and it cannot express a partially-right answer. nDCG is the field's standard
+// and its log discount is the shape click distributions actually follow, so the two together say
+// more than either: a change that moves a query from #4 to #2 barely registers in accuracy terms
+// (+20 of 100) and is a large nDCG move, while #1 to #2 is the reverse.
+//
+// SINGLE RELEVANT DOCUMENT per query, deliberately. `expect` lists ALTERNATIVES — "the topic index
+// or the article it lists, either is right" — so the ideal ranking has one relevant document at
+// position 1, not three. Summing gains over the alternatives, as textbook DCG would, would credit a
+// ranker for returning three spellings of the same answer and would make a query with more
+// acceptable answers score higher for the same reader experience.
+//
+// `grades` is honoured when a query carries one ({url: gain}), so a graded label set can be
+// introduced later without touching this. Absent, every listed URL is worth the same.
+const NDCG_K = 10;
+
+function ndcgFor(position, gain, maxGain) {
+  if (!position || position > NDCG_K) {
+    return 0;
+  }
+
+  const ideal = maxGain || 1;
+
+  return ((gain || 1) / ideal) / Math.log2(position + 1);
+}
+
 // One case = one query plus the URL(s) that would be a correct top result. `expect` may be
 // a string or an array: several pages can be equally right (a topic index and the article
 // it lists), and pretending otherwise would score a good answer as a miss.
@@ -91,25 +120,32 @@ function evaluate(ranker, cases, options) {
     try {
       hits = ranker.search(ranker.parseQuery(testCase.query)).slice(0, limit);
     } catch (error) {
-      results.push({ ...testCase, position: 0, accuracy: 0, error: String(error.message) });
+      results.push({ ...testCase, position: 0, accuracy: 0, ndcg: 0, error: String(error.message) });
       continue;
     }
 
     let position = 0;
+    let gain = 0;
 
     for (let i = 0; i < hits.length; i++) {
       const url = strict ? hits[i].record.u : page(hits[i].record.u);
 
       if (!hits[i].external && expected.includes(url)) {
         position = i + 1;
+        gain = testCase.grades ? Number(testCase.grades[url]) || 1 : 1;
         break;
       }
     }
 
+    const maxGain = testCase.grades
+      ? Math.max(...Object.values(testCase.grades).map(Number))
+      : 1;
+
     results.push({
       ...testCase,
       position,
       accuracy: accuracyFor(position),
+      ndcg: ndcgFor(position, gain, maxGain) * 100,
       returned: hits.length,
       top: hits.length ? hits[0].record.u : null,
       topScore: hits.length ? Math.round(hits[0].score) : 0,
@@ -143,6 +179,7 @@ function summarise(results) {
   return {
     total,
     accuracy: sum((r) => r.accuracy) / total,
+    ndcg: sum((r) => r.ndcg || 0) / total,
     top1: (count((r) => r.position === 1) / total) * 100,
     top3: (count((r) => r.position >= 1 && r.position <= 3) / total) * 100,
     top5: (count((r) => r.position >= 1 && r.position <= 5) / total) * 100,
@@ -174,6 +211,7 @@ function table(label, summary) {
   const lines = [
     `${label}  (n = ${summary.total})`,
     `  accuracy (KPI)   ${pct(summary.accuracy)}`,
+    `  nDCG@10          ${pct(summary.ndcg)}`,
     `  #1 exactly       ${pct(summary.top1)}`,
     `  in top 3         ${pct(summary.top3)}`,
     `  in top 5         ${pct(summary.top5)}`,
@@ -188,4 +226,4 @@ function table(label, summary) {
   return lines.join('\n');
 }
 
-module.exports = { load, evaluate, summarise, table, accuracyFor, page, median };
+module.exports = { load, evaluate, summarise, table, accuracyFor, ndcgFor, page, median };