refactor(search): the ranker moves to its own repo, pinned as a submodule

search.js is now github.com/imqueue/search-ranker, included here at
vendor/search-ranker and to be included identically by @imqueue/mcp. The reason is
drift: the MCP server carried its own implementation of the same ranking rules, the
two answered identical queries differently, and nothing compared them — recall@6 was
99.5% here against 83.9% there before anyone measured. One file with one history
cannot drift from itself.

Byte-for-byte, so this changes no answer: kpi:compare reports 0 better, 0 worse,
12,279 unchanged across the natural and artificial sets, and the served asset is the
submodule's own hash (verified in the browser — dialog opens, index loads, "retry a
failed call" returns the right answer at #1).

THE FAILURE THIS GUARDS AGAINST. buildAssetManifest() finds JS by scanning
directories, and an unpopulated submodule is an EMPTY DIRECTORY: the scan would find
no *.js, report nothing, and emit a complete-looking site with no search.js in it — a
missing asset, not a build error, and no check would object. So the ranker is handled
by NAME rather than by adding its directory to the scan, and its absence throws at
eleventy config load with `git submodule update --init` in the message. Both failure
modes were tested by causing them.

It also refuses to be shadowed: a src/**/js/search.js reappearing is a hard error
rather than "later source wins", because that file would be served in preference to
the pinned ranker with nothing saying so — the exact drift this split ends. Ordered
last in the function so the collision is detectable at all.

scripts/lib/ranker.js exists because the path was spelled five times independently
(three checks, the KPI harness, the asset scan). Moving the file meant finding all
five, and a missed one fails differently in each caller.

compare.js needed more than a new path: `--ref` names a commit in the RANKER's
history, which is no longer in this repo's history at all, so its git calls now run
with cwd inside the submodule. Uninitialised, git says only "does not exist in
'HEAD'", which names neither the cause nor the fix — so that case is caught and
explained.

submodules: true on all six checkout steps, in five workflows — every one of them
runs `npm test` or `npm run build:all`, so without it each fails at the throw above
instead of checking anything. Cloudflare Pages needs no such flag: it clones and
populates public HTTPS submodules, proven on a throwaway branch before this was
started, which is also why .gitmodules must stay public HTTPS rather than SSH.

Contributor note in the README, because a plain `git clone` yields a tree that looks
complete and a build that stops for a reason nothing on disk explains.

Author: Mikhus

.github/workflows/checks.yml

@@ -41,6 +41,13 @@ jobs:
           # file would appear to have been added by it and the check would fail on
           # every run.
           fetch-depth: 0
+          # REQUIRED: the search ranker is the imqueue/search-ranker submodule at
+          # vendor/search-ranker. Without this the checkout leaves it empty and the
+          # build throws (scripts/lib/asset-manifest.js) — which is the designed
+          # behaviour, but it means every run here fails rather than checking
+          # anything. Every workflow in this repo that runs `npm test` or
+          # `npm run build:all` needs it, and all of them do.
+          submodules: true
 
       - uses: actions/setup-node@v7
         with:

.github/workflows/indexnow.yml

@@ -21,6 +21,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7
+        with:
+          # REQUIRED by `npm run build:all` below: the search ranker is a submodule
+          # at vendor/search-ranker, and the build throws without it. See checks.yml.
+          submodules: true
 
       - uses: actions/setup-node@v7
         with:

.github/workflows/refresh-api-docs.yml

@@ -56,6 +56,9 @@ jobs:
           # date. The default shallow fetch has one commit, so every file looks as
           # though that commit added it and the check fails on every run.
           fetch-depth: 0
+          # REQUIRED for the same step: the search ranker is a submodule at
+          # vendor/search-ranker, and the build throws without it. See checks.yml.
+          submodules: true
 
       - uses: actions/setup-node@v7
         with:

.github/workflows/sync-cli-guide.yml

@@ -38,6 +38,9 @@ jobs:
           # as though that commit added it and the check fails on every run.
           # Same reason refresh-api-docs.yml sets it.
           fetch-depth: 0
+          # REQUIRED by the same step: the search ranker is a submodule at
+          # vendor/search-ranker, and the build throws without it. See checks.yml.
+          submodules: true
 
       - name: Resolve cli ref
         id: ref

.github/workflows/weekly-maintenance.yml

@@ -51,6 +51,9 @@ jobs:
           # file looks as though one commit added it. Same reason refresh-api-docs.yml
           # and sync-cli-guide.yml set it.
           fetch-depth: 0
+          # REQUIRED by the `npm test` step below: the search ranker is a submodule
+          # at vendor/search-ranker, and the build throws without it. See checks.yml.
+          submodules: true
 
       - uses: actions/setup-node@v7
         with:
@@ -122,6 +125,10 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@v7
+        with:
+          # REQUIRED: check:links:external builds both editions, and the build throws
+          # without the vendor/search-ranker submodule. See checks.yml.
+          submodules: true
       - uses: actions/setup-node@v7
         with:
           node-version: 22

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "search-ranker"]
+	path = vendor/search-ranker
+	url = https://github.com/imqueue/search-ranker.git

README.md

@@ -13,12 +13,30 @@ pages live in `src/org/` and `src/com/`, and Eleventy ignores the other one.
 
 ## Local development
 
+**Clone with submodules.** The search ranker is its own repo
+([imqueue/search-ranker](https://github.com/imqueue/search-ranker)), pinned here at
+`vendor/search-ranker` and pinned identically by the `@imqueue/mcp` server, so the site
+and the MCP tool cannot answer the same query differently. A plain `git clone` leaves that
+directory empty and the build stops with the fix in the message — deliberately, because the
+alternative is a site that builds cleanly and ships no `search.js` at all.
+
+```bash
+git clone --recurse-submodules https://github.com/imqueue/imqueue.com.git
+# already cloned?
+git submodule update --init
+```
+
 ```bash
 npm install
 npm run serve:org   # imqueue.org — http://localhost:8080
 npm run serve:com   # imqueue.com — http://localhost:8081
 ```
 
+To change the ranker, edit `vendor/search-ranker/search.js`, measure with
+`npm run kpi:compare` (never by the summary alone — read the per-query deltas), then commit
+**inside the submodule** and commit the moved pointer here. `scripts/search-kpi/README.md`
+has the numbers and the rejected experiments.
+
 ## Build
 
 ```bash

scripts/check-search-index.js

@@ -272,13 +272,13 @@ for (const name of ['search-peer-index.json', 'search-peer-text.json']) {
 }
 
 // ---- the ranker and the feeds agree about the shape ----------------------------
-// Two independent declarations, compared here on purpose. The ranker is about to live in its own
-// repository and be pinned as a submodule, while the feeds it reads are fetched LIVE — so a
-// pinned ranker reads today's feeds. Move a tuple position without moving the version and every
-// score is computed off the wrong field, with nothing to report: no exception, no empty result,
-// just quietly wrong answers. Failing the build is the only place this is cheap to catch.
+// Two independent declarations, compared here on purpose. The ranker lives in its own repository
+// and is pinned here as a submodule, while the feeds it reads are fetched LIVE — so a pinned
+// ranker reads today's feeds. Move a tuple position without moving the version and every score
+// is computed off the wrong field, with nothing to report: no exception, no empty result, just
+// quietly wrong answers. Failing the build is the only place this is cheap to catch.
 {
-  const ranker = require(path.join(__dirname, '..', 'src', '_shared', 'js', 'search.js'));
+  const ranker = require(path.join(__dirname, 'lib', 'ranker.js')).requireRanker();
   const { FEED_V } = require(path.join(__dirname, 'lib', 'search-corpus.js'));
 
   if (ranker.FEED_V !== FEED_V) {

scripts/check-search-ranking.js

@@ -4,13 +4,15 @@
 //   node scripts/check-search-ranking.js
 //
 // Relevance is the part of a search feature that regresses silently. Every weight in
-// src/org/js/search.js was set to fix a specific wrong answer, and each of those is
-// one "small improvement" away from coming back — the @imqueue MCP server's ranker
-// learned this the hard way, twice. So each case below names the signal it protects
-// and the wrong answer that was actually observed before it existed.
+// the ranker was set to fix a specific wrong answer, and each of those is one "small
+// improvement" away from coming back — the @imqueue MCP server's ranker learned this
+// the hard way, twice. So each case below names the signal it protects and the wrong
+// answer that was actually observed before it existed.
 //
 // The ranker is required directly: it exports itself when there is no `document`,
-// which is why it can be checked here instead of through a browser screenshot.
+// which is why it can be checked here instead of through a browser screenshot. It
+// lives in a submodule now, so this file is also the check that notices an
+// unpopulated one — requireRanker() throws with the fix.
 //
 // Exits non-zero on any failure; wired into `npm test`.
 
@@ -21,7 +23,7 @@ const path = require('node:path');
 
 const ROOT = path.join(__dirname, '..');
 const OUT = path.join(ROOT, '_site-org');
-const ranker = require(path.join(ROOT, 'src', '_shared', 'js', 'search.js'));
+const ranker = require('./lib/ranker.js').requireRanker();
 
 let failures = 0;
 const fail = (msg) => { failures++; console.error(`  FAIL  ${msg}`); };

scripts/check-search-ui.js

@@ -172,7 +172,17 @@ for (const [re, ok, why] of JS_CASES) {
 // ---- what search reports ----------------------------------------------------
 // The measurement is the only route to improving relevance with real readers instead of
 // hand-written ground truth (scripts/search-kpi/), so it is worth a few cases of its own.
-const searchJs = read(path.join(ROOT, 'src', '_shared', 'js', 'search.js'));
+// Read as TEXT, not required: these cases assert on source strings, which is the only way to
+// check a constant that never leaves the IIFE. From the submodule — see scripts/lib/ranker.js,
+// and check its presence first so a plain clone gets the instruction rather than an ENOENT.
+const rankerLib = require('./lib/ranker.js');
+
+if (!rankerLib.exists()) {
+  console.error(rankerLib.MISSING);
+  process.exit(1);
+}
+
+const searchJs = read(rankerLib.RANKER_FILE);
 
 // A settle window is the difference between "queries people asked" and a report full of
 // their own prefixes. Asserted as a NUMBER, not a mention: `var SETTLE = 0` would satisfy