Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ folder `/docs`.

| Path | What it is |
| --- | --- |
| `index.html` | Landing page: why the experiments were run, what System One models are, the four experiments, the protocol, the notebook log, the limits |
| `index.html` | Landing page: why the experiments were run, the premise (what Jev is), the four experiments, the process, the notebook log, a note on the level of rigor |
| `assets/` | Shared by every page: styles (`site.css`) and the theme toggle (`site.js`). Shared by the results pages: `results.css`, and `results.js` (tooltips, chart helpers, the side contents) |
| `model-router/` | The 2026-10-08 model-router run: `index.html`, `app.js`, and generated `data.js` |
| `model-router/` | The 2026-10-08 model-router run: `index.html`, `app.js`, generated `data.js`, and two captures of the page's route explorer (`screenshot-light.png`, `screenshot-dark.png`) that the landing page shows as a thumbnail |
| `godot-jev/` | The 2026-10-08 three-model godot-jev run: `index.html`, `page.css`, `app.js`, a screenshot of the game, and generated `data.js` |
| `tools/build-model-router-data.mjs` | Builds `model-router/data.js` |
| `tools/build-godot-jev-data.gd` | Builds `godot-jev/data.js` |
Expand Down
5 changes: 3 additions & 2 deletions docs/godot-jev/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,9 @@ <h1>Three Jevs parse <em>16 lines of player slang.</em></h1>
model, with the moves possible right now as the options of one Choice. I gave three Jevs, one hosted
and two running on a laptop, the same 16 lines in the same game states, three times each. All three got
the player to the crown without a wrong move. They differ in how sure they were, how often the game had
to ask what the player meant, and how long a turn took on this setup. This is a gut check on a small,
hand-written set of lines, not a benchmark: <a href="#caveats">what it does and doesn't show</a>.
to ask what the player meant, and how long a turn took on this setup. This is a gut check on a small
set of lines written for a recording, not a benchmark:
<a href="#caveats">what it does and doesn't show</a>.
</p>
<div class="hero-stats" id="hero-stats"></div>
<div class="legend-models" id="legend-models"></div>
Expand Down
230 changes: 178 additions & 52 deletions docs/index.html

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/model-router/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
const offTop = runs.map((r) => N - atDefault(r).routes[ids[TOP]]);
const sameOff = offTop.every((v) => v === offTop[0]);
$("#hero-stats").innerHTML = `
<div class="stat"><div class="v">${N}<small> tasks</small></div><div class="k">hand-written sub-agent tasks, each asked once of ${runs.length} decision models</div></div>
<div class="stat"><div class="v">${N}<small> tasks</small></div><div class="k">sub-agent tasks written and labelled by Claude, each asked once of ${runs.length} decision models</div></div>
<div class="stat"><div class="v">${under}</div><div class="k">tasks sent to a model too weak for them, by any of the three</div></div>
<div class="stat"><div class="v">${sameOff ? pct(offTop[0]) : offTop.map(pct).join(" / ")}</div><div class="k">of tasks kept off the most capable model${sameOff ? ", by all three" : ""}. ${misses} miss in ${N * runs.length} decisions.</div></div>
<div class="stat"><div class="v lat">${runs
Expand All @@ -73,7 +73,7 @@
"1 · Routing",
"On ≥ 100 real tasks, a bar where ≤ 5% go too low and ≥ 50% leave the top route",
s.map((x) => `<span class="mark-na">${pct(x.under)} low · ${pct(N - x.routes[ids[TOP]])} off top</span>`),
`<span class="stamp">Not yet evidence</span><br><small>24 starter tasks, one author</small>`,
`<span class="stamp">Not yet evidence</span><br><small>24 starter tasks, all written by Claude</small>`,
],
[
"2 · Speed",
Expand Down
4 changes: 2 additions & 2 deletions docs/model-router/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ <h1>Three Jevs pick the model for <em>24 sub-agent tasks.</em></h1>
One request to a System One model decides whether a sub-agent runs on Haiku, Sonnet or Opus. I asked
three Jevs, one hosted and two running on a laptop, to make that call for the same tasks. Their routes
nearly match. They differ in how sure they were, and in how long an answer took on this setup. This is
a gut check on a small, hand-written set of tasks, not a benchmark:
a gut check on a small set of tasks that Claude wrote and labelled, not a benchmark:
<a href="#caveats">what it does and doesn't show</a>.
</p>
<div class="hero-stats" id="hero-stats"></div>
Expand Down Expand Up @@ -462,7 +462,7 @@ <h2>What this does and doesn't show</h2>
<span class="label">It doesn't show</span>
<ul>
<li><b>24 tasks, asked once each.</b> One task is 4 percentage points.</li>
<li><b>One author wrote everything.</b> The tasks, their labels and the route descriptions came from one sitting, so tasks echo the wording they're judged against ("find the cause" against "a failure with an unknown cause").</li>
<li><b>One author wrote everything.</b> Claude wrote the tasks, their labels and the route descriptions in one sitting, so tasks echo the wording they're judged against ("find the cause" against "a failure with an unknown cause").</li>
<li><b>Labels are opinions.</b> Nobody ran these tasks on the three Claude models to see which was enough. That needs a harness (check 4).</li>
<li><b>Flash 9B's 24 of 24 has little margin.</b> On tasks 13 to 16 it was close to an even split between two routes.</li>
<li><b>The tasks don't separate the models.</b> All three are at or near 24 of 24, so this says nothing about which decides best.</li>
Expand Down
Binary file added docs/model-router/screenshot-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/model-router/screenshot-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 4 additions & 3 deletions model-router/eval/results-2026-10-08.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,10 @@ sources: `results-2026-10-08T16-41-27-867Z.json` (27B), `results-2026-10-08T18-4

An agent harness has to pick a model each time its orchestrator spawns a sub-agent. model-router
makes that choice with one request to a System One model (Jev), which returns probabilities, not
text. This is the first live run: 24 hand-written tasks, each sent once to three decision models.
One is TypeSafe's hosted Jev. The other two are [Open Jev](https://huggingface.co/openjev) models
running locally: the 27B model, and Flash 9B, a smaller one released the day before.
text. This is the first live run: 24 tasks that Claude wrote and labelled, each sent once to three
decision models. One is TypeSafe's hosted Jev. The other two are
[Open Jev](https://huggingface.co/openjev) models running locally: the 27B model, and Flash 9B, a
smaller one released the day before.

## Result

Expand Down
Loading