eval

Model-vs-rule eval

Does the model beat the obvious rule on labeled cases, or does it just mirror the detector signals? The eval sends hand-written cases through the production path and compares the model's answers with a rule written for each question. The production path is:

1. compile the case's timeline the way the browser does (scrubbing, built-in detectors, compileTimeline);
2. send it through createHandler;
3. call the real provider.

The results from the first full run are in results/ and summarized in Does it actually work?.

Run

pnpm build
pnpm --filter @adaptif/server eval --dry-run          # no key, no calls: inspect every case
pnpm --filter @adaptif/server eval --question leaving_reason --repeats 1
pnpm --filter @adaptif/server eval --verbose          # all questions, 3 repeats, per-case tables

The script reads the provider key from the environment or from a .env file in packages/server or the repository root. Any key that providerFromEnv accepts works. The script never prints the key.

Flags:

Flag	Meaning
--question <id>	Run one question. Repeat the flag to run several.
--repeats N	Ask each case N times. Default 3.
--concurrency N	Requests in flight at once. Default 4.
--provider <name>	Force a provider instead of picking one from the keys that are set.
--model <id>	Use a specific model ID.
--verbose	Print every case's answers, not just the summary.
--dry-run	Print every case without calling a model. Needs no key.
--stub-signal	Use a fake model that answers from the signals alone, to check that the scoring catches a model that only mirrors signals. Needs no key.

Each run writes its full results to .adaptif/eval/<timestamp>.json, which is gitignored.

Cases

Each file in questions/ has the question text, a rule, and the cases. The rule is the strongest reasonable rule I could write, not a straw man.

• Labels: every case has a label and a why. The cases encode the authors' theory of the user, so write new cases and labels before seeing the model's answers, and don't change them afterwards.
• Pairs: most cases come in pairs. Both cases in a pair fire the same detector signals but have different labels, so anything that reads only signals gets at most one of them right. The script refuses to run if a pair's signals don't match.
• goalNeedsPro is in the harness and is not in the committed results file. A dry run prints its cases. A live score is a new run, not a comparison against results/2026-10-03-openrouter-jev.json.

Reading the summary

• pairs model vs pairs rule is the main evidence. Pairs are built so that the rule misses one side; the question is whether the model gets both.
• only model ✓ / only rule ✓ count the cases where exactly one of them was right. sign p is the exact sign test on those cases. With about 10 cases per question, sign p will rarely be small, so treat a single question's result as go/no-go.
• mirrors signals (situations only) counts how often the model said "yes" exactly when some signal fired.
• frustrated is the control. Signals mostly decide its answers, so expect a rough tie. A big model win there suggests the cases favor the model.
• Scoring line: situations are scored at p ≥ 0.5, which is where the eval measures accuracy. Production uses the confidence presets instead; the default is 0.6.