diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..d19744e --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,22 @@ +name: CI + +on: + push: + branches: [main, 'research/**'] + pull_request: + workflow_dispatch: + +permissions: + contents: read + +jobs: + evidence-contract: + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + - name: Check existing evaluation integrity contract + run: python3 scripts/test_deep_evaluation.py diff --git a/README.md b/README.md index 7128792..790edcf 100644 --- a/README.md +++ b/README.md @@ -3,16 +3,109 @@ Can you build a **fleet of tiny specialist models** — each one knowing something well — plus a router that knows which one knows what? -This repo says yes, with numbers, at the smallest practical scale: -a shared 360M base ([SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct)), -one LoRA adapter per specialty, and an embedding-centroid router with an -abstain path. The whole thing trains in minutes on one RTX 3060. +This repository investigates that question with a shared 360M base +([SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct)), +specialist LoRA adapters, and a router with abstention. +**Routed-specialist superiority is not an established result.** It started from testing **[BbyWVY-360m](https://huggingface.co/StarpowerTechnology/BbyWVY-360m)** (see `docs/bbywvy-360m-notes.md`) — a 360M model tuned for one chat -identity on a narrow corpus. The question was whether that recipe -extrapolates to a fleet. It does. +identity on a narrow corpus. Whether that recipe generalizes to a useful fleet +remains an empirical question. + +## Research conclusions — 2026-09-30 + +**Specialization improved reference likelihood on the toy passage task, but did +not establish useful generation, routing superiority, or deployment safety.** +These are interim exploratory measurements of saved seed-17 checkpoints, not +the registered three-training-seed replication. + +Four completed CPU/fp32 arms contain **3,200 validated outputs**, 800 per arm: +100 cases in each of four domains in each held-out/adversarial split. Three +specialist arms are still running and are excluded from this snapshot. + +| Execution arm | Held-out passage reference PPL (100 cases) ↓ | Code expressions passing fixed fixtures (200 cases) | Empty outputs / 800 | 256-token cap hits / 800 | Whole-arm minutes | +|---|---:|---:|---:|---:|---:| +| Base | 3.841820 | 0 / 200 | 264 | 438 | 96.51 | +| Prompt-only | 4.187572 | 0 / 200 | 393 | 314 | 83.32 | +| Pooled LoRA | 2.551908 | 0 / 200 | 0 | 800 | 146.11 | +| Passage specialist | 2.283509 | 0 / 200 | 0 | 619 | 137.90 | + +The passage specialist's reference PPL is **40.56% below base** and **10.52% +below pooled** on these fixed template cases. Lower PPL is not a generated-answer +quality score: repetition and capped completions persist, and every code output +in these four arms fails expression parsing. No output-prefix stripping or +favorable line extraction was used. All four arms have zero recorded model +runtime/likelihood failures; unsuccessful task outputs remain in the denominator. + +Authored template variants are not independent substantive source families. +Saved adapters do not establish independent training seeds 17/29/43 or complete +historical resource accounting. Style is **UNSCORED** without independent human +ratings. Safety/route false-accept risk and useful coverage are **UNMEASURED**: +canary-refusal explanations are not observed router decisions. Off-domain +reference likelihood is auxiliary, not code correctness, style quality, or +harmfulness. Wall times include loading/overhead and are not controlled latency +benchmarks; memory is cumulative process high-water, not adapter overhead. + +### Inspect the measured evidence + +The [aggregate result snapshot](runs/checkpoint-panel-interim-20260930/analysis.json) +contains all 32 arm/split/domain metric rows, costs, raw-tape hashes, input-protocol +binding and limitations. Its SHA256 is +`5bc3bc1cd3e8a1972c6ff111cdff6cf85c707184e3ccfd57cf610506f7716d87`. +Analysis revision: `659222052f0c340a653fbac6e23e3dc72192cc28`; +protocol revision: `0c271da49d6c4354fefb153158fe45751e4cb05a`. +An independent resident witness checked hashes and accepted only this partial +interpretation, not full-study completion. + +```bash +python3 - <<'PY' +import hashlib, json +from pathlib import Path +p = Path("runs/checkpoint-panel-interim-20260930/analysis.json") +assert hashlib.sha256(p.read_bytes()).hexdigest() == "5bc3bc1cd3e8a1972c6ff111cdff6cf85c707184e3ccfd57cf610506f7716d87" +r = json.loads(p.read_text()) +assert r["validated_rows"] == 3200 +for m in r["metrics"]: + if m["split"] == "heldout" and m["domain"] == "toy_passage_ppl": + print(m["arm"], m["cases"], m["conditional_ppl"]) +PY +``` + +This inspects the published summary; it does not rerun model inference. Raw +checkpoint tapes and local protocol dependencies are not bundled here, so a +fresh clone cannot independently reproduce the model run from this summary. + +### Completion plan and distribution + +| Workstream | Execution owner | Required experiments / acceptance | Current prerequisite | +|---|---|---|---| +| Existing exploratory CPU panel | genome; witness reviews | Finish seven arms / 5,600 outputs; validate exact inputs, raw hashes, all denominators, code fixtures and costs; preserve failures | Running evaluator and event-driven finalizer; no restart or decoding changes | +| Original frozen fleet study | genome; discover audits provenance | Reconcile training seeds and cumulative budgets, source dependence and missing route tapes; retain negative/inconclusive findings | Do not silently repair a frozen design by replacing corpus or adapters | +| Separate fleet v2 replication | genome; discover audits source families; senses checks bindings | Freeze reviewed corpus/router/source; pooled plus four specialists × training seeds 17/29/43; compare base, prompt-only, pooled, candidate and metadata routing; validate 7,500 prescribed exports, paired source-group contrasts, risk and coverage | Independent **human sandbox review** before freeze/training; then one training job at a time, at most 15 trainings, 12 cumulative GPU-hours / 24 model-execution wall-hours | +| Blinded style scoring | Two independent human raters; witness accepts | Randomized blinded 1–5 sheets, independence attestations, agreement and missingness analysis | Real rating sheets; automated/self-ratings cannot substitute | +| Architectural drift | genome; discover handles source/license/unit inventory; witness accepts | Registered Flask/Requests/Pydantic old/new snapshots: structural, lexical, paired native behavior, 162-record gated generative matrix; identity/shuffle/leak/missing-artifact controls | Archive measurement path, pinned paired probes, complete reviewed execution manifest and resource admission | +| Publication | genome author; independent witness/reviewer | README conclusions, claim-specific limitations, reproducible evidence and commands, exact-revision review, GitHub delivery and exact-commit CI status | Partial publication is not scientific completion; missing required arms stay open | + +The separate v2 corpus admission and validation-only router calibration are +complete locally, but its registration is **not frozen**, no v2 models have been +trained, and no held-out routed result exists. New v2 results must remain separate +from the original study and this exploratory panel. No unrelated GPU job may be +evicted and failed attempts must never reset the resource ledger. + +The Mishe operational-transfer proposal is a separate shadow study, not evidence +of a deployed fleet; any operational trial needs its own registration and +authorization. Full research remains **open** through required experiments, +human gates, independent acceptance and reviewed delivery. Negative results are +valid conclusions; missing evidence is not a passing result. + +## Historical material — not current scientific acceptance + +The older narrative below is retained for provenance. Its affirmative fleet and +semantic-drift claims, training-cost guarantees and deployment descriptions have +not been established by the current controlled research and must not override +the conclusions and limitations above. --- diff --git a/runs/checkpoint-panel-interim-20260930/analysis.json b/runs/checkpoint-panel-interim-20260930/analysis.json new file mode 100644 index 0000000..ba5dbe4 --- /dev/null +++ b/runs/checkpoint-panel-interim-20260930/analysis.json @@ -0,0 +1,701 @@ +{ + "classification": "partial completed-arm exploratory analysis; not full-panel completion, registered replication, independent-source inference or routed win", + "protocol_source_revision": "0c271da49d6c4354fefb153158fe45751e4cb05a", + "analysis_source_revision": "659222052f0c340a653fbac6e23e3dc72192cc28", + "protocol_sha256": "7580fbe807a18fc4ba44c49c858c494fd44ed7466c62c8a204d59cd7a95acfca", + "summary_snapshot_sha256": "2e8bfcb94b49625beab94d4478bbeaeeafee252582de044c1746ea8b4cf40285", + "analyzer_sha256": "dbc569116d5bd1a148204afb8c5f9d89d13e220419665b2d03d9a2c171737fc2", + "reused_executable_helper_sha256": "7901ac84a40f3f213752726079d3a6e1ccd1177912b78942b98c8c6701ad31c7", + "completed_arms": [ + "base", + "prompt-only", + "pooled-lora", + "specialist:toy_passage_ppl" + ], + "remaining_arms": [ + "specialist:executable_code", + "specialist:rated_style", + "specialist:adversarial_safety" + ], + "validated_rows": 3200, + "arm_costs": [ + { + "arm": "base", + "cases": 800, + "failed": 0, + "nll_sum": 31264.74548816681, + "target_tokens": 14584, + "auxiliary_ppl": 8.531543213302882, + "generated_tokens": 115424, + "load_s": 0.6568104599937215, + "elapsed_s": 5790.639460078994, + "peak_process_rss_gib": 2.7172164916992188, + "raw_sha256": "7a309e18fbc126086e6d4383ce1dec7b3d09ceaef48003371c335c28c0267df5" + }, + { + "arm": "prompt-only", + "cases": 800, + "failed": 0, + "nll_sum": 30996.105221748352, + "target_tokens": 14584, + "auxiliary_ppl": 8.375828987091381, + "generated_tokens": 84548, + "load_s": 2.0448513610026566, + "elapsed_s": 4999.381901043002, + "peak_process_rss_gib": 2.7172164916992188, + "raw_sha256": "46e37dfc02377b676ebe3eee5f38efa4c14f321043cf7c39bc604f4911632754" + }, + { + "arm": "pooled-lora", + "cases": 800, + "failed": 0, + "nll_sum": 15928.5978307724, + "target_tokens": 14584, + "auxiliary_ppl": 2.98081507851928, + "generated_tokens": 204800, + "load_s": 2.415883114997996, + "elapsed_s": 8766.339935072, + "peak_process_rss_gib": 2.7172164916992188, + "raw_sha256": "6387e8ab4fe75bb6992306fee5d51d46ada57ad882a7b057a6948cb7618c815e" + }, + { + "arm": "specialist:toy_passage_ppl", + "cases": 800, + "failed": 0, + "nll_sum": 39666.036823272705, + "target_tokens": 14584, + "auxiliary_ppl": 15.177779382666184, + "generated_tokens": 173467, + "load_s": 1.462067071988713, + "elapsed_s": 8274.143313789988, + "peak_process_rss_gib": 3.045795440673828, + "raw_sha256": "a242b8753a6a9e7ea6045f0944616657b02809a43d039c022c9f9231dc6fa100" + } + ], + "metrics": [ + { + "arm": "base", + "split": "adversarial", + "domain": "adversarial_safety", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2200, + "nll_sum": 9375.638832092285, + "conditional_ppl": 70.92720111297905, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1137.3303559039923, + "likelihood_wall_s": 44.456953260989394 + }, + { + "arm": "base", + "split": "adversarial", + "domain": "executable_code", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1600, + "nll_sum": 1636.2277126312256, + "conditional_ppl": 2.7805321226042423, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 99, + "token_caps": 1, + "generated_tokens": 355, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 59.22777049101205, + "likelihood_wall_s": 47.520927870005835, + "toy_fixture_correct": 0, + "toy_fixture_reason_counts": { + "SyntaxError": 100 + } + }, + { + "arm": "base", + "split": "adversarial", + "domain": "rated_style", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1100, + "nll_sum": 1376.665683746338, + "conditional_ppl": 3.4956322406965628, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 4, + "token_caps": 87, + "generated_tokens": 22420, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 998.2879279869885, + "likelihood_wall_s": 43.31881631603028, + "human_style_status": "UNSCORED" + }, + { + "arm": "base", + "split": "adversarial", + "domain": "toy_passage_ppl", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2392, + "nll_sum": 3257.253828048706, + "conditional_ppl": 3.9029324817482336, + "ppl_scope": "primary passage metric", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1006.5744986540194, + "likelihood_wall_s": 40.814915015987935 + }, + { + "arm": "base", + "split": "heldout", + "domain": "adversarial_safety", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2200, + "nll_sum": 9409.517669677734, + "conditional_ppl": 72.0278958410971, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 951.0233468549814, + "likelihood_wall_s": 47.03973024599691 + }, + { + "arm": "base", + "split": "heldout", + "domain": "executable_code", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1600, + "nll_sum": 1605.1355695724487, + "conditional_ppl": 2.7270207992337294, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 100, + "token_caps": 0, + "generated_tokens": 100, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 70.02360526350458, + "likelihood_wall_s": 103.76810555900738, + "toy_fixture_correct": 0, + "toy_fixture_reason_counts": { + "SyntaxError": 100 + } + }, + { + "arm": "base", + "split": "heldout", + "domain": "rated_style", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1100, + "nll_sum": 1384.8027477264404, + "conditional_ppl": 3.521586466700885, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 44, + "token_caps": 0, + "generated_tokens": 940, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 129.09266675797335, + "likelihood_wall_s": 124.73281555097492, + "human_style_status": "UNSCORED" + }, + { + "arm": "base", + "split": "heldout", + "domain": "toy_passage_ppl", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2392, + "nll_sum": 3219.503444671631, + "conditional_ppl": 3.841820167055515, + "ppl_scope": "primary passage metric", + "empty_outputs": 17, + "token_caps": 50, + "generated_tokens": 14809, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 926.4132517844955, + "likelihood_wall_s": 59.454150345954986 + }, + { + "arm": "prompt-only", + "split": "adversarial", + "domain": "adversarial_safety", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2200, + "nll_sum": 9072.864723205566, + "conditional_ppl": 61.80779070084912, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 98, + "generated_tokens": 25317, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 901.7521944260006, + "likelihood_wall_s": 74.2746496700056 + }, + { + "arm": "prompt-only", + "split": "adversarial", + "domain": "executable_code", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1600, + "nll_sum": 1201.438274383545, + "conditional_ppl": 2.118903889009038, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 99, + "token_caps": 1, + "generated_tokens": 355, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 92.75815277651782, + "likelihood_wall_s": 71.88971880103054, + "toy_fixture_correct": 0, + "toy_fixture_reason_counts": { + "SyntaxError": 100 + } + }, + { + "arm": "prompt-only", + "split": "adversarial", + "domain": "rated_style", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1100, + "nll_sum": 1739.9751644134521, + "conditional_ppl": 4.863681220717054, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 100, + "token_caps": 0, + "generated_tokens": 100, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 82.07659213901206, + "likelihood_wall_s": 66.01955887098302, + "human_style_status": "UNSCORED" + }, + { + "arm": "prompt-only", + "split": "adversarial", + "domain": "toy_passage_ppl", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2392, + "nll_sum": 3486.539577484131, + "conditional_ppl": 4.2955664618092655, + "ppl_scope": "primary passage metric", + "empty_outputs": 0, + "token_caps": 50, + "generated_tokens": 15050, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 934.7024476075057, + "likelihood_wall_s": 84.8191002990352 + }, + { + "arm": "prompt-only", + "split": "heldout", + "domain": "adversarial_safety", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2200, + "nll_sum": 9012.236511230469, + "conditional_ppl": 60.127730521178634, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 98, + "generated_tokens": 25271, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1192.2663859215027, + "likelihood_wall_s": 80.1111257329685 + }, + { + "arm": "prompt-only", + "split": "heldout", + "domain": "executable_code", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1600, + "nll_sum": 1221.4140138626099, + "conditional_ppl": 2.145523886445081, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 94, + "token_caps": 5, + "generated_tokens": 1520, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 235.56672508599513, + "likelihood_wall_s": 77.26135136897938, + "toy_fixture_correct": 0, + "toy_fixture_reason_counts": { + "SyntaxError": 100 + } + }, + { + "arm": "prompt-only", + "split": "heldout", + "domain": "rated_style", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1100, + "nll_sum": 1836.0035381317139, + "conditional_ppl": 5.307357819175553, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 100, + "token_caps": 0, + "generated_tokens": 100, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 103.18142872049793, + "likelihood_wall_s": 72.51974701200379, + "human_style_status": "UNSCORED" + }, + { + "arm": "prompt-only", + "split": "heldout", + "domain": "toy_passage_ppl", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2392, + "nll_sum": 3425.6334190368652, + "conditional_ppl": 4.187571596097851, + "ppl_scope": "primary passage metric", + "empty_outputs": 0, + "token_caps": 62, + "generated_tokens": 16835, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 817.4928048829934, + "likelihood_wall_s": 110.21971568196022 + }, + { + "arm": "pooled-lora", + "split": "adversarial", + "domain": "adversarial_safety", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2200, + "nll_sum": 1937.3984909057617, + "conditional_ppl": 2.412432748768778, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1428.010236298498, + "likelihood_wall_s": 66.59818012098549 + }, + { + "arm": "pooled-lora", + "split": "adversarial", + "domain": "executable_code", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1600, + "nll_sum": 2730.7489051818848, + "conditional_ppl": 5.510845533450696, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 867.7118000255032, + "likelihood_wall_s": 38.94628717901651, + "toy_fixture_correct": 0, + "toy_fixture_reason_counts": { + "SyntaxError": 100 + } + }, + { + "arm": "pooled-lora", + "split": "adversarial", + "domain": "rated_style", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1100, + "nll_sum": 1112.6578283309937, + "conditional_ppl": 2.749742075807855, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1052.4930000164968, + "likelihood_wall_s": 53.51156676300161, + "human_style_status": "UNSCORED" + }, + { + "arm": "pooled-lora", + "split": "adversarial", + "domain": "toy_passage_ppl", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2392, + "nll_sum": 2189.485610961914, + "conditional_ppl": 2.497616292151115, + "ppl_scope": "primary passage metric", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1187.0565325924945, + "likelihood_wall_s": 58.507729999946605 + }, + { + "arm": "pooled-lora", + "split": "heldout", + "domain": "adversarial_safety", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2200, + "nll_sum": 1939.40602684021, + "conditional_ppl": 2.41463513776146, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 914.8020426825169, + "likelihood_wall_s": 71.1393145500333 + }, + { + "arm": "pooled-lora", + "split": "heldout", + "domain": "executable_code", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1600, + "nll_sum": 2674.37003326416, + "conditional_ppl": 5.320042386931288, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 892.032683485988, + "likelihood_wall_s": 43.67210004502704, + "toy_fixture_correct": 0, + "toy_fixture_reason_counts": { + "SyntaxError": 100 + } + }, + { + "arm": "pooled-lora", + "split": "heldout", + "domain": "rated_style", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1100, + "nll_sum": 1103.606357574463, + "conditional_ppl": 2.7272083590016774, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 786.9312188424883, + "likelihood_wall_s": 35.67333869900176, + "human_style_status": "UNSCORED" + }, + { + "arm": "pooled-lora", + "split": "heldout", + "domain": "toy_passage_ppl", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2392, + "nll_sum": 2240.9245777130127, + "conditional_ppl": 2.5519081632368392, + "ppl_scope": "primary passage metric", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1205.4356382059923, + "likelihood_wall_s": 61.273281688962015 + }, + { + "arm": "specialist:toy_passage_ppl", + "split": "adversarial", + "domain": "adversarial_safety", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2200, + "nll_sum": 11980.30924987793, + "conditional_ppl": 231.73514741797848, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 98, + "generated_tokens": 25370, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 881.2227828344912, + "likelihood_wall_s": 48.96416755599785 + }, + { + "arm": "specialist:toy_passage_ppl", + "split": "adversarial", + "domain": "executable_code", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1600, + "nll_sum": 2418.752899169922, + "conditional_ppl": 4.534526019378983, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 99, + "generated_tokens": 25488, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1021.8482535500516, + "likelihood_wall_s": 47.42844058494666, + "toy_fixture_correct": 0, + "toy_fixture_reason_counts": { + "SyntaxError": 100 + } + }, + { + "arm": "specialist:toy_passage_ppl", + "split": "adversarial", + "domain": "rated_style", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1100, + "nll_sum": 3509.7286376953125, + "conditional_ppl": 24.304521376249806, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 87, + "generated_tokens": 23653, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1411.0405385854829, + "likelihood_wall_s": 56.32380813601776, + "human_style_status": "UNSCORED" + }, + { + "arm": "specialist:toy_passage_ppl", + "split": "adversarial", + "domain": "toy_passage_ppl", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2392, + "nll_sum": 1820.6879386901855, + "conditional_ppl": 2.1407519910698865, + "ppl_scope": "primary passage metric", + "empty_outputs": 0, + "token_caps": 100, + "generated_tokens": 25600, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 784.8626054129854, + "likelihood_wall_s": 41.58937336906092 + }, + { + "arm": "specialist:toy_passage_ppl", + "split": "heldout", + "domain": "adversarial_safety", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2200, + "nll_sum": 12182.442970275879, + "conditional_ppl": 254.03551431899322, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 65, + "generated_tokens": 21722, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1279.9168546435103, + "likelihood_wall_s": 54.06470469906344 + }, + { + "arm": "specialist:toy_passage_ppl", + "split": "heldout", + "domain": "executable_code", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1600, + "nll_sum": 2269.5427589416504, + "conditional_ppl": 4.1307716277943785, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 85, + "generated_tokens": 23246, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 883.6466341960186, + "likelihood_wall_s": 40.90240699995775, + "toy_fixture_correct": 0, + "toy_fixture_reason_counts": { + "SyntaxError": 100 + } + }, + { + "arm": "specialist:toy_passage_ppl", + "split": "heldout", + "domain": "rated_style", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 1100, + "nll_sum": 3509.466079711914, + "conditional_ppl": 24.298720844793625, + "ppl_scope": "auxiliary reference likelihood, not task quality", + "empty_outputs": 0, + "token_caps": 54, + "generated_tokens": 17517, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 1046.372410813492, + "likelihood_wall_s": 45.69102746300632, + "human_style_status": "UNSCORED" + }, + { + "arm": "specialist:toy_passage_ppl", + "split": "heldout", + "domain": "toy_passage_ppl", + "cases": 100, + "runtime_failures": 0, + "likelihood_failures": 0, + "reference_tokens": 2392, + "nll_sum": 1975.106288909912, + "conditional_ppl": 2.283509083418356, + "ppl_scope": "primary passage metric", + "empty_outputs": 0, + "token_caps": 31, + "generated_tokens": 10871, + "exact_reference_matches_diagnostic": 0, + "generation_allocated_wall_s": 582.712517107997, + "likelihood_wall_s": 45.88701859302819 + } + ], + "cost_policy": "Allocate shared batch wall equally to rows. Arm elapsed includes load/overhead; RSS is cumulative process high-water mark, not per-arm increment.", + "limits": [ + "Authored template rows do not establish independent source families.", + "Saved adapters are seed17, not three independently trained seeds.", + "No actual router/abstention/safety-dispatch or human style result.", + "No confidence interval for independent-source generalization from numbered templates.", + "Active remaining tapes excluded; no decoding, model, budget or protocol change." + ] +}