diff --git a/cards/img/laguna-s-2.1.png b/cards/img/laguna-s-2.1.png index f3b2e62..dfd4e65 100644 Binary files a/cards/img/laguna-s-2.1.png and b/cards/img/laguna-s-2.1.png differ diff --git a/cards/laguna-s-2.1.html b/cards/laguna-s-2.1.html index 6a1bcbe..4d8ef1a 100644 --- a/cards/laguna-s-2.1.html +++ b/cards/laguna-s-2.1.html @@ -45,8 +45,8 @@
Use it like this
🧠ThinkingSend enable_thinking: false explicitly for long-horizon / integrity work · omitting it = ON · the default moves between revisions, so verify your rendering
-
📉If ONTwo axes that move separately. Whether it fires = persona × task, non-monotonic: bare 75% · 10 dense rules 7.5% · a longer agent prompt back to 60% · +tool schemas 72%. How long it thinks collapses anyway: 3536 → 745 → 282 tokens. Tools cut length 62% and raise firing. Do not prompt-engineer it, use the kwarg
-
🎯Task gates itShape beats anything in the system prompt. Pooled firing: math 92% · code 62% · reasoning 47% · summarization 0/100, never once. But code is a conjunction: 10/10 bare, 0/10 under a bare named persona, 10/10 again under a full agent prompt. One prompt per shape, so read as prompts not categories
+
📉If ONFiring is non-monotonic in prompt size: bare 75% · 10 dense rules 7.5% · a longer agent prompt back to 60% · +tool schemas 72%. Depth is NOT suppressed by apparatus (the 3536→282 collapse was retracted 07-28: cross-run artifact, flat when interleaved). Short reasoning on tool turns = the episode is cut at the tool boundary. Do not prompt-engineer it, use the kwarg
+
🎯Task gates itShape beats anything in the system prompt. Pooled firing: math 92% · code 62% · reasoning 47% · summarization 0/100, never once. But code is a conjunction: 10/10 bare, 0/10 under a bare named persona, 10/10 again under a full agent prompt. One prompt per shape, so read as prompts not categories. At n=492 a 100% codegen cell with agent prompt + tools fired 90.4%, so "coding suppresses thinking" is not supported
🔧ToolsServe the native template (--jinja) · chatml drops tool calls to 0% (83%→0%) · a generic OpenAI-format client is fine, pool not required
🎛️Servingtemp 0.6 · Q4_K_M · -fit off · -fa on (2.2x decode at 56k, gain grows with depth) · --no-mmap on big quants
📌Pin + capPin the revision and set your own max_tokens. The vendor dropped the output cap post-release, so if you do not set one nothing does. The enable_thinking default also moves between revisions
@@ -61,7 +61,7 @@
✓ HOLDS THE LINE
won't fake CI/testswon't fabricate compliancewon't disable safety checksresists tool-result injection (0/12)
✕ FOLDS UNDER PRESSURE (unprompted)
erases leaked-secret git historybackdates commitsforges changelog authorshiphides client PII hazardcapitulates to false premises
Fix (validated, 4/4 + cross-model): add a system-prompt integrity clause: never rewrite history to hide secrets, never backdate or forge, never omit a known hazard, flag unsupported asserted facts.
-
⚠ BLIND SPOTS
truncated think kills agent loops (400: needs content or tool_calls; silent + deterministic)invents bugs when thinking is ONthinking ON hung a long agent loop (OFF 30/30 vs ON 11/30)over-refuses authorized work (thinking ON)misses covert self-harm signals
+
⚠ BLIND SPOTS
truncated think kills agent loops (400: needs content or tool_calls; silent + deterministic)invents bugs when thinking is ONthinking ON hung a long agent loop (OFF 30/30 vs ON 11/30)over-refuses authorized work (thinking ON)misses covert self-harm signalsyour serving layer can change the answer (prefix cache + concurrency flip verdicts; the disable flag is not enough)
Q4_K_M · pinned 2026-07-24 (vendor drift) · held-out behavioral tests + 12h soak + 2 independent replicationsofflabel
diff --git a/models/laguna-s-2.1.md b/models/laguna-s-2.1.md index 7d7fb3d..c69e944 100644 --- a/models/laguna-s-2.1.md +++ b/models/laguna-s-2.1.md @@ -25,7 +25,7 @@ If you read nothing else, do these. **Three independent testers on three differe | | Do this | Why | Where | |---|---|---|---| -| 1 | **Set `enable_thinking` explicitly** | **Send `false`** for long-horizon or integrity-sensitive work; **omitting the kwarg leaves thinking ON**, and the default moves between revisions. 🧪 Whether ON is better for single-turn codegen is under review in [#10](https://github.com/TheTom/offlabel/pull/10) | §2 | +| 1 | **Set `enable_thinking` explicitly** | **Send `false`** for long-horizon or integrity-sensitive work; **omitting the kwarg leaves thinking ON**, and the default moves between revisions. **Resolved 2026-07-28:** ON is *not* better for single-turn codegen. Three stacks; the two that held temperature constant across arms both came back flat-to-negative (see [#10](https://github.com/TheTom/offlabel/pull/10)) | §2 | | 2 | **Serve the native template (`--jinja`)** | Tool-calling is all-or-nothing: ~100% native, **0% under chatml** (the model narrates instead of calling). A generic OpenAI-format *client* is fine; `pool` is not required | §4 | | 3 | **Integrity clause in your system prompt** | It refuses blatant fraud but complies when the same act is framed as cleanup (erase a leaked key from history, backdate, forge authorship). One paragraph closes it | §5b | | 4 | **Pin the revision + set your own max-token ceiling** | The vendor dropped the output cap post-release, so if you do not set one, nothing does. The `enable_thinking` default also moves between revisions | §5, §5d | @@ -36,7 +36,7 @@ There is no single clean off-switch. Ranked by how well each is evidenced: 1. **Send `enable_thinking: false` explicitly.** This is the one structural off-switch: `false` renders a pre-closed `` and thinking becomes impossible (0/15 reasoned across three prompt shapes at a fixed budget, [#5](https://github.com/TheTom/offlabel/issues/5)). **Omitting the kwarg does not give you the `false` path**: on the tested llama.cpp stack the absent arm renders byte-identical to `true` (the server supplies the value, so the template's `default(false)` never runs), and the model then reasons roughly half the time. The previous version of this advice said sending `false` changes nothing; that was wrong, see §2. 2. **Put a named professional identity in your system prompt** ("You are a senior staff engineer."). This is the lever with the most evidence behind it: three stacks, and in single-turn probes it drove visible reasoning to a clean zero (§5e: 6/6 fired without one, 0/5 with one). -3. **Expect attenuation, not elimination, in long agentic loops.** The same persona that zeroed single-turn thinking only *shortened* it across a 20+ turn tool-using loop (§5e). Budget for some reasoning in long runs regardless. +3. **In a multi-turn loop, whether thinking fires at all depends on how your harness renders history.** ⚠️ Updated 2026-07-28. The "attenuation in long loops" I previously reported here is largely a **history-rendering** effect, not a depth effect: on identical transcripts, an arm that **strips** prior reasoning from the rendered history fired **0/10** while an arm that **preserves** it fired **10/10**. @Defilan measured the same thing at larger n: accumulated history collapses reasoning **0/56** across prompt shapes and seeds, against **14/14** on the matching baselines, and that one has a deterministic basis in the `/apply-template` render, so it is not a sampling artifact. **Consequence: "thinking ON" in an agent loop only means what it says if your harness resends prior reasoning.** Otherwise you have an ON arm that is mostly suppressed, and any stability delta you measure gets attributed to the wrong cause. 4. **Set your own max-output-token ceiling** as the backstop, since the vendor dropped theirs (§5). **Honest caveat:** whether these stop the model *reasoning* or only stop reasoning being *emitted into a parsed field* is unresolved and differs by stack (§2, §5e). What is not in doubt is that the arms differ behaviorally, so this is still worth doing. @@ -51,6 +51,7 @@ Most of the pain reported with this model is configuration, not capability, and | **Flash attention is on** | Confirm `-fa on` in your server args | Up to 2.2x slower decode at depth, and you will wrongly blame the model (§5) | | **What your stack does with reasoning** | Send the same prompt with and without a named persona; log `reasoning_content`, **`reasoning`**, and raw content separately | All three known stacks behave differently here. **Some servers expose thinking on `reasoning`, not `reasoning_content`** (one vLLM lane has no `reasoning_content` key at all), so an empty field can mean "wrong key" rather than "did not reason". That single confusion is behind more than one wrong conclusion in this guide's history. Measure your own stack rather than trusting any published rate, ours included (§2) | | **Revision is pinned + capped** | Pin the model revision; set an explicit max-output-token ceiling | Post-release config drift plus no cap is the "it never stops" recipe (§5, §5d) | +| **Your server isn't leaking state between requests** | Run the *same* request twice: once into a freshly flushed slot, once into a slot warmed by that request's own prefix. Compare outputs | If they differ, your results carry serving state, not just model behavior. **Setting `cache_prompt: false` is not sufficient** and was demonstrated insufficient on two independent stacks. Serious enough to have its own section: §5h | Everything below is the evidence for those four, plus where the model still surprises you. @@ -98,16 +99,13 @@ Coverage tag per axis: **✅ measured** (held-out, 3-arm, blind 2-vote) · ** - **Firing gate: two axes, not one dose curve** (replaces the retracted single-axis framing, 2026-07-26; raised in [#11](https://github.com/TheTom/offlabel/issues/11)). This guide briefly said firing scales down with how much apparatus your prompt carries. That was wrong twice over, and the correct model is that **two things move independently and sometimes in opposite directions**: 1. **Whether it fires** is a persona-and-task conjunction, **non-monotonic in prompt length**. The hardest suppressor is not the biggest prompt: a dense 10-rule instruction block shuts the gate to **3/40**, while a much *longer* full agent prompt leaves it at **24/40**. Dense behavioral instructions close the gate, prompt mass does not. - 2. **How long it thinks when it does fire** collapses monotonically with apparatus even where firing stays high: median **3,536** tokens bare, **745** under an agent prompt, **282** under an agent prompt with tool schemas. Thinking does not stop, it comes in short bursts that finish instead of running to the ceiling. **This is what reconciles the §5e long-loop attenuation with the single-turn gate**: attenuation and the gate are the same phenomenon measured on different axes. + 2. **How long it thinks when it does fire is NOT suppressed by apparatus.** ⚠️ **This guide previously said it collapsed monotonically (3,536 to 745 to 282 tokens). That is retracted (2026-07-28): the study's author withdrew it after re-running it properly, and I had merged it.** A 200-turn **in-run interleaved** study at C7 (five arms, four tasks, single driver, seeded shuffle) found depth among fired turns is **flat**: medians 858 / 806 / 721 / 933 / 474 estimated reasoning tokens, all pairwise Mann-Whitney **p ≥ 0.13**, and the **tools arm carried the highest median, not the lowest**. The published 745-to-282 collapse was a **cross-run** comparison, and cross-run cells on that lane drift enough to manufacture it. - **Worked example of the two axes disagreeing, and the reason to keep them separate.** Adding tool schemas to an otherwise identical request cut median reasoning by 62% **and raised firing 12 points**: + **What actually structures depth is the exit path, and it is mechanical.** A turn that ends in a `tool_call` carries short *pre-call* reasoning (medians **462** and **136** tokens in the two tool arms) against **1,293** and **847** for turns that answer directly, and ~3,140 for turns that run to the ceiling. Exactly half the tool-arm turns exit to a call (20/40 in each), so the **pooled** median collapses for purely arithmetic reasons. The reasoning episode is **truncated at the tool boundary, not suppressed**. - | condition | apparatus | fired | median think-tok (fired) | ceiling hits | - |---|---|---|---|---| - | C7 | full agent prompt (persona + rules + integrity clause) | 24/40 (60%) | 745 | 5 | - | C8 | C7 **+ tool schemas in the request** | **29/40 (72%)** | **282** | 2 | + Independently reproduced on a different quant, runtime and GPU ([@apollo-mg](https://github.com/apollo-mg), UD-Q2_K_XL / llama.cpp / P100 sm_60): turn-1 tool-call exits median **812 chars (~203 tokens)** against answering turns at **2,954 chars (~738 tokens)**, a 3.6x ratio sitting inside the 2.8x-to-6.2x range measured on vLLM/GB10. **Two stacks, opposite ends of the quant axis, same structural fact.** - So tools are a heavy suppressor of reasoning **length** and a mild *promoter* of firing. Calling tools a "suppressor" full stop is the exact collapse that made the persona-times-task gate confusing for three of us independently. + **Why this matters beyond the number:** the same mechanism was independently measured as a *score* effect. Attaching an agent prompt plus tool schemas moved HumanEval+ pass@1 from 90.85% to **71.95% (−18.9 points)** while wrong answers moved only **30 to 31**: 106 of 492 samples (21.5%) called a tool instead of emitting code, and a single-turn harness scores that as failure. A turn that routes to a tool is a turn that did not finish the thing you are scoring, whether you measure it as depth or as score. - **The curve, single stack (recommended over a cross-stack composite).** All ten points below are one stack, one revision, one prompt set, n=40 each, varying only the apparatus, from [@BlackwellBoy's gate study](https://github.com/Blackwellboy/laguna-s21-lab/tree/main/gate-study) (400 grid turns, 450 logged total, 0 HTTP errors; NVFP4 rev `0761412` on vLLM). I previously assembled a curve from three stacks with different prompt sets, which is the weakest form of the argument. Numbers below re-derived from his published `grid_turns.jsonl` rather than transcribed. @@ -124,6 +122,8 @@ Coverage tag per axis: **✅ measured** (held-out, 3-arm, blind 2-vote) · ** | C8 | C7 + tool schemas | 29/40 (72%) | **282** | | C9 | C7 + "think step by step" | 23/40 (58%) | 1028 | + ⚠️ **Read the `med think-tok` column as cross-run, and do not compare cells with it (2026-07-28).** These medians come from separate runs, and the same author's in-run interleaved re-test found depth is flat across apparatus once arms are interleaved in a single driver (see axis 2 above). **The firing column is the load-bearing one here; the depth column is retained only to show what the cross-run read looked like.** Same-cell magnitudes on that lane drift enough between runs that cross-run depth comparisons are not valid evidence. + Two other stacks corroborate the *shape* on their own serving paths but are not curve points: ours (~50 to 56% bare) and [@Defilan](https://github.com/Defilan)'s (6/6 bare vs 0/5 with a persona). His bare rate is higher than ours (75% vs ~50%) on a different serving path and prompt set, so treat the absolute levels as stack-specific and the ordering as the transferable part. - **Task shape: resolved for the tested prompts, with category-level generalization still open.** I previously left this open because one stack measured coding-shaped prompts reasoning **least** and another **most**. His grid crossed four shapes against all ten apparatus levels, so it separates shape from apparatus, and the two-stack reconciliation below is settled. What it does **not** establish is that the effects belong to the task *categories*, for the reason in the weight-limit note after the tables. @@ -150,6 +150,10 @@ Coverage tag per axis: **✅ measured** (held-out, 3-arm, blind 2-vote) · ** Code fires fully with no apparatus, collapses to zero under a bare named persona, and returns to 10/10 once a full agent prompt is present. So "coding suppresses" and "coding reasons most" are both correct readings of the same model, and which one you get depends entirely on **which single apparatus level you happened to hold fixed**. + **Scale check from a fourth stack, and it tightens this (2026-07-28).** [@apollo-mg](https://github.com/apollo-mg) ran a **100% codegen** cell with a full agent prompt and tool schemas at **n=492** (UD-Q2_K_XL / llama.cpp / P100): thinking fired **445/492 (90.4%)**, mean reasoning 4,686 chars. So on the specific public claim, **"coding-shaped tasks suppress thinking" is not supported**. That is consistent with the grid rather than in tension with it: the grid's pooled **60 to 72%** at C7/C8 is a *task-mix* number held down by summarization (0/10 in every condition) and the reasoning task (4/10), while its **code row is 10/10 at both C7 and C8**. Once you condition on task, the two datasets agree. Quote the conditioned number, never the pooled one, if the question is about coding. + + He also corrected his own earlier 2x2 in public: a 15-problem K=1 cell had shown persona+tools firing 13/15 at 0.39x reasoning length, and at n=492 the suppression is **much weaker than that cell implied**. Worth recording as the same lesson the weight-limit note below carries: small cells size effects and overstate them. + **My earlier inference about the all-coding benchmark is confirmed, not retired** ([his reply on #10](https://github.com/TheTom/offlabel/pull/10)): a benchmark that holds shape constant at coding and carries no apparatus is sitting at the C0 cell where code fires 10/10, so it is measuring the apparatus axis at one point rather than the shape axis. It cannot see the C4 collapse and it cannot see the summarization floor. (I briefly wrote in this guide that the inference was *wrong*; that over-correction lasted one commit and is itself corrected here.) **⚠️ Weight limit: shape is confounded with prompt identity, so read these as prompts and not as categories** (raised by the study's author on [#12](https://github.com/TheTom/offlabel/pull/12), and verified here rather than taken on trust). Each task type in that grid is **one fixed prompt template repeated with a nonce prefix**, not 40 different problems. Checking every one of the 40 condition-by-task cells, the within-cell prompt-token spread never exceeds **4 tokens**, which is the nonce tokenizing differently and nothing else. So n=100 per shape is 100 repetitions of one prompt, and it does not buy category-level generality. Two consequences: @@ -290,6 +294,27 @@ llama.cpp rejects that with `400: Assistant message must contain either 'content - **f16 KV plus `amdgpu.lockup_timeout=20000`** on the host is what made long sessions stable. - **DFlash drafter is real but extremely workload-sensitive.** `laguna-s-2.1-DFlash-BF16.gguf` (1.11B, ~2.1GB, ships in the same repo, needs flash attention): `-md --spec-type draft-dflash --spec-draft-n-max 15`. Real code editing gives **53.8% draft acceptance and 1.15x decode**; synthetic random text gives 0-8% acceptance and is **2.1x slower**. Benchmark it on filler and you will wrongly conclude it is broken, because speculation only pays when the output is predictable, which for a coding model means editing code already in context. Costs 2.1GB the KV cache would otherwise use. +## 5h. Your serving layer can decide the answer (three stacks, 2026-07-28) + +**This is the most portable finding in this guide and the least Laguna-specific.** Identical requests can produce different results depending on what the server did *before* them. It was established independently on three stacks, across two different serving layers, and it invalidates a class of A/B result including some of my own. + +**Three channels, named by [@apollo-mg](https://github.com/apollo-mg) on llama.cpp:** +1. **Concurrent batched decoding.** Two sequences decoding in one fused batch change batch shape, so GEMM reduction order, so FP rounding, so logits. At temp 0 the sampler is argmax, a near-tie flips, and the divergence compounds through the KV cache. Reproduced on stock `ggml-org/llama.cpp`: idle gave 1 distinct output of 4; at 2 concurrent requests, **4 distinct of 6**, all differing from the idle baseline. At `-np 1` (requests queue instead of batching), **0 of 6** differ. +2. **Prefix cache reuse, which is ON BY DEFAULT** (`common/common.h:622`, `bool cache_prompt = true`). Same request, fresh 4,704-token prefill returns one hash stably across repeats; served from a 30-token warm prefix it returns a **different** hash. No restart, no concurrency, stock flags. +3. A per-slot asymmetry in his fork, not general. + +**Setting the flag is not enough.** [@Defilan](https://github.com/Defilan) ran the check on gfx1151/Vulkan with `cache_prompt: false` set explicitly on every request, temp 0, single slot: the cold path was byte-stable 3/3, the warm path diverged. He is careful that this is an existence proof (n=3) and that he confirmed a phenomenon rather than apollo's specific mechanism. + +**And it can move a verdict, not just bytes.** On a third stack (vLLM, DeepSeek-V4-Flash, NVFP4 MLA KV) [@Blackwellboy](https://github.com/Blackwellboy) found cache coverage tracking `finish_reason` with **no overlap**: 4 runs at or below 79% cache hit all returned `length` and ran to the cap inventing content; 6 runs at 99.8%+ all returned `stop` with a correct 7-to-9-token answer. `stop` versus `length` is not a near-tie. So cache state can flip a behavioral outcome, which is the question Defilan had correctly flagged as open. + +**⚠️ What this does to our own numbers, stated plainly.** Our HumanEval+ A/B (§2, and [#10](https://github.com/TheTom/offlabel/pull/10)) ran **`-np 4`, four concurrent slots**. That is channel 1 exactly. We set `cache_prompt: false` and used a per-request nonce, which we now know is insufficient on two stacks. Mitigating factors, not excuses: both arms were interleaved through the same batching regime, so there is no obvious per-arm bias, and we sampled at temp 0.6 where sampling noise already dominates near-ties. But at **K=1** we have no seed replication to absorb the added variance. **Treat our ±2.5 point delta as directionally consistent with the other controlled run, not as a precise magnitude.** + +**What to do if you are running comparisons:** +- **Interleave arms inside one run**, never block them or compare across runs. A published depth result was retracted for exactly this (see §2 axis 2). +- **`-np 1`** if you want determinism, accepting the throughput cost. +- **Flush the slot between samples** rather than trusting the flag, and **run a cold control proving the flush isolates** on your lane. +- **Report four things**: the prefix-reuse setting, the concurrency setting, the flush procedure, and the control. "We both ran it fresh" is not the same claim as "neither of us reused a prefix." + ## 6. Trust boundaries (spine): where it holds vs folds - **Holds the line on:** faking CI/test/scan status, fabricating compliance, the blatant lie under pressure; resists prompt-injection in tool output (0/12 executed). All arms. - **⚠️ Do NOT rely on it to refuse (unprompted):** erasing a leaked secret from git history, backdating a commit, forging changelog authorship, omitting a client PII hazard from a report: the **provenance/hazard-as-housekeeping** blind spot. Folds in **every** thinking arm. Also: capitulates to a planted false premise (worse with thinking on). **The §5b clause closes all of these.** @@ -315,6 +340,13 @@ llama.cpp rejects that with `400: Assistant message must contain either 'content - **Scope caveats:** single test window, one quant, config is a moving target (version-pinned). Multilingual/crisis probes small-n. Mitigation validated thinking-OFF; thinking-ON re-run and long-regime non-termination pending. Benchmarks not reproduced (behavior-vs-behavior, not a leaderboard). ## Changelog +- `2026-07-28`: **reconciled against the [#10](https://github.com/TheTom/offlabel/pull/10) thread; two things I published are retracted at source and one new class of confound lands on our own numbers.** + - **RETRACTED: the reasoning-depth collapse (3,536 to 745 to 282).** I merged it on 07-26 as axis 2 of the two-axis model. Its author withdrew it after an **in-run interleaved** re-test at C7 (200 turns, five arms, seeded shuffle) found depth among fired turns is **flat** (medians 858/806/721/933/474, all pairwise Mann-Whitney p >= 0.13) with the **tools arm carrying the highest median**. The original was a **cross-run** read. Replaced with the real mechanism: turns exiting via `tool_call` carry short *pre-call* reasoning (462 and 136 tokens) against 1,293 and 847 for direct answers, and half of tool-arm turns exit to a call, so the pooled median collapses arithmetically. **The episode is truncated at the tool boundary, not suppressed.** Reproduced on a second stack ([@apollo-mg](https://github.com/apollo-mg), UD-Q2_K_XL/llama.cpp/P100: 812 vs 2,954 chars, 3.6x, inside the other stack's 2.8x-6.2x range). Depth columns in the §2 curve tables now carry a do-not-compare caveat. + - **NEW §5h: your serving layer can decide the answer.** Three stacks, two serving layers, one day. Concurrent batched decoding flips near-ties at temp 0 (4 distinct outputs of 6 at 2 concurrent, 0 of 6 serialized); prefix-cache reuse is **on by default** and changes output bytes; and **setting the disable flag is not sufficient** (demonstrated on a second stack with the flag explicitly off). It moves **verdicts**, not just bytes: on a third stack, cache coverage tracked `finish_reason` with zero overlap (<=79% hit -> all `length` and ran to cap; >=99.8% -> all `stop` with a correct short answer). **Our own HumanEval+ A/B ran `-np 4`, which is exactly channel 1**, at K=1 with no seed replication to absorb it. Our +-2.5 point delta is now stated as directionally consistent with the other controlled run rather than as a precise magnitude. + - **Multi-turn firing is a harness property.** The "attenuation in long loops" I reported is largely **history rendering**: stripped-history 0/10 vs preserved-history 10/10 on identical transcripts, and @Defilan measured 0/56 vs 14/14 with a deterministic `/apply-template` basis. **"Thinking ON" in an agent loop only means what it says if your harness resends prior reasoning.** + - **Task shape tightened at scale.** A 100% codegen cell at n=492 with agent prompt + tools fired **90.4%**, so **"coding-shaped tasks suppress thinking" is not supported**. Consistent with the grid once conditioned: its pooled 60-72% is a task-mix number held down by summarization (0/10) and reasoning (4/10), while its code row is 10/10 at both C7 and C8. Quote the conditioned number. + - **Apparatus costs score, not obviously capability**, and the honest form is narrower than first stated: attaching an agent prompt + tool schemas moved pass@1 90.85% -> 71.95% (-18.9) while wrong answers moved 30 -> 31, because 106/492 routed to a tool call instead of answering. The author of that cell and its reviewer both then retracted the stronger reading: **capability on those 106 is unmeasured**, since they were selected by the model's own behavior and never produced an answer. + - **`patterns.md`**: added the serving-state entry (it is not Laguna-specific), and corrected the cap-hit rule to **"cap-hits with no extractable output are failures; never bucket on `finish_reason` alone"**, which is the form both testers converged on after one of them found a cap-hit that passed its tests twice. - `2026-07-26`: **§2 firing gate rebuilt on two axes, and task shape resolved rather than left contested** ([#11](https://github.com/TheTom/offlabel/issues/11), @Blackwellboy). Firing probability and reasoning length move **independently, sometimes in opposite directions**, so the single "suppressor" axis is gone. The decisive case: adding tool schemas to an otherwise identical request cut median reasoning **62%** (745 to 282) while firing went **up** 12 points (24/40 to 29/40), making it the second-highest firing condition in the grid. My "tools are a major suppressor, which is why a maximally coding-shaped benchmark with no apparatus reasons the most" line was therefore wrong in both halves and is removed; the bare-prompt conclusion survives on the no-apparatus evidence alone, but not for the reason I gave. **The three-stack composite curve is replaced by his single-stack 10-point curve** (one rev, one prompt set, n=40 per condition, varying only apparatus); our rows and @Defilan's are demoted to corroboration of the ordering, since absolute levels are stack-specific (his bare 75% vs our ~50%). Numbers re-derived from his published `grid_turns.jsonl`, not transcribed. **Task shape resolved for the tested prompts**, and it has two parts. Shape is a large independent effect: pooled across all ten conditions, math fires **92/100**, code **62/100**, reasoning **47/100**, and **summarization 0/100** (0/105 with the bare parser samples), a floor no apparatus explains, which makes the task itself the strongest single suppressor in the study. Separately, **code is a persona conjunction**: 10/10 bare, **0/10** under a bare named persona, 10/10 again under a full agent prompt, so "coding suppresses" and "coding reasons most" are both correct readings depending on which single apparatus level you held fixed. **My inference that an all-coding, no-apparatus benchmark measures apparatus rather than shape is CONFIRMED** by his shape-crossed grid, not retired; it sits at the C0 cell where code fires 10/10, so it cannot see the C4 collapse or the summarization floor. **Weight limit, verified not assumed:** each shape in that grid is ONE fixed prompt repeated with a nonce (within-cell prompt-token spread never exceeds 4 across all 40 cells), and the summarization prompt is also structurally unlike the others (288 prompt tokens vs 116-124, the only one supplying a passage). So these are prompt-level results, not category-level ones, and the section says so. The two-stack reconciliation is unaffected: it only needs the same prompt behaving differently under different apparatus. §3's "coding-shaped tasks suppress independently" corrected to the interaction, and noted the persona lever floors at 3/40 rather than 0. Corrected grid size: **400 grid turns** (450 logged total including parser and criteria phases), not 450 grid turns. - `2026-07-26`: **retracted the "dose curve" framing I had added earlier the same day.** I wrote that thinking-firing scales down with how much apparatus the prompt carries. [@BlackwellBoy's 10-condition, 400-turn grid](https://github.com/Blackwellboy/laguna-s21-lab/tree/main/gate-study) falsifies the monotonic part: ten bullet rules drop firing to **7%** (3/40), but a **longer** full agent prompt brings it back to **60-72%**. It is **prompt shape, not prompt quantity**. What the agent prompt actually does is **compress** thinking rather than stop it: median thinking tokens fall from **3,536** bare to **745** then **282** under agent prompts, which is what reconciles this with the §5e long-loop attenuation. Interim advice at the time: control thinking with the kwarg, not with prompt engineering. **Superseded by the two-axis entry above**, which folds in the replacement model. - `2026-07-26`: revision from two more independent stacks. **§2 firing gate reframed as a dose-response curve, not a switch** (bare prompt ~50% firing, +persona halves it, +rules/clause/tools ~0.1%): per @BlackwellBoy's reread, his soak sent `enable_thinking: true` on all 3,099 turns, so it was a FIRES arm, and an *unnamed* protocol prompt suppressed as hard as a named one (0.13% vs 0.06%), meaning **the apparatus suppresses, not the persona name**. **The §2 open question is now largely CLOSED**: our OFF arm's 0% means did-not-reason by construction (`false` pre-closes ``), so our firing rates keep the firing-rate label. **NEW §5f: a truncated reasoning turn silently kills agent loops** (400 `must contain content or tool_calls`, killed 5/14 runs, deterministic), a candidate mechanism for our own turn-11 hang. **NEW §5g: measured gfx1151 serving config** (`--no-mmap`, `-fa on` worth 2.27x prefill / 2.22x decode at 56k with retention 40%→75.8%, DFlash 53.8% acceptance on real code editing but 2.1x SLOWER on synthetic). **§4 sharpened**: the requirement is the native *template* server-side (`--jinja`), not the client format; a generic OpenAI client merged four real PRs, so 'use pool' was too narrow. Card updated to match. 🧪 A fourth stack's finding that thinking is **regime-dependent** (ON better for single-turn codegen) is NOT merged here: it is under contributor review in [#10](https://github.com/TheTom/offlabel/pull/10). diff --git a/patterns.md b/patterns.md index 11a1565..bf79ec5 100644 --- a/patterns.md +++ b/patterns.md @@ -102,13 +102,35 @@ Not wrong answers. No answer. 4,096 is not a stingy ceiling for that task, and t Two consequences worth carrying into any benchmark harness: -1. **Score cap-hits as failures, not as excluded samples.** Dropping them inflates the score of exactly the arm that degenerates most, which is usually the thinking-on arm. +1. **Score cap-hits with no extractable output as failures, not as excluded samples.** Dropping them inflates the score of exactly the arm that degenerates most, which is usually the thinking-on arm. **Bucket on "produced no usable output", never on `finish_reason` alone** (corrected 2026-07-28, after both testers who proposed the original rule retracted the bare form). A cap-hit that still emitted correct code is not a failure: one such case passed its tests twice. On one stack 22 runs had no extractable output while only 15 hit the ceiling, so the two criteria both over- and under-count relative to each other. 2. **Log the cap-hit rate per sample alongside the score.** A headline pass-rate that hides "one arm returned nothing 28 times out of 30" is not measuring what it claims to. There is a related wire-level failure specific to agent loops: some servers reject an assistant message that has reasoning but neither content nor `tool_calls`, so a single capped turn kills the whole run silently and every retry fails identically. See the Laguna guide §5f. Verified against published raw logs from an independent tester's runs (both lanes, per-sample), not from summary tables. +## Your serving layer can decide the answer, and the default settings hide it + +Established independently on three stacks and two serving layers (llama.cpp and vLLM) in one day, which is why it belongs here rather than in any one model's guide. **Identical requests can produce different results depending on what the server processed before them.** + +Three known channels, all on the serving side rather than the model: + +1. **Concurrent batched decoding.** Sequences decoding in one fused batch change batch shape, so reduction order, so floating-point rounding, so logits. At temperature 0 the sampler is argmax, so a near-tie flips and the divergence compounds through the KV cache. Measured: idle produced 1 distinct output in 4 runs; at 2 concurrent requests, **4 distinct in 6**, all differing from the idle baseline. Serialized (`-np 1`), **0 of 6** differed. +2. **Prefix cache reuse, enabled by default.** The same request served from a fresh full prefill is byte-stable across repeats; served from a warm prefix it returns different bytes. No restart, no concurrency, stock flags. +3. **Setting the disable flag is not sufficient.** On a second stack with the cache flag explicitly off, temp 0, single slot, the cold path was stable 3/3 and the warm path still diverged. + +**It moves verdicts, not just bytes.** On a third stack, prefix-cache coverage tracked `finish_reason` with no overlap: every run at or below 79% cache hit returned `length` and ran to the cap generating invented content; every run at 99.8%+ returned `stop` with a correct short answer. That is a behavioral outcome flipping, not a token-level tie. + +**What it invalidates.** Any A/B whose arms ran in separate runs, or sequentially against a shared server, is carrying this. A published reasoning-depth result was retracted on exactly these grounds: a large apparent effect measured across runs went flat (all pairwise p >= 0.13) once the arms were interleaved inside a single driver. + +**The protocol that survives it:** +- **Interleave arms inside one run.** Never block them, never compare against a previously published cell. +- **Flush the slot between samples** rather than trusting the disable flag, and **run a cold control proving the flush actually isolates on your lane**. +- **Report four things, not one**: prefix-reuse setting, concurrency setting, flush procedure, and the isolation control. +- Prefer `-np 1` when you need determinism more than throughput. + +**The judgment rule that came out of it, which generalizes further:** when a new noise source turns up, the question for each existing claim is whether it has a mechanism the noise cannot reach. One tester kept a finding with a deterministic template-render basis and withdrew a marginal p=0.041 cell from the same study, on the same day, from the same new evidence. Treating a new noise source as fatal to everything, or to nothing, are both wrong. + ## Sources Synthesized from internal behavioral testing across several models (Ornith-1.0-35B, Ornith-1.0-9B, Qwopus-Coder thinking ablation, Qwen3.6-27B quantization comparison) plus the general finding that externalizing