Reconcile the guide with the #10 thread: two retractions at source, one new confound - #19
Reconcile the guide with the #10 thread: two retractions at source, one new confound#19TheTom wants to merge 2 commits into
Conversation
…found The thread moved a long way in two days and several things I published are no longer supported. Reconciling rather than letting the guide drift. RETRACTED, and it was mine to carry: the reasoning-depth collapse (3,536 to 745 to 282) that I merged on 07-26 as axis 2 of the two-axis model. Its author withdrew it after an in-run interleaved re-test at C7 (200 turns, five arms, seeded shuffle) found depth among fired turns is flat: medians 858/806/721/ 933/474, all pairwise Mann-Whitney p >= 0.13, with the tools arm carrying the HIGHEST median. The published collapse was a cross-run read. The real mechanism replaces it and is better: turns exiting via tool_call carry short pre-call reasoning (462 and 136 tokens) against 1,293 and 847 for turns that answer directly. Half of tool-arm turns exit to a call, so the pooled median collapses arithmetically. The episode is truncated at the tool boundary, not suppressed. Reproduced on a second stack at the opposite end of the quant axis (UD-Q2_K_XL/llama.cpp/P100: 812 vs 2,954 chars, 3.6x, inside the other stack's range). Depth columns in the curve tables now carry a do-not-compare caveat. NEW 5h, and the most portable thing in the guide: your serving layer can decide the answer. Concurrent batched decoding flips near-ties at temp 0 (4 distinct outputs of 6 at 2 concurrent, 0 of 6 serialized); prefix-cache reuse is on by default and changes output bytes; and setting the disable flag is NOT sufficient, shown on a second stack with the flag explicitly off. It moves verdicts rather than only bytes: on a third stack cache coverage tracked finish_reason with zero overlap. That lands on us. Our HumanEval+ A/B ran -np 4, which is exactly the concurrency channel, at K=1 with no seed replication to absorb it. The +-2.5 point delta is now stated as directionally consistent with the other controlled run, not as a precise magnitude. Also folded: - multi-turn firing is a harness property, not model attenuation. Stripped history 0/10 vs preserved 10/10 on identical transcripts; 0/56 vs 14/14 at larger n with a deterministic template-render basis. "Thinking ON" in a loop only means what it says if the harness resends prior reasoning. - task shape at scale: a 100% codegen cell at n=492 fired 90.4%, so "coding suppresses thinking" is not supported. Consistent with the grid once conditioned, since its pooled 60-72% is held down by summarization at 0/10. - apparatus costs score, not obviously capability, in the narrower form both the author and reviewer settled on: capability on the 106 routed samples is unmeasured, since they were selected by the model's own behaviour. patterns.md gains the serving-state entry, which is not Laguna-specific, and the cap-hit rule is corrected to "no extractable output", never finish_reason alone, after one tester found a cap-hit that passed its tests twice. Card and PNG regenerated to match.
Three stacks now. The two that held temperature identical across arms both came back flat-to-negative (-1.02 and -2.44 on HumanEval+); the only arm showing a gain ran its two legs at different temperatures.
|
Drafted with AI assistance (Claude); reviewed and submitted by @Blackwellboy. Reviewed against the source data rather than against the summary. Taking your The thing you asked me to confirmConfirmed. The tools arm carried the highest median, not the lowest. Every figure in your section 1 matches the run:
The cold-versus-warm verdict result is also right: four runs at or below Four things to change, in descending order of how much they matter1. The card states this more strongly than the run supportsThe card now reads "Depth is NOT suppressed by apparatus". That is stronger What the run supports is in its own limitations section: n-fired is 19 to 27 Suggested wording, close to what you already have: "the 3536 to 282 collapse 2. The 1,293 and 847 comparators are n=4 cells, and that is our faultThis is the one I would most want fixed, and the error is ours before it is Your section 2 mechanism reads "short pre-call reasoning (medians 462 and 136 So the contrast as stated is a 20-turn median against a 4-turn median. The Why you could not have known: our per-finish table states 3. The replacement mechanism drops one of the three we gaveOur writeup attributes the original collapse to three things together: The middle one does independent work and it is worth keeping, because it bites As written, your version is cleaner than the data. One sentence restores it. 4.
|
|
Tom, had Opus double check everything. One clarification specifically:
§5h states the channels correctly, and yes — he attributes channel 2 to the upstream default, which is right. Worth confirming one thing explicitly in your review, because it's the difference between a llama.cpp property and a buun-fork property: the channel-1 numbers he quotes (1 of 4 idle, 4 of 6 concurrent) are our genuine-upstream figures, not the fork's (which were 1 of 5 and 3 of 6). He picked the stronger, more general ones. Correct, but worth saying so it can't drift later. Also correct: he omits our third channel (VBR first-use slot) from §5h. That one is fork-specific and doesn't belong in a portable section. |
|
Pre-registered: does the history-strip drive the firing decay in a real production agentic coder? (A/B running, numbers to follow) #19 point 3 makes the multi-turn attenuation a harness property: strip prior reasoning from rendered history and firing goes 0/10, preserve it and 10/10; I measured the same at 0/56 vs 14/14 on a deterministic The harness. Foreman (our open LLMKube coder) runs Laguna S2.1 on Strix over real 70–120-turn issue-fix tasks, temp 0.2, Exploratory scaffolding (not the claim). Across 7 stored runs / 512 turns in the stripped default, firing (reasoning > ~20 tok) is ~8% overall, median 0, and — the reason I'm testing this — it decays by run position: 37% in the first decile → ~0% by mid-run, then flat. That is the attenuation, as a curve, in a confirmed-stripped harness. But it is confounded: turn index co-varies with context growth and task phase (early = explore/read → think; late = mechanical edits → don't), and the agent also windows old turns ( The pre-registered A/B (isolates the resend effect). Same issue, same intent, same serving config, N=4 per arm:
Primary outcome (frozen before results): firing-rate-vs-run-position curve, S vs P. Prediction from the history-rendering model: P sustains firing later into the run; S decays. Secondary: overall firing rate, reasoning-length slope vs turn index (mixed-effects, run as random slope). Rigor gates. A/A control FIRST — two Arm-S runs; if their decay curves differ by more than the S-vs-P effect we'd claim, the run-to-run noise floor swamps it and I don't report. Threshold, tokenization, and the What I'm asking before the numbers land: is the strip-toggle the right isolation, or is there a confound in running this on a coding task specifically that I should control for (given point 4 — coding fires ~90% at n=492)? The A/A is running now; I'll post S-vs-P when it clears its own gate. |
|
Answering before your numbers land, since that is worth more than answering after. The strip-toggle is the right isolation. Same binary, same env, one gated flag, arms differing only in resend is exactly the comparison that separates "the model needs to see its prior reasoning" from "something else about long runs". The A/A control first is the right call and it is the part most people skip. The coding-task confound you suspect is real, and it is ours to disclose rather than yours to have caught. It comes out of our own published figure, the one in #10: coding fires at roughly 90% at n=492. That is the highest firing rate of any task type we have measured. So a coding task starts near the firing ceiling, which means a decay curve measured on coding has somewhere to fall that a lower-firing task type does not. Some part of your 37% to ~0% could be regression toward the task-type baseline as the run shifts from explore-and-read work to mechanical edits, rather than the history strip doing it. We put that number into circulation without noting it constrains exactly this kind of design. If your A/B had landed and someone read the decay as pure history-rendering, the figure that made that reading plausible would have been ours. Two ways to separate it, cheapest last. The clean separator is task type as a second factor: run the same S-vs-P contrast on a task class that does not start at the ceiling, and see whether the resend effect survives. If P sustains firing on both, the mechanism is history-rendering. If P only sustains on the low-firing class, the coding decay was substantially phase. At N=4 per arm that may not be affordable, and you should not spend your power on it. The cheap version costs nothing and does not touch your pre-registration: report the firing curve split by phase rather than only by run position. You have already identified that phase co-varies with turn index, and That is a reporting change, not a design change, so it stays inside what you pre-registered. One thing worth freezing now if you have not. Your exploratory pass has median 0 firing with ~8% overall, which means the distribution is mostly zeros with a tail. A mean-based curve on that will be noisy at N=4. If you have not fixed it already, decide before you look whether the primary is the firing rate per position bin or the proportion of runs still firing at position k, because those two can disagree and picking after the fact is where a real effect turns into a chosen one. And your null is a real result. If preserve does not arrest the decay, that says the decay is context or phase and not history-rendering, which would put a bound on our #19 point 3 that we do not currently have. We would publish that with the same weight. Our own evidence for the mechanism is a probe-scale 0/10 versus 10/10 and your 0/56 versus 14/14 on |
|
This is the reason to pre-register before the numbers, and I'm adopting both, frozen now while the A/A is still running and before I've looked at any firing curve from it. Phase split, adopted as the primary confound control. Each turn gets classified EXPLORE (before the run's first edit tool call) vs EDIT (from the first edit onward), and I report the S-vs-P firing gap within the edit phase, where both arms are doing the same mechanical work. If the gap survives there, it is net of the explore-to-edit regression your n=492 ceiling introduces; if it only lives in the explore phase, I'll say the coding decay was substantially phase. That your 90% figure is what makes this confound bite, and that you flagged it against your own published number, is the version of this I want on the record. Primary metric frozen. Given the mostly-zeros distribution you're right to worry about a mean curve at N=4. Frozen: primary is per-turn fired(0/1) in a mixed-effects logistic, Both are analysis choices, not design changes, so N stays 4 per arm and the pre-registration holds. On the null: agreed, and that's the outcome I'd find most useful to us both. A long-horizon production agent putting a bound on point 3 is worth as much as confirming it, and it's the surface neither the 0/10 probe nor the /apply-template 0/56 reaches. Numbers when the A/A clears its gate. |
|
Numbers, and they are a null: the A/A control did not clear its own gate, so there is no strip-toggle A/B to report. Here is exactly what happened and what it does and does not mean. I ran the controlled task (#1311, a well-specified GPUQuota fix) four times as Arm S, two with MCP off and two with MCP on. Per-turn firing (reasoning > ~20 tok) was 2 to 4% in every run, and the largest reasoning length in any single turn, across all four runs, was ~112 characters. No turn over 500 characters, anywhere. The big-reasoning tail that produced my exploratory 37%-to-~0% decay is simply absent here, so there is no decay curve to measure and the resend contrast has nothing to act on. Gate not passed, Arm P not run. Two things fall out, and I want to be precise about both. What this does establish: my exploratory decay does not reproduce in a controlled production run. It lived in the big-reasoning turns of a heterogeneous batch, and those turns are the model over-generating reasoning until it hits max_tokens on harder or vaguer tasks. Aggregating them across mixed task types manufactured a decay. It is the verbosity phenomenon, not a clean per-turn history-rendering effect. And MCP is not the driver: MCP-on and MCP-off were indistinguishable, so my earlier guess there was wrong. What it does not establish: it does not touch point 3 or the /apply-template 0/56 vs 14/14. That result is about whether stripping suppresses firing when firing would occur, on a deterministic basis. Here firing barely occurs at all, regardless of the strip, because a well-specified fix makes the model go near-pure-action, tool calls with almost no reasoning and no narration. The production surface I tested does not supply the reasoning the strip effect needs, so I cannot offer a production test of point 3 with this task. I can only report that this task class does not elicit the phenomenon. @Blackwellboy, both of your pre-registration calls were right, and I am glad I froze them before looking. The phase split had nothing to bite on because the whole curve was near zero, and the metric freeze is exactly what stops me from fishing a decay out of one or two firing events per run. And your framing that a null bounding point 3 is worth as much as confirming it is why I am posting this rather than quietly re-scoping. The honest next step, if it is worth the runs, is a task class that reliably elicits sustained reasoning, harder or more open-ended than a spelled-out fix, and running the strip contrast there. That is a new design, not this one. I will not claim a production bound on point 3 until I have a task where the reasoning actually shows up. |
4 participants
The #10 thread moved a long way over two days. Several things this guide publishes are no longer supported, including one I merged myself, so this brings the repo back in line rather than letting it drift.
Nothing here is new testing. It is reconciliation, and every number is attributed to whoever measured it.
1. RETRACTED: the reasoning-depth collapse (3,536 to 745 to 282)
This is the one that matters most, because I merged it in #12 as axis 2 of the two-axis model, and its author has since withdrawn it.
@Blackwellboy re-ran it properly: a 200-turn in-run interleaved study at C7, five arms, four tasks, single driver, seeded shuffle. Depth among fired turns is flat: medians 858 / 806 / 721 / 933 / 474 estimated reasoning tokens, all pairwise Mann-Whitney p >= 0.13, and the tools arm carried the highest median, not the lowest. The published collapse was a cross-run comparison, and cross-run cells on that lane drift enough to manufacture it.
The replacement mechanism is better than the thing it replaces. A turn that ends in a
tool_callcarries short pre-call reasoning (medians 462 and 136 tokens) against 1,293 and 847 for turns that answer directly. Exactly half of tool-arm turns exit to a call, so the pooled median collapses arithmetically. The reasoning episode is truncated at the tool boundary, not suppressed.Independently reproduced at the opposite end of the quant axis by @apollo-mg (UD-Q2_K_XL / llama.cpp / P100 sm_60): turn-1 tool exits at 812 chars against answering turns at 2,954, a 3.6x ratio sitting inside the 2.8x-6.2x range measured on vLLM/GB10.
The
med think-tokcolumn in both §2 curve tables now carries an explicit do-not-compare caveat rather than being deleted, so the record of what the cross-run read looked like survives.2. NEW §5h: your serving layer can decide the answer
The most portable finding in the guide and the least Laguna-specific. Three stacks, two serving layers, one day:
-np 1, 0 of 6.common/common.h:622) and changes output bytes with no restart and no concurrency.cache_prompt: falseexplicitly set, temp 0, single slot, and the warm path still diverged while the cold path was stable 3/3.finish_reasonwith zero overlap. Every run at or below 79% hit returnedlengthand ran to the cap inventing content; every run at 99.8%+ returnedstopwith a correct 7-to-9-token answer.This lands on our own numbers and the section says so. Our HumanEval+ A/B ran
-np 4, which is exactly the concurrency channel, at K=1 with no seed replication to absorb it. Our +-2.5 point delta is now stated as directionally consistent with the other controlled run rather than as a precise magnitude.3. Multi-turn firing is a harness property, not model attenuation
The "attenuation in long agentic loops" this guide reported is largely history rendering. On identical transcripts, stripping prior reasoning from rendered history fired 0/10; preserving it fired 10/10. @Defilan measured the same at larger n, 0/56 against 14/14, with a deterministic
/apply-templatebasis.Consequence, now stated in the minimize-thinking list: "thinking ON" in an agent loop only means what it says if your harness resends prior reasoning. Otherwise you are comparing a mostly-suppressed ON arm against OFF and attributing the delta to the wrong thing.
4. Task shape at scale
@apollo-mg ran a 100% codegen cell with agent prompt and tool schemas at n=492: thinking fired 445/492 (90.4%). So "coding-shaped tasks suppress thinking" is not supported.
That is consistent with the grid rather than against it, once conditioned: the grid's pooled 60-72% at C7/C8 is a task-mix number held down by summarization (0/10 everywhere) and the reasoning task (4/10), while its code row is 10/10 at both C7 and C8. The guide now says to quote the conditioned number, never the pooled one, when the question is about coding.
He also corrected his own 15-problem K=1 2x2 in public after n=492 shrank the effect it had sized. Recorded, because it is the same lesson as the prompt-identity weight limit already in §2.
5. Apparatus costs score, in the narrower form that survived review
Attaching an agent prompt plus tool schemas moved pass@1 from 90.85% to 71.95% (-18.9 points) while wrong answers moved only 30 to 31: 106 of 492 samples routed to a tool call instead of answering, which a single-turn harness scores as failure.
Both the author of that cell and its reviewer then retracted the stronger reading, and the guide carries the narrow one: capability on those 106 is unmeasured, because they were selected by the model's own behaviour under the apparatus and never produced an answer.
6.
patterns.mdfinish_reasonalone." That is the form both testers converged on after @apollo-mg found a cap-hit that passed its tests twice, and after @Blackwellboy corrected his own restatement of it.7. Quickstart
Row 1's "under review in #10" pointer is resolved: ON is not better for single-turn codegen. Three stacks, and the two that held temperature identical across arms both came back flat-to-negative.
Review asks
Still open, and deliberately not written as settled
The fed-back-tools arm you two have pre-registered is the discriminator for whether routing prevents loops or relabels them, and nothing in this PR pre-empts it. Card and PNG regenerated to match;
check_guides.pygreen.